Restate-Kafka Quickstart

In this guide, you will learn how to connect your Restate service handler to a Kafka topic. Restate lets you connect any handler to a Kafka topic, and invoke the handler for each event that arrives on the topic. This way, you can use Restate to process events in a lightweight, flexible, transactional way.

Get the Greeter service template and run the service

You can choose any of the SDK languages for this quickstart. You do not need to adapt the handler code to be able to read from Kafka. Any handler can be connected to Kafka.

Running Restate Server

Start the Restate Server:

restate-server

Let the Restate Server know about the Greeter service by registering it:

restate deployments register localhost:9080

Start Kafka and the Kafka ingress integration

Let’s start a Kafka cluster with a single broker, create a topic named greetings, and run the Kafka ingress integration next to it.You can run your Kafka cluster in your preferred way. Here, we will use Docker Compose.Create a docker-compose.yaml file with the following content:

docker-compose.yaml

services:
    broker:
        image: confluentinc/cp-kafka:7.5.0
        container_name: broker
        ports:
            - "9092:9092"
            - "9101:9101"
        environment:
            KAFKA_BROKER_ID: 1
            KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
            KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
            KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
            KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
            KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
            KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
            KAFKA_PROCESS_ROLES: broker,controller
            KAFKA_NODE_ID: 1
            KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker:29093
            KAFKA_LISTENERS: PLAINTEXT://broker:29092,CONTROLLER://broker:29093,PLAINTEXT_HOST://0.0.0.0:9092
            KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
            KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
            KAFKA_LOG_DIRS: /tmp/kraft-combined-logs
            CLUSTER_ID: MkU3OEVBNTcwNTJENDM2Qk

    init-kafka:
        image: confluentinc/cp-kafka:7.5.0
        depends_on:
            - broker
        entrypoint: [ '/bin/sh', '-c' ]
        command: |
            "# blocks until kafka is reachable
            kafka-topics --bootstrap-server broker:29092 --list
            echo -e 'Creating kafka topics'
            kafka-topics --bootstrap-server broker:29092 --create --if-not-exists --topic greetings --replication-factor 1 --partitions 1

            echo -e 'Successfully created the following topics:'
            kafka-topics --bootstrap-server broker:29092 --list"

    restate-kafka-integration:
        image: ghcr.io/restatedev/ingress-integration-kafka:latest
        depends_on:
            init-kafka:
                condition: service_completed_successfully
        extra_hosts:
            - "host.docker.internal:host-gateway"
        environment:
            KAFKA_BOOTSTRAP_SERVERS: broker:29092
            KAFKA_GROUP_ID: greetings-to-restate
            KAFKA_TOPICS: greetings
            KAFKA_AUTO_OFFSET_RESET: earliest
            RESTATE_INGRESS_URL: http://host.docker.internal:8080
            RESTATE_RECORD_MAPPER_SERVICE: Greeter
            RESTATE_RECORD_MAPPER_HANDLER: greet

The restate-kafka-integration container consumes the greetings topic and invokes Greeter/greet for each record. It reaches the Restate Server running on your machine via host.docker.internal.Start the containers:

docker compose up

Invoke the handler by publishing an event

Create a Kafka producer and publish an event to the greetings topic:

docker exec -it broker kafka-console-producer --bootstrap-server broker:29092 --topic greetings

Then type a message and press enter. The greeter takes a String name as an input:

{"name":"Sarah"}

For some greeter templates, you might need to send just the name as "Sarah".

We invoked a Restate service over Kafka

You now see your handler getting invoked.The way this worked is that the Kafka ingress integration consumed the record from the topic and sent it to Restate, which durably persisted it, similar to what it does for HTTP invocations.It then pushed the message to the handler, as opposed to the pull mechanism that you would have with a Kafka consumer.If the handler fails, Restate will retry the request until it eventually succeeds.Have a look at the Kafka ingress integration docs to learn how to map Kafka records to Restate invocations, connect to secured clusters, and read record metadata in your handler. Have a look at the Event Processing use case page to learn about what you can do with Restate and event processing.

Event processing use cases: Have a look at examples of how Restate gives you lightweight, transactional event processing.
Have a look at the examples. There, you can find examples on Durable Execution and event processing.
Kafka ingress integration reference
Blog post on Kafka integration

Recipes

Development Guides

Deployment Guides

Integrations

Restate-Kafka Quickstart

​Related resources

Related resources