# Cluster state endpoint
Source: https://docs.restate.dev/admin-api/cluster_health/cluster-state-endpoint

schemas/openapi-admin.json get /cluster-health



# Delete deployment
Source: https://docs.restate.dev/admin-api/deployment/delete-deployment

schemas/openapi-admin.json delete /deployments/{deployment}
Delete a deployment. Currently, only forced deletions are supported.
**Use with caution**: forcing a deployment deletion can break in-flight invocations.



# Get deployment
Source: https://docs.restate.dev/admin-api/deployment/get-deployment

schemas/openapi-admin.json get /deployments/{deployment}
Returns detailed information about a registered deployment, including deployment metadata and the services it exposes.



# List deployments
Source: https://docs.restate.dev/admin-api/deployment/list-deployments

schemas/openapi-admin.json get /deployments
Returns a list of all registered deployments, including their endpoints and associated services.



# Register deployment
Source: https://docs.restate.dev/admin-api/deployment/register-deployment

schemas/openapi-admin.json post /deployments
Registers a new deployment (HTTP or Lambda). Restate will invoke the endpoint to discover available services and handlers,
and make them available for invocation. For more information, see the [deployment documentation](https://docs.restate.dev/services/versioning#registering-a-deployment).



# Update deployment
Source: https://docs.restate.dev/admin-api/deployment/update-deployment

schemas/openapi-admin.json patch /deployments/{deployment}
Updates an existing deployment configuration, such as the endpoint address or invocation headers.
By default, service schemas are not re-discovered. Set `overwrite: true` to trigger re-discovery.



# Health check endpoint
Source: https://docs.restate.dev/admin-api/health/health-check-endpoint

schemas/openapi-admin.json get /health



# Query the system and service state by using SQL.
Source: https://docs.restate.dev/admin-api/introspection/query-the-system-and-service-state-by-using-sql

schemas/openapi-admin.json post /query



# Cancel an invocation
Source: https://docs.restate.dev/admin-api/invocation/cancel-an-invocation

schemas/openapi-admin.json patch /invocations/{invocation_id}/cancel
Gracefully cancels an invocation. The invocation is terminated, but its progress is persisted, allowing consistency guarantees to be maintained.
For more information, see the [cancellation documentation](https://docs.restate.dev/services/invocation/managing-invocations#cancel).



# Delete an invocation
Source: https://docs.restate.dev/admin-api/invocation/delete-an-invocation

schemas/openapi-admin.json delete /invocations/{invocation_id}
Use kill_invocation/cancel_invocation/purge_invocation instead.



# Kill an invocation
Source: https://docs.restate.dev/admin-api/invocation/kill-an-invocation

schemas/openapi-admin.json patch /invocations/{invocation_id}/kill
Forcefully terminates an invocation. **Warning**: This operation does not guarantee consistency for virtual object instance state,
in-flight invocations to other services, or other side effects. Use with caution.
For more information, see the [cancellation documentation](https://docs.restate.dev/services/invocation/managing-invocations#kill).



# Pause an invocation
Source: https://docs.restate.dev/admin-api/invocation/pause-an-invocation

schemas/openapi-admin.json patch /invocations/{invocation_id}/pause



# Purge a completed invocation
Source: https://docs.restate.dev/admin-api/invocation/purge-a-completed-invocation

schemas/openapi-admin.json patch /invocations/{invocation_id}/purge
Deletes all state associated with a completed invocation, including its journal and metadata.
This operation only applies to invocations that have already completed. For more information,
see the [purging documentation](https://docs.restate.dev/services/invocation/managing-invocations#purge).



# Purge invocation journal
Source: https://docs.restate.dev/admin-api/invocation/purge-invocation-journal

schemas/openapi-admin.json patch /invocations/{invocation_id}/purge-journal
Deletes only the journal entries for a completed invocation, while retaining its metadata.
This operation only applies to invocations that have already completed.



# Restart invocation as new
Source: https://docs.restate.dev/admin-api/invocation/restart-invocation-as-new

schemas/openapi-admin.json patch /invocations/{invocation_id}/restart-as-new
Creates a new invocation from a completed invocation, optionally copying partial progress from the original invocation's journal.
The new invocation will have a different invocation ID. Use the `from` parameter to specify how much of the original journal to preserve.



# Resume an invocation
Source: https://docs.restate.dev/admin-api/invocation/resume-an-invocation

schemas/openapi-admin.json patch /invocations/{invocation_id}/resume
Resumes a paused or suspended invocation. If the invocation is backing off due to a retry, this will immediately trigger the retry.
Optionally, you can change the deployment ID that will be used when the invocation resumes. For more information see [resume documentation](https://docs.restate.dev/services/invocation/managing-invocations#resume)



# Get service
Source: https://docs.restate.dev/admin-api/service/get-service

schemas/openapi-admin.json get /services/{service}
Returns detailed metadata about a specific service, including its type, handlers, and configuration settings.



# Get service OpenAPI definition
Source: https://docs.restate.dev/admin-api/service/get-service-openapi-definition

schemas/openapi-admin.json get /services/{service}/openapi
Returns the OpenAPI 3.1 specification for the service, describing all handlers and their request/response schemas.



# List services
Source: https://docs.restate.dev/admin-api/service/list-services

schemas/openapi-admin.json get /services
Returns a list of all registered services, including their metadata and configuration.



# Modify service configuration
Source: https://docs.restate.dev/admin-api/service/modify-service-configuration

schemas/openapi-admin.json patch /services/{service}
Updates the configuration of a registered service, such as public visibility, retention policies, and timeout settings.
Note: Service re-discovery will update these settings based on the service endpoint configuration.



# Modify service state
Source: https://docs.restate.dev/admin-api/service/modify-service-state

schemas/openapi-admin.json post /services/{service}/state
Modifies the K/V state of a Virtual Object. For a detailed description of this API and how to use it, see the [state documentation](https://docs.restate.dev/operate/invocation#modifying-service-state).



# Get service handler
Source: https://docs.restate.dev/admin-api/service_handler/get-service-handler

schemas/openapi-admin.json get /services/{service}/handlers/{handler}
Returns detailed metadata about a specific handler within a service, including its input/output types and handler type.



# List service handlers
Source: https://docs.restate.dev/admin-api/service_handler/list-service-handlers

schemas/openapi-admin.json get /services/{service}/handlers
Returns a list of all handlers (methods) available in the specified service.



# Create subscription
Source: https://docs.restate.dev/admin-api/subscription/create-subscription

schemas/openapi-admin.json post /subscriptions
Creates a new subscription that connects an event source (e.g., a Kafka topic) to a Restate service handler.
For more information, see the [subscription documentation](https://docs.restate.dev/operate/invocation#managing-kafka-subscriptions).



# Delete subscription
Source: https://docs.restate.dev/admin-api/subscription/delete-subscription

schemas/openapi-admin.json delete /subscriptions/{subscription}
Deletes a subscription. This will stop events from the source from being forwarded to the sink.



# Get subscription
Source: https://docs.restate.dev/admin-api/subscription/get-subscription

schemas/openapi-admin.json get /subscriptions/{subscription}
Returns the details of a specific subscription, including its source, sink, and configuration options.



# List subscriptions
Source: https://docs.restate.dev/admin-api/subscription/list-subscriptions

schemas/openapi-admin.json get /subscriptions
Returns a list of all registered subscriptions, optionally filtered by source or sink.



# Get version information
Source: https://docs.restate.dev/admin-api/version/get-version-information

schemas/openapi-admin.json get /version
Returns the server version, supported Admin API versions, and the advertised ingress endpoint.



# AI Agent Quickstart
Source: https://docs.restate.dev/ai-quickstart

Build and run your first AI agent with Restate and popular AI SDKs

This guide takes you through building your first AI agent with Restate and popular AI SDKs.

We will run a simple weather agent that can answer questions about the weather using durable execution to ensure reliability.

<img alt="AI Agent Quickstart" />

Select your AI SDK:

<Tabs>
  <Tab title="TypeScript + Vercel AI" icon={"/img/languages/typescript.svg"}>
    <Info>
      **Prerequisites**:

      * [Node.js](https://nodejs.org/en/) >= v20
      * OpenAI API key (get one at [OpenAI](https://platform.openai.com/))
    </Info>

    <Steps>
      <Step title="Install Restate Server & CLI">
        Restate is a single self-contained binary. No external dependencies needed.

        <Tabs>
          <Tab title="Homebrew">
            ```shell theme={null}
            brew install restatedev/tap/restate-server restatedev/tap/restate
            ```

            Start the server:

            ```shell theme={null}
            restate-server
            ```
          </Tab>

          <Tab title="Download binaries">
            Download prebuilt binaries from the [releases page](https://github.com/restatedev/restate/releases/latest):

            <CodeGroup>
              ```shell MacOS-x64 theme={null}
              BIN=/usr/local/bin && RESTATE_PLATFORM=x86_64-apple-darwin && \
              curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
              tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
              tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
              chmod +x restate restate-server && \
              sudo mv restate $BIN && \
              sudo mv restate-server $BIN
              ```

              ```shell MacOS-arm64 theme={null}
              BIN=/usr/local/bin && RESTATE_PLATFORM=aarch64-apple-darwin && \
              curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
              tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
              tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
              chmod +x restate restate-server && \
              sudo mv restate $BIN && \
              sudo mv restate-server $BIN
              ```

              ```shell Linux-x64 theme={null}
              BIN=$HOME/.local/bin && RESTATE_PLATFORM=x86_64-unknown-linux-musl && \
              curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
              tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
              tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
              chmod +x restate restate-server && \
              mv restate $BIN && \
              mv restate-server $BIN
              ```

              ```shell Linux-arm64 theme={null}
              BIN=$HOME/.local/bin && RESTATE_PLATFORM=aarch64-unknown-linux-musl && \
              curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
              tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
              tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
              chmod +x restate restate-server && \
              mv restate $BIN && \
              mv restate-server $BIN
              ```
            </CodeGroup>

            Start the server:

            ```shell theme={null}
            restate-server
            ```
          </Tab>

          <Tab title="npm">
            ```shell theme={null}
            npm install --global @restatedev/restate-server@latest @restatedev/restate@latest
            ```

            Start the server:

            ```shell theme={null}
            restate-server
            ```
          </Tab>

          <Tab title="Docker">
            Run the Restate Server:

            ```shell theme={null}
            docker run --name restate_dev --rm \
            -p 8080:8080 -p 9070:9070 -p 9071:9071 \
            --add-host=host.docker.internal:host-gateway \
            docker.restate.dev/restatedev/restate:latest
            ```

            Run CLI commands:

            ```shell theme={null}
            docker run -it --network=host \
            docker.restate.dev/restatedev/restate-cli:latest \
            invocations ls
            ```

            Replace `invocations ls` with any CLI subcommand.
          </Tab>
        </Tabs>

        You can find the Restate UI running on port 9070 (`http://localhost:9070`) after starting the Restate Server.
      </Step>

      <Step title="Get the AI Agent template">
        Get the weather agent template for the [Vercel AI SDK](https://ai-sdk.dev/docs/foundations/overview) and Restate:

        ```shell theme={null}
        git clone https://github.com/restatedev/ai-examples.git &&
        cd ai-examples/vercel-ai/template &&
        npm install
        ```

        <GitHubLink />
      </Step>

      <Step title="Run the AI Agent service">
        Export your OpenAI key and run the agent:

        ```shell theme={null}
        export OPENAI_API_KEY=your_openai_api_key_here
        npm run dev
        ```

        The weather agent is now listening on port 9080.
      </Step>

      <Step title="Register the service">
        Tell Restate where the service is running (`http://localhost:9080`), so Restate can discover and register the services and handlers behind this endpoint.
        You can do this via the UI (`http://localhost:9070`) or via:

        <CodeGroup>
          ```shell CLI theme={null}
          restate deployments register http://localhost:9080
          ```

          ```shell curl theme={null}
          curl localhost:9070/deployments --json '{"uri": "http://localhost:9080"}'
          ```
        </CodeGroup>

        <Expandable title="Output">
          <CodeGroup>
            ```shell CLI theme={null}
            ❯ SERVICES THAT WILL BE ADDED:
            - agent
            Type: Service
            HANDLER  INPUT                                     OUTPUT
            run      value of content-type 'application/json'  value of content-type 'application/json'


            ✔ Are you sure you want to apply those changes? · yes
            ✅ DEPLOYMENT:
            SERVICE  REV
            agent    1
            ```

            ```shell curl theme={null}
            {
                "id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                "services": [
                    {
                        "name": "Agent",
                        "handlers": [
                            {
                                "name": "run",
                                "ty": "Shared",
                                "input_description": "one of [\"none\", \"value of content-type 'application/json'\"]",
                                "output_description": "value of content-type 'application/json'"
                            }
                        ],
                        "ty": "Service",
                        "deployment_id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                        "revision": 1,
                        "public": true,
                        "idempotency_retention": "1day"
                    }
                ]
            }
            ```
          </CodeGroup>
        </Expandable>

        If you run Restate with Docker, register `http://host.docker.internal:9080` instead of `http://localhost:9080`.

        <Accordion title="Restate Cloud">
          When using [Restate Cloud](https://restate.dev/cloud), your service must be accessible over the public internet so Restate can invoke it.
          If you want to develop with a local service, you can expose it using our [tunnel](/deploy/server/cloud/#registering-restate-services-with-your-environment) feature.
        </Accordion>
      </Step>

      <Step title="Send weather requests to the AI Agent">
        Invoke the agent via the Restate UI playground: go to `http://localhost:9070`, click on your service and then on playground.

        <Frame>
          <img alt="Restate UI Playground" />
        </Frame>

        Or invoke via `curl`:

        ```shell theme={null}
        curl localhost:8080/agent/run --json '"What is the weather in Detroit?"'
        ```

        Output: `The weather in Detroit is currently 17°C with misty conditions.`.
      </Step>

      <Step title="Congratulations, you just ran a Durable AI Agent!">
        The agent you just invoked uses Durable Execution to make agents resilient to failures. Restate persisted all LLM calls and tool execution steps, so if anything fails, the agent can resume exactly where it left off.

        We did this by using Restate's `durableCalls` middleware to persist LLM responses and using [Restate Context actions](/foundations/actions) (e.g. `ctx.run`) to make the tool executions resilient:

        ```ts expandable {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/vercel-ai/template/src/app.ts?collapse_imports"}  theme={null}
        async function weatherAgent(restate: restate.Context, prompt: string) {
          // The durableCalls middleware persists each LLM response in Restate,
          // so they can be restored on retries without re-calling the LLM
          const model = wrapLanguageModel({
            model: openai("gpt-4o"),
            middleware: durableCalls(restate, { maxRetryAttempts: 3 }),
          });

          const { text } = await generateText({
            model,
            system: "You are a helpful agent that provides weather updates.",
            prompt,
            tools: {
              getWeather: tool({
                description: "Get the current weather for a given city.",
                inputSchema: z.object({ city: z.string() }),
                execute: async ({ city }) => {
                  // call tool wrapped as Restate durable step
                  return await restate.run("get weather", () => fetchWeather(city));
                },
              }),
            },
            stopWhen: [stepCountIs(5)],
            providerOptions: { openai: { parallelToolCalls: false } },
          });

          return text;
        }

        // create a Restate Service as the callable entrypoint
        // for our durable agent function
        const agent = restate.service({
          name: "agent",
          handlers: {
            run: async (ctx: restate.Context, prompt: string) => {
              return weatherAgent(ctx, prompt);
            },
          },
        });

        // Serve the entry-point via an HTTP/2 server
        restate.serve({
          services: [agent],
        });
        ```

        <GitHubLink />

        The Invocations tab of the Restate UI shows us how Restate captured each LLM call and tool step in a journal:

        <Frame>
          <img alt="Restate UI Journal Entries" />
        </Frame>

        <Accordion title="See how a failing tool call is retried">
          Ask about the weather in Denver:

          ```shell theme={null}
          curl localhost:8080/agent/run --json '"What is the weather in Denver?"'
          ```

          You can see in the service logs and in the Restate UI how each LLM call and tool step gets durably executed.
          We can see how the weather tool is currently stuck, because the weather API is down.

          <Frame>
            <img alt="Restate UI Durable Execution" />
          </Frame>

          This was a mimicked failure. To fix the problem, remove the line `failOnDenver` from the `fetchWeather` function in the `utils.ts` file:

          ```ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/vercel-ai/template/src/utils/weather.ts#weather"}  theme={null}
          export async function fetchWeather(city: string) {
            failOnDenver(city);
            const output = await fetchWeatherFromAPI(city);
            return parseWeatherResponse(output);
          }
          ```

          Once you restart the service, the agent resumes at the weather tool call and successfully completes the request.
        </Accordion>

        **Next step:**
        Follow the [Tour of Agents](/tour/vercel-ai-agents) to learn how to build agents with Restate and Vercel AI SDK, OpenAI Agents SDK, etc.
      </Step>
    </Steps>
  </Tab>

  <Tab title="Python + OpenAI" icon={"/img/languages/python.svg"}>
    <Info>
      **Prerequisites**:

      * Python >= v3.12
      * [uv](https://docs.astral.sh/uv/getting-started/installation/)
      * OpenAI API key (get one at [OpenAI](https://platform.openai.com/))
    </Info>

    <iframe title="YouTube video player" />

    <Steps>
      <Step title="Install Restate Server & CLI">
        Restate is a single self-contained binary. No external dependencies needed.

        <Tabs>
          <Tab title="Homebrew">
            ```shell theme={null}
            brew install restatedev/tap/restate-server restatedev/tap/restate
            ```

            Start the server:

            ```shell theme={null}
            restate-server
            ```
          </Tab>

          <Tab title="Download binaries">
            Download prebuilt binaries from the [releases page](https://github.com/restatedev/restate/releases/latest):

            <CodeGroup>
              ```shell MacOS-x64 theme={null}
              BIN=/usr/local/bin && RESTATE_PLATFORM=x86_64-apple-darwin && \
              curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
              tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
              tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
              chmod +x restate restate-server && \
              sudo mv restate $BIN && \
              sudo mv restate-server $BIN
              ```

              ```shell MacOS-arm64 theme={null}
              BIN=/usr/local/bin && RESTATE_PLATFORM=aarch64-apple-darwin && \
              curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
              tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
              tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
              chmod +x restate restate-server && \
              sudo mv restate $BIN && \
              sudo mv restate-server $BIN
              ```

              ```shell Linux-x64 theme={null}
              BIN=$HOME/.local/bin && RESTATE_PLATFORM=x86_64-unknown-linux-musl && \
              curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
              tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
              tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
              chmod +x restate restate-server && \
              mv restate $BIN && \
              mv restate-server $BIN
              ```

              ```shell Linux-arm64 theme={null}
              BIN=$HOME/.local/bin && RESTATE_PLATFORM=aarch64-unknown-linux-musl && \
              curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
              tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
              tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
              chmod +x restate restate-server && \
              mv restate $BIN && \
              mv restate-server $BIN
              ```
            </CodeGroup>

            Start the server:

            ```shell theme={null}
            restate-server
            ```
          </Tab>

          <Tab title="npm">
            ```shell theme={null}
            npm install --global @restatedev/restate-server@latest @restatedev/restate@latest
            ```

            Start the server:

            ```shell theme={null}
            restate-server
            ```
          </Tab>

          <Tab title="Docker">
            Run the Restate Server:

            ```shell theme={null}
            docker run --name restate_dev --rm \
            -p 8080:8080 -p 9070:9070 -p 9071:9071 \
            --add-host=host.docker.internal:host-gateway \
            docker.restate.dev/restatedev/restate:latest
            ```

            Run CLI commands:

            ```shell theme={null}
            docker run -it --network=host \
            docker.restate.dev/restatedev/restate-cli:latest \
            invocations ls
            ```

            Replace `invocations ls` with any CLI subcommand.
          </Tab>
        </Tabs>

        You can find the Restate UI running on port 9070 (`http://localhost:9070`) after starting the Restate Server.
      </Step>

      <Step title="Get the AI Agent template">
        Get the weather agent template for the [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/) and Restate:

        ```shell theme={null}
        git clone https://github.com/restatedev/ai-examples.git &&
        cd ai-examples/openai-agents/template
        ```

        <GitHubLink />
      </Step>

      <Step title="Run the AI Agent service">
        Export your OpenAI key and run the agent:

        ```shell theme={null}
        export OPENAI_API_KEY=your_openai_api_key_here
        uv run .
        ```

        The weather agent is now listening on port 9080.
      </Step>

      <Step title="Register the agent service">
        Tell Restate where the service is running (`http://localhost:9080`), so Restate can discover and register the services and handlers behind this endpoint.
        You can do this via the UI (`http://localhost:9070`) or via:

        <CodeGroup>
          ```shell CLI theme={null}
          restate deployments register http://localhost:9080
          ```

          ```shell curl theme={null}
          curl localhost:9070/deployments --json '{"uri": "http://localhost:9080"}'
          ```
        </CodeGroup>

        <Expandable title="Output">
          <CodeGroup>
            ```shell CLI theme={null}
            ❯ SERVICES THAT WILL BE ADDED:
            - Agent
            Type: Service
            HANDLER  INPUT                                     OUTPUT
            run      value of content-type 'application/json'  value of content-type 'application/json'


            ✔ Are you sure you want to apply those changes? · yes
            ✅ DEPLOYMENT:
            SERVICE  REV
            Agent    1
            ```

            ```shell curl theme={null}
            {
                "id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                "services": [
                    {
                        "name": "Agent",
                        "handlers": [
                            {
                                "name": "run",
                                "ty": "Shared",
                                "input_description": "one of [\"none\", \"value of content-type 'application/json'\"]",
                                "output_description": "value of content-type 'application/json'"
                            }
                        ],
                        "ty": "Service",
                        "deployment_id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                        "revision": 1,
                        "public": true,
                        "idempotency_retention": "1day"
                    }
                ]
            }
            ```
          </CodeGroup>
        </Expandable>

        If you run Restate with Docker, register `http://host.docker.internal:9080` instead of `http://localhost:9080`.

        <Accordion title="Restate Cloud">
          When using [Restate Cloud](https://restate.dev/cloud), your service must be accessible over the public internet so Restate can invoke it.
          If you want to develop with a local service, you can expose it using our [tunnel](/deploy/server/cloud/#registering-restate-services-with-your-environment) feature.
        </Accordion>
      </Step>

      <Step title="Send weather requests to the AI Agent">
        Invoke the agent via the Restate UI playground: go to `http://localhost:9070`, click on your service and then on playground.

        <Frame>
          <img alt="Restate UI Playground" />
        </Frame>

        Or invoke via `curl`:

        ```shell theme={null}
        curl localhost:8080/agent/run --json '{
                "message": "What is the weather in Detroit?"
            }'
        ```

        Output: `The weather in Detroit is currently 17°C with misty conditions.`
      </Step>

      <Step title="Congratulations, you just ran a Durable AI Agent!">
        The agent you just invoked uses Durable Execution to make agents resilient to failures. Restate persisted all LLM calls and tool execution steps, so if anything fails, the agent can resume exactly where it left off.

        We did this by using Restate's `DurableRunner` to persist LLM responses, and by using [Restate Context actions](/foundations/actions) (e.g. `restate_context().run_typed`) to make the tool executions resilient:

        ```python expandable {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/openai-agents/template/agent.py?collapse_imports"}  theme={null}
        @durable_function_tool
        async def get_weather(req: WeatherRequest) -> WeatherResponse:
            """Get the current weather for a given city."""
            # Do durable steps using the Restate context
            return await restate_context().run_typed(
                "Get weather", fetch_weather, city=req.city
            )


        weather_agent = Agent(
            name="WeatherAgent",
            instructions="You are a helpful agent that provides weather updates.",
            tools=[get_weather],
        )


        agent_service = restate.Service("agent")


        @agent_service.handler()
        async def run(_ctx: restate.Context, req: WeatherPrompt) -> str:
            # Runner that persists the agent execution for recoverability
            result = await DurableRunner.run(weather_agent, req.message)
            return result.final_output
        ```

        <GitHubLink />

        The Invocations tab of the Restate UI shows us how Restate captured each LLM call and tool step in a journal:

        <Frame>
          <img alt="Restate UI Journal Entries" />
        </Frame>

        <Accordion title="See how a failing tool call is retried">
          Ask about the weather in Denver:

          ```shell theme={null}
          curl localhost:8080/agent/run --json '{
                  "message": "What is the weather in Denvewr?"
              }'
          ```

          You can see in the Restate UI how each LLM call and tool step gets durably executed.
          We can see how the weather tool is currently stuck, because the weather API is down.

          <Frame>
            <img alt="Restate UI Durable Execution" />
          </Frame>

          This was a mimicked failure. To fix the problem, remove the line `fail_on_denver` from the `fetch_weather` function in the `utils.py` file:

          ```python {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/openai-agents/template/utils/utils.py#weather"}  theme={null}
          async def fetch_weather(city: str) -> WeatherResponse:
              fail_on_denver(city)
              weather_data = await call_weather_api(city)
              return parse_weather_data(weather_data)
          ```

          Once you restart the service, the agent resumes at the weather tool call and successfully completes the request.
        </Accordion>
      </Step>
    </Steps>
  </Tab>
</Tabs>

<Tip>
  [Once you are done with the quickstart, check out how to write Restate apps with AI assistants.](/develop/ai-assistant)
</Tip>


# AI Examples
Source: https://docs.restate.dev/ai/ai-examples





# AI / Agents with Restate
Source: https://docs.restate.dev/ai/index

Learn how to build durable AI agents and integrate popular AI SDKs with Restate.

Restate makes AI agents and workflows innately resilient. Restate provides the reliability infrastructure you need to run AI workloads in production - from simple LLM chains to complex multi-agent systems.

<Card title="restatedev/ai-examples" icon="github" href="https://github.com/restatedev/ai-examples">
  Browse through all AI examples here
</Card>

## Why Restate?

Restate makes building AI workflows and agent easy:

* ✅ Recovery from failures - Never lose agent progress again
* ✅ Built-in session management - Store context in Restate's K/V store
* ✅ Complete observability - Trace every decision and action
* ✅ Composable patterns - From simple agents to complex multi-agent systems
* ✅ Production safety - Approvals, timeouts, rollbacks, and more

Whether you're building chatbots, autonomous agents, or AI-powered workflows, Restate handles the complexity of distributed execution so you can focus on your AI logic.

## LLM & Agent SDK Integrations

Get started quickly by integrating Restate with your favorite LLM and Agent SDKs. Restate works with most AI SDKs that allow you to wrap model calls and tools, enabling durable execution for your AI workloads.

<CardGroup>
  <Card title="Vercel AI SDK" icon="https://mintcdn.com/restate-6d46e1dc/VzG7GAPBH0jzpzUw/img/ai/sdk-integrations/vercel.svg?fit=max&auto=format&n=VzG7GAPBH0jzpzUw&q=85&s=4967ab5afdb94b8334f29f964de9dcb9" href="/ai/sdk-integrations/vercel-ai-sdk" />

  <Card title="OpenAI Agents SDK" icon="https://mintcdn.com/restate-6d46e1dc/eyiUDPHMMaoJj2hw/img/ai/sdk-integrations/openai.webp?fit=max&auto=format&n=eyiUDPHMMaoJj2hw&q=85&s=bd5490a9489b6c4b602e28fe0fa0e6d5" href="/ai/sdk-integrations/openai-agents-sdk" />

  <Card title="LiteLLM" href="/ai/sdk-integrations/litellm" icon="https://mintcdn.com/restate-6d46e1dc/eyiUDPHMMaoJj2hw/img/ai/sdk-integrations/lite-llm_icon.webp?fit=max&auto=format&n=eyiUDPHMMaoJj2hw&q=85&s=26d2b56cbaae9765079c2b25b855791e" />

  <Card title="Google ADK" href="/ai/sdk-integrations/google-adk" icon="https://mintcdn.com/restate-6d46e1dc/eyiUDPHMMaoJj2hw/img/ai/sdk-integrations/google-adk.png?fit=max&auto=format&n=eyiUDPHMMaoJj2hw&q=85&s=efb9e6bea73fd103d930af1d22c3234c" />

  <Card title="Integrating with other AI SDKs" href="/ai/sdk-integrations/integration-guide" />
</CardGroup>

## Composable AI Patterns

If you prefer to **own your control flow** and only want to use an LLM SDK for model calls, Restate turns your custom logic into durable, fault-tolerant workflows.

<CardGroup>
  <Card title="Prompt chaining" href="/ai/patterns/prompt-chaining" icon="link">
    Build fault-tolerant processing pipelines with automatic retries and recovery.
  </Card>

  <Card title="Tools and Workflows" href="/ai/patterns/tools" icon="wrench">
    Implement recoverable routing and tool execution with Restate.
  </Card>

  <Card title="Multi-Agent Systems" href="/ai/patterns/multi-agent" icon="users">
    Coordinate multiple AI agents to collaborate on complex tasks and workflows.
  </Card>

  <Card title="Sessions & chat" href="/ai/patterns/sessions-and-chat" icon="comments">
    Build stateful chat sessions with persistent context and concurrency control.
  </Card>

  <Card title="Human-in-the-loop" href="/ai/patterns/human-in-the-loop" icon="user-check">
    Integrate human feedback and approval steps into AI workflows and agent decision-making.
  </Card>

  <Card title="Parallelization" href="/ai/patterns/parallelization" icon="bolt">
    Execute multiple tools and agents concurrently with deterministic recovery.
  </Card>

  <Card title="Competitive racing" href="/ai/patterns/competitive-racing" icon="flag-checkered">
    Start multiple workflows/agents, return the first result, cancel the rest.
  </Card>

  <Card title="Notify when ready" href="/ai/patterns/notify-when-ready" icon="bell">
    Resilient async notifications for late responses of long-running agents.
  </Card>

  <Card title="Roll back on failures" href="/ai/patterns/rollback" icon="arrow-u-turn-up-left">
    Undo completed work when an agent fails to execute the entire workflow.
  </Card>
</CardGroup>

## Other Resources

Blog posts:

* Announcements:
  * [Vercel AI SDK Integration](https://www.restate.dev/blog/building-durable-agents-with-vercel-and-restate)
  * [OpenAI Agents SDK](https://www.restate.dev/blog/durable-orchestration-for-ai-agents-with-restate-and-openai-sdk)
  * [Google ADK](https://www.restate.dev/blog/build-resilient-ai-agents-with-restate-and-google-adk)
* [Durable AI Loops: Fault Tolerance across Frameworks and without Handcuffs](https://www.restate.dev/blog/resilient-serverless-agents)
* [AI Agents should be serverless and durable](https://www.restate.dev/blog/resilient-serverless-agents)
* [A Durable Coding Agent — with Modal and Restate](https://www.restate.dev/blog/durable-coding-agent-with-restate-and-modal)

Webinars:

* [Video: Durable AI Agents with Restate - Community Meeting July 2025](https://www.youtube.com/watch?v=BawfutguT5E)


# Chat UI Integration
Source: https://docs.restate.dev/ai/patterns/chat-ui-integration

Call Restate agents from frontend applications, such as Next.js, React or any web framework.

This guide explains how to invoke Restate services from frontend applications using the Restate SDK clients or plain HTTP requests.

## Calling agents from the frontend

You can call your agents in two ways:

1. Via the SDK clients (type-safe, recommended). Supported for [TypeScript](/services/invocation/clients/typescript-sdk),
   [Python](/services/invocation/clients/python-sdk),
   [Java](/services/invocation/clients/java-sdk),
   [Go](/services/invocation/clients/go-sdk).
2. Via [plain HTTP requests](/services/invocation/http) (works anywhere in any language).

Restate supports calling your agents via:

* **Request-response**: short operations where the user is waiting (\< few seconds)
* **Fire-and-forget**: longer operations where you'll receive updates via a separate channel ([pubsub](/ai/patterns/streaming-responses)/webhooks) or want to retrieve the result later
* **Scheduled/delayed calls**: schedules a request to an agent with a delay

### Example: TypeScript SDK client

Install the Restate SDK clients package:

```bash theme={null}
npm install @restatedev/restate-sdk-clients
```

[Export the type definition](/develop/ts/service-communication#advanced:-sharing-service-type-definitions) of your agent:

```typescript {"CODE_LOAD::ts/src/ai/guides/chat-ui/my_agent.ts#export"}  theme={null}
export type MyAgent = typeof myAgent;
```

Create a client connection to the Restate ingress, then use the client for your service type to send a request:

```typescript {"CODE_LOAD::ts/src/ai/guides/chat-ui/ui_sdk_clients.ts#here"}  theme={null}
// import * as clients from "@restatedev/restate-sdk-clients";
// import {MyAgent} from "./my_agent";

const ingress = clients.connect({ url: "http://localhost:8080" });

// Send a message without waiting for completion
const handle = await ingress
  .serviceSendClient<MyAgent>({ name: "my-agent" })
  .run({ message: "How are you?" });
```

Consult the [client SDK docs](/services/invocation/clients/typescript-sdk) to learn about calling Virtual Objects or doing other types of calls.

### Example: Next.js API Route

You can use the SDK clients to send requests from your NextJS app to Restate services.

Here's an example of an API route that calls a Restate service:

```typescript app/agent/[topic]/api/route.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/vercel-ai/nextjs-example-app/app/agent/%5Btopic%5D/api/route.ts"}  theme={null}
import { NextRequest } from "next/server";
import * as clients from "@restatedev/restate-sdk-clients";
import { Agent } from "@/restate/services/agent";

export async function POST(
  request: NextRequest,
  { params }: { params: Promise<{ topic: string }> },
) {
  const { topic } = await params;
  const { message } = await request.json();
  const ingressUrl = process.env.INGRESS_URL || "http://localhost:8080";

  const ingress = clients.connect({ url: ingressUrl });

  await ingress
    .serviceSendClient<Agent>({ name: "agent" })
    .chat({ prompt: message, topic });

  return Response.json({ ok: true });
}
```

[Check out the full NextJS example here.](https://github.com/restatedev/ai-examples/tree/main/vercel-ai/nextjs-example-app)

### Using HTTP requests

You can also call Restate services using plain HTTP requests without the SDK.

For example, in JavaScript:

```javascript {"CODE_LOAD::ts/src/ai/guides/chat-ui/http_requests.js#here"}  theme={null}
// Request-response
const response = await fetch("http://localhost:8080/my-agent/run", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ message: "How are you?" }),
});
const result = await response.json();

// Fire-and-forget: add /send to the URL
await fetch("http://localhost:8080/my-agent/run/send", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ message: "How are you?" }),
});
```

## Request deduplication / idempotency

You can let Restate deduplicate requests by adding an idempotency key to the header of your request.

```javascript {"CODE_LOAD::ts/src/ai/guides/chat-ui/http_requests.js#idempotency"}  theme={null}
await fetch("http://localhost:8080/my-agent/run/send", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "idempotency-key": "abc-123",
  },
  body: JSON.stringify("How are you?"),
});
```

This can help with implementing retries and avoiding rage clicking to lead to many invocations.

## Re-attach to executions

Restate lets you re-attach to ongoing executions. This is useful to retrieve the response from another processs, for example after a crash or as part of your application logic.idempotency

There are three ways to do this:

1. If you use an idempotency key, then sending the same request with the same idempotency key will re-attach you to the original invocation.

2. If you have the invocation ID, then you can use that to attach to it:

```javascript {"CODE_LOAD::ts/src/ai/guides/chat-ui/http_requests.js#attach"}  theme={null}
// start the invocation
const handle = await fetch("http://localhost:8080/agent/run/send", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify("How are you?"),
});

// retrieve the invocationId
const { invocationId, status } = await handle.json();

// retrieve the result
const result = await fetch(
  `http://localhost:8080/restate/invocation/${invocationId}/attach`,
  { method: "GET" }
);
await result.json();
```

3. If you started the invocation with an idempotency key, then you can use that to attach to it. This does not start the invocation as opposed to point 1:

```javascript {"CODE_LOAD::ts/src/ai/guides/chat-ui/http_requests.js#attach_idem"}  theme={null}
// start the invocation with idempotency key
const idempotencyKey = "abc-123";
const result = await fetch(
  `http://localhost:8080/restate/invocation/agent/run/${idempotencyKey}/attach`,
  { method: "GET" }
);
await result.json();
```

Consult the [HTTP requests](/services/invocation/http#attach-to-an-invocation) page for more information.


# Competitive Racing
Source: https://docs.restate.dev/ai/patterns/competitive-racing

Run agents in parallel, return the fastest response and cancel the others.

Execute multiple AI approaches or strategies simultaneously and return the result from whichever completes first successfully. This pattern is ideal when you have multiple ways to solve the same problem and want to minimize latency by racing them against each other.

Useful for:

* Querying multiple AI models (e.g., GPT-4, Claude, Gemini) and returning the fastest response
* Running different agents, prompts or strategies in parallel and using the first successful outcome

## How does Restate help?

The benefits of using Restate for competitive racing patterns are:

* **Durable coordination**: Restate turns Promises/Futures into durable, distributed constructs that persist across failures and process restarts. Race multiple approaches and return the first successful result.
* **Cancel slow tasks**: Failed or slower approaches can be cancelled, preventing resource waste
* **Serverless scaling**: Deploy racing strategies on serverless infrastructure for automatic scaling while the main process remains suspended
* Works with **any LLM SDK** (Vercel AI, LangChain, LiteLLM, etc.) and **any programming language** supported by Restate (TypeScript, Python, Go, etc.).

## Example

When you need a quick response and have access to multiple AI models, race them against each other to get the fastest result:

<CodeGroup>
  ```ts TypeScript {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/typescript-patterns/src/racing-agents.ts#here"}  theme={null}
  async function run(
    ctx: Context,
    { message }: { message: string },
  ): Promise<string> {
    // Start both service calls concurrently
    const slowCall = ctx.serviceClient(racingAgent).thinkLonger({ message });
    const slowResponse = slowCall.map((res) => ({ tag: "slow", res }));

    const fastCall = ctx.serviceClient(racingAgent).respondQuickly({ message });
    const fastResponse = fastCall.map((res) => ({ tag: "fast", res }));

    const pending = [slowResponse, fastResponse];

    // Wait for the first one to complete
    const { tag, res } = await RestatePromise.any(pending);

    if (tag === "fast") {
      console.log("Quick response won the race!");
      const slowInvocationId = await slowCall.invocationId;
      ctx.cancel(slowInvocationId);
    } else {
      console.log("Deep analysis won the race!");
      const quickInvocationId = await fastCall.invocationId;
      ctx.cancel(quickInvocationId);
    }

    return res ?? "LLM gave no response";
  }
  ```

  ```python Python {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/python-patterns/app/racing_agents.py?collapse_prequel"}  theme={null}
  racing_agent = Service("RacingAgent")


  @racing_agent.handler()
  async def run(ctx: Context, query: Question) -> str | None:
      """Run two approaches in parallel and return the fastest response."""
      # Start both service calls concurrently
      slow_response = ctx.service_call(think_longer, arg=query)
      quick_response = ctx.service_call(respond_quickly, arg=query)

      done, pending = await restate.wait_completed(slow_response, quick_response)

      # cancel the pending calls
      for f in pending:
          call_future = typing.cast(RestateDurableCallFuture, f)
          ctx.cancel_invocation(await call_future.invocation_id())

      # return the fastest result
      return await done[0]




  @racing_agent.handler()
  async def think_longer(ctx: Context, req: Question) -> str | None:
      output = await ctx.run_typed(
          "Deep analysis",
          llm_call,
          RunOptions(max_attempts=3),
          messages=f"Analyze this thoroughly: {req}",
      )
      return output.content


  @racing_agent.handler()
  async def respond_quickly(ctx: Context, req: Question) -> str | None:
      output = await ctx.run_typed(
          "Quick response",
          llm_call,
          RunOptions(max_attempts=3),
          messages=f"Quick answer: {req}",
      )
      return output.content
  ```
</CodeGroup>

View on GitHub: [TS](https://github.com/restatedev/ai-examples/blob/typescript_patterns/typescript-patterns/src/racing-agents.ts) /
[Python](https://github.com/restatedev/ai-examples/blob/main/python-patterns/app/racing_agents.py)

In the Restate UI, you can see how multiple approaches are started simultaneously, with the first successful result being returned while other tasks are automatically cancelled:

<img alt="Competitive racing execution - UI" />

<Accordion title="Run the example">
  <Steps>
    <Step title="Requirements">
      * AI SDK of your choice (e.g., OpenAI, LangChain, Pydantic AI, LiteLLM, etc.) to make LLM calls.
      * API key for your model provider.
    </Step>

    <Step title="Download the example">
      <CodeGroup>
        ```shell TypeScript theme={null}
        git clone https://github.com/restatedev/ai-examples.git &&
        cd typescript-patterns &&
        npm install
        ```

        ```shell Python theme={null}
        git clone https://github.com/restatedev/ai-examples.git &&
        cd python-patterns
        ```
      </CodeGroup>
    </Step>

    <Step title="Start the Restate Server">
      ```shell theme={null}
      restate-server
      ```
    </Step>

    <Step title="Start the Service">
      Export the API key of your model provider as an environment variable and then start the agent. For example, for OpenAI:

      <CodeGroup>
        ```shell TypeScript theme={null}
        export OPENAI_API_KEY=your_openai_api_key
        npm run dev
        ```

        ```shell Python theme={null}
        export OPENAI_API_KEY=your_openai_api_key
        uv run .
        ```
      </CodeGroup>
    </Step>

    <Step title="Register the services">
      <Tabs>
        <Tab title="UI">
          <img alt="Service Registration" />
        </Tab>

        <Tab title="CLI">
          ```shell theme={null}
          restate deployments register localhost:9080
          ```
        </Tab>
      </Tabs>
    </Step>

    <Step title="Send a request">
      In the UI (`http://localhost:9070`), click on the `run` handler of the `RacingAgent` service to open the playground and send a prompt to race multiple models:

      <img alt="Agent racing - UI" />
    </Step>

    <Step title="Check the Restate UI">
      In the UI, you can see how multiple models are queried simultaneously, with the fastest response winning the race:

      <img alt="Competitive racing execution - UI" />
    </Step>
  </Steps>
</Accordion>


# Human-in-the-loop steps
Source: https://docs.restate.dev/ai/patterns/human-in-the-loop

Build resilient human approval steps in your agent workflows.

Build resilient workflows that include human approval steps or external signals without worrying about failures or interruptions.

Sometimes you need to include a human evaluator, approval step, or another external signal in an agentic workflow.
With Restate, you can model this by creating a Durable Promise and awaiting its completion (via a callback). Without worrying about failures or interruptions.

## How does Restate help?

The benefits of using Restate here are:

* **Durable promises**: Create promises that survive process restarts and failures
* **Automatic recovery**: Resume exactly where you left off after any interruption
* Works with **any LLM SDK** (Vercel AI, LangChain, LiteLLM, etc.) and **any programming language** supported by Restate (TypeScript, Python, Go, etc.).
* **Serverless-friendly waiting**: Suspend execution during approval wait times - pay for active work, not idle time:

<img />

## Example

Use `ctx.awakeable()` to create durable promises that can be resolved externally. The workflow suspends during the wait and resumes automatically when the promise is resolved.

<CodeGroup>
  ```ts TypeScript {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/typescript-patterns/src/human-in-the-loop.ts#here"}  theme={null}
  const tools = {
    getHumanReview: tool({
      description: "Request human review if policy violation is uncertain.",
      inputSchema: z.object({}),
    }),
  };

  async function moderate(ctx: Context, { message }: { message: string }) {
    const prompt = `You are a content moderation agent. Decide if the content violates policy: ${message}`;
    const { text, toolCalls } = await ctx.run(
      "LLM call",
      // Use your preferred LLM SDK here
      async () => llmCall(prompt, tools),
      { maxRetryAttempts: 3 },
    );

    if (toolCalls?.[0]?.toolName === "getHumanReview") {
      // Create a recoverable approval promise
      const approval = ctx.awakeable<string>();
      await ctx.run("Ask review", () => notifyModerator(message, approval.id));

      // Suspend until moderator resolves the approval
      // Check the service logs to see how to resolve it over HTTP, e.g.:
      // curl http://localhost:8080/restate/awakeables/sign_.../resolve --json '"approved"'
      return approval.promise;
    }

    return text;
  }
  ```

  ```python Python {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/python-patterns/app/human_in_the_loop.py?collapse_prequel"}  theme={null}
  content_moderator = restate.Service("HumanInTheLoopService")


  @content_moderator.handler()
  async def moderate(ctx: restate.Context, content: Content) -> str | None:
      """Moderate content with optional human review."""

      # Run LLM moderation
      result = await ctx.run_typed(
          "moderate",
          llm_call,  # Use your preferred LLM SDK here
          RunOptions(max_attempts=3),
          messages=f"""You are a content moderation agent.
          Decide if the content violates policy: {content.message}
          Request human review via tools if policy violation is uncertain.""",
          tools=[tool("get_human_review", "Request human review")],
      )

      # Handle human review request
      if result.tool_calls and result.tool_calls[0].function.name == "get_human_review":
          # Create a recoverable approval promise
          approval_id, approval_promise = ctx.awakeable(type_hint=str)

          await ctx.run_typed(
              "notify moderator",
              notify_moderator,
              content=content,
              approval_id=approval_id,
          )

          # Suspend until moderator resolves the approval
          # Check the service logs to see how to resolve it over HTTP, e.g.:
          # curl http://localhost:8080/restate/awakeables/sign_.../resolve --json '"approved"'
          return await approval_promise

      return result.content
  ```
</CodeGroup>

View on GitHub: [TS](https://github.com/restatedev/ai-examples/blob/typescript_patterns/typescript-patterns/src/human-in-the-loop.ts) /
[Python](https://github.com/restatedev/ai-examples/blob/main/python-patterns/app/human_in_the_loop.py)

Check out the SDK documentation for more details on the awakeables and durable promises API ([TS](/develop/ts/external-events) / [Python](/develop/python/external-events)).

<Tip>
  This pattern is implementable with any of our SDKs and any AI SDK.
  If you need help with a specific SDK, please reach out to us via [Discord](https://discord.restate.dev) or [Slack](https://slack.restate.dev).
</Tip>

<Accordion title="Run the example">
  <Steps>
    <Step title="Requirements">
      * AI SDK of your choice (e.g., OpenAI, LangChain, Pydantic AI, LiteLLM, etc.) to make LLM calls.
      * API key for your model provider.
    </Step>

    <Step title="Download the example">
      <CodeGroup>
        ```shell TypeScript theme={null}
        git clone https://github.com/restatedev/ai-examples.git &&
        cd typescript-patterns &&
        npm install
        ```

        ```shell Python theme={null}
        git clone https://github.com/restatedev/ai-examples.git &&
        cd python-patterns
        ```
      </CodeGroup>
    </Step>

    <Step title="Start the Restate Server">
      ```shell theme={null}
      restate-server
      ```
    </Step>

    <Step title="Start the Service">
      Export the API key of your model provider as an environment variable and then start the agent. For example, for OpenAI:

      <CodeGroup>
        ```shell TypeScript theme={null}
        export OPENAI_API_KEY=your_openai_api_key
        npm run dev
        ```

        ```shell Python theme={null}
        export OPENAI_API_KEY=your_openai_api_key
        uv run .
        ```
      </CodeGroup>
    </Step>

    <Step title="Register the services">
      <Tabs>
        <Tab title="UI">
          <img alt="Service Registration" />
        </Tab>

        <Tab title="CLI">
          ```shell theme={null}
          restate deployments register localhost:9080
          ```
        </Tab>
      </Tabs>
    </Step>

    <Step title="Send a request">
      <Tabs>
        <Tab title="UI">
          In the UI (`http://localhost:9070`), click on the `moderate` handler of the `CallChainingService` to open the playground and send a default request:

          <img alt="Human approval playground - UI" />
        </Tab>

        <Tab title="curl">
          Or use `curl` to send a request directly to the service:

          <CodeGroup>
            ```shell TypeScript theme={null}
            curl localhost:8080/HumanInTheLoopService/moderate \
            --json '{"message": "Very explicit content that might violate the policy."}'
            ```

            ```shell Python theme={null}
            curl localhost:8080/HumanInTheLoopService/moderate \
            --json '{"message": "Very explicit content that might violate the policy."}'
            ```
          </CodeGroup>
        </Tab>
      </Tabs>

      This will block on the human approval step, so you will not see a response yet.

      You can see in the Invocations Tab of the UI how the workflow suspends after waiting for a minute:

      <img alt="Human approval suspended journal" />
    </Step>

    <Step title="Resolve the approval">
      Check the service logs to find the curl command to resolve the approval. It will look like this:

      ```shell theme={null}
      curl http://localhost:8080/restate/awakeables/{awakeable_id}/resolve --json '"approved"'
      ```

      Replace `{awakeable_id}` with the actual ID from the logs.
    </Step>

    <Step title="Check the Restate UI">
      You can see in the Invocations Tab of the UI how the workflow suspends during the human approval step and resumes once the promise is resolved.

      <img alt="Human approval - journal" />
    </Step>
  </Steps>
</Accordion>


# Multi-Agent Systems
Source: https://docs.restate.dev/ai/patterns/multi-agent

Implement recoverable routing of tasks to tools and agents with Restate.

Build resilient multi-agent systems that automatically recover from failures and scale across distributed infrastructure. Restate provides two approaches for routing requests to specialized agents: **local agents** for simple routing within a single service, and **remote agents** for separate scaling or isolation.

## What Restate offers for multi-agent systems

At the core of every multi-agent system is a routing mechanism that decides which agent should handle each request. Restate makes this routing resilient by ensuring all decisions and agent interactions are durably logged and automatically retried.

### Two routing approaches

**Local Agent Routing**

* Route to different specialized prompts within the same service
* Best for: Simple agent specialization with shared context and without infrastructure complexity
* Use when: You want different AI personalities/expertise but don't need separate deployments

**Remote Agent Routing**

* Route to independent services running across different infrastructure
* Best for: Systems requiring independent scaling, isolation, or different technology stacks
* Use when: You need agents in different languages/SDKs, separate scaling, or team ownership boundaries

### How Restate ensures resilience

**Durable routing decisions**: When your LLM decides to route to an agent, that decision is automatically persisted. If anything fails, Restate resumes from exactly where it left off—no duplicate work, no lost context.

**Failure-resistant communication**: Whether calling local or remote agents, all interactions are wrapped in Restate's durable execution. Network failures, service restarts, and timeouts are handled automatically with retries and recovery.

**End-to-end observability**: The Restate UI shows the complete execution trace across all agent calls, making it easy to debug complex multi-agent workflows and understand routing decisions.

Works with **any LLM SDK** (Vercel AI, LangChain, LiteLLM, etc.) and **any programming language** supported by Restate (TypeScript, Python, Go, etc.).

## Routing to local agents

Local agent routing lets you create specialized AI assistants within a single service by using different prompts and personalities. The LLM first decides which specialist is needed, then you call the LLM again with a specialized prompt for that agent.

This approach is perfect when you want to create focused expertise areas (like billing, technical support, or sales) without the complexity of separate services.

<CodeGroup>
  ```ts TypeScript {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/typescript-patterns/src/routing-to-agent.ts#here"}  theme={null}
  const SPECIALISTS = {
    billingAgent: {
      description: "Expert in payments, charges, and refunds",
      prompt:
        "You are a billing support agent specializing in payments, charges, and refunds.",
    },
    accountAgent: {
      description: "Expert in login issues and security",
      prompt:
        "You are an account support agent specializing in login issues and security.",
    },
    productAgent: {
      description: "Expert in features and how-to guides",
      prompt:
        "You are a product support agent specializing in features and how-to guides.",
    },
  } as const;

  type Specialist = keyof typeof SPECIALISTS;

  async function answer(ctx: Context, { message }: { message: string }) {
    // 1. First, decide if a specialist is needed
    const routingDecision = await ctx.run(
      "Pick specialist",
      // Use your preferred LLM SDK here - specify agents as tools
      async () => llmCall(message, createTools(SPECIALISTS)),
      { maxRetryAttempts: 3 },
    );

    // 2. No specialist needed? Give a general answer
    if (!routingDecision.toolCalls || routingDecision.toolCalls.length === 0) {
      return routingDecision.text;
    }

    // 3. Get the specialist's name
    const specialist = routingDecision.toolCalls[0].toolName as Specialist;

    // 4. Ask the specialist to answer
    const { text } = await ctx.run(
      `Ask ${specialist}`,
      async () =>
        llmCall([
          { role: "user", content: message },
          { role: "system", content: SPECIALISTS[specialist].prompt },
        ]),
      { maxRetryAttempts: 3 },
    );

    return text;
  }
  ```

  ```python Python {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/python-patterns/app/routing_to_agent.py?collapse_prequel"}  theme={null}
  router = restate.Service("AgentRouter")

  # Our team of AI specialists
  SPECIALISTS = {
      "BillingAgent": "Expert in payments, charges, and refunds",
      "AccountAgent": "Expert in login issues and security",
      "ProductAgent": "Expert in features and how-to guides",
  }


  @router.handler()
  async def answer(ctx: restate.Context, question: Question) -> str | None:
      """Classify request and route to appropriate specialized agent."""

      # 1. First, decide if a specialist is needed
      routing_decision = await ctx.run_typed(
          "Pick specialist",
          llm_call,  # Use your preferred LLM SDK here
          RunOptions(max_attempts=3),
          messages=f"""You are a customer service routing system. 
          Choose the appropriate specialist, or respond directly if no specialist is needed. 
          {question.message}""",
          tools=[tool(name=name, description=desc) for name, desc in SPECIALISTS.items()],
      )

      # 2. No specialist needed? Give a general answer
      if not routing_decision.tool_calls:
          return routing_decision.content

      # 3. Get the specialist's name
      specialist = routing_decision.tool_calls[0].function.name or "ProductAgent"

      # 4. Ask the specialist to answer
      response = await ctx.run_typed(
          f"Ask {specialist}",
          llm_call,
          RunOptions(max_attempts=3),
          messages=f"""You are a {SPECIALISTS.get(specialist)} specialist."
          Answer the question: {question.message}""",
      )

      return response.content
  ```
</CodeGroup>

View on GitHub: [TS](https://github.com/restatedev/ai-examples/blob/typescript_patterns/typescript-patterns/src/routing-to-agent.ts) /
[Python](https://github.com/restatedev/ai-examples/blob/main/python-patterns/app/routing_to_agent.py)

In the Restate UI, you can see how the LLM decides to forward the request to the specialized support agents, and how the response is processed:

<img alt="Dynamic routing based on LLM output - UI" />

<Accordion title="Run the example">
  <Steps>
    <Step title="Requirements">
      * AI SDK of your choice (e.g., OpenAI, LangChain, Pydantic AI, LiteLLM, etc.) to make LLM calls.
      * API key for your model provider.
    </Step>

    <Step title="Download the example">
      <CodeGroup>
        ```shell TypeScript theme={null}
        git clone https://github.com/restatedev/ai-examples.git &&
        cd typescript-patterns &&
        npm install
        ```

        ```shell Python theme={null}
        git clone https://github.com/restatedev/ai-examples.git &&
        cd python-patterns
        ```
      </CodeGroup>
    </Step>

    <Step title="Start the Restate Server">
      ```shell theme={null}
      restate-server
      ```
    </Step>

    <Step title="Start the Service">
      Export the API key of your model provider as an environment variable and then start the agent. For example, for OpenAI:

      <CodeGroup>
        ```shell TypeScript theme={null}
        export OPENAI_API_KEY=your_openai_api_key
        npm run dev
        ```

        ```shell Python theme={null}
        export OPENAI_API_KEY=your_openai_api_key
        uv run .
        ```
      </CodeGroup>
    </Step>

    <Step title="Register the services">
      <Tabs>
        <Tab title="UI">
          <img alt="Service Registration" />
        </Tab>

        <Tab title="CLI">
          ```shell theme={null}
          restate deployments register localhost:9080
          ```
        </Tab>
      </Tabs>
    </Step>

    <Step title="Send a request">
      In the UI (`http://localhost:9070`), click on the `answer` handler of the `AgentRouter` service to open the playground and send a default request:

      <img alt="Multi-agent routing - UI" />
    </Step>

    <Step title="Check the Restate UI">
      In the UI, you can see how the LLM decides to forward the request to the specialized support agents, and how the response is processed:

      <img alt="Dynamic routing based on LLM output - UI" />
    </Step>
  </Steps>
</Accordion>

## Routing to remote agents

Deploy specialized agents as separate services when you need independent scaling, isolation, or different technology stacks.

Remote agents run as independent services that communicate over HTTP.
Restate makes these calls look like local function calls while providing end-to-end durability and failure recovery.

The example below shows dynamic routing where the LLM's decision determines which remote service to call. Each specialist agent runs as its own service with a standard `run` handler.

<CodeGroup>
  ```ts TypeScript {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/typescript-patterns/src/routing-to-remote-agent.ts#here"}  theme={null}
  // Define your agents as tools as your AI SDK requires (here Vercel AI SDK)
  const SPECIALISTS = {
    BillingAgent: { description: "Expert in payments, charges, and refunds" },
    AccountAgent: { description: "Expert in login issues and security" },
    ProductAgent: { description: "Expert in features and how-to guides" },
  } as const;
  type Specialist = keyof typeof SPECIALISTS;

  async function answer(ctx: Context, { message }: { message: string }) {
    // 1. First, decide if a specialist is needed
    const messages: ModelMessage[] = [
      {
        role: "system",
        content:
          "You are a routing agent. Route the question to a specialist or respond directly if no specialist is needed.",
      },
      { role: "user", content: message },
    ];
    const routingDecision = await ctx.run(
      "Pick specialist",
      // Use your preferred LLM SDK here - specify agents as tools
      async () => llmCall(messages, createTools(SPECIALISTS)),
      { maxRetryAttempts: 3 },
    );

    // 2. No specialist needed? Give a general answer
    if (!routingDecision.toolCalls || routingDecision.toolCalls.length === 0) {
      return routingDecision.text;
    }

    // 3. Get the specialist's name
    const specialist = routingDecision.toolCalls[0].toolName as Specialist;

    // 4. Call the specialist over HTTP
    return ctx.genericCall<string, string>({
      service: specialist,
      method: "run",
      parameter: message,
      inputSerde: restate.serde.json,
      outputSerde: restate.serde.json,
    });
  }
  ```

  ```python Python {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/python-patterns/app/routing_to_remote_agent.py?collapse_prequel"}  theme={null}
  remote_agent_router = restate.Service("RemoteAgentRouter")

  # Classify the request
  SPECIALISTS = {
      "BillingAgent": "Expert in payments, charges, and refunds",
      "AccountAgent": "Expert in login issues and security",
      "ProductAgent": "Expert in features and how-to guides",
  }


  @remote_agent_router.handler()
  async def answer(ctx: restate.Context, question: Question) -> str | None:
      """Classify request and route to appropriate specialized agent."""

      # 1. First, decide if a specialist is needed
      routing_decision = await ctx.run_typed(
          "Pick specialist",
          llm_call,  # Use your preferred AI SDK here
          RunOptions(max_attempts=3),
          messages=question.message,
          tools=[tool(name=name, description=desc) for name, desc in SPECIALISTS.items()],
      )

      # 2. No specialist needed? Give a general answer
      if not routing_decision.tool_calls:
          return routing_decision.content

      # 3. Get the specialist's name
      specialist = routing_decision.tool_calls[0].function.name
      if not specialist:
          return "Unable to determine specialist"

      # 4. Call the specialist over HTTP
      response = await ctx.generic_call(
          specialist,
          "run",
          arg=question.model_dump_json().encode(),
      )
      return response.decode("utf-8")
  ```
</CodeGroup>

View on GitHub: [TS](https://github.com/restatedev/ai-examples/blob/typescript_patterns/typescript-patterns/src/routing-to-remote-agent.ts) /
[Python](https://github.com/restatedev/ai-examples/blob/main/python-patterns/app/routing_to_remote_agent.py)

Each agent is then implemented in its own service. For example, here is the billing agent implementation.

<Accordion title="Billing Agent implementation">
  <CodeGroup>
    ```ts TypeScript {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/typescript-patterns/src/utils/utils.ts#here"}  theme={null}
    export const billingAgent = restate.service({
      name: "BillingAgent",
      handlers: {
        run: async (ctx: Context, question: string): Promise<string> => {
          const { text } = await ctx.run(
            "LLM call",
            async () =>
              llmCall(`You are a billing support specialist.
                Acknowledge the billing issue, explain charges clearly, provide next steps with timeline.
                ${question}`),
            { maxRetryAttempts: 3 },
          );
          return text;
        },
      },
    });
    ```

    ```python Python {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/python-patterns/app/util/util.py#here"}  theme={null}
    billing_agent_svc = restate.Service("BillingAgent")


    @billing_agent_svc.handler("run")
    async def get_billing_support(ctx: restate.Context, question: Question) -> str | None:
        result = await ctx.run_typed(
            "LLM call",
            llm_call,
            RunOptions(max_attempts=3),
            messages=f"""You are a billing support specialist.
            Acknowledge the billing issue, explain charges clearly, provide next steps with timeline.
            {question.message}""",
        )
        return result.content
    ```
  </CodeGroup>
</Accordion>

For more details on service communication, see: [TypeScript](/develop/ts/service-communication) / [Python](/develop/python/service-communication).

The Restate UI shows how the LLM decides to forward the request to the specialized remote support agents, and shows the execution trace of the nested calls in a single view.
This is useful for debugging and monitoring complex multi-agent workflows.

<img alt="Dynamic routing based on LLM output - UI" />

<Accordion title="Run the example">
  <Steps>
    <Step title="Requirements">
      * AI SDK of your choice (e.g., OpenAI, LangChain, Pydantic AI, LiteLLM, etc.) to make LLM calls.
      * API key for your model provider.
    </Step>

    <Step title="Download the example">
      <CodeGroup>
        ```shell TypeScript theme={null}
        git clone https://github.com/restatedev/ai-examples.git &&
        cd typescript-patterns &&
        npm install
        ```

        ```shell Python theme={null}
        git clone https://github.com/restatedev/ai-examples.git &&
        cd python-patterns
        ```
      </CodeGroup>
    </Step>

    <Step title="Start the Restate Server">
      ```shell theme={null}
      restate-server
      ```
    </Step>

    <Step title="Start the Service">
      Export the API key of your model provider as an environment variable and then start the agent. For example, for OpenAI:

      <CodeGroup>
        ```shell TypeScript theme={null}
        export OPENAI_API_KEY=your_openai_api_key
        npm run dev
        ```

        ```shell Python theme={null}
        export OPENAI_API_KEY=your_openai_api_key
        uv run .
        ```
      </CodeGroup>
    </Step>

    <Step title="Register the services">
      <Tabs>
        <Tab title="UI">
          <img alt="Service Registration" />
        </Tab>

        <Tab title="CLI">
          ```shell theme={null}
          restate deployments register localhost:9080
          ```
        </Tab>
      </Tabs>
    </Step>

    <Step title="Send a request">
      In the UI (`http://localhost:9070`), click on the `answer` handler of the `RemoteAgentRouter` service to open the playground and send a default request:

      <img alt="Multi-agent routing - UI" />
    </Step>

    <Step title="Check the Restate UI">
      In the UI, you can see how the LLM decides to forward the request to the specialized support agents, and the nested execution trace of the remote calls:

      <img alt="Dynamic routing based on LLM output - UI" />
    </Step>
  </Steps>
</Accordion>


# Notify when ready
Source: https://docs.restate.dev/ai/patterns/notify-when-ready

Build async notification systems for your agents with just a few lines of code.

Let users subscribe to notifications for long-running agent tasks instead of waiting.
With Restate's durable promises, you can coordinate between agent execution and notification handlers, no extra infrastructure needed.

## How does Restate help?

Building this yourself requires message queues, databases, and polling workers. Restate replaces all that with a few lines of code:

* **Durable promises**: Create promises that survive failures and restarts
* **Zero infrastructure**: No queues, databases, or polling workers needed
* **No race conditions**: Late subscribers immediately receive already-resolved results
* **Serverless-friendly**: Handlers suspend while waiting - pay only for active compute
* Works with **any LLM SDK** and **any language** supported by Restate

## Example

The pattern works in two steps:

**Step 1:** The agent handler processes the request and resolves a durable promise with the result.

**Step 2:** The notification handler awaits that promise and sends the notification when it resolves.

```python Python {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/openai-agents/examples/notify_when_ready/agent.py?collapse_imports"}  theme={null}
agent_service = restate.Workflow("AsyncNotificationsAgent")

agent = Agent(
    name="Assistant",
    instructions="You are a helpful assistant.",
)


@agent_service.main()
async def run(ctx: restate.WorkflowContext, user_query: str):
    # Process the user's query with the AI agent
    response = await DurableRunner.run(agent, user_query)

    # Notify other handlers of the response
    await ctx.promise("agent_response").resolve(response.final_output)

    # Return synchronous response
    return response.final_output


@agent_service.handler()
async def on_notify(ctx: restate.WorkflowContext, email: str):
    # Wait for the agent's response
    response = await ctx.promise("agent_response", type_hint=str).value()

    # Send the email
    await ctx.run_typed("Email", send_email, email=email, body=response)
```

View on GitHub: [Python](https://github.com/restatedev/ai-examples/blob/main/openai-agents/examples/notify_when_ready/agent.py)

Check out the SDK documentation for more details on the durable promises API ([TS](/develop/ts/external-events) / [Python](/develop/python/external-events)).

<Tip>
  This pattern is implementable with any of our SDKs and any AI SDK.
  If you need help with a specific SDK, please reach out to us via [Discord](https://discord.restate.dev) or [Slack](https://slack.restate.dev).
</Tip>

<Accordion title="Run the example">
  <Steps>
    <Step title="Requirements">
      * AI SDK of your choice (e.g., OpenAI, LangChain, Pydantic AI, LiteLLM, etc.) to make LLM calls.
      * API key for your model provider.
    </Step>

    <Step title="Download the example">
      ```shell Python theme={null}
      git clone https://github.com/restatedev/ai-examples.git &&
      cd ai-examples/openai-agents/examples/notify_when_ready
      ```
    </Step>

    <Step title="Start the Restate Server">
      ```shell theme={null}
      restate-server
      ```
    </Step>

    <Step title="Start the Service">
      Export the API key of your model provider as an environment variable and then start the agent. For example, for OpenAI:

      ```shell Python theme={null}
      export OPENAI_API_KEY=your_openai_api_key
      uv run .
      ```
    </Step>

    <Step title="Register the services">
      <Tabs>
        <Tab title="UI">
          <img alt="Service Registration" />
        </Tab>

        <Tab title="CLI">
          ```shell theme={null}
          restate deployments register localhost:9080
          ```
        </Tab>
      </Tabs>
    </Step>

    <Step title="Send a request">
      In the UI (`http://localhost:9070`), click on the `run` handler of the `AsyncNotificationsAgent` to open the playground and send a request to the `run` handler and then the `on_notify` handler.

      Or send requests via the terminal.

      First, start the agent asynchronously by adding `/send` to the end of the URL:

      ```json theme={null}
      curl localhost:8080/AsyncNotificationsAgent/msg-123/run/send \
        --json '"Write a 1000-word description of Durable Execution"'
      ```

      Then, asynchronously call the `on_notify` handler to send the agent response via email once it's ready:

      ```json theme={null}
      curl localhost:8080/AsyncNotificationsAgent/msg-123/on_notify/send \
        --json '"me@mail.com"'
      ```

      Notice that the key `msg-123` is identical for both requests, to send them to the same workflow instance.
    </Step>

    <Step title="Check the Restate UI">
      In the Restate UI, you see how the notify handler waited on the agent response and then sent the email.

      <img />
    </Step>
  </Steps>
</Accordion>


# Parallelizing Tools and Agents
Source: https://docs.restate.dev/ai/patterns/parallelization

Execute multiple AI tools and agent tasks in parallel with automatic recovery and coordination.

Execute multiple AI tools or agent tasks simultaneously to improve performance and efficiency. Restate ensures that all parallel operations are durably logged and automatically coordinated, enabling recovery from failures while maintaining consistency across concurrent executions.

## How does Restate help?

The benefits of using Restate for parallel agent and tool execution are:

* **Guaranteed execution**: Restate lets you schedule tasks asynchronously and guarantees that all tasks will run, with retries and recovery on failures
* **Durable coordination**: Restate turns Promises/Futures into durable, distributed constructs that are persisted in Restate and can be recovered and awaited on another process
* **Serverless scaling**: You can deploy the subtask executors on serverless infrastructure, like AWS Lambda, to let them scale automatically. The main task, that is idle while waiting on the subtasks, gets suspended until it can make progress
* **Independent failure handling**: Failed operations are automatically retried without affecting successful ones
* Works with **any LLM SDK** (Vercel AI, LangChain, LiteLLM, etc.) and **any programming language** supported by Restate (TypeScript, Python, Go, etc.).

## Parallelizing tool calls

When an LLM decides to call multiple tools, you can execute all tool calls in parallel instead of sequentially. This significantly reduces latency when tools are independent.

Wrap tool executions in `ctx.run()` to ensure durability, and use `RestatePromise.all()` (TypeScript) or `restate.gather()` (Python) to coordinate parallel execution.

<CodeGroup>
  ```ts TypeScript {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/typescript-patterns/src/parallel-tools.ts#here"}  theme={null}
  // Define your tools as your AI SDK requires (here Vercel AI SDK)
  const tools = {
    get_weather: tool({
      description: "Get the current weather for a location",
      inputSchema: z.object({ city: z.string() }),
    }),
  };

  async function run(ctx: Context, { message }: { message: string }) {
    const history: ModelMessage[] = [{ role: "user", content: message }];

    while (true) {
      // Use your preferred LLM SDK here
      let { text, toolCalls, messages } = await ctx.run(
        "LLM call",
        async () => llmCall(history, tools),
        { maxRetryAttempts: 3 },
      );
      history.push(...messages);

      if (!toolCalls || toolCalls.length === 0) {
        return text;
      }

      // Run all tool calls in parallel
      let toolPromises = [];
      for (let { toolCallId, toolName, input } of toolCalls) {
        const { city } = input as { city: string };
        const promise = ctx.run(`Get weather ${city}`, () => fetchWeather(city));
        toolPromises.push({ toolCallId, toolName, promise });
      }

      // Wait for all tools to complete in parallel
      await RestatePromise.all(toolPromises.map(({ promise }) => promise));

      // Append all results to messages
      for (const { toolCallId, toolName, promise } of toolPromises) {
        messages.push(toolResult(toolCallId, toolName, await promise));
      }
    }
  }
  ```

  ```python Python {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/python-patterns/app/parallel_tools.py?collapse_prequel"}  theme={null}
  parallel_tools_agent = restate.Service("ParallelToolAgent")


  @parallel_tools_agent.handler()
  async def run(ctx: Context, prompt: WeatherPrompt) -> str | None:
      """Main agent loop with tool calling"""
      messages = [{"role": "user", "content": prompt.message}]

      while True:
          # Call LLM with durable execution
          response = await ctx.run_typed(
              "LLM call",
              llm_call,  # Use your preferred LLM SDK here
              RunOptions(max_attempts=3),
              messages=messages,
              tools=[
                  tool(
                      name="get_weather",
                      description="Get the current weather for a location",
                      parameters=WeatherRequest.model_json_schema(),
                  )
              ],
          )
          messages.append(response.dict())

          if not response.tool_calls:
              return response.content

          # Run all tool calls in parallel
          tool_promises = {}
          for tool_call in response.tool_calls:
              if tool_call.function.name == "get_weather":
                  req = WeatherRequest.model_validate_json(tool_call.function.arguments)
                  tool_promises[tool_call.id] = ctx.run_typed(
                      f"Get weather {req.city}",
                      get_weather,
                      req=req,
                  )

          #  Wait for all tools to complete and append results
          await restate.gather(*tool_promises.values())
          for tool_id, promise in tool_promises.items():
              output = await promise
              messages.append(tool_result(tool_id, "get_weather", str(output)))
  ```
</CodeGroup>

When you run the example below, you can see how multiple tool calls are executed in parallel, significantly reducing the total execution time compared to sequential processing:

<img alt="Parallel tool execution - UI" />

<Accordion title="Run the example">
  <Steps>
    <Step title="Requirements">
      * AI SDK of your choice (e.g., OpenAI, LangChain, Pydantic AI, LiteLLM, etc.) to make LLM calls.
      * API key for your model provider.
    </Step>

    <Step title="Download the example">
      <CodeGroup>
        ```shell TypeScript theme={null}
        git clone https://github.com/restatedev/ai-examples.git &&
        cd typescript-patterns &&
        npm install
        ```

        ```shell Python theme={null}
        git clone https://github.com/restatedev/ai-examples.git &&
        cd python-patterns
        ```
      </CodeGroup>
    </Step>

    <Step title="Start the Restate Server">
      ```shell theme={null}
      restate-server
      ```
    </Step>

    <Step title="Start the Service">
      Export the API key of your model provider as an environment variable and then start the agent. For example, for OpenAI:

      <CodeGroup>
        ```shell TypeScript theme={null}
        export OPENAI_API_KEY=your_openai_api_key
        npm run dev
        ```

        ```shell Python theme={null}
        export OPENAI_API_KEY=your_openai_api_key
        uv run .
        ```
      </CodeGroup>
    </Step>

    <Step title="Register the services">
      <Tabs>
        <Tab title="UI">
          <img alt="Service Registration" />
        </Tab>

        <Tab title="CLI">
          ```shell theme={null}
          restate deployments register localhost:9080
          ```
        </Tab>
      </Tabs>
    </Step>

    <Step title="Send a request">
      In the UI (`http://localhost:9070`), click on the `run` handler of the `ParallelToolAgent` service to open the playground and send a request that triggers multiple tool calls:

      <img alt="Parallel tool calls - UI" />
    </Step>

    <Step title="Check the Restate UI">
      In the UI, you can see how multiple tool calls are executed concurrently, with all operations completing in parallel:

      <img alt="Parallel tool execution - UI" />
    </Step>
  </Steps>
</Accordion>

## Parallelizing agents

Execute multiple independent agents simultaneously, such as analyzing different aspects of the same input or processing multiple requests concurrently.

This pattern is useful when you need to perform multiple analysis tasks that don't depend on each other, like sentiment analysis, key point extraction, and summarization.

<CodeGroup>
  ```ts TypeScript {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/typescript-patterns/src/parallel-agents.ts#here"}  theme={null}
  async function analyze(ctx: Context, { message }: { message: string }) {
    // Create parallel tasks - each runs independently
    const tasks = [
      ctx.run(
        "Analyze sentiment",
        // Use your preferred LLM SDK here
        async () => llmCall(`Analyze sentiment: ${message}`),
        { maxRetryAttempts: 3 },
      ),
      ctx.run(
        "Extract key points",
        async () => llmCall(`Extract 3 key points as bullets: ${message}`),
        { maxRetryAttempts: 3 },
      ),
      ctx.run(
        "Summarize",
        async () => llmCall(`Summarize in one sentence: ${message}`),
        { maxRetryAttempts: 3 },
      ),
    ];

    // Wait for all tasks to complete and return the results
    const results = await RestatePromise.all(tasks);
    return results.map((res) => res.text);
  }
  ```

  ```python Python {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/python-patterns/app/parallel_agents.py?collapse_prequel"}  theme={null}
  parallelization_svc = restate.Service("ParallelAgentsService")


  @parallelization_svc.handler()
  async def analyze(ctx: restate.Context, text: Text) -> list[str | None]:
      """Analyzes multiple aspects of the text in parallel."""

      # Create parallel tasks - each runs independently
      tasks = [
          ctx.run_typed(
              "Analyze sentiment",
              llm_call,  # Use your preferred LLM SDK here
              RunOptions(max_attempts=3),
              messages=f"Analyze sentiment (positive/negative/neutral): {text}",
          ),
          ctx.run_typed(
              "Extract key points",
              llm_call,
              RunOptions(max_attempts=3),
              messages=f"Extract 3 key points as bullets: {text}",
          ),
          ctx.run_typed(
              "Summarize",
              llm_call,
              RunOptions(max_attempts=3),
              messages=f"Summarize in one sentence: {text}",
          ),
      ]

      # Wait for all tasks to complete
      await restate.gather(*tasks)

      # Gather and collect results
      return [(await task).content for task in tasks]
  ```
</CodeGroup>

In the Restate UI, you can see how multiple analysis tasks are executed concurrently, each with independent retry policies and failure handling:

<img alt="Parallel agent execution - UI" />

<Accordion title="Run the example">
  <Steps>
    <Step title="Requirements">
      * AI SDK of your choice (e.g., OpenAI, LangChain, Pydantic AI, LiteLLM, etc.) to make LLM calls.
      * API key for your model provider.
    </Step>

    <Step title="Download the example">
      <CodeGroup>
        ```shell TypeScript theme={null}
        git clone https://github.com/restatedev/ai-examples.git &&
        cd typescript-patterns &&
        npm install
        ```

        ```shell Python theme={null}
        git clone https://github.com/restatedev/ai-examples.git &&
        cd python-patterns
        ```
      </CodeGroup>
    </Step>

    <Step title="Start the Restate Server">
      ```shell theme={null}
      restate-server
      ```
    </Step>

    <Step title="Start the Service">
      Export the API key of your model provider as an environment variable and then start the agent. For example, for OpenAI:

      <CodeGroup>
        ```shell TypeScript theme={null}
        export OPENAI_API_KEY=your_openai_api_key
        npm run dev
        ```

        ```shell Python theme={null}
        export OPENAI_API_KEY=your_openai_api_key
        uv run .
        ```
      </CodeGroup>
    </Step>

    <Step title="Register the services">
      <Tabs>
        <Tab title="UI">
          <img alt="Service Registration" />
        </Tab>

        <Tab title="CLI">
          ```shell theme={null}
          restate deployments register localhost:9080
          ```
        </Tab>
      </Tabs>
    </Step>

    <Step title="Send a request">
      In the UI (`http://localhost:9070`), click on the `analyze_text` handler of the `ParallelAgentsService` service to open the playground and send a text for analysis:

      <img alt="Parallel agent analysis - UI" />
    </Step>

    <Step title="Check the Restate UI">
      In the UI, you can see how multiple analysis tasks are executed in parallel, with each task having its own execution trace and retry policy:

      <img alt="Parallel agent execution - UI" />
    </Step>
  </Steps>
</Accordion>


# Prompt Chaining
Source: https://docs.restate.dev/ai/patterns/prompt-chaining

Build fault-tolerant processing pipelines with automatic retries and recovery.

Build fault-tolerant processing pipelines where each step transforms the previous step's output.
If any step fails, Restate automatically resumes from that exact point.

For example:

```
+----------+    +---------------+    +-------------+    +--------------+
|  Input   | => | Extract       | => | Sort        | => | Format as    |
| Message  |    | Metrics       |    | Results     |    | Table        |
+----------+    +---------------+    +-------------+    +--------------+
```

## How does Restate help?

The benefits of using Restate here are:

* **Automatic retries** of failed tasks: LLM API down, timeouts, infrastructure failures, etc.
* **Recovery of previous progress**: After a failure, Restate recovers the progress the execution did before the crash.
* Works with **any LLM SDK** (Vercel AI, LangChain, LiteLLM, etc.) and **any programming language** supported by Restate (TypeScript, Python, Go, etc.).

## Example

Wrap each step in the chain with `ctx.run()` to ensure fault tolerance and automatic recovery. Restate uses durable execution to persist the result of each step as it completes, so if any step fails, Restate will retry from that exact point without losing previous progress or re-executing completed steps.

<CodeGroup>
  ```ts TypeScript {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/typescript-patterns/src/chaining.ts#here"}  theme={null}
  async function process(ctx: Context, report: { message: string }) {
    // Step 1: Extract metrics
    const extract = await ctx.run(
      "Extract metrics",
      // Use your preferred LLM SDK here
      async () =>
        llmCall(`Extract numerical values and their metrics from the text. 
              Format as 'Metric Name: Value' per line. Input: ${report.message}`),
      { maxRetryAttempts: 3 },
    );

    // Step 2: Process the result from Step 1
    const sortedMetrics = await ctx.run(
      "Sort metrics",
      async () =>
        llmCall(`Sort lines in descending order by value: ${extract.text}`),
      { maxRetryAttempts: 3 },
    );

    // Step 3: Format as table
    const table = await ctx.run(
      "Format as table",
      async () =>
        llmCall(`Format the data as a markdown table: ${sortedMetrics.text}`),
      { maxRetryAttempts: 3 },
    );

    return table.text;
  }
  ```

  ```python Python {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/python-patterns/app/chaining.py?collapse_prequel"}  theme={null}
  call_chaining_svc = restate.Service("CallChainingService")


  @call_chaining_svc.handler()
  async def process(ctx: restate.Context, report: Report) -> str | None:
      """Sequentially chains multiple LLM calls, each transforming the prior output."""

      # Step 1: Extract metrics
      extract = await ctx.run_typed(
          "Extract metrics",
          llm_call,  # Use your preferred LLM SDK here
          RunOptions(max_attempts=3),  # Avoid infinite retries
          messages=f"""Extract numerical values and their metrics from the text. 
          Format as 'Metric Name: Value' per line. Input: {report.message}""",
      )

      # Step 2: Sort by value
      sorted_metrics = await ctx.run_typed(
          "Sort metrics",
          llm_call,
          RunOptions(max_attempts=3),
          messages=f"Sort lines in descending order by value: {extract}",
      )

      # Step 3: Format as table
      table = await ctx.run_typed(
          "Format as table",
          llm_call,
          RunOptions(max_attempts=3),
          messages=f"Format the data as a markdown table:{sorted_metrics}",
      )

      return table.content
  ```
</CodeGroup>

View on GitHub: [TS](https://github.com/restatedev/ai-examples/blob/typescript_patterns/typescript-patterns/src/chaining.ts) /
[Python](https://github.com/restatedev/ai-examples/blob/main/python-patterns/app/chaining.py)

The Restate UI shows how each step in the chain is executed and persisted:

<img alt="Chaining LLM calls - UI" />

<Tip>
  This pattern is implementable with any of our SDKs and any AI SDK.
  If you need help with a specific SDK, please reach out to us via [Discord](https://discord.restate.dev) or [Slack](https://slack.restate.dev).
</Tip>

<Accordion title="Run the example">
  <Steps>
    <Step title="Requirements">
      * AI SDK of your choice (e.g., OpenAI, LangChain, Pydantic AI, LiteLLM, etc.) to make LLM calls.
      * API key for your model provider.
    </Step>

    <Step title="Download the example">
      <CodeGroup>
        ```shell TypeScript theme={null}
        git clone https://github.com/restatedev/ai-examples.git &&
        cd typescript-patterns &&
        npm install
        ```

        ```shell Python theme={null}
        git clone https://github.com/restatedev/ai-examples.git &&
        cd python-patterns
        ```
      </CodeGroup>
    </Step>

    <Step title="Start the Restate Server">
      ```shell theme={null}
      restate-server
      ```
    </Step>

    <Step title="Start the Service">
      Export the API key of your model provider as an environment variable and then start the agent. For example, for OpenAI:

      <CodeGroup>
        ```shell TypeScript theme={null}
        export OPENAI_API_KEY=your_openai_api_key
        npm run dev
        ```

        ```shell Python theme={null}
        export OPENAI_API_KEY=your_openai_api_key
        uv run .
        ```
      </CodeGroup>
    </Step>

    <Step title="Register the services">
      <Tabs>
        <Tab title="UI">
          <img alt="Service Registration" />
        </Tab>

        <Tab title="CLI">
          ```shell theme={null}
          restate deployments register localhost:9080
          ```
        </Tab>
      </Tabs>
    </Step>

    <Step title="Send a request">
      In the UI (`http://localhost:9070`), click on the `process` handler of the `CallChainingService` to open the playground and send a default request:

      <img alt="Chaining LLM calls - UI" />
    </Step>

    <Step title="Check the Restate UI">
      You see in the Invocations Tab of the UI how the LLM is called multiple times, and how each result is persisted in Restate:

      <img alt="Chaining LLM calls - UI" />
    </Step>
  </Steps>
</Accordion>


# Rollback on Failures
Source: https://docs.restate.dev/ai/patterns/rollback

Implement rollback mechanisms for agent failures.

Implement compensation and rollback mechanisms for agents that need to undo partial work when failures occur. When agents perform multiple actions and something goes wrong, you need to systematically undo the changes to maintain consistency.

<Tip>
  This pattern is also called sagas in the context of microservices. [See the guide.](/guides/sagas)
</Tip>

## How does Restate help?

Restate provides durable execution for both the main workflow and compensation actions:

* **Guaranteed compensation**: If the main workflow fails, compensation handlers are reliably executed
* **No state management**: No manual state tracking or extra infra required
* **Observability**: Track both forward progress and rollback operations in the Restate UI
* Works with **any AI SDK** and **any programming language** supported by Restate

## Example

Track the rollback actions as you go, let the agent raise terminal tool errors, and execute the rollback actions in reverse order.

Here is an example of a travel booking agent that first reserves a hotel, flight and car, and then either confirms them or rolls back if any step fails with a terminal error (e.g. car type not available).

We let tools add rollback actions to the list for each booking step the do.
The `run` handler catches any terminal errors and runs all the rollback actions.

<CodeGroup>
  ```typescript Vercel AI SDK {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/vercel-ai/examples/src/rollback-agent.ts#here"}  theme={null}
  const book = async (ctx: Context, { id, prompt }: BookingRequest) => {
    const undo_list: { (): restate.RestatePromise<any> }[] = [];

    const model = wrapLanguageModel({
      model: openai("gpt-4o"),
      middleware: durableCalls(ctx, { maxRetryAttempts: 3 }),
    });

    try {
      const { text } = await generateText({
        model,
        system: `Book a complete travel package with the requirements in the prompt.
          Use tools to first book the hotel, then the flight.`,
        prompt,
        tools: {
          bookHotel: tool({
            description: "Book a hotel reservation",
            inputSchema: HotelBookingSchema,
            execute: async (req: HotelBooking) => {
              undo_list.push(() => ctx.run("🏨-cancel", () => cancelHotel(id)));
              return ctx.run("🏨-book", () => reserveHotel(id, req));
            },
          }),
          bookFlight: tool({
            description: "Book a flight",
            inputSchema: FlightBookingSchema,
            execute: async (req: FlightBooking) => {
              undo_list.push(() => ctx.run("✈️-cancel", () => cancelFlight(id)));
              return ctx.run("✈️-book", () => reserveFlight(id, req));
            },
          }),
        },
        stopWhen: [stepCountIs(10)],
        onStepFinish: rethrowTerminalToolError,
        providerOptions: { openai: { parallelToolCalls: false } },
      });
      return text;
    } catch (error) {
      console.log("Rolling back bookings");
      for (const undo_step of undo_list.reverse()) {
        await undo_step();
      }
      throw error;
    }
  };

  export default restate.service({
    name: "BookingWithRollbackAgent",
    handlers: {
      book: restate.createServiceHandler({ input: schema(BookingRequest) }, book),
    },
  });
  ```

  ```python OpenAI Agents {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/openai-agents/examples/rollback/agent.py?collapse_imports"}  theme={null}
  class BookingContext(BaseModel):
      model_config = ConfigDict(arbitrary_types_allowed=True)
      booking_id: str
      on_rollback: list[Callable] = Field(default=[])


  # Functions raise terminal errors instead of feeding them back to the agent
  @durable_function_tool
  async def book_hotel(
      wrapper: RunContextWrapper[BookingContext], booking: HotelBooking
  ) -> BookingResult:
      """Book a hotel"""
      ctx = restate_context()
      booking_ctx, booking_id = wrapper.context, wrapper.context.booking_id
      # Register a rollback action for each step, in case of failures further on in the workflow
      booking_ctx.on_rollback.append(
          lambda: ctx.run_typed("🏨 Cancel hotel", cancel_hotel, id=booking_id)
      )

      # Execute the workflow step
      return await ctx.run_typed(
          "🏨 Book hotel", reserve_hotel, id=booking_id, booking=booking
      )


  @durable_function_tool
  async def book_flight(
      wrapper: RunContextWrapper[BookingContext], booking: FlightBooking
  ) -> BookingResult:
      """Book a flight"""
      ctx = restate_context()
      booking_ctx, booking_id = wrapper.context, wrapper.context.booking_id
      booking_ctx.on_rollback.append(
          lambda: ctx.run_typed("✈️ Cancel flight", cancel_flight, id=booking_id)
      )
      return await ctx.run_typed(
          "✈️ Book flight", reserve_flight, id=booking_id, booking=booking
      )


  # ... Do the same for cars ...


  agent = Agent[BookingContext](
      name="BookingWithRollbackAgent",
      instructions="Book a complete travel package with the requirements in the prompt."
      "Use tools to first book the hotel, then the flight.",
      tools=[book_hotel, book_flight],
  )


  agent_service = restate.Service("BookingWithRollbackAgent")


  @agent_service.handler()
  async def book(_ctx: restate.Context, req: BookingPrompt) -> str:
      booking_ctx = BookingContext(booking_id=req.booking_id)
      try:
          result = await DurableRunner.run(agent, req.message, context=booking_ctx)
      except restate.TerminalError as e:
          # Run all the rollback actions on terminal errors
          for compensation in reversed(booking_ctx.on_rollback):
              await compensation()
          raise e

      return result.final_output
  ```
</CodeGroup>

View on GitHub: [TS](https://github.com/restatedev/ai-examples/tree/main/vercel-ai/examples/src/rollback-agent.ts) /
[Python](https://github.com/restatedev/ai-examples/blob/main/openai-agents/examples/rollback/agent.py)

Compensation actions should be idempotent since they may be retried. Design them to safely handle cases where the resource to be cleaned up no longer exists.
Have a look [at the sagas guide](/guides/sagas#advanced:-idempotency-and-compensations) for more information.

The Restate UI shows both the forward execution and rollback operations when failures occur:

<Frame>
  <img alt="Invocation overview" />
</Frame>

<Tip>
  This pattern is implementable with any of our SDKs and any AI SDK.
  If you need help with a specific SDK, please reach out to us via [Discord](https://discord.restate.dev) or [Slack](https://slack.restate.dev).
</Tip>

<Accordion title="Run the example">
  <Steps>
    <Step title="Requirements">
      * AI SDK of your choice (e.g., OpenAI, LangChain, Pydantic AI, LiteLLM, etc.) to make LLM calls.
      * API key for your model provider.
    </Step>

    <Step title="Download the example">
      <CodeGroup>
        ```shell TypeScript theme={null}
        git clone git@github.com:restatedev/ai-examples.git
        cd ai-examples/vercel-ai/examples
        npm install
        ```

        ```shell Python theme={null}
        git clone https://github.com/restatedev/ai-examples.git &&
        cd ai-examples/openai-agents/examples/rollback
        ```
      </CodeGroup>
    </Step>

    <Step title="Start the Restate Server">
      ```shell theme={null}
      restate-server
      ```
    </Step>

    <Step title="Start the Service">
      Export the API key of your model provider as an environment variable and then start the agent. For example, for OpenAI:

      <CodeGroup>
        ```shell TypeScript theme={null}
        export OPENAI_API_KEY=your_openai_api_key
        npm run dev
        ```

        ```shell Python theme={null}
        export OPENAI_API_KEY=your_openai_api_key
        uv run .
        ```
      </CodeGroup>
    </Step>

    <Step title="Register the services">
      <Tabs>
        <Tab title="UI">
          <img alt="Service Registration" />
        </Tab>

        <Tab title="CLI">
          ```shell theme={null}
          restate deployments register localhost:9080
          ```
        </Tab>
      </Tabs>
    </Step>

    <Step title="Send a request">
      In the UI (`http://localhost:9070`), click on the `book` handler of the `BookingWithRollbackAgent` to open the playground and send a default request:

      ```json theme={null}
      {
          "bookingId": "booking_123",
          "prompt": "I need to book a business trip to San Francisco from March 15-17. Flying from JFK, need a hotel downtown for 1 guest."
      }
      ```
    </Step>

    <Step title="Check the Restate UI">
      You can see in the Invocations Tab how the workflow executes forward steps, encounters a failure, then executes compensation actions in reverse order:

      <Frame>
        <img alt="Invocation overview" />
      </Frame>
    </Step>
  </Steps>
</Accordion>


# Durable Sessions
Source: https://docs.restate.dev/ai/patterns/sessions

Build persistent chat sessions that survive interruptions and maintain conversation state.

Build persistent, stateful chat sessions that handle long-running conversations across multiple interactions and users.

A user might start a conversation now, respond hours later, and return again after a few days. Multiple users may be having separate conversations going on, and a single conversation may be open in multiple browser windows.

This guide helps you with implementing persistent chat sessions with Restate.

## Virtual Objects

To implement stateful entities like chat sessions, or stateful agents, Restate provides the service type **Virtual Objects**.

Each Virtual Object instance maintains isolated state and is identified by a unique key.

Here is an example of a Virtual Object that represents chat sessions:

<img alt="Objects" />

<CodeGroup>
  ```ts TypeScript {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/typescript-patterns/src/chat.ts#here"}  theme={null}
  export default restate.object({
    name: "Chat",
    handlers: {
      message: restate.createObjectHandler(
        { input: zodPrompt(examplePrompt) },
        async (ctx: ObjectContext, { message }: { message: string }) => {
          const messages = (await ctx.get<Array<ModelMessage>>("memory")) ?? [];
          messages.push({ role: "user", content: message });

          // Use your preferred LLM SDK here
          const result = await ctx.run("LLM call", async () => llmCall(messages));

          messages.push({ role: "assistant", content: result.text });
          ctx.set("memory", messages);

          return result.text;
        },
      ),
      getHistory: restate.createObjectSharedHandler(
        async (ctx: restate.ObjectSharedContext) =>
          ctx.get<Array<ModelMessage>>("memory"),
      ),
    },
  });
  ```

  ```python Python {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/python-patterns/app/chat.py?collapse_prequel"}  theme={null}
  chat = restate.VirtualObject("Chat")


  @chat.handler()
  async def message(ctx: restate.ObjectContext, msg: ChatMessage) -> str | None:
      """A long-lived stateful chat session that allows for ongoing conversation."""

      # Retrieve conversation memory from Restate
      messages = await ctx.get("memory", type_hint=list[dict]) or []
      messages.append({"role": "user", "content": msg.message})

      result = await ctx.run_typed(
          "LLM call",
          llm_call,  # Use your preferred LLM SDK here
          RunOptions(max_attempts=3),
          messages=messages,
      )

      # Update conversation memory in Restate
      messages.append({"role": "assistant", "content": result.content})
      ctx.set("memory", messages)

      return result.content


  @chat.handler(kind="shared")
  async def get_history(ctx: restate.ObjectSharedContext):
      return await ctx.get("memory", type_hint=list[dict]) or []
  ```
</CodeGroup>

View on GitHub: [TS](https://github.com/restatedev/ai-examples/blob/main/typescript-patterns/src/chat.ts) /
[Python](https://github.com/restatedev/ai-examples/blob/main/python-patterns/app/chat.py)

This example stores the chat messages in Restate. You can also store any other K/V state, like user preferences.

Virtual Objects provide:

* **Durable state**: Conversation history or any other K/V state (e.g. user preferences) that persists across failures and restarts
* **Session isolation**: Each chat gets isolated state with automatic concurrency control ([see below](/ai/patterns/sessions#built-in-concurrency-control)
* Works with **any LLM SDK** (Vercel AI, LangChain, LiteLLM, etc.) and **any programming language** supported by Restate (TypeScript, Python, Go, etc.).

The UI lets you query the state of each chat session:

<img alt="Chat session state - UI" />

<Tip>
  This pattern is complementary to AI memory solutions like mem0 or graffiti. You can use Virtual Objects to enforce session concurrency and queueing while storing the agent's memory in specialized memory systems.
</Tip>

<Tip>
  This pattern is implementable with any of our SDKs and any AI SDK.
  If you need help with a specific SDK, please reach out to us via [Discord](https://discord.restate.dev) or [Slack](https://slack.restate.dev).
</Tip>

<Accordion title="Run the example">
  <Steps>
    <Step title="Requirements">
      * AI SDK of your choice (e.g., OpenAI, LangChain, Pydantic AI, LiteLLM, etc.) to make LLM calls.
      * API key for your model provider.
    </Step>

    <Step title="Download the example">
      <CodeGroup>
        ```shell TypeScript theme={null}
        git clone https://github.com/restatedev/ai-examples.git &&
        cd typescript-patterns &&
        npm install
        ```

        ```shell Python theme={null}
        git clone https://github.com/restatedev/ai-examples.git &&
        cd python-patterns
        ```
      </CodeGroup>
    </Step>

    <Step title="Start the Restate Server">
      ```shell theme={null}
      restate-server
      ```
    </Step>

    <Step title="Start the Service">
      Export the API key of your model provider as an environment variable and then start the agent. For example, for OpenAI:

      <CodeGroup>
        ```shell TypeScript theme={null}
        export OPENAI_API_KEY=your_openai_api_key
        npm run dev
        ```

        ```shell Python theme={null}
        export OPENAI_API_KEY=your_openai_api_key
        uv run .
        ```
      </CodeGroup>
    </Step>

    <Step title="Register the services">
      <Tabs>
        <Tab title="UI">
          <img alt="Service Registration" />
        </Tab>

        <Tab title="CLI">
          ```shell theme={null}
          restate deployments register localhost:9080
          ```
        </Tab>
      </Tabs>
    </Step>

    <Step title="Send messages to a chat session">
      <Tabs>
        <Tab title="UI">
          In the UI (`http://localhost:9070`), click on the `message` handler of the `Chat` service to open the playground.
          Enter a key for the chat session (e.g., `session123`) and send messages to start a conversation.

          <img alt="Chat playground" />
        </Tab>

        <Tab title="curl">
          Send a request to the service:

          <CodeGroup>
            ```shell TypeScript theme={null}
            curl localhost:8080/Chat/session123/message \
              --json '{"message": "Write a poem about Durable Execution?"}'
            ```

            ```shell Python theme={null}
            curl localhost:8080/Chat/session123/message \
              --json '{"message": "Write a poem about Durable Execution?"}'
            ```
          </CodeGroup>
        </Tab>
      </Tabs>

      The session state (conversation history) is automatically persisted and maintained across calls.
      Send additional messages with the same session ID to see how the conversation context is preserved. For example, ask to shorten the poem.
    </Step>

    <Step title="Check the Restate UI">
      In the State Tab, you can view what is stored in Restate for each chat session:

      <img alt="Chat session state - UI" />
    </Step>
  </Steps>
</Accordion>

## Built-in concurrency control

Restate’s Virtual Objects have built-in queuing and consistency guarantees per object key. Handlers either have read-write access (`ObjectContext`) or read-only access (shared object context).

* Only one handler with write access can run at a time per object key to prevent concurrent/lost writes or race conditions (for example `message()`).
* Handlers with read-only access can run concurrently to the write-access handlers (for example `getHistory()`).

<img alt="Queue" />

**Seeing concurrency control in action:**

In the chat service, the `message` handler is an exclusive handler, while the `getHistory` handler is a shared handler.

Let's send some messages to a chat session:

```bash theme={null}
curl localhost:8080/Chat/session123/message/send --json '{"message": "make a poem about durable execution"}' &
curl localhost:8080/Chat/session456/message/send --json '{"message": "what are the benefits of durable execution?"}' &
curl localhost:8080/Chat/session789/message/send --json '{"message": "how does workflow orchestration work?"}' &
curl localhost:8080/Chat/session123/message/send --json '{"message": "can you make it rhyme better?"}' &
curl localhost:8080/Chat/session456/message/send --json '{"message": "what about fault tolerance in distributed systems?"}' &
curl localhost:8080/Chat/session789/message/send --json '{"message": "give me a practical example"}' &
curl localhost:8080/Chat/session101/message/send --json '{"message": "explain event sourcing in simple terms"}' &
curl localhost:8080/Chat/session202/message/send --json '{"message": "what is the difference between async and sync processing?"}'
```

The UI shows how Restate queues the requests per session to ensure consistency. Only one chat agent is running per session ID:

<Frame>
  <img alt="Conversation State Management" />
</Frame>

## Retrieving state

The state you store in Virtual Objects lives forever. If you want to resume a session, this means simply sending a new message to the same virtual object.

To retrieve the state, we can add a handler which reads the state. Have a look at the `getHistory`/`get_history` handler in the example above.

Call the handler to get the history:

<CodeGroup>
  ```shell TypeScript theme={null}
  curl localhost:8080/Chat/user123/getHistory
  ```

  ```shell Python theme={null}
  curl localhost:8080/Chat/user123/get_history
  ```
</CodeGroup>

This is a [shared handler](/foundations/handlers#handler-behavior), meaning it can only read state (not write). This allows it to run concurrently with the `message` handler.


# Streaming Responses
Source: https://docs.restate.dev/ai/patterns/streaming-responses

Stream updates from Restate Services to the UI via Server-Sent Events.

This guide shows how to stream real-time updates from Restate services to a chat UI.

## Via Server-Sent Events (SSE)

Restate has a TypeScript library to implement streaming updates from Restate services to frontends via SSE.
You can use this, for example, to stream agent updates to your chat UI.

<Card title="restatedev/pubsub" icon="github" href="https://github.com/restatedev/pubsub">
  Explore the library.
</Card>

To use the library, have a look at the example and the steps below.

<Card title="nextjs-example-app" icon="github" href="https://github.com/restatedev/ai-examples/tree/main/vercel-ai/nextjs-example-app">
  NextJS Chat app backed by an AI SDK Agent that streams updates using SSE.
</Card>

<Steps>
  <Step title="Install dependencies">
    ```bash theme={null}
    npm install @restatedev/pubsub @restatedev/pubsub-client
    ```
  </Step>

  <Step title="Register the pubsub object">
    Add the pubsub Virtual Object to your Restate endpoint:

    ```typescript restate/endpoint.ts {"CODE_LOAD::ts/src/ai/guides/streaming-responses/endpoint.ts"}  theme={null}
    import * as restate from "@restatedev/restate-sdk/fetch";
    import { createPubsubObject } from "@restatedev/pubsub";
    import { agent } from "./services/agent";

    const pubsub = createPubsubObject("pubsub", {});

    export const endpoint = restate.createEndpointHandler({
        services: [agent, pubsub],
    });
    ```

    <GitHubLink />
  </Step>

  <Step title="Publish messages from your service">
    Use the pubsub client to publish messages:

    ```typescript restate/services/agent.ts {"CODE_LOAD::ts/src/ai/guides/streaming-responses/services/agent.ts"}  theme={null}
    import * as restate from "@restatedev/restate-sdk";
    import { createPubsubClient } from "@restatedev/pubsub-client";

    const pubsub = createPubsubClient({
        url: "http://localhost:8080",
        name: "pubsub",
    });

    // In your handler:
    export const agent = restate.service({
        name: "MyAgent",
        handlers: {
            run: async (ctx: restate.Context, prompt: string) => {
                await pubsub.publish("session-123", { role: "user", content: prompt }, ctx.rand.uuidv4());
            },
        },
    });
    ```

    <GitHubLink />

    Note that the UUID, in the publish step, is an idempotency key that Restate will use to deduplicate events.
    This is necessary since this publish step goes via the Restate ingress.
    Without this idempotency key, each replay/retry leads to duplicate events.
    Alternatively, you can use [service-to-service calls](/develop/python/service-communication#sending-messages) to publish events.
  </Step>

  <Step title="Create an SSE endpoint">
    For example for NextJS, create an API route that streams pubsub messages:

    ```typescript app/pubsub/[topic]/route.ts {"CODE_LOAD::ts/src/ai/guides/streaming-responses/pubsub-client.ts"}  theme={null}
    import { createPubsubClient } from "@restatedev/pubsub-client";
    import { NextRequest } from "next/server";

    const pubsub = createPubsubClient({
        url: process.env.INGRESS_URL || "http://localhost:8080",
        name: "pubsub",
    });

    export async function GET(request: NextRequest, { params }: any) {
        const topic = (await params).topic;
        const offset = Number(request.nextUrl.searchParams.get("offset") || 0);

        const stream = pubsub.sse({
            topic,
            offset,
            signal: request.signal,
        });

        return new Response(stream, {
            headers: { "Content-Type": "text/event-stream" },
        });
    }
    ```

    <GitHubLink />
  </Step>

  <Step title="Subscribe from the frontend">
    Use the browser's `EventSource` API to receive updates:

    ```typescript app/agent/[topic]/page.tsx {"CODE_LOAD::ts/src/ai/guides/streaming-responses/frontend.tsx#here"}  theme={null}
    let offset = 0;
    const evtSource = new EventSource(`/pubsub/${topic}?offset=${offset}`);

    evtSource.onmessage = (event) => {
        const data = JSON.parse(event.data);
        // show the new messages in the UI...
    };
    ```

    <GitHubLink />
  </Step>
</Steps>

<Tip>
  We are planning to add support for SSE with the Python SDK.
  Contact us on [Discord](https://discord.restate.dev) or or [Slack](https://slack.restate.dev) for more info.
</Tip>

## Streaming token-by-token

Token-by-token streaming is not yet supported but is on the roadmap.
Contact us on [Discord](https://discord.restate.dev) or or [Slack](https://slack.restate.dev) for more info.


# Tools and Workflows
Source: https://docs.restate.dev/ai/patterns/tools

Implement recoverable routing of tasks to tools and agents with Restate.

Build resilient AI agents that route requests to appropriate tools and workflows. Restate ensures all routing decisions, tool calls, and interactions are durably logged with automatic retries and recovery.

## How does Restate help?

Restate provides **durable execution** for tool routing and execution:

* **Durable routing decisions**: LLM routing choices are persisted and recovered after failures
* **Resilient tool execution**: Tool calls are wrapped with retries and failure recovery
* **Distributed tool communication**: Call remote tools as regular functions with end-to-end durability
* **Workflow orchestration**: Implement complex multi-step processes with state management and scheduling
* Works with **any LLM SDK** (Vercel AI, LangChain, LiteLLM, etc.) and **any programming language** supported by Restate (TypeScript, Python, Go, etc.).

Tools can either be executed **locally** within the same service/process or as **remote services**. Restate supports both options, allowing you to choose the best fit for your architecture and use case.

### Option 1: Local tools with durable execution

**What it provides:**

* Execute tools within the same service/process
* Automatic retries and failure recovery for each tool call

**Best for:** Tools with few steps, built-in SDK tools (e.g. search, database queries)

**What it looks like:** Tools defined as local functions. You can either:

1. Wrap the entire tool execution in `ctx.run()` for durability.
2. Do multiple Restate Context actions within the tool for finer-grained durability and retries.

### Option 2: Remote tools as separate services

**What it provides:**

* Tools can scale independently
* Tools can run on different infrastructure (e.g. serverless or Kubernetes) or languages
* Tools can run asynchronously: agent can kick off work and return early
* Durable communication between services

**Best for:** Complex tools, multi-step workflows, asynchronous or long-running tasks

**What it looks like:** Tools deployed as separate Restate services, called durably over HTTP from the main agent service.

## Example

This example implements an example of a local tool (querying the user database) and a remote workflow (creating a support ticket) for a technical support agent.

1. **Define tools** according to your AI SDK requirements
2. **Wrap LLM calls** in `ctx.run()` for persistence
3. **Process tool calls** in a loop until the LLM returns a final answer
   * Local tools get wrapped in `ctx.run()` for durability and retries (see tool call to query user database)
   * Remote tools get called with the SDK client (see tool call to create support ticket)

The following examples show how to implement local tool routing with popular AI SDKs:

<CodeGroup>
  ```ts TypeScript {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/typescript-patterns/src/routing-to-tools.ts#here"}  theme={null}
  // Define your tools as your AI SDK requires (here Vercel AI SDK)
  const tools = {
    queryUserDatabase: tool({
      description: "Get user account and billing info",
      inputSchema: z.object({}),
    }),
    createSupportTicket: tool({
      description: "Create support tickets",
      inputSchema: z.object({
        description: z.string().describe("Detailed description of the issue"),
      }),
    }),
  };

  async function route(ctx: Context, req: { message: string; userId: string }) {
    const messages: ModelMessage[] = [
      {
        role: "system",
        content:
          "You are a support agent. Use the available tools to answer the user's question. " +
          "If you don't know the answer, create a support ticket",
      },
      { role: "user", content: req.message },
    ];

    while (true) {
      // Call the LLM using your favorite AI SDK
      const result = await ctx.run(
        "LLM call",
        // Use your preferred LLM SDK here
        async () => llmCall(messages, tools),
        { maxRetryAttempts: 3 },
      );
      messages.push(...result.messages);

      if (result.finishReason !== "tool-calls") return result.text;

      for (const { toolName, toolCallId, input } of result.toolCalls) {
        let output: string;
        // Use ctx.run to ensure durable execution of tool calls
        switch (toolName) {
          case "queryUserDatabase":
            // Example of a local tool
            output = await ctx.run("Query DB", () => queryUserDb(req.userId));
            break;
          case "createSupportTicket":
            // Example of a remote tool/workflow
            output = await ctx
              .serviceClient(crmService)
              .createSupportTicket(input as SupportTicket);
            break;
          default:
            output = `Tool not found: ${toolName}`;
        }
        messages.push(toolResult(toolCallId, toolName, output));
      }
    }
  }
  ```

  ```python Python {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/python-patterns/app/routing_to_tool.py?collapse_prequel"}  theme={null}
  tool_router = restate.Service("ToolRouter")

  # Define tools as required by your LLM SDK
  TOOLS = [
      tool("query_user_database", "Get user account and billing info"),
      tool(
          "create_support_ticket",
          "Create support tickets",
          SupportTicket.model_json_schema(),
      ),
  ]


  @tool_router.handler()
  async def route(ctx: restate.Context, question: Question) -> str | None:
      """Route to appropriate tool and execute until final answer"""
      messages = [{"role": "user", "content": question.message}]

      while True:
          response = await ctx.run_typed(
              "LLM call",
              llm_call,  # Use your preferred LLM SDK here
              RunOptions(max_attempts=3, type_hint=Message),
              messages=messages,
              tools=TOOLS,
          )
          messages.append(response.dict())

          if not response.tool_calls:
              return response.content

          for tool_call in response.tool_calls:
              fn = tool_call.function
              tool_name = fn.name or "unknown"
              result = ""
              match tool_name:
                  case "query_user_database":
                      # Example of a local tool
                      result = await ctx.run_typed(
                          "Query DB", query_user_db, user_id=question.user_id
                      )
                  case "create_support_ticket":
                      # Example of a remote tool/workflow
                      ticket = SupportTicket.model_validate_json(fn.arguments)
                      result = await ctx.service_call(create_support_ticket, arg=ticket)
                  case _:
                      result = f"Tool not found: {tool_name}"

              messages.append(tool_result(tool_call.id, tool_name, result))
  ```
</CodeGroup>

View on GitHub: [TS](https://github.com/restatedev/ai-examples/blob/typescript_patterns/typescript-patterns/src/routing-to-tools.ts) /
[Python](https://github.com/restatedev/ai-examples/blob/main/python-patterns/app/routing_to_tool.py)

When you run the example below, you can see how the LLM decides to forward the request to the technical support tools, and how the response is processed:

<img alt="Dynamic routing based on LLM output - UI" />

<Accordion title="Run the example">
  <Steps>
    <Step title="Requirements">
      * AI SDK of your choice (e.g., OpenAI, LangChain, Pydantic AI, LiteLLM, etc.) to make LLM calls.
      * API key for your model provider.
    </Step>

    <Step title="Download the example">
      <CodeGroup>
        ```shell TypeScript theme={null}
        git clone https://github.com/restatedev/ai-examples.git &&
        cd typescript-patterns &&
        npm install
        ```

        ```shell Python theme={null}
        git clone https://github.com/restatedev/ai-examples.git &&
        cd python-patterns
        ```
      </CodeGroup>
    </Step>

    <Step title="Start the Restate Server">
      ```shell theme={null}
      restate-server
      ```
    </Step>

    <Step title="Start the Service">
      Export the API key of your model provider as an environment variable and then start the agent. For example, for OpenAI:

      <CodeGroup>
        ```shell TypeScript theme={null}
        export OPENAI_API_KEY=your_openai_api_key
        npm run dev
        ```

        ```shell Python theme={null}
        export OPENAI_API_KEY=your_openai_api_key
        uv run .
        ```
      </CodeGroup>
    </Step>

    <Step title="Register the services">
      <Tabs>
        <Tab title="UI">
          <img alt="Service Registration" />
        </Tab>

        <Tab title="CLI">
          ```shell theme={null}
          restate deployments register localhost:9080
          ```
        </Tab>
      </Tabs>
    </Step>

    <Step title="Send a request">
      In the UI (`http://localhost:9070`), click on the `route` handler of the `ToolRouter` service to open the playground and send a default request:

      <img alt="Dynamic routing LLM calls - UI" />

      Or send other requests to test different tool routing scenarios.
    </Step>

    <Step title="Check the Restate UI">
      In the UI, you can see how the LLM decides to forward the request to the technical support tools, and how the response is processed:

      <img alt="Dynamic routing based on LLM output - UI" />
    </Step>
  </Steps>
</Accordion>


# Restate & Google ADK
Source: https://docs.restate.dev/ai/sdk-integrations/google-adk

Learn how to use the Google ADK with Restate.

Integrate Google's Agent Development Kit (ADK) with Restate for fault-tolerant agent execution with automatic retries and durable state.

<Card icon="https://mintcdn.com/restate-6d46e1dc/eyiUDPHMMaoJj2hw/img/ai/sdk-integrations/google-adk.png?fit=max&auto=format&n=eyiUDPHMMaoJj2hw&q=85&s=efb9e6bea73fd103d930af1d22c3234c" title="Learn more about Google ADK" href="https://ai-sdk.dev" />

## Quick Start

* [Quickstart](https://github.com/restatedev/restate-google-adk-example)
* [Interactive Tour](/tour/google-adk)

<iframe title="YouTube video player" />

<Info>
  If you are interested in integrating Restate with Google ADK, please reach out to us via [Discord](https://discord.restate.dev) or [Slack](https://slack.restate.dev).
</Info>

## Learn More

* [Announcement:](https://www.restate.dev/blog/build-resilient-ai-agents-with-restate-and-google-adk) Build Resilient AI Agents with Restate and Google ADK


# Integrating Restate with LLM / Agent SDKs
Source: https://docs.restate.dev/ai/sdk-integrations/integration-guide

Learn what's required to integrate Restate with LLM / Agent SDKs.

Restate has a flexible programming model that allows it to be integrated with many LLM or Agent SDKs to add fault tolerance to your AI applications. While each SDK has its own API, the integration approach follows similar principles.

This guide covers the key patterns and requirements for integrating Restate with different types of SDKs.

<Info>
  If you are interested in the integration of a specific LLM / Agent SDK, please reach out to us via [Discord](https://discord.restate.dev) or [Slack](https://slack.restate.dev).
</Info>

## Agent SDKs vs. LLM SDKs

For simplicity, let's make a distinction between the following two types of SDKs:

<img alt="Diagram showing LLM SDKs vs Agent SDKs architecture" />

**1. LLM SDKs** - These provide simplified interfaces for making LLM calls:

* Abstract away HTTP request/response handling and parsing
* Offer unified APIs across multiple model providers
* Support tool definition but not automatic function calling or allow disabling it
* Leave execution control in your hands
* **Best fit for Restate integration**
* **Examples**: LiteLLM (Python), Vercel AI SDK (TypeScript), LangChain4j (Java), Spring AI (Java)

**2. Agent SDKs** - These manage both LLM calls and the agent execution loop:

* Provide high-level DSLs for defining agent behavior
* Handle the complete agent loop automatically
* Use configuration-driven approaches
* Manage tool execution and agent-to-agent communication
* **Require deeper integration with Restate**
* **Examples**: OpenAI Agents SDK, Vercel AI SDK (TypeScript), CrewAI, AutoGen

## Integrating LLM SDKs

With LLM SDKs, you maintain full control over when and how LLM calls are made. The integration is straightforward:

**Wrap LLM calls in `ctx.run()` blocks** to make them durable and fault-tolerant:

```typescript theme={null}
// Instead of direct LLM calls
const result = await openai.chat.completions.create({...});

// Wrap them in ctx.run()
const result = await ctx.run("analyze-text", async () => {
  return openai.chat.completions.create({...});
});
```

**Benefits:**

* Automatic retries on failures (API timeouts, rate limits, network issues)
* Progress preservation - completed steps aren't re-executed
* Works with any LLM SDK without modification
* You own the control flow and can use Restate to make it resilient
* Mix LLM-based steps and traditional workflows with a single resiliency strategy

**Examples:**
Find complete integration examples here:[TypeScript](https://github.com/restatedev/ai-examples/tree/main/typescript-patterns) / [Python](https://github.com/restatedev/ai-examples/tree/main/python-patterns)

## Integrating Agent SDKs

Agent SDKs require more careful integration since they control the execution loop. You have two approaches:

### Approach 1: Wrap the Entire Agent Execution

The simplest approach is to wrap the complete agent execution in a single `ctx.run()` block:

```typescript theme={null}
const result = await ctx.run("agent-execution", async () => {
  return await agent.execute(input);
});
```

This provides coarse-grained fault tolerance but treats the entire agent as a single recoverable step.

### Approach 2: Fine-Grained Integration

For better resilience, integrate Restate more deeply within the agent execution.

This requires a deep understanding of the Agent SDK you are using and careful designing and testing to avoid non-determinism issues or infinite retries in production.

**Key Requirements:**

* **Wrap model calls**: Ensure all LLM requests are wrapped in `ctx.run()`. We recommend setting the maximum number of retry attempts to prevent infinite retries and high LLM costs. You can do this via the `ctx.run` options ([TS](/develop/ts/durable-steps#error-handling-and-retry-policies)/[Python](/develop/python/durable-steps#error-handling-and-retry-policies)).
* **Pass Restate Context to tools**: Make the Restate Context available in your tool implementations to be able to execute durable steps. The recommended way to do this depends on the Agent SDK. Have a look at our [existing integrations](/ai#ai-%26-agent-sdk-integrations) for inspiration. Alternatively, you integrate more deeply with the Agent SDK and let it wrap each tool execution in `ctx.run`, to treat it as a single recoverable step.
* **Handle terminal errors**: Some Agent SDKs catch all errors that happen during tool execution and use them as input to the next iteration of the agent. Restate uses the concept of [Terminal Errors](/guides/error-handling) for errors which should not be retried. You might want to properly propagate these errors to make sure the agent doesn't retry them.
* **Propagate suspensions**: Restate [suspends](https://www.restate.dev/blog/resilient-serverless-agents#cost-efficiency-scaling-to-zero-while-waiting) your agent execution when your agent is idly waiting. It does this by throwing a suspension error. Similar to terminal errors, you need to make sure that these suspension errors do not get ingested by the LLM but get re-raised.
* **Disable streaming LLM responses**: Restate's `ctx.run()`- blocks do not support streaming responses. Therefore, you should turn off streaming for model responses or wait for the full response to arrive.
* **Disable parallelism**: Configure the agent to execute tools sequentially. Agent SDKs use the native async execution libraries to parallelize tool calls (`asyncio.gather` for Python, and `Promise.all` for TypeScript). This does not work with [Restate's replay mechanism](/foundations/key-concepts#durable-execution) because the order of completion may differ on retries, leading to a non-deterministic journal. Therefore, you need to disable this form of parallelism and implement parallelism yourself with Restate's concurrent task primitives, as shown in [this guide](/ai/patterns/parallelization).
* **Ensure determinism**: If the Agent SDK generates IDs, timestamps or other non-deterministic values, then these need to be persisted in Restate to make [replay deterministic](/foundations/key-concepts#durable-execution).
* **Persist built-in tool results**: Many agent SDKs have built-in tools such as web search or database queries. Make sure that these tool executions are wrapped in `ctx.run` to make them deterministic on replay.
* **State management**: Optionally, use Restate for agent session state. Some SDKs let you implement a session store interface to plug in your preferred session state store. For this, you can use Restate's K/V store in combination with Virtual Objects keyed by session ID.

<Info>
  These are the requirements we bumped into when integrating Restate with libraries like Vercel AI SDK and OpenAI Agents SDK. New requirements might need to be added for other SDKs.

  If you are considering an integration, please reach out to us via [Discord](https://discord.restate.dev) or [Slack](https://slack.restate.dev).
</Info>

**Benefits of fine-grained integration:**

* Individual steps within tool executions are fault-tolerant
* Partial progress is preserved during failures
* Better observability into agent execution steps
* No re-execution of expensive or slow operations


# Restate & LiteLLM
Source: https://docs.restate.dev/ai/sdk-integrations/litellm

Learn how to use the LiteLLM with Restate.

Integrating Restate with LiteLLM is straightforward. LiteLLM's design allows for easy [wrapping of model calls](/ai/sdk-integrations/integration-guide#integrating-ai-sdks), making it a great fit for Restate's durable AI capabilities.

<Card icon="https://mintcdn.com/restate-6d46e1dc/eyiUDPHMMaoJj2hw/img/ai/sdk-integrations/lite-llm_icon.webp?fit=max&auto=format&n=eyiUDPHMMaoJj2hw&q=85&s=26d2b56cbaae9765079c2b25b855791e" title="Learn more about LiteLLM" href="https://www.litellm.ai/" />

Have a look at the following resources to get started with Restate and LiteLLM:

* [Examples](https://github.com/restatedev/ai-examples/tree/main/python-patterns)
* [Composable AI patterns](/ai#composable-ai-patterns): Most patterns have been implemented using LiteLLM for model calls.

## Learn more

* [Blog:](https://www.restate.dev/blog/resilient-serverless-agents) AI Agents should be serverless


# Restate & OpenAI Agents SDK
Source: https://docs.restate.dev/ai/sdk-integrations/openai-agents-sdk

Learn how to use the OpenAI Agents SDK with Restate.

Integrate OpenAI Agents SDK with Restate for fault-tolerant agent execution with automatic retries and durable state.

<Card icon="https://mintcdn.com/restate-6d46e1dc/eyiUDPHMMaoJj2hw/img/ai/sdk-integrations/openai.webp?fit=max&auto=format&n=eyiUDPHMMaoJj2hw&q=85&s=bd5490a9489b6c4b602e28fe0fa0e6d5" title="Learn more about OpenAI Agents SDK" href="https://openai.github.io/openai-agents-python/" />

## Quick Start

* [Quickstart guide](/ai-quickstart#python-%2B-openai)
* [Interactive tour](/tour/openai-agents)

<iframe title="YouTube video player" />

## Learn More

* [Announcement:](https://www.restate.dev/blog/durable-orchestration-for-ai-agents-with-restate-and-openai-sdk) Durable Orchestration for AI Agents with Restate and OpenAI SDK
* [Blog:](https://www.restate.dev/blog/resilient-serverless-agents) Durable AI Loops: Fault Tolerance across Frameworks and without Handcuffs
* [Blog:](https://www.restate.dev/blog/resilient-serverless-agents) AI Agents should be serverless and durable


# Restate & Vercel AI SDK
Source: https://docs.restate.dev/ai/sdk-integrations/vercel-ai-sdk

Learn how to use the Vercel AI SDK with Restate.

The Vercel AI SDK is an excellent fit for Restate integration.

<Card icon="https://mintcdn.com/restate-6d46e1dc/VzG7GAPBH0jzpzUw/img/ai/sdk-integrations/vercel.svg?fit=max&auto=format&n=VzG7GAPBH0jzpzUw&q=85&s=4967ab5afdb94b8334f29f964de9dcb9" title="Learn more about Vercel AI SDK" href="https://ai-sdk.dev" />

## Quick Start

* [Quickstart guide](/ai-quickstart#typescript-%2B-vercel-ai)
* [Interactive tour](/tour/vercel-ai-agents)
* [npm package `@restatedev/vercel-ai-middleware`](https://www.npmjs.com/package/@restatedev/vercel-ai-middleware)

## Examples & Templates

* [Template](https://github.com/restatedev/ai-examples/tree/main/vercel-ai/template)
* [Examples](https://github.com/restatedev/ai-examples/tree/main/vercel-ai/examples)
* [Next.js template](https://github.com/restatedev/ai-examples/tree/main/vercel-ai/nextjs-template)
* [Next.js example app](https://github.com/restatedev/ai-examples/tree/main/vercel-ai/nextjs-example-app)
* [TypeScript patterns](https://github.com/restatedev/ai-examples/tree/main/typescript-patterns)

## Learn More

* [Announcement:](https://www.restate.dev/blog/building-durable-agents-with-vercel-and-restate) Building Durable AI Agents with Restate + Vercel AI SDK
* [Blog:](https://www.restate.dev/blog/resilient-serverless-agents) Durable AI Loops: Fault Tolerance across Frameworks and without Handcuffs
* [Blog:](https://www.restate.dev/blog/resilient-serverless-agents) AI Agents should be serverless and durable
* [Blog:](https://www.restate.dev/blog/durable-coding-agent-with-restate-and-modal) Durable Coding Agent with Modal and Restate
* [Video:](https://www.youtube.com/watch?v=BawfutguT5E) Community Meeting July 2025 - Durable AI Agents


# Connecting services to Restate Cloud
Source: https://docs.restate.dev/cloud/connecting-services

Learn how to connect your services to Restate Cloud.

When using Restate Cloud, you can run your services anywhere: on Kubernetes, as serverless functions, on AWS Lambda, or in private environments.
The only requirement is that your services need to be reachable from Restate Cloud's infrastructure.

You can connect your services to Restate Cloud in several ways, depending on where they run:

* Connect [Kubernetes services](/cloud/connecting-services#connecting-kubernetes-services) via a secure tunnel with Restate Operator.
* Connect [serverless functions (Vercel, Cloudflare Workers, Deno Deploy, etc.) or other public endpoints](/cloud/connecting-services#serverless-functions-and-other-public-endpoints), by signing requests with your cloud environment's public key.
* Connect [AWS Lambda functions](#connecting-aws-lambda-services), by granting Restate Cloud permission to assume a role in your AWS account.
* Connect [services in private environments](#connecting-services-in-private-environments), by setting up a tunnel.

If you prefer a video walkthrough, check out this webinar on getting started with cloud:

<iframe title="YouTube video player" />

## Kubernetes services

You can connect your Kubernetes services to Restate Cloud using a secure tunnel provided by the Restate Operator.

This allows you to run your services inside a private Kubernetes cluster without exposing them to the public internet.

The operator will:

* Establish a tunnel to Restate Cloud
* Register your services automatically
* Manage multiple deployment revisions
* Scale down old revisions when no longer needed

<Info>
  Try connecting a greeter service on a local kind Kubernetes cluster to Restate Cloud by following this [guide](/guides/connecting-k8s-services-to-cloud).
</Info>

### Operator deployment

Install the Restate Operator via Helm:

```bash theme={null}
helm install restate-operator \
  oci://ghcr.io/restatedev/restate-operator-helm \
  --namespace restate-operator \
  --create-namespace
```

To install the operator, you need to be able to create namespaces and CRDs.

### Create a Restate Cloud secret

Create an API key via `Developers` tab of the Restate Cloud UI and give it `Full` permissions.
The API key starts with `key_`. Save it in a file named `token` and create the secret in Kubernetes:

```bash theme={null}
kubectl create secret generic my-cloud-environment-secret \
  --from-file=token=./token -n restate-operator
```

### Set up a secure tunnel

Set up a secure tunnel between the Kubernetes cluster and Restate Cloud with the `RestateCloudEnvironment` CRD:

```yaml theme={null}
apiVersion: restate.dev/v1beta1
kind: RestateCloudEnvironment
metadata:
  name: my-cloud-environment
spec:
  environmentId: env_...
  signingPublicKey: publickeyv1_...
  region: eu
  authentication:
    secret:
      name: my-cloud-environment-secret
      key: token
```

* The `environmentId` can be found in the Restate Cloud UI in the top left corner when you select your environment.
  This ID starts with `env_`.

* The `signingPublicKey` can be found in the Restate Cloud UI under `Developers`. Scroll down to the `Security` section to find the public key under `HTTP endpoints`.
  The public key starts with `publickeyv1_`.

* Set the `region` field to the region your Restate Cloud environment is hosted in. Possible values are `us` and `eu`.
  You can find the region next to your environment ID in the Restate Cloud UI.

To see all the available configuration options for the `RestateCloudEnvironment` resource, refer to the Custom Resource Definition (CRD): [Pkl](https://github.com/restatedev/restate-operator/blob/main/crd/RestateCloudEnvironment.pkl) (recommended) or [YAML](https://github.com/restatedev/restate-operator/blob/main/crd/restatecloudenvironments.yaml).

<Info>
  [Learn more about how the tunnel works below.](/cloud/connecting-services#how-does-the-tunnel-work)
</Info>

### Create your RestateDeployment

Create a `RestateDeployment` manifest for your service, as described in the [deployment documentation](/services/deploy/kubernetes#deployment-with-restate-operator), and specify the cloud environment to register with in the `spec`:

```yaml theme={null}
spec:
  restate:
    register:
      cloud: my-cloud-environment
```

Once you apply the manifest, the Restate Operator will set up the secure tunnel and automatically register your service with Restate Cloud.

## Serverless Functions and other public endpoints

Restate can invoke your services over the public internet.
This is useful if you are deploying your services as serverless functions, e.g. on Vercel, Cloudflare Workers, or Deno Deploy.
You can also securely deploy Restate services you operate as directly reachable endpoints.

<Info>
  For AWS Lambda, please see the [AWS Lambda functions](#aws-lambda-functions) section below.
</Info>

It is important to secure access to your publicly routable services so that only the designated Restate environment can call them.
The easiest way to do this is with Restate's native request identity feature.
All requests to your service will be signed with a unique environment-specific private key.
The Restate SDK will reject any requests that do not have a valid signature.

You can find your cloud environment's public key in the Cloud UI's Developers tab under Security -> HTTP Services.
Since it is not secret, it is safe to include the public key directly in your service source code or configuration files:

<CodeGroup>
  ```typescript TypeScript {"CODE_LOAD::ts/src/develop/serving.ts#identity"}  theme={null}
  restate.serve({
    services: [myService],
    identityKeys: ["publickeyv1_w7YHemBctH5Ck2nQRQ47iBBqhNHy4FV7t2Usbye2A6f"],
  });
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/develop/ServingIdentity.java#here"}  theme={null}
  import dev.restate.sdk.auth.signing.RestateRequestIdentityVerifier;
  import dev.restate.sdk.endpoint.Endpoint;
  import dev.restate.sdk.http.vertx.RestateHttpServer;

  class MySecureApp {
    public static void main(String[] args) {
      var endpoint =
          Endpoint.bind(new MyService())
              .withRequestIdentityVerifier(
                  RestateRequestIdentityVerifier.fromKeys(
                      "publickeyv1_w7YHemBctH5Ck2nQRQ47iBBqhNHy4FV7t2Usbye2A6f"));
      RestateHttpServer.listen(endpoint);
    }
  }
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/ServingIdentity.kt#here"}  theme={null}
  import dev.restate.sdk.auth.signing.RestateRequestIdentityVerifier
  import dev.restate.sdk.http.vertx.RestateHttpServer
  import dev.restate.sdk.kotlin.endpoint.endpoint

  fun main() {
    RestateHttpServer.listen(
        endpoint {
          bind(MyService())
          requestIdentityVerifier =
              RestateRequestIdentityVerifier.fromKeys(
                  "publickeyv1_w7YHemBctH5Ck2nQRQ47iBBqhNHy4FV7t2Usbye2A6f",
              )
        })
  }
  ```

  ```python Python {"CODE_LOAD::python/src/develop/serving.py#identity"}  theme={null}
  app = restate.app(
      services=[my_service],
      identity_keys=["publickeyv1_w7YHemBctH5Ck2nQRQ47iBBqhNHy4FV7t2Usbye2A6f"],
  )
  ```

  ```go Go {"CODE_LOAD::go/develop/serving.go#identity"}  theme={null}
  if err := server.NewRestate().
    Bind(restate.Reflect(MyService{})).
    WithIdentityV1("publickeyv1_w7YHemBctH5Ck2nQRQ47iBBqhNHy4FV7t2Usbye2A6f").
    Start(context.Background(), ":9080"); err != nil {
    log.Fatal(err)
  }
  ```
</CodeGroup>

You can find more information on how to deploy Restate to various serverless platforms in the following guides:

* [Vercel](https://docs.restate.dev/services/deploy/vercel)
* [Cloudflare Workers](https://docs.restate.dev/services/deploy/cloudflare-workers)
* [Deno Deploy](/services/deploy/deno-deploy)

Note that, if you are using public endpoints, HTTPS must be used between Restate and your services. Consider using a load balancer like AWS Network Load Balancer which handles TLS termination. For optimum performance, the load balancer in front of your services must support HTTP/2.

## AWS Lambda functions

Restate Cloud can securely invoke your AWS Lambda functions by assuming an AWS role which you manage, and has appropriate permissions to invoke the relevant functions.

### Setting up the invoker role

Restate Cloud must be able to assume an AWS role with permission to perform `lambda:InvokeFunction`, within the same AWS account where the Lambda function is deployed. As Restate Cloud is a multi-tenant system, the role must be set up to only allow specifically enumerated Restate Cloud environment(s) you trust to assume it.

Note that the Restate Cloud role is separate from your Lambda function’s execution role.
The execution role is used by your function to perform its tasks, while the invoker role grants Restate Cloud permission to invoke your service handler functions — and nothing else.

You can set up the role directly in IAM or use an infrastructure-as-code tool such as AWS CDK to define it.

<Info>
  If your Lambda has an appropriate trust policy, you do not
  need to secure incoming requests any further. If you choose to however, the
  identity verification checks will work on Lambda endpoints as well.
</Info>

#### IAM trust policy for Restate Cloud

To create a new IAM role, you can copy the trust policy directly from the Restate Cloud UI, by visiting [Developers > Security > AWS Lambda](https://cloud.restate.dev/to/developers/integration#lambda).
The trust policy allows the Restate Cloud IAM identity to assume the invoker role, only on behalf of the specified environment ID.

Once you have created the invoker role, you can register your Lambda-backed service(s) with your Restate environment by specifying the role to be assumed:

```shell theme={null}
restate deployments register <LAMBDA_FUNCTION_ARN> --assume-role-arn <ROLE_ARN>
```

#### Using AWS CDK

You can use AWS CDK together with the [Restate CDK construct library](https://www.npmjs.com/package/@restatedev/restate-cdk) to deploy
Lambda handlers, manage the invoker role, and automatically register Lambda-backed services with a Restate (Cloud) environment.

Here is an example CDK stack that deploys a Lambda function and registers it with Restate Cloud:

```ts expandable {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/integrations/deployment-lambda-cdk/lib/lambda-ts-cdk-stack.ts?remove_comments"}  theme={null}

import * as restate from "@restatedev/restate-cdk";
import * as cdk from "aws-cdk-lib";
import * as lambda from "aws-cdk-lib/aws-lambda";
import * as lambda_nodejs from "aws-cdk-lib/aws-lambda-nodejs";
import * as secrets from "aws-cdk-lib/aws-secretsmanager";
import { Construct } from "constructs";

export class LambdaTsCdkStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props: cdk.StackProps) {
    super(scope, id, props);

    const handler = new lambda_nodejs.NodejsFunction(this, "GreeterService", {
      runtime: lambda.Runtime.NODEJS_20_X,
      entry: "lib/lambda/handler.ts",
      architecture: lambda.Architecture.ARM_64,
      bundling: {
        minify: true,
        sourceMap: true,
      },
    });

    if (!process.env.RESTATE_ENV_ID || !process.env.RESTATE_API_KEY) {
      throw new Error(
        "Required environment variables RESTATE_ENV_ID and RESTATE_API_KEY are not set, please see README.",
      );
    }

    const restateEnvironment = new restate.RestateCloudEnvironment(this, "RestateCloud", {
      environmentId: process.env.RESTATE_ENV_ID! as restate.EnvironmentId,
      apiKey: new secrets.Secret(this, "RestateCloudApiKey", {
        secretStringValue: cdk.SecretValue.unsafePlainText(process.env.RESTATE_API_KEY!),
      }),
    });
    const deployer = new restate.ServiceDeployer(this, "ServiceDeployer");


    deployer.deployService("Greeter", handler.currentVersion, restateEnvironment);
    new cdk.CfnOutput(this, "restateIngressUrl", { value: restateEnvironment.ingressUrl });
  }
}
```

View an example of deploying to AWS Lambda with SDK for your SDK here: [TS](https://github.com/restatedev/examples/tree/main/typescript/integrations/deployment-lambda-cdk), [Go](https://github.com/restatedev/examples/tree/main/go/integrations/go-lambda-cdk), [Java](https://github.com/restatedev/examples/tree/main/java/integrations/java-gradle-lambda-cdk), [Kotlin](https://github.com/restatedev/examples/tree/main/kotlin/integrations/kotlin-gradle-lambda-cdk)

* The `RESTATE_ENV_ID` can be found in the Restate Cloud UI in the top left corner when you select your environment.
  This ID starts with `env_`.

* For the `RESTATE_API_KEY`, create an API key via `Developers` tab of the Restate Cloud UI and give it `Full` permissions.

The `RestateCloudEnvironment` automatically creates an IAM role with an appropriate trust policy for the matching Restate Cloud environment. The `ServiceDeployer` construct automatically grants this role permission to invoke any AWS Lambda functions registered with the environment. For more details, consult the relevant [Restate CDK](https://www.npmjs.com/package/@restatedev/restate-cdk) constructs' API documentation.

The CDK also registers your service with Restate Cloud, so no need to do this manually.

If something isn't working, the environment logs in the Cloud UI may help you find the issue.

## Services in private environments

You can connect services running in a locked-down private environment to Restate Cloud using a secure tunnel.

This tunnel allows you to expose your services to Restate Cloud without opening up ports to the public internet.
It also acts as an authenticating proxy, making it appear as if your Restate Cloud environment is inside your private network.

The tunnel lets you use native access control mechanisms, like VPC Security Groups and Kubernetes network policies, to manage access to your Cloud environment. It also lets you control connections from the Cloud environment to the rest of your network by restricting which internal services the tunnel client can connect to.

<Info>
  If you are running on Kubernetes, then you can [use the Restate Operator to create a `RestateCloudEnvironment`](/cloud/connecting-services#kubernetes-services).
  This tunnel automatically sets up the tunnel for you.
  You only need to run the tunnel client manually if you’re not running on Kubernetes.
</Info>

### Set up a secure tunnel

The tunnel can be hosted as a cloud VM in your VPC, a sidecar to your service, or a dedicated pod in your container orchestrator.

You can deploy multiple copies for redundancy.

To run the tunnel:

```bash theme={null}
# export RESTATE_ENVIRONMENT_ID=env_...
# export RESTATE_BEARER_TOKEN=key_...
# export RESTATE_TUNNEL_NAME=test-tunnel
# export RESTATE_SIGNING_PUBLIC_KEY=publickeyv1_...
# export RESTATE_CLOUD_REGION=eu

docker run \
  -e RESTATE_ENVIRONMENT_ID \
  -e RESTATE_BEARER_TOKEN \
  -e RESTATE_TUNNEL_NAME \
  -e RESTATE_SIGNING_PUBLIC_KEY \
  -e RESTATE_CLOUD_REGION \
  -p 8080:8080 \
  -p 9090:9090 \
  -p 9070:9070 \
  -it ghcr.io/restatedev/restate-cloud-tunnel-client:latest
```

<AccordionGroup>
  <Accordion title="Environment variables">
    * `RESTATE_ENVIRONMENT_ID` is the environment id (including the `env_` prefix).
    * `RESTATE_BEARER_TOKEN` is the API key you created in step 1.
    * `RESTATE_TUNNEL_NAME` is a name for the tunnel. Choose a unique DNS-friendly tunnel name, e.g. `prod-tunnel`.
    * `RESTATE_SIGNING_PUBLIC_KEY` is the public key you copied from the Cloud UI in step 1.
    * `RESTATE_CLOUD_REGION` is the region of your Restate Cloud environment, e.g. `eu` or `us`.
    * You can run `latest` or pin the current version, e.g. `0.4.0`
    * The health check URL is at `:9090/health`
  </Accordion>

  <Accordion title="Tunnel ports">
    The tunnel client exposes the following ports

    * `9090` tunnel's own health status
    * `8080` Restate Ingress (authenticating proxy)
    * `9070` Restate Admin API (authenticating proxy)
  </Accordion>
</AccordionGroup>

In addition to exposing local services to Restate Cloud, by default the tunnel client will also serve unauthenticated ingress and admin endpoints from Restate Cloud on its local 8080 and 9080 ports. This behaviour can be disabled with `RESTATE_REMOTE_PROXY=false`. If left enabled, be careful to restrict access to these ports; access to them is equivalent to having the configured `RESTATE_BEARER_TOKEN`.

<Info>
  [Learn more about how the tunnel works below.](/cloud/connecting-services#how-does-the-tunnel-work)
</Info>

### Register your service

If your setup is correct, you can now register your service with the Restate Cloud environment:

```bash theme={null}
restate deployments register --tunnel-name <tunnel-name> <service address to be reached through the tunnel>
```

You can now invoke your service as usual.

### How does the tunnel work

* Restate CLI is communicating with your Restate environment at `https://*.eu.restate.cloud:9070` using your user ID token and asks the admin API to perform a discovery at a special tunnel URL.
* The Restate Cloud end of the tunnel receives the request from your environment and forwards the traffic to the tunnel container.
* The tunnel container is forwarding the traffic to the Restate service endpoint.

<img alt="Invocation overview" />

## Advanced topics

### Client-side encryption

You can optionally configure client-side encryption, which makes your data opaque to Restate Cloud.
In this case, the SDK encrypts journal entries after serializing them but before sending them to Restate Cloud.

**What is encrypted?** Only the following journal entries are encrypted:

* The input and output parameters to the handler
* `ctx.run` success results
* RPC calls parameters and return values (for service-to-service invocations)
* State values
* Awakeables and Durable Promise results

It is not possible to encrypt any other journal entries.

**How to enable encryption?** You need to implement a [`JournalValueCodec`](https://github.com/restatedev/sdk-typescript/blob/main/packages/restate-sdk-core/src/entry_codec.ts#L26) and provide it to the SDK when creating the endpoint.
Currently, this feature is only available in the TypeScript SDK.

<Card href={"https://github.com/restatedev/journal-encryption"}>
  Have a look at a reference implementation that uses Amazon's KMS to manage encryption keys.
</Card>


# Getting Started with Restate Cloud
Source: https://docs.restate.dev/cloud/getting-started

Learn how to use Restate Cloud, the fully managed version of Restate.

Restate Cloud is a fully managed serverless version of Restate.
This allows you to focus on implementing your services, while Restate Cloud handles all aspects of availability and durability for your invocations, workflows, and state.

<Card title="Restate Cloud" icon={"cloud"} href={"https://restate.dev/cloud/"}>
  Learn more about what Restate Cloud has to offer and pricing plans.
</Card>

Your services can run anywhere: on Kubernetes, as serverless functions, or in private environments.
Restate Cloud lets you connect your services securely, and provides a great local developer experience.

## Get started with the free tier

Restate Cloud offers a free tier that lets you get started quickly, with no credit card required.

<Card title="Restate Cloud Free Tier" icon="user-plus" href={"https://cloud.restate.dev/"}>
  Sign up for free and get started with Restate Cloud.
</Card>

After signing up, you'll be prompted to create an account and an environment.

An environment is a unique Restate cluster instance, managed by Restate.

Once you have your environment set up, you can follow the quickstart tour to get a feeling for what Restate Cloud has to offer: register the example service, invoke it, and explore the Restate Cloud UI.

<Frame>
  <img alt="Cloud login quickstart" />
</Frame>

## Local development with Restate Cloud

<Steps>
  <Step title="Connect your CLI to your environment">
    You can start using your new environment straight away using the [Restate CLI](/installation#running-restate-server--cli-locally).

    Log in to Restate Cloud:

    ```bash theme={null}
    restate cloud login
    ```

    Set up a new CLI profile for your environment:

    ```bash theme={null}
    restate cloud env configure
    ```

    Tell the CLI to use the new environment:

    ```bash theme={null}
    restate config use-env <name>
    ```

    <Info>
      At any time, you can switch your CLI back to point to a Restate server running
      on your machine with `restate config use-environment local` (see the
      [CLI config docs](/references/cli-config#)).
    </Info>
  </Step>

  <Step title="Expose your local service to Restate Cloud">
    After connecting the CLI to your environment, you can use the built-in tunnel feature to expose your local service to Restate Cloud.

    For example to expose a service running on `localhost:9080`, run:

    ```bash theme={null}
    restate cloud env tunnel --tunnel-name my-tunnel
    restate deployments register --tunnel-name my-tunnel http://localhost:9080
    ```
  </Step>

  <Step title="Invoke your local service">
    To invoke your service locally via Restate Cloud, you can create a tunnel that tunnel exposes the ingress port of your Restate Cloud instance as if it was running on your machine:

    ```bash theme={null}
    restate cloud env tunnel --remote-port 8080
    ```

    Now you can call your service handlers over HTTP on `localhost:8080` (Restate Cloud's ingress port):

    ```bash theme={null}
    curl localhost:8080/MyService/myHandler
    ```
  </Step>
</Steps>

## Connecting Restate services to your environment

When using Restate Cloud, you can run your services anywhere: on Kubernetes, as serverless functions, on AWS Lambda, or in private environments. The only requirement is that your services need to be reachable from Restate Cloud’s infrastructure.

Have a look at the [docs on connecting services to Restate Cloud](/cloud/connecting-services) to learn how to connect your services to your Restate Cloud environment.

## Invoking services with API keys

To call your services from other apps or scripts, send HTTP requests to your environment’s ingress URL using your API key.

You can find the ingress URL in the Developers tab of the Cloud UI.
Usually it has the form `https://<env_id>.env.<region>.restate.cloud:8080`, where the `<env_id>` is your environment ID without the `env_` prefix, and `<region>` is either `us` or `eu`.

To create an API key, go to the Developers tab in the Cloud UI.

Now you can call your service handlers by including the API Key as a Bearer token like this:

```bash theme={null}
curl -H "Authorization: Bearer $RESTATE_AUTH_TOKEN" https://201hy10cd3h6426jy80tb32n6en.env.us.restate.cloud:8080/MyService/MyHandler
curl -H "Authorization: Bearer $RESTATE_AUTH_TOKEN" https://201hy10cd3h6426jy80tb32n6en.env.us.restate.cloud:9070/deployments
```

You can also use the CLI with this token:

```bash theme={null}
export RESTATE_HOST_SCHEME=https RESTATE_HOST=201hy10cd3h6426jy80tb32n6en.env.us.restate.cloud RESTATE_AUTH_TOKEN=$RESTATE_AUTH_TOKEN
restate whoami
```

<Info>
  You can use this approach for sending requests to the Admin API from scripts, CI pipelines, API gateways, or services that exist outside Restate.
</Info>


# AI-Assisted Development
Source: https://docs.restate.dev/develop/ai-assistant

Developing Restate apps with AI coding agents.

The Restate documentation site includes several features to help you leverage AI coding agents like Cursor and Claude Code in your development workflow.

## Contextual menu

Each page of the Restate documentation has a contextual menu in the top right corner that provides quick access to various AI resources:

<img alt="Contextual menu" />

You can use this to quickly hydrate your AI chat with the content of the current page, or to add the Restate docs MCP server to Cursor, or VSCode.

## Restate skill

To add the Restate skill to your AI code assistant:

```shell theme={null}
npx skills add https://docs.restate.dev
```

We are continuously improving the Restate skill, let us know if you have any feedback via [Discord](https://discord.restate.dev) or [Slack](https://slack.restate.dev).

## Restate Docs as MCP server

The Restate documentation is available as an MCP server, which you can add to Cursor or other AI coding agents that support MCP.

You can either use the shortcuts in the [contextual menu](/develop/ai-assistant#contextual-menu) or follow the following guidelines:

<Tabs>
  <Tab title="Claude">
    1. Navigate to the `Connectors` page in the Claude settings.
    2. Select `Add custom connector`.
    3. Add the MCP server name and URL: `restate-docs`, `https://docs.restate.dev/mcp`.
    4. Select `Add`.
    5. When using Claude, select the attachments button (the plus icon).
    6. Select your MCP server.
  </Tab>

  <Tab title="Claude Code">
    To add the Restate documentation MCP server to Claude Code, run the following command in your terminal:

    ```shell theme={null}
    claude mcp add --transport http restate-docs https://docs.restate.dev/mcp
    ```
  </Tab>

  <Tab title="Cursor">
    1. Use `Command` + `Shift` + `P` (`Ctrl` + `Shift` + `P` on Windows) to open the command palette.
    2. Search for `“Open MCP settings”`.
    3. Select `Add custom MCP`. This will open the `mcp.json` file.
    4. In `mcp.json`, configure your server:

    ```json theme={null}
    {
        "mcpServers": {
            "restate-docs": {
                "url": "https://docs.restate.dev/mcp"
            }
        }
    }
    ```
  </Tab>

  <Tab title="VS Code">
    1. Create a `.vscode/mcp.json` file.
    2. In `mcp.json`, configure your server:

    ```json theme={null}
    {
        "servers": {
        "restate-docs": {
                "type": "http",
                "url": "https://docs.restate.dev/mcp"
            }
        }
    }
    ```
  </Tab>
</Tabs>

More information in the [Mintlify documentation](https://www.mintlify.com/docs/ai/model-context-protocol#using-your-mcp-server).

## AGENTS.md: Coding agent rules

AGENTS.md is a simple markdown file for defining agent instructions.

To improve the performance of coding agents when implementing Restate applications, add the following rules to your agent's context.

* [TypeScript SDK AGENTS.md](/develop/ts/agents.md)
* [Java SDK AGENTS.md](/develop/java/agents.md)
* [Python SDK AGENTS.md](/develop/python/agents.md)
* [Go SDK AGENTS.md](/develop/go/agents.md)

### Cursor

Download the AGENTS.md file and put it [in the `.cursor/rules` folder](https://cursor.com/docs/context/rules#agentsmd) at the root of your project:

<CodeGroup>
  ```shell Typescript theme={null}
  mkdir -p ./.cursor/rules && wget -O ./.cursor/rules/AGENTS.md https://docs.restate.dev/develop/ts/agents.md
  ```

  ```shell Java theme={null}
  mkdir -p ./.cursor/rules && wget -O ./.cursor/rules/AGENTS.md https://docs.restate.dev/develop/java/agents.md
  ```

  ```shell Python theme={null}
  mkdir -p ./.cursor/rules && wget -O ./.cursor/rules/AGENTS.md https://docs.restate.dev/develop/python/agents.md
  ```

  ```shell Go theme={null}
  mkdir -p ./.cursor/rules && wget -O ./.cursor/rules/AGENTS.md https://docs.restate.dev/develop/go/agents.md
  ```
</CodeGroup>

### Claude Code

Download the CLAUDE.md file and put it [in the `.claude` folder](https://www.anthropic.com/engineering/claude-code-best-practices) at the root of your project:

<CodeGroup>
  ```shell Typescript theme={null}
  mkdir -p ./.claude && wget -O ./.claude/CLAUDE.md https://docs.restate.dev/develop/ts/agents.md
  ```

  ```shell Java theme={null}
  mkdir -p ./.claude && wget -O ./.claude/CLAUDE.md https://docs.restate.dev/develop/java/agents.md
  ```

  ```shell Python theme={null}
  mkdir -p ./.claude && wget -O ./.claude/CLAUDE.md https://docs.restate.dev/develop/python/agents.md
  ```

  ```shell Go theme={null}
  mkdir -p ./.claude && wget -O ./.claude/CLAUDE.md https://docs.restate.dev/develop/go/agents.md
  ```
</CodeGroup>

## llms.txt and llms-full.txt

The Restate documentation is available in markdown, for easy ingestion by LLMs.

Add `.md` to any page’s URL to view a Markdown version.

The documentation also includes [`llms.txt`](https://docs.restate.dev/llms.txt) (navigation structure) and [`llms-full.txt`](https://docs.restate.dev/llms-full.txt) (full documentation) files.


# Code Generation
Source: https://docs.restate.dev/develop/go/code-generation

Use proto for type-safe servers and clients.

The Go SDK supports generating typed clients and server interfaces from
[Protocol Buffer service definitions](https://protobuf.dev/programming-guides/proto3/#services).
A full example can be found in the [SDK repository](https://github.com/restatedev/sdk-go/tree/main/examples/codegen).

Code generation relies on the tool `protoc-gen-go-restate` which you can install using
`go install github.com/restatedev/sdk-go/protoc-gen-go-restate@latest`, in addition to `protoc`
which you can install from the [Protobuf compiler repo](https://github.com/protocolbuffers/protobuf#protobuf-compiler-installation).

## Defining a service

To get started, create a Protocol Buffer definition file defining a service:

```proto service.proto {"CODE_LOAD::go/develop/codegen/proto/service.proto#service"}  theme={null}
package greeter;

service Greeter {
  rpc SayHello (HelloRequest) returns (HelloResponse) {}
}

message HelloRequest {
  string name = 1;
}

message HelloResponse {
  string message = 1;
}
```

You can compile this file into Go types and your Restate generated code with:

```shell theme={null}
protoc --go_out=. --go_opt=paths=source_relative --go-restate_out=. --go-restate_opt=paths=source_relative proto/service.proto
```

This creates two files: `service.pb.go` containing Go types, and `service_restate.pb.go` which contains several useful objects:

1. `NewGreeterClient`, which returns an interface for making type-safe calls to the Greeter service. The default serialization
   for request and response data is the [canonical Protobuf JSON encoding](https://protobuf.dev/programming-guides/proto3/#json),
   but this is configurable.
2. `GreeterServer`, an interface for server code to implement. An implementor can be converted into a service definition with
   `NewGreeterServer`.
3. `UnimplementedGreeterServer`, an empty struct that implements `GreeterServer` by just returning 'not implemented' terminal
   errors for each method. You can choose to embed this struct into your own implementation struct, in which case new methods
   in the Protobuf definition will not cause compilation errors in your Go code, which can help with backwards compatibility.

<Info>
  Both `NewGreeterClient` and `NewGreeterServer` accept option parameters which can be used to set a different serialization codec.
  For example, you can use Protobuf by providing `restate.WithProto` to both functions.
</Info>

## Implementing the server

Next, you can implement the `GreeterServer` interface and bind it to your Restate server.

```go {"CODE_LOAD::go/develop/codegen/codegen.go#server"}  theme={null}
type greeter struct {
  proto.UnimplementedGreeterServer
}

func (greeter) SayHello(ctx restate.Context, req *proto.HelloRequest) (*proto.HelloResponse, error) {
  return &proto.HelloResponse{
    Message: fmt.Sprintf("%s!", req.Name),
  }, nil
}

func main() {
  if err := server.NewRestate().
    Bind(proto.NewGreeterServer(greeter{})).
    Start(context.Background(), ":9080"); err != nil {
    log.Fatal(err)
  }
}
```

## Using the client

`NewGreeterClient` will return a similar client to what you'd get from `restate.Service`, with additional
type safety for method names and request/response types.

```go {"CODE_LOAD::go/develop/codegen/codegen.go#client"}  theme={null}
client := proto.NewGreeterClient(ctx)
resp, err := client.SayHello().Request(&proto.HelloRequest{Name: "world"})
if err != nil {
  return err
}
```

## Defining Virtual Objects and Workflows

To define a Virtual Object or Workflow, you will need to provide the option `dev.restate.sdk.go.service_type` when defining the service.
This option is defined by the Go SDK's proto file
[`dev/restate/sdk/go.proto`](https://github.com/restatedev/sdk-go/blob/main/proto/dev/restate/sdk/go.proto) which must be imported in your own file,
and a directory containing it must be provided to protoc eg with `-I $GOPATH/src/github.com/restatedev/sdk-go/proto`

You may additionally provide the option `dev.restate.sdk.go.handler_type` to denote that the handler is a shared handler.

```proto service.proto {"CODE_LOAD::go/develop/codegen/proto/service.proto#virtual_object"}  theme={null}
import "dev/restate/sdk/go.proto";

service Counter {
  option (dev.restate.sdk.go.service_type) = VIRTUAL_OBJECT;
  rpc Add (AddRequest) returns (AddResponse) {}
  rpc Get (GetRequest) returns (GetResponse) {
    option (dev.restate.sdk.go.handler_type) = SHARED;
  }
}

service Workflow {
  option (dev.restate.sdk.go.service_type) = WORKFLOW;
  rpc Run (RunRequest) returns (RunResponse) {}
  rpc GetStatus (GetStatusRequest) returns (GetStatusResponse) {}
}
```

Clients are instantiated with a key:

```go {"CODE_LOAD::go/develop/codegen/codegen.go#virtual_object_client"}  theme={null}
client := proto.NewCounterClient(ctx, "key-1")
resp, err := client.Add().Request(&proto.AddRequest{Delta: 1})
if err != nil {
  return err
}
```

<Accordion title="Easier Protobuf with Buf">
  Instead of manually wrangling `protoc` to handle generation and proto imports,
  it can be a lot easier to use [Buf](https://buf.build/).

  Buf determines what code to generate based on a `buf.gen.yaml` file:

  ```yaml theme={null}
  version: v2
  managed:
    enabled: true
  plugins:
    - remote: buf.build/protocolbuffers/go:v1.34.2
      out: .
      opt: paths=source_relative
    - local: protoc-gen-go-restate
      out: .
      opt: paths=source_relative
  inputs:
    - directory: .
  ```

  And to allow the import of `dev/restate/sdk/go.proto`, you can specify the dependency in your `buf.yaml`:

  ```yaml theme={null}
  version: v2
  lint:
    use:
      - DEFAULT
  breaking:
    use:
      - FILE
  deps:
    - buf.build/restatedev/sdk-go
  ```

  Then you can run `buf generate` to generate the code.
</Accordion>


# Concurrent Tasks
Source: https://docs.restate.dev/develop/go/concurrent-tasks

Execute multiple tasks concurrently and gather results.

When building resilient applications, you often need to perform multiple operations in parallel to improve performance and user experience. Restate provides durable concurrency primitives that allow you to run tasks concurrently while maintaining deterministic execution during replays.

## When to use concurrent tasks

Use concurrent tasks when you need to:

* Call multiple external services simultaneously (e.g., fetching data from different APIs)
* Race multiple operations and use the first result (e.g., trying multiple LLM providers)
* Implement timeouts by racing an operation against a timer
* Perform batch operations where individual tasks can run in parallel

## Key benefits

* **Deterministic replay**: Restate logs the order of completion, ensuring consistent behavior during failures
* **Fault tolerance**: If your handler fails, tasks that were already completed will be replayed with their results, while pending tasks will be retried

## Parallelizing tasks

Start multiple durable operations concurrently by calling them without immediately awaiting:

```go {"CODE_LOAD::go/develop/journalingresults.go#parallel"}  theme={null}
call1 := restate.RunAsync(ctx, func(ctx restate.RunContext) (UserData, error) {
  return fetchUserData(123)
})
call2 := restate.RunAsync(ctx, func(ctx restate.RunContext) (OrderHistory, error) {
  return fetchOrderHistory(123)
})
call3 := restate.Service[int](ctx, "AnalyticsService", "CalculateMetrics").RequestFuture(123)

user, _ := call1.Result()
orders, _ := call2.Result()
metrics, _ := call3.Response()
```

Check out the guide on [parallelizing work](guides/parallelizing-work).

## Retrieving results

Operations such as calls, awakeables, and [`restate.After`](https://pkg.go.dev/github.com/restatedev/sdk-go#After) return futures which can
be safely waited concurrently using [`restate.Wait`](https://pkg.go.dev/github.com/restatedev/sdk-go#Wait). Restate will log the order in which they complete, to make this deterministic on replay.

<Warning>
  **Don't use Go routines**:

  Do not try to combine blocking Restate operations using goroutines, channels or select statements. These cannot be used deterministically, and will likely lead to non-determinism errors upon replay. The only place it is safe to use these types is inside of a restate.Run function.
</Warning>

### Select the first successful completion

```go {"CODE_LOAD::go/develop/journalingresults.go#race"}  theme={null}
sleepFuture := restate.After(ctx, 30*time.Second)
callFuture := restate.Service[string](ctx, "MyService", "MyHandler").RequestFuture("hi")

fut, err := restate.WaitFirst(ctx, sleepFuture, callFuture)
if err != nil {
  return "", err
}
switch fut {
case sleepFuture:
  if err := sleepFuture.Done(); err != nil {
    return "", err
  }
  return "sleep won", nil
case callFuture:
  result, err := callFuture.Response()
  if err != nil {
    return "", err
  }
  return fmt.Sprintf("call won with result: %s", result), nil
}
```

### Wait for all tasks to complete

```go {"CODE_LOAD::go/develop/journalingresults.go#all"}  theme={null}
callFuture1 := restate.Service[string](ctx, "MyService", "MyHandler").RequestFuture("hi")
callFuture2 := restate.Service[string](ctx, "MyService", "MyHandler").RequestFuture("hi again")

// Collect all results
var subResults []string
for fut, err := range restate.Wait(ctx, callFuture1, callFuture2) {
  if err != nil {
    return "", err
  }
  response, err := fut.(restate.ResponseFuture[string]).Response()
  if err != nil {
    return "", err
  }
  subResults = append(subResults, response)
}
```


# Durable Steps
Source: https://docs.restate.dev/develop/go/durable-steps

Persist results of operations.

Restate uses an execution log to replay operations after failures and suspensions. Non-deterministic operations (database calls, HTTP requests, UUID generation) must be wrapped to ensure deterministic replay.

## Run

Use `Run` to safely wrap any non-deterministic operation, like HTTP calls or database responses, and have Restate store its result in the execution log.

```go {"CODE_LOAD::go/develop/journalingresults.go#side_effect"}  theme={null}
result, err := restate.Run(ctx, func(ctx restate.RunContext) (string, error) {
  return doDbRequest()
})
if err != nil {
  return err
}
```

Note that inside `Run`, you cannot use the Restate context (e.g., `get`, `sleep`, or nested `Run`).
You should only use methods available on the [`RunContext`](https://pkg.go.dev/github.com/restatedev/sdk-go#RunContext) provided to your function.

<AccordionGroup>
  <Accordion title="Serialization">
    You can return any payload that can be serialized. By default, serialization is
    done with [`JSONCodec`](https://pkg.go.dev/github.com/restatedev/sdk-go/encoding#PayloadCodec)
    which uses `encoding/json`. If you don't need to return anything, you can use
    `restate.Void{}` which serialises to a nil byte slice.
  </Accordion>

  <Accordion title="Error handling and retry policies">
    Failures in `Run` are treated the same as any other handler error. Restate will retry it unless configured otherwise or unless a [`TerminalError`](/develop/go/error-handling) is thrown.

    You can customize how `Run` retries via:

    ```go {"CODE_LOAD::go/develop/journalingresults.go#side_effect_retry"}  theme={null}
    result, err := restate.Run(ctx,
      func(ctx restate.RunContext) (string, error) {
        return doDbRequest()
      },
      // After 10 seconds, give up retrying
      restate.WithMaxRetryDuration(time.Second*10),
      // On the first retry, wait 100 milliseconds before next attempt
      restate.WithInitialRetryInterval(time.Millisecond*100),
      // Grow retry interval with factor 2
      restate.WithRetryIntervalFactor(2.0),
      // Optional: provide a name for the operation to be visible in the
      // observability tools.
      restate.WithName("my_db_request"),
    )
    if err != nil {
      return err
    }
    ```

    * You can limit retries by time or count
    * When the policy is exhausted, a `TerminalError` is thrown
    * See the [Error Handling Guide](/guides/error-handling) and the [Sagas Guide](/guides/sagas) for patterns like compensation
  </Accordion>

  <Accordion title="Increasing timeouts">
    If Restate doesn't receive new journal entries from a service for more than one minute (by default), it will automatically suspend the invocation and retry it.

    However, some business logic can take longer to complete—for example, an LLM call that takes up to 3 minutes to respond.

    In such cases, you can adjust the service’s [inactivity timeout and abort timeout](/services/configuration) settings to accommodate longer execution times.

    For more information, see the [error handling guide](/guides/error-handling).
  </Accordion>

  <Accordion title="Context propagation">
    The Go SDK supports wrapping external contexts within Restate contexts, enabling seamless integration with libraries relying on Go's context propagation like OpenTelemetry.

    Use [`WrapContext`](https://pkg.go.dev/github.com/restatedev/sdk-go#WrapContext) to embed an external context into your Restate context:

    ```go {"CODE_LOAD::go/develop/journalingresults.go#wrap_context"}  theme={null}
    // Wrap an external context (like OpenTelemetry) into the Restate context
    externalCtx := context.Background()
    ctx = restate.WrapContext(ctx, externalCtx)

    // Use the wrapped context in Run blocks
    _, err := restate.Run(ctx, func(ctx restate.RunContext) (string, error) {
      // The external context is now available here
      return callExternalAPI(ctx)
    })
    ```

    You can also propagate custom values using [`WithValue`](https://pkg.go.dev/github.com/restatedev/sdk-go#WithValue):

    ```go {"CODE_LOAD::go/develop/journalingresults.go#with_value"}  theme={null}
    // Propagate custom values through the context
    type contextKey string
    const userIDKey contextKey = "userID"

    // Set a value in the context
    ctx = restate.WithValue(ctx, userIDKey, "user-123")

    // The value is available in Run blocks
    _, err := restate.Run(ctx, func(ctx restate.RunContext) (string, error) {
      // Extract the value using standard context APIs
      userID := ctx.Value(userIDKey).(string)
      return processWithUserID(userID)
    })
    ```

    This feature enables you to maintain context across Restate operations, preserving tracing information and custom metadata throughout your handler execution.
  </Accordion>
</AccordionGroup>

## Deterministic randoms

The SDK provides deterministic helpers for random values — seeded by the invocation ID — so they return the **same result on retries**.

### UUIDs

To generate stable UUIDs for things like idempotency keys:

```go {"CODE_LOAD::go/develop/journalingresults.go#uuid"}  theme={null}
uuid := restate.UUID(ctx)
```

Do not use this in cryptographic contexts.

### Random numbers

Methods exist on `restate.Rand(ctx)` for generating `float64` and `uint64`, or
otherwise `restate.Rand(ctx).Source()` can be provided to `math/rand/v2` as a
source for any random operation:

```go {"CODE_LOAD::go/develop/journalingresults.go#random_nb"}  theme={null}
randomInt := restate.Rand(ctx).Uint64()
randomFloat := restate.Rand(ctx).Float64()
mathRandV2 := rand.New(restate.RandSource(ctx))
```


# Scheduling & Timers
Source: https://docs.restate.dev/develop/go/durable-timers

Durable timers, scheduled actions, and sleep, backed by Restate.

Restate provides durable, fault-tolerant timers that allow you to:

* [Sleep](#durable-sleep): Pause a handler for a given duration
* [Send delayed messages](#scheduling-async-tasks): Schedule handler invocations in the future
* [Set timeouts](#timers-and-timeouts) for async operations
* Implement patterns like [cron jobs](#cron-jobs)

## How it works

Restate tracks and manages timers, ensuring they survive failures and restarts.
For example, if a service is sleeping for 12 hours and fails after 8 hours, then Restate will make sure it only sleeps 4 hours more.

If your handler runs on function-as-a-service platforms like AWS Lambda, Restate suspends the handler while it is sleeping, to free up resources.
Since these platforms charge on execution time, this saves costs.

## Durable sleep

To pause a handler for a set duration:

```go {"CODE_LOAD::go/develop/durabletimers.go#here"}  theme={null}
if err := restate.Sleep(ctx, 10*time.Second); err != nil {
  return "", err
}
```

There is no hard limit on how long you can sleep. You can sleep for months or even years, but keep in mind:

* If you sleep in an [exclusive handler](/foundations/handlers#handler-behavior) in a Virtual Object, all other calls to this object will be queued.
* You need to keep the deployment version until the invocation completes (see [versioning](/services/versioning)).
  So for long sleeps, we recommend breaking this up into multiple handlers that call each other with [delayed messages](/develop/go/service-communication#delayed-messages).

<Accordion title="Clock synchronization Restate Server vs. SDK">
  The Restate SDK calculates the wake-up time based on the delay you specify.
  The Restate Server then uses this calculated time to wake up the handler.
  If the Restate Server and the SDK have different system clocks, the sleep duration might not be accurate.
  So make sure that the system clock of the Restate Server and the SDK have the same timezone and are synchronized.
  A mismatch can cause timers to fire earlier or later than expected.
</Accordion>

## Scheduling async tasks

To invoke a handler at a later time, use [delayed messages](/develop/go/service-communication#delayed-messages).

<Accordion title="Delayed messages vs. sleep and send">
  In theory, you can schedule future work in Restate in two ways:

  1. **Delayed messages** (recommended)
  2. **Sleep + send** - sleeping in the current handler, then sending a message

  At first sight, both approaches might seem to achieve similar results.
  However, we recommend to use delayed messages for the following reasons:

  * **Handler completes immediately**: The calling handler can finish execution and complete the invocation without waiting for the delay to finish
  * **No Virtual Object blocking**: Other requests to the same Virtual Object can be processed during the delay period
  * **Better for service versioning**: No long-running invocations that require keeping old service deployments around for a long time (see [service versioning](/services/versioning))
</Accordion>

## Timers and timeouts

Most context actions are async actions that return a future.

You can use [Restate's combinators](/develop/go/concurrent-tasks) to race an async operation against a timeout:

```go {"CODE_LOAD::go/develop/durabletimers.go#timer"}  theme={null}
sleepFuture := restate.After(ctx, 30*time.Second)
callFuture := restate.Service[string](ctx, "MyService", "MyHandler").RequestFuture("hi")

fut, err := restate.WaitFirst(ctx, sleepFuture, callFuture)
if err != nil {
  return "", err
}
switch fut {
case sleepFuture:
  if err := sleepFuture.Done(); err != nil {
    return "", err
  }
  return "sleep won", nil
case callFuture:
  result, err := callFuture.Response()
  if err != nil {
    return "", err
  }
  return fmt.Sprintf("call won with result: %s", result), nil
}
```

## Cron jobs

Restate does not yet include native cron support, but you can implement your own cron scheduler using:

* Durable timers
* Virtual Objects
* A repeat loop or sleep-schedule pattern

Check out the guide on [implementing cron jobs](/guides/cron).


# Error Handling
Source: https://docs.restate.dev/develop/go/error-handling

Stop infinite retries with Terminal Errors.

Restate handles retries for failed invocations.

<Info>
  Check out the [Error Handling guide](/guides/error-handling) to learn more about how Restate handles transient errors, terminal errors, retries, and timeouts.
</Info>

## Retry strategies

By default, Restate does infinite retries with an exponential backoff strategy.

Check out the [error handling guide](/guides/error-handling) to learn how to customize this.

## Terminal Errors

For failures for which you do not want retries, but instead want the invocation to end and the error message
to be propagated back to the caller, you can throw a **terminal error**.

You can throw a `TerminalError` with an optional HTTP status code and a message anywhere in your handler, as follows:

```go {"CODE_LOAD::go/develop/errorhandling.go#here"}  theme={null}
return restate.TerminalError(fmt.Errorf("Something went wrong."), 500)
```

You can catch terminal errors, and build your control flow around it.

<Info>
  When you throw a terminal error, you might need to undo the actions you did earlier in your handler to make sure that your system remains in a consistent state.
  Have a look at our [sagas guide](/guides/sagas) to learn more.
</Info>


# External Events
Source: https://docs.restate.dev/develop/go/external-events

Handle external events and human-in-the-loop patterns with durable waiting primitives.

Sometimes your handlers need to pause and wait for external processes to complete. This is common in:

* **Human-in-the-loop workflows** (approvals, reviews, manual steps)
* **External system integration** (waiting for webhooks, async APIs)
* **AI agent patterns** (tool execution, human oversight)

This pattern is also known as the **callback** or **task token** pattern.

## Two Approaches

Restate provides two primitives for handling external events:

| Primitive            | Use Case                   | Key Feature                          |
| -------------------- | -------------------------- | ------------------------------------ |
| **Awakeables**       | Services & Virtual Objects | Unique ID-based completion           |
| **Durable Promises** | Workflows only             | Named promises for simpler signaling |

## How it works

Implementing this pattern in a distributed system is tricky, since you need to ensure that the handler can recover from failures and resume waiting for the external event.

Restate promises are durable and distributed. They survive crashes and can be resolved or rejected by any handler in the workflow.

To save costs on FaaS deployments, Restate lets the handler [suspend](/foundations/key-concepts#suspensions-on-faas) while awaiting the promise, and invokes it again when the result is available.

## Awakeables

**Best for:** Services and Virtual Objects where you need to coordinate with external systems.

### Creating and waiting for awakeables

1. **Create an awakeable** - Get a unique ID and promise
2. **Send the ID externally** - Pass the awakeable ID to your external system
3. **Wait for result** - Your handler [suspends](/foundations/key-concepts#suspensions-on-faas) until the external system responds

```go {"CODE_LOAD::go/develop/awakeable.go#here"}  theme={null}
awakeable := restate.Awakeable[string](ctx)
awakeableId := awakeable.Id()

if _, err := restate.Run(ctx, func(ctx restate.RunContext) (string, error) {
  return requestHumanReview(awakeableId)
}); err != nil {
  return err
}

review, err := awakeable.Result()
if err != nil {
  return err
}
```

<Accordion title="Serialization">
  You can resolve an awakeable with any payload that can be serialized. By default,
  serialization is done with [`JSONCodec`](https://pkg.go.dev/github.com/restatedev/sdk-go/encoding#PayloadCodec)
  which uses `encoding/json`. If you don't need to provide anything, you can
  use `restate.Void{}` which serializes to a nil byte slice.
</Accordion>

<Info>
  Note that if you wait for an awakeable in an [exclusive handler](/foundations/handlers#handler-behavior) in a Virtual Object, all other calls to this object will be queued.
</Info>

### Resolving/rejecting Awakeables

External processes complete awakeables in two ways:

* **Resolve** with success data → handler continues normally
* **Reject** with error reason → throws a [terminal error](/develop/go/error-handling) in the waiting handler

#### Via SDK (from other handlers)

**Resolve:**

```go {"CODE_LOAD::go/develop/awakeable.go#resolve"}  theme={null}
restate.ResolveAwakeable(ctx, awakeableId, "Looks good!")
```

**Reject:**

```go {"CODE_LOAD::go/develop/awakeable.go#reject"}  theme={null}
restate.RejectAwakeable(ctx, awakeableId, fmt.Errorf("Cannot do review"))
```

#### Via HTTP API

External systems can complete awakeables using Restate's HTTP API:

**Resolve with data:**

```shell theme={null}
curl localhost:8080/restate/awakeables/sign_1PePOqp/resolve \
  --json '"Looks good!"'
```

**Reject with error:**

```shell theme={null}
curl localhost:8080/restate/awakeables/sign_1PePOqp/reject \
  -H 'content-type: text/plain' \
  -d 'Review rejected: insufficient documentation'
```

## Durable Promises

**Best for:** Workflows where you need to signal between different workflow handlers.

**Key differences from awakeables:**

* No ID management - use logical names instead
* Scoped to workflow execution lifetime

Use this for:

* Sending data to the run handler
* Have handlers wait for events emitted by the run handler

<Info>
  After a workflow's run handler completes, other handlers can still be called for up to 24 hours (default).
  The results of resolved Durable Promises remain available during this time.
  Update the retention time via the [service configuration](/services/configuration).
</Info>

### Creating and waiting for promises

Wait for a promise by name:

```go {"CODE_LOAD::go/develop/durablepromise.go#promise"}  theme={null}
review, err := restate.Promise[string](ctx, "review").Result()
```

### Resolving/rejecting promises

Resolve/reject from any workflow handler:

```go {"CODE_LOAD::go/develop/durablepromise.go#resolve_promise"}  theme={null}
err := restate.Promise[string](ctx, "review").Resolve(review)
if err != nil {
  return err
}
```

### Complete workflow example

```go expandable {"CODE_LOAD::go/develop/durablepromise.go#review"}  theme={null}
type ReviewWorkflow struct{}

func (ReviewWorkflow) Run(ctx restate.WorkflowContext, documentId string) (string, error) {
  // Send document for review
  if _, err := restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
    return restate.Void{}, askReview(documentId)
  }); err != nil {
    return "", err
  }

  // Wait for external review submission
  review, err := restate.Promise[string](ctx, "review").Result()
  if err != nil {
    return "", err
  }

  // Process the review result
  return processReview(documentId, review)
}

func (ReviewWorkflow) SubmitReview(ctx restate.WorkflowSharedContext, review string) error {
  // Signal the waiting run handler
  return restate.Promise[string](ctx, "review").Resolve(review)
}
```

### Two signaling patterns

**External → Workflow** (shown above): External handlers signal the run handler

* Use for human approvals, external API responses, manual interventions
* External handlers call the handler which resolves the promise

**Workflow → External**: Run handler signals other handlers waiting for workflow events

* Use for step completion notifications, status updates, result broadcasting
* Run handler resolves promises that external handlers are awaiting

## Best Practices

* **Use awakeables** for services/objects coordinating with external systems
* **Use durable promises** for workflow signaling
* **Always handle rejections** to gracefully manage failures
* **Include timeouts** for long-running external processes


# Logging
Source: https://docs.restate.dev/develop/go/logging

Configure the log level of your services.

The Go SDK uses the standard library [`log/slog`](https://pkg.go.dev/log/slog)
package for logging.

By default logs will go to the slog default handler. However, you may provide
any slog handler upon creating the Restate server instance, which allows you to
direct the logs to other log libraries:

```go {"CODE_LOAD::go/develop/logging.go#handler"}  theme={null}
myHandler := slog.NewJSONHandler(os.Stdout, nil)
server.NewRestate().
  WithLogger(myHandler, true).
  Bind(restate.Reflect(Monitoring{})).
  Start(context.Background(), "0.0.0.0:9080")
```

**Avoiding duplicate logging**

If you use the default slog logger or another logger, log statements will be
printed over and over again during replays.
To avoid this, you can use the Restate context logger, which is a
`*slog.Logger` that suppresses duplicate log statements during replays:

```go {"CODE_LOAD::go/develop/logging.go#logger"}  theme={null}
ctx.Log().Info("This will not be printed again during replays")
ctx.Log().Debug("This will not be printed again during replays")
```

When providing a custom log handler using
[`WithLogger`](https://pkg.go.dev/github.com/restatedev/sdk-go/server#Restate.WithLogger),
you can provide `false` as the second argument, in which case logs will not be
dropped during replay allowing you to handle them as you prefer. You can still
determine whether they would have been dropped using
[`rcontext.LogContextFrom`](https://pkg.go.dev/github.com/restatedev/sdk-go/rcontext#LogContextFrom)
on the context passed to your log handler.


# Service Communication
Source: https://docs.restate.dev/develop/go/service-communication

Call other services from your handler.

Your Restate handler can call other handlers in three ways:

* **[Request-response calls](#request-response-calls)**: Call and wait for a response
* **[One-way messages](#sending-messages)**: Send a message and continue
* **[Delayed messages](#delayed-messages)**: Send a message after a delay

<Info>
  To call a service from an external application, see the [HTTP](/services/invocation/http), [Kafka](/services/invocation/kafka), or [SDK Clients](/services/invocation/clients/typescript-sdk) documentation.
</Info>

<Info>[Learn how Restate how it works](/foundations/key-concepts#resilient-communication)</Info>

## Request-response calls

To call a Restate handler and wait for its result:

```go {"CODE_LOAD::go/develop/servicecommunication.go#request_response"}  theme={null}
// To call a Service:
svcResponse, err := restate.Service[string](ctx, "MyService", "MyHandler").
  Request("Hi")
if err != nil {
  return err
}

// To call a Virtual Object:
objResponse, err := restate.Object[string](ctx, "MyObject", "Mary", "MyHandler").
  Request("Hi")
if err != nil {
  return err
}

// To call a Workflow:
// `run` handler — can only be called once per workflow ID
wfResponse, err := restate.Workflow[bool](ctx, "MyWorkflow", "my-workflow-id", "Run").
  Request("Hi")
if err != nil {
  return err
}
// Other handlers can be called anytime within workflow retention
status, err := restate.Workflow[restate.Void](ctx, "MyWorkflow", "my-workflow-id", "GetStatus").
  Request("Hi again")
if err != nil {
  return err
}
```

<Accordion title="Workflow retention">
  After a workflow's run handler completes, other handlers can still be called for up to 24 hours (default).
  Update this via the [service configuration](/services/configuration).
</Accordion>

<Info>
  Request-response calls between [exclusive handlers](/foundations/handlers#handler-behavior) of Virtual Objects may lead to deadlocks:

  * Cross deadlock: A → B and B → A (same keys).
  * Cycle deadlock: A → B → C → A.

  Use the UI or CLI to [cancel](/services/invocation/managing-invocations#cancel) and unblock deadlocked invocations.
</Info>

## Sending messages

To send a message to another Restate handler without waiting for a response:

```go {"CODE_LOAD::go/develop/servicecommunication.go#one_way"}  theme={null}
// To message a Service:
restate.ServiceSend(ctx, "MyService", "MyHandler").Send("Hi")

// To message a Virtual Object:
restate.ObjectSend(ctx, "MyObject", "Mary", "MyHandler").Send("Hi")

// To message a Workflow:
// `run` handler — can only be called once per workflow ID
restate.WorkflowSend(ctx, "MyWorkflow", "my-workflow-id", "Run").
  Send("Hi")
// Other handlers can be called anytime within workflow retention
restate.WorkflowSend(ctx, "MyWorkflow", "my-workflow-id", "InteractWithWorkflow").
  Send("Hi again")
```

Restate handles message delivery and retries, so the handler can complete and return without waiting for the message to be processed.

<Info>
  Calls to a Virtual Object execute in order of arrival, serially.
  Example:

  ```go {"CODE_LOAD::go/develop/servicecommunication.go#ordering"}  theme={null}
  restate.ObjectSend(ctx, "MyService", "Mary", "MyHandler").Send("I'm call A")
  restate.ObjectSend(ctx, "MyService", "Mary", "MyHandler").Send("I'm call B")
  ```

  Call A is guaranteed to execute before B. However, other invocations may interleave between A and B.
</Info>

## Delayed messages

To send a message after a delay:

```go {"CODE_LOAD::go/develop/servicecommunication.go#delayed"}  theme={null}
// To message a Service with a delay:
restate.ServiceSend(ctx, "MyService", "MyHandler").
  Send("Hi", restate.WithDelay(5*time.Hour))

// To message a Virtual Object with a delay:
restate.ObjectSend(ctx, "MyObject", "Mary", "MyHandler").
  Send("Hi", restate.WithDelay(5*time.Hour))

// To message a Workflow with a delay:
restate.WorkflowSend(ctx, "MyWorkflow", "my-workflow-id", "Run").
  Send("Hi", restate.WithDelay(5*time.Hour))
```

<Info>
  Learn [how this is different](/develop/ts/durable-timers#scheduling-async-tasks) from sleeping and then sending a message.
</Info>

## Using an idempotency key

To prevent duplicate executions of the same call, add an idempotency key:

```go {"CODE_LOAD::go/develop/servicecommunication.go#idempotency_key"}  theme={null}
restate.
  ServiceSend(ctx, "MyService", "MyHandler").
  // Send attaching idempotency key
  Send("Hi", restate.WithIdempotencyKey("my-idempotency-key"))
```

Restate automatically deduplicates calls made during the same handler execution, so there's no need to provide an idempotency key in that case.
However, if multiple handlers might call the same service independently, you can use an idempotency key to ensure deduplication across those calls.

## Attach to an invocation

To wait for or get the result of a previously sent message:

```go {"CODE_LOAD::go/develop/servicecommunication.go#attach"}  theme={null}
// Execute the request and retrieve the invocation id
invocationId := restate.
  ServiceSend(ctx, "MyService", "MyHandler").
  // Optional: send attaching idempotency key
  Send("Hi", restate.WithIdempotencyKey("my-idempotency-key")).
  GetInvocationId()

// Later re-attach to the request
response, err := restate.AttachInvocation[string](ctx, invocationId).Response()
```

* With an idempotency key: Wait for completion and retrieve the result.
* Without an idempotency key: Can only wait, not retrieve the result.

## Cancel an invocation

To cancel a running handler:

```go {"CODE_LOAD::go/develop/servicecommunication.go#cancel"}  theme={null}
// Execute the request and retrieve the invocation id
invocationId := restate.
  ServiceSend(ctx, "MyService", "MyHandler").
  Send("Hi").
  GetInvocationId()

// I don't need this invocation anymore, let me just cancel it
restate.CancelInvocation(ctx, invocationId)
```

## See also

* **[SDK Clients](/develop/go/service-communication)**: Call Restate services from external applications
* **[Error Handling](/develop/go/error-handling)**: Handle failures and terminal errors in service calls
* **[Durable Timers](/develop/go/durable-timers)**: Implement timeouts for your service calls
* **[Sagas](/guides/sagas)**: Roll back or compensate for canceled service calls.


# Services
Source: https://docs.restate.dev/develop/go/services

Implementing Restate services with the Go SDK.

The Restate Go SDK is open source and available on [GitHub](https://github.com/restatedev/sdk-go).

The Restate SDK lets you implement **handlers**. Handlers can be part of a **[Basic Service](/foundations/services#basic-service)**, a **[Virtual Object](/foundations/services#virtual-object)**, or a **[Workflow](/foundations/services#workflow)**. This page shows how to define them with the Go SDK.

## Prerequisites

* Go: >= 1.21.0

## Getting started

<Tip>
  Get started quickly with the [Go Quickstart](/quickstart#go).
</Tip>

Add the `github.com/restatedev/sdk-go` dependency to your project to start developing Restate services.

## Basic Services

[Basic Services](/foundations/services) group related **handlers** and expose them as callable endpoints:

```go {"CODE_LOAD::go/develop/myservice/main.go?collapse_prequel"}  theme={null}
type MyService struct{}

func (MyService) MyHandler(ctx restate.Context, greeting string) (string, error) {
  return fmt.Sprintf("%s!", greeting), nil
}

func main() {
  if err := server.NewRestate().
    Bind(restate.Reflect(MyService{})).
    Start(context.Background(), "0.0.0.0:9080"); err != nil {
    log.Fatal(err)
  }
}
```

* Define a Service by implementing exported handlers on any struct.
* Handlers have the `Context` as the first argument.
  Within the handler, you use the `Context` to interact with Restate.
  The SDK stores the actions you do on the context in the Restate journal to make them durable.
* The handler input parameters and return type can be of any type, as long as
  they can be serialized. By default, serialization is done with [`JSONCodec`](https://pkg.go.dev/github.com/restatedev/sdk-go/encoding#PayloadCodec)
  which uses `encoding/json`. Input types, output types, and even errors are not mandatory
  and can be omitted if your handler doesn't need them.
* The service will be reachable under the struct name `MyService`, so the service can be
  called at `<RESTATE_INGRESS_URL>/MyService/MyHandler`.
* Pass the `MyService` struct to `restate.Reflect` which uses reflection to turn
  the methods into handlers. It will skip unexported methods and those that
  don't have the expected signature - see the [package documentation](https://pkg.go.dev/github.com/restatedev/sdk-go#Reflect)
  for a list of allowed signatures.
* Finally, create a server listening on the specified address and bind the
  service(s) to it.

<Tip>
  The Go SDK also supports defining handlers and input/output types using
  code generation from [Protocol Buffers](https://protobuf.dev/). See
  [Code Generation](/develop/go/code-generation) for more details.
</Tip>

## Virtual Objects

[Virtual Objects](/foundations/services) are services that are stateful and key-addressable — each object instance has a unique ID and persistent state.

```go {"CODE_LOAD::go/develop/myvirtualobject/main.go?collapse_prequel"}  theme={null}
type MyObject struct{}

func (MyObject) MyHandler(ctx restate.ObjectContext, greeting string) (string, error) {
  return fmt.Sprintf("%s %s!", greeting, restate.Key(ctx)), nil
}

func (MyObject) MyConcurrentHandler(ctx restate.ObjectSharedContext, greeting string) (string, error) {
  return fmt.Sprintf("%s %s!", greeting, restate.Key(ctx)), nil
}

func main() {
  if err := server.NewRestate().
    Bind(restate.Reflect(MyObject{})).
    Start(context.Background(), "0.0.0.0:9080"); err != nil {
    log.Fatal(err)
  }
}
```

* Define a Virtual Object by implementing exported handlers on any struct.
* You can retrieve the key of the object you are in via `restate.Key(ctx)`.
* Virtual Objects can have [exclusive and shared handlers](/foundations/handlers#handler-behavior).

- Exclusive handlers receive an `ObjectContext`, allowing read/write access to object state.
- Shared handlers are wrapped in `handlers.object.shared(...)` and use the `ObjectSharedContext`

## Workflows

[Workflows](/foundations/services) are long-lived processes with a defined lifecycle. They run once per key and are ideal for orchestrating multi-step operations, which require external interaction via signals and queries.

```go {"CODE_LOAD::go/develop/myworkflow/main.go?collapse_prequel"}  theme={null}
type MyWorkflow struct{}

func (MyWorkflow) Run(ctx restate.WorkflowContext, req string) (string, error) {
  // implement the workflow logic here
  return "success", nil
}

func (MyWorkflow) InteractWithWorkflow(ctx restate.WorkflowSharedContext) error {
  // implement interaction logic here
  // e.g. resolve a promise that the workflow is waiting on
  return nil
}

func main() {
  server := server.NewRestate().
    Bind(restate.Reflect(MyWorkflow{}))

  if err := server.Start(context.Background(), ":9080"); err != nil {
    slog.Error("application exited unexpectedly", "err", err.Error())
    os.Exit(1)
  }
}
```

* Define a Workflow by implementing exported handlers on any struct.

- Every workflow **must** include a `Run` handler:
  * This is the main orchestration entry point
  * It runs exactly once per workflow execution and uses the `WorkflowContext`
  * Resubmission of the same workflow will fail with "Previously accepted". The invocation ID can be found in the request header `x-restate-id`.
  * Use `restate.Key(ctx)` to access the workflow's unique ID
- Additional handlers must use the `WorkflowSharedContext` and can signal or query the workflow. They can run concurrently with the `Run` handler and until the retention time expires.

## Configuring services

Check out the [service configuration docs](/services/configuration) to learn how to configure service behavior, including timeouts and retention policies.


# Serving
Source: https://docs.restate.dev/develop/go/serving

Create an endpoint to serve your services.

The Restate SDK is served as a HTTP handler. You can choose to run this in a
standalone HTTP2 server, or connect the handler into a FaaS system like Lambda.

## Creating a HTTP2 server

1. Create the Restate Server
2. Bind one or multiple services to it.
3. Listen on the specified port for connections and requests.

```go {"CODE_LOAD::go/develop/serving.go#endpoint"}  theme={null}
if err := server.NewRestate().
  Bind(restate.Reflect(MyService{})).
  Bind(restate.Reflect(MyObject{})).
  Bind(restate.Reflect(MyWorkflow{})).
  Start(context.Background(), ":9080"); err != nil {
  log.Fatal(err)
}
```

<Accordion title="Customizing the HTTP2 server">
  If you need to customize the HTTP2 server, or serve over HTTP1.1
  you can call `.Handler()` instead of `Start()`, and then use the
  handler as normal. To discover services over HTTP1.1 you must
  provide the `--use-http1.1` CLI flag.

  ```go {"CODE_LOAD::go/develop/serving.go#custom_endpoint"}  theme={null}
  handler, err := server.NewRestate().
  Bind(restate.Reflect(MyService{})).
  Bind(restate.Reflect(MyObject{})).
  Bind(restate.Reflect(MyWorkflow{})).
  Handler()
  if err != nil {
  log.Fatal(err)
  }
  ```

  By default, this handler will advertise itself as working
  bidirectionally; the SDK will try to get completions from the runtime
  during execution.

  However, you can use the method `.Bidirectional(false)` on the endpoint
  builder to change this on platforms that do not support bidirectional
  communication, such as Lambda. If you don't do this your handler may get
  stuck.
</Accordion>

## Creating a Lambda handler

To register your service as a Lambda function change the endpoint into a Lambda handler
with `.LambdaHandler()`, and pass this handler to [`lambda.Start`](https://pkg.go.dev/github.com/aws/aws-lambda-go/lambda#Start).

```go {"CODE_LOAD::go/develop/servinglambda.go#lambda"}  theme={null}
handler, err := server.NewRestate().
  Bind(restate.Reflect(MyService{})).
  Bind(restate.Reflect(MyObject{})).
  Bind(restate.Reflect(MyWorkflow{})).
  Bidirectional(false).
  LambdaHandler()
if err != nil {
  log.Fatal(err)
}
lambda.Start(handler)
```

Have a look at the [deployment section](/services/deploy/lambda) for guidance on how to deploy your services on AWS Lambda.

The implementation of your services and handlers remains the same for both deployment options.

## Validating request identity

SDKs can validate that incoming requests come from a particular Restate
instance. You can find out more about request identity in the [Security docs](/services/security#locking-down-service-access)

```go {"CODE_LOAD::go/develop/serving.go#identity"}  theme={null}
if err := server.NewRestate().
  Bind(restate.Reflect(MyService{})).
  WithIdentityV1("publickeyv1_w7YHemBctH5Ck2nQRQ47iBBqhNHy4FV7t2Usbye2A6f").
  Start(context.Background(), ":9080"); err != nil {
  log.Fatal(err)
}
```


# State
Source: https://docs.restate.dev/develop/go/state

Store key-value state in Restate.

Restate lets you persist key-value (K/V) state using its embedded K/V store.

## Key characteristics

<Info>
  [Learn about Restate's embedded K/V store](/foundations/key-concepts#consistent-state).
</Info>

**State is only available for Virtual Objects and Workflows.**

Scope & retention:

* For Virtual Objects: State is scoped per object key and retained indefinitely. It is persisted and shared across all invocations for that object until explicitly cleared.
* For Workflows: State is scoped per workflow execution (workflow ID) and retained only for the duration of the workflow’s configured retention time.

Access Rules:

* [Exclusive handlers](/foundations/handlers#handler-behavior) (e.g., `run()` in workflows) can read and write state.
* [Shared handlers](/foundations/handlers#handler-behavior) can only read state and cannot mutate it.

You can inspect and edit the K/V state via the UI and the [CLI](/services/introspection#inspecting-application-state).

## List all state keys

To retrieve all keys for which the current Virtual Object has stored state:

```go {"CODE_LOAD::go/develop/state.go#statekeys"}  theme={null}
stateKeys, err := restate.Keys(ctx)
if err != nil {
  return err
}
```

## Get state value

To read a value by key:

```go {"CODE_LOAD::go/develop/state.go#get"}  theme={null}
myString := "my-default"
if s, err := restate.Get[*string](ctx, "my-string-key"); err != nil {
  return err
} else if s != nil {
  myString = *s
}

myNumber, err := restate.Get[int](ctx, "my-number-key")
if err != nil {
  return err
}
```

Returns null if the key doesn't exist.

## Set state value

To write or update a value:

```go {"CODE_LOAD::go/develop/state.go#set"}  theme={null}
restate.Set(ctx, "my-key", "my-new-value")
```

## Clear state key

To delete a specific key:

```go {"CODE_LOAD::go/develop/state.go#clear"}  theme={null}
restate.Clear(ctx, "my-key")
```

## Clear all state keys

To remove all stored state for the current Virtual Object:

```go {"CODE_LOAD::go/develop/state.go#clear_all"}  theme={null}
restate.ClearAll(ctx)
```

## Advanced: Eager vs. lazy state loading

Restate supports two modes for loading state in handlers:

### Eager state (default)

* **How it works**: State is automatically sent with the request when invoking a handler
* **Benefits**: State is available immediately when the handler starts executing
* **Behavior**: All reads and writes to state are local to the handler execution
* **Best for**: Small to medium state objects that are frequently accessed

### Lazy state

* **How it works**: State is fetched on-demand on `get` calls from the Restate Server
* **Benefits**: Reduces initial request size and memory usage
* **Setup**: Enable lazy state in the [service or handler configuration](/services/configuration)
* **Best for**: Large state objects that aren't needed in every handler execution


# Testing
Source: https://docs.restate.dev/develop/go/testing

Utilities to test your handler logic.

The Go SDK provides some testing utilities to test your Restate handlers.

This uses [Testcontainers](https://testcontainers.com/) to run a Restate Server in a Docker container and provides a client to let you test your Restate handlers.

## Testing handlers

If you have a service as follows:

```go {"CODE_LOAD::go/develop/testing/svc.go#here"}  theme={null}
type Greeter struct{}

func (Greeter) Greet(ctx restate.Context, req string) (string, error) {
  return fmt.Sprintf("Hello %s!", req), nil
}
```

Then you can test it via:

```go {"CODE_LOAD::go/develop/testing/t.go#here"}  theme={null}
import (
  "testing"

  restate "github.com/restatedev/sdk-go"
  restateingress "github.com/restatedev/sdk-go/ingress"
  restatetest "github.com/restatedev/sdk-go/testing"
  "github.com/stretchr/testify/require"
)

func TestWithTestcontainers(t *testing.T) {
  tEnv := restatetest.Start(t, restate.Reflect(Greeter{}))
  client := tEnv.Ingress()

  out, err := restateingress.Service[string, string](client, "Greeter", "Greet").
    Request(t.Context(), "Francesco")
  require.NoError(t, err)
  require.Equal(t, "You said hi to Francesco!", out)
}
```


# Concurrent Tasks
Source: https://docs.restate.dev/develop/java/concurrent-tasks

Execute multiple tasks concurrently and gather results.

When building resilient applications, you often need to perform multiple operations in parallel to improve performance and user experience. Restate provides durable concurrency primitives that allow you to run tasks concurrently while maintaining deterministic execution during replays.

## When to use concurrent tasks

Use concurrent tasks when you need to:

* Call multiple external services simultaneously (e.g., fetching data from different APIs)
* Race multiple operations and use the first result (e.g., trying multiple LLM providers)
* Implement timeouts by racing an operation against a timer
* Perform batch operations where individual tasks can run in parallel

## Key benefits

* **Deterministic replay**: Restate logs the order of completion, ensuring consistent behavior during failures
* **Fault tolerance**: If your handler fails, tasks that were already completed will be replayed with their results, while pending tasks will be retried

## Parallelizing tasks

Start multiple durable operations concurrently by calling them without immediately awaiting:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/JournalingResults.java#parallel"}  theme={null}
  // Start operations concurrently using DurableFuture
  var call1 = ctx.runAsync(UserData.class, () -> fetchUserData(123));
  var call2 = ctx.runAsync(OrderHistory.class, () -> fetchOrderHistory(123));
  var call3 = AnalyticsServiceClient.fromContext(ctx).calculateMetric(123);

  // Now wait for results as needed
  UserData user = call1.await();
  OrderHistory orders = call2.await();
  Integer metric = call3.await();
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/JournalingResults.kt#parallel"}  theme={null}
  val call1 = ctx.runAsync<UserData> { fetchUserData(123) }
  val call2 = ctx.runAsync<OrderHistory> { fetchOrderHistory(123) }
  val call3 = AnalyticsServiceClient.fromContext(ctx).calculateMetric(123)

  // Now wait for results as needed
  val user: UserData = call1.await()
  val orders: OrderHistory = call2.await()
  val metric: Int = call3.await()
  ```
</CodeGroup>

Check out the guide on [parallelizing work](guides/parallelizing-work).

## Retrieving results

Restate provides several patterns for coordinating concurrent tasks. All patterns use `DurableFuture` combinators that log the order of completion, ensuring deterministic behavior during replays.

### Wait for the first completion

**Select** creates a `DurableFuture` that returns on the first completed `DurableFuture` of the provided ones.
The semantics are similar to [`CompleteableFuture.anyOf()`](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/concurrent/CompletableFuture.html#anyOf\(java.util.concurrent.CompletableFuture...\)), but the outcome is stored in the Restate journal to be deterministically replayable.

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/JournalingResults.java#combine_any"}  theme={null}
  boolean res = Select.<Boolean>select().or(a1).or(a2).or(a3).await();
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/JournalingResults.kt#combine_any"}  theme={null}
  val resSelect =
  select {
        a1.onAwait { it }
        a2.onAwait { it }
        a3.onAwait { it }
      }
      .await()
  ```
</CodeGroup>

### Wait for all tasks to complete

**Await all** creates a `DurableFuture` that awaits for all the provided DurableFutures to resolve.
The semantics are similar to [`CompleteableFuture.allOf()`](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/concurrent/CompletableFuture.html#allOf\(java.util.concurrent.CompletableFuture...\)), but the outcome is stored in the Restate journal to be deterministically replayable.

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/JournalingResults.java#combine_all"}  theme={null}
  DurableFuture.all(a1, a2, a3).await();
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/JournalingResults.kt#combine_all"}  theme={null}
  listOf(a1, a2, a3).awaitAll()
  ```
</CodeGroup>


# Durable Steps
Source: https://docs.restate.dev/develop/java/durable-steps

Persist results of operations.

Restate uses an execution log to replay operations after failures and suspensions. Non-deterministic operations (database calls, HTTP requests, UUID generation) must be wrapped to ensure deterministic replay.

## Run

Use `ctx.run` to safely wrap any non-deterministic operation, like HTTP calls or database responses, and have Restate store its result in the execution log.

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/JournalingResults.java#side_effect"}  theme={null}
  String output = ctx.run(String.class, () -> doDbRequest());
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/JournalingResults.kt#side_effect"}  theme={null}
  val output = ctx.runBlock { doDbRequest() }
  ```
</CodeGroup>

Note that inside `ctx.run`, you cannot use the Restate context (e.g., `ctx.get`, `ctx.sleep`, or nested `ctx.run`).

<AccordionGroup>
  <Accordion title="Asynchronous run actions">
    To run an action asynchronously, use `ctx.runAsync`:

    <CodeGroup>
      ```java Java {"CODE_LOAD::java/src/main/java/develop/JournalingResults.java#async_side_effect"}  theme={null}
      DurableFuture<String> myRunFuture = ctx.runAsync(String.class, () -> doSomethingSlow());
      ```

      ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/JournalingResults.kt#async_side_effect"}  theme={null}
      val myRunFuture = ctx.runAsync { doSomethingSlow() }
      ```
    </CodeGroup>

    You can use this to combine your run actions with other asynchronous operations, like waiting for a timer or an awakeable.

    Check out [Concurrent tasks](/develop/java/concurrent-tasks).
  </Accordion>

  <Accordion title="Serialization">
    Have a look at the [serialization docs](/develop/java/serialization) to learn how to serialize and deserialize your objects.
  </Accordion>

  <Accordion title="Error handling and retry policies">
    Failures in `ctx.run` are treated the same as any other handler error. Restate will retry it unless configured otherwise or unless a [`TerminalException`](/develop/java/error-handling) is thrown.

    You can customize how `ctx.run` retries via:

    <CodeGroup>
      ```java Java {"CODE_LOAD::java/src/main/java/develop/RetryRunService.java#here"}  theme={null}
      try {
        RetryPolicy myRunRetryPolicy =
            RetryPolicy.defaultPolicy()
                .setInitialDelay(Duration.ofMillis(500))
                .setExponentiationFactor(2)
                .setMaxDelay(Duration.ofSeconds(10))
                .setMaxAttempts(10)
                .setMaxDuration(Duration.ofMinutes(5));
        ctx.run("my-run", myRunRetryPolicy, () -> writeToOtherSystem());
      } catch (TerminalException e) {
        // Handle the terminal error after retries exhausted
        // For example, undo previous actions (see sagas guide) and
        // propagate the error back to the caller
      }
      ```

      ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/RetryRunService.kt#here"}  theme={null}
      try {
        val myRunRetryPolicy =
            RetryPolicy(
                initialDelay = 5.seconds,
                exponentiationFactor = 2.0f,
                maxDelay = 60.seconds,
                maxAttempts = 10,
                maxDuration = 5.minutes)
        ctx.runBlock("write", myRunRetryPolicy) { writeToOtherSystem() }
      } catch (e: TerminalException) {
        // Handle the terminal error after retries exhausted
        // For example, undo previous actions (see sagas guide) and
        // propagate the error back to the caller
        throw e
      }
      ```
    </CodeGroup>

    * You can limit retries by time or count
    * When the policy is exhausted, a `TerminalException` is thrown
    * See the [Error Handling Guide](/guides/error-handling) and the [Sagas Guide](/guides/sagas) for patterns like compensation
  </Accordion>

  <Accordion title="Increasing timeouts">
    If Restate doesn't receive new journal entries from a service for more than one minute (by default), it will automatically suspend the invocation and retry it.

    However, some business logic can take longer to complete—for example, an LLM call that takes up to 3 minutes to respond.

    In such cases, you can adjust the service’s [inactivity timeout and abort timeout](/services/configuration) settings to accommodate longer execution times.

    For more information, see the [error handling guide](/guides/error-handling).
  </Accordion>
</AccordionGroup>

## Deterministic randoms

The SDK provides deterministic helpers for random values — seeded by the invocation ID — so they return the **same result on retries**.

### UUIDs

To generate stable UUIDs for things like idempotency keys:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/JournalingResults.java#uuid"}  theme={null}
  UUID uuid = ctx.random().nextUUID();
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/JournalingResults.kt#uuid"}  theme={null}
  val uuid = ctx.random().nextUUID()
  ```
</CodeGroup>

Do not use this in cryptographic contexts.

### Random numbers

To generate a deterministic float between `0` and `1`:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/JournalingResults.java#random_nb"}  theme={null}
  int value = ctx.random().nextInt();
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/JournalingResults.kt#random_nb"}  theme={null}
  val value = ctx.random().nextInt()
  ```
</CodeGroup>

This behaves like the standard library `Random` class but is deterministically replayable.
You can use any of the methods of [`java.util.Random`](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/Random.html) to generate random numbers: for example, `nextBoolean()`, `nextLong()`, `nextFloat()`, etc.


# Scheduling & Timers
Source: https://docs.restate.dev/develop/java/durable-timers

Durable timers, scheduled actions, and sleep, backed by Restate.

Restate provides durable, fault-tolerant timers that allow you to:

* [Sleep](#durable-sleep): Pause a handler for a given duration
* [Send delayed messages](#scheduling-async-tasks): Schedule handler invocations in the future
* [Set timeouts](#timers-and-timeouts) for async operations
* Implement patterns like [cron jobs](#cron-jobs)

## How it works

Restate tracks and manages timers, ensuring they survive failures and restarts.
For example, if a service is sleeping for 12 hours and fails after 8 hours, then Restate will make sure it only sleeps 4 hours more.

If your handler runs on function-as-a-service platforms like AWS Lambda, Restate suspends the handler while it is sleeping, to free up resources.
Since these platforms charge on execution time, this saves costs.

## Durable sleep

To pause a handler for a set duration:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/DurableTimers.java#sleep"}  theme={null}
  ctx.sleep(Duration.ofSeconds(10));
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/DurableTimers.kt#sleep"}  theme={null}
  ctx.sleep(10.seconds)
  ```
</CodeGroup>

There is no hard limit on how long you can sleep. You can sleep for months or even years, but keep in mind:

* If you sleep in an [exclusive handler](/foundations/handlers#handler-behavior) in a Virtual Object, all other calls to this object will be queued.
* You need to keep the deployment version until the invocation completes (see [versioning](/services/versioning)).
  So for long sleeps, we recommend breaking this up into multiple handlers that call each other with [delayed messages](/develop/java/service-communication#delayed-messages).

<Accordion title="Clock synchronization Restate Server vs. SDK">
  The Restate SDK calculates the wake-up time based on the delay you specify.
  The Restate Server then uses this calculated time to wake up the handler.
  If the Restate Server and the SDK have different system clocks, the sleep duration might not be accurate.
  So make sure that the system clock of the Restate Server and the SDK have the same timezone and are synchronized.
  A mismatch can cause timers to fire earlier or later than expected.
</Accordion>

## Scheduling async tasks

To invoke a handler at a later time, use [delayed messages](/develop/java/service-communication#delayed-messages).

<Accordion title="Delayed messages vs. sleep and send">
  In theory, you can schedule future work in Restate in two ways:

  1. **Delayed messages** (recommended)
  2. **Sleep + send** - sleeping in the current handler, then sending a message

  At first sight, both approaches might seem to achieve similar results.
  However, we recommend to use delayed messages for the following reasons:

  * **Handler completes immediately**: The calling handler can finish execution and complete the invocation without waiting for the delay to finish
  * **No Virtual Object blocking**: Other requests to the same Virtual Object can be processed during the delay period
  * **Better for service versioning**: No long-running invocations that require keeping old service deployments around for a long time (see [service versioning](/services/versioning))
</Accordion>

## Timers and timeouts

Most context actions are async actions that return a `DurableFuture`.

Restate's `DurableFuture` supports setting timeouts, allowing you to bound how long your code waits for an operation.

When an operation times out, it throws a Restate `TimeoutException`.

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/DurableTimers.java#timer"}  theme={null}
  try {
    MyServiceClient.fromContext(ctx).myHandler("Hi!").await(Duration.ofSeconds(5));
  } catch (TimeoutException e) {
    // Handle the timeout error
  }
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/DurableTimers.kt#timer"}  theme={null}
  try {
    MyServiceClient.fromContext(ctx).myHandler("Hi!").await(5.seconds)
  } catch (e: TimeoutException) {
    // Handle the timeout
  }
  ```
</CodeGroup>

For alternative ways of racing async operations, have a look at the [Durable Future combinators docs](/develop/java/concurrent-tasks).

## Cron jobs

Restate does not yet include native cron support, but you can implement your own cron scheduler using:

* Durable timers
* Virtual Objects
* A repeat loop or sleep-schedule pattern

Check out the guide on [implementing cron jobs](/guides/cron).


# Error Handling
Source: https://docs.restate.dev/develop/java/error-handling

Stop infinite retries with Terminal Errors.

Restate handles retries for failed invocations.

<Info>
  Check out the [Error Handling guide](/guides/error-handling) to learn more about how Restate handles transient errors, terminal errors, retries, and timeouts.
</Info>

## Retry strategies

By default, Restate does infinite retries with an exponential backoff strategy.

Check out the [error handling guide](/guides/error-handling) to learn how to customize this.

## Terminal Errors

For failures for which you do not want retries, but instead want the invocation to end and the error message
to be propagated back to the caller, you can throw a **terminal error**.

You can throw a `TerminalException` with an optional HTTP status code and a message anywhere in your handler, as follows:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/ErrorHandling.java#here"}  theme={null}
  throw new TerminalException(500, "Something went wrong");
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/ErrorHandling.kt#here"}  theme={null}
  throw TerminalException(500, "Something went wrong")
  ```
</CodeGroup>

You can catch terminal errors, and build your control flow around it.

<Info>
  When throwing a terminal error, undo earlier handler actions to keep the system consistent. See our [sagas guide](/guides/sagas) for details.
</Info>


# External Events
Source: https://docs.restate.dev/develop/java/external-events

Handle external events and human-in-the-loop patterns with durable waiting primitives.

Sometimes your handlers need to pause and wait for external processes to complete. This is common in:

* **Human-in-the-loop workflows** (approvals, reviews, manual steps)
* **External system integration** (waiting for webhooks, async APIs)
* **AI agent patterns** (tool execution, human oversight)

This pattern is also known as the **callback** or **task token** pattern.

## Two Approaches

Restate provides two primitives for handling external events:

| Primitive            | Use Case                   | Key Feature                          |
| -------------------- | -------------------------- | ------------------------------------ |
| **Awakeables**       | Services & Virtual Objects | Unique ID-based completion           |
| **Durable Promises** | Workflows only             | Named promises for simpler signaling |

## How it works

Implementing this pattern in a distributed system is tricky, since you need to ensure that the handler can recover from failures and resume waiting for the external event.

Restate promises are durable and distributed. They survive crashes and can be resolved or rejected by any handler in the workflow.

To save costs on FaaS deployments, Restate lets the handler [suspend](/foundations/key-concepts#suspensions-on-faas) while awaiting the promise, and invokes it again when the result is available.

## Awakeables

**Best for:** Services and Virtual Objects where you need to coordinate with external systems.

### Creating and waiting for awakeables

1. **Create an awakeable** - Get a unique ID and Future
2. **Send the ID externally** - Pass the awakeable ID to your external system
3. **Wait for result** - Your handler [suspends](/foundations/key-concepts#suspensions-on-faas) until the external system responds

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/Awakeables.java#here"}  theme={null}
  // Create awakeable and get unique ID
  Awakeable<String> awakeable = ctx.awakeable(String.class);
  String awakeableId = awakeable.id();

  // Send ID to external system (email, queue, webhook, etc.)
  ctx.run(() -> requestHumanReview(name, awakeableId));

  // Handler suspends here until external completion
  String review = awakeable.await();
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/Awakeables.kt#here"}  theme={null}
  // Create awakeable and get unique ID
  val awakeable = ctx.awakeable<String>()
  val awakeableId = awakeable.id

  // Send ID to external system (email, queue, webhook, etc.)
  ctx.runBlock { requestHumanReview(awakeableId) }

  // Handler suspends here until external completion
  val review = awakeable.await()
  ```
</CodeGroup>

<Accordion title="Serialization">
  To customize serialization, visit the [docs](/develop/java/serialization).
</Accordion>

<Info>
  Note that if you wait for an awakeable in an [exclusive handler](/foundations/handlers#handler-behavior) in a Virtual Object, all other calls to this object will be queued.
</Info>

### Resolving/rejecting Awakeables

External processes complete awakeables in two ways:

* **Resolve** with success data → handler continues normally
* **Reject** with error reason → throws a [terminal error](/develop/java/error-handling) in the waiting handler

#### Via SDK (from other handlers)

**Resolve:**

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/Awakeables.java#resolve"}  theme={null}
  // Complete with success data
  ctx.awakeableHandle(awakeableId).resolve(String.class, "Looks good!");
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/Awakeables.kt#resolve"}  theme={null}
  // Complete with success data
  ctx.awakeableHandle(awakeableId).resolve("Looks good!")
  ```
</CodeGroup>

**Reject:**

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/Awakeables.java#reject"}  theme={null}
  // Complete with error
  ctx.awakeableHandle(awakeableId).reject("This cannot be reviewed.");
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/Awakeables.kt#reject"}  theme={null}
  // Complete with error
  ctx.awakeableHandle(awakeableId).reject("This cannot be reviewed.")
  ```
</CodeGroup>

#### Via HTTP API

External systems can complete awakeables using Restate's HTTP API:

**Resolve with data:**

```shell theme={null}
curl localhost:8080/restate/awakeables/sign_1PePOqp/resolve \
--json '"Looks good!"'
```

**Reject with error:**

```shell theme={null}
curl localhost:8080/restate/awakeables/sign_1PePOqp/reject \
-H 'content-type: text/plain' \
-d 'Review rejected: insufficient documentation'
```

## Durable Promises

**Best for:** Workflows where you need to signal between different workflow handlers.

**Key differences from awakeables:**

* No ID management - use logical names instead
* Scoped to workflow execution lifetime

Use this for:

* Sending data to the run handler
* Have handlers wait for events emitted by the run handler

<Info>
  After a workflow's run handler completes, other handlers can still be called for up to 24 hours (default).
  The results of resolved Durable Promises remain available during this time.
  Update the retention time via the [service configuration](/services/configuration).
</Info>

### Creating and waiting for promises

Create a promise key:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/ReviewWorkflow.java#promise_key"}  theme={null}
  private static final DurablePromiseKey<String> REVIEW_PROMISE =
      DurablePromiseKey.of("review", String.class);
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/ReviewWorkflow.kt#promise_key"}  theme={null}
  val REVIEW_PROMISE = durablePromiseKey<String>("review")
  ```
</CodeGroup>

Wait for a promise by name:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/ReviewWorkflow.java#promise"}  theme={null}
  String review = ctx.promise(REVIEW_PROMISE).future().await();
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/ReviewWorkflow.kt#promise"}  theme={null}
  val review: String = ctx.promise(REVIEW_PROMISE).future().await()
  ```
</CodeGroup>

Your workflow can [suspend](/foundations/key-concepts#suspensions-on-faas) until the promise is resolved or rejected.

### Resolving/rejecting promises

Resolve/reject from any workflow handler:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/ReviewWorkflow.java#resolve_promise"}  theme={null}
  ctx.promiseHandle(REVIEW_PROMISE).resolve(review);
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/ReviewWorkflow.kt#resolve_promise"}  theme={null}
  ctx.promiseHandle(REVIEW_PROMISE).resolve(review)
  ```
</CodeGroup>

### Complete workflow example

<CodeGroup>
  ```java Java expandable {"CODE_LOAD::java/src/main/java/develop/ReviewWorkflow.java#here"}  theme={null}
  @Workflow
  public class ReviewWorkflow {
    private static final DurablePromiseKey<String> REVIEW_PROMISE =
        DurablePromiseKey.of("review", String.class);


    @Workflow
    public String run(WorkflowContext ctx, String documentId) {
      // Send document for review
      ctx.run(() -> askReview(documentId));

      // Wait for external review submission
      String review = ctx.promise(REVIEW_PROMISE).future().await();

      // Process the review result
      return processReview(documentId, review);
    }

    @Shared
    public void submitReview(SharedWorkflowContext ctx, String review) {
      // Signal the waiting run handler
      ctx.promiseHandle(REVIEW_PROMISE).resolve(review);
    }
  }
  ```

  ```kotlin Kotlin expandable {"CODE_LOAD::kotlin/src/main/kotlin/develop/ReviewWorkflow.kt#here"}  theme={null}
  @Workflow
  class ReviewWorkflow {

    companion object {
      val REVIEW_PROMISE = durablePromiseKey<String>("review")
    }

    @Workflow
    suspend fun run(ctx: WorkflowContext, documentId: String): String {
      // Send document for review
      ctx.runBlock { askReview(documentId) }

      // Wait for external review submission
      val review: String = ctx.promise(REVIEW_PROMISE).future().await()

      // Process the review result
      return processReview(documentId, review)
    }

    @Shared
    suspend fun submitReview(ctx: SharedWorkflowContext, review: String) {
      // Signal the waiting run handler
      ctx.promiseHandle(REVIEW_PROMISE).resolve(review)
    }
  }
  ```
</CodeGroup>

### Two signaling patterns

**External → Workflow** (shown above): External handlers signal the run handler

* Use for human approvals, external API responses, manual interventions
* External handlers call the handler which resolves the promise

**Workflow → External**: Run handler signals other handlers waiting for workflow events

* Use for step completion notifications, status updates, result broadcasting
* Run handler resolves promises that external handlers are awaiting

## Best Practices

* **Use awakeables** for services/objects coordinating with external systems
* **Use durable promises** for workflow signaling
* **Always handle rejections** to gracefully manage failures
* **Include timeouts** for long-running external processes


# Logging
Source: https://docs.restate.dev/develop/java/logging

Configure the log level of your services.

The Java SDK uses [log4j2](https://logging.apache.org/log4j/2.x/manual/configuration.html) as logging facade.

To configure the logging, add the file `resources/log4j2.properties`:

```properties theme={null}
# Set to debug or trace if log4j initialization is failing
status = warn

# Console appender configuration
appender.console.type = Console
appender.console.name = consoleLogger
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss} %-5p %notEmpty{[%X{restateInvocationTarget}]}%notEmpty{[%X{restateInvocationId}]} %c - %m%n

# Filter out logging during replay
appender.console.filter.replay.type = ContextMapFilter
appender.console.filter.replay.onMatch = DENY
appender.console.filter.replay.onMismatch = NEUTRAL
appender.console.filter.replay.0.type = KeyValuePair
appender.console.filter.replay.0.key = restateInvocationStatus
appender.console.filter.replay.0.value = REPLAYING

# Restate logs to debug level
logger.app.name = dev.restate
logger.app.level = info
logger.app.additivity = false
logger.app.appenderRef.console.ref = consoleLogger

# Root logger
rootLogger.level = info
rootLogger.appenderRef.stdout.ref = consoleLogger
```

If you want to do filtering of the logs, you can use `log4j2` filters.

Logging on the `INFO` level is enough for most use cases, but you can set the log level of
the `dev.restate` classes to `DEBUG` and `TRACE` if you want more info about the internal SDK operations.

The SDK injects the following additional metadata to the logging context that can be used for filtering as well:

* `restateInvocationTarget`: Invocation Target, e.g. `counter.Counter/Add`.
* `restateInvocationId`: [Invocation ID](/services/invocation/http#invocation-identifier).
* `restateInvocationStatus`: Invocation status, can be `WAITING_START`, `REPLAYING`, `PROCESSING`, `CLOSED`.


# Serialization
Source: https://docs.restate.dev/develop/java/serialization

Customize serialization for SDK actions.

Restate sends data over the network for storing state, journaling actions, awakeables, etc.
Therefore, Restate needs to serialize and deserialize the journal entries.

## Default serialization

The SDK uses by default [Jackson Databind](https://github.com/FasterXML/jackson) for the Java API, and [Kotlinx serialization](https://kotlinlang.org/docs/serialization.html) for the Kotlin API.
For example, for state keys:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/SerializationExample.java#state_keys"}  theme={null}
  // Primitive types
  var myString = StateKey.of("myString", String.class);
  // Generic types need TypeRef (similar to Jackson's TypeReference)
  var myMap = StateKey.of("myMap", TypeTag.of(new TypeRef<Map<String, String>>() {}));
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/SerializationExample.kt#state_keys"}  theme={null}
  // These imports provide extension methods with reified generics
  import dev.restate.sdk.kotlin.*

  // Primitive types
  val myString = stateKey<String>("my-string")
  // Generic types
  val myMap = stateKey<Map<String, String>>("my-map")
  // Custom generic type
  val myPerson = stateKey<Person<String>>("my-person")
  ```
</CodeGroup>

The same applies to journaling actions, awakeables, etc.

## Customizing serialization

There are different entrypoints in the SDK to customize (de)serialization, depending on your needs.

### Custom Jackson `ObjectMapper`

You can customize the Jackson's [`ObjectMapper`](https://javadoc.io/doc/com.fasterxml.jackson.core/jackson-databind/latest/com/fasterxml/jackson/databind/ObjectMapper.html) by subclassing the [`JacksonSerdeFactory`](https://restatedev.github.io/sdk-java/javadocs/dev/restate/sdk/serde/jackson/JacksonSerdeFactory) class:

```java {"CODE_LOAD::java/src/main/java/develop/SerializationExample.java#custom_jackson"}  theme={null}
class MyJacksonSerdeFactory extends JacksonSerdeFactory {
  public MyJacksonSerdeFactory() {
    super(new ObjectMapper().configure(SerializationFeature.INDENT_OUTPUT, true));
  }
}
```

And annotating your services with [`@CustomSerdeFactory`](https://restatedev.github.io/sdk-java/javadocs/dev/restate/sdk/annotation/CustomSerdeFactory):

```java {"CODE_LOAD::java/src/main/java/develop/SerializationExample.java#custom_jackson_service"}  theme={null}
@CustomSerdeFactory(MyJacksonSerdeFactory.class)
@Service
class ServiceWithCustomJacksonObjectMapper {
  @Handler
  public String greet(Context context) {
    return "Hello world";
  }
}
```

### Custom Kotlinx Serialization `Json`

You can customize the Kotlinx Serialization [`Json`](https://kotlinlang.org/api/kotlinx.serialization/kotlinx-serialization-json/kotlinx.serialization.json/-json/) by subclassing the [`KotlinSerializationSerdeFactory`](https://restatedev.github.io/sdk-java/ktdocs/sdk-api-kotlin/dev.restate.sdk.kotlin.serialization/-kotlin-serialization-serde-factory/) class:

```kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/CustomSerializationExample.kt#custom"}  theme={null}
class MyJsonSerdeFactory : KotlinSerializationSerdeFactory(json = Json { prettyPrint = true })
```

And annotating your services with [`@CustomSerdeFactory`](https://restatedev.github.io/sdk-java/ktdocs/sdk-common/dev.restate.sdk.annotation/-custom-serde-factory/):

```kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/CustomSerializationExample.kt#custom_service"}  theme={null}
@CustomSerdeFactory(MyJsonSerdeFactory::class)
@Service
class ServiceWithCustomSerdeFactory {
  @Handler suspend fun greet(ctx: Context) = "Hello world!"
}
```

### Use custom serializer for a specific operation

If you need for a specific operation (e.g. for `Context.run` or `Context.Awakeable`) a custom (de)serializer, you can implement one using the interface [`Serde`](https://restatedev.github.io/sdk-java/javadocs/dev/restate/serde/Serde):

```java {"CODE_LOAD::java/src/main/java/develop/SerializationExample.java#customserdes"}  theme={null}
class MyPersonSerde implements Serde<Person> {
  @Override
  public Slice serialize(Person person) {
    // convert value to a byte array, then wrap in a Slice
    return Slice.wrap(person.toBytes());
  }

  @Override
  public Person deserialize(Slice slice) {
    // convert value to Person
    return Person.fromBytes(slice.toByteArray());
  }
}
```

And then use it, for example, in combination with `ctx.run`:

```java {"CODE_LOAD::java/src/main/java/develop/SerializationExample.java#use_person_serde"}  theme={null}
ctx.run(new MyPersonSerde(), () -> new Person());
```

### Use another serialization library

If you want to use a different serialization library throughout your service, we suggest implementing [`SerdeFactory`](https://restatedev.github.io/sdk-java/javadocs/dev/restate/serde/SerdeFactory) and annotating the service class with [`@CustomSerdeFactory`](https://restatedev.github.io/sdk-java/javadocs/dev/restate/sdk/annotation/CustomSerdeFactory). Refer to the Javadocs for more details.


# Service Communication
Source: https://docs.restate.dev/develop/java/service-communication

Call other services from your handler.

Your Restate handler can call other handlers in three ways:

* **[Request-response calls](#request-response-calls)**: Call and wait for a response
* **[One-way messages](#sending-messages)**: Send a message and continue
* **[Delayed messages](#delayed-messages)**: Send a message after a delay

<Info>
  To call a service from an external application, see the [HTTP](/services/invocation/http), [Kafka](/services/invocation/kafka), or [SDK Clients](/services/invocation/clients/java-sdk) documentation.
</Info>

<Tip>[Why use Restate for service communication?](/foundations/key-concepts#resilient-communication)</Tip>

## Generating service clients

The Restate Java SDK automatically generates clients for each of your services when you build the project.

If you don't see the generated clients, make sure you [added the code generator](develop/java/overview#getting-started) and have built the project with `./gradlew build` or `mvn compile exec:java`.

## Request-response calls

To call a Restate handler, use the generated clients and wait for its result:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/ServiceCommunication.java#request_response"}  theme={null}
  // To call a Service:
  String svcResponse = MyServiceClient.fromContext(ctx).myHandler(request).await();

  // To call a Virtual Object:
  String objResponse = MyObjectClient.fromContext(ctx, objectKey).myHandler(request).await();

  // To call a Workflow:
  // `run` handler — can only be called once per workflow ID
  String wfResponse = MyWorkflowClient.fromContext(ctx, workflowId).run(request).await();
  // Other handlers can be called anytime within workflow retention
  String status =
      MyWorkflowClient.fromContext(ctx, workflowId).interactWithWorkflow(request).await();
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/ServiceCommunication.kt#request_response"}  theme={null}
  // To call a Service:
  val svcResponse = MyServiceClient.fromContext(ctx).myHandler(request).await()

  // To call a Virtual Object:
  val objResponse = MyObjectClient.fromContext(ctx, objectKey).myHandler(request).await()

  // To call a Workflow:
  // `run` handler — can only be called once per workflow ID
  val wfResponse = MyWorkflowClient.fromContext(ctx, workflowId).run(request).await()
  // Other handlers can be called anytime within workflow retention
  MyWorkflowClient.fromContext(ctx, workflowId).interactWithWorkflow(request).await()
  ```
</CodeGroup>

Use the generic clients when you don't have access to typed clients or need dynamic service names:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/ServiceCommunication.java#request_response_generic"}  theme={null}
  Target target = Target.service("MyService", "myHandler"); // or virtualObject or workflow
  String response =
      ctx.call(Request.of(target, TypeTag.of(String.class), TypeTag.of(String.class), request))
          .await();
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/ServiceCommunication.kt#request_response_generic"}  theme={null}
  val target = Target.service("MyService", "myHandler")
  val response =
      ctx.call(Request.of(target, typeTag<String>(), typeTag<String>(), request)).await()
  ```
</CodeGroup>

<Accordion title="Workflow retention">
  After a workflow's run handler completes, other handlers can still be called for up to 24 hours (default).
  Update this via the [service configuration](/services/configuration).
</Accordion>

<Info>
  Request-response calls between [exclusive handlers](/foundations/handlers#handler-behavior) of Virtual Objects may lead to deadlocks:

  * Cross deadlock: A → B and B → A (same keys).
  * Cycle deadlock: A → B → C → A.

  Use the UI or CLI to [cancel](/services/invocation/managing-invocations#cancel) and unblock deadlocked invocations.
</Info>

## Sending messages

To send a message to another Restate handler without waiting for a response:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/ServiceCommunication.java#one_way"}  theme={null}
  MyServiceClient.fromContext(ctx).send().myHandler(request);
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/ServiceCommunication.kt#one_way"}  theme={null}
  MyServiceClient.fromContext(ctx).send().myHandler(request)
  ```
</CodeGroup>

Restate handles message delivery and retries, so the handler can complete and return without waiting for the message to be processed.

Use generic clients when you don't have the service definition:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/ServiceCommunication.java#one_way_generic"}  theme={null}
  Target target = Target.service("MyService", "myHandler"); // or virtualObject or workflow
  ctx.send(Request.of(target, TypeTag.of(String.class), TypeTag.of(String.class), request));
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/ServiceCommunication.kt#one_way_generic"}  theme={null}
  val target = Target.service("MyService", "myHandler")
  ctx.send(Request.of(target, typeTag<String>(), typeTag<String>(), request))
  ```
</CodeGroup>

<Info>
  Calls to a Virtual Object execute in order of arrival, serially.
  Example:

  ```java {"CODE_LOAD::java/src/main/java/develop/ServiceCommunication.java#ordering"}  theme={null}
  MyObjectClient.fromContext(ctx, objectKey).send().myHandler("I'm call A");
  MyObjectClient.fromContext(ctx, objectKey).send().myHandler("I'm call B");
  ```

  Call A is guaranteed to execute before B. However, other invocations may interleave between A and B.
</Info>

## Delayed messages

To send a message after a delay, use the generated clients with `.send()` and the `Duration` as second parameter:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/ServiceCommunication.java#delayed"}  theme={null}
  MyServiceClient.fromContext(ctx).send().myHandler(request, Duration.ofDays(5));
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/ServiceCommunication.kt#delayed"}  theme={null}
  MyServiceClient.fromContext(ctx).send().myHandler(request, 5.days)
  ```
</CodeGroup>

Or with the generic clients:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/ServiceCommunication.java#delayed_generic"}  theme={null}
  Target target = Target.service("MyService", "myHandler"); // or virtualObject or workflow
  ctx.send(
      Request.of(target, TypeTag.of(String.class), TypeTag.of(String.class), request),
      Duration.ofDays(5));
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/ServiceCommunication.kt#delayed_generic"}  theme={null}
  val target = Target.service("MyService", "myHandler")
  Request.of(target, typeTag<String>(), typeTag<String>(), request).send(ctx, 5.days)
  ```
</CodeGroup>

<Info>
  Learn [how this is different](/develop/java/durable-timers#scheduling-async-tasks) from sleeping and then sending a message.
</Info>

## Using an idempotency key

To prevent duplicate executions of the same call, add an idempotency key:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/ServiceCommunication.java#idempotency_key"}  theme={null}
  // For a request-response call
  MyServiceClient.fromContext(ctx).myHandler(request, req -> req.idempotencyKey("abc123"));
  // For a message
  MyServiceClient.fromContext(ctx).send().myHandler(request, req -> req.idempotencyKey("abc123"));
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/ServiceCommunication.kt#idempotency_key"}  theme={null}
  // For a regular call
  MyServiceClient.fromContext(ctx).myHandler(request) { idempotencyKey = "abc123" }
  // For a one way call
  MyServiceClient.fromContext(ctx).send().myHandler(request) { idempotencyKey = "abc123" }
  ```
</CodeGroup>

Restate automatically deduplicates calls made during the same handler execution, so there's no need to provide an idempotency key in that case.
However, if multiple handlers might call the same service independently, you can use an idempotency key to ensure deduplication across those calls.

## Attach to an invocation

To wait for or get the result of a previously sent message:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/ServiceCommunication.java#attach"}  theme={null}
  var handle =
      MyServiceClient.fromContext(ctx)
          .send()
          .myHandler(request, req -> req.idempotencyKey("abc123"));
  var response = handle.attach().await();
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/ServiceCommunication.kt#attach"}  theme={null}
  val handle =
      MyServiceClient.fromContext(ctx).send().myHandler(request) { idempotencyKey = "abc123" }
  val response = handle.attach().await()
  ```
</CodeGroup>

* With an idempotency key: Wait for completion and retrieve the result.
* Without an idempotency key: Can only wait, not retrieve the result.

## Cancel an invocation

To cancel a running handler:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/ServiceCommunication.java#cancel"}  theme={null}
  var handle = MyServiceClient.fromContext(ctx).send().myHandler(request);
  handle.cancel();
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/ServiceCommunication.kt#cancel"}  theme={null}
  val handle = MyServiceClient.fromContext(ctx).send().myHandler(request)
  handle.cancel()
  ```
</CodeGroup>

## See also

* **[SDK Clients](/develop/java/service-communication)**: Call Restate services from external applications
* **[Error Handling](/develop/java/error-handling)**: Handle failures and terminal errors in service calls
* **[Durable Timers](/develop/java/durable-timers)**: Implement timeouts for your service calls
* **[Serialization](/develop/java/serialization)**: Customize how data is serialized between services
* **[Sagas](/guides/sagas)**: Roll back or compensate for canceled service calls.


# Services
Source: https://docs.restate.dev/develop/java/services

Implementing Restate services with the Java/Kotlin SDK.

The Restate Java/Kotlin SDK is open source and available on [GitHub](https://github.com/restatedev/sdk-java).

The Restate SDK lets you implement **handlers**. Handlers can be part of a **[Basic Service](/foundations/services#basic-service)**, a **[Virtual Object](/foundations/services#virtual-object)**, or a **[Workflow](/foundations/services#workflow)**. This page shows how to define them with the Java/Kotlin SDK.

## Prerequisites

* [JDK](https://whichjdk.com/) >= 17

## Getting started

<Tip>
  Get started quickly with the [Java](/quickstart#java) or [Kotlin](/quickstart#kotlin) Quickstart.
</Tip>

To start building Restate services, you need two dependencies in your Maven/Gradle project manifest:

* The SDK dependency.
* The annotation processor dependency.

The SDK dependency comes in different flavors: Java or Kotlin API, HTTP or Lambda.
Choose the one you need depending on the language you want to use and whether you want to deploy the service as an HTTP server or as AWS Lambda.

<CodeGroup>
  ```kt Java/Gradle theme={null}
  // Annotation processor
  annotationProcessor("dev.restate:sdk-api-gen:2.4.1")

  // For deploying as HTTP service
  implementation("dev.restate:sdk-java-http:2.4.1")
  // Or for deploying using AWS Lambda
  implementation("dev.restate:sdk-java-lambda:2.4.1")
  ```

  ```xml Java/Maven theme={null}
  <properties>
      <restate.version>2.4.1</restate.version>
  </properties>
  <dependencies>
      <!-- For deploying as HTTP service -->
      <dependency>
          <groupId>dev.restate</groupId>
          <artifactId>sdk-java-http</artifactId>
          <version>${restate.version}</version>
      </dependency>
      <!-- For deploying using AWS Lambda -->
      <dependency>
          <groupId>dev.restate</groupId>
          <artifactId>sdk-java-http</artifactId>
          <version>${restate.version}</version>
      </dependency>
  </dependencies>
  <build>
      <plugins>
          <plugin>
              <groupId>org.apache.maven.plugins</groupId>
              <artifactId>maven-compiler-plugin</artifactId>
              <configuration>
                  <annotationProcessorPaths>
                      <!-- Setup annotation processor -->
                      <path>
                          <groupId>dev.restate</groupId>
                          <artifactId>sdk-api-gen</artifactId>
                          <version>${restate.version}</version>
                      </path>
                  </annotationProcessorPaths>
              </configuration>
          </plugin>
      </plugins>
  </build>
  ```

  ```kt Kotlin/Gradle theme={null}
  // KSP annotation processor
  ksp("dev.restate:sdk-api-kotlin-gen:2.4.1")

  // For deploying as HTTP service
  implementation("dev.restate:sdk-kotlin-http:2.4.1")
  // Or for deploying using AWS Lambda
  implementation("dev.restate:sdk-kotlin-lambda:2.4.1")
  ```
</CodeGroup>

<Accordion title="Troubleshooting: Cannot resolve symbol '<MyServiceName>Client'">
  Make sure the annotation processor is correctly configured, as described above, and **build the project** at least once.
</Accordion>

<Accordion title="IntelliJ won't update the Client classes when building">
  Make sure you have annotation processing enabled in the IntelliJ compiler, check the relevant [documentation](https://www.jetbrains.com/help/idea/annotation-processors-support.html).
  An example setup is available here: [compiler.xml](https://github.com/restatedev/examples/blob/main/java/templates/java-gradle/.idea/compiler.xml)
</Accordion>

## Basic Services

[Basic Services](/foundations/services) group related **handlers** and expose them as callable endpoints:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/MyService.java?collapse_prequel"}  theme={null}
  @Service
  public class MyService {
    @Handler
    public String myHandler(Context ctx, String greeting) {
      return greeting + "!";
    }

    public static void main(String[] args) {
      RestateHttpServer.listen(Endpoint.bind(new MyService()));
    }
  }
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/MyService.kt?collapse_prequel"}  theme={null}
  @Service
  class MyService {
    @Handler suspend fun myHandler(ctx: Context, greeting: String) = "$greeting!"
  }

  fun main() {
    RestateHttpServer.listen(endpoint { bind(MyService()) })
  }
  ```
</CodeGroup>

* Define a service using the [`@Service`](http://docs.restate.dev/javadocs/dev/restate/sdk/annotation/Service.html) and [`@Handler`](http://docs.restate.dev/javadocs/dev/restate/sdk/annotation/Handler.html) annotations
* Each handler can be called at `<RESTATE_INGRESS>/MyService/myHandler`. To override the service name (default is simple class name), use the annotation [`@Name`](http://docs.restate.dev/javadocs/dev/restate/sdk/annotation/Name.html).
* Handlers take the `Context` ([JavaDocs](https://restatedev.github.io/sdk-java/javadocs/dev/restate/sdk/ObjectContext)/[KotlinDocs](https://restatedev.github.io/sdk-java/ktdocs/sdk-api-kotlin/dev.restate.sdk.kotlin/-context/)) as the first argument.
* The input parameter (at most one) and return type are optional and can be of any type. See [serialization](/develop/java/serialization) for more details.
* Create an endpoint to expose the service over HTTP (port `9080` by default).

## Virtual Objects

[Virtual Objects](/foundations/services) are services that are stateful and key-addressable — each object instance has a unique ID and persistent state.

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/MyObject.java?collapse_prequel"}  theme={null}
  @VirtualObject
  public class MyObject {

    @Handler
    public String myHandler(ObjectContext ctx, String greeting) {
      String objectId = ctx.key();

      return greeting + " " + objectId + "!";
    }

    @Shared
    public String myConcurrentHandler(SharedObjectContext ctx, String input) {
      return "my-output";
    }

    public static void main(String[] args) {
      RestateHttpServer.listen(Endpoint.bind(new MyObject()));
    }
  }
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/MyObject.kt?collapse_prequel"}  theme={null}
  @VirtualObject
  class MyObject {

    @Handler
    suspend fun myHandler(ctx: ObjectContext, greeting: String): String {
      val objectKey = ctx.key()

      return "$greeting $objectKey!"
    }

    @Shared suspend fun myConcurrentHandler(ctx: SharedObjectContext, input: String) = "my-output"
  }

  fun main() {
    RestateHttpServer.listen(endpoint { bind(MyObject()) })
  }
  ```
</CodeGroup>

* Use the `@VirtualObject` annotation.
* The first argument of the handler must be the `ObjectContext` parameter ([JavaDocs](https://restatedev.github.io/sdk-java/javadocs/dev/restate/sdk/ObjectContext)/[KotlinDocs](https://restatedev.github.io/sdk-java/ktdocs/sdk-api-kotlin/dev.restate.sdk.kotlin/-object-context/)).
* Each instance is identified by a key (accessible via `ctx.key()`).
* Virtual Objects can have [exclusive and shared handlers](/foundations/handlers#handler-behavior).
* Exclusive handlers receive an `ObjectContext`, allowing read/write access to object state.
* Shared handlers use the [`@Shared`](http://docs.restate.dev/javadocs/dev/restate/sdk/annotation/Shared.html) annotation and the `SharedObjectContext` ([JavaDocs](https://restatedev.github.io/sdk-java/javadocs/dev/restate/sdk/SharedObjectContext)/[KotlinDocs](https://restatedev.github.io/sdk-java/ktdocs/sdk-api-kotlin/dev.restate.sdk.kotlin/-shared-object-context/)).

## Workflows

[Workflows](/foundations/services) are long-lived processes with a defined lifecycle. They run once per key and are ideal for orchestrating multi-step operations, which require external interaction via signals and queries.

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/MyWorkflow.java?collapse_prequel"}  theme={null}
  @Workflow
  public class MyWorkflow {

    @Workflow
    public String run(WorkflowContext ctx, String input) {

      // implement workflow logic here

      return "success";
    }

    @Shared
    public String interactWithWorkflow(SharedWorkflowContext ctx, String input) {
      // implement interaction logic here
      return "my result";
    }

    public static void main(String[] args) {
      RestateHttpServer.listen(Endpoint.bind(new MyWorkflow()));
    }
  }
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/MyWorkflow.kt?collapse_prequel"}  theme={null}
  @Workflow
  class MyWorkflow {

    @Workflow
    suspend fun run(ctx: WorkflowContext, input: String): String {
      // implement workflow logic here

      return "success"
    }

    @Handler
    suspend fun interactWithWorkflow(ctx: SharedWorkflowContext, input: String): String {
      // implement interaction logic here
      return "my result"
    }
  }

  fun main() {
    RestateHttpServer.listen(endpoint { bind(MyWorkflow()) })
  }
  ```
</CodeGroup>

* Create the workflow by using the [`@Workflow`](http://docs.restate.dev/javadocs/dev/restate/sdk/annotation/Workflow.html) annotation.
* Every workflow **must** include a `run` handler:
  * This is the main orchestration entry point
  * It runs exactly once per workflow execution and uses the `WorkflowContext` ([JavaDocs](https://restatedev.github.io/sdk-java/javadocs/dev/restate/sdk/WorkflowContext)/[KotlinDocs](https://restatedev.github.io/sdk-java/ktdocs/sdk-api-kotlin/dev.restate.sdk.kotlin/-workflow-context/))
  * Resubmission of the same workflow will fail with "Previously accepted". The invocation ID can be found in the request header `x-restate-id`.
  * Use `ctx.key()` to access the workflow's unique ID
* Additional handlers must use the `SharedWorkflowContext` ([JavaDocs](https://restatedev.github.io/sdk-java/javadocs/dev/restate/sdk/SharedWorkflowContext)/[KotlinDocs](https://restatedev.github.io/sdk-java/ktdocs/sdk-api-kotlin/dev.restate.sdk.kotlin/-shared-workflow-context/)) and can signal or query the workflow. They can run concurrently with the `run` handler and until the retention time expires.

## Configuring services

Check out the [service configuration docs](/services/configuration) to learn how to configure service behavior, including timeouts and retention policies.


# Serving
Source: https://docs.restate.dev/develop/java/serving

Create an endpoint to serve your services.

Restate services can run in two ways: as an HTTP endpoint or as AWS Lambda functions.

## Creating an HTTP endpoint

1. Use either  or  as SDK dependency.
2. Create an endpoint
3. Bind one or multiple services to it
4. Listen on the specified port (default `9080`) for connections and requests.

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/ServingHttp.java#here"}  theme={null}
  import dev.restate.sdk.endpoint.Endpoint;
  import dev.restate.sdk.http.vertx.RestateHttpServer;

  class MyApp {
    public static void main(String[] args) {
      RestateHttpServer.listen(
          Endpoint.bind(new MyService()).bind(new MyObject()).bind(new MyWorkflow()), 8080);
    }
  }
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/ServingHttp.kt#here"}  theme={null}
  fun main() {
    RestateHttpServer.listen(
        endpoint {
          bind(MyService())
          bind(MyObject())
          bind(MyWorkflow())
        })
  }
  ```
</CodeGroup>

## Creating a Lambda handler

1. Use either  or   as SDK dependency.
2. Extend the class `BaseRestateLambdaHandler`
3. Override the register method
4. Bind one or multiple services to the builder

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/ServingLambda.java#here"}  theme={null}
  import dev.restate.sdk.endpoint.Endpoint;
  import dev.restate.sdk.lambda.BaseRestateLambdaHandler;

  class MyLambdaHandler extends BaseRestateLambdaHandler {
    @Override
    public void register(Endpoint.Builder builder) {
      builder.bind(new MyService()).bind(new MyObject());
    }
  }
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/MyLambdaHandler.kt#here"}  theme={null}
  import dev.restate.sdk.endpoint.Endpoint
  import dev.restate.sdk.lambda.BaseRestateLambdaHandler

  class MyLambdaHandler : BaseRestateLambdaHandler() {
    override fun register(builder: Endpoint.Builder) {
      builder.bind(MyService()).bind(MyObject())
    }
  }
  ```
</CodeGroup>

The implementation of your services and handlers remains the same for both deployment options.
Have a look at the [deployment section](/services/deploy/lambda) for guidance on how to deploy your services on AWS Lambda.

<Accordion title="Using Java 21 Virtual Threads">
  If you use a JVM >= 21, you can use virtual threads to run your services:

  <CodeGroup>
    ```java Java {"CODE_LOAD::java/src/main/java/develop/ServingVirtualThreads.java#here"}  theme={null}
    builder.bind(
        new Greeter(),
        HandlerRunner.Options.withExecutor(Executors.newVirtualThreadPerTaskExecutor()));
    ```

    ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/ServingVirtualThreads.kt#here"}  theme={null}
    builder.bind(
        Greeter(),
        HandlerRunner.Options(
            coroutineContext = Executors.newVirtualThreadPerTaskExecutor().asCoroutineDispatcher(),
        ),
    )
    ```
  </CodeGroup>
</Accordion>

## Validating request identity

SDKs can validate that incoming requests come from a particular Restate
instance. You can find out more about request identity in the
[Security docs](/services/security#locking-down-service-access). You will need to use the request identity dependency .

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/ServingIdentity.java#here"}  theme={null}
  import dev.restate.sdk.auth.signing.RestateRequestIdentityVerifier;
  import dev.restate.sdk.endpoint.Endpoint;
  import dev.restate.sdk.http.vertx.RestateHttpServer;

  class MySecureApp {
    public static void main(String[] args) {
      var endpoint =
          Endpoint.bind(new MyService())
              .withRequestIdentityVerifier(
                  RestateRequestIdentityVerifier.fromKeys(
                      "publickeyv1_w7YHemBctH5Ck2nQRQ47iBBqhNHy4FV7t2Usbye2A6f"));
      RestateHttpServer.listen(endpoint);
    }
  }
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/ServingIdentity.kt#here"}  theme={null}
  import dev.restate.sdk.auth.signing.RestateRequestIdentityVerifier
  import dev.restate.sdk.http.vertx.RestateHttpServer
  import dev.restate.sdk.kotlin.endpoint.endpoint

  fun main() {
    RestateHttpServer.listen(
        endpoint {
          bind(MyService())
          requestIdentityVerifier =
              RestateRequestIdentityVerifier.fromKeys(
                  "publickeyv1_w7YHemBctH5Ck2nQRQ47iBBqhNHy4FV7t2Usbye2A6f",
              )
        })
  }
  ```
</CodeGroup>


# State
Source: https://docs.restate.dev/develop/java/state

Store key-value state in Restate.

Restate lets you persist key-value (K/V) state using its embedded K/V store.

## Key characteristics

<Info>
  [Learn about Restate's embedded K/V store](/foundations/key-concepts#consistent-state).
</Info>

**State is only available for Virtual Objects and Workflows.**

Scope & retention:

* For Virtual Objects: State is scoped per object key and retained indefinitely. It is persisted and shared across all invocations for that object until explicitly cleared.
* For Workflows: State is scoped per workflow execution (workflow ID) and retained only for the duration of the workflow’s configured retention time.

Access Rules:

* [Exclusive handlers](/foundations/handlers#handler-behavior) (e.g., `run()` in workflows) can read and write state.
* [Shared handlers](/foundations/handlers#handler-behavior) can only read state and cannot mutate it.

You can inspect and edit the K/V state via the UI and the [CLI](/services/introspection#inspecting-application-state).

## List all state keys

To retrieve all keys for which the current Virtual Object has stored state:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/State.java#statekeys"}  theme={null}
  Collection<String> keys = ctx.stateKeys();
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/State.kt#statekeys"}  theme={null}
  val keys = ctx.stateKeys()
  ```
</CodeGroup>

## Get state value

To read a value by key.

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/State.java#get"}  theme={null}
  // Getting String value
  StateKey<String> STRING_STATE_KEY = StateKey.of("my-key", String.class);
  String stringState = ctx.get(STRING_STATE_KEY).orElse("my-default");

  // Getting integer value
  StateKey<Integer> INT_STATE_KEY = StateKey.of("my-key", Integer.class);
  int intState = ctx.get(INT_STATE_KEY).orElse(0);
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/State.kt#get"}  theme={null}
  // Getting String value
  val STRING_STATE_KEY = stateKey<String>("my-key")
  val stringState: String? = ctx.get(STRING_STATE_KEY)

  // Getting integer value
  val INT_STATE_KEY = stateKey<Int>("my-key")
  val intState: Int? = ctx.get(INT_STATE_KEY)
  ```
</CodeGroup>

See the [serialization docs](/develop/java/serialization) to customize the state serializer.

## Set state value

To write or update a value:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/State.java#set"}  theme={null}
  StateKey<String> STRING_STATE_KEY = StateKey.of("my-key", String.class);
  ctx.set(STRING_STATE_KEY, "my-new-value");
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/State.kt#set"}  theme={null}
  val STRING_STATE_KEY = stateKey<String>("my-key")
  ctx.set(STRING_STATE_KEY, "my-new-value")
  ```
</CodeGroup>

## Clear state key

To delete a specific key:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/State.java#clear"}  theme={null}
  StateKey<String> STRING_STATE_KEY = StateKey.of("my-key", String.class);
  ctx.clear(STRING_STATE_KEY);
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/State.kt#clear"}  theme={null}
  val STRING_STATE_KEY = stateKey<String>("my-key")
  ctx.clear(STRING_STATE_KEY)
  ```
</CodeGroup>

## Clear all state keys

To remove all stored state for the current Virtual Object:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/State.java#clear_all"}  theme={null}
  ctx.clearAll();
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/State.kt#clear_all"}  theme={null}
  ctx.clearAll()
  ```
</CodeGroup>

## Advanced: Eager vs. lazy state loading

Restate supports two modes for loading state in handlers:

### Eager state (default)

* **How it works**: State is automatically sent with the request when invoking a handler
* **Benefits**: State is available immediately when the handler starts executing
* **Behavior**: All reads and writes to state are local to the handler execution
* **Best for**: Small to medium state objects that are frequently accessed

### Lazy state

* **How it works**: State is fetched on-demand on `get` calls from the Restate Server
* **Benefits**: Reduces initial request size and memory usage
* **Setup**: Enable lazy state in the [service or handler configuration](/services/configuration)
* **Best for**: Large state objects that aren't needed in every handler execution


# Testing
Source: https://docs.restate.dev/develop/java/testing

Utilities to test your handler logic.

The Java SDK comes with the module `sdk-testing` that integrates with JUnit 5 and [Testcontainers](https://testcontainers.com/) to start up a Restate container together with your services code and automatically register them.

Add the following dependency to your project: .

## Using the JUnit 5 Extension

Given the service to test `MyService`, annotate the test class with `@RestateTest` and annotate the services to bind with `@BindService`:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/MyServiceTest.java#extension"}  theme={null}
  @RestateTest
  class MyServiceTest {

    @BindService MyService service = new MyService();

    // Your tests

  }
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/MyServiceTest.kt#extension"}  theme={null}
  @RestateTest
  class MyServiceTest {

    @BindService val service = MyService()

    // Your tests

  }
  ```
</CodeGroup>

Note that the extension will start one Restate server for the whole test class. For more details, checkout [`RestateTest` Javadocs](https://restatedev.github.io/sdk-java/javadocs/dev/restate/sdk/testing/RestateTest.html).

Once the extension is set, you can implement your test methods as usual, and inject a [`Client`](https://restatedev.github.io/sdk-java/javadocs/dev/restate/client/Client.html) using [`@RestateClient`](https://restatedev.github.io/sdk-java/javadocs/dev/restate/sdk/testing/RestateClient.html) to interact with Restate and the registered services:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/MyServiceTestMethod.java#test"}  theme={null}
  @Test
  void testMyHandler(@RestateClient Client ingressClient) {
    // Create the service client from the injected ingress client
    var client = MyServiceClient.fromClient(ingressClient);

    // Send request to service and assert the response
    var response = client.myHandler("Hi");
    assertEquals(response, "Hi!");
  }
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/MyServiceTestMethod.kt#test"}  theme={null}
  @Test
  fun testMyHandler(@RestateClient ingressClient: Client) = runTest {
    // Create the service client from the injected ingress client
    val client = MyServiceClient.fromClient(ingressClient)

    // Send request to service and assert the response
    val response = client.myHandler("Hi")
    assertEquals(response, "Hi!")
  }
  ```
</CodeGroup>

## Usage without JUnit 5

You can use the testing tools without JUnit 5 by using [`RestateRunner`](https://restatedev.github.io/sdk-java/javadocs/dev/restate/sdk/testing/RestateRunner.html) directly.
For more details, refer to the Javadocs.


# Concurrent Tasks
Source: https://docs.restate.dev/develop/python/concurrent-tasks

Execute multiple tasks concurrently and gather results.

When building resilient applications, you often need to perform multiple operations in parallel to improve performance and user experience. Restate provides durable concurrency primitives that allow you to run tasks concurrently while maintaining deterministic execution during replays.

## When to use concurrent tasks

Use concurrent tasks when you need to:

* Call multiple external services simultaneously (e.g., fetching data from different APIs)
* Race multiple operations and use the first result (e.g., trying multiple LLM providers)
* Implement timeouts by racing an operation against a timer
* Perform batch operations where individual tasks can run in parallel

## Key benefits

* **Deterministic replay**: Restate logs the order of completion, ensuring consistent behavior during failures
* **Fault tolerance**: If your handler fails, tasks that were already completed will be replayed with their results, while pending tasks will be retried

## Parallelizing tasks

Start multiple durable operations concurrently by calling them without immediately awaiting:

```python {"CODE_LOAD::python/src/develop/journaling_results.py#parallel"}  theme={null}
# Start operations concurrently
call1 = ctx.run_typed("fetch_user", fetch_user_data, user_id=123)
call2 = ctx.run_typed("fetch_orders", fetch_order_history, user_id=123)
call3 = ctx.service_call(calculate_metrics, arg=123)

# Now wait for results as needed
user = await call1
orders = await call2
metrics = await call3
```

Check out the guide on [parallelizing work](guides/parallelizing-work).

## Retrieving results

Restate provides several patterns for coordinating concurrent tasks. All patterns use `RestateDurableFuture` combinators that log the order of completion, ensuring deterministic behavior during replays.

### Waiting for first completion

There are two ways to do this.

#### Select

Use `restate.select()` to race multiple operations and handle the first one that completes. This is ideal for implementing timeouts or waiting for external confirmations:

```python {"CODE_LOAD::python/src/develop/journaling_results.py#select"}  theme={null}
_, confirmation_future = ctx.awakeable(type_hint=str)
match await restate.select(
    confirmation=confirmation_future, timeout=ctx.sleep(timedelta(days=1))
):
    case ["confirmation", "ok"]:
        return "success!"
    case ["confirmation", "deny"]:
        raise TerminalError("Confirmation was denied!")
    case _:
        raise TerminalError("Verification timer expired!")
```

#### Wait completed

Use `restate.wait_completed()` when you want to wait for at least one task to complete.
This returns a tuple of two lists: the first list contains the futures that are completed,
the second list contains the futures that are not completed.
This gives you the option to, for example, cancel the pending futures:

```python {"CODE_LOAD::python/src/develop/journaling_results.py#wait_completed"}  theme={null}
claude = ctx.service_call(claude_sonnet, arg=f"What is the weather?")
openai = ctx.service_call(open_ai, arg=f"What is the weather?")

done, pending = await restate.wait_completed(claude, openai)

# collect the completed results
results = [await f for f in done]

# cancel the pending calls
for f in pending:
    call_future = typing.cast(RestateDurableCallFuture, f)
    ctx.cancel_invocation(await call_future.invocation_id())
```

#### Key Differences

* **Return behavior**: `select` returns as soon as the first future completes, while `wait_completed` waits for at least one to complete and returns both completed and pending futures.
* **Use cases**:
  * Use `select` when you want to race multiple operations and act on whichever completes first.
  * Use `wait_completed` when you want to handle completed futures immediately while potentially canceling or managing pending ones,
* **Pattern matching**: `select` uses pattern matching to determine which future completed, while `wait_completed` separates futures into completed and pending collections.

### Waiting for all tasks to complete

```python {"CODE_LOAD::python/src/develop/journaling_results.py#all"}  theme={null}
claude = ctx.service_call(claude_sonnet, arg=f"What is the weather?")
openai = ctx.service_call(open_ai, arg=f"What is the weather?")

results_done = await restate.gather(claude, openai)
results = [await result for result in results_done]
```

### Processing results as they complete

Use `restate.as_completed()` to process results in the order they finish:

```python {"CODE_LOAD::python/src/develop/journaling_results.py#as_completed"}  theme={null}
call1 = ctx.run_typed(
    "LLM call", call_llm, prompt="What is the weather?", model="gpt-4"
)
call2 = ctx.run_typed(
    "LLM call", call_llm, prompt="What is the weather?", model="gpt-3.5-turbo"
)
async for future in restate.as_completed(call1, call2):
    # do something with the completed future
    print(await future)
```


# Durable Steps
Source: https://docs.restate.dev/develop/python/durable-steps

Persist results of operations.

Restate uses an execution log to replay operations after failures and suspensions. Non-deterministic operations (database calls, HTTP requests, UUID generation) must be wrapped to ensure deterministic replay.

## Run

Use `ctx.run` to safely wrap any non-deterministic operation, like HTTP calls or database responses, and have Restate store its result in the execution log.

```python {"CODE_LOAD::python/src/develop/journaling_results.py#side_effect"}  theme={null}
async def call_llm(prompt: str) -> str:
    # ... implement ...
    return "llm response"

# specify the (async) function to call and its arguments
result = await ctx.run_typed("LLM call", call_llm, prompt="What is the weather?")

# or use a lambda to capture a single value
my_number = await ctx.run_typed("generate number", lambda: random.randint(0, 10))
```

Note that inside `ctx.run`, you cannot use the Restate context (e.g., `ctx.get`, `ctx.sleep`, or nested `ctx.run`).

<AccordionGroup>
  <Accordion title="Serialization">
    By default, the SDK serializes the journal entry with the [`json`](https://docs.python.org/3/library/json.html#) library.
    Alternatively, you can specify a [Pydantic model](/develop/python/serialization#pydantic) or [custom serializer](/develop/python/serialization#custom-serialization).
  </Accordion>

  <Accordion title="Error handling and retry policies">
    Failures in `ctx.run` are treated the same as any other handler error. Restate will retry it unless configured otherwise or unless a [`TerminalError`](/develop/python/error-handling) is thrown.

    You can customize how `ctx.run` retries via:

    ```py {"CODE_LOAD::python/src/develop/retries.py#here"}  theme={null}
    try:
        await ctx.run_typed(
            "write",
            write_to_other_system,
            restate.RunOptions(
                # Initial retry interval
                initial_retry_interval=timedelta(milliseconds=100),
                # Retry policies are exponential, the retry interval will double on each attempt
                retry_interval_factor=2.0,
                # Maximum retry interval
                max_retry_interval=timedelta(seconds=10),
                # Max duration of retries before giving up
                max_duration=timedelta(minutes=5),
                # Max attempts (including the initial) before giving up
                max_attempts=10,
            ))
    except TerminalError as err:
        # Handle the terminal error after retries exhausted
        # For example, undo previous actions (see sagas guide) and
        # propagate the error back to the caller
        raise err
    ```

    * You can limit retries by time or count
    * When the policy is exhausted, a `TerminalError` is thrown
    * See the [Error Handling Guide](/guides/error-handling) and the [Sagas Guide](/guides/sagas) for patterns like compensation
  </Accordion>

  <Accordion title="Increasing timeouts">
    If Restate doesn't receive new journal entries from a service for more than one minute (by default), it will automatically suspend the invocation and retry it.

    However, some business logic can take longer to complete—for example, an LLM call that takes up to 3 minutes to respond.

    In such cases, you can adjust the service’s [inactivity timeout and abort timeout](/services/configuration) settings to accommodate longer execution times.

    For more information, see the [error handling guide](/guides/error-handling).
  </Accordion>
</AccordionGroup>

## Deterministic randoms

When you do non-deterministic operations, like generating UUIDs or random numbers, you must ensure that the results are deterministic on replay.

For example, to generate stable UUIDs for things like idempotency keys:

```python {"CODE_LOAD::python/src/develop/journaling_results.py#uuid"}  theme={null}
my_uuid = ctx.uuid()
```

The SDK provides deterministic helpers for random values — seeded by the invocation ID — so they return the **same result on retries**.

### UUIDs

To generate stable UUIDs for things like idempotency keys:

```python {"CODE_LOAD::python/src/develop/journaling_results.py#uuid"}  theme={null}
my_uuid = ctx.uuid()
```

Do not use this in cryptographic contexts.

### Random numbers

To generate a deterministic float between `0` and `1`:

```python {"CODE_LOAD::python/src/develop/journaling_results.py#random_nb"}  theme={null}
ctx.random().random()
```

This behaves like `Math.random()` but is deterministically replayable.

### Deterministic time

To get the current millis since midnight, January 1, 1970, that is consistent across retries:

```python {"CODE_LOAD::python/src/develop/journaling_results.py#time"}  theme={null}
current_time = await ctx.time()
```


# Scheduling & Timers
Source: https://docs.restate.dev/develop/python/durable-timers

Durable timers, scheduled actions, and sleep, backed by Restate.

Restate provides durable, fault-tolerant timers that allow you to:

* [Sleep](#durable-sleep): Pause a handler for a given duration
* [Send delayed messages](#scheduling-async-tasks): Schedule handler invocations in the future
* [Set timeouts](#timers-and-timeouts) for async operations
* Implement patterns like [cron jobs](#cron-jobs)

## How it works

Restate tracks and manages timers, ensuring they survive failures and restarts.
For example, if a service is sleeping for 12 hours and fails after 8 hours, then Restate will make sure it only sleeps 4 hours more.

If your handler runs on function-as-a-service platforms like AWS Lambda, Restate suspends the handler while it is sleeping, to free up resources.
Since these platforms charge on execution time, this saves costs.

## Durable sleep

To pause a handler for a set duration:

```python {"CODE_LOAD::python/src/develop/durable_timers.py#here"}  theme={null}
await ctx.sleep(delta=timedelta(seconds=10))
```

There is no hard limit on how long you can sleep. You can sleep for months or even years, but keep in mind:

* If you sleep in an [exclusive handler](/foundations/handlers#handler-behavior) in a Virtual Object, all other calls to this object will be queued.
* You need to keep the deployment version until the invocation completes (see [versioning](/services/versioning)).
  So for long sleeps, we recommend breaking this up into multiple handlers that call each other with [delayed messages](/develop/python/service-communication#delayed-messages).

<Accordion title="Clock synchronization Restate Server vs. SDK">
  The Restate SDK calculates the wake-up time based on the delay you specify.
  The Restate Server then uses this calculated time to wake up the handler.
  If the Restate Server and the SDK have different system clocks, the sleep duration might not be accurate.
  So make sure that the system clock of the Restate Server and the SDK have the same timezone and are synchronized.
  A mismatch can cause timers to fire earlier or later than expected.
</Accordion>

## Scheduling async tasks

To invoke a handler at a later time, use [delayed messages](/develop/python/service-communication#delayed-messages).

<Accordion title="Delayed messages vs. sleep and send">
  In theory, you can schedule future work in Restate in two ways:

  1. **Delayed messages** (recommended)
  2. **Sleep + send** - sleeping in the current handler, then sending a message

  At first sight, both approaches might seem to achieve similar results.
  However, we recommend to use delayed messages for the following reasons:

  * **Handler completes immediately**: The calling handler can finish execution and complete the invocation without waiting for the delay to finish
  * **No Virtual Object blocking**: Other requests to the same Virtual Object can be processed during the delay period
  * **Better for service versioning**: No long-running invocations that require keeping old service deployments around for a long time (see [service versioning](/services/versioning))
</Accordion>

## Timers and timeouts

Most context actions are async actions that return a `RestateDurableFuture`.

You can race these against a timer to bound how long your code waits for an operation.

For example, to either wait for the greeting or for the timeout:

```py {"CODE_LOAD::python/src/develop/durable_timers.py#timer"}  theme={null}
match await restate.select(
    greeting=ctx.service_call(my_handler, "Hi"),
    timeout=ctx.sleep(timedelta(seconds=5)),
):
    case ["greeting", greeting]:
        print("Greeting:", greeting)
    case _:
        print("Timeout occurred")
```

For alternative ways of racing async operations, have a look at the [`RestateDurableFuture` Combinators](/develop/python/concurrent-tasks).

## Cron jobs

Restate does not yet include native cron support, but you can implement your own cron scheduler using:

* Durable timers
* Virtual Objects
* A repeat loop or sleep-schedule pattern

Check out the guide on [implementing cron jobs](/guides/cron).


# Error Handling
Source: https://docs.restate.dev/develop/python/error-handling

Stop infinite retries with Terminal Errors.

When your code fails, Restate automatically retries the invocation. Understanding when to stop retrying is critical to building reliable applications.

<Info>
  Check out the [Error Handling guide](/guides/error-handling) to learn more about how Restate handles transient errors, terminal errors, retries, and timeouts.
</Info>

## Understanding Retryable vs Terminal Errors

**Retryable errors** are temporary problems that might succeed if retried:

* Database connection timeout
* Network issues
* Temporary service unavailability

**Terminal errors** are permanent failures that won't be fixed by retrying:

* Invalid user input (wrong format, missing required fields)
* Authorization failures (user doesn't have permission)
* Business logic violations (insufficient balance, duplicate order)

**By default, Restate retries ALL errors**, except `TerminalError`s.

## Raising `TerminalError`

Use `TerminalError` to signal permanent failures.

When you raise `TerminalError` in your handler code, it stops the invocation and marks it as permanently failed:

```python {"CODE_LOAD::python/src/develop/error_handling.py#terminal"}  theme={null}
from restate.exceptions import TerminalError

raise TerminalError("Something went wrong.")
```

When you raise `TerminalError` inside a `ctx.run` block, it fails that specific step, allowing your handler to continue and handle the error:

```python {"CODE_LOAD::python/src/develop/error_handling.py#run"}  theme={null}
try:
    # Await a ctx.run_typed raising an error
    def do_transaction():
        raise TerminalError("Can't write")
    await ctx.run_typed("do transaction", do_transaction)
except TerminalError as err:
    # Handle the terminal error raised by ctx.run_typed
    # For example, undo previous actions...
    await ctx.run_typed("undo transaction", undo_transaction)
    # ...and propagate the error
    raise err
```

Inside `ctx.run`, **all errors are retried** unless you `raise TerminalError` explicitly or set up a [run retry policy](/develop/python/durable-steps#run).

Common use cases for terminal error are:

* Input validation failures: `raise TerminalError("Invalid email format")`
* Business rule violations: `raise TerminalError("Insufficient balance")`
* Resource not found: `raise TerminalError("User ID not found")`

<Info>
  When you throw a terminal error, you may need to undo previous actions to keep your system consistent. Check out our [sagas guide](/guides/sagas) to learn about compensations.
</Info>

## Handling Errors

Most of the time, you only need to catch `TerminalError` to handle permanent failures.

<Warning>
  **DO NOT** use bare `except:` or `except Exception:` in Restate handlers!
  This will catch internal SDK exceptions that **you must not handle**, and which resulting action can lead to unexpected behavior.
</Warning>

❌ **Wrong** - This leads to unexpected behavior:

```python theme={null}
try:
    # Do something with ctx
except:  # BAD: Catches internal SDK exceptions!
    # What happens here will not get recorded!
```

✅ **Correct** - Only catch what you need:

```python theme={null}
try:
    # Do something with ctx
except TerminalError:  # GOOD: Only catches terminal errors
    # Handle permanent failures
```

<Accordion title="Advanced: using finally for resource cleanup">
  <Note>
    Only use `finally` blocks if you're managing resources (files, connections, locks) that must be released even when retries happen.
  </Note>

  When using `finally`, understand what goes where:

  ```python theme={null}
  try:
      # Do something with ctx
  except TerminalError as e:
      # Handle permanent failures
  finally:
      # Release resources acquired during THIS invocation attempt
      # This runs even if the invocation will be retried
  ```

  You can alternatively use `restate.is_internal_exception(e)` to identify whether an exception is a Restate SDK internal exception that should be ignored.

  **Key principle:**

  * `except TerminalError`: For compensations and business logic when the invocation permanently fails
  * `finally`: For releasing resources (files, connections) acquired during the current attempt
</Accordion>

## Retry strategies

By default, Restate does infinite retries with an exponential backoff strategy.

Check out the [error handling guide](/guides/error-handling) to learn how to customize this.


# External Events
Source: https://docs.restate.dev/develop/python/external-events

Handle external events and human-in-the-loop patterns with durable waiting primitives.

Sometimes your handlers need to pause and wait for external processes to complete. This is common in:

* **Human-in-the-loop workflows** (approvals, reviews, manual steps)
* **External system integration** (waiting for webhooks, async APIs)
* **AI agent patterns** (tool execution, human oversight)

This pattern is also known as the **callback** or **task token** pattern.

## Two Approaches

Restate provides two primitives for handling external events:

| Primitive            | Use Case                   | Key Feature                          |
| -------------------- | -------------------------- | ------------------------------------ |
| **Awakeables**       | Services & Virtual Objects | Unique ID-based completion           |
| **Durable Promises** | Workflows only             | Named promises for simpler signaling |

## How it works

Implementing this pattern in a distributed system is tricky, since you need to ensure that the handler can recover from failures and resume waiting for the external event.

Restate promises are durable and distributed. They survive crashes and can be resolved or rejected by any handler in the workflow.

To save costs on FaaS deployments, Restate lets the handler [suspend](/foundations/key-concepts#suspensions-on-faas) while awaiting the promise, and invokes it again when the result is available.

## Awakeables

**Best for:** Services and Virtual Objects where you need to coordinate with external systems.

### Creating and waiting for awakeables

1. **Create an awakeable** - Get a unique ID and promise
2. **Send the ID externally** - Pass the awakeable ID to your external system
3. **Wait for result** - Your handler [suspends](/foundations/key-concepts#suspensions-on-faas) until the external system responds

```py {"CODE_LOAD::python/src/develop/awakeables.py#here"}  theme={null}
id, promise = ctx.awakeable(type_hint=str)

await ctx.run_typed("trigger task", request_human_review, name=name, id=id)

review = await promise
```

<Accordion title="Serialization">
  By default, the SDK serializes the journal entry with the [`json`](https://docs.python.org/3/library/json.html#) library.
  Alternatively, you can specify a [Pydantic model](/develop/python/serialization#pydantic) or [custom serializer](/develop/python/serialization#custom-serialization).
</Accordion>

<Info>
  Note that if you wait for an awakeable in an [exclusive handler](/foundations/handlers#handler-behavior) in a Virtual Object, all other calls to this object will be queued.
</Info>

### Resolving/rejecting Awakeables

External processes complete awakeables in two ways:

* **Resolve** with success data → handler continues normally
* **Reject** with error reason → throws a [terminal error](/develop/python/error-handling) in the waiting handler

#### Via SDK (from other handlers)

**Resolve:**

```python {"CODE_LOAD::python/src/develop/awakeables.py#resolve"}  theme={null}
ctx.resolve_awakeable(name, review)
```

**Reject:**

```python {"CODE_LOAD::python/src/develop/awakeables.py#reject"}  theme={null}
ctx.reject_awakeable(name, "Cannot be reviewed")
```

#### Via HTTP API

External systems can complete awakeables using Restate's HTTP API:

**Resolve with data:**

```shell theme={null}
curl localhost:8080/restate/awakeables/sign_1PePOqp/resolve \
  --json '"Looks good!"'
```

**Reject with error:**

```shell theme={null}
curl localhost:8080/restate/awakeables/sign_1PePOqp/reject \
  -H 'content-type: text/plain' \
  -d 'Review rejected: insufficient documentation'
```

## Durable Promises

**Best for:** Workflows where you need to signal between different workflow handlers.

**Key differences from awakeables:**

* No ID management - use logical names instead
* Scoped to workflow execution lifetime

Use this for:

* Sending data to the run handler
* Have handlers wait for events emitted by the run handler

<Info>
  After a workflow's run handler completes, other handlers can still be called for up to 24 hours (default).
  The results of resolved Durable Promises remain available during this time.
  Update the retention time via the [service configuration](/services/configuration).
</Info>

### Creating and waiting for promises

Wait for a promise by name:

```py {"CODE_LOAD::python/src/develop/durable_promise.py#promise"}  theme={null}
review = await ctx.promise("review", type_hint=str).value()
```

### Resolving/rejecting promises

Resolve/reject from any workflow handler:

```py {"CODE_LOAD::python/src/develop/durable_promise.py#resolve_promise"}  theme={null}
await ctx.promise("review", type_hint=str).resolve(review)
```

### Complete workflow example

```py expandable {"CODE_LOAD::python/src/develop/durable_promise.py#review"}  theme={null}
review_workflow = restate.Workflow("ReviewWorkflow")


@review_workflow.main()
async def run(ctx: restate.WorkflowContext, document_id: str):
    # Send document for review
    await ctx.run_typed("ask review", ask_review, document_id=document_id)

    # Wait for external review submission
    review = await ctx.promise("review", type_hint=str).value()

    # Process the review result
    return process_review(document_id, review)


@review_workflow.handler()
async def submit_review(ctx: restate.WorkflowSharedContext, review: str):
    # Signal the waiting run handler
    await ctx.promise("review", type_hint=str).resolve(review)


app = restate.app([review_workflow])
```

### Two signaling patterns

**External → Workflow** (shown above): External handlers signal the run handler

* Use for human approvals, external API responses, manual interventions
* External handlers call the handler which resolves the promise

**Workflow → External**: Run handler signals other handlers waiting for workflow events

* Use for step completion notifications, status updates, result broadcasting
* Run handler resolves promises that external handlers are awaiting

## Best Practices

* **Use awakeables** for services/objects coordinating with external systems
* **Use durable promises** for workflow signaling
* **Always handle rejections** to gracefully manage failures
* **Include timeouts** for long-running external processes


# Serialization
Source: https://docs.restate.dev/develop/python/serialization

Customize serialization for SDK actions.

Restate sends data over the network for storing state, journaling actions, awakeables, etc.
Therefore, Restate needs to serialize and deserialize the journal entries.

## Default Serialization

By default, the SDK serializes the journal entry with the [`json`](https://docs.python.org/3/library/json.html#) library.
If this does not work for your data type, then you need to specify a custom serializer, as shown below.

## Pydantic

[Pydantic](https://docs.pydantic.dev/latest/) is a data validation and parsing library for Python.
You can use Pydantic models to define the structure of your data: handler input/output, state, etc.

### Using Pydantic

Make sure to install the optional `serde` dependency of the Restate SDK: `restate-sdk[serde]`.

Then do the following:

```python {"CODE_LOAD::python/src/develop/serialization.py#using_pydantic"}  theme={null}
class Delivery(BaseModel):
    timestamp: datetime
    dimensions: tuple[int, int]


class CompletedDelivery(BaseModel):
    status: str
    timestamp: datetime


# For the input/output serialization of your handlers
@my_object.handler()
async def deliver(ctx: ObjectContext, delivery: Delivery) -> CompletedDelivery:

    # To get state
    await ctx.get("delivery", type_hint=Delivery)

    # To serialize awakeable payloads
    ctx.awakeable(type_hint=Delivery)

    # To serialize the results of actions
    await ctx.run_typed("some-task", do_something, restate.RunOptions(type_hint=Delivery))

    return CompletedDelivery(status="delivered", timestamp=datetime.now())
```

### Pydantic & OpenAPI

Pydantic integrates well with [OpenAPI](https://www.openapis.org/). Restate generates the OpenAPI specifications for you.
If you use Pydantic, you can use the OpenAPI-generated clients to interact with your services.
You can find example clients in the UI playground (click on your service in the overview and then on playground).

## msgspec

[msgspec](https://jcristharif.com/msgspec/) is a fast serialization and validation library, with builtin support for JSON, MessagePack, YAML, and TOML.
You can use msgspec Structs to define the structure of your data: handler input/output, state, etc.

### Using msgspec

Make sure to install the optional `serde` dependency of the Restate SDK: `restate-sdk[serde]`.

Then do the following:

```python {"CODE_LOAD::python/src/develop/serialization.py#using_msgspec"}  theme={null}
class Package(msgspec.Struct):
    timestamp: datetime
    dimensions: tuple[int, int]


class CompletedPackage(msgspec.Struct):
    status: str
    timestamp: datetime


# For the input/output serialization of your handlers
@my_object.handler()
async def ship(ctx: ObjectContext, package: Package) -> CompletedPackage:

    # To get state
    await ctx.get("package", type_hint=Package)

    # To serialize awakeable payloads
    ctx.awakeable(type_hint=Package)

    # To serialize the results of actions
    await ctx.run_typed("some-task", do_package_task, restate.RunOptions(type_hint=Package))

    return CompletedPackage(status="shipped", timestamp=datetime.now())
```

### msgspec & OpenAPI

msgspec integrates well with [OpenAPI](https://www.openapis.org/). Restate generates the OpenAPI specifications for you.
If you use msgspec, you can use the OpenAPI-generated clients to interact with your services.
You can find example clients in the UI playground (click on your service in the overview and then on playground).

## Dataclasses

You can also use Python's built-in `dataclasses` to define the structure of your data.
Make sure to install the optional `serde` dependency of the Restate SDK `restate-sdk[serde]`.

Then add a type hint in a similar way as you would with Pydantic.

## Custom Serialization

To write a custom serializer, you implement the `Serde` interface.

For example a custom JSON serializer could look like this:

```python {"CODE_LOAD::python/src/develop/serialization.py#custom"}  theme={null}
class MyData(typing.TypedDict):
    """Represents a response from the GPT model."""

    some_value: str
    my_number: int


class MySerde(Serde[MyData]):
    def deserialize(self, buf: bytes) -> typing.Optional[MyData]:
        if not buf:
            return None
        data = json.loads(buf)
        return MyData(some_value=data["some_value"], my_number=data["some_number"])

    def serialize(self, obj: typing.Optional[MyData]) -> bytes:
        if obj is None:
            return bytes()
        data = {"some_value": obj["some_value"], "some_number": obj["my_number"]}
        return bytes(json.dumps(data), "utf-8")
```

You then use this serializer in your handlers, as follows:

```python {"CODE_LOAD::python/src/develop/serialization.py#using_custom_serde"}  theme={null}
# For the input/output serialization of your handlers
@my_object.handler(input_serde=MySerde(), output_serde=MySerde())
async def my_handler(ctx: ObjectContext, greeting: str) -> str:

    # To serialize state
    await ctx.get("my_state", serde=MySerde())
    ctx.set("my_state", MyData(some_value="Hi", my_number=15), serde=MySerde())

    # To serialize awakeable payloads
    ctx.awakeable(serde=MySerde())

    # etc.

    return "some-output"
```


# Service Communication
Source: https://docs.restate.dev/develop/python/service-communication

Call other services from your handler.

Your Restate handler can call other handlers in three ways:

* **[Request-response calls](#request-response-calls)**: Call and wait for a response
* **[One-way messages](#sending-messages)**: Send a message and continue
* **[Delayed messages](#delayed-messages)**: Send a message after a delay

<Info>
  To call a service from an external application, see the [HTTP](/services/invocation/http) or [Kafka](/services/invocation/kafka) documentation.
</Info>

<Info>[Learn how Restate how it works](/foundations/key-concepts#resilient-communication)</Info>

## Request-response calls

To call a Restate handler and wait for its result:

```python {"CODE_LOAD::python/src/develop/service_communication.py#request_response"}  theme={null}
# import my_service # Import the service module to get to the handler, not the service itself

# To call a Service:
response = await ctx.service_call(my_service_handler, arg="Hi")

# To call a Virtual Object:
response = await ctx.object_call(my_object_handler, key="Mary", arg="Hi")

# To call a Workflow:
# `run` handler — can only be called once per workflow ID
response = await ctx.workflow_call(run, key="my_workflow_id", arg="Hi")
# Other handlers can be called anytime within workflow retention
response = await ctx.workflow_call(
    interact_with_workflow, key="my_workflow_id", arg="Hi"
)
```

Use generic calls when you don't have the service definition or need dynamic service names:

```python {"CODE_LOAD::python/src/develop/service_communication.py#request_response_generic"}  theme={null}
response_bytes = await ctx.generic_call(
    "MyObject", "my_handler", key="Mary", arg=json.dumps("Hi").encode("utf-8")
)
```

<Accordion title="Workflow retention">
  After a workflow's run handler completes, other handlers can still be called for up to 24 hours (default).
  Update this via the [service configuration](/services/configuration).
</Accordion>

<Info>
  Request-response calls between [exclusive handlers](/foundations/handlers#handler-behavior) of Virtual Objects may lead to deadlocks:

  * Cross deadlock: A → B and B → A (same keys).
  * Cycle deadlock: A → B → C → A.

  Use the UI or CLI to [cancel](/services/invocation/managing-invocations#cancel) and unblock deadlocked invocations.
</Info>

## Sending messages

To send a message to another Restate handler without waiting for a response:

```python {"CODE_LOAD::python/src/develop/service_communication.py#one_way"}  theme={null}
# To message a Service:
ctx.service_send(my_service_handler, arg="Hi")

# To message a Virtual Object:
ctx.object_send(my_object_handler, key="Mary", arg="Hi")

# To message a Workflow:
# `run` handler — can only be called once per workflow ID
ctx.workflow_send(run, key="my_wf_id", arg="Hi")
# Other handlers can be called anytime within workflow retention
ctx.workflow_send(interact_with_workflow, key="my_wf_id", arg="Hi")
```

Restate handles message delivery and retries, so the handler can complete and return without waiting for the message to be processed.

Use generic send when you don't have the service definition:

```python {"CODE_LOAD::python/src/develop/service_communication.py#one_way_generic"}  theme={null}
ctx.generic_send("MyService", "my_handler", arg=json.dumps("Hi").encode("utf-8"))
```

<Info>
  Calls to a Virtual Object execute in order of arrival, serially.
  Example:

  ```python {"CODE_LOAD::python/src/develop/service_communication.py#ordering"}  theme={null}
  ctx.object_send(my_object_handler, key="Mary", arg="I'm call A")
  ctx.object_send(my_object_handler, key="Mary", arg="I'm call B")
  ```

  Call A is guaranteed to execute before B. However, other invocations may interleave between A and B.
</Info>

## Delayed messages

To send a message after a delay:

```python {"CODE_LOAD::python/src/develop/service_communication.py#delayed"}  theme={null}
# To message a Service with a delay:
ctx.service_send(my_service_handler, arg="Hi", send_delay=timedelta(hours=5))

# To message a Virtual Object with a delay:
ctx.object_send(
    my_object_handler, key="Mary", arg="Hi", send_delay=timedelta(hours=5)
)

# To message a Workflow with a delay:
ctx.workflow_send(
    run, key="my_workflow_id", arg="Hi", send_delay=timedelta(hours=5)
)
```

Use generic send with a delay when you don't have the service definition:

```python {"CODE_LOAD::python/src/develop/service_communication.py#delayed_generic"}  theme={null}
ctx.generic_send(
    "MyService",
    "my_handler",
    arg=json.dumps("Hi").encode("utf-8"),
    send_delay=timedelta(hours=5),
)
```

<Info>
  Learn [how this is different](/develop/python/durable-timers#scheduling-async-tasks) from sleeping and then sending a message.
</Info>

## Using an idempotency key

To prevent duplicate executions of the same call, add an idempotency key:

```python {"CODE_LOAD::python/src/develop/service_communication.py#idempotency_key"}  theme={null}
await ctx.service_call(
    my_service_handler,
    arg="Hi",
    idempotency_key="my-idempotency-key",
)
```

Restate automatically deduplicates calls made during the same handler execution, so there's no need to provide an idempotency key in that case.
However, if multiple handlers might call the same service independently, you can use an idempotency key to ensure deduplication across those calls.

## Attach to an invocation

To wait for or get the result of a previously sent message:

```python {"CODE_LOAD::python/src/develop/service_communication.py#attach"}  theme={null}
# Send a request, get the invocation id
handle = ctx.service_send(
    my_service_handler, arg="Hi", idempotency_key="my-idempotency-key"
)
invocation_id = await handle.invocation_id()

# Now re-attach
result = await ctx.attach_invocation(invocation_id, type_hint=str)
```

* With an idempotency key: Wait for completion and retrieve the result.
* Without an idempotency key: Can only wait, not retrieve the result.

## Cancel an invocation

To cancel a running handler:

```python {"CODE_LOAD::python/src/develop/service_communication.py#cancel"}  theme={null}
ctx.cancel_invocation(invocation_id)
```

## See also

* **[Error Handling](/develop/python/error-handling)**: Handle failures and terminal errors in service calls
* **[Durable Timers](/develop/python/durable-timers)**: Implement timeouts for your service calls
* **[Serialization](/develop/python/serialization)**: Customize how data is serialized between services
* **[Sagas](/guides/sagas)**: Roll back or compensate for canceled service calls.


# Services
Source: https://docs.restate.dev/develop/python/services

Implementing Restate services with the Python SDK.

The Restate Python SDK is open source and available on [GitHub](https://github.com/restatedev/sdk-python).

The Restate SDK lets you implement **handlers**. Handlers can be part of a **[Basic Service](/foundations/services#basic-service)**, a **[Virtual Object](/foundations/services#virtual-object)**, or a **[Workflow](/foundations/services#workflow)**. This page shows how to define them with the Python SDK.

## Prerequisites

* Python >= v3.11

## Getting started

<Tip>
  Get started quickly with the [Python Quickstart](/quickstart#python).
</Tip>

Add the `restate_sdk[serde]` dependency to your Python project to start developing Restate services.

## Basic Services

[Basic Services](/foundations/services) group related **handlers** and expose them as callable endpoints:

```python {"CODE_LOAD::python/src/develop/my_service.py"}  theme={null}
import restate

my_service = restate.Service("MyService")


@my_service.handler("myHandler")
async def my_handler(ctx: restate.Context, greeting: str) -> str:
    return f"{greeting}!"


app = restate.app([my_service])
```

* Initialize the Service and specify the service name (here `MyService`).
* Annotate each handler with `@my_service.handler()`. Optionally, you can override the handler name (here `myHandler`).
* Each handler can then be called at `<RESTATE_INGRESS>/myService/myHandler`
* Handlers take the `restate.Context` as the first argument.
* Handlers can take one optional JSON-serializable input and must return an optional output. These can be of any primitive type, `TypedDict` or Pydantic model (or see [custom serialization](/develop/python/serialization) for advanced types).
* Finally, initialize the app, bind the service(s) to it, and serve it. Look at the [serving docs](/develop/python/serving) to learn more about serving your app.

## Virtual Objects

[Virtual Objects](/foundations/services) are services that are stateful and key-addressable — each object instance has a unique ID and persistent state.

```python {"CODE_LOAD::python/src/develop/my_virtual_object.py"}  theme={null}
import restate

my_object = restate.VirtualObject("MyVirtualObject")


@my_object.handler("myHandler")
async def my_handler(ctx: restate.ObjectContext, greeting: str) -> str:
    return f"{greeting} {ctx.key()}!"


@my_object.handler(kind="shared")
async def my_concurrent_handler(ctx: restate.ObjectSharedContext, greeting: str) -> str:
    return f"{greeting} {ctx.key()}!"


app = restate.app([my_object])
```

* Initialize a `restate.VirtualObject` and specify the object's name (here `MyVirtualObject`).
* Each instance is identified by a key (accessible via `ctx.key()`).
* Virtual Objects can have [exclusive and shared handlers](/foundations/handlers#handler-behavior).
* Exclusive handlers receive an `ObjectContext`, allowing read/write access to object state.
* Shared handlers have the annotation `kind="shared"` and receive an `ObjectSharedContext`.

## Workflows

[Workflows](/foundations/services) are long-lived processes with a defined lifecycle. They run once per key and are ideal for orchestrating multi-step operations, which require external interaction via signals and queries.

```python {"CODE_LOAD::python/src/develop/my_workflow.py"}  theme={null}
import restate

my_workflow = restate.Workflow("MyWorkflow")


@my_workflow.main()
async def run(ctx: restate.WorkflowContext, req: str) -> str:
    # ... implement workflow logic here ---
    return "success"


@my_workflow.handler()
async def interact_with_workflow(ctx: restate.WorkflowSharedContext, req: str):
    # ... implement interaction logic here ...
    return


app = restate.app([my_workflow])
```

* Initialize a `restate.Workflow` and specify its name (here `MyWorkflow`).
* Every workflow **must** include a `run` handler:
  * This is the main orchestration entry point and is annotated with `@my_workflow.main()`.
  * It runs exactly once per workflow execution and uses the `WorkflowContext`
  * Resubmission of the same workflow will fail with "Previously accepted". The invocation ID can be found in the request header `x-restate-id`.
  * Use `ctx.key()` to access the workflow's unique ID
* Additional handlers are annotated with `@my_workflow.handler()`. They must use the `WorkflowSharedContext` and can signal or query the workflow. They can run concurrently with the `run` handler and until the retention time expires.

## Configuring services

Check out the [service configuration docs](/services/configuration) to learn how to configure service behavior, including timeouts and retention policies.


# Serving
Source: https://docs.restate.dev/develop/python/serving

Create an endpoint to serve your services.

Restate services can run in a few ways: as an HTTP handler, or as an AWS Lambda handler.

## Creating the app

Create the app and bind one or multiple services to it:

```python {"CODE_LOAD::python/src/develop/serving.py#endpoint"}  theme={null}
import restate

app = restate.app(services=[my_service, my_object])
```

<Accordion title="Mounting Restate app on a FastAPI app">
  If you are using [FastAPI](https://fastapi.tiangolo.com/), you can mount the Restate app on a route as follows:

  ```python {"CODE_LOAD::python/src/develop/serving.py#fastapi"}  theme={null}
  from fastapi import FastAPI

  app = FastAPI()

  app.mount("/restate/v1", restate.app(services=[my_service, my_object]))
  ```

  Then, [register the service](/services/versioning#registering-a-deployment) with Restate specifying this route, here `localhost:9080/restate/v1`.
</Accordion>

## Serving the app

The Python SDK follows the [ASGI](https://asgi.readthedocs.io/en/latest/introduction.html) standard for the serving of the services.

[Hypercorn](https://pypi.org/project/Hypercorn/) and [Uvicorn](https://www.uvicorn.org/) are two popular ASGI servers that you can use to serve your Restate services.

### Hypercorn

The templates and examples use [Hypercorn](https://pypi.org/project/Hypercorn/) to serve the services.

#### Hypercorn development server

To use Hypercorn during development, you can run the app with:

```python {"CODE_LOAD::python/src/develop/serving.py#hypercorn"}  theme={null}
if __name__ == "__main__":
    import hypercorn
    import hypercorn.asyncio
    import asyncio

    conf = hypercorn.Config()
    conf.bind = ["0.0.0.0:9080"]
    asyncio.run(hypercorn.asyncio.serve(app, conf))
```

Check out the [Quickstart](/quickstart) for more instructions.

#### Hypercorn production server

To run the app in production, we recommend using a configuration file for Hypercorn:

```toml hypercorn-config.toml theme={null}
bind = "0.0.0.0:9080"
h2_max_concurrent_streams = 2147483647
keep_alive_max_requests = 2147483647
keep_alive_timeout = 2147483647
workers = 8
```

Run the app with:

```shell theme={null}
python -m hypercorn --config hypercorn-config.toml example:app
```

This serves the app you specified in an `example.py` file, which contains [the `app`](/develop/python/serving#creating-the-app).

### Uvicorn

You can also use [Uvicorn](https://www.uvicorn.org/) to serve [the app](/develop/python/serving#creating-the-app).
To run the app with Uvicorn:

```shell theme={null}
uvicorn example:app
```

Uvicorn does not support HTTP/2, so you need to tell Restate to use HTTP/1.1 when [registering the deployment](/services/versioning#deployments-supporting-only-http1-1).

## Validating request identity

SDKs can validate that incoming requests come from a particular Restate
instance. You can find out more about request identity in the [Security docs](/services/security#locking-down-service-access)

```python {"CODE_LOAD::python/src/develop/serving.py#identity"}  theme={null}
app = restate.app(
    services=[my_service],
    identity_keys=["publickeyv1_w7YHemBctH5Ck2nQRQ47iBBqhNHy4FV7t2Usbye2A6f"],
)
```


# State
Source: https://docs.restate.dev/develop/python/state

Store key-value state in Restate.

Restate lets you persist key-value (K/V) state using its embedded K/V store.

## Key characteristics

<Info>
  [Learn about Restate's embedded K/V store](/foundations/key-concepts#consistent-state).
</Info>

**State is only available for Virtual Objects and Workflows.**

Scope & retention:

* For Virtual Objects: State is scoped per object key and retained indefinitely. It is persisted and shared across all invocations for that object until explicitly cleared.
* For Workflows: State is scoped per workflow execution (workflow ID) and retained only for the duration of the workflow’s configured retention time.

Access Rules:

* [Exclusive handlers](/foundations/handlers#handler-behavior) (e.g., `run()` in workflows) can read and write state.
* [Shared handlers](/foundations/handlers#handler-behavior) can only read state and cannot mutate it.

You can inspect and edit the K/V state via the UI and the [CLI](/services/introspection#inspecting-application-state).

## List all state keys

To retrieve all keys for which the current Virtual Object has stored state:

```python {"CODE_LOAD::python/src/develop/state.py#statekeys"}  theme={null}
state_keys = ctx.state_keys()
```

## Get state value

To read a value by key:

```python {"CODE_LOAD::python/src/develop/state.py#get"}  theme={null}
my_string = await ctx.get("my-string-key", type_hint=str) or "default-key"
my_number = await ctx.get("my-number-key", type_hint=int) or 123
```

The return value is `None` if no value was stored.

See the [serialization docs](/develop/python/serialization) to customize the state serializer.

## Set state value

To write or update a value:

```python {"CODE_LOAD::python/src/develop/state.py#set"}  theme={null}
ctx.set("my-key", "my-new-value")
```

## Clear state key

To delete a specific key:

```python {"CODE_LOAD::python/src/develop/state.py#clear"}  theme={null}
ctx.clear("my-key")
```

## Clear all state keys

To remove all stored state for the current Virtual Object:

```python {"CODE_LOAD::python/src/develop/state.py#clear_all"}  theme={null}
ctx.clear_all()
```

## Advanced: Eager vs. lazy state loading

Restate supports two modes for loading state in handlers:

### Eager state (default)

* **How it works**: State is automatically sent with the request when invoking a handler
* **Benefits**: State is available immediately when the handler starts executing
* **Behavior**: All reads and writes to state are local to the handler execution
* **Best for**: Small to medium state objects that are frequently accessed

### Lazy state

* **How it works**: State is fetched on-demand on `get` calls from the Restate Server
* **Benefits**: Reduces initial request size and memory usage
* **Setup**: Enable lazy state in the [service or handler configuration](/services/configuration)
* **Best for**: Large state objects that aren't needed in every handler execution


# Testing
Source: https://docs.restate.dev/develop/python/testing

Utilities to test your handler logic.

The Python SDK has a testing harness to test your Restate handlers.

This uses [Testcontainers](https://testcontainers.com/) to run a Restate Server in a Docker container and provides a client to let you test your Restate handlers.

## Setup

Install the package:

```bash theme={null}
pip install restate_sdk[harness]
```

## Testing handlers

If you have a service as follows:

```python {"CODE_LOAD::python/src/develop/testing.py#service"}  theme={null}
import restate

greeter = restate.Service("greeter")


@greeter.handler()
async def greet(ctx: restate.Context, name: str) -> str:
    return f"Hello {name}!"


app = restate.app(services=[greeter])
```

Then you can test it via:

```python {"CODE_LOAD::python/src/develop/testing.py#testing"}  theme={null}
import restate

with restate.test_harness(app) as harness:
    restate_client = harness.ingress_client()
    print(restate_client.post("/greeter/greet", json="Alice").json())
```


# Rust SDK
Source: https://docs.restate.dev/develop/rust





# Concurrent Tasks
Source: https://docs.restate.dev/develop/ts/concurrent-tasks

Execute multiple tasks concurrently and gather results.

When building resilient applications, you often need to perform multiple operations in parallel to improve performance and user experience. Restate provides durable concurrency primitives that allow you to run tasks concurrently while maintaining deterministic execution during replays.

## When to use concurrent tasks

Use concurrent tasks when you need to:

* Call multiple external services simultaneously (e.g., fetching data from different APIs)
* Race multiple operations and use the first result (e.g., trying multiple LLM providers)
* Implement timeouts by racing an operation against a timer
* Perform batch operations where individual tasks can run in parallel

## Key benefits

* **Deterministic replay**: Restate logs the order of completion, ensuring consistent behavior during failures
* **Fault tolerance**: If your handler fails, tasks that were already completed will be replayed with their results, while pending tasks will be retried

## Parallelizing tasks

Start multiple durable operations concurrently by calling them without immediately awaiting:

```typescript {"CODE_LOAD::ts/src/develop/journaling_results.ts#parallel"}  theme={null}
const call1 = ctx.run("fetch_user", async () =>
  fetchUserData({ userId: 123 })
);
const call2 = ctx.run("fetch_orders", async () =>
  fetchOrderHistory({ userId: 123 })
);
const call3 = ctx
  .serviceClient(analyticsService)
  .calculateMetrics({ userId: 123 });

const user = await call1;
const orders = await call2;
const metrics = await call3;
```

Check out the guide on [parallelizing work](guides/parallelizing-work).

## Retrieving results

Restate provides several patterns for coordinating concurrent tasks. All patterns use `RestatePromise` combinators that log the order of completion, ensuring deterministic behavior during replays.

### Wait for all tasks to complete

Similar to [`Promise.all`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all), waits for all promises to resolve successfully:

```typescript {"CODE_LOAD::ts/src/develop/journaling_results.ts#all"}  theme={null}
const sleepPromise = ctx.sleep({ milliseconds: 100 });
const callPromise = ctx.serviceClient(myService).myHandler("Hi");
const externalCallPromise = ctx.run(() => httpCall());

const resultArray = await RestatePromise.all([
  sleepPromise,
  callPromise,
  externalCallPromise,
]);
```

### Wait for the first successful completion

Similar to [`Promise.any`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/any), returns the first promise that resolves successfully:

```typescript {"CODE_LOAD::ts/src/develop/journaling_results.ts#any"}  theme={null}
const sleepPromise1 = ctx.sleep({ milliseconds: 100 });
const sleepPromise2 = ctx.sleep({ milliseconds: 200 });
const callPromise = ctx.serviceClient(myService).myHandler("Hi");

const firstResult = await RestatePromise.any([
  sleepPromise1,
  sleepPromise2,
  callPromise,
]);
```

### Wait for the first to complete

Similar to [`Promise.race`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/race), returns the first promise to settle (resolve or reject):

```typescript {"CODE_LOAD::ts/src/develop/journaling_results.ts#race"}  theme={null}
const sleepPromise3 = ctx.sleep({ milliseconds: 100 });
const callPromise2 = ctx.serviceClient(myService).myHandler("Hi");

const firstToComplete = await RestatePromise.race([
  sleepPromise3,
  callPromise2,
]);
```

### Wait for all to settle

Similar to [`Promise.allSettled`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/allSettled), waits for all promises to complete regardless of success or failure:

```typescript {"CODE_LOAD::ts/src/develop/journaling_results.ts#allSettled"}  theme={null}
const sleepPromise4 = ctx.sleep({ milliseconds: 100 });
const callPromise3 = ctx.serviceClient(myService).myHandler("Hi");
const externalCallPromise2 = ctx.run(() => httpCall());

const allResults = await RestatePromise.allSettled([
  sleepPromise4,
  callPromise3,
  externalCallPromise2,
]);
```


# Durable Steps
Source: https://docs.restate.dev/develop/ts/durable-steps

Persist results of operations.

Restate uses an execution log to replay operations after failures and suspensions. Non-deterministic operations (database calls, HTTP requests, UUID generation) must be wrapped to ensure deterministic replay.

## Run

Use `ctx.run` to safely wrap any non-deterministic operation, like HTTP calls or database responses, and have Restate store its result in the execution log.

```typescript {"CODE_LOAD::ts/src/develop/journaling_results.ts#side_effect"}  theme={null}
const result = await ctx.run<string>(async () => doDbRequest());
```

Note that inside `ctx.run`, you cannot use the Restate context (e.g., `ctx.get`, `ctx.sleep`, or nested `ctx.run`).

<AccordionGroup>
  <Accordion title="Serialization">
    By default, results are serialized using [JSON](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON). See the [serialization docs](/develop/ts/serialization) to customize this.
  </Accordion>

  <Accordion title="Error handling and retry policies">
    Failures in `ctx.run` are treated the same as any other handler error. Restate will retry it unless configured otherwise or unless a [`TerminalError`](/develop/ts/error-handling) is thrown.

    You can customize how `ctx.run` retries via:

    ```typescript {"CODE_LOAD::ts/src/develop/retries.ts#here"}  theme={null}
    try {
      const myRunRetryPolicy = {
        initialRetryInterval: { milliseconds: 500 },
        retryIntervalFactor: 2,
        maxRetryInterval: { seconds: 1 },
        maxRetryAttempts: 5,
        maxRetryDuration: { seconds: 1 },
      };
      await ctx.run("write", () => writeToOtherSystem(), myRunRetryPolicy);
    } catch (e) {
      if (e instanceof restate.TerminalError) {
        // Undo or compensate here (see Sagas guide)
      }
      throw e;
    }
    ```

    * You can limit retries by time or count
    * When the policy is exhausted, a `TerminalError` is thrown
    * See the [Error Handling Guide](/guides/error-handling) and the [Sagas Guide](/guides/sagas) for patterns like compensation
  </Accordion>

  <Accordion title="Increasing timeouts">
    If Restate doesn't receive new journal entries from a service for more than one minute (by default), it will automatically suspend the invocation and retry it.

    However, some business logic can take longer to complete—for example, an LLM call that takes up to 3 minutes to respond.

    In such cases, you can adjust the service’s [inactivity timeout and abort timeout](/services/configuration) settings to accommodate longer execution times.

    For more information, see the [error handling guide](/guides/error-handling).
  </Accordion>
</AccordionGroup>

## Deterministic randoms

The SDK provides deterministic helpers for random values — seeded by the invocation ID — so they return the **same result on retries**.

### UUIDs

To generate stable UUIDs for things like idempotency keys:

```typescript {"CODE_LOAD::ts/src/develop/journaling_results.ts#uuid"}  theme={null}
const uuid = ctx.rand.uuidv4();
```

Do not use this in cryptographic contexts.

### Random numbers

To generate a deterministic float between `0` and `1`:

```typescript {"CODE_LOAD::ts/src/develop/journaling_results.ts#random_nb"}  theme={null}
const randomNumber = ctx.rand.random();
```

This behaves like `Math.random()` but is deterministically replayable.

### Deterministic time

To get the current millis since midnight, January 1, 1970, that is consistent across retries:

```typescript {"CODE_LOAD::ts/src/develop/journaling_results.ts#time"}  theme={null}
const now = await ctx.date.now();
```


# Scheduling & Timers
Source: https://docs.restate.dev/develop/ts/durable-timers

Durable timers, scheduled actions, and sleep, backed by Restate.

Restate provides durable, fault-tolerant timers that allow you to:

* [Sleep](#durable-sleep): Pause a handler for a given duration
* [Send delayed messages](#scheduling-async-tasks): Schedule handler invocations in the future
* [Set timeouts](#timers-and-timeouts) for async operations
* Implement patterns like [cron jobs](#cron-jobs)

## How it works

Restate tracks and manages timers, ensuring they survive failures and restarts.
For example, if a service is sleeping for 12 hours and fails after 8 hours, then Restate will make sure it only sleeps 4 hours more.

If your handler runs on function-as-a-service platforms like AWS Lambda, Restate suspends the handler while it is sleeping, to free up resources.
Since these platforms charge on execution time, this saves costs.

## Durable sleep

To pause a handler for a set duration:

```typescript {"CODE_LOAD::ts/src/develop/durable_timers.ts#sleep"}  theme={null}
await ctx.sleep({ seconds: 10 });
```

There is no hard limit on how long you can sleep. You can sleep for months or even years, but keep in mind:

* If you sleep in an [exclusive handler](/foundations/handlers#handler-behavior) in a Virtual Object, all other calls to this object will be queued.
* You need to keep the deployment version until the invocation completes (see [versioning](/services/versioning)).
  So for long sleeps, we recommend breaking this up into multiple handlers that call each other with [delayed messages](/develop/ts/service-communication#delayed-messages).

<Accordion title="Clock synchronization Restate Server vs. SDK">
  The Restate SDK calculates the wake-up time based on the delay you specify.
  The Restate Server then uses this calculated time to wake up the handler.
  If the Restate Server and the SDK have different system clocks, the sleep duration might not be accurate.
  So make sure that the system clock of the Restate Server and the SDK have the same timezone and are synchronized.
  A mismatch can cause timers to fire earlier or later than expected.
</Accordion>

## Scheduling async tasks

To invoke a handler at a later time, use [delayed messages](/develop/ts/service-communication#delayed-messages).

<Accordion title="Delayed messages vs. sleep and send">
  In theory, you can schedule future work in Restate in two ways:

  1. **Delayed messages** (recommended)
  2. **Sleep + send** - sleeping in the current handler, then sending a message

  At first sight, both approaches might seem to achieve similar results.
  However, we recommend to use delayed messages for the following reasons:

  * **Handler completes immediately**: The calling handler can finish execution and complete the invocation without waiting for the delay to finish
  * **No Virtual Object blocking**: Other requests to the same Virtual Object can be processed during the delay period
  * **Better for service versioning**: No long-running invocations that require keeping old service deployments around for a long time (see [service versioning](/services/versioning))
</Accordion>

## Timers and timeouts

Most context actions are async actions that return a `RestatePromise`.

`RestatePromise` supports setting timeouts, allowing you to bound how long your code waits for an operation.

When an operation times out, it throws a `TimeoutError`.

```typescript {"CODE_LOAD::ts/src/develop/durable_timers.ts#timer"}  theme={null}
try {
  await ctx
    .serviceClient(myService)
    .myHandler("Hi")
    .orTimeout({ seconds: 5 });
} catch (error) {
  if (error instanceof restate.TimeoutError) {
    console.error("Operation timed out:", error);
  } else {
    throw error; // Re-throw other errors
  }
}
```

For alternative ways of racing async operations, have a look at the [`RestatePromise` Combinator docs](/develop/ts/concurrent-tasks).

## Cron jobs

Restate does not yet include native cron support, but you can implement your own cron scheduler using:

* Durable timers
* Virtual Objects
* A repeat loop or sleep-schedule pattern

Check out the guide on [implementing cron jobs](/guides/cron).


# Error Handling
Source: https://docs.restate.dev/develop/ts/error-handling

Stop infinite retries with Terminal Errors.

Restate handles retries for failed invocations.

<Info>
  Check out the [Error Handling guide](/guides/error-handling) to learn more about how Restate handles transient errors, terminal errors, retries, and timeouts.
</Info>

## Retry strategies

By default, Restate does infinite retries with an exponential backoff strategy.

Check out the [error handling guide](/guides/error-handling) to learn how to customize this.

## Terminal Errors

For failures for which you do not want retries, but instead want the invocation to end and the error message
to be propagated back to the caller, you can throw a **terminal error**.

You can throw a `TerminalError` with an optional HTTP status code and a message anywhere in your handler, as follows:

```typescript {"CODE_LOAD::ts/src/develop/error_handling.ts#terminal"}  theme={null}
throw new TerminalError("Something went wrong.", { errorCode: 500 });
```

You can catch terminal errors, and build your control flow around it.

<Info>
  When you throw a terminal error, you might need to undo the actions you did earlier in your handler to make sure that your system remains in a consistent state.
  Have a look at our [sagas guide](/guides/sagas) to learn more.
</Info>

## Mapping errors to `TerminalError`

If you're using external libraries (e.g., for validation), you might want to automatically convert certain error types into terminal errors.

You can do this using the `asTerminalError` option in your [service configuration](/services/configuration).

For example, to fail with `TerminalError` for each `MyValidationError`, do the following:

```typescript {"CODE_LOAD::ts/src/develop/error_handling.ts#as_terminal"}  theme={null}
class MyValidationError extends Error {}

const greeter = restate.service({
  name: "greeter",
  handlers: {
    greet: async (ctx: restate.Context, name: string) => {
      if (name.length === 0) {
        throw new MyValidationError("Length too short");
      }
      return `Hello ${name}`;
    },
  },
  options: {
    asTerminalError: (err) => {
      if (err instanceof MyValidationError) {
        // My validation error is terminal
        return new restate.TerminalError(err.message, { errorCode: 400 });
      }
    },
  },
});
```


# External Events
Source: https://docs.restate.dev/develop/ts/external-events

Handle external events and human-in-the-loop patterns with durable waiting primitives.

Sometimes your handlers need to pause and wait for external processes to complete. This is common in:

* **Human-in-the-loop workflows** (approvals, reviews, manual steps)
* **External system integration** (waiting for webhooks, async APIs)
* **AI agent patterns** (tool execution, human oversight)

This pattern is also known as the **callback** or **task token** pattern.

## Two Approaches

Restate provides two primitives for handling external events:

| Primitive            | Use Case                   | Key Feature                          |
| -------------------- | -------------------------- | ------------------------------------ |
| **Awakeables**       | Services & Virtual Objects | Unique ID-based completion           |
| **Durable Promises** | Workflows only             | Named promises for simpler signaling |

## How it works

Implementing this pattern in a distributed system is tricky, since you need to ensure that the handler can recover from failures and resume waiting for the external event.

Restate promises are durable and distributed. They survive crashes and can be resolved or rejected by any handler in the workflow.

To save costs on FaaS deployments, Restate lets the handler [suspend](/foundations/key-concepts#suspensions-on-faas) while awaiting the promise, and invokes it again when the result is available.

## Awakeables

**Best for:** Services and Virtual Objects where you need to coordinate with external systems.

### Creating and waiting for awakeables

1. **Create an awakeable** - Get a unique ID and promise
2. **Send the ID externally** - Pass the awakeable ID to your external system
3. **Wait for result** - Your handler [suspends](/foundations/key-concepts#suspensions-on-faas) until the external system responds

```ts {"CODE_LOAD::ts/src/develop/awakeable.ts#here"}  theme={null}
// Create awakeable and get unique ID
const { id, promise } = ctx.awakeable<string>();

// Send ID to external system (email, queue, webhook, etc.)
await ctx.run(() => requestHumanReview(name, id));

// Handler suspends here until external completion
const review = await promise;
```

<Accordion title="Serialization">
  Both awakeables and durable promises use built-in [JSON](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON) for (de)serialization. Complex objects are automatically serialized/deserialized. See the [serialization docs](/develop/ts/serialization) for customization options.
</Accordion>

<Info>
  Note that if you wait for an awakeable in an [exclusive handler](/foundations/handlers#handler-behavior) in a Virtual Object, all other calls to this object will be queued.
</Info>

### Resolving/rejecting Awakeables

External processes complete awakeables in two ways:

* **Resolve** with success data → handler continues normally
* **Reject** with error reason → throws a [terminal error](/develop/ts/error-handling) in the waiting handler

#### Via SDK (from other handlers)

**Resolve:**

```ts {"CODE_LOAD::../snippets/ts/src/develop/awakeable.ts#resolve"}  theme={null}
// Complete with success data
ctx.resolveAwakeable(id, { approved: true, comments: "Looks good!" });
```

**Reject:**

```ts {"CODE_LOAD::../snippets/ts/src/develop/awakeable.ts#reject"}  theme={null}
// Complete with error
ctx.rejectAwakeable(id, "This cannot be reviewed.");
```

#### Via HTTP API

External systems can complete awakeables using Restate's HTTP API:

**Resolve with data:**

```shell theme={null}
curl localhost:8080/restate/awakeables/sign_1PePOqp/resolve \
  --json '"Looks good!"'
```

**Reject with error:**

```shell theme={null}
curl localhost:8080/restate/awakeables/sign_1PePOqp/reject \
  -H 'content-type: text/plain' \
  -d 'Review rejected: insufficient documentation'
```

## Durable Promises

**Best for:** Workflows where you need to signal between different workflow handlers.

**Key differences from awakeables:**

* No ID management - use logical names instead
* Scoped to workflow execution lifetime

Use this for:

* Sending data to the run handler
* Have handlers wait for events emitted by the run handler

<Info>
  After a workflow's run handler completes, other handlers can still be called for up to 24 hours (default).
  The results of resolved Durable Promises remain available during this time.
  Update the retention time via the [service configuration](/services/configuration).
</Info>

### Creating and waiting for promises

Wait for a promise by name:

```ts {"CODE_LOAD::../snippets/ts/src/develop/awakeable.ts#promise"}  theme={null}
const review = await ctx.promise<string>("review");
```

### Resolving/rejecting promises

Resolve/reject from any workflow handler:

```ts {"CODE_LOAD::../snippets/ts/src/develop/awakeable.ts#resolve_promise"}  theme={null}
await ctx.promise<string>("review").resolve(review);
```

### Complete workflow example

```ts expandable {"CODE_LOAD::../snippets/ts/src/develop/awakeable.ts#review"}  theme={null}
restate.workflow({
  name: "reviewWorkflow",
  handlers: {
    // Main workflow execution
    run: async (ctx: WorkflowContext, documentId: string) => {
      // Send document for review
      await ctx.run(() => askReview(documentId));

      // Wait for external review submission
      const review = await ctx.promise<string>("review");

      // Process the review result
      return processReview(documentId, review);
    },

    // External endpoint to submit reviews
    submitReview: async (
      ctx: restate.WorkflowSharedContext,
      review: string
    ) => {
      // Signal the waiting run handler
      await ctx.promise<string>("review").resolve(review);
    },
  },
});
```

### Two signaling patterns

**External → Workflow** (shown above): External handlers signal the run handler

* Use for human approvals, external API responses, manual interventions
* External handlers call the handler which resolves the promise

**Workflow → External**: Run handler signals other handlers waiting for workflow events

* Use for step completion notifications, status updates, result broadcasting
* Run handler resolves promises that external handlers are awaiting

## Best Practices

* **Use awakeables** for services/objects coordinating with external systems
* **Use durable promises** for workflow signaling
* **Always handle rejections** to gracefully manage failures
* **Include timeouts** for long-running external processes


# Logging
Source: https://docs.restate.dev/develop/ts/logging

Configure the log level of your services.

The TypeScript SDK includes an internal logger to help you monitor and debug your services.

## Log Levels

You can control the verbosity of logs using environment variables:

* **Set the log level:**\
  Use the `RESTATE_LOGGING` environment variable.\
  Possible values: `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`.

* **Default log level:**
  * `INFO` if `NODE_ENV=production`
  * `DEBUG` otherwise

* **Verbose journal logging:**\
  If you set `RESTATE_LOGGING=TRACE`, you can enable even more detailed journal logs with:\
  `RESTATE_JOURNAL_LOGGING=TRACE`

## Console Logging

By default, using the Node.js console logger will print log statements repeatedly during replays.\
**To avoid duplicate logs during replays, use the Restate context logger.**

The context logger wraps the console and suppresses duplicate log statements during replays:

```typescript {"CODE_LOAD::ts/src/develop/logging.ts"}  theme={null}
import * as restate from "@restatedev/restate-sdk";

const service = restate.object({
  name: "Greeter",
  handlers: {
    greet: async (ctx: restate.ObjectContext, name: string) => {
      ctx.console.info("This will not be printed again during replays");
      ctx.console.debug("This will not be printed again during replays");
      // Any other console logging method can be used
    },
  },
});
```

* The context logger uses the same log level filtering as described above.
* Use `ctx.console` for all logging inside handlers to ensure clean, non-redundant logs.


# Serialization
Source: https://docs.restate.dev/develop/ts/serialization

Customize serialization for SDK actions.

Restate sends data over the network for storing state, journaling actions, awakeables, etc.
Therefore, Restate needs to serialize and deserialize the journal entries.

## Default serialization

By default, Typescript SDK uses the built-in [JSON](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON) support to perform (de)serialization.

## Standard Schema & Zod

You can use any [Standard schema](http://standardschema.dev/) compliant library for your handler input/output schemas, such as [Zod](https://zod.dev/):

```typescript {"CODE_LOAD::ts/src/develop/serialization_standard_schema.ts#standard_schema"}  theme={null}
import * as restate from "@restatedev/restate-sdk";
import { z } from "zod";

const Greeting = z.object({
  name: z.string(),
});

const GreetingResponse = z.object({
  result: z.string(),
});

const greeter = restate.service({
  name: "Greeter",
  handlers: {
    greet: restate.handlers.handler(
      {
        input: restate.serde.schema(Greeting),
        output: restate.serde.schema(GreetingResponse),
      },
      async (ctx: restate.Context, { name }) => {
        return { result: `You said hi to ${name}!` };
      }
    ),
  },
});
```

<Accordion title="For Zod < 4.2">
  If you're using Zod versions earlier than 4.2, you should use the ad-hoc Zod integration instead: [`@restatedev/restate-sdk-zod`](https://www.npmjs.com/package/@restatedev/restate-sdk-zod).

  Then do the following:

  ```typescript {"CODE_LOAD::ts/src/develop/serialization.ts#zod"}  theme={null}
  import * as restate from "@restatedev/restate-sdk";
  import { z } from "zod";
  import { serde } from "@restatedev/restate-sdk-zod";

  const Greeting = z.object({
    name: z.string(),
  });

  const GreetingResponse = z.object({
    result: z.string(),
  });

  const greeter = restate.service({
    name: "Greeter",
    handlers: {
      greet: restate.handlers.handler(
        { input: serde.zod(Greeting), output: serde.zod(GreetingResponse) },
        async (ctx: restate.Context, { name }) => {
          return { result: `You said hi to ${name}!` };
        }
      ),
    },
  });
  ```
</Accordion>

## Custom serialization

It is possible to implement customized serialization using the `Serde` interface.

For example, to implement custom serializers for the handler input and output:

```typescript {"CODE_LOAD::ts/src/develop/serialization.ts#service_definition"}  theme={null}
const myService = restate.service({
  name: "MyService",
  handlers: {
    myHandler: restate.handlers.handler(
      {
        // Set the input serde here
        input: restate.serde.binary,
        // Set the output serde here
        output: restate.serde.binary,
      },
      async (ctx: Context, data: Uint8Array): Promise<Uint8Array> => {
        // Process the request
        return data;
      }
    ),
  },
});
```

When sending a request to a handler configured with custom serde(s) you always need to manually specify them, because the client does not automatically infer what serde(s) should be used.
To customize the serde to use on requests:

```typescript {"CODE_LOAD::ts/src/develop/serialization.ts#client"}  theme={null}
ctx.serviceClient(myService).myHandler(
  input,
  restate.rpc.opts({
    input: restate.serde.binary,
    output: restate.serde.binary,
  })
);
```

To customize the serde for other actions on the context:

```typescript {"CODE_LOAD::ts/src/develop/serialization.ts#actions"}  theme={null}
ctx.get("my-binary-data", restate.serde.binary);
ctx.set("my-binary-data", new Uint8Array(), restate.serde.binary);

ctx.awakeable(restate.serde.binary);

await ctx.run("my-side-effect", () => new Uint8Array(), {
  serde: restate.serde.binary,
});
```


# Service Communication
Source: https://docs.restate.dev/develop/ts/service-communication

Call other services from your handler.

Your Restate handler can call other handlers in three ways:

* **[Request-response calls](#request-response-calls)**: Call and wait for a response
* **[One-way messages](#sending-messages)**: Send a message and continue
* **[Delayed messages](#delayed-messages)**: Send a message after a delay

<Info>
  To call a service from an external application, see the [HTTP](/services/invocation/http), [Kafka](/services/invocation/kafka), or [SDK Clients](/services/invocation/clients/typescript-sdk) documentation.
</Info>

<Info>[Learn how Restate how it works](/foundations/key-concepts#resilient-communication)</Info>

## Request-response calls

To call a Restate handler and wait for its result:

```ts {"CODE_LOAD::ts/src/develop/service_communication.ts#request_response"}  theme={null}
// To call a Service:
const svcResponse = await ctx.serviceClient(myService).myHandler("Hi");

// To call a Virtual Object:
const objResponse = await ctx
  .objectClient(myObject, "Mary")
  .myHandler("Hi");

// To call a Workflow:
// `run` handler — can only be called once per workflow ID
const wfResponse = await ctx
  .workflowClient(myWorkflow, "wf-id")
  .run("Hi");
// Other handlers can be called anytime within workflow retention
const result = await ctx
  .workflowClient(myWorkflow, "wf-id")
  .interactWithWorkflow();
```

Use generic calls when you don't have the service definition or need dynamic service names:

```ts {"CODE_LOAD::ts/src/develop/service_communication.ts#generic_call"}  theme={null}
const response = await ctx.genericCall({
  service: "MyObject",
  method: "myHandler",
  parameter: "Hi",
  key: "Mary", // drop this for Service calls
  inputSerde: restate.serde.json,
  outputSerde: restate.serde.json,
});
```

<Accordion title="Workflow retention">
  After a workflow's run handler completes, other handlers can still be called for up to 24 hours (default).
  Update this via the [service configuration](/services/configuration).
</Accordion>

<Info>
  Request-response calls between [exclusive handlers](/foundations/handlers#handler-behavior) of Virtual Objects may lead to deadlocks:

  * Cross deadlock: A → B and B → A (same keys).
  * Cycle deadlock: A → B → C → A.

  Use the UI or CLI to [cancel](/services/invocation/managing-invocations#cancel) and unblock deadlocked invocations.
</Info>

## Sending messages

To send a message to another Restate handler without waiting for a response:

```ts {"CODE_LOAD::ts/src/develop/service_communication.ts#one_way"}  theme={null}
// To message a Service:
ctx.serviceSendClient(myService).myHandler("Hi");

// To message a Virtual Object:
ctx.objectSendClient(myObject, "Mary").myHandler("Hi");

// To message a Workflow:
// `run` handler — can only be called once per workflow ID
ctx.workflowSendClient(myWorkflow, "wf-id").run("Hi");
// Other handlers can be called anytime within workflow retention
ctx.workflowSendClient(myWorkflow, "wf-id").interactWithWorkflow();
```

Restate handles message delivery and retries, so the handler can complete and return without waiting for the message to be processed.

Use generic send when you don't have the service definition:

```ts {"CODE_LOAD::ts/src/develop/service_communication.ts#generic_send"}  theme={null}
ctx.genericSend({
  service: "MyService",
  method: "myHandler",
  parameter: "Hi",
  inputSerde: restate.serde.json,
});
```

<Info>
  Calls to a Virtual Object execute in order of arrival, serially.
  Example:

  ```typescript {"CODE_LOAD::ts/src/develop/service_communication.ts#ordering"}  theme={null}
  ctx.objectSendClient(myObject, "Mary").myHandler("I'm call A");
  ctx.objectSendClient(myObject, "Mary").myHandler("I'm call B");
  ```

  Call A is guaranteed to execute before B. However, other invocations may interleave between A and B.
</Info>

## Delayed messages

To send a message after a delay:

```ts {"CODE_LOAD::ts/src/develop/service_communication.ts#delayed"}  theme={null}
// To message a Service with a delay:
ctx
  .serviceSendClient(myService)
  .myHandler("Hi", restate.rpc.sendOpts({ delay: { hours: 5 } }));

// To message a Virtual Object with a delay:
ctx
  .objectSendClient(myObject, "Mary")
  .myHandler("Hi", restate.rpc.sendOpts({ delay: { hours: 5 } }));

// To message a Workflow with a delay:
ctx
  .workflowSendClient(myWorkflow, "Mary")
  .run("Hi", restate.rpc.sendOpts({ delay: { hours: 5 } }));
```

Use generic send with a delay when you don't have the service definition:

```ts {"CODE_LOAD::ts/src/develop/service_communication.ts#generic_delayed"}  theme={null}
ctx.genericSend({
  service: "MyService",
  method: "myHandler",
  parameter: "Hi",
  inputSerde: restate.serde.json,
  delay: { seconds: 5 },
});
```

<Info>
  Learn [how this is different](/develop/ts/durable-timers#scheduling-async-tasks) from sleeping and then sending a message.
</Info>

## Using an idempotency key

To prevent duplicate executions of the same call, add an idempotency key:

```ts {"CODE_LOAD::ts/src/develop/service_communication.ts#idempotency_key"}  theme={null}
// For request-response
const response = await ctx.serviceClient(myService).myHandler(
  "Hi",
  restate.rpc.opts({
    idempotencyKey: "my-idempotency-key",
  })
);
// For sending a message
ctx.serviceSendClient(myService).myHandler(
  "Hi",
  restate.rpc.sendOpts({
    idempotencyKey: "my-idempotency-key",
  })
);
```

Restate automatically deduplicates calls made during the same handler execution, so there's no need to provide an idempotency key in that case.
However, if multiple handlers might call the same service independently, you can use an idempotency key to ensure deduplication across those calls.

## Attach to an invocation

To wait for or get the result of a previously sent message:

```ts {"CODE_LOAD::ts/src/develop/service_communication.ts#attach"}  theme={null}
const handle = ctx.serviceSendClient(myService).myHandler(
  "Hi",
  restate.rpc.sendOpts({
    idempotencyKey: "my-idempotency-key",
  })
);
const invocationId = await handle.invocationId;

// Later...
const response = ctx.attach(invocationId);
```

* With an idempotency key: Wait for completion and retrieve the result.
* Without an idempotency key: Can only wait, not retrieve the result.

## Cancel an invocation

To cancel a running handler:

```ts {"CODE_LOAD::ts/src/develop/service_communication.ts#cancel"}  theme={null}
const handle = ctx.serviceSendClient(myService).myHandler("Hi");
const invocationId = await handle.invocationId;

// Cancel the invocation
ctx.cancel(invocationId);
```

## Advanced: sharing service type definitions

When calling services, you need access to their type definitions for type safety. Here are four approaches to share service definitions without exposing implementation details:

<AccordionGroup>
  <Accordion title="Option 1: Export type-only service definition">
    Export only the service type information from where you define your service:

    ```typescript {"CODE_LOAD::ts/src/develop/my_service.ts#api_export"}  theme={null}
    export const myService = restate.service({
      name: "MyService",
      handlers: {
        myHandler: async (ctx: restate.Context, greeting: string) => {
          return `${greeting}!`;
        },
      },
    });

    export type MyService = typeof myService;
    ```

    Import and use this definition in calling handlers:

    ```ts {"CODE_LOAD::ts/src/develop/service_communication.ts#export_definition"}  theme={null}
    const response = await ctx
      .serviceClient<MyService>({ name: "MyService" })
      .myHandler("Hi");
    ```
  </Accordion>

  <Accordion title="Option 2: Define service interface manually">
    Create a TypeScript interface matching the service's handler signatures. Useful when the service is in a different language or when you can't import the type definition:

    ```ts {"CODE_LOAD::ts/src/develop/service_communication.ts#interface"}  theme={null}
    interface MyService {
      myHandler(ctx: unknown, greeting: string): Promise<string>;
    }
    const response = await ctx
      .serviceClient<MyService>({ name: "MyService" })
      .myHandler("Hi");
    ```
  </Accordion>

  <Accordion title="Option 3: Import as dev dependency">
    When your service is published as a separate package, import it as a dev dependency to access its types:

    ```ts {"CODE_LOAD::ts/src/develop/service_communication.ts#dev_dependency"}  theme={null}
    // import type { MyService } from "my-service-package";

    const response = await ctx
      .serviceClient<MyService>({ name: "MyService" })
      .myHandler("Hi");
    ```
  </Accordion>
</AccordionGroup>

## See also

* **[SDK Clients](/develop/ts/service-communication)**: Call Restate services from external applications
* **[Error Handling](/develop/ts/error-handling)**: Handle failures and terminal errors in service calls
* **[Durable Timers](/develop/ts/durable-timers)**: Implement timeouts for your service calls
* **[Serialization](/develop/ts/serialization)**: Customize how data is serialized between services
* **[Sagas](/guides/sagas)**: Roll back or compensate for canceled service calls.


# Services
Source: https://docs.restate.dev/develop/ts/services

Implementing Restate services with the TypeScript SDK.

The Restate TypeScript SDK is open source and available on [GitHub](https://github.com/restatedev/sdk-typescript).

The Restate SDK lets you implement **handlers**. Handlers can be part of a **[Basic Service](/foundations/services#basic-service)**, a **[Virtual Object](/foundations/services#virtual-object)**, or a **[Workflow](/foundations/services#workflow)**. This page shows how to define them with the TypeScript SDK.

## Prerequisites

* [NodeJS](https://nodejs.org/en/) >= v20 or [Bun](https://bun.sh/docs/installation) or [Deno](https://deno.land/#installation)
* [npm CLI](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm) >= 9.6.7

## Getting started

<Tip>
  Get started quickly with the [TypeScript Quickstart](/quickstart).
</Tip>

Add the [`@restatedev/restate-sdk`](https://www.npmjs.com/package/@restatedev/restate-sdk) dependency to your project to start developing Restate services.

## Basic Services

[Basic Services](/foundations/services) group related **handlers** and expose them as callable endpoints:

```ts {"CODE_LOAD::ts/src/develop/service.ts"}  theme={null}
import * as restate from "@restatedev/restate-sdk";

export const myService = restate.service({
  name: "MyService",
  handlers: {
    myHandler: async (ctx: restate.Context, greeting: string) => {
      return `${greeting}!`;
    },
  },
});

restate.serve({ services: [myService] });
```

* Define a service using [`restate.service`](https://restatedev.github.io/sdk-typescript/functions/_restatedev_restate-sdk.service).
* The service has a name and a list of handlers.
* Each handler has a name and can be called at `<RESTATE_INGRESS>/myService/myHandler`
* Handlers take the [`Context`](https://restatedev.github.io/sdk-typescript/interfaces/_restatedev_restate-sdk.Context) as the first argument.
* Handlers can take one optional JSON-serializable input and must return a JSON-serializable output (see [custom serialization](/develop/ts/serialization) for advanced types).
* Serve the service over HTTP (port `9080` by default).

## Virtual Objects

[Virtual Objects](/foundations/services) are services that are stateful and key-addressable — each object instance has a unique ID and persistent state.

```ts {"CODE_LOAD::ts/src/develop/virtual_object.ts"}  theme={null}
import * as restate from "@restatedev/restate-sdk";

export const myObject = restate.object({
  name: "MyObject",
  handlers: {
    myHandler: async (ctx: restate.ObjectContext, greeting: string) => {
      return `${greeting} ${ctx.key}!`;
    },
    myConcurrentHandler: restate.handlers.object.shared(
      async (ctx: restate.ObjectSharedContext, greeting: string) => {
        return `${greeting} ${ctx.key}!`;
      }
    ),
  },
});

restate.serve({ services: [myObject] });
```

* Define a Virtual Object using [`restate.object(...)`](https://restatedev.github.io/sdk-typescript/functions/_restatedev_restate-sdk.object)
* Each instance is identified by a key (accessible via `ctx.key`).
* Virtual Objects can have [exclusive and shared handlers](/foundations/handlers#handler-behavior).
* Exclusive handlers receive an [`ObjectContext`](https://restatedev.github.io/sdk-typescript/interfaces/_restatedev_restate-sdk.ObjectContext), allowing read/write access to object state.
* Shared handlers are wrapped in `handlers.object.shared(...)` and use the [`ObjectSharedContext`](https://restatedev.github.io/sdk-typescript/interfaces/_restatedev_restate-sdk.ObjectSharedContext)
* Serve the Virtual Object over HTTP (port `9080` by default).

## Workflows

[Workflows](/foundations/services) are long-lived processes with a defined lifecycle. They run once per key and are ideal for orchestrating multi-step operations, which require external interaction via signals and queries.

```ts {"CODE_LOAD::ts/src/develop/workflow.ts"}  theme={null}
import * as restate from "@restatedev/restate-sdk";

export const myWorkflow = restate.workflow({
  name: "MyWorkflow",
  handlers: {
    run: async (ctx: restate.WorkflowContext, req: string) => {
      // implement workflow logic here

      return "success";
    },

    interactWithWorkflow: async (ctx: restate.WorkflowSharedContext) => {
      // implement interaction logic here
      // e.g. resolve a promise that the workflow is waiting on
    },
  },
});

restate.serve({ services: [myWorkflow] });
```

* Define a workflow with [`restate.workflow(...)`](https://restatedev.github.io/sdk-typescript/functions/_restatedev_restate-sdk.workflow)
* Every workflow **must** include a `run` handler:
  * This is the main orchestration entry point
  * It runs exactly once per workflow execution and uses the [`WorkflowContext`](https://restatedev.github.io/sdk-typescript/interfaces/_restatedev_restate-sdk.WorkflowContext)
  * Resubmission of the same workflow will fail with "Previously accepted". The invocation ID can be found in the request header `x-restate-id`.
  * Use `ctx.key` to access the workflow's unique ID
* Additional handlers must use the [`WorkflowSharedContext`](https://restatedev.github.io/sdk-typescript/interfaces/_restatedev_restate-sdk.WorkflowSharedContext) and can signal or query the workflow. They can run concurrently with the `run` handler and until the retention time expires.
* Serve the Workflow over HTTP (port `9080` by default).

## Configuring services

Check out the [service configuration docs](/services/configuration) to learn how to configure service behavior, including timeouts and retention policies.


# Serving
Source: https://docs.restate.dev/develop/ts/serving

Create an endpoint to serve your services.

Restate services can run in a few ways: as a Node.js HTTP handler, as an AWS
Lambda handler, or on other Javascript runtimes like Bun, Deno and Cloudflare
Workers.

## Creating a Node.js HTTP handler

Use `restate.serve` to serve the provided services, starting an HTTP/2 server on port `9080`.

```typescript {"CODE_LOAD::ts/src/develop/serving.ts#endpoint"}  theme={null}
import * as restate from "@restatedev/restate-sdk";
restate.serve({
  services: [myService, myVirtualObject, myWorkflow],
});
```

<Accordion title="Customizing the HTTP2 server">
  If you need to manually control or customize the HTTP2 server, use `restate.createEndpointHandler` to create a Node HTTP/2 handler, and then use it to manually instantiate the HTTP server:

  ```ts {"CODE_LOAD::ts/src/develop/serving.ts#custom_endpoint"}  theme={null}
  const http2Handler = restate.createEndpointHandler({
    services: [myService, myVirtualObject, myWorkflow],
  });
  const httpServer = http2.createServer(http2Handler);
  httpServer.listen();
  ```
</Accordion>

## Creating a Lambda handler

To register your service as a Lambda function, use the `/lambda` import
component and use `restate.createEndpointHandler`:

```typescript {"CODE_LOAD::ts/src/develop/serving_lambda.ts#lambda"}  theme={null}
import * as restate from "@restatedev/restate-sdk/lambda";
export const handler = restate.createEndpointHandler({
  services: [myService, myVirtualObject, myWorkflow],
});
```

The implementation of your services and handlers remains the same for both deployment options.

Have a look at the [deployment section](/services/deploy/lambda)
for guidance on how to deploy your services on AWS Lambda.

## Creating a Deno/Cloudflare Workers handler

Other Javascript runtimes like Deno and Cloudflare Workers have
built on top of the [Fetch Standard](https://github.com/whatwg/fetch) for
defining HTTP server handlers. To register your service as a fetch handler, use
the `/fetch` import component.

```typescript {"CODE_LOAD::ts/src/develop/serving_fetch.ts#fetch"}  theme={null}
import * as restate from "@restatedev/restate-sdk/fetch";
const handler = restate.createEndpointHandler({
  services: [myService, myVirtualObject, myWorkflow],
});
// Cloudflare expects the handler as a default export
export default handler;
// Deno expects to be passed the fetch function
Deno.serve({ port: 9080 }, handler);
```

By default, a fetch handler will not advertise itself as working
bidirectionally; the SDK will end the HTTP request at each suspension point,
and the Restate runtime will re-invoke the service when there is more work to
do.

However, you can use the option `bidirectional: true` to change this on supported platforms,
which will improve latencies once the service is re-registered with the runtime.

* Deno (including Deno Deploy) supports HTTP2 and therefore bidirectional mode can be enabled.
* Cloudflare Workers do not support end-to-end HTTP2 or bidirectional HTTP1.1,
  and enabling bidirectional mode will cause invocations to stall and time out.
  Services running on Workers must be discovered with the `--use-http1.1`
  CLI flag.

<Accordion title="Cloudflare Workers and minification">
  Cloudflare Workers minification is not working correctly with the Restate SDK. If you see an issue similar to:

  ```
  ✘ [ERROR] restate-cloudflare-worker: Uncaught TypeError: Cannot read properties of undefined (reading '__wbindgen_malloc')
  ```

  Then most likely you have enabled minification when deploying. Disable it with `minify = false` in your `workers.toml` file.
</Accordion>

## Validating request identity

SDKs can validate that incoming requests come from a particular Restate
instance. You can find out more about request identity in the [Security docs](/services/security#locking-down-service-access)

```typescript {"CODE_LOAD::ts/src/develop/serving.ts#identity"}  theme={null}
restate.serve({
  services: [myService],
  identityKeys: ["publickeyv1_w7YHemBctH5Ck2nQRQ47iBBqhNHy4FV7t2Usbye2A6f"],
});
```


# State
Source: https://docs.restate.dev/develop/ts/state

Store key-value state in Restate.

Restate lets you persist key-value (K/V) state using its embedded K/V store.

## Key characteristics

<Info>
  [Learn about Restate's embedded K/V store](/foundations/key-concepts#consistent-state).
</Info>

**State is only available for Virtual Objects and Workflows.**

Scope & retention:

* For Virtual Objects: State is scoped per object key and retained indefinitely. It is persisted and shared across all invocations for that object until explicitly cleared.
* For Workflows: State is scoped per workflow execution (workflow ID) and retained only for the duration of the workflow’s configured retention time.

Access Rules:

* [Exclusive handlers](/foundations/handlers#handler-behavior) (e.g., `run()` in workflows) can read and write state.
* [Shared handlers](/foundations/handlers#handler-behavior) can only read state and cannot mutate it.

You can inspect and edit the K/V state via the UI and the [CLI](/services/introspection#inspecting-application-state).

## List all state keys

To retrieve all keys for which the current Virtual Object has stored state:

```typescript {"CODE_LOAD::ts/src/develop/state.ts#statekeys"}  theme={null}
const stateKeys = ctx.stateKeys();
```

## Get state value

To read a value by key:

```typescript {"CODE_LOAD::ts/src/develop/state.ts#get"}  theme={null}
const myString = (await ctx.get<string>("my-string-key")) ?? "my-default";
const myNumber = (await ctx.get<number>("my-number-key")) ?? 0;
```

Returns null if the key doesn't exist.

See the [serialization docs](/develop/ts/serialization) to customize the state serializer.

## Set state value

To write or update a value:

```typescript {"CODE_LOAD::ts/src/develop/state.ts#set"}  theme={null}
ctx.set("my-key", "my-new-value");
```

## Clear state key

To delete a specific key:

```typescript {"CODE_LOAD::ts/src/develop/state.ts#clear"}  theme={null}
ctx.clear("my-key");
```

## Clear all state keys

To remove all stored state for the current Virtual Object:

```typescript {"CODE_LOAD::ts/src/develop/state.ts#clear_all"}  theme={null}
ctx.clearAll();
```

## Advanced: Eager vs. lazy state loading

Restate supports two modes for loading state in handlers:

### Eager state (default)

* **How it works**: State is automatically sent with the request when invoking a handler
* **Benefits**: State is available immediately when the handler starts executing
* **Behavior**: All reads and writes to state are local to the handler execution
* **Best for**: Small to medium state objects that are frequently accessed

### Lazy state

* **How it works**: State is fetched on-demand on `get` calls from the Restate Server
* **Benefits**: Reduces initial request size and memory usage
* **Setup**: Enable lazy state in the [service or handler configuration](/services/configuration)
* **Best for**: Large state objects that aren't needed in every handler execution

## Advanced: Typed State

You can specify the types of your Object's K/V state, to ensure type safety and better developer experience:

```typescript {"CODE_LOAD::ts/src/develop/state.ts#typed_state"}  theme={null}
type GreeterContext = restate.ObjectContext<{
  name: string;
  count: number;
}>;

export const greeter = restate.object({
  name: "Greeter",
  handlers: {
    greet: async (ctx: GreeterContext, name: string) => {
      const count: number = (await ctx.get("count")) ?? 0;
      ctx.set("name", name);
    },
  },
});
```


# Testing
Source: https://docs.restate.dev/develop/ts/testing

Utilities to test your handler logic.

The Typescript SDK has a companion library which makes it easy to test against a Restate container:
[`@restatedev/restate-sdk-testcontainers`](https://www.npmjs.com/package/@restatedev/restate-sdk-testcontainers).

This uses [Testcontainers](https://testcontainers.com/) to run a Restate Server in a Docker container and let you test your Restate handlers.

## Setup

`RestateTestEnvironment.start` creates a Restate container and executes a user-provided closure to register services.
An optional second argument allows you to specify a custom [Testcontainer](https://node.testcontainers.org/) for Restate.

```typescript {"CODE_LOAD::ts/src/develop/testing.test.ts#setup"}  theme={null}
describe("ExampleObject", () => {
  // import { RestateTestEnvironment } from "@restatedev/restate-sdk-testcontainers";
  let restateTestEnvironment: RestateTestEnvironment;
  // import * as clients from "@restatedev/restate-sdk-clients";
  let restateIngress: clients.Ingress;

  beforeAll(async () => {
    restateTestEnvironment = await RestateTestEnvironment.start({
      services: [router],
    });
    restateIngress = clients.connect({ url: restateTestEnvironment.baseUrl() });
  }, 20_000);

  afterAll(async () => {
    if (restateTestEnvironment !== undefined) {
      await restateTestEnvironment.stop();
    }
  });
```

## Calling services

The Restate ingress client can be used as usual (see the [clients documentation](/services/invocation/clients/typescript-sdk))

```typescript {"CODE_LOAD::ts/src/develop/testing.test.ts#methods"}  theme={null}
it("Can call methods", async () => {
  const client = restateIngress.objectClient(router, "myKey");

  await client.greet("Test!");
});
```

## Checking and mutating state

The `stateOf` method on the `RestateTestEnvironment` class can be used to obtain a handle on the Virtual Object / Workflow state
for a particular key.

```typescript {"CODE_LOAD::ts/src/develop/testing.test.ts#state"}  theme={null}
it("Can read state", async () => {
  const state = restateTestEnvironment.stateOf(router, "myKey");

  expect(await state.getAll()).toStrictEqual({});
  expect(await state.get("count")).toBeNull();
});

it("Can write state", async () => {
  const state = restateTestEnvironment.stateOf(router, "myKey");

  await state.setAll({
    count: 123,
  });
  await state.set("count", 321);
});
```

## Typed state

`stateOf` can be provided with a type for the services state, to allow for type-safe state operations.

```typescript {"CODE_LOAD::ts/src/develop/testing.test.ts#typedstate"}  theme={null}
type ServiceState = { count: number };

it("Can operate on typed state", async () => {
  const state = restateTestEnvironment.stateOf<ServiceState>(router, "myKey");

  await state.setAll({ count: 1 });
  await state.set("count", 2);
});
```


# Documentation
Source: https://docs.restate.dev/docs

Build, deploy, and operate resilient applications with Restate

Complete reference for building, deploying, and operating resilient applications with Restate.

## Build your services

Choose your SDK and start building:

<Columns>
  <Card title="TypeScript" href="/develop/ts/services" icon="https://mintcdn.com/restate-6d46e1dc/poNk335qY7Qd1eFK/img/languages/typescript.svg?fit=max&auto=format&n=poNk335qY7Qd1eFK&q=85&s=598b5b5f71cbe5919c3bf52082a5098f" />

  <Card title="Java" href="/develop/java/services" icon="https://mintcdn.com/restate-6d46e1dc/poNk335qY7Qd1eFK/img/languages/java.svg?fit=max&auto=format&n=poNk335qY7Qd1eFK&q=85&s=506480b935094cfeabb2fa13f2cb855b" />

  <Card title="Kotlin" href="/develop/java/services" icon="https://mintcdn.com/restate-6d46e1dc/poNk335qY7Qd1eFK/img/languages/kotlin.svg?fit=max&auto=format&n=poNk335qY7Qd1eFK&q=85&s=c03087d21df2ee0fdb1e063a98f47aea" />

  <Card title="Python" href="/develop/python/services" icon="https://mintcdn.com/restate-6d46e1dc/poNk335qY7Qd1eFK/img/languages/python.svg?fit=max&auto=format&n=poNk335qY7Qd1eFK&q=85&s=006b00cef491afa790377c07c3abdcfe" />

  <Card title="Go" href="/develop/go/services" icon="https://mintcdn.com/restate-6d46e1dc/poNk335qY7Qd1eFK/img/languages/go.svg?fit=max&auto=format&n=poNk335qY7Qd1eFK&q=85&s=b2be5e89ba4c743e5b62192286852450" />

  <Card title="Rust" href="https://docs.rs/restate-sdk/latest/restate_sdk/" icon="https://mintcdn.com/restate-6d46e1dc/poNk335qY7Qd1eFK/img/languages/rust.svg?fit=max&auto=format&n=poNk335qY7Qd1eFK&q=85&s=03f463d359c3c3b6a20842640ce013a0" />
</Columns>

## Deploy and operate your services

<CardGroup>
  <Card title="Deploy" icon="box" href="/services/deploy/kubernetes">
    Deploy to Kubernetes, AWS Lambda, Vercel, Cloudflare Workers, or Deno Deploy
  </Card>

  <Card title="Invoke" icon="phone" href="/services/invocation/http">
    Call services via HTTP, SDK clients, or Kafka events
  </Card>

  <Card title="Versioning" icon="code-branch" href="/services/versioning">
    Manage service versions and compatibility
  </Card>

  <Card title="Monitor & Inspect" icon="magnifying-glass" href="/services/introspection">
    Query system state and inspect running services
  </Card>
</CardGroup>

## Hosting Restate

Choose between managed cloud or self-hosted deployment:

<Columns>
  <Card title="Restate Cloud" href="/cloud/getting-started" icon="cloud">
    **Managed platform** with instant setup, automatic scaling, and built-in monitoring.

    Perfect for getting started quickly without infrastructure management.
  </Card>

  <Card title="Self-Hosted" href="/server/overview" icon="server">
    **Full control** over your infrastructure with flexible deployment options.

    Single node, cluster, and Kubernetes deployment available.
  </Card>
</Columns>

## References

<CardGroup>
  <Card title="Architecture & Config" href="/references/architecture" icon="gear">
    System design • [Server config](/references/server-config)
  </Card>

  <Card title="API References" icon="code">
    [TypeScript](https://restatedev.github.io/sdk-typescript) • [Java](https://restatedev.github.io/sdk-java/javadocs) • [Kotlin](https://restatedev.github.io/sdk-java/ktdocs) • [Go](https://pkg.go.dev/github.com/restatedev/sdk-go)
  </Card>
</CardGroup>

## New to Restate?

<Columns>
  <Card title="Quickstart" href="/quickstart" icon="rocket">
    Build your first service
  </Card>

  <Card title="Use Cases" icon="lightbulb">
    [AI agents](/use-cases/ai-agents) • [Microservices](/use-cases/microservice-orchestration) • [Workflows](/use-cases/workflows) • [Event Processing](/use-cases/event-processing)
  </Card>

  <Card title="Concepts" href="/foundations/key-concepts" icon="cube">
    Core concepts and building blocks
  </Card>
</Columns>


# Examples
Source: https://docs.restate.dev/examples





# Actions
Source: https://docs.restate.dev/foundations/actions

Essential context actions for building reliable handlers

Context actions are methods available on the Restate Context object (`ctx`) that provide Restate's core capabilities.
These actions enable durable execution, state management, service communication, and timing control.

### Durable steps

Use `run` to safely wrap any non-deterministic operation, like HTTP calls or database responses, and have Restate persist its result.

<CodeGroup>
  ```typescript TypeScript {"CODE_LOAD::ts/src/foundations/actions/actions.ts#durable_steps"}  theme={null}
  // External API call
  const apiResult = await ctx.run("fetch-data", async () => {
    const response = await fetch("https://api.example.com/data");
    return response.json();
  });

  // Database operation
  const dbResult = await ctx.run("update-user", () => {
    return updateUserDatabase(userId, { name: "John" });
  });

  // Idempotency key generation
  const id = ctx.rand.uuidv4();
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/foundations/actions/Actions.java#durable_steps"}  theme={null}
  // External API call
  String apiResult = ctx.run(String.class, () -> fetchData("https://api.example.com/data"));

  // Database operation
  boolean dbResult = ctx.run(Boolean.class, () -> updateUserDatabase(userId, user));

  // Idempotency key generation
  String id = ctx.random().nextUUID().toString();
  ```

  ```python Python {"CODE_LOAD::python/src/foundations/actions/actions.py#durable_steps"}  theme={null}
  # External API call
  api_result = await ctx.run_typed(
      "fetch-data", fetch_url, url="https://api.example.com/data"
  )

  # Database operation
  db_result = await ctx.run_typed(
      "update-user", update_user_database, id=user_id, data={"name": "John"}
  )

  # Idempotency key generation
  id = ctx.uuid()
  ```

  ```go Go {"CODE_LOAD::go/foundations/actions/actions.go#durable_steps"}  theme={null}
  // External API call
  apiResult, err := restate.Run(ctx, func(ctx restate.RunContext) (string, error) {
    return fetchData("https://api.example.com/data")
  })
  if err != nil {
    return err
  }

  // Database operation
  success, err := restate.Run(ctx, func(ctx restate.RunContext) (bool, error) {
    return updateUserDatabase(userId, user)
  })
  if err != nil {
    return err
  }

  // Idempotency key generation
  id := restate.UUID(ctx).String()
  ```
</CodeGroup>

Without `run()`, these operations would produce different results during replay, breaking deterministic recovery.

## State management

Available in Virtual Object and Workflow handlers for persistent key-value storage.

### Get

Retrieve stored state by key.

<CodeGroup>
  ```typescript TypeScript {"CODE_LOAD::ts/src/foundations/actions/actions.ts#state_get"}  theme={null}
  // Get with type and default value
  const profile = await ctx.get<UserProfile>("profile");
  const count = (await ctx.get<number>("count")) ?? 0;
  const cart = (await ctx.get<ShoppingCart>("cart")) ?? [];
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/foundations/actions/StateExample.java#state_get"}  theme={null}
  // Get with type and default value
  UserProfile profile = ctx.get(PROFILE).orElse(null);
  int count = ctx.get(COUNT).orElse(0);
  ShoppingCart cart = ctx.get(CART).orElse(new ShoppingCart());
  ```

  ```python Python {"CODE_LOAD::python/src/foundations/actions/actions.py#state_get"}  theme={null}
  # Get with type and default value
  profile = await ctx.get("profile", type_hint=UserProfile)
  count = await ctx.get("count", type_hint=int) or 0
  cart = await ctx.get("cart", type_hint=ShoppingCart) or ShoppingCart()
  ```

  ```go Go {"CODE_LOAD::go/foundations/actions/actions.go#state_get"}  theme={null}
  // Get with type and default value
  profile, err := restate.Get[UserProfile](ctx, "profile")
  if err != nil {
    return err
  }

  count, err := restate.Get[int](ctx, "count")
  if err != nil {
    return err
  }

  cart, err := restate.Get[ShoppingCart](ctx, "cart")
  if err != nil {
    return err
  }
  ```
</CodeGroup>

### Set

Store state that persists across function invocations.

<CodeGroup>
  ```typescript TypeScript {"CODE_LOAD::ts/src/foundations/actions/actions.ts#state_set"}  theme={null}
  // Store simple values
  ctx.set("lastLogin", request.date);
  ctx.set("count", count + 1);

  // Store complex objects
  ctx.set("profile", {
    name: "John Doe",
    email: "john@example.com",
  });
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/foundations/actions/StateExample.java#state_set"}  theme={null}
  // Store simple values
  ctx.set(COUNT, count + 1);
  ctx.set(LAST_LOGIN, request.date());

  // Store complex objects
  ctx.set(PROFILE, new UserProfile("John Doe", "john@example.com"));
  ```

  ```python Python {"CODE_LOAD::python/src/foundations/actions/actions.py#state_set"}  theme={null}
  # Store simple values
  ctx.set("lastLogin", request["date"])
  ctx.set("count", count + 1)

  # Store complex objects
  ctx.set("profile", UserProfile(name="John Doe", email="john@example.com"))
  ```

  ```go Go {"CODE_LOAD::go/foundations/actions/actions.go#state_set"}  theme={null}
  // Store simple values
  restate.Set(ctx, "lastLogin", request.Date)
  restate.Set(ctx, "count", count+1)

  // Store complex objects
  restate.Set(ctx, "profile", UserProfile{
    Name:  "John Doe",
    Email: "john@example.com",
  })
  ```
</CodeGroup>

### Clear

State is retained indefinitely for Virtual Objects, or for the configured retention period for Workflows.

To clear state:

<CodeGroup>
  ```typescript TypeScript {"CODE_LOAD::ts/src/foundations/actions/actions.ts#state_clear"}  theme={null}
  // Clear specific keys
  ctx.clear("shoppingCart");
  ctx.clear("sessionToken");

  // Clear all user data
  ctx.clearAll();
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/foundations/actions/StateExample.java#state_clear"}  theme={null}
  // Clear specific keys
  ctx.clear(StateKey.of("shoppingCart", ShoppingCart.class));
  ctx.clear(StateKey.of("sessionToken", String.class));

  // Clear all user data
  ctx.clearAll();
  ```

  ```python Python {"CODE_LOAD::python/src/foundations/actions/actions.py#state_clear"}  theme={null}
  # Clear specific keys
  ctx.clear("shoppingCart")
  ctx.clear("sessionToken")

  # Clear all user data
  ctx.clear_all()
  ```
</CodeGroup>

## Service communication

### Request-response calls

Make request-response calls to other services. Your function waits for the result.

<CodeGroup>
  ```typescript TypeScript {"CODE_LOAD::ts/src/foundations/actions/actions.ts#service_calls"}  theme={null}
  // Call another service
  const validation = await ctx
    .serviceClient(ValidationService)
    .validateOrder(order);

  // Call Virtual Object function
  const profile = await ctx.objectClient(UserAccount, userId).getProfile();

  // Submit Workflow
  const result = await ctx
    .workflowClient(OrderWorkflow, orderId)
    .run(order);
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/foundations/actions/Actions.java#service_calls"}  theme={null}
  // Call another service
  var validation = ValidationServiceClient.fromContext(ctx).validateOrder(req.order()).await();

  // Call Virtual Object function
  var profile = UserAccountClient.fromContext(ctx, req.userId()).getProfile().await();

  // Submit Workflow
  var result = OrderWorkflowClient.fromContext(ctx, req.orderId()).run(req.order()).await();
  ```

  ```python Python {"CODE_LOAD::python/src/foundations/actions/actions.py#service_calls"}  theme={null}
  # Call another service
  from .validation_service import validate_order

  validation = await ctx.service_call(validate_order, order)

  # Call Virtual Object function
  from .user_account import get_profile

  profile: UserProfile = await ctx.object_call(get_profile, key=user_id, arg=None)

  # Submit Workflow
  from .order_workflow import run

  result = await ctx.workflow_call(run, key=order_id, arg=order)
  ```

  ```go Go {"CODE_LOAD::go/foundations/actions/actions.go#service_calls"}  theme={null}
  // Call another service
  validation, err := restate.Service[bool](ctx, "ValidationService", "ValidateOrder").Request(order)
  if err != nil {
    return err
  }

  // Call Virtual Object function
  profile, err := restate.Object[UserProfile](ctx, "UserAccount", userId, "GetProfile").Request(restate.Void{})
  if err != nil {
    return err
  }

  // Submit Workflow
  _, err = restate.Workflow[restate.Void](ctx, "OrderWorkflow", orderId, "Run").Request(order)
  if err != nil {
    return err
  }
  ```
</CodeGroup>

### Sending messages

Make one-way calls that don't return results. Your function continues immediately.

<CodeGroup>
  ```typescript TypeScript {"CODE_LOAD::ts/src/foundations/actions/actions.ts#sending_messages"}  theme={null}
  // Fire-and-forget notification
  ctx
    .serviceSendClient(NotificationService)
    .sendEmail({ userId, message: "Welcome!" });

  // Background analytics
  ctx
    .serviceSendClient(AnalyticsService)
    .recordEvent({ kind: "user_signup", userId });

  // Cleanup task
  ctx.objectSendClient(ShoppingCartObject, userId).emtpyExpiredCart();
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/foundations/actions/Actions.java#sending_messages"}  theme={null}
  // Fire-and-forget notification
  NotificationServiceClient.fromContext(ctx)
      .send()
      .sendEmail(new EmailRequest(userId, "Welcome!"));

  // Background analytics
  AnalyticsServiceClient.fromContext(ctx).send().recordEvent(event);

  // Cleanup task
  ShoppingCartObjectClient.fromContext(ctx, userId).send().emptyExpiredCart();
  ```

  ```python Python {"CODE_LOAD::python/src/foundations/actions/actions.py#sending_messages"}  theme={null}
  # Fire-and-forget notification
  from .notification_service import send_email

  ctx.service_send(send_email, {"userId": user_id, "message": "Welcome!"})

  # Background analytics
  from .analytics_service import record_event

  ctx.service_send(record_event, {"kind": "user_signup", "userId": user_id})

  # Cleanup task
  from .shopping_cart_object import empty_expired_cart

  ctx.object_send(empty_expired_cart, key=user_id, arg=None)
  ```

  ```go Go {"CODE_LOAD::go/foundations/actions/actions.go#sending_messages"}  theme={null}
  // Fire-and-forget notification
  restate.ServiceSend(ctx, "NotificationService", "SendEmail").Send(message)

  // Background analytics
  restate.ServiceSend(ctx, "AnalyticsService", "RecordEvent").Send(event)

  // Cleanup task
  restate.ObjectSend(ctx, "ShoppingCartObject", userId, "EmptyExpiredCart").Send(restate.Void{})
  ```
</CodeGroup>

### Delayed messages

Schedule handlers to run in the future.

<CodeGroup>
  ```typescript TypeScript {"CODE_LOAD::ts/src/foundations/actions/actions.ts#delayed_messages"}  theme={null}
  // Schedule reminder for tomorrow
  ctx.serviceSendClient(ReminderService).sendReminder(
    { userId, message },
    sendOpts({
      delay: { days: 1 },
    })
  );
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/foundations/actions/Actions.java#delayed_messages"}  theme={null}
  // Schedule reminder for tomorrow
  ReminderServiceClient.fromContext(ctx).send().sendReminder(reminderRequest, Duration.ofDays(1));
  ```

  ```python Python {"CODE_LOAD::python/src/foundations/actions/actions.py#delayed_messages"}  theme={null}
  # Schedule reminder for tomorrow
  from .notification_service import send_reminder

  ctx.service_send(
      send_reminder,
      {"userId": user_id, "message": message},
      send_delay=timedelta(days=1),
  )
  ```

  ```go Go {"CODE_LOAD::go/foundations/actions/actions.go#delayed_messages"}  theme={null}
  // Schedule reminder for tomorrow
  restate.ServiceSend(ctx, "ReminderService", "SendReminder").Send(
    message,
    restate.WithDelay(24*time.Hour),
  )
  ```
</CodeGroup>

## Durable timers and timeouts

Pause function execution for a specific duration.

<CodeGroup>
  ```typescript TypeScript {"CODE_LOAD::ts/src/foundations/actions/actions.ts#durable_timers"}  theme={null}
  // Sleep for specific duration
  await ctx.sleep({ minutes: 5 }); // 5 minutes

  // Wait for action or timeout
  const result = await ctx
    .workflowClient(OrderWorkflow, orderId)
    .run(order)
    .orTimeout({ minutes: 5 });
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/foundations/actions/Actions.java#durable_timers"}  theme={null}
  // Sleep for specific duration
  ctx.sleep(Duration.ofMinutes(5)); // 5 minutes

  // Wait for action or timeout
  try {
    OrderWorkflowClient.fromContext(ctx, req.orderId())
        .run(req.order())
        .await(Duration.ofMinutes(5));
  } catch (TimeoutException e) {
    // Handle timeout
  }
  ```

  ```python Python {"CODE_LOAD::python/src/foundations/actions/actions.py#durable_timers"}  theme={null}
  # Sleep for specific duration
  await ctx.sleep(timedelta(minutes=5))  # 5 minutes

  # Wait for action or timeout
  from .order_workflow import run

  match await restate.select(
      result=ctx.workflow_call(run, key=order_id, arg=order),
      timeout=ctx.sleep(timedelta(minutes=5)),
  ):
      case ["result", result]:
          return result
      case _:
          print("Order processing timed out")
  ```

  ```go Go {"CODE_LOAD::go/foundations/actions/actions.go#durable_timers"}  theme={null}
  // Sleep for specific duration
  if err := restate.Sleep(ctx, 5*time.Minute); err != nil {
    return err
  }

  // Wait for action or timeout
  sleepFuture := restate.After(ctx, 5*time.Minute)
  callFuture := restate.Workflow[restate.Void](ctx, "OrderWorkflow", orderId, "Run").RequestFuture(order)

  fut, err := restate.WaitFirst(ctx, sleepFuture, callFuture)
  if err != nil {
    return err
  }
  switch fut {
  case sleepFuture:
    if err := sleepFuture.Done(); err != nil {
      return err
    }
    // Timeout occurred
  case callFuture:
    if _, err := callFuture.Response(); err != nil {
      return err
    }
    // Call completed
  }
  ```
</CodeGroup>

Handlers consume no resources while sleeping and resume at exactly the right time, even across restarts ([see suspensions](/foundations/key-concepts#suspensions-on-faas)).

## Workflow events

Use durable promises to wait for external events or human input in your workflows.

Create promises that external systems can resolve to send data to your workflow.

<CodeGroup>
  ```typescript TypeScript {"CODE_LOAD::ts/src/foundations/actions/actions.ts#workflow_promises"}  theme={null}
  // Wait for external event
  const paymentResult = await ctx.promise<PaymentResult>(
    "payment-completed"
  );

  // Wait for human approval
  const approved = await ctx.promise<boolean>("manager-approval");

  // Wait for multiple events
  const [payment, inventory] = await RestatePromise.all([
    ctx.promise<PaymentResult>("payment").get(),
    ctx.promise<InventoryResult>("inventory").get(),
  ]);
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/foundations/actions/WorkflowExample.java#workflow_promises"}  theme={null}
  // Wait for external event
  PaymentResult paymentResult = ctx.promise(PAYMENT_COMPLETED).future().await();

  // Wait for human approval
  Boolean approved = ctx.promise(MANAGER_APPROVAL).future().await();

  // Wait for multiple events
  var paymentFuture = ctx.promise(PAYMENT).future();
  var inventoryFuture = ctx.promise(INVENTORY).future();
  PaymentResult payment = paymentFuture.await();
  InventoryResult inventory = inventoryFuture.await();
  ```

  ```python Python {"CODE_LOAD::python/src/foundations/actions/actions.py#workflow_promises"}  theme={null}
  # Wait for external event
  payment_result = await ctx.promise(
      "payment-completed", type_hint=PaymentResult
  ).value()

  # Wait for human approval
  approved = await ctx.promise("manager-approval", type_hint=bool).value()

  # Wait for multiple events using gather
  payment_promise = ctx.promise("payment", type_hint=PaymentResult)
  inventory_promise = ctx.promise("inventory", type_hint=InventoryConfirmation)
  payment, inventory = await restate.gather(
      payment_promise.value(), inventory_promise.value()
  )
  ```

  ```go Go {"CODE_LOAD::go/foundations/actions/actions.go#workflow_promises"}  theme={null}
  // Wait for external event
  paymentResult, err := restate.Promise[PaymentResult](ctx, "payment-completed").Result()
  if err != nil {
    return err
  }

  // Wait for human approval
  approved, err := restate.Promise[bool](ctx, "manager-approval").Result()
  if err != nil {
    return err
  }

  // Wait for multiple events
  paymentFuture := restate.Promise[PaymentResult](ctx, "payment")
  inventoryFuture := restate.Promise[InventoryResult](ctx, "inventory")

  payment, err := paymentFuture.Result()
  if err != nil {
    return err
  }

  inventory, err := inventoryFuture.Result()
  if err != nil {
    return err
  }
  ```
</CodeGroup>

Resolve promises from signal handlers.

<CodeGroup>
  ```typescript TypeScript {"CODE_LOAD::ts/src/foundations/actions/actions.ts#signal_functions"}  theme={null}
  // In a signal function
  confirmPayment: async (
    ctx: WorkflowSharedContext,
    result: PaymentResult
  ) => {
    await ctx.promise("payment-completed").resolve(result);
  },

  // In a signal function
  approveRequest: async (ctx: WorkflowSharedContext, approved: boolean) => {
    await ctx.promise("manager-approval").resolve(approved);
  },
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/foundations/actions/WorkflowExample.java#signal_functions"}  theme={null}
  // In a signal function
  @Shared
  public void confirmPayment(SharedWorkflowContext ctx, PaymentResult result) {
    ctx.promiseHandle(PAYMENT_COMPLETED).resolve(result);
  }

  // In a signal function
  @Shared
  public void approveRequest(SharedWorkflowContext ctx, Boolean approved) {
    ctx.promiseHandle(MANAGER_APPROVAL).resolve(approved);
  }
  ```

  ```python Python {"CODE_LOAD::python/src/foundations/actions/actions.py#signal_functions"}  theme={null}
  # In a signal function
  @workflow_example_workflow.handler()
  async def confirm_payment(ctx: WorkflowSharedContext, result: PaymentResult) -> None:
      await ctx.promise("payment-completed", type_hint=PaymentResult).resolve(result)


  # In a signal function
  @workflow_example_workflow.handler()
  async def approve_request(ctx: WorkflowSharedContext, approved: bool) -> None:
      await ctx.promise("manager-approval", type_hint=bool).resolve(approved)
  ```

  ```go Go {"CODE_LOAD::go/foundations/actions/actions.go#signal_functions"}  theme={null}
  // In a signal function
  func (WorkflowExample) ConfirmPayment(ctx restate.WorkflowSharedContext, result PaymentResult) error {
    if err := restate.Promise[PaymentResult](ctx, "payment-completed").Resolve(result); err != nil {
      return err
    }
    return nil
  }

  // In a signal function
  func (WorkflowExample) ApproveRequest(ctx restate.WorkflowSharedContext, approved bool) error {
    if err := restate.Promise[bool](ctx, "manager-approval").Resolve(approved); err != nil {
      return err
    }
    return nil
  }
  ```
</CodeGroup>

<Info>To implement a similar pattern in Basic Services or Virtual Objects, have a look at awakeables ([TS](/develop/ts/external-events)/[Java/Kotlin](/develop/java/external-events)/[Go](/develop/ts/external-events)/[Python](/develop/python/external-events)).</Info>


# Handlers
Source: https://docs.restate.dev/foundations/handlers

Learn how to create, invoke and manage handlers in Restate services

Handlers are the core building blocks of Restate services. Each service has a list of handlers: durable functions that can be invoked over HTTP, Kafka, or using typed clients.

## Writing handlers

Handlers receive a Restate context object (`ctx`) and a single, optional input:

<CodeGroup>
  ```typescript TypeScript {"CODE_LOAD::ts/src/foundations/functions/functions.ts#here"}  theme={null}
  async function myHandler(ctx: restate.Context, input: MyInput) {
    const result = await ctx.run("process", () => processData(input));
    return { success: true, result };
  }
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/foundations/functions/Functions.java#here"}  theme={null}
  @Handler
  public ProcessingResult myHandler(Context ctx, MyInput myInput) {
    return ctx.run(ProcessingResult.class, () -> processData(myInput));
  }
  ```

  ```python Python {"CODE_LOAD::python/src/foundations/functions/functions.py#here"}  theme={null}
  async def my_handler(ctx: Context, req: MyInput) -> ProcessingResult:
      return await ctx.run_typed("process", process_data, req=req)
  ```

  ```go Go {"CODE_LOAD::go/foundations/functions/functions.go#here"}  theme={null}
  func MyHandler(ctx restate.Context, input MyInput) (ProcessingResult, error) {
    return restate.Run(ctx, func(ctx restate.RunContext) (ProcessingResult, error) {
      return processData(input)
    })
  }
  ```
</CodeGroup>

<Info>
  The input and output can be any JSON-serializable type. For other types, consult the serialization documentation of your SDK.
</Info>

## Context types

The context type you use depends on your service type:

* **`Context`** - Basic Services (stateless handlers)
* **`ObjectContext`** - Virtual Objects (exclusive handlers with state access)
* **`ObjectSharedContext`** - Virtual Objects (concurrent read-only handlers)
* **`WorkflowContext`** - Workflows (main run handler)
* **`WorkflowSharedContext`** - Workflows (signal/query handlers)

## Handler behavior

Virtual Objects and Workflows support two handler types:

**Exclusive handlers** (`ObjectContext`, `WorkflowContext`) can read and write state but only one runs at a time per key to prevent conflicts.

**Shared handlers** (`ObjectSharedContext`, `WorkflowSharedContext`) can only read state but run concurrently without blocking—useful for querying status during long-running operations.


# Invocations
Source: https://docs.restate.dev/foundations/invocations

Invoke handlers in Restate services over HTTP, Kafka or with typed clients.

An invocation is a request to execute a handler that is part of a Restate service.

There are three ways to invoke a handler: over HTTP, using typed clients, or through Kafka topics.

## HTTP invocations

To call a function over HTTP, send a request to the Restate Server, specifying the service and the function you want to invoke.

For example, to call the `processPayment` function of the `PaymentService`:

```bash theme={null}
curl -X POST localhost:8080/PaymentService/processPayment \
--json '{"amount": 100, "currency": "USD"}'
```

The Restate Server ingress endpoint is here hosted at `localhost:8080`.

To call Virtual Objects or Workflows, you need to specify the object key or workflow ID in the URL.

For example, to call `updateBalance` of the `UserAccount` object with key `user123`:

```bash theme={null}
curl -X POST localhost:8080/UserAccount/user123/updateBalance \
--json '{"amount": 50}'
```

Check out the [HTTP invocation](/services/invocation/http) docs to learn more.

## Typed client invocations

Use typed clients from external applications to invoke Restate handlers:

<CodeGroup>
  ```typescript TypeScript {"CODE_LOAD::ts/src/foundations/invocations/client.ts#here"}  theme={null}
  //import * as clients from "@restatedev/restate-sdk-clients";
  const restateClient = clients.connect({ url: "http://localhost:8080" });

  // To call a service:
  const greet = await restateClient
    .serviceClient(greeterService)
    .greet({ greeting: "Hi" });
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/foundations/invocations/Client.java#here"}  theme={null}
  var restate = Client.connect("http://localhost:8080");

  // To call a service:
  MyServiceClient.fromClient(restate).myHandler(req);
  ```

  ```go Go {"CODE_LOAD::go/foundations/invocations/client.go#here"}  theme={null}
  restateClient := restateingress.NewClient("http://localhost:8080")

  // To call a service
  response, err := restateingress.Service[*MyInput, *MyOutput](
    restateClient, "MyService", "MyHandler").
    Request(context.Background(), &input)
  if err != nil {
    return err
  }
  ```
</CodeGroup>

For service-to-service calls within handlers, see [Service Communication](/foundations/actions#service-communication).

## Kafka invocations

You can connect your Restate handlers to Kafka topics, to let Restate invoke them for each message.

Consult the [Kafka Quickstart](/guides/kafka-quickstart) to get started.

## Idempotency

Add an idempotency key to your request header to let Restate deduplicate retries:

```bash theme={null}
curl -X POST localhost:8080/PaymentService/processPayment \
-H 'idempotency-key: payment-123' \
--json '{"amount": 100, "currency": "USD"}'
```

On retry, Restate returns the first invocation’s result or lets you attach to it if still running.

## Inspecting invocations

Use the Restate UI to monitor and troubleshoot invocations:

<Frame>
  <img alt="Durable Execution" />
</Frame>

## Attaching to invocations

Attach to ongoing invocations to retrieve their results:

<CodeGroup>
  ```typescript TypeScript {"CODE_LOAD::ts/src/foundations/invocations/client.ts#attach"}  theme={null}
  const handle = ctx
    .serviceSendClient(myService)
    .myHandler("Hi", sendOpts({ idempotencyKey: "my-key" }));
  const invocationId = await handle.invocationId;

  // Later...
  const response = ctx.attach(invocationId);
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/foundations/invocations/Client.java#attach"}  theme={null}
  var handle =
      MyServiceClient.fromContext(ctx)
          .send()
          .myHandler("Hi", opt -> opt.idempotencyKey("my-key"));
  var response = handle.attach().await();
  ```

  ```python Python {"CODE_LOAD::python/src/foundations/invocations/client.py#attach"}  theme={null}
  handle = ctx.service_send(my_handler, "Hi", idempotency_key="my-key")
  invocation_id = await handle.invocation_id()

  # Later...
  await ctx.attach_invocation(invocation_id)
  ```

  ```go Go {"CODE_LOAD::go/foundations/invocations/client.go#attach"}  theme={null}
  // Execute the request and retrieve the invocation id
  invocationId := restate.
    ServiceSend(ctx, "MyService", "MyHandler").
    Send("Hi", restate.WithIdempotencyKey("my-idempotency-key")).
    GetInvocationId()

  // Later re-attach to the request
  response, err := restate.AttachInvocation[string](ctx, invocationId).Response()
  ```
</CodeGroup>

Or over HTTP:

```bash theme={null}
curl localhost:8080/restate/invocation/inv_1234567890abcdef/attach
```

This is useful when another process needs the result of an ongoing operation or wants to check whether it has completed.

## Cancelling invocations

Cancel running invocations when they're no longer needed:

<CodeGroup>
  ```bash CLI theme={null}
  restate invocations cancel inv_1234567890abcdef
  ```

  ```bash HTTP theme={null}
  curl -X PATCH http://localhost:9070/invocations/inv_1gdJBtdVEcM942bjcDmb1c1khoaJe11Hbz/cancel
  ```
</CodeGroup>

Or programmatically:

<CodeGroup>
  ```typescript TypeScript {"CODE_LOAD::ts/src/foundations/invocations/client.ts#cancel"}  theme={null}
  const handle = ctx.serviceSendClient(myService).myHandler("Hi");
  const invocationId = await handle.invocationId;

  // Cancel the invocation
  ctx.cancel(invocationId);
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/foundations/invocations/Client.java#cancel"}  theme={null}
  var handle = MyServiceClient.fromContext(ctx).send().myHandler("Hi");

  // Cancel the invocation
  handle.cancel();
  ```

  ```python Python {"CODE_LOAD::python/src/foundations/invocations/client.py#cancel"}  theme={null}
  handle = ctx.service_send(my_handler, "Hi", idempotency_key="my-key")
  invocation_id = await handle.invocation_id()

  # Cancel the invocation
  ctx.cancel_invocation(invocation_id)
  ```

  ```go Go {"CODE_LOAD::go/foundations/invocations/client.go#cancel"}  theme={null}
  // Execute the request and retrieve the invocation id
  invocationId := restate.
    ServiceSend(ctx, "MyService", "MyHandler").
    Send("Hi").
    GetInvocationId()

  // I don't need this invocation anymore, let me just cancel it
  restate.CancelInvocation(ctx, invocationId)
  ```
</CodeGroup>

To roll back the actions the handler already completed before cancellation, check out at the [sagas guide](/guides/sagas).


# Key Concepts
Source: https://docs.restate.dev/foundations/key-concepts

Core concepts of Restate applications

This page explains the key concepts and architecture that make Restate applications reliable and resilient.

## Application Structure

A Restate application consists of the following components:

<img alt="Application Structure" />

### Restate Services

Your business logic lives in **services**: regular applications that embed the Restate SDK. Services contain **handlers** (durable functions) that process requests and execute business logic.

* **Deploy anywhere**: Services run in your infrastructure as containers, serverless functions, VMs, or Kubernetes pods
* **Familiar programming model**: Services look and feel like normal applications with added reliability by using the Restate SDK. Restate has SDKs for TypeScript, Java, Kotlin, Python, Go, and Rust.

### Restate Server

The **Restate Server** sits in front of your services, similar to a reverse proxy or message broker. It handles incoming requests, manages service discovery, and provides durable execution capabilities.

The server is a single binary written in Rust with a stream-processing architecture for **low latency and high throughput**.

It can be deployed in a high-availability cluster or as a single-node instance.

<Card title="Restate Cloud" href="https://restate.dev/cloud/" icon="cloud">
  Look at the options for Restate Cloud for a fully managed Restate deployment.
</Card>

### Invocations

An **invocation** represents a request to execute a handler. Restate tracks each invocation through completion, ensuring it runs exactly once regardless of failures.

You can invoke Restate handlers over HTTP, Kafka, or programmatically.

## Durable Execution

Restate's core feature is **durable execution**: the ability to make any code execution reliable and fault-tolerant without changing how you write business logic.

### How it works

Restate tracks every step of your code execution in a **journal**. When you call other services, update databases, set timers, or perform any side-effecting operation, Restate records both the operation and its result. If your function crashes or fails, Restate replays the journal, skipping completed steps and resuming from exactly where it left off.

<img alt="Durable Execution" />

In this example, if sending the receipt fails, Restate automatically retries the function but skips the payment processing since it already completed successfully. Each step is recorded in the journal and won't repeat on retry.

<Info>
  For a detailed technical walkthrough of the lifecycle of a request, read this **[guide](/guides/request-lifecycle)**.
</Info>

### What it covers

Restate's implementation of Durable Execution protects your applications from common failure scenarios:

<AccordionGroup>
  <Accordion title="Infrastructure failures (e.g. server crashes, VM restarts)">
    * **Service crashes or restarts**: Any ongoing executions are retried on new instances.
    * **Restate Server crashes or restarts**: The Restate Server can be deployed as a high-availability cluster, so if a node fails, others can take over without losing any execution state.
      If Restate is deployed as a single node, it can still recover from crashes by recovering from persistent disk.
  </Accordion>

  <Accordion title="API timeouts and downtime">
    If a handler calls an API that times out, Restate will retry the run action with exponential backoff. You can tune the retry policy through the [service configuration](/services/configuration#retries).
  </Accordion>

  <Accordion title="Duplicate requests">
    If you add an idempotency key to your request headers, Restate will automatically ensure that requests are deduplicated.
    Duplicate requests will return the same result as the original request.
  </Accordion>

  <Accordion title="Network outages, partitions and zombie processes">
    * **Between Server and service**: If the service cannot send journal entries to the Restate Server, it will stop the handler execution. The Restate Server will retry the execution on another service instance.
    * **Between nodes in a Restate cluster**: Restate clusters rely on a consensus algorithm to accept new entries in the log. The consensus algorithm requires a majority of nodes to be available, so if a network partition occurs, the cluster will still be able to process requests as long as a majority can still communicate.
  </Accordion>
</AccordionGroup>

## Resilient Communication

Beyond executing individual handlers reliably, Restate also ensures that communication between services is resilient.

<Frame>
  <img alt="communication" />
</Frame>

Restate proxies all communication towards and between services.
It provides a **resilient RPC framework**:

* Automatic retries and recovery: Failed calls are retried until they succeed
* No duplicates: Same call won’t execute twice
* Full observability: See all calls and call chains in the UI
* No message queues needed: Restate handles message delivery

## Consistent State

The Restate Server includes an embedded **key-value store** for persisting application state in Virtual Objects and Workflows.

<img alt="Virtual Objects" />

Key characteristics:

* **Usage**: Implement consistent state machines, session state, or agent context and memory.
* **Scoped per entity**: Each Virtual Object and Workflow execution has its own isolated state.
* **Automatically journaled**: State updates are recorded alongside execution steps for consistency. State is never out of sync with the execution.
* **Single-writer guarantee**: Only one handler can modify state at a time, preventing race conditions.
* **Stateless services**: The Restate Server stores all state and execution history, delivering them with each request. Your services remain stateless, can scale horizontally, and run stateful logic even on serverless platforms.

<Info>
  Learn more about when to store state in Restate vs. in a database with [this guide](/guides/databases).
</Info>

## Suspensions on FaaS

When running handlers on function-as-a-service platforms (FaaS), like AWS Lambda, Restate can suspend functions while they wait for external events (like user input or long-running tasks).

For example, if a handler needs to wait for a user to approve an order, Restate can suspend the function until the user responds.

This lets you run long-running workflows on FaaS platforms **without paying for wait time**.

## Next Steps

Now that you understand the core concepts, dive deeper into how to build with Restate:

* **[Services](/foundations/services)**: Understand the three service types and when to use each
* **[Handlers](/foundations/handlers)**: Learn how to write handlers that take advantage of durable execution
* **[Actions](/foundations/actions)**: Master the context helpers for state management, service calls, and timing control


# Services
Source: https://docs.restate.dev/foundations/services

Understanding Restate's three service types and when to use each

Restate provides three service types optimized for different use cases.

## Service types comparison

|                  | **Basic Service**                            | **Virtual Object**                                                               | **Workflow**                                               |
| ---------------- | -------------------------------------------- | -------------------------------------------------------------------------------- | ---------------------------------------------------------- |
| **What**         | Independent stateless handlers               | Stateful entity with a unique key                                                | Multi-step processes that execute exactly-once per ID      |
| **State**        | None                                         | Isolated per object key                                                          | Isolated per workflow instance                             |
| **Concurrency**  | Unlimited parallel execution                 | Single writer per key (+ concurrent readers)                                     | Single `run` handler per ID (+ concurrent signals/queries) |
| **Key Features** | Durable execution, service calls             | Built-in K/V state, single-writer consistency                                    | Durable promises, signals, lifecycle management            |
| **Best For**     | ETL, sagas, parallelization, background jobs | User accounts, shopping carts, agents, state machines, stateful event processing | Approvals, onboarding workflows, multi-step flows          |

## Basic Service

Basic Services group related handlers as callable endpoints.

<CodeGroup>
  ```typescript TypeScript {"CODE_LOAD::ts/src/foundations/services/basic_service.ts#here"}  theme={null}
  const subscriptionService = restate.service({
    name: "SubscriptionService",
    handlers: {
      add: async (ctx: restate.Context, req: SubscriptionRequest) => {
        const paymentId = ctx.rand.uuidv4();

        const payRef = await ctx.run(() =>
          createRecurringPayment(req.creditCard, paymentId)
        );

        for (const subscription of req.subscriptions) {
          await ctx.run(() =>
            createSubscription(req.userId, subscription, payRef)
          );
        }
      },
    },
  });
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/foundations/services/SubscriptionService.java#here"}  theme={null}
  @Service
  public class SubscriptionService {

    @Handler
    public void add(Context ctx, SubscriptionRequest req) {
      var paymentId = ctx.random().nextUUID().toString();

      String payRef =
          ctx.run("pay", String.class, () -> createRecurringPayment(req.creditCard(), paymentId));

      for (String subscription : req.subscriptions()) {
        ctx.run("add-" + subscription, () -> createSubscription(req.userId(), subscription, payRef));
      }
    }
  }
  ```

  ```python Python {"CODE_LOAD::python/src/foundations/services/basic_service.py#here"}  theme={null}
  subscription_service = restate.Service("SubscriptionService")


  @subscription_service.handler()
  async def add(ctx: Context, req: SubscriptionRequest) -> None:
      payment_id = str(uuid.uuid4())

      pay_ref = await ctx.run_typed(
          "pay",
          create_recurring_payment,
          credit_card=req.credit_card,
          payment_id=payment_id,
      )

      for subscription in req.subscriptions:
          await ctx.run_typed(
              "add-" + subscription,
              create_subscription,
              user_id=req.user_id,
              subscription=subscription,
              pay_ref=pay_ref,
          )
  ```

  ```go Go {"CODE_LOAD::go/foundations/services/basic_service.go#here"}  theme={null}
  type SubscriptionService struct{}

  func (SubscriptionService) Add(ctx restate.Context, req SubscriptionRequest) error {
    paymentId := restate.UUID(ctx).String()

    payRef, err := restate.Run(ctx, func(ctx restate.RunContext) (string, error) {
      return createRecurringPayment(req.CreditCard, paymentId)
    })
    if err != nil {
      return err
    }

    for _, subscription := range req.Subscriptions {
      _, err := restate.Run(ctx, func(ctx restate.RunContext) (string, error) {
        return createSubscription(req.UserId, subscription, payRef)
      })
      if err != nil {
        return err
      }
    }

    return nil
  }
  ```
</CodeGroup>

**Characteristics:**

* Use Durable Execution to run requests to completion
* Scale horizontally with high concurrency
* No shared state between requests

**Use for:** API calls, sagas, background jobs, task parallelization, ETL operations.

## Virtual Object

Stateful entities identified by a unique key.

<img alt="Virtual Objects" />

<CodeGroup>
  ```typescript TypeScript {"CODE_LOAD::ts/src/foundations/services/object.ts#here"}  theme={null}
  const cartObject = restate.object({
    name: "ShoppingCart",
    handlers: {
      addItem: async (ctx: restate.ObjectContext, item: Item) => {
        const items = (await ctx.get<Item[]>("cart")) ?? [];
        items.push(item);
        ctx.set("cart", items);
        return items;
      },

      getTotal: restate.handlers.object.shared(
        async (ctx: restate.ObjectSharedContext) => {
          const items = (await ctx.get<Item[]>("cart")) ?? [];
          return items.reduce((sum, item) => sum + item.price * item.quantity, 0);
        }
      ),
    },
  });
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/foundations/services/ShoppingCartObject.java#here"}  theme={null}
  @VirtualObject
  public class ShoppingCartObject {
    private static final StateKey<Cart> CART = StateKey.of("cart", Cart.class);

    @Handler
    public Cart addItem(ObjectContext ctx, Item item) {
      var cart = ctx.get(CART).orElse(new Cart());
      var newCart = cart.addItem(item);
      ctx.set(CART, newCart);
      return cart;
    }

    @Shared
    public double getTotal(SharedObjectContext ctx) {
      var cart = ctx.get(CART).orElse(new Cart());
      return cart.getTotalPrice();
    }
  }
  ```

  ```python Python {"CODE_LOAD::python/src/foundations/services/object.py#here"}  theme={null}
  cart_object = restate.VirtualObject("ShoppingCart")


  @cart_object.handler()
  async def add_item(ctx: ObjectContext, item: Item) -> Cart:
      cart = await ctx.get("cart", type_hint=Cart) or Cart()
      cart.items.append(item)
      ctx.set("cart", cart)
      return cart


  @cart_object.handler(kind="shared")
  async def get_total(ctx: ObjectSharedContext) -> float:
      cart = await ctx.get("cart", type_hint=Cart) or Cart()
      return sum(item.price * item.quantity for item in cart.items)
  ```

  ```go Go {"CODE_LOAD::go/foundations/services/object.go#here"}  theme={null}
  type ShoppingCartObject struct{}

  func (ShoppingCartObject) AddItem(ctx restate.ObjectContext, item Item) ([]Item, error) {
    items, err := restate.Get[[]Item](ctx, "items")
    if err != nil {
      return nil, err
    }
    items = append(items, item)
    restate.Set(ctx, "items", items)
    return items, nil
  }

  func (ShoppingCartObject) GetTotal(ctx restate.ObjectSharedContext) (float64, error) {
    items, err := restate.Get[[]Item](ctx, "items")
    if err != nil {
      return 0, err
    }
    total := 0.0
    for _, item := range items {
      total += item.Price * float64(item.Quantity)
    }
    return total, nil
  }
  ```
</CodeGroup>

**Characteristics:**

* Use Durable Execution to run requests to completion
* K/V state retained indefinitely and shared across requests
* Horizontal scaling with state consistency:
  * At most one handler with write access can run at a time per object key. Mimicks a queue per object key.
  * Concurrent execution across different object keys
  * Concurrent execution of shared handlers (read-only)

<img alt="Virtual Objects" />

**Use for:** Modeling entities like user accounts, shopping carts, chat sessions, AI agents, state machines, or any business entity needing persistent state.

## Workflow

Workflows orchestrate multi-step processes with guaranteed once-per-ID execution.

<CodeGroup>
  ```typescript TypeScript {"CODE_LOAD::ts/src/foundations/services/workflow.ts#here"}  theme={null}
  const signupWorkflow = restate.workflow({
    name: "UserSignup",
    handlers: {
      run: async (
        ctx: restate.WorkflowContext,
        user: { name: string; email: string }
      ) => {
        // workflow ID = user ID; workflow runs once per user
        const userId = ctx.key;

        await ctx.run("create", () => createUserEntry({ userId, user }));

        const secret = ctx.rand.uuidv4();
        await ctx.run("mail", () => sendVerificationEmail({ user, secret }));

        const clickSecret = await ctx.promise<string>("email-link-clicked");
        return clickSecret === secret;
      },

      click: async (
        ctx: restate.WorkflowSharedContext,
        request: { secret: string }
      ) => {
        await ctx.promise<string>("email-link-clicked").resolve(request.secret);
      },
    },
  });
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/foundations/services/SignupWorkflow.java#here"}  theme={null}
  @Workflow
  public class SignupWorkflow {
    private static final DurablePromiseKey<String> EMAIL_LINK_CLICKED =
        DurablePromiseKey.of("email-link-clicked", String.class);

    @Workflow
    public boolean run(WorkflowContext ctx, User user) {
      // workflow ID = user ID; workflow runs once per user
      String userId = ctx.key();

      ctx.run(() -> createUserEntry(userId, user));

      String secret = ctx.random().nextUUID().toString();
      ctx.run(() -> sendVerificationEmail(user, secret));

      String clickSecret = ctx.promise(EMAIL_LINK_CLICKED).future().await();
      return clickSecret.equals(secret);
    }

    @Shared
    public void click(SharedWorkflowContext ctx, ClickRequest request) {
      ctx.promiseHandle(EMAIL_LINK_CLICKED).resolve(request.secret());
    }
  }
  ```

  ```python Python {"CODE_LOAD::python/src/foundations/services/workflow.py#here"}  theme={null}
  signup_workflow = restate.Workflow("UserSignup")


  @signup_workflow.main()
  async def run(ctx: WorkflowContext, user: User) -> bool:
      # workflow ID = user ID; workflow runs once per user
      user_id = ctx.key()

      await ctx.run_typed("create", create_user_entry, user_id=user_id, user=user)

      secret = str(ctx.uuid())
      await ctx.run_typed("mail", send_verification_email, user=user, secret=secret)

      click_secret = await ctx.promise("email-link-clicked", type_hint=str).value()
      return click_secret == secret


  @signup_workflow.handler()
  async def click(ctx: WorkflowSharedContext, secret: str) -> None:
      await ctx.promise("email-link-clicked", type_hint=str).resolve(secret)
  ```

  ```go Go {"CODE_LOAD::go/foundations/services/workflow.go#here"}  theme={null}
  type SignupWorkflow struct{}

  func (SignupWorkflow) Run(ctx restate.WorkflowContext, user User) (bool, error) {
    // workflow ID = user ID; workflow runs once per user
    userId := restate.Key(ctx)

    _, err := restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
      return createUserEntry(userId, user)
    })
    if err != nil {
      return false, err
    }

    secret := restate.UUID(ctx).String()
    _, err = restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
      return sendVerificationEmail(user, secret)
    })
    if err != nil {
      return false, err
    }

    clickSecret, err := restate.Promise[string](ctx, "email-link-clicked").Result()
    if err != nil {
      return false, err
    }

    return clickSecret == secret, nil
  }

  func (SignupWorkflow) Click(ctx restate.WorkflowSharedContext, secret string) error {
    return restate.Promise[string](ctx, "email-link-clicked").Resolve(secret)
  }
  ```
</CodeGroup>

**Characteristics:**

* Use Durable Execution to run requests to completion
* The `run` handler executes exactly once per workflow ID
* Other handlers run concurrently with the `run` handler to signal, query state, or wait for events
* Optimized APIs for workflow interaction and lifecycle management

**Use for:** Processes requiring interaction capabilities like approval flows, user onboarding, multi-step transactions, and complex orchestration.

## Choosing the right service type

**Start with Basic Services** for most business logic, data processing, and API integrations.

**Use Virtual Objects** to model stateful entities.

**Use Workflows** for multi-step processes that execute exactly-once and require interaction.

You can combine these service types within the same application for different aspects of your business logic.

## Deployments, Endpoints, and Versions

Services deploy behind endpoints. Multiple services can bind to the same endpoint.

<CodeGroup>
  ```typescript TypeScript {"CODE_LOAD::ts/src/foundations/services/app.ts#here"}  theme={null}
  restate.serve({
    services: [subscriptionService, cartObject, signupWorkflow],
    port: 9080,
  });
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/foundations/services/App.java#here"}  theme={null}
  public class App {
    public static void main(String[] args) {
      RestateHttpServer.listen(
          Endpoint.bind(new SubscriptionService())
              .bind(new ShoppingCartObject())
              .bind(new SignupWorkflow()));
    }
  }
  ```

  ```python Python {"CODE_LOAD::python/src/foundations/services/app.py#here"}  theme={null}
  app = restate.app([subscription_service, cart_object, signup_workflow])

  if __name__ == "__main__":
      conf = hypercorn.Config()
      conf.bind = ["0.0.0.0:9080"]
      asyncio.run(hypercorn.asyncio.serve(app, conf))
  ```

  ```go Go {"CODE_LOAD::go/foundations/services/app.go#here"}  theme={null}
  func main() {
    if err := server.NewRestate().
      Bind(restate.Reflect(SubscriptionService{})).
      Bind(restate.Reflect(ShoppingCartObject{})).
      Bind(restate.Reflect(SignupWorkflow{})).
      Start(context.Background(), ":9080"); err != nil {
      log.Fatal(err)
    }
  }
  ```
</CodeGroup>

Services run on your preferred platform: serverless (AWS Lambda), containers (Kubernetes), or dedicated servers.

Restate handles versioning through immutable deployments where each deployment represents a specific, unchangeable version of your service code. After deploying your
services to an endpoint, you must register that endpoint with Restate so it can discover and route requests to it:

```shell theme={null}
restate deployments register http://my-service:9080
```

When you update your services, you deploy the new version to a new endpoint and register it with Restate, which automatically routes new requests to the latest version while existing requests continue on their
original deployment until completion.

See [deployment](/deploy/services/kubernetes) and [versioning](/services/versioning) docs for details.


# Local Restate Cluster with Docker
Source: https://docs.restate.dev/guides/cluster

Learn how to deploy a Restate cluster using Docker Compose.

This guide shows how to deploy a distributed Restate cluster consisting of three nodes using Docker and Docker compose.

<Info title="Prerequisites">
  * [Docker](https://docs.docker.com/get-started/get-docker/)
  * [Docker Compose](https://docs.docker.com/compose/install/)
</Info>

<Steps>
  <Step title="Deploy the Restate cluster using Docker">
    You can run a Restate Cluster using [Docker](https://docs.docker.com/get-started/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/).

    To deploy a 3-node distributed Restate cluster, create a file `docker-compose.yml` and run `docker compose up`.

    ```yaml expandable lines docker-compose.yml theme={null}
    x-environment: &common-env
      RESTATE_CLUSTER_NAME: "restate-cluster"
      # For more on logging, see: https://docs.restate.dev/operate/monitoring/logging
      RESTATE_LOG_FILTER: "restate=info"
      RESTATE_DEFAULT_REPLICATION: 2  # We require minimum of 2 nodes to accept writes
      # The addresses where nodes can reach each other over the "internal" Docker Compose network
      RESTATE_METADATA_CLIENT__ADDRESSES: '["http://restate-1:5122","http://restate-2:5122","http://restate-3:5122"]'
      # Partition snapshotting, see: https://docs.restate.dev/operate/snapshots
      RESTATE_WORKER__SNAPSHOTS__DESTINATION: "s3://restate/snapshots"
      RESTATE_WORKER__SNAPSHOTS__SNAPSHOT_INTERVAL_NUM_RECORDS: "1000"
      RESTATE_WORKER__SNAPSHOTS__AWS_REGION: "local"
      RESTATE_WORKER__SNAPSHOTS__AWS_ENDPOINT_URL: "http://minio:9000"
      RESTATE_WORKER__SNAPSHOTS__AWS_ALLOW_HTTP: true
      RESTATE_WORKER__SNAPSHOTS__AWS_ACCESS_KEY_ID: "minioadmin"
      RESTATE_WORKER__SNAPSHOTS__AWS_SECRET_ACCESS_KEY: "minioadmin"

    x-defaults: &defaults
      image: docker.restate.dev/restatedev/restate:latest
      extra_hosts:
        - "host.docker.internal:host-gateway"
      volumes:
        - restate-data:/restate-data

    services:
      restate-1:
        <<: *defaults
        ports:
          - "8080:8080"  # Ingress
          - "9070:9070"  # Admin
          - "5122:5122"  # Node-to-node communication
        environment:
          <<: *common-env
          RESTATE_NODE_NAME: restate-1
          RESTATE_FORCE_NODE_ID: 1
          RESTATE_ADVERTISED_ADDRESS: "http://restate-1:5122"  # Other Restate nodes must be able to reach us using this address
          RESTATE_AUTO_PROVISION: "true"                       # Only the first node provisions the cluster

      restate-2:
        <<: *defaults
        ports:
          - "25122:5122"
          - "29070:9070"
          - "28080:8080"
        environment:
          <<: *common-env
          RESTATE_NODE_NAME: restate-2
          RESTATE_FORCE_NODE_ID: 2
          RESTATE_ADVERTISED_ADDRESS: "http://restate-2:5122"
          RESTATE_AUTO_PROVISION: "false"

      restate-3:
        <<: *defaults
        ports:
          - "35122:5122"
          - "39070:9070"
          - "38080:8080"
        environment:
          <<: *common-env
          RESTATE_NODE_NAME: restate-3
          RESTATE_FORCE_NODE_ID: 3
          RESTATE_ADVERTISED_ADDRESS: "http://restate-3:5122"
          RESTATE_AUTO_PROVISION: "false"

      minio:
        image: quay.io/minio/minio
        entrypoint: "/bin/sh"
        # Ensure a bucket called "restate" exists on startup:
        command: "-c 'mkdir -p /data/restate && /usr/bin/minio server --quiet /data'"
        ports:
          - "9000:9000"

    # We create a volume to persist data across container starts; delete it via `docker volume rm restate-data` if you want to start a fresh cluster
    volumes:
      restate-data:
    ```

    The cluster uses the `replicated` Bifrost provider and replicates log writes to a minimum of two nodes.
    Since we are running with 3 nodes, the cluster can tolerate one node failure without becoming unavailable.
    By default, partition state is replicated to all workers (though each partition has only one acting leader at a time).

    The `replicated` metadata cluster consists of all nodes since they all run the `metadata-server` role.
    Since the `replicated` metadata cluster requires a majority quorum to operate, the cluster can tolerate one node failure without becoming unavailable.

    Take a look at the [cluster deployment documentation](/server/clusters) for more information on how to configure and deploy a distributed Restate cluster.

    In this example we also deployed a Minio server to host the cluster snapshots bucket. Visit [Snapshots](/server/snapshots) to learn more about whis is strongly recommended for all clusters.
  </Step>

  <Step title="Check the cluster status">
    You can check the status of the cluster by running the `restatectl status` command on any of the started Restate servers.
    Note, it might take a few seconds until the cluster has fully started and the status is available.

    ```shell theme={null}
    docker compose exec restate-1 restatectl status
    ```
  </Step>

  <Step title="Start a local Restate service">
    Follow the [Quickstart guide](/quickstart) to create a simple Restate service.
  </Step>

  <Step title="Register the service endpoint">
    You can register the service endpoint at any of the started Restate nodes since they all run the `admin` role.

    ```shell theme={null}
    restate dp register http://host.docker.internal:9080
    ```

    Or alternatively you can open the Restate UI at [http://localhost:9080](http://localhost:9080) and register the service endpoint there.
  </Step>

  <Step title="Invoke the service">
    You can invoke the registered service at any of the started Restate nodes since they all run the ingress.

    ```shell theme={null}
    curl localhost:8080/Greeter/greet --json '"Sarah"' &&
    curl localhost:28080/Greeter/greet --json '"Bob"' &&
    curl localhost:38080/Greeter/greet --json '"Eve"'
    ```
  </Step>

  <Step title="Kill and restart Restate servers">
    Try killing and restarting one of the Restate nodes and see how the cluster reacts.

    ```shell theme={null}
    docker compose kill restate-1 &&
    sleep 5 &&
    docker compose up -d restate-1
    ```
  </Step>

  <Step title="Create snapshots">
    Try instructing the partition processors to create a snapshot of their state in the object store bucket:

    ```shell theme={null}
    docker compose exec restate-1 restatectl snapshot create
    ```

    Navigate to the Minio console at [http://localhost:9000](http://localhost:9000) and browse the bucket contents (default credentials: `minioadmin`/`minioadmin`).
  </Step>

  <Step title="🎉Congratulations, you managed to run your first distributed Restate cluster and simulated some failures!" />
</Steps>

Here are some next steps for you to try:

* Try to configure a 5-node Restate cluster that can tolerate up to two node failures.
* Trim the logs (either manually, or by setting up automatic trimming) *before* adding more nodes.
* Try to deploy a 3-node Restate cluster using Kubernetes.


# Connecting Kubernetes Services to Restate Cloud
Source: https://docs.restate.dev/guides/connecting-k8s-services-to-cloud

Learn how to connect services on Kubernetes to Restate Cloud.

This guide walks you through connecting Kubernetes services to Restate Cloud with Restate Operator.

We will use [kind](https://kind.sigs.k8s.io/) to create a local Kubernetes cluster for testing purposes.

<Steps>
  <Step title="Prerequisites">
    Install the following on your local machine:

    * [Docker](https://docs.docker.com/)
    * [Helm](https://helm.sh/docs/intro/install/)
    * [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl)
    * [Restate](/installation)
    * [Restate Cloud account](https://cloud.restate.dev/) (free tier available)
    * [kind](https://kind.sigs.k8s.io/docs/user/quick-start)

    This guide was written with kind v0.30.0, [Restate Operator](https://github.com/restatedev/restate-operator/tree/main) v1.8.1, and Restate v1.5.0.
    Contact us on [Discord](https://discord.restate.dev) or [Slack](https://slack.restate.dev) if something does not work as expected.
  </Step>

  <Step title="Create a Restate Cloud environment">
    Open the [Restate Cloud UI](https://cloud.restate.dev) and create a new environment.

    Note the environment id (`env_...`) and request signing key (under security -> HTTP services: `publickeyv1_...`), as you will need them later.
  </Step>

  <Step title="Create a kind cluster">
    ```bash theme={null}
    kind create cluster
    ```

    Set `kubectl` context to `kind-kind`:

    ```bash theme={null}
    kubectl cluster-info --context kind-kind
    ```
  </Step>

  <Step title="Install the Restate Operator">
    Install the Restate Operator via Helm:

    ```bash theme={null}
    helm install restate-operator \
      oci://ghcr.io/restatedev/restate-operator-helm \
      --namespace restate-operator \
      --create-namespace
    ```

    To install the operator, you need to be able to create namespaces and CRDs.
  </Step>

  <Step title="Create a Restate Cloud environment secret">
    Create an API key via `Developers` tab of the Restate Cloud UI and give it `Full` permissions.

    The API key starts with `api_`. Put it in a file named `token` and add the secret:

    ```bash theme={null}
    kubectl create secret generic my-cloud-environment-secret \
    --from-file=token=./token -n restate-operator
    ```
  </Step>

  <Step title="Set up a secure tunnel between the kind cluster and Restate Cloud">
    Create a `RestateCloudEnvironment` manifest in a file called `restate-cloud-env.yaml`:

    ```yaml restate-cloud-env.yaml theme={null}
    apiVersion: restate.dev/v1beta1
    kind: RestateCloudEnvironment
    metadata:
      name: my-cloud-environment
    spec:
      environmentId: env_...
      signingPublicKey: publickeyv1_...
      region: eu
      authentication:
        secret:
          name: my-cloud-environment-secret
          key: token
    ```

    * The `environmentId` can be found in the Restate Cloud UI in the top left corner when you select your environment.
      This ID starts with `env_`.

    * The `signingPublicKey` can be found in the Restate Cloud UI under `Developers`. Scroll down to the `Security` section to find the public key under `HTTP endpoints`.
      The public key starts with `publickeyv1_`.

    * Set the `region` field to the region your Restate Cloud environment is hosted in. Possible values are `us` and `eu`.
      You can find the region next to your environment ID in the Restate Cloud UI.

    Finally, apply the manifest:

    ```bash theme={null}
    kubectl apply -f restate-cloud-env.yaml
    ```

    This deploys a tunnel service in the `restate-operator` namespace that connects the kind cluster to your Restate Cloud environment.

    And wait until the tunnel is ready:

    ```bash theme={null}
    kubectl get pods -w -n restate-operator
    ```
  </Step>

  <Step title="Build and upload a Restate service Docker image to the kind cluster">
    For example, download the TypeScript Hello World service:

    ```bash theme={null}
    restate example typescript-hello-world &&
    cd typescript-hello-world &&
    npm install
    ```

    Build a Docker image for the service:

    ```bash theme={null}
    docker build -t my-restate-service:0.0.1 .
    ```

    Upload the image to the kind cluster:

    ```bash theme={null}
    kind load docker-image my-restate-service:0.0.1
    ```
  </Step>

  <Step title="Deploy the Restate service">
    Create a `RestateDeployment` manifest for the service in a file called `service-deployment.yaml`:

    ```yaml service-deployment.yaml theme={null}
    apiVersion: restate.dev/v1beta1
    kind: RestateDeployment
    metadata:
      name: my-app
    spec:
      replicas: 1
      restate:
        register:
          cloud: my-cloud-environment
      selector:
        matchLabels:
          app: my-app
      template:
        metadata:
          labels:
            app: my-app
        spec:
          containers:
          - name: app
            image: my-restate-service:0.0.1
            ports:
            - name: restate
              containerPort: 9080
    ```

    Apply the manifest to deploy the service:

    ```bash theme={null}
    kubectl apply -f service-deployment.yaml -n restate-test
    ```

    You should now see the deployed service listed in the Restate Cloud UI:

    <Frame>
      <img />
    </Frame>

    The Restate Operator automatically [registered](/services/versioning#registering-a-deployment) the service with the Restate cluster.
  </Step>

  <Step title="Invoke the service">
    In the Restate Cloud UI playground, you can now invoke the service. In the overview page, click on the `greet` handler of your service to open the playground.
    Then send a request:

    <Frame>
      <img />
    </Frame>
  </Step>

  <Step title="🎉 You did it!">
    You have successfully connected a Restate service to Restate Cloud!

    Check out the [Restate Cloud docs](/cloud/connecting-services) for further information.
  </Step>
</Steps>

After you are done experimenting, you can delete the kind cluster:

```bash theme={null}
kind delete cluster
```


# Cron Jobs
Source: https://docs.restate.dev/guides/cron

Schedule tasks periodically with Restate

A cron job is a scheduled task that runs periodically at a specified time or interval. It is often used for background tasks like cleanup or sending notifications.

## How does Restate help?

Restate has many features that make it a good fit for implementing cron jobs:

* **Durable timers**: Schedule tasks reliably for future execution.
* **Task resiliency**: Automatic retries until success.
* **Task control**: Inspect and cancel running jobs.
* **K/V state**: Store and query cron job details using K/V storage.
* **FaaS support**: Integrates with platforms like AWS Lambda, scaling to zero when idle.
* **Scalability**: Handles many parallel jobs; scales horizontally.
* **Observability**: View execution history and job status in the Restate UI.

Restate doesn't have built-in cron functionality, but its building blocks make it easy to build reliable, custom schedulers.

## Example

To implement cron jobs, you can copy over the following file ([TS](https://github.com/restatedev/examples/blob/main/typescript/patterns-use-cases/src/cron/cron_service.ts) /
[Java](https://github.com/restatedev/examples/blob/main/java/patterns-use-cases/src/main/java/my/example/cron/Cron.java) /
[Go](https://github.com/restatedev/examples/blob/main/go/patterns-use-cases/src/cron/cron.go)) to your project:

<CodeGroup>
  ```ts TypeScript expandable {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/patterns-use-cases/src/cron/cron_service.ts?collapse_prequel"}  theme={null}
  export const cronJobInitiator = restate.service({
    name: "CronJobInitiator",
    handlers: {
      create: async (ctx: restate.Context, request: JobRequest) => {
        // Create a new job ID and initiate the cron job object for that ID
        // We can then address this job object by its ID
        const jobId = ctx.rand.uuidv4();
        const job = await ctx.objectClient(cronJob, jobId).initiate(request);
        return `Job created with ID ${jobId} and next execution time ${job.next_execution_time}`;
      },
    },
  });

  export const cronJob = restate.object({
    name: "CronJob",
    handlers: {
      initiate: async (ctx: restate.ObjectContext, request: JobRequest): Promise<JobInfo> => {
        if (await ctx.get<JobInfo>(JOB_STATE)) {
          throw new TerminalError("Job already exists for this ID.");
        }

        return await scheduleNextExecution(ctx, request);
      },
      execute: async (ctx: restate.ObjectContext) => {
        const jobState = await ctx.get<JobInfo>(JOB_STATE);
        if (!jobState) {
          throw new TerminalError("Job not found.");
        }

        // execute the task
        const { service, method, key, payload } = jobState.request;
        if (payload) {
          ctx.genericSend({
            service,
            method,
            parameter: payload,
            key,
            inputSerde: serde.json,
          });
        } else {
          ctx.genericSend({
            service,
            method,
            parameter: undefined,
            key,
            inputSerde: serde.empty,
          });
        }

        await scheduleNextExecution(ctx, jobState.request);
      },
      cancel: async (ctx: restate.ObjectContext) => {
        // Cancel the next execution
        const jobState = await ctx.get<JobInfo>(JOB_STATE);
        if (jobState) {
          ctx.cancel(jobState.next_execution_id);
        }

        // Clear the job state
        ctx.clearAll();
      },
      getInfo: restate.handlers.object.shared(async (ctx: restate.ObjectSharedContext) => {
        return ctx.get<JobInfo>(JOB_STATE);
      }),
    },
  });

  const scheduleNextExecution = async (
    ctx: restate.ObjectContext,
    request: JobRequest,
  ): Promise<JobInfo> => {
    // Parse cron expression
    // Persist current date in Restate for deterministic replay
    const currentDate = await ctx.date.now();
    let interval;
    try {
      interval = CronExpressionParser.parse(request.cronExpression, { currentDate });
    } catch (e) {
      throw new TerminalError(`Invalid cron expression: ${(e as Error).message}`);
    }

    const next = interval.next().toDate();
    const delay = next.getTime() - currentDate;

    // Schedule next execution for this job
    const thisJobId = ctx.key; // This got generated by the CronJobInitiator
    const handle = ctx.objectSendClient(cronJob, thisJobId, { delay }).execute();

    // Store the job information
    const jobState = {
      request,
      next_execution_time: next.toString(),
      next_execution_id: await handle.invocationId,
    };
    ctx.set<JobInfo>(JOB_STATE, jobState);
    return jobState;
  };
  ```

  ```java Java expandable {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/patterns-use-cases/src/main/java/my/example/cron/Cron.java"}  theme={null}
  package my.example.cron;

  import static com.cronutils.model.CronType.UNIX;

  import com.cronutils.model.definition.CronDefinitionBuilder;
  import com.cronutils.model.time.ExecutionTime;
  import com.cronutils.parser.CronParser;
  import dev.restate.common.Request;
  import dev.restate.common.Target;
  import dev.restate.sdk.Context;
  import dev.restate.sdk.ObjectContext;
  import dev.restate.sdk.SharedObjectContext;
  import dev.restate.sdk.annotation.*;
  import dev.restate.sdk.common.StateKey;
  import dev.restate.sdk.common.TerminalException;
  import dev.restate.serde.TypeTag;
  import java.time.ZonedDateTime;
  import java.util.Optional;

  /*
   * A distributed cron service built with Restate that schedules tasks based on cron expressions.
   *
   * Features:
   * - Create cron jobs with standard cron expressions (e.g., "0 0 * * *" for daily at midnight)
   * - Schedule any Restate service handler or virtual object method
   * - Guaranteed execution with Restate's durability
   * - Cancel and inspect running jobs
   *
   * Usage:
   * 1. Send requests to CronInitiator.create() to start new jobs
   * 2. Each job gets a unique ID and runs as a CronJob virtual object
   * 3. Jobs automatically reschedule themselves after each execution
   */
  public class Cron {

    public record JobRequest(
        String cronExpression, // e.g. "0 0 * * *" (every day at midnight)
        String service,
        String method, // Handler to execute with this schedule
        Optional<String> key, // Optional Virtual Object key of the task to call
        Optional<String> payload) {} // Optional data to pass to the handler

    public record JobInfo(JobRequest request, String nextExecutionTime, String nextExecutionId) {}

    @Name("CronJobInitiator")
    @Service
    public static class JobInitiator {
      @Handler
      public String create(Context ctx, JobRequest request) {
        // Create a new job ID and initiate the cron job object for that ID
        // We can then address this job object by its ID
        var jobId = ctx.random().nextUUID().toString();
        var cronJob = CronJobClient.fromContext(ctx, jobId).initiate(request).await();
        return String.format(
            "Job created with ID %s and next execution time %s", jobId, cronJob.nextExecutionTime());
      }
    }

    @Name("CronJob")
    @VirtualObject
    public static class Job {

      private final StateKey<JobInfo> JOB_STATE = StateKey.of("job-state", JobInfo.class);
      private final CronParser PARSER =
          new CronParser(CronDefinitionBuilder.instanceDefinitionFor(UNIX));

      @Handler
      public JobInfo initiate(ObjectContext ctx, JobRequest request) {
        if (ctx.get(JOB_STATE).isPresent()) {
          throw new TerminalException("Job already exists for this ID");
        }
        return scheduleNextExecution(ctx, request);
      }

      @Handler
      public void execute(ObjectContext ctx) {
        JobRequest request =
            ctx.get(JOB_STATE).orElseThrow(() -> new TerminalException("Job not found")).request;

        executeTask(ctx, request);
        scheduleNextExecution(ctx, request);
      }

      @Handler
      public void cancel(ObjectContext ctx) {
        ctx.get(JOB_STATE)
            .ifPresent(jobState -> ctx.invocationHandle(jobState.nextExecutionId).cancel());

        // Clear the job state
        ctx.clearAll();
      }

      @Shared
      public Optional<JobInfo> getInfo(SharedObjectContext ctx) {
        return ctx.get(JOB_STATE);
      }

      private void executeTask(ObjectContext ctx, JobRequest job) {
        Target target =
            (job.key.isPresent())
                ? Target.virtualObject(job.service, job.method, job.key.get())
                : Target.service(job.service, job.method);
        var request =
            (job.payload.isPresent())
                ? Request.of(
                    target, TypeTag.of(String.class), TypeTag.of(Void.class), job.payload.get())
                : Request.of(target, new byte[0]);
        ctx.send(request);
      }

      private JobInfo scheduleNextExecution(ObjectContext ctx, JobRequest request) {
        // Parse cron expression
        ExecutionTime executionTime;
        try {
          executionTime = ExecutionTime.forCron(PARSER.parse(request.cronExpression));
        } catch (IllegalArgumentException e) {
          throw new TerminalException("Invalid cron expression: " + e.getMessage());
        }

        // Calculate next execution time
        var now = ctx.run(ZonedDateTime.class, ZonedDateTime::now);
        var delay =
            executionTime
                .timeToNextExecution(now)
                .orElseThrow(() -> new TerminalException("Cannot determine next execution time"));
        var next =
            executionTime
                .nextExecution(now)
                .orElseThrow(() -> new TerminalException("Cannot determine next execution time"));

        // Schedule next execution for this job
        String thisJobId = ctx.key(); // This got generated by the CronJobInitiator
        var handle = CronJobClient.fromContext(ctx, thisJobId).send().execute(delay);

        // Save job state
        var jobState = new JobInfo(request, next.toString(), handle.invocationId());
        ctx.set(JOB_STATE, jobState);
        return jobState;
      }
    }
  }
  ```

  ```go Go expandable {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/patterns-use-cases/src/cron/cron.go"}  theme={null}
  package main

  import (
    "fmt"
    "time"

    restate "github.com/restatedev/sdk-go"
    "github.com/robfig/cron/v3"
  )

  // JobRequest represents the structure for creating a cron job
  type JobRequest struct {
    CronExpression string `json:"cronExpression"` // The cron expression e.g. "0 0 * * *" (every day at midnight)
    Service        string `json:"service"`
    Method         string `json:"method"`            // Handler to execute with this schedule
    Key            string `json:"key,omitempty"`     // Optional: Virtual Object key of task to call
    Payload        string `json:"payload,omitempty"` // Optional payload to pass to the handler
  }

  // JobInfo represents the stored job information
  type JobInfo struct {
    Request           JobRequest `json:"request"`
    NextExecutionTime time.Time  `json:"next_execution_time"`
    NextExecutionID   string     `json:"next_execution_id"`
  }

  const JOB_STATE = "job-state" // K/V state key for storing job info in the Restate

  // CronJobInitiator service for creating new cron jobs
  //
  // A distributed cron service built with Restate that schedules tasks based on cron expressions.
  //
  // Features:
  // - Create cron jobs with standard cron expressions (e.g., "0 0 * * *" for daily at midnight)
  // - Schedule any Restate service handler or virtual object method
  // - Guaranteed execution with Restate's durability
  // - Cancel and inspect running jobs
  //
  // Usage:
  // 1. Send requests to CronJobInitiator.Create() to start new jobs
  // 2. Each job gets a unique ID and runs as a CronJob virtual object
  // 3. Jobs automatically reschedule themselves after each execution

  type CronJobInitiator struct{}

  func (CronJobInitiator) Create(ctx restate.Context, req JobRequest) (string, error) {
    // Create a new job ID and initiate the cron job object for that ID
    // We can then address this job object by its ID
    jobID := restate.UUID(ctx).String()

    fmt.Printf("Creating new cron job with ID %s for service %s and method %s", jobID, req.Service, req.Method)
    job, err := restate.Object[*JobInfo](ctx, "CronJob", jobID, "Initiate").Request(req)
    if err != nil {
      return "", err
    }

    return fmt.Sprintf("Job created with ID %s and next execution time %s",
      jobID, job.NextExecutionTime.Format(time.RFC3339)), nil
  }

  type CronJob struct{}

  func (CronJob) Initiate(ctx restate.ObjectContext, req JobRequest) (*JobInfo, error) {
    // Check if jobState already exists
    jobState, err := restate.Get[*JobInfo](ctx, JOB_STATE)
    if err != nil {
      return nil, err
    }
    if jobState != nil {
      return nil, restate.TerminalErrorf("jobState already exists for this ID")
    }

    return scheduleNextExecution(ctx, req)
  }

  func (CronJob) Execute(ctx restate.ObjectContext) error {
    // Get the job information
    jobState, err := restate.Get[*JobInfo](ctx, JOB_STATE)
    if err != nil {
      return err
    }
    if jobState == nil {
      return restate.TerminalErrorf("job not found")
    }

    // Add key if it's a virtual object call
    req := jobState.Request
    fmt.Printf("Executing job with ID: %s for service %s for method %s", restate.Key(ctx), req.Service, req.Method)
    if req.Key != "" {
      restate.ObjectSend(ctx, req.Service, req.Key, req.Method).Send(req.Payload)
    } else {
      restate.ServiceSend(ctx, req.Service, req.Method).Send(req.Payload)
    }

    // Schedule the next execution
    _, err = scheduleNextExecution(ctx, req)
    return err
  }

  func (CronJob) Cancel(ctx restate.ObjectContext) error {
    // Get the job to cancel the next execution
    job, err := restate.Get[*JobInfo](ctx, JOB_STATE)
    if err != nil {
      return err
    }
    if job == nil {
      return restate.TerminalError(fmt.Errorf("job not found for cancellation"), 404)
    }
    restate.CancelInvocation(ctx, job.NextExecutionID)

    restate.ClearAll(ctx)
    return nil
  }

  func (CronJob) GetInfo(ctx restate.ObjectSharedContext) (*JobInfo, error) {
    return restate.Get[*JobInfo](ctx, JOB_STATE)
  }

  // scheduleNextExecution calculates and schedules the next execution of the cron job
  func scheduleNextExecution(ctx restate.ObjectContext, req JobRequest) (*JobInfo, error) {
    // Parse cron expression
    parser := cron.NewParser(cron.Minute | cron.Hour | cron.Dom | cron.Month | cron.Dow)
    schedule, err := parser.Parse(req.CronExpression)
    if err != nil {
      return nil, restate.TerminalErrorf("invalid cron expression: %v", err)
    }

    // Get current time deterministically from Restate
    currentTime, _ := restate.Run(ctx, func(ctx restate.RunContext) (time.Time, error) {
      return time.Now(), nil
    })

    // Calculate next execution time
    nextTime := schedule.Next(currentTime)
    delay := nextTime.Sub(currentTime)

    // Schedule next execution for this job
    thisJobID := restate.Key(ctx) // This got generated by the CronJobInitiator
    handle := restate.ObjectSend(ctx, "CronJob", thisJobID, "Execute").Send(nil, restate.WithDelay(delay))

    // Store the job information
    jobState := &JobInfo{
      Request:           req,
      NextExecutionTime: nextTime,
      NextExecutionID:   handle.GetInvocationId(),
    }
    restate.Set(ctx, JOB_STATE, jobState)
    return jobState, nil
  }
  ```
</CodeGroup>

This contains two services: a `CronJobInitiator` Service, and a `CronJob` Virtual Object:

<img alt="Cron Jobs Diagram" />

Bind both services to your endpoint, and register them.

Usage:

* Send requests to `CronJobInitiator.create()` to start new jobs with standard [cron expressions](https://www.baeldung.com/cron-expressions):

```json theme={null}
{
    "cronExpression": "0 0 * * *", # E.g. run every day at midnight
    "service": "TaskService", # Schedule any Restate handler
    "method": "executeTask",
    "key": "taskId", # Optional, Virtual Object key
    "payload": "Hello midnight!"
}
```

* Each job gets a unique ID and runs as a CronJob Virtual Object.
* Jobs automatically reschedule themselves after each execution.

<Note>
  This pattern is implementable with any of our SDKs. We are still working on translating all patterns to all SDK languages.
  If you need help with a specific language, please reach out to us via [Discord](https://discord.restate.dev) or [Slack](https://slack.restate.dev).
</Note>

## Running the example

<Steps>
  <Step title="Download the example">
    <CodeGroup>
      ```bash TypeScript theme={null}
      restate example typescript-patterns-use-cases && cd typescript-patterns-use-cases
      ```

      ```bash Java theme={null}
      restate example java-patterns-use-cases && cd java-patterns-use-cases
      ```

      ```bash Go theme={null}
      restate example go-patterns-use-cases && cd go-patterns-use-cases
      ```
    </CodeGroup>
  </Step>

  <Step title="Start the Restate Server">
    ```bash theme={null}
    restate-server
    ```
  </Step>

  <Step title="Start the Service">
    <CodeGroup>
      ```bash TypeScript theme={null}
      npm install
      npx tsx watch ./src/cron/task_service.ts
      ```

      ```bash Java theme={null}
      ./gradlew -PmainClass=my.example.cron.TaskService run
      ```

      ```bash Go theme={null}
      go run ./src/cron
      ```
    </CodeGroup>
  </Step>

  <Step title="Register the services">
    ```bash theme={null}
    restate deployments register localhost:9080
    ```
  </Step>

  <Step title="Send a request">
    For example, run `executeTask` every minute:

    <CodeGroup>
      ```bash TypeScript theme={null}
      curl localhost:8080/CronJobInitiator/create --json '{
        "cronExpression": "* * * * *",
        "service": "TaskService",
        "method": "executeTask",
        "payload": "Hello new minute!"
      }'
      ```

      ```bash Java theme={null}
      curl localhost:8080/CronJobInitiator/create --json '{
        "cronExpression": "* * * * *",
        "service": "TaskService",
        "method": "executeTask",
        "payload": "Hello new minute!"
      }'
      ```

      ```bash Go theme={null}
      curl localhost:8080/CronJobInitiator/Create --json '{
        "cronExpression": "* * * * *",
        "service": "TaskService",
        "method": "executeTask",
        "payload": "Hello new minute!"
      }'
      ```
    </CodeGroup>

    For example, or run `executeTask` at midnight:

    <CodeGroup>
      ```bash TypeScript theme={null}
      curl localhost:8080/CronJobInitiator/create --json '{
        "cronExpression": "0 0 * * *",
        "service": "TaskService",
        "method": "executeTask",
        "payload": "Hello midnight!"
      }'
      ```

      ```bash Java theme={null}
      curl localhost:8080/CronJobInitiator/create --json '{
        "cronExpression": "0 0 * * *",
        "service": "TaskService",
        "method": "executeTask",
        "payload": "Hello midnight!"
      }'
      ```

      ```bash Go theme={null}
      curl localhost:8080/CronJobInitiator/Create --json '{
        "cronExpression": "0 0 * * *",
        "service": "TaskService",
        "method": "executeTask",
        "payload": "Hello midnight!"
      }'
      ```
    </CodeGroup>

    You can also use the cron service to execute handlers on Virtual Objects, by specifying the Virtual Object key in the request.

    You will get back a response with the job ID.

    Using the job ID, you can then get information about the job:

    <CodeGroup>
      ```bash TypeScript theme={null}
      curl localhost:8080/CronJob/myJobId/getInfo
      ```

      ```bash Java theme={null}
      curl localhost:8080/CronJob/myJobId/getInfo
      ```

      ```bash Go theme={null}
      curl localhost:8080/CronJob/myJobId/GetInfo
      ```
    </CodeGroup>

    Or cancel the job later:

    <CodeGroup>
      ```bash TypeScript theme={null}
      curl localhost:8080/CronJob/myJobId/cancel
      ```

      ```bash Java theme={null}
      curl localhost:8080/CronJob/myJobId/cancel
      ```

      ```bash Go theme={null}
      curl localhost:8080/CronJob/myJobId/Cancel
      ```
    </CodeGroup>
  </Step>

  <Step title="Check the scheduled tasks and state">
    In the UI, you can see how the tasks are scheduled, and how the state of the cron jobs is stored in Restate.

    <img alt="Cron Service Schedule" />

    <img alt="Cron State UI" />

    You can kill and restart any of the services or the Restate Server, and the scheduled tasks will still be there.
  </Step>
</Steps>

## Adapt to your use case

Note that this implementation is fully resilient, but you might need to make some adjustments to make this fit your use case:

* Take into account time zones.
* Adjust how you want to handle tasks that fail until the next task gets scheduled. With the current implementation, you would have concurrent executions of the same cron job (one retrying and the other starting up).
  If you want to cancel the failing task when a new one needs to start, you can do the following: at the beginning of the `execute` call, retrieve the `next_execution_id` from the job state and check if it is completed by [attaching to it](/develop/ts/service-communication#re-attach-to-an-invocation) with [a timeout set to 0](/develop/ts/durable-timers#timers-and-timeouts). If it is not completed, [cancel it](/develop/ts/service-communication#cancel-an-invocation) and start the new iteration.


# Databases and Restate
Source: https://docs.restate.dev/guides/databases

Learn when and how to use databases in combination with Restate.

Restate is not only a system for resilience and durability in communication and orchestration, it can also store data like a database system, to enable writing long-lived stateful logic with strong out-of-the-box resilience.

To avoid confusion: In this page, we refer to the type of state that outlives a single workflow or durable handler execution, and that uses Restate’s [Virtual Objects](/foundations/services#virtual-object). This is a feature not available in typical workflow systems or durable execution frameworks.

This guide discusses thoughts on when to use Restate for that type of state, when to use a database, and how to integrate Restate and databases.

## What state to store in Restate, what to store in databases

State stored directly in Restate ([Virtual Objects](/foundations/services#virtual-object)) has a set of advantages:

* **Ultra robust**: No coordination between systems is needed. Instead, state access/updates directly participate in the durable execution logic (go through the same consensus log), making the state always aligned with the business logic: no lost updates, duplications of state changes, stale views of state, etc.
* **Simple correctness model**: The single-writer-per-key and linearizable consistency model means developers don’t need to worry about locks, race conditions, dirty reads, change visibility, zombie processes, etc.
* **Efficient**: Computation and access patterns to state align naturally, you cannot get into situations where multiple function executions try to access/update the same database entry and contend on locks or interfere with each other's transaction.
  No write amplification through separate durability in an external system (instead state is simply committed with the durable execution journal data).
* **Serverless**: when Restate invokes a handler, it attaches the relevant state to the request. On serverless, for example AWS lambda, this means your state is locally available when the function executes, no access delays, synchronization, wait times, etc.
* **No additional dependency** - the K/V store is integrated in the core of Restate. You don't need to do anything extra to start using state when developing locally, or migrating the Restate app to cloud/production.

State stored directly in Restate (Virtual Objects) has some limitations:

* K/V interface with single-key transactions (though a single key can store a document with flexible structures)
* SQL is supported only for analytics / introspection, not for updates/transactions.
* Modifiable only from the Virtual Object, not from other services. Any modification needs to be sent as a request to the Virtual Object.
* While state can be exported in a standard way (psql client), it is managed by a less standard system, compared to some databases.

## When to use which?

Use a database:

* When you need complex access patterns, full SQL, text search, time-series analysis, etc.
* For core business data, like your user database, products database, history of transactions, etc. that you want to access from other services as well

Use Restate for any state requiring tight integration with function logic, resilience, and correctness:

* State that is part of transactional state machines (payment status, activation status, ...)
* Session state (shopping cart, ongoing orders, LLM context, AI agent context, ...)
* State that is part of distributed orchestration and coordination (e.g., versions/life-cycle of shared resources, transaction IDs, reference counts, distributed locks/semaphores/leases, ...)
* Agents / Digital-twins (agent/twin memory and context)
* State in event processing pipelines (aggregates, joins)
* State of control planes (desired cluster status, resource references, ...)
* Consistent state/metadata overlay over eventual consistent infra

## How to interact with Databases from Restate

This set of examples shows various patterns to access databases from Restate handlers.

```ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/patterns-use-cases/src/database/main.ts"}  theme={null}
import * as restate from "@restatedev/restate-sdk";
import { Sequelize, Transaction } from "sequelize";
import { runQueryAs2pcTxn } from "./2phasecommit";

// ----------------------------------------------------------------------------
//
//  This example shows various patterns to access databases from Restate
//  handlers.
//
//  The basics are:
//
//  (1) You don't need to do anything special, you can just interact with
//      your database the same way as from other microservices or
//      workflow activities.
//
//  (2) But you can use Restate's state, journal, and concurrency mechanisms
//      as helpers to improve common access problems, solve race conditions,
//      or avoid inconsistencies in the presence of retries, concurrent requests,
//      or zombie processes.
//
// The below example illustrate this step by step.
//
// ----------------------------------------------------------------------------

type Row = {
  userId: string; // the primary key
  name: string;
  address: string;
  credits: number;
  version?: number; // the row version (used by some access methods)
};

const db = new Sequelize("postgres://restatedb:restatedb@localhost:5432/exampledb", {
  logging: false,
});

// ----------------------------------------------------------------------------

/**
 * This service implements the database accesses similar to the way you
 * might access the database from other microservices.
 * It doesn't use any Restate features to help making the DB access consistent.
 */
const simpledbAccess = restate.service({
  name: "simple",
  handlers: {
    /**
     * Simple read from a database row.
     * This is the same as it would be from any non-Restate service.
     */
    read: async (_ctx: restate.Context, userid: string) => {
      const [results, _meta] = await db.query(
        `SELECT *
                   FROM users
                  WHERE userid = '${userid}'`,
      );
      return results.length > 0 ? results[0] : "(row not found)";
    },

    /**
     * This performs a read in a Restate durable action, persisting the result.
     * The read access is the same as in a non-Restate service, but the result
     * is persisted in the execution journal.
     *
     * This is useful for example, if you have decisions/branches based on
     * the result and want to ensure consistent and deterministic behavior
     * on retries (where the value in the database might have changed
     * in-between retries).
     */
    durableRead: async (ctx: restate.Context, userid: string) => {
      const credits = await ctx.run("read credits", async () => {
        const [results, _metadata] = await db.query(
          `SELECT credits
                       FROM users
                      WHERE userid = '${userid}'`,
        );
        return (results[0] as any)?.credits;
      });

      if (credits === 0) {
        // Execute some conditional logic, e.g., alerting, sending reminder.
        // Automatic retries (on failure) will follow this branch again (using
        // Restate's journalled value for 'credits') so code in this branch will
        // reliably run to conclusion
        console.log("Found an account without balance, doing something about it...");
      }

      return credits;
    },

    /**
     * Inserts a row into the database. No Restate-specific handling, this
     * simply uses the database's primary key uniqueness to avoid duplicate
     * entries.
     */
    insert: async (_ctx: restate.Context, row: Row) => {
      const { userId, name, address, credits } = row;

      const [_, numInserted] = await db.query(
        `INSERT INTO users (userid, name, address, credits, version)
                      VALUES ('${userId}', '${name}', '${address}', ${credits}, 0)
                 ON CONFLICT (userid) DO NOTHING`,
      );
      return numInserted === 1;
    },

    /**
     * Updates a row in a database. This handler makes no specific effort to
     * add idempotency, so it purely relies on database / application semantics.
     * It is vulnerable to concurrent updates overwriting each other, but can make
     * sense if this is fine for a specific type of (rare) update.
     *
     * For an example that use Restate to add consistency, see the examples
     * further below.
     */
    update: async (_ctx: restate.Context, update: { userId: string; newName: string }) => {
      const { userId, newName } = update;

      // Update row - we execute this statement directly, because it is the only
      // step in this handler. If the handler contains multiple steps, wrap this in
      // a `ctx.run( () => { ... } )` block to ensure it does not get re-executed once
      // during retries of successive steps

      const [_, meta] = await db.query(
        `UPDATE users
                    SET name = '${newName}'
                  WHERE userID = '${userId}'`,
      );
      return (meta as any).rowCount === 1;
    },
  },
});

// ----------------------------------------------------------------------------

/*
 * When updating single rows by primary key (on a database or a key/value store),
 * you can use Virtual Objects to serialize access per key. This happens automatically
 * if you access the database from a Virtual Object and use the same key for object and
 * primary key in the database.
 *
 * Most databases internally serialize updates on the same row anyways (either through locks,
 * or by detecting conflicts during when attempting to commit transactions and rolling back), so
 * making the access sequential from the application can help to avoid lock congestion and
 * avoid pathological degradation of optimistic concurrency control around contended keys.
 *
 * Furthermore, this update pattern makes it possible to use conditional updates with versions
 * that don't get duplicated on retries, resulting in proper exactly-once semantics.
 */
const keyedDbAccess = restate.object({
  name: "keyed",
  handlers: {
    /**
     * Updates the credits for a row in a database.
     * This variant makes no effort to ensure the update does not get duplicated.
     *
     * In practice, this will can reasonably safe, because there is a very small window
     * time where the query gets re-executed after success, whihc is when the query succeeded,
     * but Restate did not see the completion of the handler.
     *
     * Note that this is especially rare, since the single-writer-per-key semantics of the
     * Virtual Object ensure that no contention on rows can happen in the database.
     */
    update: async (ctx: restate.ObjectContext, credits: number) => {
      const userid = ctx.key;

      await db.query(
        `UPDATE users
                    SET credits = credits + ${credits}
                  WHERE userid = '${userid}'`,
      );
    },

    /**
     * Updates the 'credits' field and uses the 'version' number to make the update
     * idempotent.
     *
     * The approach is to separate reads (current credits, version) and
     * updates into separate queries. The reads are kept in the execution journal, and
     * the updates are conditional on the fact that the version did not change in between.
     * This is sometimes referred to as a _semantic lock pattern_.
     *
     * Because the reads go through a `ctx.run` step (strong consensus inside Restate),
     * it is guaranteed that the same invocation will always work off of the same version.
     * Regardless of partial failures/retries/zombies/network partitions, the code will
     * always observe a single version.
     *
     * If all updates to the row go through this handler (or other handlers in this
     * Virtual Object that use the versioning scheme) then no duplicates are possible.
     */
    updateConditional: async (ctx: restate.ObjectContext, addCredits: number) => {
      const userid = ctx.key;

      // first read the current credits and the version.
      // Because the reads go through a `ctx.run` step (strong consensus), the
      // a single invocation and all if its retries can never see different values
      // for version and credits.
      const [credits, version] = await ctx.run("read credits, version", async () => {
        const [results, _] = await db.query(
          `SELECT credits, version
                       FROM users
                      WHERE userid = '${userid}'`,
        );

        if (results.length !== 1) {
          throw new restate.TerminalError(`No single row with userid = '${userid}'`);
        }
        const row = results[0] as any;
        return [Number(row.credits), Number(row.version)];
      });

      // Make the update conditional on the fact that the version is still the same (see
      // 'AND version = ...'). If the version is no longer the same, it means that a
      // previous execution attempt updated this and crashed before completing the handler).
      // It cannot be completed by a concurrent query, because all operations on the userid
      // are serialized through the Virtual Object.
      const [_, meta] = await db.query(
        `UPDATE users
                    SET credits = ${credits + addCredits},
                        version = ${version + 1}
                  WHERE userid = '${userid}'
                    AND version = ${version}`,
      );

      if ((meta as any).rowCount !== 1) {
        console.debug("Update was not applied - version no longer matches.");
      }
    },

    /**
     * Reads a row from the database.
     *
     * This one is markes as a SHARED handler, which means it doesn't queue
     * with the other invocations for this key. That way reads execute immediately
     * and concurrently, while writes execute sequentially.
     */
    read: restate.handlers.object.shared(async (ctx: restate.ObjectSharedContext) => {
      const userid = ctx.key;

      const [results, _meta] = await db.query(
        `SELECT *
                       FROM users
                      WHERE userid = '${userid}'`,
      );
      return results[0] ?? "(row not found)";
    }),
  },
});

// ----------------------------------------------------------------------------

/**
 * This service uses a second table in the database to remember idempotency tokens
 * for each operation. The idempotency tokens are inserted into the database in the
 * same transaction as the main operation.
 *
 * The code uses Restate's features to deterministically generate the tokens and
 * to expire them after a day.
 */
const idempotencyKeyDbAccess = restate.service({
  name: "idempotency",
  handlers: {
    /**
     * Adds credits to a single user entry. This operation is _idempotent_ within a
     * time window of a day.
     *
     * This handler uses a second table in the database to remember idempotency tokens
     * for each update. The idempotency tokens are inserted into the database in the
     * same transaction as the update.
     *
     * The code uses Restate's features to deterministically generate the tokens and
     * to expire them after a day.
     */
    update: async (ctx: restate.Context, update: { userId: string; addCredits: number }) => {
      // create a persistent idempotency key. we use Restate's random number generator,
      // because that is the most efficient way to generate a durable deterministic ID.
      const idempotencyKey = ctx.rand.uuidv4();

      // expire the idempotency key from the database after one day
      // by scheduling a call to the handler that deletes the key
      ctx
        .serviceSendClient(idempotencyKeyDbAccess)
        .expireIdempotencyKey(idempotencyKey, restate.rpc.sendOpts({ delay: { days: 1 } }));

      // checking the existence of the idempotency key and making the update happens
      // in one database transaction
      const tx = await db.transaction({
        isolationLevel: Transaction.ISOLATION_LEVELS.SERIALIZABLE,
      });
      try {
        // first test whether the idempotency token exists by checking whether
        // we can insert one into the table (primary key constraint prevents
        // duplicates)
        const [_tokenRows, numInserted] = await db.query(
          `INSERT INTO user_idempotency (id)
                        VALUES ('${idempotencyKey}')
                    ON CONFLICT (id) DO NOTHING`,
          { transaction: tx },
        );

        if (numInserted !== 1) {
          // idempotency key was inserted before, so this must be a retry
          await tx.rollback();
          return;
        }

        // now make the actual update
        await db.query(
          `UPDATE users
                        SET credits = credits + ${update.addCredits}
                      WHERE userid = '${update.userId}'`,
          { transaction: tx },
        );

        // if everything succeeds, commit idempotency token and update atomically
        await tx.commit();
      } catch (e) {
        // speed up release of DB locks by eagerly aborting transaction
        await tx.rollback();
        throw e;
      }
    },

    /**
     * Deletes the idempotency key from the database.
     * This handler is involked as a scheduled call from the handler that performs
     * the update.
     */
    expireIdempotencyKey: async (_ctx: restate.Context, key: string) => {
      await db.query(
        `DELETE FROM user_idempotency
                       WHERE id = '${key}'`,
      );
    },
  },
});

// ----------------------------------------------------------------------------

/**
 * Exactly-once updates to the database using a 2-phase commit integration between
 * Restate and PostegreSQL.
 *
 * This uses PostgreSQL's PREPARE TRANSACTION feature to ensure that Restate
 * kicks off a transaction once and only once, and it thus works without any
 * additional idempotency or versioning mechanisms.
 * BUT: It requires the Postgres database to enable prepared transactions in the
 * configuration via `max_prepared_transactions = 100`.
 *
 * This is a _good use case_ of the 2-phase-commit protocol, because it is not
 * the (rightfully) criticized use case of running distributed transactions across
 * services and thus couple availability and liveliness of multiple services together.
 * This case here keeps the same participants and dependencies and only uses 2pc as a
 * way to split a single-participant transaction into two stages so that Restate can
 * avoid executing the query against the database multiple times under retries.
 *
 * NOTE: This code currently works only on services which have a bidi streaming connection,
 * (so any service deployed as http/2 or http1 bidi) due to the fact that it needs to
 * differentiate re-tries of blocks from first executions, which currently works only
 * for streaming connections. This is simply a missing feature and not a fundamental
 * limitation.
 */
const twoPhaseCommitDbAccess = restate.service({
  name: "twoPhaseCommit",
  handlers: {
    update: async (ctx: restate.Context, update: { userId: string; addCredits: number }) => {
      const { userId, addCredits } = update;

      const query = `UPDATE users
                           SET credits = credits + ${addCredits}
                           WHERE userid = '${userId}'`;

      await runQueryAs2pcTxn(ctx, db, query);
    },
  },
});

// ----------------------------------------------------------------------------

restate.serve({
  services: [simpledbAccess, keyedDbAccess, idempotencyKeyDbAccess, twoPhaseCommitDbAccess],
  port: 9080,
});
```

### Two-phase commit

```ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/patterns-use-cases/src/database/2phasecommit.ts"}  theme={null}
import * as restate from "@restatedev/restate-sdk";
import { randomUUID } from "node:crypto";
import { Sequelize, Transaction } from "sequelize";

/**
 * Example 2-phase-commit integration library for Restate and PostegreSQL.
 *
 * This uses PostgreSQL's PREPARE TRANSACTION feature to ensure that Restate
 * kicks off a transaction once and only once.
 *
 * Postgres' interface is far from optimal here, for example it does not provide a way
 * to block/fence off discarded transaction IDs or establish other relationship between
 * transaction IDs. As a result, this code is needs to be very aggressive in cleaning
 * up previous transactions, to avoid leaving database locks in place.
 *
 * This code currently works only on services which have a bidi streaming connection !!!
 */
export async function runQueryAs2pcTxn(
  ctx: restate.Context,
  dbConnection: Sequelize,
  query: string,
) {
  const action = async (db: Sequelize, txn: Transaction) => {
    await db.query(query, { transaction: txn });
  };
  return runAs2pcTxn(ctx, dbConnection, action);
}

/**
 * See docs for {@link runQueryAs2pcTxn}.
 *
 * IMPORTANT: When using function, all queries issues must make use of the `tnx` object:
 * 
 * await db.query(
 *     "UDATE ... SET ... WHERE ...",
 *     { transaction: txn }
 * );
 * 
 */
export async function runAs2pcTxn(
  ctx: restate.Context,
  dbConnection: Sequelize,
  action: (dbConnection: Sequelize, txn: Transaction) => Promise<void>,
) {
  // this code only works on a streaming bi-directional connection, see below.
  checkRunsOnBidi(ctx);

  // We loop here until we are are sure that there successfully made it through
  // (1) running DB transaction, (2) preparing for commit (pre-commit), (3) recording
  // that in the Restate journal.
  // If we don't make it through the full squence in one execution attempt, we clear the
  // pending transaction and try again under a different transaction ID. Successive re-tries
  // and recoveries from failures also make sure they clean up the pending transaction.
  // In the absence of failures, the first iteration will succeed.
  let txnIdToCommit: string | undefined = undefined;
  while (txnIdToCommit === undefined) {
    // We generate the transaction ID and use a trick here to find out if we
    // actually generated the ID, or whether the ID was restored from the journal.
    // That is the reason this code only works on long-lived bidi connections (not Lambda),
    // because on Lambda, every `ctx.run` block is only ack-ed through a re-invocation
    // and journal replay.
    let executedThisTime: boolean = false;
    const txnId: string = await ctx.run("generate transaction id", () => {
      executedThisTime = true;
      return randomUUID();
    });

    // this is our cleanup hook for this specific transaction ID, to run in case
    // we need to ensure the transaction is cleared
    async function cleanup() {
      try {
        await dbConnection.query(`ROLLBACK PREPARED '${txnId}'`);
      } catch (e: any) {
        if (e.original?.code === "42704" && e instanceof Error) {
          // code 42704 means "no txn with that ID found".
          // => transaction was not prepared (crashed before) or already rolled back
          // => we can ignore this
          return;
        }
        console.error(e.message);
        throw e;
      }
    }

    try {
      const txnRan = await ctx.run("run txn attempt (or cleanup old txn)", async () => {
        // If we did not generate this transaction ID, a previous execution attempt did.
        // We don't know whether that execution made it to the point that it already prepared
        // the query (pre-committed) and crashed just before recording the completion of the
        // `ctx.run` block in the Restate journal. As a result, we clean up that transaction ID
        // to be on the safe side and not leave lingering database locks.
        if (!executedThisTime) {
          cleanup();
          return false; // try again with a different txnId
        }

        // run:
        //   (1) the query transaction
        //   (2) pre-commit step (prepare transaction)
        //   (3) record this in the Restate journal

        const txn = await dbConnection.transaction();
        try {
          // (1) run the main app-defined database query (or queries)
          await action(dbConnection, txn);

          // (2) pre-commit - after this step, the txn is immutable and not rolled back
          // any more after closing/disposing/loss-of-connection
          await dbConnection.query(`PREPARE TRANSACTION '${txnId}'`, {
            transaction: txn,
          });
          // (3) if this durable block returns ('true' recoded in journal) step (3) will
          // be completed
          return true;
        } catch (e) {
          await txn.rollback();
          throw e;
        }
      });

      if (txnRan) {
        txnIdToCommit = txnId;
      }
    } catch (e) {
      // clean up in case the query didn't go through, to speed up release
      // of possibly prepared txn
      await cleanup();
      throw e;
    }
  }

  // now commit the prepared transaction - this step is idempotent, so if it was
  // already committed, this does nothing
  await ctx.run("commit prepared transaction", () =>
    dbConnection.query(`COMMIT PREPARED '${txnIdToCommit}'`),
  );
}

function checkRunsOnBidi(ctx: restate.Context): boolean {
  // this needs to be replaced by the actual check, which needs an additional
  // property to be exposed in Restate
  return true; // dummy value for now
}
```

## Running the examples

This is purely optional, the example code and comments document the behavior well.
Running the example can be interesting, though, if you want to play with specific failure scenarios, like pausing/killing processes at specific points and observe the behavior.

[Check out the readme of the example on how to run the example.](https://github.com/restatedev/examples/blob/main/typescript/patterns-use-cases/README.md#database-interaction-patterns)


# Durable Webhooks
Source: https://docs.restate.dev/guides/durable-webhooks

Process webhook events from external services with exactly-once delivery guarantees.

Restate handlers can be used as **durable processors of webhook events**.

<img />

## How does Restate help?

* Restate persists all incoming events, and ensures that they are **processed exactly once**, across failures and restarts. Restate guarantees your handler runs till completion.
* Let Restate **deduplicate** events on an [idempotency key](/invoke/http#invoke-a-handler-idempotently). If the sender of the event retries, Restate will not process the event again.
* Use any of Restate's [**durable SDK constructs**](/concepts/durable_building_blocks) when processing the events: durable calls/messaging to other services, durable timers, scheduling tasks, K/V state, concurrency guarantees etc.
* Any handler can be a durable webhook endpoint. **No need to do anything special or extra!**

Just point your webhook endpoint to your handler: `restate:8080/MyService/myHandler`.

## Example

This example processes webhook callbacks from a payment provider.

The payment provider notifies us about payment success or failure of invoices by sending webhook events to our handler.
The handler then routes the event to the correct processor via a one-way message.

<CodeGroup>
  ```ts TypeScript {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/patterns-use-cases/src/webhookcallbacks/webhook_callback_router.ts?collapse_prequel"}  theme={null}
  const webhookCallbackRouter = restate.service({
    name: "WebhookCallbackRouter",
    handlers: {
      // Any handler can be a durable webhook processor that never loses events
      // You don't need to do anything special for this.
      // Just point your webhook to the handler endpoint: restate:8080/WebhookCallbackRouter/onStripeEvent
      onStripeEvent: async (ctx: restate.Context, event: StripeEvent) => {
        if (event.type === "invoice.payment_failed") {
          ctx.objectSendClient(PaymentTracker, event.data.object.id).onPaymentFailed(event);
        } else if (event.type === "invoice.payment_succeeded") {
          ctx.objectSendClient(PaymentTracker, event.data.object.id).onPaymentSuccess(event);
        }
      },
    },
  });

  restate.serve({
    services: [webhookCallbackRouter],
    port: 9080,
  });
  ```

  ```go Go {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/patterns-use-cases/src/webhookcallbacks/callbackrouter.go?collapse_prequel"}  theme={null}
  type WebhookCallbackRouter struct{}

  // Any handler can be a durable webhook processor that never loses events
  // You don't need to do anything special for this.
  // Just point your webhook to the handler endpoint: restate:8080/WebhookCallbackRouter/OnStripeEvent

  func (WebhookCallbackRouter) OnStripeEvent(ctx restate.Context, event StripeEvent) error {
    if event.Type == "invoice.payment_failed" {
      restate.ObjectSend(ctx, "PaymentTracker", "onPaymentFailed", event.Data.Object.ID).
        Send(event)
    } else if event.Type == "invoice.payment_succeeded" {
      restate.ObjectSend(ctx, "PaymentTracker", "onPaymentSuccess", event.Data.Object.ID).
        Send(event)
    }
    return nil
  }
  ```
</CodeGroup>

<Info>
  This pattern is implementable with any of our SDKs. We are still working on translating all patterns to all SDK languages.
  If you need help with a specific language, please reach out to us via [Discord](https://discord.restate.dev) or [Slack](https://slack.restate.dev).
</Info>


# Restate and Encore
Source: https://docs.restate.dev/guides/encore

Learn how to run Restate service handlers inside an Encore TypeScript backend.

[Encore](https://encore.dev) is an open-source backend framework for TypeScript (and Go) with type-safe APIs,
declarative infrastructure, and built-in observability.
You can run Restate service handlers inside an Encore app by exposing them via
[raw endpoints](https://encore.dev/docs/ts/primitives/raw-endpoints), which give you direct access to the
Node.js request/response objects.

This lets you use Encore for your API layer, databases, and infrastructure while Restate handles durable execution.

<Info>
  **Prerequisites**

  * [The prerequisites for running Restate TS services](/get_started/quickstart)
</Info>

<Steps>
  <Step title="Create an Encore app">
    Follow the [Encore quickstart](https://encore.dev/docs/ts/quick-start) to install the CLI,
    then create a new app:

    ```shell theme={null}
    encore app create
    ```
  </Step>

  <Step title="Install the Restate SDK">
    Add the Restate TypeScript SDK to your Encore project:

    ```shell theme={null}
    npm install @restatedev/restate-sdk
    ```
  </Step>

  <Step title="Define your Restate service handlers">
    Create a Restate service with your handler logic. This is standard Restate code:

    ```ts order-processor/restate.ts {"CODE_LOAD::ts/src/guides/encore/restate.ts"} theme={null}
    import * as restate from "@restatedev/restate-sdk";

    export const orderProcessor = restate.service({
        name: "OrderProcessor",
        handlers: {
            process: async (ctx: restate.Context, order: { id: string; item: string }) => {
                const confirmation = await ctx.run(() => chargePayment(order));
                await ctx.run(() => reserveInventory(order));
                await ctx.run(() => sendConfirmation(order, confirmation));
                return { status: "completed", orderId: order.id };
            },
        },
    });

    export type OrderProcessor = typeof orderProcessor;

    async function chargePayment(order: { id: string; item: string }) {
        console.log(`Charging payment for order ${order.id}`);
        return `conf-${order.id}`;
    }

    async function reserveInventory(order: { id: string; item: string }) {
        console.log(`Reserving inventory for ${order.item}`);
    }

    async function sendConfirmation(order: { id: string; item: string }, confirmation: string) {
        console.log(`Sending confirmation ${confirmation} for order ${order.id}`);
    }
    ```

    Each `ctx.run()` call is journaled by Restate. If the process crashes mid-execution, Restate replays completed
    steps and resumes where it left off.
  </Step>

  <Step title="Expose the handler via an Encore raw endpoint">
    Create the Encore service and wire the Restate endpoint handler to a raw endpoint.

    Since Encore's raw endpoints use Node.js HTTP types while the Restate SDK provides a Fetch API handler,
    we need a small adapter to bridge between the two:

    ```ts order-processor/encore.restate.ts {"CODE_LOAD::ts/src/guides/encore/encore.restate.ts"} theme={null}
    import { Service } from "encore.dev/service";
    export default new Service("order-processor");
    ```

    ```ts order-processor/api.ts {"CODE_LOAD::ts/src/guides/encore/api.ts"} theme={null}
    import { api } from "encore.dev/api";
    import { createEndpointHandler } from "@restatedev/restate-sdk/fetch";
    import { orderProcessor } from "./restate";

    const handler = createEndpointHandler({
        services: [orderProcessor],
    });

    // Catch-all raw endpoint that Restate Server calls into
    export const restateEndpoint = api.raw(
        { expose: true, path: "/restate/!rest", method: "*" },
        async (req, resp) => {
            const url = new URL(req.url ?? "/", `http://${req.headers.host}`);
            const body =
                req.method !== "GET" && req.method !== "HEAD"
                    ? await new Promise<Buffer>((resolve, reject) => {
                        const chunks: Buffer[] = [];
                        req.on("data", (chunk: Buffer) => chunks.push(chunk));
                        req.on("end", () => resolve(Buffer.concat(chunks)));
                        req.on("error", reject);
                    })
                    : undefined;

            const webReq = new Request(url.toString(), {
                method: req.method,
                headers: req.headers as Record<string, string>,
                body,
            });

            const webResp = await handler(webReq);

            resp.writeHead(
                webResp.status,
                Object.fromEntries(webResp.headers.entries()),
            );
            const buf = await webResp.arrayBuffer();
            resp.end(Buffer.from(buf));
        },
    );
    ```

    Encore now serves the Restate handler at `/restate/*`. The Restate Server will call this
    endpoint to discover and invoke your handlers.
  </Step>

  <Step title="Start Restate Server and Encore">
    In one terminal, start the Restate Server:

    ```shell theme={null}
    restate-server
    ```

    In another terminal, start your Encore app:

    ```shell theme={null}
    encore run
    ```
  </Step>

  <Step title="Register your service with Restate">
    Tell the Restate Server where to find your handlers:

    ```shell theme={null}
    restate deployments register --use-http1.1 http://localhost:4000/restate
    ```

    You should see the discovered `OrderProcessor` service printed out.

    <Info>
      The `--use-http1.1` flag is needed because Encore's raw endpoints serve HTTP/1.1.
    </Info>
  </Step>

  <Step title="Invoke your handler">
    Send a request to Restate Server's ingress (port 8080), which routes it to your Encore-hosted handler:

    ```shell theme={null}
    curl localhost:8080/OrderProcessor/process \
      --json '{"id": "order-123", "item": "widget"}'
    ```

    You can also invoke the handler from other Encore API endpoints using the Restate client:

    ```ts order-processor/orders.ts {"CODE_LOAD::ts/src/guides/encore/orders.ts"} theme={null}
    import { api } from "encore.dev/api";
    import * as restate from "@restatedev/restate-sdk-clients";
    import { OrderProcessor } from "./restate";

    const restateClient = restate.connect({ url: "http://localhost:8080" });

    export const createOrder = api(
        { method: "POST", path: "/orders", expose: true },
        async (req: { item: string }): Promise<{ orderId: string }> => {
            const orderId = `order-${Date.now()}`;
            await restateClient
                .serviceSendClient<OrderProcessor>({ name: "OrderProcessor" })
                .process({ id: orderId, item: req.item });
            return { orderId };
        },
    );
    ```
  </Step>

  <Step title="You're running Restate handlers inside Encore!" />
</Steps>

## Related resources

* [Encore documentation](https://encore.dev/docs/ts)
* [Encore raw endpoints](https://encore.dev/docs/ts/primitives/raw-endpoints)
* [`encore-restate-gen`](https://github.com/sebastianhindhede/encore-restate-gen): a CLI tool that auto-generates Restate service definitions and typed clients from your Encore project
* [Microservice orchestration use cases](/use-cases/microservice-orchestration)
* [Sagas guide](/guides/sagas)


# Error Handling
Source: https://docs.restate.dev/guides/error-handling

Learn how to handle transient and terminal errors in your applications.

Restate handles retries for failed invocations. By default, Restate infinitely retries all errors with an exponential backoff strategy.

This guide helps you fine-tune the retry behavior for your use cases.

## Infrastructure errors (transient) vs. application errors (terminal)

In Restate, we distinguish between two types of errors: transient errors and terminal errors.

* **Transient errors** are temporary and can be retried. They are typically caused by infrastructure issues (network problems, service overload, API unavailability,...).
* **Terminal errors** are permanent and should not be retried. They are typically caused by application logic (invalid input, business rule violation, ...).

## Handling transient errors via retries

Restate assumes by default that all errors are transient errors and therefore retryable.
If you do not want an error to be retried, you need to specifically label it as a terminal error ([see below](#application-errors-terminal)).

Restate lets you configure the retry strategy at different levels: for the invocation and at the run-block-level.

### At Invocation level

Restate retries executing invocations that can't make any progress according to a retry policy.
This policy controls the retry intervals, the maximum number of attempts and whether to **pause** or **kill** the invocation when the attempts are exhausted.

To see the current retry policy for your services, click on your service in the overview page of the UI:

<Frame>
  <img alt="Restart from prefix" />
</Frame>

The retry policy can be set on each individual handler, or for all the handlers of a service, or globally in the Restate configuration directly.

<AccordionGroup>
  <Accordion title="Configure for services/handlers">
    To configure the retry policy on a service/handler level, check [retry service configuration](/services/configuration#retries).
  </Accordion>

  <Accordion title="Configure Restate server defaults">
    Via the [`restate-server` configuration file](/server/configuration):

    ```toml restate.toml theme={null}
    [invocation.default-retry-policy]
    initial-interval = "50ms"
    exponentiation-factor = 2.0
    max-attempts = 70
    max-interval = "60s"
    on-max-attempts = "pause"  # Use "pause" (default) or "kill"
    ```

    Then run the Restate Server with:

    ```shell theme={null}
    restate-server --config-file restate.toml
    ```

    Or you can set these options via env variables:

    ```dotenv theme={null}
    RESTATE_DEFAULT_RETRY_POLICY__INITIAL_INTERVAL="10s"
    RESTATE_DEFAULT_RETRY_POLICY__MAX_ATTEMPTS=100
    RESTATE_DEFAULT_RETRY_POLICY__MAX_INTERVAL="10s"
    ```

    This retry policy will retry the invocation 100 times, after which the invocation will be paused if no progress can be made. To resume a paused invocation, check the paragraph below.

    You can also retry forever, without ever pausing or killing the invocation:

    ```dotenv theme={null}
    RESTATE_DEFAULT_RETRY_POLICY__MAX_ATTEMPTS=unlimited
    ```

    Check the [configuration documentation](/server/configuration) and [reference](/references/server-config) for the `default-retry-policy`.

    When a retry policy is unset, Restate by default will retry undefinitely, alike setting `max-attempts = "unlimited"`.
  </Accordion>
</AccordionGroup>

### At the Run-Block-Level

Handlers use run blocks to execute non-deterministic actions, often involving other systems and services (API call, DB write, ...).
These run blocks are especially prone to transient failures, and you might want to configure a specific retry policy for them.

<CodeGroup>
  ```ts TypeScript {"CODE_LOAD::ts/src/guides/retries.ts#here"}  theme={null}
  const myRunRetryPolicy = {
    initialRetryInterval: { milliseconds: 500 },
    retryIntervalFactor: 2,
    maxRetryInterval: { seconds: 1 },
    maxRetryAttempts: 5,
    maxRetryDuration: { seconds: 1 },
  };
  await ctx.run("write", () => writeToOtherSystem(), myRunRetryPolicy);
  ```

  ```python Python {"CODE_LOAD::python/src/guides/retries.py#here"}  theme={null}
  await ctx.run_typed(
      "write",
      write_to_other_system,
      restate.RunOptions(
          # Max number of retry attempts to complete the action.
          max_attempts=3,
          # Max duration for retrying, across all retries.
          max_retry_duration=timedelta(seconds=10),
      )
  )
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/guides/RetryRunService.java#here"}  theme={null}
  RetryPolicy myRunRetryPolicy =
      RetryPolicy.exponential(Duration.ofMillis(500), 2)
          .setMaxDelay(Duration.ofSeconds(10))
          .setMaxAttempts(10)
          .setMaxDuration(Duration.ofMinutes(5));
  ctx.run("my-run", myRunRetryPolicy, () -> writeToOtherSystem());
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/guides/RetryRunService.kt#here"}  theme={null}
  val myRunRetryPolicy = retryPolicy {
    initialDelay = 5.seconds
    exponentiationFactor = 2.0f
    maxDelay = 60.seconds
    maxAttempts = 10
    maxDuration = 5.minutes
  }
  ctx.runBlock("write", myRunRetryPolicy) { writeToOtherSystem() }
  ```

  ```go Go {"CODE_LOAD::go/guides/retries.go#here"}  theme={null}
  result, err := restate.Run(ctx,
    func(ctx restate.RunContext) (string, error) {
      return writeToOtherSystem()
    },
    // After 10 seconds, give up retrying
    restate.WithMaxRetryDuration(time.Second*10),
    // On the first retry, wait 100 milliseconds before next attempt
    restate.WithInitialRetryInterval(time.Millisecond*100),
    // Grow retry interval with factor 2
    restate.WithRetryIntervalFactor(2.0),
  )
  if err != nil {
    return err
  }
  ```
</CodeGroup>

Note that these retries are coordinated and initiated by the Restate Server.
So the handler goes through the regular retry cycle of suspension and re-invocation.

If you set a maximum number of attempts, then the run block will fail with a TerminalException once the retries are exhausted.

<Info title="Other levels of retry policies">
  Service-level retry policies are planned and will come soon.
</Info>

## Application errors (terminal)

By default, Restate infinitely retries all errors.
In some cases, you might not want to retry an error (e.g. because of business logic, because the issue is not transient, ...).

For these cases you can throw a terminal error. Terminal errors are permanent and are not retried by Restate.

You can throw a terminal error as follows:

<CodeGroup>
  ```ts TypeScript {"CODE_LOAD::ts/src/develop/error_handling.ts#terminal"}  theme={null}
  throw new TerminalError("Something went wrong.", { errorCode: 500 });
  ```

  ```python Python {"CODE_LOAD::python/src/develop/error_handling.py#terminal"}  theme={null}
  from restate.exceptions import TerminalError

  raise TerminalError("Something went wrong.")
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/develop/ErrorHandling.java#here"}  theme={null}
  throw new TerminalException(500, "Something went wrong");
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/ErrorHandling.kt#here"}  theme={null}
  throw TerminalException(500, "Something went wrong")
  ```

  ```go Go {"CODE_LOAD::go/develop/errorhandling.go#here"}  theme={null}
  return restate.TerminalError(fmt.Errorf("Something went wrong."), 500)
  ```

  ```rust Rust {"CODE_LOAD::rust/src/guides/retries.rs#terminal_error"}  theme={null}
  Err(TerminalError::new("This is a terminal error"))
  ```
</CodeGroup>

You can throw terminal errors from any place in your handler, including run blocks.

Unless catched, terminal errors stop the execution and are propagated back to the caller.
If the caller is another Restate service, the terminal error will propagate across RPCs, and will get thrown at the line where the RPC was made.
If this is not caught, it will propagate further up the call stack until it reaches the original caller.

You can catch terminal errors just like any other error, and build control flow around this.
For example, the catch block can run undo actions for the actions you did earlier in your handler, to bring it to a consistent state before rethrowing the terminal error.

For example, to catch a terminal error of a run block:

<CodeGroup>
  ```ts TypeScript {"CODE_LOAD::ts/src/guides/retries.ts#catch"}  theme={null}
  try {
    // Fails with a terminal error after 3 attempts or if the function throws one
    await ctx.run("write", () => writeToOtherSystem(), {
      maxRetryAttempts: 3,
    });
  } catch (e) {
    if (e instanceof restate.TerminalError) {
      // Handle the terminal error: undo previous actions and
      // propagate the error back to the caller
    }
    throw e;
  }
  ```

  ```python Python {"CODE_LOAD::python/src/guides/retries.py#catch"}  theme={null}
  try:
      # Fails with a terminal error after 3 attempts or if the function throws one
      await ctx.run_typed("write", write_to_other_system, restate.RunOptions(max_attempts=3))
  except TerminalError as err:
      # Handle the terminal error: undo previous actions and
      # propagate the error back to the caller
      raise err
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/guides/RetryRunService.java#catch"}  theme={null}
  try {
    // Fails with a terminal error after 3 attempts or if the function throws one
    ctx.run("my-run", RetryPolicy.defaultPolicy().setMaxAttempts(3), () -> writeToOtherSystem());
  } catch (TerminalException e) {
    // Handle the terminal error: undo previous actions and
    // propagate the error back to the caller
    throw e;
  }
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/guides/RetryRunService.kt#catch"}  theme={null}
  try {
    // Fails with a terminal error after 3 attempts or if the function throws one
    ctx.runBlock(
        "write",
        RetryPolicy(
            initialDelay = 500.milliseconds, maxAttempts = 3, exponentiationFactor = 2.0f)) {
          writeToOtherSystem()
        }
  } catch (e: TerminalException) {
    // Handle the terminal error: undo previous actions and
    // propagate the error back to the caller
    throw e
  }
  ```

  ```go Go {"CODE_LOAD::go/guides/retries.go#catch"}  theme={null}
  result, err := restate.Run(ctx, func(ctx restate.RunContext) (string, error) {
    return writeToOtherSystem()
  })
  if err != nil {
    if restate.IsTerminalError(err) {
      // Handle the terminal error: undo previous actions and
      // propagate the error back to the caller
    }
    return err
  }
  ```

  ```rust Rust {"CODE_LOAD::rust/src/guides/retries.rs#catch"}  theme={null}
  // Fails with a terminal error after 3 attempts or if the function throws one
  if let Err(e) = ctx
      .run(|| write_to_other_system())
      .retry_policy(RunRetryPolicy::default().max_attempts(3))
      .await
  {
      // Handle the terminal error: undo previous actions and
      // propagate the error back to the caller
      return Err(e);
  }
  ```
</CodeGroup>

<Info>
  When you throw a terminal error, you might need to undo the actions you did earlier in your handler to make sure that your system remains in a consistent state.
  Have a look at our [sagas guide](/guides/sagas) to learn more.
</Info>

## Cancellations are Terminal Errors

You can cancel invocations via the [CLI](/services/invocation/managing-invocations#cancel), UI and programmatically.
When you cancel an invocation, it throws a terminal error in the handler processing the invocation the next time it awaits a Promise or Future of a Restate Context action (e.g. run block, RPC, sleep,...; `RestatePromise` in TypeScript, `DurableFuture` in Java).
Unless caught, This terminal error will propagate up the call stack until it reaches the original caller.

Here again, the handler needs to have [compensation logic](/guides/sagas) in place to make sure the system remains in a consistent state, when you cancel an invocation.

## Timeouts between Restate and the service

There are two types of timeouts describing the behavior between Restate and the service.

### Inactivity timeout

When the Restate Server does not receive a next journal entry from a running handler within the inactivity timeout, it will ask the handler to suspend.
This timer guards against stalled service/handler invocations. Once it expires, Restate triggers a graceful termination by asking the service invocation to suspend (which preserves intermediate progress).

By default, the inactivity timeout is set to one minute.

You can increase the inactivity timeout if you have long-running `ctx.run` blocks, that lead to long pauses between journal entries. Otherwise, this timeout might kill the ongoing execution.

### Abort timeout

This timer guards against stalled service/handler invocations that are supposed to terminate.
The abort timeout is started after the 'inactivity timeout' has expired and the service/handler invocation has been asked to gracefully terminate.
Once the timer expires, it will abort the service/handler invocation.

By default, the abort timeout is set to ten minutes.
This timer potentially interrupts user code.
If the user code needs longer to gracefully terminate, then this value needs to be set accordingly.

If you have long-running `ctx.run` blocks, you need to increase both timeouts to prevent the handler from terminating prematurely.

### Configuring the timeouts

As with the retry policy, you can configure these timeouts on specific handlers, on all the handlers of a service, or globally in the Restate configuration directly.

<AccordionGroup>
  <Accordion title="Configure for services/handlers">
    To configure these timeouts on a service/handler level, use the UI or check [timeouts service configuration](/services/configuration#timeouts).
  </Accordion>

  <Accordion title="Configure Restate server defaults">
    Via the [`restate-server` configuration file](/server/configuration):

    ```toml restate.toml theme={null}
    [worker.invoker]
    inactivity-timeout = "1m"
    abort-timeout = "1m"
    ```

    ```shell theme={null}
    restate-server --config-file restate.toml
    ```

    Both timeouts follow the [jiff](https://docs.rs/jiff/latest/jiff/struct.Span.html) format.

    Or set it [via environment variables](/server/configuration#environment-variables), for example:

    ```shell theme={null}
    RESTATE_WORKER__INVOKER__INACTIVITY_TIMEOUT=5m \
    RESTATE_WORKER__INVOKER__ABORT_TIMEOUT=5m \
    restate-server
    ```
  </Accordion>
</AccordionGroup>

## Common patterns

These are some common patterns for handling errors in Restate:

### Sagas

Have a look at the [sagas guide](/guides/sagas) to learn how to revert your system back to a consistent state after a terminal error.
Keep track of compensating actions throughout your business logic and apply them in the catch block after a terminal error.

### Dead-letter queue

A [dead-letter queue (DLQ)](https://aws.amazon.com/what-is/dead-letter-queue/) is a queue where you can send messages that could not be processed due to errors.

You can implement this in Restate by wrapping your handler in a try-catch block. In the catch block you can forward the failed invocation to a DLQ Kafka topic or a catch-all handler which for example reports them or backs them up.

<Accordion title="Catching failed invocations before handler execution starts">
  Some errors might happen before the handler code gets invoked/starts running (e.g. service does not exist, request decoding errors in SDK HTTP server, ...).
  By default, Restate fails these requests with `400`.

  Handle these as follows:

  * In case the caller waited for the response of the failed call, the caller can handle the propagation to the DLQ.
  * If the caller did not wait for the response (one-way send), you would lose these messages.
  * Decoding errors can be caught by doing the decoding inside the handler.
    The called handler then takes raw input and does the decoding and validation itself.
    In this case, it would be included in the try-catch block which would do the dispatching:

  <CodeGroup>
    ```ts TypeScript {"CODE_LOAD::ts/src/guides/retries.ts#raw"}  theme={null}
    myHandler: async (ctx: restate.Context) => {
    try {
    const rawRequest = ctx.request().body;
    const decodedRequest = decodeRequest(rawRequest);

    // ... rest of your business logic ...
    } catch (e) {
    if (e instanceof restate.TerminalError) {
      // Propagate to DLQ/catch-all handler
    }
    throw e;
    }
    },
    ```

    ```python Python {"CODE_LOAD::python/src/guides/retries.py#raw"}  theme={null}
    @my_service.handler()
    async def my_handler(ctx: Context):
    try:
        raw_request = ctx.request().body
        decoded_request = decode_request(raw_request)

        # ... rest of your business logic ...

    except TerminalError as err:
        # Propagate to DLQ/catch-all handler
        raise err
    ```

    ```java Java {"CODE_LOAD::java/src/main/java/guides/RetryRunService.java#raw"}  theme={null}
    @Handler
    public void myHandler(Context ctx, @Accept("*/*") @Raw byte[] request) {
    try {
    var decodedRequest = decodeRequest(request);

    // ... rest of your business logic ...

    } catch (TerminalException e) {
    // Propagate to DLQ/catch-all handler
    }
    }
    ```

    ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/guides/RetryRunService.kt#raw"}  theme={null}
    @Handler
    suspend fun myHandler(ctx: Context, @Accept("*/*") @Raw request: ByteArray) {
    try {
    val decodedRequest = decodeRequest(request)

    // ... rest of your business logic ...

    } catch (e: TerminalException) {
    // Propagate to DLQ/catch-all handler
    throw e
    }
    }
    ```

    ```go Go {"CODE_LOAD::go/guides/retries.go#raw"}  theme={null}
    func (MyService) myHandler(ctx restate.Context) (string, error) {
    rawRequest := ctx.Request().Body
    decodedRequest, err := decodeRequest(rawRequest)
    if err != nil {
    if restate.IsTerminalError(err) {
      // Propagate to DLQ/catch-all handler
    }
    return "", err
    }

    // ... rest of your business logic ...
    return decodedRequest, nil
    }
    ```

    ```rust Rust {"CODE_LOAD::rust/src/guides/retries.rs#raw"}  theme={null}
    // Use Vec<u8> to represent a binary request
    async fn my_handler(&self, ctx: Context<'_>, request: Vec<u8>) -> Result<(), HandlerError> {
    let decoded_request = decode_request(&request)?;

    // ... rest of you business logic ...

    Ok(())
    }
    ```
  </CodeGroup>

  The other errors mainly occur due to misconfiguration of your setup (e.g. wrong service name, wrong handler name, forgot service registration...).
  You cannot handle those.
</Accordion>

### Timeouts for context actions

You can set timeouts for context actions like calls, awakeables, etc. to bound the time they take:

<CodeGroup>
  ```ts TypeScript {"CODE_LOAD::ts/src/guides/retries.ts#timeout"}  theme={null}
  try {
    // If the timeout hits first, it throws a `TimeoutError`.
    // If you do not catch it, it will lead to a retry.
    await ctx
      .serviceClient(myService)
      .myHandler("hello")
      .orTimeout({ seconds: 5 });

    const { id, promise } = ctx.awakeable();
    // do something that will trigger the awakeable
    await promise.orTimeout({ seconds: 5 });
  } catch (e) {
    if (e instanceof restate.TimeoutError) {
      // Handle the timeout error
    }
    throw e;
  }
  ```

  ```python Python {"CODE_LOAD::python/src/guides/retries.py#timeout"}  theme={null}
  match await restate.select(
      greeting=ctx.service_call(my_service_handler, "value"),
      timeout=ctx.sleep(timedelta(seconds=5)),
  ):
      case ["greeting", greeting]:
          print("Greeting:", greeting)
      case ["timeout", _]:
          print("Timeout occurred")
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/guides/RetryRunService.java#timeout"}  theme={null}
  try {
    // If the timeout hits first, it throws a `TimeoutError`.
    // If you do not catch it, it will lead to a retry.
    MyServiceClient.fromContext(ctx).myHandler("Hello").await(Duration.ofSeconds(5));

    var awakeable = ctx.awakeable(Boolean.class);
    // ...Do something that will trigger the awakeable
    awakeable.await(Duration.ofSeconds(5));

  } catch (TimeoutException e) {
    // Handle the timeout error
  }
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/guides/RetryRunService.kt#timeout"}  theme={null}
  try {
    ctx.awakeable<String>().withTimeout(5.seconds).await()
  } catch (e: TimeoutException) {
    // Handle the timeout
  }
  ```

  ```go Go {"CODE_LOAD::go/guides/retries.go#timeout"}  theme={null}
  awakeable := restate.Awakeable[string](ctx)
  timeout := restate.After(ctx, 5*time.Second)
  fut, err := restate.WaitFirst(ctx, awakeable, timeout)
  if err != nil {
    return err
  }
  switch fut {
  case awakeable:
    result, err := awakeable.Result()
    if err != nil {
      return err
    }
    slog.Info("Awakeable resolved first with: " + result)
  case timeout:
    if err := timeout.Done(); err != nil {
      return err
    }
    slog.Info("Timeout hit first")
  }
  ```
</CodeGroup>


# Guides
Source: https://docs.restate.dev/guides/index

Learn how to do common tasks with Restate.

## Recipes

<CardGroup>
  <Card title="Cron Jobs" href="/guides/cron">
    Schedule tasks periodically with Restate
  </Card>

  <Card title="Durable Webhooks" href="/guides/durable-webhooks">
    Process webhook events from external services with exactly-once delivery guarantees.
  </Card>

  <Card title="Parallelizing Work" href="/guides/parallelizing-work">
    Execute a list of tasks in parallel and then gather their result.
  </Card>

  <Card title="Rate Limiting" href="/guides/rate-limiting">
    Control request rates and prevent service overload with Restate
  </Card>

  <Card title="Sagas" href="/guides/sagas">
    Implementing undo operations in case of failures, to keep your system consistent
  </Card>
</CardGroup>

## Development Guides

<CardGroup>
  <Card title="Databases and Restate" href="/guides/databases">
    Learn when and how to use databases in combination with Restate.
  </Card>

  <Card title="Error Handling" href="/guides/error-handling">
    Learn how to handle transient and terminal errors in your applications.
  </Card>

  <Card title="Request Lifecycle" href="/guides/request-lifecycle">
    Deep dive into the lifecycle of a request in Restate
  </Card>
</CardGroup>

## Deployment Guides

<CardGroup>
  <Card title="Connecting Kubernetes Services to Restate Cloud" href="/guides/connecting-k8s-services-to-cloud">
    Learn how to connect services on Kubernetes to Restate Cloud.
  </Card>

  <Card title="Kubernetes deployments with Helm" href="/guides/restate-on-kind-with-helm">
    Learn how to deploy a Restate cluster using Helm on a kind Kubernetes cluster.
  </Card>

  <Card title="Kubernetes deployments with Restate Operator" href="/guides/restate-on-kind-with-operator">
    Learn how to deploy single-node Restate with Restate operator on a kind Kubernetes cluster.
  </Card>

  <Card title="Local Restate Cluster with Docker" href="/guides/cluster">
    Learn how to deploy a Restate cluster using Docker Compose.
  </Card>

  <Card title="Scaling to Multi-Node Deployments" href="/guides/local-to-replicated">
    Migrate a single node to a multi-node cluster.
  </Card>
</CardGroup>

## Integrations

<CardGroup>
  <Card title="Restate and Encore" href="/guides/encore">
    Learn how to run Restate service handlers inside an Encore TypeScript backend.
  </Card>

  <Card title="Restate-Kafka Quickstart" href="/guides/kafka-quickstart">
    Learn how to connect your Restate service to a Kafka topic.
  </Card>

  <Card title="XState" href="/guides/xstate">
    Integrate Restate with XState to implement durable state machines.
  </Card>
</CardGroup>


# Restate-Kafka Quickstart
Source: https://docs.restate.dev/guides/kafka-quickstart

Learn how to connect your Restate service to a Kafka topic.

In this guide, you will learn how to connect your Restate service handler to a Kafka topic.

Restate lets you connect any handler to a Kafka topic, and invoke the handler for each event that arrives on the topic.
This way, you can use Restate to process events in a lightweight, flexible, transactional way.

<Steps>
  <Step title={<span><a href={"/get_started/quickstart"}>Get the Greeter service template and run the service</a></span>}>
    You can choose any of the SDK languages for this quickstart.
    You do not need to adapt the handler code to be able to read from Kafka.
    Any handler can be connected to Kafka.
  </Step>

  <Step title="Start the Kafka cluster">
    Let's start a Kafka cluster with a single broker, and create a topic named `greetings`.

    You can run your Kafka cluster in your preferred way. Here, we will use Docker Compose to start a Kafka cluster.

    Create a `docker-compose.yaml` file with the following content:

    ```yaml docker-compose.yaml expandable theme={null}
    version: '3'
    services:
        broker:
            image: confluentinc/cp-kafka:7.5.0
            container_name: broker
            ports:
                - "9092:9092"
                - "9101:9101"
            environment:
                KAFKA_BROKER_ID: 1
                KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
                KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
                KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
                KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
                KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
                KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
                KAFKA_PROCESS_ROLES: broker,controller
                KAFKA_NODE_ID: 1
                KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker:29093
                KAFKA_LISTENERS: PLAINTEXT://broker:29092,CONTROLLER://broker:29093,PLAINTEXT_HOST://0.0.0.0:9092
                KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
                KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
                KAFKA_LOG_DIRS: /tmp/kraft-combined-logs
                CLUSTER_ID: MkU3OEVBNTcwNTJENDM2Qk

        init-kafka:
            image: confluentinc/cp-kafka:7.5.0
            depends_on:
                - broker
            entrypoint: [ '/bin/sh', '-c' ]
            command: |
                "# blocks until kafka is reachable
                kafka-topics --bootstrap-server broker:29092 --list
                echo -e 'Creating kafka topics'
                kafka-topics --bootstrap-server broker:29092 --create --if-not-exists --topic greetings --replication-factor 1 --partitions 1

                echo -e 'Successfully created the following topics:'
                kafka-topics --bootstrap-server broker:29092 --list"
    ```

    Start the Kafka broker:

    ```shell theme={null}
    docker compose up
    ```
  </Step>

  <Step title="Running Restate Server">
    Now, let's start the Restate Server and let it know about the Kafka cluster via the configuration file.

    Store the following configuration in a file named `restate.toml`:

    ```toml restate.toml theme={null}
    [[ingress.kafka-clusters]]
    name = "my-cluster"
    brokers = ["PLAINTEXT://localhost:9092"]
    ```

    Start the Restate Server from the same location as the configuration file:

    ```shell theme={null}
    restate-server --config-file restate.toml
    ```
  </Step>

  <Step title="Register the service">
    Let the Restate Server know about the Greeter service by registering it:

    ```shell theme={null}
    restate deployments register localhost:9080
    ```
  </Step>

  <Step title="Connect the handler to the topic">
    Now, we need to make Restate subscribe to the Kafka topics and tell it where it should push the events that arrive on the topic.

    Execute the following curl command to create a subscription, and invoke the handler for each event:

    ```shell theme={null}
    curl localhost:9070/subscriptions --json '{
        "source": "kafka://my-cluster/greetings",
        "sink": "service://Greeter/greet",
        "options": {"auto.offset.reset": "earliest"}
    }'
    ```

    For Go, you need to capitalize the handler name: `service://Greeter/Greet`.

    This curl command calls the Admin API of the Restate Server and tells it to invoke the `greet` handler of the `Greeter` service for each event that arrives on the `greetings` topic in the `my-cluster` Kafka cluster.
  </Step>

  <Step title="Invoke the handler by publishing an event">
    Create a Kafka producer and publish an event to the `greetings` topic:

    ```shell theme={null}
    docker exec -it broker kafka-console-producer --bootstrap-server broker:29092 --topic greetings
    ```

    Then type a message and press enter. The greeter takes a String name as an input:

    ```
    {"name":"Sarah"}
    ```

    For some greeter templates, you might need to send just the name as `"Sarah"`.
  </Step>

  <Step title="We invoked a Restate service over Kafka">
    You now see your handler getting invoked.

    The way this worked is that Restate reads the message off the Kafka topic and durably persisted it, similar to what it does for HTTP invocations.

    It then **pushed** the message to the handler, as opposed to the pull mechanism that you would have with a Kafka consumer.

    If the handler fails, Restate will retry the request until it eventually succeeds.

    When invoking Basic Services, Restates ignores the key of the message.

    When invoking Virtual Objects, Restate uses the key of the Kafka message as the Virtual Object key.
    Whereas a Kafka partition contains multiple keys, Restate effectively keeps track of a queue per key.

    Have a look at the [Event Processing use case page](/use-cases/event-processing) to learn about what you can do with Restate and event processing.
  </Step>

  <Step title="Cleanup: removing the subscription">
    You can see the subscriptions that are active via the Admin API:

    ```shell theme={null}
    curl localhost:9070/subscriptions
    ```

    Example output:

    ```json theme={null}
    {
        "subscriptions": [
            {
                "id": "sub_11XHoawrCiWtv8kzhEyGtsR",
                "source": "kafka://my-cluster/my-topic",
                "sink": "service://Greeter/greet",
                "options": {
                    "auto.offset.reset": "earliest",
                    "client.id": "restate",
                    "group.id": "sub_11XHoawrCiWtv8kzhEyGtsR"
                }
            }
        ]
    }
    ```

    As you can see, subscriptions have an ID that starts with `sub_`.

    Now you can use the subscription ID to delete the subscription:

    ```shell theme={null}
    curl -X DELETE localhost:9070/subscriptions/sub_11XHoawrCiWtv8kzhEyGtsR
    ```
  </Step>
</Steps>

## Related resources

* [Event processing use cases](/use-cases/event-processing): Have a look at examples of how Restate gives you lightweight, transactional event processing.
* Have a look at [the examples](https://github.com/restatedev/examples). There, you can find examples on Durable Execution and event processing.
* [Reference docs for invoking over Kafka](/services/invocation/kafka)
* [Blog post on Kafka integration](https://restate.dev/blog/restate--kafka-event-driven-apps-where-event-driven-is-an-implementation-detail/)


# Scaling to Multi-Node Deployments
Source: https://docs.restate.dev/guides/local-to-replicated

Migrate a single node to a multi-node cluster.

This guide shows how to scale an existing single-node deployment to a multi-node cluster. It assumes you have a running single-node Restate server that is running the replicated loglet and replicated metadata server, which are enabled by default in Restate >= v1.4.0.

Older versions of Restate ( v1.3.2) use the local loglet and local metadata server by default. The local loglet and local metadata server are suitable for development and single-node deployments. We recommend using the replicated loglet and replicated metadata server to ensure high availability and durability. They are also required for multi-node clusters. Starting with version v1.4.0, existing logs and metadata will be automatically migrated to the replicated equivalents.

<Steps>
  <Step title="Upgrade to latest Restate version">
    Make sure to [upgrade](/server/upgrading) your single-node deployment to the latest Restate version before adding more nodes.
  </Step>

  <Step title="Verify that node is running the replicated metadata server">
    Check that the metadata service is running using the [`restatectl`](/server/clusters#controlling-clusters-with-restatectl) tool.

    ```shell theme={null}
    restatectl metadata-server list-servers
    ```

    You should see a single member node providing metadata service:

    ```
    Metadata service
    NODE  STATUS  VERSION  LEADER  MEMBERS  APPLIED  COMMITTED  TERM  LOG-LENGTH  SNAP-INDEX  SNAP-SIZE
    N1    Member  v1       N1      [N1]     2        2          2     1           1           6.7 KiB
    ```

    If you see the node as unreachable with an error reason of "Unimplemented", verify that you are running the latest version. The older local metadata server is no longer supported in Restate v1.4.0 and newer.
  </Step>

  <Step title="Verify that node is running the replicated loglet">
    The default configuration is cluster-ready. However, if you have explicitly specified server roles in configuration, you should make sure these include the `log-server` role. Similarly, if you have explicitly set the loglet provider to be `local`, you should remove this. While the local loglet is still supported, the default type is `replicated` starting from v1.4.0. If you have a configuration file and would like to make these settings explicit, it should contain the following:

    ```toml restate.toml theme={null}
    roles = [
    "worker",
    "admin",
    "metadata-server",
    "log-server",                # needed for replicated loglet
    "http-ingress",
    ]

    [bifrost]
    default-provider = "replicated"  # default
    ```

    Confirm that cluster configuration uses the replicated loglet as the default log provider.

    ```shell theme={null}
    restatectl config get
    ```

    In the default configuration you should expect to see:

    ```
    ⚙️ Cluster Configuration
    ├ Number of partitions: 24
    ├ Partition replication: {node: 1}
    └ Logs Provider: replicated
    ├ Log replication: {node: 1}
    └ Nodeset size: 1
    ```

    You can confirm that the type of logs in use by the server using the command:

    ```shell theme={null}
    restatectl logs list
    ```

    If you have enabled the `log-server` role and left the default provider unset (or set it to `replicated`), and still do not see the cluster configuration you can change the cluster log configuration using:

    ```shell theme={null}
    restatectl config set --log-provider replicated --log-replication 1
    ```

    This command sets the default log provider to `replicated` with a default replication of `1`.
    As long as you have a single-node deployment, you must set the replication to `1`.
    Otherwise, the server will become unavailable because it cannot provision the new log segments.
  </Step>

  <Step title="Configure snapshot repository">
    If you plan to extend your single-node deployment to a multi-node deployment, you also need to [configure the snapshot repository](/server/snapshots#configuring-automatic-snapshotting).
    This allows new nodes to join the cluster by restoring the latest snapshot.

    ```toml restate.toml theme={null}
    [worker.snapshots]
    destination = "s3://snapshots-bucket/cluster-prefix"
    ```
  </Step>

  <Step title="Create snapshots to allow other nodes to join">
    For other nodes to join, you need to snapshot every partition because the local loglet is not accessible from other nodes.
    Run the following command to create a snapshot for each partition.

    ```shell theme={null}
    restatectl snapshots create --trim-log
    ```

    Note that this also instructs Restate to trim the logs after partition state has been successfully published to the snapshot repository. This ensures that the logs no longer reference historic local loglets that may have existed on the original node.
  </Step>

  <Step title="Turn a single-node into a multi-node deployment">
    To add more nodes to your cluster, you need to start new Restate servers with the same `cluster-name` and configure the metadata client with the address of at least one existing node running the metadata-server role.

    ```toml restate.toml theme={null}
    cluster-name = "my-cluster"
    auto-provision = false

    [metadata-client]
    # Point to at least one peer running metadata-server role
    addresses = ["http://metadata-node.cluster:5122"]
    ```

    <Info>
      Nodes automatically include themselves if they run the metadata-server role, so you only need to list peer addresses.
    </Info>

    <Warning>
      Newly added nodes must set `auto-provision = false` to prevent them from starting a new cluster instead of joining the existing one.
    </Warning>

    Metadata is critical to the operation of your cluster and we recommend that you run the `metadata-server` role on additional nodes. Make the cluster metadata service resilient to node failures by specifying the full list of metadata servers on all cluster nodes.

    ```toml restate.toml theme={null}
    [metadata-client]
    addresses = ["http://node1.cluster:5122", "http://node2.cluster:5122", "http://node3.cluster:5122"]
    ```
  </Step>

  <Step title="Verify that your cluster consists of multiple nodes">
    If everything is set up correctly, you should see the new nodes in the cluster status.

    ```shell theme={null}
    restatectl status
    ```
  </Step>

  <Step title="🎉 Congratulations, you migrated your single-node deployment to a multi-node deployment!" />
</Steps>

## See also

* Try [growing your cluster](/server/clusters#growing-the-cluster) to tolerate node failures


# Parallelizing Work
Source: https://docs.restate.dev/guides/parallelizing-work

Execute a list of tasks in parallel and then gather their result.

This guide shows how to use the Restate to **execute a list of tasks in parallel and then gather their result**, also known as fan-out, fan-in.

<img />

## How does Restate help?

* Restate lets you schedule the tasks asynchronously and guarantees that all tasks will run, with **retries and recovery** on failures.
* Restate **turns Promises/Futures into durable, distributed constructs** that are persisted in Restate and can be recovered and awaited on another process.
* You can deploy the subtask executors on **serverless** infrastructure, like AWS Lambda, to let them scale automatically. The main task, that is idle while waiting on the subtasks, gets suspended until it can make progress.

**Fan out**: You can fan out tasks with Restate by creating a handler that processes a single subtask,
and then scheduling it repeatedly from another handler.

**Fan in**: You can fan in the results of the subtasks by using Restate's Promise Combinators to wait for all promises to resolve.

## Example

The example implements a worker service:

1. It splits a task into subtasks.
2. It schedules all the subtasks. Each subtask results in a promise that gets added to a list.
3. The result is gathered by waiting for all promises to resolve.

You can run this on FaaS infrastructure, like AWS Lambda, and it will scale automatically.
The `run` handler will then suspend while it waits for all subtasks to finish.
Restate will then resume the handler when all subtasks are done.

<CodeGroup>
  ```ts TypeScript {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/patterns-use-cases/src/parallelizework/fan_out_worker.ts?collapse_prequel"}  theme={null}
  const fanOutWorker = restate.service({
    name: "worker",
    handlers: {
      run: async (ctx: Context, task: Task): Promise<Result> => {
        // Split the task in subtasks
        const subtasks: SubTask[] = await ctx.run("split task", () => split(task));

        // Fan out the subtasks - run them in parallel
        const resultPromises = [];
        for (const subtask of subtasks) {
          const subResultPromise = ctx.serviceClient(fanOutWorker).runSubtask(subtask);
          resultPromises.push(subResultPromise);
        }

        // Fan in - Aggregate the results
        const results = await RestatePromise.all(resultPromises);
        return aggregate(ctx, results);
      },

      // Can also run on FaaS
      runSubtask: async (ctx: Context, subtask: SubTask): Promise<SubTaskResult> => {
        // Processing logic goes here ...
        // Can be moved to a separate service to scale independently
        return executeSubtask(ctx, subtask);
      },
    },
  });

  restate.serve({
    services: [fanOutWorker],
    port: 9080,
  });
  ```

  ```java Java {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/patterns-use-cases/src/main/java/my/example/parallelizework/FanOutWorker.java?collapse_prequel"}  theme={null}
  @Service
  public class FanOutWorker {

    @Handler
    public Result run(Context ctx, Task task) {
      // Split the task in subtasks
      var subTasks = ctx.run(new TypeRef<>() {}, () -> split(task));

      // Fan out the subtasks - run them in parallel
      List<DurableFuture<?>> resultFutures = new ArrayList<>();
      for (SubTask subTask : subTasks) {
        resultFutures.add(FanOutWorkerClient.fromContext(ctx).runSubtask(subTask));
      }

      DurableFuture.all(resultFutures).await();

      // Fan in - Aggregate the results
      var results = resultFutures.stream().map(future -> (SubTaskResult) future.await()).toList();
      return aggregate(results);
    }

    // Can also run on FaaS
    @Handler
    public SubTaskResult runSubtask(Context ctx, SubTask subTask) {
      // Processing logic goes here ...
      // Can be moved to a separate service to scale independently
      return executeSubtask(ctx, subTask);
    }

    public static void main(String[] args) {
      RestateHttpServer.listen(Endpoint.bind(new FanOutWorker()));
    }
  }
  ```

  ```kotlin Kotlin {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/kotlin/patterns-use-cases/src/main/kotlin/my/example/parallelizework/FanOutWorker.kt?collapse_prequel"}  theme={null}
  @Service
  class FanOutWorker {
    @Handler
    suspend fun run(ctx: Context, task: Task): TaskResult {
      // Split the task in subtasks
      val subTasks = ctx.runBlock { task.split() }

      // Fan out the subtasks - run them in parallel
      // Fan in - Await all results and aggregate
      val results = subTasks.map { FanOutWorkerClient.fromContext(ctx).runSubtask(it) }.awaitAll()

      return results.aggregate()
    }

    @Handler
    suspend fun runSubtask(ctx: Context, subTask: SubTask): SubTaskResult {
      // Processing logic goes here ...
      // Can be moved to a separate service to scale independently
      return subTask.execute(ctx)
    }
  }

  fun main() {
    RestateHttpServer.listen(endpoint { bind(FanOutWorker()) })
  }
  ```

  ```python Python {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/python/patterns-use-cases/parallelizework/app.py?collapse_prequel"}  theme={null}
  fan_out_worker = restate.Service("FanOutWorker")


  @fan_out_worker.handler()
  async def run(ctx: restate.Context, task: Task) -> Result:
      # Split the task in subtasks
      subtasks = await ctx.run_typed("split task", split, task=task)

      # Fan out the subtasks - run them in parallel
      result_promises = [
          ctx.run_typed(f"execute {subtask}", execute_subtask, subtask=subtask)
          for subtask in subtasks.subtasks
      ]

      # Fan in - Aggregate the results
      results_done = await restate.gather(*result_promises)
      results = [await result for result in results_done]

      return aggregate(results)


  app = restate.app([fan_out_worker])

  if __name__ == "__main__":
      import hypercorn
      import asyncio

      conf = hypercorn.Config()
      conf.bind = ["0.0.0.0:9080"]
      asyncio.run(hypercorn.asyncio.serve(app, conf))
  ```

  ```go Go {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/patterns-use-cases/src/parallelizework/fanoutworker.go?collapse_prequel"}  theme={null}
  type FanOutWorker struct{}

  func (FanOutWorker) Run(ctx restate.Context, task Task) (Result, error) {
    // Split the task into subtasks
    subtasks, err := split(task)
    if err != nil {
      return Result{}, err
    }

    // Fan out the subtasks - run them in parallel
    subtaskFutures := make([]restate.Future, 0, len(subtasks))
    for _, subtask := range subtasks {
      subtaskFutures = append(subtaskFutures,
        restate.Service[SubTaskResult](ctx, "FanOutWorker", "RunSubtask").RequestFuture(subtask))
    }

    // Fan in - Aggregate the results
    subResults := make([]SubTaskResult, 0, len(subtasks))
    for fut, err := range restate.Wait(ctx, subtaskFutures...) {
      if err != nil {
        return Result{}, err
      }
      response, err := fut.(restate.ResponseFuture[SubTaskResult]).Response()
      if err != nil {
        return Result{}, err
      }
      subResults = append(subResults, response)
    }

    // Fan in - Aggregate the results
    return aggregate(subResults)
  }

  // RunSubtask can also run on FaaS
  func (FanOutWorker) RunSubtask(ctx restate.Context, subtask SubTask) (SubTaskResult, error) {
    // Processing logic goes here ...
    // Can be moved to a separate service to scale independently
    return executeSubtask(ctx, subtask)
  }

  func main() {
    server := server.NewRestate().
      Bind(restate.Reflect(FanOutWorker{}))

    if err := server.Start(context.Background(), ":9080"); err != nil {
      slog.Error("application exited unexpectedly", "err", err.Error())
      os.Exit(1)
    }
  }
  ```
</CodeGroup>

In this example, we parallelize RPC calls, but this can also be used to parallelize `ctx.run` actions.

<Info>
  This pattern is implementable with any of our SDKs. We are still working on translating all patterns to all SDK languages.
  If you need help with a specific language, please reach out to us via [Discord](https://discord.restate.dev) or [Slack](https://slack.restate.dev).
</Info>

## Running the example

<Steps>
  <Step title="Download the example">
    <CodeGroup>
      ```shell TypeScript theme={null}
      restate example typescript-patterns-use-cases && cd typescript-patterns-use-cases
      ```

      ```shell Java theme={null}
      restate example java-patterns-use-cases && cd java-patterns-use-cases
      ```

      ```shell Kotlin theme={null}
      restate example kotlin-patterns-use-cases && cd kotlin-patterns-use-cases
      ```

      ```shell Python theme={null}
      restate example python-patterns-use-cases && cd python-patterns-use-cases
      ```

      ```shell Go theme={null}
      restate example go-patterns-use-cases && cd go-patterns-use-cases
      ```
    </CodeGroup>
  </Step>

  <Step title="Start the Restate Server">
    ```shell theme={null}
    restate-server
    ```
  </Step>

  <Step title="Start the Service">
    <CodeGroup>
      ```shell TypeScript theme={null}
      npm install
      npx tsx watch ./src/parallelizework/fan_out_worker.ts
      ```

      ```shell Java theme={null}
      ./gradlew -PmainClass=my.example.parallelizework.FanOutWorker run
      ```

      ```shell Kotlin theme={null}
      ./gradlew -PmainClass=my.example.parallelizework.FanOutWorkerKt run
      ```

      ```shell Python theme={null}
      uv run parallelizework/app.py
      ```

      ```shell Go theme={null}
      go run ./src/parallelizework
      ```
    </CodeGroup>
  </Step>

  <Step title="Register the services">
    ```shell theme={null}
    restate deployments register localhost:9080
    ```
  </Step>

  <Step title="Send a request">
    <CodeGroup>
      ```shell TypeScript theme={null}
      curl localhost:8080/worker/run \
          --json '{"description": "get out of bed,shower,make coffee,have breakfast"}'
      ```

      ```shell Java theme={null}
      curl localhost:8080/FanOutWorker/run \
          --json '{"description": "get out of bed,shower,make coffee,have breakfast"}'
      ```

      ```shell Kotlin theme={null}
      curl localhost:8080/FanOutWorker/run \
          --json '{"description": "get out of bed,shower,make coffee,have breakfast"}'
      ```

      ```shell Python theme={null}
      curl localhost:8080/FanOutWorker/run \
          --json '{"description": "get out of bed,shower,make coffee,have breakfast"}'
      ```

      ```shell Go theme={null}
      curl localhost:8080/FanOutWorker/Run \
          --json '{"description": "get out of bed,shower,make coffee,have breakfast"}'
      ```
    </CodeGroup>
  </Step>

  <Step title="Check the service logs">
    See how all tasks get spawned in parallel, finish at different times, and then get aggregated.

    <CodeGroup>
      ```shell TypeScript theme={null}
      [restate] [worker/runSubtask][inv_17jBqoqRG0TN3msVqHEpZn2aQMOX5kSKrf][2025-01-17T08:51:44.993Z] INFO:  Started executing subtask: get out of bed
      [restate] [worker/runSubtask][inv_1f8R1NuF0LF27EdQ0R6s7PR8hld245OM8h][2025-01-17T08:51:44.995Z] INFO:  Started executing subtask: shower
      [restate] [worker/runSubtask][inv_101oPhGwxQqZ0sQebkQnpGyV9Rp3oj9CSJ][2025-01-17T08:51:44.997Z] INFO:  Started executing subtask: make coffee
      [restate] [worker/runSubtask][inv_1eKDShaxMCEB6DXasrR5OtRXJEvA2je33X][2025-01-17T08:51:44.998Z] INFO:  Started executing subtask: have breakfast
      [restate] [worker/runSubtask][inv_17jBqoqRG0TN3msVqHEpZn2aQMOX5kSKrf][2025-01-17T08:51:47.003Z] INFO:  Execution subtask finished: get out of bed
      [restate] [worker/runSubtask][inv_101oPhGwxQqZ0sQebkQnpGyV9Rp3oj9CSJ][2025-01-17T08:51:48.007Z] INFO:  Execution subtask finished: make coffee
      [restate] [worker/runSubtask][inv_1f8R1NuF0LF27EdQ0R6s7PR8hld245OM8h][2025-01-17T08:51:48.999Z] INFO:  Execution subtask finished: shower
      [restate] [worker/runSubtask][inv_1eKDShaxMCEB6DXasrR5OtRXJEvA2je33X][2025-01-17T08:51:49.001Z] INFO:  Execution subtask finished: have breakfast
      [restate] [worker/run][inv_18QHSeAYfvim1oNXRl9I5105veQcTW3BEl][2025-01-17T08:51:49.007Z] INFO:  Aggregated result: get out of bed: DONE,shower: DONE,make coffee: DONE,have breakfast: DONE
      ```

      ```shell Java theme={null}
      2025-01-17 10:00:58 INFO  [FanOutWorker/run][inv_1jNoSMJtWluo4Ir43OUyDAxD9weMAQ4OeR] dev.restate.sdk.core.InvocationStateMachine - Start invocation
      2025-01-17 10:00:58 INFO  [FanOutWorker/runSubtask][inv_1kdpBvVXdqyo3saU6KThul6Jgkfot6LcRP] dev.restate.sdk.core.InvocationStateMachine - Start invocation
      2025-01-17 10:00:58 INFO  [FanOutWorker/runSubtask][inv_1kdpBvVXdqyo3saU6KThul6Jgkfot6LcRP] my.example.parallelizework.utils.Utils - Started executing subtask: get out of bed
      2025-01-17 10:00:58 INFO  [FanOutWorker/runSubtask][inv_162MCD5ertQ65pdG0uDIYRMgLBFYZkNPnb] dev.restate.sdk.core.InvocationStateMachine - Start invocation
      2025-01-17 10:00:58 INFO  [FanOutWorker/runSubtask][inv_162MCD5ertQ65pdG0uDIYRMgLBFYZkNPnb] my.example.parallelizework.utils.Utils - Started executing subtask: shower
      2025-01-17 10:00:58 INFO  [FanOutWorker/runSubtask][inv_10bPiFTjBUXX35qtzOTPNr0vfgoYYVehpf] dev.restate.sdk.core.InvocationStateMachine - Start invocation
      2025-01-17 10:00:58 INFO  [FanOutWorker/runSubtask][inv_10bPiFTjBUXX35qtzOTPNr0vfgoYYVehpf] my.example.parallelizework.utils.Utils - Started executing subtask: make coffee
      2025-01-17 10:00:58 INFO  [FanOutWorker/runSubtask][inv_1115lzidXq7M7CtLZn0aEyUvMC4zkXYLWF] dev.restate.sdk.core.InvocationStateMachine - Start invocation
      2025-01-17 10:00:58 INFO  [FanOutWorker/runSubtask][inv_1115lzidXq7M7CtLZn0aEyUvMC4zkXYLWF] my.example.parallelizework.utils.Utils - Started executing subtask: have breakfast
      2025-01-17 10:00:59 INFO  [FanOutWorker/runSubtask][inv_162MCD5ertQ65pdG0uDIYRMgLBFYZkNPnb] my.example.parallelizework.utils.Utils - Execution subtask finished: shower
      2025-01-17 10:00:59 INFO  [FanOutWorker/runSubtask][inv_162MCD5ertQ65pdG0uDIYRMgLBFYZkNPnb] dev.restate.sdk.core.InvocationStateMachine - End invocation
      2025-01-17 10:01:00 INFO  [FanOutWorker/runSubtask][inv_1kdpBvVXdqyo3saU6KThul6Jgkfot6LcRP] my.example.parallelizework.utils.Utils - Execution subtask finished: get out of bed
      2025-01-17 10:01:00 INFO  [FanOutWorker/runSubtask][inv_1kdpBvVXdqyo3saU6KThul6Jgkfot6LcRP] dev.restate.sdk.core.InvocationStateMachine - End invocation
      2025-01-17 10:01:04 INFO  [FanOutWorker/runSubtask][inv_10bPiFTjBUXX35qtzOTPNr0vfgoYYVehpf] my.example.parallelizework.utils.Utils - Execution subtask finished: make coffee
      2025-01-17 10:01:04 INFO  [FanOutWorker/runSubtask][inv_10bPiFTjBUXX35qtzOTPNr0vfgoYYVehpf] dev.restate.sdk.core.InvocationStateMachine - End invocation
      2025-01-17 10:01:05 INFO  [FanOutWorker/runSubtask][inv_1115lzidXq7M7CtLZn0aEyUvMC4zkXYLWF] my.example.parallelizework.utils.Utils - Execution subtask finished: have breakfast
      2025-01-17 10:01:05 INFO  [FanOutWorker/runSubtask][inv_1115lzidXq7M7CtLZn0aEyUvMC4zkXYLWF] dev.restate.sdk.core.InvocationStateMachine - End invocation
      2025-01-17 10:01:05 INFO  [FanOutWorker/run][inv_1jNoSMJtWluo4Ir43OUyDAxD9weMAQ4OeR] my.example.parallelizework.utils.Utils - Aggregated result: get out of bed: DONE, shower: DONE, make coffee: DONE, have breakfast: DONE
      2025-01-17 10:01:05 INFO  [FanOutWorker/run][inv_1jNoSMJtWluo4Ir43OUyDAxD9weMAQ4OeR] dev.restate.sdk.core.InvocationStateMachine - End invocation
      ```

      ```shell Kotlin theme={null}
      2025-03-06 12:20:18 INFO  [FanOutWorker/run][inv_1fGEUyfogPKK5cbSCSWzpCkcDpkQIKSzMB] dev.restate.sdk.core.InvocationStateMachine - Start invocation
      2025-03-06 12:20:18 INFO  [FanOutWorker/runSubtask][inv_146fBfVLISKb2sCWqesf6uMReXdRKvqmv7] dev.restate.sdk.core.InvocationStateMachine - Start invocation
      2025-03-06 12:20:18 INFO  [FanOutWorker/runSubtask][inv_146fBfVLISKb2sCWqesf6uMReXdRKvqmv7] FanOutWorker - Started executing subtask: get out of bed
      2025-03-06 12:20:18 INFO  [FanOutWorker/runSubtask][inv_18T9WW6paOhm6eciCeBt5iqHXRY4h2NRvP] dev.restate.sdk.core.InvocationStateMachine - Start invocation
      2025-03-06 12:20:18 INFO  [FanOutWorker/runSubtask][inv_18T9WW6paOhm6eciCeBt5iqHXRY4h2NRvP] FanOutWorker - Started executing subtask: shower
      2025-03-06 12:20:18 INFO  [FanOutWorker/runSubtask][inv_10kE3b5UcL8L64ghFpHQjeeAEowKNis4dH] dev.restate.sdk.core.InvocationStateMachine - Start invocation
      2025-03-06 12:20:18 INFO  [FanOutWorker/runSubtask][inv_10kE3b5UcL8L64ghFpHQjeeAEowKNis4dH] FanOutWorker - Started executing subtask: make coffee
      2025-03-06 12:20:18 INFO  [FanOutWorker/runSubtask][inv_1fCFmQ9ulbxL2MBwYCDRgV8Was8PDBedW1] dev.restate.sdk.core.InvocationStateMachine - Start invocation
      2025-03-06 12:20:18 INFO  [FanOutWorker/runSubtask][inv_1fCFmQ9ulbxL2MBwYCDRgV8Was8PDBedW1] FanOutWorker - Started executing subtask: have breakfast
      2025-03-06 12:20:21 INFO  [FanOutWorker/runSubtask][inv_10kE3b5UcL8L64ghFpHQjeeAEowKNis4dH] FanOutWorker - Execution subtask finished: make coffee
      2025-03-06 12:20:21 INFO  [FanOutWorker/runSubtask][inv_10kE3b5UcL8L64ghFpHQjeeAEowKNis4dH] dev.restate.sdk.core.InvocationStateMachine - End invocation
      2025-03-06 12:20:24 INFO  [FanOutWorker/runSubtask][inv_146fBfVLISKb2sCWqesf6uMReXdRKvqmv7] FanOutWorker - Execution subtask finished: get out of bed
      2025-03-06 12:20:24 INFO  [FanOutWorker/runSubtask][inv_146fBfVLISKb2sCWqesf6uMReXdRKvqmv7] dev.restate.sdk.core.InvocationStateMachine - End invocation
      2025-03-06 12:20:25 INFO  [FanOutWorker/runSubtask][inv_1fCFmQ9ulbxL2MBwYCDRgV8Was8PDBedW1] FanOutWorker - Execution subtask finished: have breakfast
      2025-03-06 12:20:25 INFO  [FanOutWorker/runSubtask][inv_1fCFmQ9ulbxL2MBwYCDRgV8Was8PDBedW1] dev.restate.sdk.core.InvocationStateMachine - End invocation
      2025-03-06 12:20:27 INFO  [FanOutWorker/runSubtask][inv_18T9WW6paOhm6eciCeBt5iqHXRY4h2NRvP] FanOutWorker - Execution subtask finished: shower
      2025-03-06 12:20:27 INFO  [FanOutWorker/runSubtask][inv_18T9WW6paOhm6eciCeBt5iqHXRY4h2NRvP] dev.restate.sdk.core.InvocationStateMachine - End invocation
      2025-03-06 12:20:27 INFO  [FanOutWorker/run][inv_1fGEUyfogPKK5cbSCSWzpCkcDpkQIKSzMB] FanOutWorker - Aggregated result: get out of bed: DONE, shower: DONE, make coffee: DONE, have breakfast: DONE
      2025-03-06 12:20:27 INFO  [FanOutWorker/run][inv_1fGEUyfogPKK5cbSCSWzpCkcDpkQIKSzMB] dev.restate.sdk.core.InvocationStateMachine - End invocation
      ```

      ```shell Python theme={null}
      [2025-01-17 12:00:05,183] [12245] [INFO] - Started executing subtask: get out of bed
      [2025-01-17 12:00:05,184] [12247] [INFO] - Started executing subtask: shower
      [2025-01-17 12:00:05,184] [12245] [INFO] - Started executing subtask: make coffee
      [2025-01-17 12:00:05,185] [12245] [INFO] - Started executing subtask: have breakfast
      [2025-01-17 12:00:05,188] [12245] [INFO] - Execution subtask finished: make coffee
      [2025-01-17 12:00:08,193] [12245] [INFO] - Execution subtask finished: get out of bed
      [2025-01-17 12:00:10,194] [12247] [INFO] - Execution subtask finished: shower
      [2025-01-17 12:00:15,196] [12245] [INFO] - Execution subtask finished: have breakfast
      [2025-01-17 12:00:15,198] [12245] [INFO] - Aggregated result: get out of bed: DONE,shower: DONE,make coffee: DONE,have breakfast: DONE
      ```

      ```shell Go theme={null}
      2025/01/16 16:41:22 INFO Handling invocation method=FanOutWorker/Run invocationID=inv_1lkcVTBmCorR3fSPhE0pNTiO8XFXoV34C5
      2025/01/16 16:41:22 INFO Handling invocation method=FanOutWorker/RunSubtask invocationID=inv_1jpZWOrDK45b2ZwWCapl68GgXzNfoOh0BP
      2025/01/16 16:41:22 Started executing subtask: get out of bed
      2025/01/16 16:41:22 INFO Handling invocation method=FanOutWorker/RunSubtask invocationID=inv_10eVGnmjP1ET4PgI3z82rvXcVlCnSOep3P
      2025/01/16 16:41:22 Started executing subtask: shower
      2025/01/16 16:41:22 INFO Handling invocation method=FanOutWorker/RunSubtask invocationID=inv_1i3RduoDMNnb4ideAtWaWCLRDaIO62eghP
      2025/01/16 16:41:22 Started executing subtask: make coffee
      2025/01/16 16:41:22 INFO Handling invocation method=FanOutWorker/RunSubtask invocationID=inv_142WnXnWDxfy6k4JanZ7DQVqAL6zmktuxP
      2025/01/16 16:41:22 Started executing subtask: have breakfast
      2025/01/16 16:41:24 Execution subtask finished: get out of bed
      2025/01/16 16:41:24 INFO Invocation completed successfully method=FanOutWorker/RunSubtask invocationID=inv_1jpZWOrDK45b2ZwWCapl68GgXzNfoOh0BP
      2025/01/16 16:41:25 Execution subtask finished: shower
      2025/01/16 16:41:25 INFO Invocation completed successfully method=FanOutWorker/RunSubtask invocationID=inv_10eVGnmjP1ET4PgI3z82rvXcVlCnSOep3P
      2025/01/16 16:41:25 Execution subtask finished: have breakfast
      2025/01/16 16:41:25 INFO Invocation completed successfully method=FanOutWorker/RunSubtask invocationID=inv_142WnXnWDxfy6k4JanZ7DQVqAL6zmktuxP
      2025/01/16 16:41:26 Execution subtask finished: make coffee
      2025/01/16 16:41:26 INFO Invocation completed successfully method=FanOutWorker/RunSubtask invocationID=inv_1i3RduoDMNnb4ideAtWaWCLRDaIO62eghP
      2025/01/16 16:41:26 Aggregated result: get out of bed: DONE,shower: DONE,have breakfast: DONE,make coffee: DONE
      2025/01/16 16:41:26 INFO Invocation completed successfully method=FanOutWorker/Run invocationID=inv_1lkcVTBmCorR3fSPhE0pNTiO8XFXoV34C5
      ```
    </CodeGroup>
  </Step>
</Steps>

## Related resources

* Concurrent tasks docs: [TS](/develop/ts/concurrent-tasks) /
  [Java/Kotlin](/develop/java/concurrent-tasks) /
  [Python](/develop/python/concurrent-tasks) /
  [Go](/develop/go/concurrent-tasks) /
  [Rust](https://docs.rs/restate-sdk/latest/restate_sdk/macro.select.html)


# Rate Limiting
Source: https://docs.restate.dev/guides/rate-limiting

Control request rates and prevent service overload with Restate

Rate limiting is a technique used to control the number of requests or operations that a service can handle within a specific time period. It helps prevent system overload and ensures fair resource usage.

## How does Restate help?

Restate provides several features that make it well-suited for implementing rate limiting:

* **Durable state**: Store and manage rate limit counters reliably.
* **Virtual objects**: Isolated rate limiters per key (user, API endpoint, etc.).
* **Durable timers**: Schedule token refills and cleanup operations.

Restate doesn't have built-in rate limiting functionality, but its building blocks make it easy to build this.

## Example

This implementation provides a token bucket rate limiter that can control the rate of operations for any service or resource. You can copy the following files to your project:

The **limiter client interface**, which you can use in your services:

<CodeGroup>
  ```ts TypeScript expandable {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/patterns-use-cases/src/ratelimit/limiter_client.ts"}  theme={null}
  import { Context, TerminalError } from "@restatedev/restate-sdk";
  import type { Limiter as LimiterObject, Reservation as ReservationResponse } from "./limiter";

  export interface Reservation extends ReservationResponse {
    // cancel indicates that the reservation holder will not perform the reserved action
    // and reverses the effects of this Reservation on the rate limit as much as possible,
    // considering that other reservations may have already been made.
    cancel(): void;
  }

  export interface Limiter {
    // limit returns the maximum overall event rate.
    limit(): Promise<number>;
    // burst returns the maximum burst size. Burst is the maximum number of tokens
    // that can be consumed in a single call to allow, reserve, or wait, so higher
    // Burst values allow more events to happen at once.
    // A zero Burst allows no events, unless limit == Inf.
    burst(): Promise<number>;
    // tokens returns the number of tokens available at time t (defaults to now).
    tokens(): Promise<number>;
    // allow reports whether n events may happen at time t.
    // Use this method if you intend to drop / skip events that exceed the rate limit.
    // Otherwise use reserve or wait.
    allow(n?: number): Promise<boolean>;
    // reserve returns a Reservation that indicates how long the caller must wait before n events happen.
    // The limiter takes this Reservation into account when allowing future events.
    // The returned Reservation’s ok parameter is false if n exceeds the limiter's burst size, or provided waitLimitMillis.
    // Use this method if you wish to wait and slow down in accordance with the rate limit without dropping events.
    // If you need to cancel the delay, use wait instead.
    // To drop or skip events exceeding rate limit, use allow instead.
    reserve(n?: number, waitLimitMillis?: number): Promise<Reservation>;
    // setLimit sets a new limit for the limiter. The new limit, and burst, may be violated
    // or underutilized by those which reserved (using reserve or wait) but did not yet act
    // before setLimit was called.
    setLimit(newLimit: number): Promise<void>;
    // setBurst sets a new burst size for the limiter.
    setBurst(newBurst: number): Promise<void>;
    // setRate sets a new limit and burst size for the limiter.
    setRate(newLimit: number, newBurst: number): Promise<void>;
    // waitN blocks until the limiter permits n events to happen.
    // It returns an error if n exceeds the limiter's burst size, the invocation is canceled,
    // or the wait would be longer than the deadline.
    // The burst limit is ignored if the rate limit is Inf.
    wait(n?: number, waitLimitMillis?: number): Promise<void>;
  }

  export namespace Limiter {
    export function fromContext(ctx: Context, limiterID: string): Limiter {
      const client = ctx.objectClient<LimiterObject>({ name: "limiter" }, limiterID);
      return {
        async limit() {
          return (await client.state()).limit;
        },
        async burst() {
          return (await client.state()).burst;
        },
        async tokens() {
          return client.tokens();
        },
        async allow(n?: number) {
          const r = await client.reserve({
            n,
            waitLimitMillis: 0,
          });
          return r.ok;
        },
        async reserve(n?: number, waitLimitMillis?: number) {
          const r = await client.reserve({
            n,
            waitLimitMillis,
          });
          return {
            cancel() {
              ctx
                .objectSendClient<LimiterObject>({ name: "limiter" }, limiterID)
                .cancelReservation(r);
            },
            ...r,
          };
        },
        async setLimit(newLimit: number) {
          return client.setRate({
            newLimit,
          });
        },
        async setBurst(newBurst: number) {
          return client.setRate({
            newBurst,
          });
        },
        async setRate(newLimit: number, newBurst: number) {
          return client.setRate({
            newLimit,
            newBurst,
          });
        },
        async wait(n: number = 1, waitLimitMillis?: number) {
          // Reserve
          const r = await this.reserve(n, waitLimitMillis);
          if (!r.ok) {
            if (waitLimitMillis === undefined) {
              throw new TerminalError(`rate: Wait(n=${n}) would exceed the limiters burst`, {
                errorCode: 429,
              });
            } else {
              throw new TerminalError(
                `rate: Wait(n=${n}) would either exceed the limiters burst or the provided waitLimitMillis`,
                { errorCode: 429 },
              );
            }
          }
          // Wait if necessary
          const delay = delayFrom(r, r.creationDate);
          if (delay == 0) {
            return;
          }

          try {
            await ctx.sleep(delay);
          } catch (e) {
            // this only happens on invocation cancellation - cancel the reservation in the background
            r.cancel();
            throw e;
          }
        },
      };
    }
  }

  // delayFrom returns the duration in millis for which the reservation holder must wait
  // before taking the reserved action.  Zero duration means act immediately.
  // Infinity means the limiter cannot grant the tokens requested in this
  // Reservation within the maximum wait time.
  function delayFrom(r: ReservationResponse, date: number): number {
    if (!r.ok) {
      return Infinity;
    }
    const delay = r.dateToAct - date;
    if (delay < 0) {
      return 0;
    }
    return Math.floor(delay);
  }
  ```

  ```go Go expandable {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/patterns-use-cases/src/ratelimit/client/limitclient.go"}  theme={null}
  package client

  import (
    "time"

    "github.com/restatedev/examples/go/patterns-use-cases/src/ratelimit/types"
    restate "github.com/restatedev/sdk-go"
  )

  type Limiter struct {
    ctx       restate.Context
    limiterID string
  }

  func NewLimiter(ctx restate.Context, limiterID string) *Limiter {
    return &Limiter{
      ctx:       ctx,
      limiterID: limiterID,
    }
  }

  func (lim *Limiter) state() (types.LimiterState, error) {
    return restate.Object[types.LimiterState](lim.ctx, "Limiter", lim.limiterID, "State").Request(restate.Void{})
  }

  // Limit returns the maximum overall event rate.
  func (lim *Limiter) Limit() (types.Limit, error) {
    state, err := lim.state()
    if err != nil {
      return 0.0, err
    }
    return state.Limit, nil
  }

  // Burst returns the maximum burst size. Burst is the maximum number of tokens
  // that can be consumed in a single call to Allow, Reserve, or Wait, so higher
  // Burst values allow more events to happen at once.
  // A zero Burst allows no events, unless limit == Inf.
  func (lim *Limiter) Burst() (int, error) {
    state, err := lim.state()
    if err != nil {
      return 0, err
    }
    return state.Burst, nil
  }

  // Tokens returns the number of tokens available now.
  func (lim *Limiter) Tokens() (float64, error) {
    return restate.Object[float64](lim.ctx, "Limiter", lim.limiterID, "Tokens").Request(restate.Void{})
  }

  // Allow reports whether an event may happen now.
  func (lim *Limiter) Allow() (bool, error) {
    return lim.AllowN(1)
  }

  // AllowN reports whether n events may happen now.
  // Use this method if you intend to drop / skip events that exceed the rate limit.
  // Otherwise use Reserve or Wait.
  func (lim *Limiter) AllowN(n int) (bool, error) {
    r, err := restate.Object[types.Reservation](lim.ctx, "Limiter", lim.limiterID, "ReserveN").Request(types.ReserveRequest{
      N:                n,
      MaxFutureReserve: 0,
    })
    if err != nil {
      return false, err
    }
    return r.Ok, nil
  }

  type Reservation struct {
    lim *Limiter
    r   types.Reservation
  }

  // Cancel indicates that the reservation holder will not perform the reserved action
  // and reverses the effects of this Reservation on the rate limit as much as possible,
  // considering that other reservations may have already been made.
  func (r *Reservation) Cancel() {
    restate.ObjectSend(r.lim.ctx, "Limiter", r.lim.limiterID, "CancelReservation").Send(r.r)
  }

  // Reserve is shorthand for ReserveN(1).
  func (lim *Limiter) Reserve() (*Reservation, error) {
    return lim.ReserveN(1)
  }

  // ReserveN returns a Reservation that indicates how long the caller must wait before n events happen.
  // The Limiter takes this Reservation into account when allowing future events.
  // The returned Reservation’s OK() method returns false if n exceeds the Limiter's burst size.
  // Usage example:
  //
  //  r := lim.ReserveN(1)
  //  if !r.OK() {
  //    // Not allowed to act! Did you remember to set lim.burst to be > 0 ?
  //    return
  //  }
  //  restate.Sleep(r.Delay())
  //  Act()
  //
  // Use this method if you wish to wait and slow down in accordance with the rate limit without dropping events.
  // If you need to respect a deadline or cancel the delay, use Wait instead.
  // To drop or skip events exceeding rate limit, use Allow instead.
  func (lim *Limiter) ReserveN(n int) (*Reservation, error) {
    return lim.reserveN(n, types.InfDuration)
  }

  func (lim *Limiter) reserveN(n int, maxFutureReserve time.Duration) (*Reservation, error) {
    r, err := restate.Object[types.Reservation](lim.ctx, "Limiter", lim.limiterID, "ReserveN").Request(types.ReserveRequest{
      N:                n,
      MaxFutureReserve: maxFutureReserve,
    })
    if err != nil {
      return nil, err
    }

    return &Reservation{
      lim: lim,
      r:   r,
    }, nil
  }

  // Wait is shorthand for WaitN(1, types.InfDuration).
  func (lim *Limiter) Wait() (err error) {
    return lim.WaitN(1, types.InfDuration)
  }

  // SetLimit sets a new Limit for the limiter. The new Limit, and Burst, may be violated
  // or underutilized by those which reserved (using Reserve or Wait) but did not yet act
  // before SetLimit was called.
  func (lim *Limiter) SetLimit(limit types.Limit) error {
    _, err := restate.Object[types.Reservation](lim.ctx, "Limiter", lim.limiterID, "SetRate").Request(types.SetRateRequest{
      Limit: &limit,
    })
    return err
  }

  // SetBurst sets a new burst size for the limiter.
  func (lim *Limiter) SetBurst(burst int) error {
    _, err := restate.Object[types.Reservation](lim.ctx, "Limiter", lim.limiterID, "SetRate").Request(types.SetRateRequest{
      Burst: &burst,
    })
    return err
  }

  // SetRate is a convenience method to call both SetLimit and SetBurst atomically.
  func (lim *Limiter) SetRate(limit types.Limit, burst int) error {
    _, err := restate.Object[types.Reservation](lim.ctx, "Limiter", lim.limiterID, "SetRate").Request(types.SetRateRequest{
      Limit: &limit,
      Burst: &burst,
    })
    return err
  }

  // WaitN blocks until lim permits n events to happen.
  // It returns an error if n exceeds the Limiter's burst size, the invocation is
  // canceled, or the expected wait time exceeds the maxFutureReserve
  // The burst limit is ignored if the rate limit is Inf.
  func (lim *Limiter) WaitN(n int, maxFutureReserve time.Duration) (err error) {
    r, err := lim.reserveN(n, maxFutureReserve)
    if err != nil {
      return err
    }

    if !r.r.Ok {
      if maxFutureReserve == types.InfDuration {
        return restate.WithErrorCode(restate.TerminalErrorf("rate: Wait(n=%d) would exceed the limiters burst", n), 429)
      } else {
        return restate.WithErrorCode(restate.TerminalErrorf("rate: Wait(n=%d) would either exceed the limiters burst or the provided maxFutureReserve", n), 429)
      }
    }

    // Wait if necessary
    delay := r.DelayFrom(r.r.CreationTime)
    if delay == 0 {
      return
    }

    if err := restate.Sleep(lim.ctx, delay); err != nil {
      // this only happens on invocation cancellation - cancel the reservation in the background
      r.Cancel()
      return err
    }

    return nil
  }

  // Delay is shorthand for DelayFrom(time.Now()) using a deterministic timestamp.
  func (r *Reservation) Delay() time.Duration {
    t, _ := restate.Run(r.lim.ctx, func(ctx restate.RunContext) (time.Time, error) {
      return time.Now(), nil
    })
    return r.DelayFrom(t)
  }

  // DelayFrom returns the duration for which the reservation holder must wait
  // before taking the reserved action.  Zero duration means act immediately.
  // InfDuration means the limiter cannot grant the tokens requested in this
  // Reservation within the maximum wait time.
  func (r *Reservation) DelayFrom(t time.Time) time.Duration {
    if !r.r.Ok {
      return types.InfDuration
    }
    delay := r.r.TimeToAct.Sub(t)
    if delay < 0 {
      return 0
    }
    return delay
  }
  ```
</CodeGroup>

The **limiter implementation**, which manages the token bucket state and logic:

<CodeGroup>
  ```ts TypeScript expandable {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/patterns-use-cases/src/ratelimit/limiter.ts"}  theme={null}
  // a faithful reimplementation of https://pkg.go.dev/golang.org/x/time/rate#Limiter
  // using virtual object state

  import { object, ObjectContext } from "@restatedev/restate-sdk";

  type LimiterState = {
    state: LimiterStateInner;
  };
  type LimiterStateInner = {
    limit: number;
    burst: number;
    tokens: number;
    // last is the last time the limiter's tokens field was updated, in unix millis
    last: number;
    // lastEvent is the latest time of a rate-limited event (past or future), in unix millis
    lastEvent: number;
  };

  export interface Reservation {
    ok: boolean;
    tokens: number;
    creationDate: number;
    dateToAct: number;
    // This is the Limit at reservation time, it can change later.
    limit: number;
  }

  export const limiter = object({
    name: "limiter",
    handlers: {
      state: async (ctx: ObjectContext<LimiterState>): Promise<LimiterStateInner> => {
        return getState(ctx);
      },
      tokens: async (ctx: ObjectContext<LimiterState>): Promise<number> => {
        // deterministic date not needed, as there is only an output entry
        const tokens = advance(await getState(ctx), Date.now());
        return tokens;
      },
      reserve: async (
        ctx: ObjectContext<LimiterState>,
        { n = 1, waitLimitMillis = Infinity }: { n?: number; waitLimitMillis?: number },
      ): Promise<Reservation> => {
        let lim = await getState(ctx);

        if (lim.limit == Infinity) {
          // deterministic date is not necessary, as this is part of a response body, which won't be replayed.
          const now = Date.now();
          return {
            ok: true,
            tokens: n,
            creationDate: now,
            dateToAct: now,
            limit: 0,
          };
        }

        let r: Reservation;
        ({ lim, r } = await ctx.run(() => {
          const now = Date.now();
          let tokens = advance(lim, now);

          // Calculate the remaining number of tokens resulting from the request.
          tokens -= n;

          // Calculate the wait duration
          let waitDurationMillis = 0;
          if (tokens < 0) {
            waitDurationMillis = durationFromTokens(lim.limit, -tokens);
          }

          // Decide result
          const ok = n <= lim.burst && waitDurationMillis <= waitLimitMillis;

          // Prepare reservation
          const r = {
            ok,
            tokens: 0,
            creationDate: now,
            dateToAct: 0,
            limit: lim.limit,
          } satisfies Reservation;

          if (ok) {
            r.tokens = n;
            r.dateToAct = now + waitDurationMillis;

            // Update state
            lim.last = now;
            lim.tokens = tokens;
            lim.lastEvent = r.dateToAct;
          }

          return { lim, r };
        }));

        setState(ctx, lim);

        return r;
      },
      setRate: async (
        ctx: ObjectContext<LimiterState>,
        { newLimit, newBurst }: { newLimit?: number; newBurst?: number },
      ) => {
        if (newLimit === undefined && newBurst === undefined) {
          return;
        }

        let lim = await getState(ctx);

        lim = await ctx.run(() => {
          const now = Date.now();
          const tokens = advance(lim, now);

          lim.last = now;
          lim.tokens = tokens;
          if (newLimit !== undefined) lim.limit = newLimit;
          if (newBurst !== undefined) lim.burst = newBurst;

          return lim;
        });

        setState(ctx, lim);
      },
      cancelReservation: async (ctx: ObjectContext<LimiterState>, r: Reservation) => {
        let lim = await getState(ctx);

        lim = await ctx.run(() => {
          const now = Date.now();

          if (lim.limit == Infinity || r.tokens == 0 || r.dateToAct < now) {
            return lim;
          }

          // calculate tokens to restore
          // The duration between lim.lastEvent and r.timeToAct tells us how many tokens were reserved
          // after r was obtained. These tokens should not be restored.
          const restoreTokens = r.tokens - tokensFromDuration(r.limit, lim.lastEvent - r.dateToAct);
          if (restoreTokens <= 0) {
            return lim;
          }
          // advance time to now
          let tokens = advance(lim, now);
          // calculate new number of tokens
          tokens += restoreTokens;
          if (tokens > lim.burst) {
            tokens = lim.burst;
          }
          // update state
          lim.last = now;
          lim.tokens = tokens;
          if (r.dateToAct == lim.lastEvent) {
            const prevEvent = r.dateToAct + durationFromTokens(r.limit, -r.tokens);
            if (prevEvent >= now) {
              lim.lastEvent = prevEvent;
            }
          }

          return lim;
        });

        setState(ctx, lim);
      },
    },
  });

  function advance(lim: LimiterStateInner, date: number): number {
    let last = lim.last;
    if (date <= last) {
      last = date;
    }

    // Calculate the new number of tokens, due to time that passed.
    const elapsedMillis = date - last;
    const delta = tokensFromDuration(lim.limit, elapsedMillis);
    let tokens = lim.tokens + delta;
    if (tokens > lim.burst) {
      tokens = lim.burst;
    }

    return tokens;
  }

  async function getState(ctx: ObjectContext<LimiterState>): Promise<LimiterStateInner> {
    return (
      (await ctx.get("state")) ?? {
        limit: 0,
        burst: 0,
        tokens: 0,
        last: 0,
        lastEvent: 0,
      }
    );
  }

  async function setState(ctx: ObjectContext<LimiterState>, lim: LimiterStateInner) {
    ctx.set("state", lim);
  }

  function durationFromTokens(limit: number, tokens: number): number {
    if (limit <= 0) {
      return Infinity;
    }

    return (tokens / limit) * 1000;
  }

  function tokensFromDuration(limit: number, durationMillis: number): number {
    if (limit <= 0) {
      return 0;
    }
    return (durationMillis / 1000) * limit;
  }

  export type Limiter = typeof limiter;
  ```

  ```go Go expandable {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/patterns-use-cases/src/ratelimit/service/limiter.go"}  theme={null}
  package service

  import (
    "math"
    "time"

    "github.com/restatedev/examples/go/patterns-use-cases/src/ratelimit/types"
    restate "github.com/restatedev/sdk-go"
  )

  type Limiter struct{}

  func (Limiter) State(ctx restate.ObjectContext) (types.LimiterState, error) {
    return restate.Get[types.LimiterState](ctx, "state")
  }

  func (Limiter) Tokens(ctx restate.ObjectContext) (float64, error) {
    lim, err := restate.Get[types.LimiterState](ctx, "state")
    if err != nil {
      return 0.0, err
    }

    // deterministic date not needed, as there is only an output entry
    tokens := advance(&lim, time.Now())
    return tokens, nil
  }

  func (Limiter) ReserveN(ctx restate.ObjectContext, req types.ReserveRequest) (types.Reservation, error) {
    lim, err := restate.Get[types.LimiterState](ctx, "state")
    if err != nil {
      return types.Reservation{}, err
    }

    if lim.Limit == types.Inf {
      // deterministic date is not necessary, as this is part of a response body, which won't be replayed.
      t := time.Now()
      return types.Reservation{
        Ok:           true,
        CreationTime: t,
        Tokens:       req.N,
        TimeToAct:    t,
      }, nil
    }

    type runResult struct {
      types.LimiterState `json:"limiterState"`
      types.Reservation  `json:"reservation"`
    }

    result, err := restate.Run(ctx, func(ctx restate.RunContext) (runResult, error) {
      t := time.Now()
      tokens := advance(&lim, t)

      // Calculate the remaining number of tokens resulting from the request.
      tokens -= float64(req.N)

      // Calculate the wait duration
      var waitDuration time.Duration
      if tokens < 0 {
        waitDuration = durationFromTokens(lim.Limit, -tokens)
      }

      // Decide result
      ok := req.N <= lim.Burst && waitDuration <= req.MaxFutureReserve

      // Prepare reservation
      r := types.Reservation{
        Ok:           ok,
        CreationTime: t,
        Limit:        lim.Limit,
      }
      if ok {
        r.Tokens = req.N
        r.TimeToAct = t.Add(waitDuration)

        // Update state
        lim.Last = t
        lim.Tokens = tokens
        lim.LastEvent = r.TimeToAct
      }

      return runResult{lim, r}, nil
    })
    if err != nil {
      return types.Reservation{}, err
    }

    restate.Set(ctx, "state", result.LimiterState)

    return result.Reservation, nil
  }

  func (Limiter) SetRate(ctx restate.ObjectContext, req types.SetRateRequest) error {
    if req.Limit == nil && req.Burst == nil {
      return nil
    }

    lim, err := restate.Get[types.LimiterState](ctx, "state")
    if err != nil {
      return err
    }

    lim, err = restate.Run(ctx, func(ctx restate.RunContext) (types.LimiterState, error) {
      t := time.Now()
      tokens := advance(&lim, t)

      lim.Last = t
      lim.Tokens = tokens

      if req.Limit != nil {
        lim.Limit = *req.Limit
      }
      if req.Burst != nil {
        lim.Burst = *req.Burst
      }

      return lim, nil
    })
    if err != nil {
      return err
    }

    restate.Set(ctx, "state", lim)
    return nil
  }

  func (Limiter) CancelReservation(ctx restate.ObjectContext, r types.Reservation) error {
    lim, err := restate.Get[types.LimiterState](ctx, "state")
    if err != nil {
      return err
    }

    lim, err = restate.Run(ctx, func(ctx restate.RunContext) (types.LimiterState, error) {
      t := time.Now()

      if r.Limit == types.Inf || r.Tokens == 0 || r.TimeToAct.Before(t) {
        return lim, nil
      }

      // calculate tokens to restore
      // The duration between lim.lastEvent and r.timeToAct tells us how many tokens were reserved
      // after r was obtained. These tokens should not be restored.
      restoreTokens := float64(r.Tokens) - tokensFromDuration(r.Limit, lim.LastEvent.Sub(r.TimeToAct))
      if restoreTokens <= 0 {
        return lim, nil
      }
      // advance time to now
      tokens := advance(&lim, t)
      // calculate new number of tokens
      tokens += restoreTokens
      if burst := float64(lim.Burst); tokens > burst {
        tokens = burst
      }
      // update state
      lim.Last = t
      lim.Tokens = tokens
      if r.TimeToAct == lim.LastEvent {
        prevEvent := r.TimeToAct.Add(durationFromTokens(r.Limit, float64(-r.Tokens)))
        if !prevEvent.Before(t) {
          lim.LastEvent = prevEvent
        }
      }

      return lim, nil
    })

    restate.Set(ctx, "state", lim)

    return nil
  }

  func advance(lim *types.LimiterState, t time.Time) float64 {
    last := lim.Last
    if t.Before(last) {
      last = t
    }

    // Calculate the new number of tokens, due to time that passed.
    elapsed := t.Sub(last)
    delta := tokensFromDuration(lim.Limit, elapsed)
    tokens := lim.Tokens + delta
    if burst := float64(lim.Burst); tokens > burst {
      tokens = burst
    }
    return tokens
  }

  // durationFromTokens is a unit conversion function from the number of tokens to the duration
  // of time it takes to accumulate them at a rate of limit tokens per second.
  func durationFromTokens(limit types.Limit, tokens float64) time.Duration {
    if limit <= 0 {
      return types.InfDuration
    }

    duration := (tokens / float64(limit)) * float64(time.Second)

    // Cap the duration to the maximum representable int64 value, to avoid overflow.
    if duration > float64(math.MaxInt64) {
      return types.InfDuration
    }

    return time.Duration(duration)
  }

  // tokensFromDuration is a unit conversion function from a time duration to the number of tokens
  // which could be accumulated during that duration at a rate of limit tokens per second.
  func tokensFromDuration(limit types.Limit, d time.Duration) float64 {
    if limit <= 0 {
      return 0
    }
    return d.Seconds() * float64(limit)
  }
  ```
</CodeGroup>

This implementation provides a `RateLimiter` Virtual Object which implements the **Token Bucket Algorithm**:

* Tokens are added at a specified rate (limit)
* Tokens are consumed when operations are performed
* A burst capacity allows for short bursts of activity

**Key Methods** (via the client interface):

* `limit()` / `burst()`: Get maximum event rate and burst size
* `tokens()`: Get number of available tokens
* `wait()`: Block until events are permitted to happen
* `setRate()` / `setLimit()` / `setBurst()`: Configure rate limiting parameters

## Usage Example

Here's how to use the rate limiter in your services:

<CodeGroup>
  ```ts TypeScript {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/patterns-use-cases/src/ratelimit/service.ts"}  theme={null}
  import { Context, service } from "@restatedev/restate-sdk";
  import { Limiter } from "./limiter_client";

  const LIMITER_NAME = "myService-expensiveMethod";

  export const myService = service({
    name: "myService",
    handlers: {
      expensiveMethod: async (ctx: Context) => {
        const limiter = Limiter.fromContext(ctx, LIMITER_NAME);
        await limiter.wait();
        console.log("expensive!");
      },
      expensiveMethodBatch: async (ctx: Context, batchSize: number = 20) => {
        const limiter = Limiter.fromContext(ctx, LIMITER_NAME);
        await limiter.wait(batchSize);
        for (let i = 0; i < batchSize; i++) {
          console.log("expensive!");
        }
      },
    },
  });

  export type MyService = typeof myService;
  ```

  ```go Go {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/patterns-use-cases/src/ratelimit/example/main.go?collapse_prequel"}  theme={null}
  type LimitedTask struct{}

  func (LimitedTask) RunTask(ctx restate.Context) error {
    limiter := client.NewLimiter(ctx, "LimitedTask-RunTask")
    if err := limiter.Wait(); err != nil {
      return err
    }

    // Implement doing the work...

    return nil
  }

  func main() {
    server := server.NewRestate().
      Bind(restate.Reflect(service.Limiter{})).
      Bind(restate.Reflect(LimitedTask{}))

    if err := server.Start(context.Background(), ":9080"); err != nil {
      slog.Error("application exited unexpectedly", "err", err.Error())
      os.Exit(1)
    }
  }
  ```
</CodeGroup>

## Running the example

<Steps>
  <Step title="Download the example">
    <CodeGroup>
      ```bash TypeScript theme={null}
      restate example typescript-patterns-use-cases && cd typescript-patterns-use-cases
      ```

      ```bash Go theme={null}
      restate example go-patterns-use-cases && cd go-patterns-use-cases
      ```
    </CodeGroup>
  </Step>

  <Step title="Start the Restate Server">
    ```bash theme={null}
    restate-server
    ```
  </Step>

  <Step title="Start the Service">
    <CodeGroup>
      ```bash TypeScript theme={null}
      npm install
      npx tsx watch ./src/ratelimit/app.ts
      ```

      ```bash Go theme={null}
      go run ./src/ratelimit/example/main.go
      ```
    </CodeGroup>
  </Step>

  <Step title="Register the services">
    ```bash theme={null}
    restate deployments register localhost:9080
    ```
  </Step>

  <Step title="Set up rate limiting">
    Set up the limiter named `myService-expensiveMethod` with a rate limit of 1 per second:

    <CodeGroup>
      ```bash TypeScript theme={null}
      curl localhost:8080/limiter/myService-expensiveMethod/setRate \
          --json '{"newLimit": 1, "newBurst": 1}'
      ```

      ```bash Go theme={null}
      curl localhost:8080/Limiter/LimitedTask-RunTask/SetRate \
          --json '{"limit": 1, "burst": 1}'
      ```
    </CodeGroup>

    Try sending multiple requests quickly to see the rate limiting in action.
  </Step>

  <Step title="Send requests to the rate limited handler">
    You can send requests that are subject to the limiter like this:

    <CodeGroup>
      ```bash TypeScript theme={null}
      # send one request
      curl localhost:8080/myService/expensiveMethod

      # send lots
      for i in $(seq 1 30); do curl localhost:8080/myService/expensiveMethod && echo "request completed"; done
      ```

      ```bash Go theme={null}
      # send one request
      curl localhost:8080/LimitedTask/RunTask

      # send lots
      for i in $(seq 1 30); do curl localhost:8080/LimitedTask/RunTask && echo "request completed"; done
      ```
    </CodeGroup>

    You should observe that only one request is processed per second.
    You can then try changing the limit or the burst and sending more requests.
  </Step>

  <Step title="Observe in the Restate UI">
    In the Restate UI, you can observe:

    * Invocations getting scheduled one per second in the Invocations tab
    * Rate limiter state and token counts in the State tab
  </Step>
</Steps>


# Request Lifecycle
Source: https://docs.restate.dev/guides/request-lifecycle

Deep dive into the lifecycle of a request in Restate

This guide provides a detailed technical walkthrough of how Restate processes requests from start to finish, including the mechanisms that enable durable execution and failure recovery.

## Request Processing Lifecycle

When Restate receives a request to invoke a handler, it follows this detailed process:

### 1. Persist Request

Restate persists the request and creates a new execution journal for this invocation. From this point, Restate guarantees that it will process the request to completion, even in the face of failures.

### 2. Forward to Service

Restate opens a bidirectional streaming connection (HTTP/2) to the service for which the request is meant. This connection will be used to stream messages back and forth between the Server and the SDK. It then forwards the request to the service.

### 3. Invoke Handler

The Restate SDK, embedded in the service, calls the handler with the request data and a context object that provides access to Restate features like state management, service calls, and timers.

### 4. Track Execution

As the handler runs, it uses the Restate Context to call other services, update state, or perform side effects. The Restate SDK under-the-hood sends a message to the Restate server for each Context operation. The Restate Server then records it in the execution journal.

### 5. Handle Failures

If the handler crashes or fails, Restate either loses the connection or receives an error message. In this case, the Restate Server will send a retry request to the service, including the latest journal.

Failure detection mechanisms:

* Connection between Server and service dropped
* Explicit error messages (not terminal errors)
* Inactivity timeouts reached

### 6. Replay Journal

The service receives the retry request and restarts the execution of the handler. Whenever the handler performs an action on the Restate context (like calling another service, updating state, or setting a timer), the SDK checks the journal for the last recorded result. If it finds a previous result, it skips executing that action and uses the recorded result instead. This way, the handler resumes from exactly where it left off.

### 7. Complete Execution

Once the handler completes successfully, the Restate SDK proxies the result back to the client via the Restate Server. The server then marks the invocation as complete in the journal.

## Cross-service interactions

Note that inter-handler communication between two durable handlers goes through Restate (not direct service-to-service). The caller appends an invocation event to its journal, which Restate Server turns into an invocation for the target service, providing end-to-end idempotency.

## Request-response Deployment Targets

Some deployment targets don't support bidirectional streaming connections (HTTP/2), such as AWS Lambda. In these cases, the Restate SDK uses a request-response pattern (HTTP/1.1) instead.

In this case, the SDK will try to make as much progress as possible.
When it hits a point where it needs to wait for a response (like a service call, sleep, or run action), it will send the new journal entries to the Restate Server and suspend the execution.
The server will send a retry request once the response is available.
Execution then resumes by replaying the journal up to the suspension point.


# Kubernetes deployments with Helm
Source: https://docs.restate.dev/guides/restate-on-kind-with-helm

Learn how to deploy a Restate cluster using Helm on a kind Kubernetes cluster.

This tutorial will guide you through deploying a single-node Restate cluster on a local Kubernetes cluster using Helm.

We will use [kind](https://kind.sigs.k8s.io/) to create a local Kubernetes cluster for testing purposes.

<Steps>
  <Step title="Prerequisites">
    Install the following on your local machine:

    * [Docker](https://docs.docker.com/)
    * [Helm](https://helm.sh/docs/intro/install/)
    * [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl)
    * [Restate](/installation)
    * [kind](https://kind.sigs.k8s.io/docs/user/quick-start)

    This guide was written with kind v0.30.0, and Restate v1.5.0.
    Contact us on [Discord](https://discord.restate.dev) or [Slack](https://slack.restate.dev) if something does not work as expected.
  </Step>

  <Step title="Create a kind cluster">
    ```bash theme={null}
    kind create cluster
    ```

    Set `kubectl` context to `kind-kind`:

    ```bash theme={null}
    kubectl cluster-info --context kind-kind
    ```
  </Step>

  <Step title="Install the single-node Restate Helm chart">
    ```bash theme={null}
    helm install restate oci://ghcr.io/restatedev/restate-helm \
      --namespace restate
      --create-namespace
    ```

    This will deploy a single-node Restate cluster in the `restate` namespace.

    Wait until the `restate-0` pod is in the `Ready` state:

    ```bash theme={null}
    kubectl get pods -n restate -w
    ```
  </Step>

  <Step title="Forward the ports of the Restate ingress and the UI to your local machine">
    ```bash theme={null}
    kubectl port-forward svc/restate -n restate 8080:8080 9070:9070
    ```

    Now you can access the Restate UI at `http://localhost:9070`.
  </Step>

  <Step title="Build and upload a Restate service Docker image to the kind cluster">
    For example, download the TypeScript Hello World service:

    ```bash theme={null}
    restate example typescript-hello-world &&
    cd typescript-hello-world &&
    npm install
    ```

    Build a Docker image for the service:

    ```bash theme={null}
    docker build -t my-restate-service:0.0.1 .
    ```

    Upload the image to the kind cluster:

    ```bash theme={null}
    kind load docker-image my-restate-service:0.0.1
    ```
  </Step>

  <Step title="Deploy the Restate service">
    Create a `RestateDeployment` manifest for the service in a file called `service-deployment.yaml`:

    ```yaml service-deployment.yaml theme={null}
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: my-app
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: my-app
      template:
        metadata:
          labels:
            app: my-app
        spec:
          containers:
          - name: app
            image: my-restate-service:0.0.1
            ports:
            - name: http
              containerPort: 9080
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: my-app
    spec:
      selector:
        app: my-app
      ports:
      - name: http
        port: 9080
        targetPort: 9080
    ```

    Apply the manifest to deploy the service:

    ```bash theme={null}
    kubectl apply -f service-deployment.yaml -n restate
    ```
  </Step>

  <Step title="Register the service">
    In the Restate UI, you can now register the deployment `http://my-app.restate.svc.cluster.local:9080/` via the UI at `http://localhost:9070`.

    Once it is registered, you should see the deployed service listed in the Restate UI:

    <Frame>
      <img />
    </Frame>
  </Step>

  <Step title="Invoke the service">
    In the Restate UI playground, you can now invoke the service. In the overview page, click on the `greet` handler of your service to open the playground.
    Then send a request:

    <Frame>
      <img />
    </Frame>
  </Step>

  <Step title="🎉 You did it!">
    You have successfully deployed a Restate cluster and a Restate service on a local kind Kubernetes cluster using the Helm chart!

    Check out the [Restate Helm deployment docs](/server/deploy/kubernetes#helm-chart) for more information.
  </Step>
</Steps>

After you are done experimenting, you can delete the kind cluster:

```bash theme={null}
kind delete cluster
```


# Kubernetes deployments with Restate Operator
Source: https://docs.restate.dev/guides/restate-on-kind-with-operator

Learn how to deploy single-node Restate with Restate operator on a kind Kubernetes cluster.

This guide walks you through deploying a Restate cluster on Kubernetes using the Restate Operator.

We will use [kind](https://kind.sigs.k8s.io/) to create a local Kubernetes cluster for testing purposes.

<Steps>
  <Step title="Prerequisites">
    Install the following on your local machine:

    * [Docker](https://docs.docker.com/)
    * [Helm](https://helm.sh/docs/intro/install/)
    * [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl)
    * [Restate](/installation)
    * [kind](https://kind.sigs.k8s.io/docs/user/quick-start)

    This guide was written with kind v0.30.0, [Restate Operator](https://github.com/restatedev/restate-operator/tree/main) v1.8.1, and Restate v1.5.0.
    Contact us on [Discord](https://discord.restate.dev) or [Slack](https://slack.restate.dev) if something does not work as expected.
  </Step>

  <Step title="Create a kind cluster">
    ```bash theme={null}
    kind create cluster
    ```

    Set `kubectl` context to `kind-kind`:

    ```bash theme={null}
    kubectl cluster-info --context kind-kind
    ```
  </Step>

  <Step title="Install the Restate Operator">
    Install the Restate Operator via Helm:

    ```bash theme={null}
    helm install restate-operator \
      oci://ghcr.io/restatedev/restate-operator-helm \
      --namespace restate-operator \
      --create-namespace
    ```

    To install the operator, you need to be able to create namespaces and CRDs.
  </Step>

  <Step title="Create a Restate cluster">
    Create a `RestateCluster` manifest in a file called `restate-cluster.yaml`:

    ```yaml restate-cluster.yaml theme={null}
    apiVersion: restate.dev/v1
    kind: RestateCluster
    metadata:
      name: restate-test
    spec:
      compute:
        image: restatedev/restate:1.5
      storage:
        storageRequestBytes: 2147483648 # 2 GiB
      security:
        disableNetworkPolicies: true
    ```

    Note that we disable network policies here for simplicity, but in a production environment, you should configure appropriate network policies for security.

    To learn more about the `RestateCluster` spec options, see the [`RestateCluster` pkl definition](https://github.com/restatedev/restate-operator/blob/main/crd/RestateCluster.pkl).

    Apply the manifest:

    ```bash theme={null}
    kubectl apply -n restate-test -f restate-cluster.yaml
    ```

    A single-node Restate deployment will be created in the `restate-test` namespace.

    Wait until the `restate-0` pod is in the `Ready` state:

    ```bash theme={null}
    kubectl get pods -n restate-test -w
    ```
  </Step>

  <Step title="Forward the ports of the Restate ingress and the UI to your local machine">
    ```bash theme={null}
    kubectl port-forward svc/restate -n restate-test 8080:8080 9070:9070
    ```

    Now you can access the Restate UI at `http://localhost:9070`.
  </Step>

  <Step title="Build and upload a Restate service Docker image to the kind cluster">
    For example, download the TypeScript Hello World service:

    ```bash theme={null}
    restate example typescript-hello-world &&
    cd typescript-hello-world &&
    npm install
    ```

    Build a Docker image for the service:

    ```bash theme={null}
    docker build -t my-restate-service:0.0.1 .
    ```

    Upload the image to the kind cluster:

    ```bash theme={null}
    kind load docker-image my-restate-service:0.0.1
    ```
  </Step>

  <Step title="Deploy the Restate service">
    Create a `RestateDeployment` manifest for the service in a file called `service-deployment.yaml`:

    ```yaml service-deployment.yaml theme={null}
    apiVersion: restate.dev/v1beta1
    kind: RestateDeployment
    metadata:
      name: my-app
    spec:
      replicas: 1
      restate:
        register:
          service:
            name: restate
            namespace: restate-test
      selector:
        matchLabels:
          app: my-app
      template:
        metadata:
          labels:
            app: my-app
        spec:
          containers:
          - name: app
            image: my-restate-service:0.0.1
            ports:
            - name: restate
              containerPort: 9080
    ```

    Apply the manifest to deploy the service:

    ```bash theme={null}
    kubectl apply -f service-deployment.yaml -n restate-test
    ```

    You should now see the deployed service listed in the Restate UI:

    <Frame>
      <img />
    </Frame>

    The Restate Operator automatically [registered](/services/versioning#registering-a-deployment) the service with the Restate cluster.
  </Step>

  <Step title="Invoke the service">
    In the Restate UI playground, you can now invoke the service. In the overview page, click on the `greet` handler of your service to open the playground.
    Then send a request:

    <Frame>
      <img />
    </Frame>
  </Step>

  <Step title="🎉 You did it!">
    You have successfully deployed a Restate cluster and a Restate service on a local kind Kubernetes cluster using the Restate Operator!

    Check out the [Restate Operator docs](/server/deploy/kubernetes#restate-kubernetes-operator) for further information.
  </Step>
</Steps>

After you are done experimenting, you can delete the kind cluster:

```bash theme={null}
kind delete cluster
```


# Sagas
Source: https://docs.restate.dev/guides/sagas

Implementing undo operations in case of failures, to keep your system consistent

A **Saga** is a design pattern for handling transactions that span multiple services. It breaks the process into a sequence of local operations, each with a corresponding **compensating action**. If a failure occurs partway through, these compensations are triggered to **undo** completed steps, ensuring your system stays consistent even when things go wrong.

## How does Restate help?

Restate makes implementing resilient sagas simple:

* **Durable Execution**: Restate guarantees completion and automatically retries from failure points. No manual state tracking or retry logic needed
* **Code-first approach**: Define sagas using regular code, no DSLs required

<img alt="Sagas UI" />

## Example

A travel booking workflow: book a flight, rent a car, then book a hotel. If any step fails (e.g. hotel full), we roll back previous steps to maintain consistency.

<img alt="Sagas example diagram" />

**Implementation:**

* Wrap business logic in a try-block, throw terminal errors for compensation cases
* Add compensations to a list for each step
* In catch block, run compensations in reverse order and rethrow

Note: Golang uses `defer` for compensations.

<CodeGroup>
  ```ts TypeScript {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/patterns-use-cases/src/sagas/booking_workflow.ts?collapse_prequel?remove_comments"}  theme={null}
  const bookingWorkflow = restate.service({
    name: "BookingWorkflow",
    handlers: {
      run: async (ctx: restate.Context, req: BookingRequest) => {
        const { customerId, flight, car, hotel } = req;
        const compensations = [];

        try {
          compensations.push(() => ctx.run("Cancel flight", () => flightClient.cancel(customerId)));
          await ctx.run("Book flight", () => flightClient.book(customerId, flight));

          compensations.push(() => ctx.run("Cancel car", () => carRentalClient.cancel(customerId)));
          await ctx.run("Book car", () => carRentalClient.book(customerId, car));

          compensations.push(() => ctx.run("Cancel hotel", () => hotelClient.cancel(customerId)));
          await ctx.run("Book hotel", () => hotelClient.book(customerId, hotel));
        } catch (e) {
          if (e instanceof restate.TerminalError) {
            for (const compensation of compensations.reverse()) {
              await compensation();
            }
          }
          throw e;
        }
      },
    },
  });

  restate.serve({
    services: [bookingWorkflow],
    port: 9080,
  });
  ```

  ```java Java {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/patterns-use-cases/src/main/java/my/example/sagas/BookingWorkflow.java?collapse_prequel?remove_comments"}  theme={null}
  @Service
  public class BookingWorkflow {

    public record BookingRequest(
        String customerId, FlightRequest flight, CarRequest car, HotelRequest hotel) {}

    @Handler
    public void run(Context ctx, BookingRequest req) throws TerminalException {
      List<Runnable> compensations = new ArrayList<>();

      try {
        compensations.add(() -> ctx.run("Cancel flight", () -> FlightClient.cancel(req.customerId)));
        ctx.run("Book flight", () -> FlightClient.book(req.customerId, req.flight()));

        compensations.add(() -> ctx.run("Cancel car", () -> CarRentalClient.cancel(req.customerId)));
        ctx.run("Book car", () -> CarRentalClient.book(req.customerId, req.car()));

        compensations.add(() -> ctx.run("Cancel hotel", () -> HotelClient.cancel(req.customerId)));
        ctx.run("Book hotel", () -> HotelClient.book(req.customerId, req.hotel()));
      }
      catch (TerminalException e) {
        for (Runnable compensation : compensations) {
          compensation.run();
        }
        throw e;
      }
    }

    public static void main(String[] args) {
      RestateHttpServer.listen(Endpoint.bind(new BookingWorkflow()));
    }
  }
  ```

  ```kotlin Kotlin {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/kotlin/patterns-use-cases/src/main/kotlin/my/example/sagas/BookingWorkflow.kt?collapse_prequel?remove_comments"}  theme={null}
  @Service
  class BookingWorkflow {

    @Handler
    suspend fun run(ctx: Context, req: BookingRequest) {

      val compensations = mutableListOf<suspend () -> Unit>()

      try {
        compensations.add { ctx.runBlock("Cancel flight") { cancelFlight(req.customerId) } }
        ctx.runBlock("Book flight") { bookFlight(req.customerId, req.flight) }

        compensations.add { ctx.runBlock("Cancel car") { cancelCar(req.customerId) } }
        ctx.runBlock("Book car") { bookCar(req.customerId, req.car) }

        compensations.add { ctx.runBlock("Cancel hotel") { cancelHotel(req.customerId) } }
        ctx.runBlock("Book hotel") { bookHotel(req.customerId, req.hotel) }
      }
      catch (e: TerminalException) {
        compensations.reversed().forEach { it() }
        throw e
      }
    }
  }

  fun main() {
    RestateHttpServer.listen(endpoint { bind(BookingWorkflow()) })
  }
  ```

  ```python Python {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/python/patterns-use-cases/sagas/app.py?collapse_prequel?remove_comments"}  theme={null}
  booking_workflow = restate.Service("BookingWorkflow")


  @booking_workflow.handler()
  async def run(ctx: restate.Context, req: BookingRequest):
      compensations = []

      try:
          compensations.append(
              lambda: ctx.run_typed("Cancel flight", flight_client.cancel, customer_id=req.customer_id)
          )
          await ctx.run_typed("Book flight", flight_client.book, customer_id=req.customer_id, flight=req.flight)

          compensations.append(
              lambda: ctx.run_typed("Cancel car", car_rental_client.cancel, customer_id=req.customer_id)
          )
          await ctx.run_typed("Book car", car_rental_client.book, customer_id=req.customer_id, car=req.car)

          compensations.append(
              lambda: ctx.run_typed("Cancel hotel", hotel_client.cancel, customer_id=req.customer_id)
          )
          await ctx.run_typed("Book hotel", hotel_client.book, customer_id=req.customer_id, hotel=req.hotel)

      except TerminalError as e:
          for compensation in reversed(compensations):
              await compensation()
          raise e


  app = restate.app([booking_workflow])

  if __name__ == "__main__":
      import hypercorn
      import asyncio

      conf = hypercorn.Config()
      conf.bind = ["0.0.0.0:9080"]
      asyncio.run(hypercorn.asyncio.serve(app, conf))
  ```

  ```go Go {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/patterns-use-cases/src/sagas/bookingworkflow.go?collapse_prequel?remove_comments"}  theme={null}
  type BookingWorkflow struct{}

  func (BookingWorkflow) Run(ctx restate.Context, req BookingRequest) (err error) {

    var compensations []func() (restate.Void, error)

    defer func() {
      if err != nil {
        for _, compensation := range slices.Backward(compensations) {
          if _, compErr := compensation(); compErr != nil {
            err = compErr
          }
        }
      }
    }()

    compensations = append(compensations, func() (restate.Void, error) {
      return restate.Run(ctx,
        func(ctx restate.RunContext) (restate.Void, error) {
          return CancelFlight(req.CustomerId)
        },
        restate.WithName("Cancel flight"),
      )
    })
    if _, err = restate.Run(ctx,
      func(ctx restate.RunContext) (restate.Void, error) {
        return BookFlight(req.CustomerId, req.Flight)
      },
      restate.WithName("Book flight"),
    ); err != nil {
      return err
    }

    compensations = append(compensations, func() (restate.Void, error) {
      return restate.Run(ctx,
        func(ctx restate.RunContext) (restate.Void, error) {
          return CancelCar(req.CustomerId)
        },
        restate.WithName("Cancel car"),
      )
    })
    if _, err = restate.Run(ctx,
      func(ctx restate.RunContext) (restate.Void, error) {
        return BookCar(req.CustomerId, req.Car)
      },
      restate.WithName("Book car"),
    ); err != nil {
      return err
    }

    compensations = append(compensations, func() (restate.Void, error) {
      return restate.Run(ctx,
        func(ctx restate.RunContext) (restate.Void, error) {
          return CancelHotel(req.CustomerId)
        },
        restate.WithName("Cancel hotel"),
      )
    })
    if _, err = restate.Run(ctx,
      func(ctx restate.RunContext) (restate.Void, error) {
        return BookHotel(req.CustomerId, req.Hotel)
      },
      restate.WithName("Book hotel"),
    ); err != nil {
      return err
    }

    return nil
  }

  func main() {
    if err := server.NewRestate().
      Bind(restate.Reflect(BookingWorkflow{})).
      Start(context.Background(), ":9080"); err != nil {
      log.Fatal(err)
    }
  }
  ```
</CodeGroup>

View on GitHub:
[TS](https://github.com/restatedev/examples/blob/main/typescript/patterns-use-cases/src/sagas/booking_workflow.ts) /
[Java](https://github.com/restatedev/examples/blob/main/java/patterns-use-cases/src/main/java/my/example/sagas/BookingWorkflow.java) /
[Kotlin](https://github.com/restatedev/examples/blob/main/kotlin/patterns-use-cases/src/main/kotlin/my/example/sagas/BookingWorkflow.kt) /
[Python](https://github.com/restatedev/examples/blob/main/python/patterns-use-cases/sagas/app.py) /
[Go](https://github.com/restatedev/examples/blob/main/go/patterns-use-cases/src/sagas/bookingworkflow.go)

<Note>
  This pattern is implementable with any of our SDKs. We are still working on translating all patterns to all SDK languages.
  If you need help with a specific language, please reach out to us via [Discord](https://discord.restate.dev) or [Slack](https://slack.restate.dev).
</Note>

## When to use Sagas

Restate automatically retries transient failures (network hiccups, temporary outages). For non-transient failures, sagas are essential:

1. **Business logic failures**: When failures are business decisions (e.g. "Hotel is full"), retrying won't help. Throw a terminal error to trigger compensations.

2. **User/system cancellations**: When you [cancel](/services/invocation/managing-invocations#cancel) long-running invocations, sagas undo previous operations to maintain consistency.

## Running the example

<Steps>
  <Step title="Download the example">
    <CodeGroup>
      ```bash TypeScript theme={null}
      restate example typescript-patterns-use-cases && cd typescript-patterns-use-cases
      ```

      ```bash Java theme={null}
      restate example java-patterns-use-cases && cd java-patterns-use-cases
      ```

      ```bash Kotlin theme={null}
      restate example kotlin-patterns-use-cases && cd kotlin-patterns-use-cases
      ```

      ```bash Python theme={null}
      restate example python-patterns-use-cases && cd python-patterns-use-cases
      ```

      ```bash Go theme={null}
      restate example go-patterns-use-cases && cd go-patterns-use-cases
      ```
    </CodeGroup>
  </Step>

  <Step title="Start the Restate Server">
    ```bash theme={null}
    restate-server
    ```
  </Step>

  <Step title="Start the Service">
    <CodeGroup>
      ```bash TypeScript theme={null}
      npm install
      npx tsx watch ./src/sagas/booking_workflow.ts
      ```

      ```bash Java theme={null}
      ./gradlew -PmainClass=my.example.sagas.BookingWorkflow run
      ```

      ```bash Kotlin theme={null}
      ./gradlew -PmainClass=my.example.sagas.BookingWorkflowKt run
      ```

      ```bash Python theme={null}
      uv run sagas/app.py
      ```

      ```bash Go theme={null}
      go run ./src/sagas
      ```
    </CodeGroup>
  </Step>

  <Step title="Register the services">
    ```bash theme={null}
    restate deployments register localhost:9080
    ```
  </Step>

  <Step title="Send a request">
    <CodeGroup>
      ```bash TypeScript theme={null}
      curl localhost:8080/BookingWorkflow/run --json '{
              "flight": {
                  "flightId": "12345",
                  "passengerName": "John Doe"
              },
                  "car": {
                  "pickupLocation": "Airport",
                  "rentalDate": "2024-12-16"
              },
                  "hotel": {
                  "arrivalDate": "2024-12-16",
                  "departureDate": "2024-12-20"
              }
          }'
      ```

      ```bash Java theme={null}
      curl localhost:8080/BookingWorkflow/run --json '{
              "flight": {
                  "flightId": "12345",
                  "passengerName": "John Doe"
              },
                  "car": {
                  "pickupLocation": "Airport",
                  "rentalDate": "2024-12-16"
              },
                  "hotel": {
                  "arrivalDate": "2024-12-16",
                  "departureDate": "2024-12-20"
              }
          }'
      ```

      ```bash Kotlin theme={null}
      curl localhost:8080/BookingWorkflow/run --json '{
              "flight": {
                  "flightId": "12345",
                  "passengerName": "John Doe"
              },
                  "car": {
                  "pickupLocation": "Airport",
                  "rentalDate": "2024-12-16"
              },
                  "hotel": {
                  "arrivalDate": "2024-12-16",
                  "departureDate": "2024-12-20"
              }
          }'
      ```

      ```bash Python theme={null}
      curl localhost:8080/BookingWorkflow/run --json '{
              "flight": {
                  "flightId": "12345",
                  "passengerName": "John Doe"
              },
                  "car": {
                  "pickupLocation": "Airport",
                  "rentalDate": "2024-12-16"
              },
                  "hotel": {
                  "arrivalDate": "2024-12-16",
                  "departureDate": "2024-12-20"
              }
          }'
      ```

      ```bash Go theme={null}
      curl localhost:8080/BookingWorkflow/Run --json '{
              "flight": {
                  "flightId": "12345",
                  "passengerName": "John Doe"
              },
                  "car": {
                  "pickupLocation": "Airport",
                  "rentalDate": "2024-12-16"
              },
                  "hotel": {
                  "arrivalDate": "2024-12-16",
                  "departureDate": "2024-12-20"
              }
          }'
      ```
    </CodeGroup>
  </Step>

  <Step title="Check the UI or service logs">
    See in the Restate UI (`localhost:9070`) how all steps were executed, and how the compensations were triggered because the hotel was full.

    <img alt="Sagas UI" />
  </Step>
</Steps>

## Advanced: Idempotency and compensations

Sagas in Restate are flexible and powerful since they're implemented in user code. However, you need to make sure compensations are idempotent.
The example uses customer ID for idempotency, preventing duplicate bookings on retries. The API provider deduplicates requests based on this ID.

Different APIs require different approaches:

1. **Two-phase APIs**: First *reserve*, then *confirm* or *cancel*. Register the compensation after reservation, when you have the resource ID.
   This type of API usually auto-cancels reservations after a timeout.

<CodeGroup>
  ```ts TypeScript {"CODE_LOAD::ts/src/guides/sagas/booking_workflow.ts#twostep"}  theme={null}
  const bookingId = await ctx.run(() =>
    flightClient.reserve(customerId, flight)
  );
  compensations.push(() => ctx.run(() => flightClient.cancel(bookingId)));

  // ... do other work, like reserving a car, etc. ...

  await ctx.run(() => flightClient.confirm(bookingId));
  ```

  ```python Python {"CODE_LOAD::python/src/guides/sagas/app.py#twostep"}  theme={null}
  booking_id = await ctx.run_typed(
      "reserve", flight_client.reserve, customer_id=customer_id, flight=flight
  )
  compensations.append(
      lambda: ctx.run_typed("cancel", flight_client.cancel, booking_id=booking_id)
  )

  #  ... do other work, like reserving a car, etc. ...

  await ctx.run_typed("confirm", flight_client.confirm, booking_id=booking_id)
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/guides/sagas/BookingWorkflow.java#twostep"}  theme={null}
  String bookingId = ctx.run(String.class, () -> FlightClient.reserve(flight));
  compensations.add(() -> ctx.run(() -> FlightClient.cancel(bookingId)));

  // ... do other work, like reserving a car, etc. ...

  ctx.run(() -> FlightClient.confirm(bookingId));
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/guides/sagas/BookingWorkflow.kt#twostep"}  theme={null}
  // For each action, we register a compensation that will be executed on failures
  val bookingId = ctx.runBlock { reserveFlight(customerId, flight) }
  compensations.add { ctx.runBlock { cancelFlight(bookingId) } }

  // ... do other work, like reserving a car, etc. ...

  compensations.add { ctx.runBlock { confirmFlight(bookingId) } }
  ```

  ```go Go {"CODE_LOAD::go/guides/sagas/bookingworkflow.go#twostep"}  theme={null}
  bookingId, err := restate.Run(ctx, func(ctx restate.RunContext) (string, error) {
    return ReserveFlight(req.CustomerId, req.Flight)
  })
  if err != nil {
    return err
  }

  compensations = append(compensations, func() (restate.Void, error) {
    return restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
      return CancelFlight(bookingId)
    })
  })

  // ... do other work, like reserving a car, etc. ...

  if _, err = restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
    return ConfirmFlight(bookingId)
  }); err != nil {
    return err
  }
  ```
</CodeGroup>

2. **One-shot APIs with idempotency key**: Generate idempotency key, persist in Restate, register compensation (e.g. `refund`), then do action (e.g. `charge`). Register compensation first in case action succeeded but confirmation was lost.

<CodeGroup>
  ```ts TypeScript {"CODE_LOAD::ts/src/guides/sagas/booking_workflow.ts#idempotency"}  theme={null}
  const paymentId = ctx.rand.uuidv4();
  compensations.push(() => ctx.run(() => paymentClient.refund(paymentId)));
  await ctx.run(() => paymentClient.charge(paymentInfo, paymentId));
  ```

  ```python Python {"CODE_LOAD::python/src/guides/sagas/app.py#idempotency"}  theme={null}
  payment_id = str(ctx.uuid())
  compensations.append(lambda: ctx.run_typed("refund", refund, payment_id=payment_id))
  await ctx.run_typed("charge", charge, payment_info=payment_info, payment_id=payment_id)
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/guides/sagas/BookingWorkflow.java#idempotency"}  theme={null}
  String paymentId = ctx.random().nextUUID().toString();
  compensations.add(() -> ctx.run(() -> PaymentClient.refund(paymentId)));
  ctx.run(() -> PaymentClient.charge(paymentInfo, paymentId));
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/guides/sagas/BookingWorkflow.kt#idempotency"}  theme={null}
  val paymentId = ctx.random().nextUUID().toString()
  compensations.add { ctx.runBlock { refund(paymentId) } }
  ctx.runBlock { charge(paymentInfo, paymentId) }
  ```

  ```go Go {"CODE_LOAD::go/guides/sagas/bookingworkflow.go#idempotency"}  theme={null}
  paymentID := restate.UUID(ctx).String()

  // Register the refund as a compensation, using the idempotency key
  compensations = append(compensations, func() (restate.Void, error) {
    return restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
      return Refund(paymentID)
    })
  })

  // Do the payment using the idempotency key
  if _, err = restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
    return Charge(paymentID, req.PaymentInfo)
  }); err != nil {
    return err
  }
  ```
</CodeGroup>

## Related resources

* [Error Handling guide](/guides/error-handling)
* [Cancellation of invocations](/services/invocation/managing-invocations#cancel)
* [Blog post: Graceful cancellations: How to keep your application and workflow state consistent 💪](https://restate.dev/blog/graceful-cancellations-how-to-keep-your-application-and-workflow-state-consistent/)


# XState
Source: https://docs.restate.dev/guides/xstate

Integrate Restate with XState to implement durable state machines.

Restate integrates with [XState](https://xstate.js.org/). Have a look at the [repo](https://github.com/restatedev/xstate) to learn more.


# Welcome to Restate!
Source: https://docs.restate.dev/index

Build innately resilient backends and AI agents

Restate is a lightweight runtime to turn AI agents, workflows, and backend services into durable processes. Focus on your logic, not failure mechanics.
Write normal code and let Restate handles resilience and consistency automatically.

## Key capabilities

**Durable execution**: Code automatically stores completed steps and resumes from where it left off when recovering from failures.

**Built-in state**: Maintain state beyond workflow executions and share it between functions with strong consistency guarantees.

**Reliable communication**: Call services sync or async with guaranteed execution and exactly-once semantics.

**Time-based coordination**: Sleep, schedule, and wait for external events with durable timers.

**Workflows**: Coordinate long-running processes, human approvals, listen to webhooks and other signals.

## Common use cases

<Columns>
  <Card title="AI agents" href="/use-cases/ai-agents">
    Manage stateful AI agents with reliable tool usage and long-running conversations.
  </Card>

  <Card title="Workflows" href="/use-cases/workflows">
    Build approval processes, multi-step operations, and business workflows that survive failures.
  </Card>

  <Card title="Microservice orchestration" href="/use-cases/microservice-orchestration">
    Coordinate calls across multiple services with automatic retries and failure handling.
  </Card>

  <Card title="Event processing" href="/use-cases/event-processing">
    Process events with exactly-once guarantees and automatic retry handling.
  </Card>
</Columns>

## First time here?

<Columns>
  <Card title="Quickstart" href="/quickstart" icon={"rocket"}>
    Build your first Restate service in minutes.
  </Card>

  <Card title="Concepts" href="/foundations/key-concepts" icon={"cube"}>
    Understand the core building blocks.
  </Card>

  <Card title="Tour of Restate" icon={"graduation-cap"}>
    Learn how to build common applications with Restate:

    [AI agents](/tour/vercel-ai-agents) •
    [Workflows](/tour/workflows) •
    [Microservice orchestration](/tour/microservice-orchestration)
  </Card>

  <Card title="Ask AI" href="?assistant=open" icon={"robot"}>
    Chat with our AI assistant to get answers to your questions about Restate.
  </Card>
</Columns>

## Learning resources

<CardGroup>
  <Card title="Examples" icon="code-branch" href="https://github.com/restatedev/examples">
    A collection of examples that illustrate how to use Restate to solve common application challenges.
  </Card>

  <Card title="AI Recipes" icon="robot" href="/ai">
    Learn how to build durable AI applications with Restate: agents, chatbots, multi-agent systems, ...
  </Card>

  <Card title="Guides" icon="book" href="/guides">
    Learn how to do common tasks with Restate: patterns, integrations, deployment tutorials, ...
  </Card>
</CardGroup>

## Reference

<CardGroup>
  <Card title="SDKs" icon="code">
    Implement Restate applications in one of the available SDKs.

    <div>
      [TypeScript](/develop/ts/services) •
      [Java](/develop/java/services) •
      [Kotlin](/develop/java/services) •
      [Python](/develop/python/services) •
      [Go](/develop/go/services) •
      [Rust](https://docs.rs/restate-sdk/latest/restate_sdk/)
    </div>
  </Card>

  <Card title="Service Lifecycle" icon="box">
    Deploy and operate services on your preferred platform.

    [Deploy](/deploy/services/kubernetes) •
    [Invoke](/services/invocation/http) •
    [Versioning](/services/versioning) •
    [Monitor & Inspect](/services/introspection)
  </Card>

  <Card title="Host Restate" icon="cloud" href="/cloud/getting-started">
    Get started immediately with Restate Cloud, or host your own Restate server.
  </Card>
</CardGroup>

## Community

<CardGroup>
  <Card title="Need help?" icon="circle-info">
    Join the Restate Discord or Slack communities.

    <br />

    <div>
      [<Icon icon="discord" />](https://discord.restate.dev)

      [<Icon icon="slack" />](https://slack.restate.dev)
    </div>
  </Card>

  <Card title="Stay up to date" icon="calendar" href="https://lu.ma/restatedev">
    Follow us on Twitter, LinkedIn, Bluesky.

    <br />

    <div>
      [<Icon icon="twitter" />](https://twitter.com/restatedev)

      [<Icon icon="linkedin" />](https://www.linkedin.com/company/restatedev)

      [<Icon icon="bluesky" />](https://bsky.app/profile/restate.dev)
    </div>
  </Card>
</CardGroup>

<CardGroup>
  <Card title="YouTube Channel" href="https://www.youtube.com/@restatedev">
    <Frame>
      <iframe />
    </Frame>

    Watch intro videos, community meetings and talks about Restate.
  </Card>

  <Card title="Events" href="https://lu.ma/restatedev">
    <Frame>
      <iframe />
    </Frame>

    Subscribe and attend our events.
  </Card>
</CardGroup>


# Installation
Source: https://docs.restate.dev/installation

Learn how to set up your local Restate development environment.

Set up Restate locally in minutes.

## Install Restate Server & CLI

Restate is a single self-contained binary. No external dependencies needed.

<Tabs>
  <Tab title="Homebrew">
    ```shell theme={null}
    brew install restatedev/tap/restate-server restatedev/tap/restate
    ```

    Start the server:

    ```shell theme={null}
    restate-server
    ```

    Find the CLI at:

    ```shell theme={null}
    restate --help
    ```
  </Tab>

  <Tab title="Download binaries">
    Download prebuilt binaries from the [releases page](https://github.com/restatedev/restate/releases/latest):

    <CodeGroup>
      ```shell MacOS-x64 theme={null}
      BIN=/usr/local/bin && RESTATE_PLATFORM=x86_64-apple-darwin && \
      curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
      tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
      tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
      chmod +x restate restate-server && \
      sudo mv restate $BIN && \
      sudo mv restate-server $BIN
      ```

      ```shell MacOS-arm64 theme={null}
      BIN=/usr/local/bin && RESTATE_PLATFORM=aarch64-apple-darwin && \
      curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
      tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
      tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
      chmod +x restate restate-server && \
      sudo mv restate $BIN && \
      sudo mv restate-server $BIN
      ```

      ```shell Linux-x64 theme={null}
      BIN=$HOME/.local/bin && RESTATE_PLATFORM=x86_64-unknown-linux-musl && \
      curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
      tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
      tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
      chmod +x restate restate-server && \
      mv restate $BIN && \
      mv restate-server $BIN
      ```

      ```shell Linux-arm64 theme={null}
      BIN=$HOME/.local/bin && RESTATE_PLATFORM=aarch64-unknown-linux-musl && \
      curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
      tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
      tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
      chmod +x restate restate-server && \
      mv restate $BIN && \
      mv restate-server $BIN
      ```
    </CodeGroup>

    Start the server:

    ```shell theme={null}
    restate-server
    ```

    Find the CLI at:

    ```shell theme={null}
    restate --help
    ```
  </Tab>

  <Tab title="npm">
    ```shell theme={null}
    npm install --global @restatedev/restate-server@latest @restatedev/restate@latest
    ```

    Start the server:

    ```shell theme={null}
    restate-server
    ```

    Find the CLI at:

    ```shell theme={null}
    restate --help
    ```
  </Tab>

  <Tab title="Docker">
    Run the Restate Server:

    ```shell theme={null}
    docker run --name restate_dev --rm \
    -p 8080:8080 -p 9070:9070 -p 9071:9071 \
    --add-host=host.docker.internal:host-gateway \
    docker.restate.dev/restatedev/restate:latest
    ```

    Run CLI commands:

    ```shell theme={null}
    docker run -it --network=host \
    docker.restate.dev/restatedev/restate-cli:latest \
    invocations ls
    ```

    Replace `invocations ls` with any CLI subcommand.
  </Tab>
</Tabs>

You can find the Restate UI running on port 9070 (`http://localhost:9070`) after starting the Restate Server.

<AccordionGroup>
  <Accordion title="Lightweight local dev server">
    You can use `restate up` to start a lightweight, local dev server.
  </Accordion>

  <Accordion title="Random ports">
    You can start `restate-server` binding to random ports via:

    ```shell theme={null}
    restate-server --use-random-ports=true --no-logo
    ```

    The chosen ports will be displayed in the first three standard output lines.
  </Accordion>
</AccordionGroup>

Also check out the [CLI](/references/cli-config) or [Server configuration options](/server/configuration).

<Info title="Telemetry">
  Restate Server collects anonymized telemetry about versions and uptime via [Scarf](https://about.scarf.sh).\
  We don't have access to your IP or any information about your cluster.
  To disable, set `DO_NOT_TRACK=1`.
</Info>

## Restate UI

The UI is bundled with the Restate Server and available at [http://localhost:9070](http://localhost:9070) when running locally. Use the UI to manage, debug, and configure your applications.

<Frame>
  <img alt="Restart from prefix" />
</Frame>

The UI gives you the best experience when working with Restate.

## Useful CLI Commands

With the CLI installed, try these commands:

<AccordionGroup>
  <Accordion title="Check which server you are connected to">
    ```shell theme={null}
    restate whoami
    ```
  </Accordion>

  <Accordion title="Register a new service deployment">
    (If using Docker, use `http://host.docker.internal:9080`)

    ```shell theme={null}
    restate deployments register localhost:9080
    ```
  </Accordion>

  <Accordion title="List deployments">
    ```shell theme={null}
    restate deployments list
    ```
  </Accordion>

  <Accordion title="List invocations">
    ```shell theme={null}
    restate invocation list
    ```
  </Accordion>

  <Accordion title="Cancel/kill a single or batch of invocations">
    ```shell theme={null}
    # a single invocation
    restate invocation cancel my_invocation_id
    # cancel all invocations of a service, object, or handler
    restate invocation cancel MyService
    restate invocation cancel MyService/myHandler
    restate invocation cancel MyObject/myObjectKey
    restate invocation cancel MyObject/myObjectKey/myHandler
    ```

    Use `restate invocation kill` to force kill.
  </Accordion>

  <Accordion title="Clear the K/V state of a Virtual Object or Workflow">
    ```shell theme={null}
    restate kv clear MyObject
    restate kv clear MyObject/myObjectKey
    ```
  </Accordion>

  <Accordion title="Wipe the server during local development">
    Remove the `restate-data` directory to wipe all invocations, state, registered services, etc.

    ```shell theme={null}
    rm -rf restate-data
    ```
  </Accordion>

  <Accordion title="Execute a SQL query on invocation or application state">
    See [SQL introspection docs](/services/introspection#inspecting-invocations) for examples.
    Use `--json` for JSON output.

    ```shell theme={null}
    restate sql "query"
    ```
  </Accordion>

  <Accordion title="View your service configuration">
    ```shell theme={null}
    restate service config view
    ```
  </Accordion>
</AccordionGroup>

See also the [introspection page](/services/introspection) for more CLI debugging commands.

## Advanced: Installing `restatectl`

`restatectl` is a command-line tool for managing Restate clusters. It provides commands for cluster management, introspection, and debugging.

This tool is specifically designed for system operators to manage Restate servers and is particularly useful in a cluster environment.

<Tabs>
  <Tab title="Homebrew">
    Install with:

    ```shell theme={null}
    brew install restatedev/tap/restatectl
    ```

    Then run:

    ```shell theme={null}
    restatectl --help
    ```
  </Tab>

  <Tab title="Download binaries">
    Install restatectl by downloading the binary with `curl` from the [releases page](https://github.com/restatedev/restate/releases/latest), and make them executable:

    <CodeGroup>
      ```shell Linux-x64 theme={null}
      BIN=$HOME/.local/bin && RESTATE_PLATFORM=x86_64-unknown-linux-musl && \
      curl -LO https://restate.gateway.scarf.sh/latest/restatectl-$RESTATE_PLATFORM.tar.xz && \
      tar -xvf restatectl-$RESTATE_PLATFORM.tar.xz --strip-components=1 restatectl-$RESTATE_PLATFORM/restatectl && \
      chmod +x restatectl && \

      # Move the binary to a directory in your PATH, for example ~/.local/bin:
      mv restatectl $BIN
      ```

      ```shell Linux-arm64 theme={null}
      BIN=$HOME/.local/bin && RESTATE_PLATFORM=aarch64-unknown-linux-musl && \
      curl -LO https://restate.gateway.scarf.sh/latest/restatectl-$RESTATE_PLATFORM.tar.xz && \
      tar -xvf restatectl-$RESTATE_PLATFORM.tar.xz --strip-components=1 restatectl-$RESTATE_PLATFORM/restatectl && \
      chmod +x restatectl && \

      # Move the binary to a directory in your PATH, for example ~/.local/bin:
      mv restatectl $BIN
      ```

      ```shell MacOS-x64 theme={null}
      BIN=/usr/local/bin && RESTATE_PLATFORM=x86_64-apple-darwin && \
      curl -LO https://restate.gateway.scarf.sh/latest/restatectl-$RESTATE_PLATFORM.tar.xz && \
      tar -xvf restatectl-$RESTATE_PLATFORM.tar.xz --strip-components=1 restatectl-$RESTATE_PLATFORM/restatectl && \
      chmod +x restatectl && \

      # Move the binary to a directory in your PATH, for example /usr/local/bin (needs sudo):
      sudo mv restatectl $BIN
      ```

      ```shell MacOS-arm64 theme={null}
      BIN=/usr/local/bin && RESTATE_PLATFORM=aarch64-apple-darwin && \
      curl -LO https://restate.gateway.scarf.sh/latest/restatectl-$RESTATE_PLATFORM.tar.xz && \
      tar -xvf restatectl-$RESTATE_PLATFORM.tar.xz --strip-components=1 restatectl-$RESTATE_PLATFORM/restatectl && \
      chmod +x restatectl && \

      # Move the binary to a directory in your PATH, for example /usr/local/bin (needs sudo):
      sudo mv restatectl $BIN
      ```
    </CodeGroup>

    Then run:

    ```shell theme={null}
    restatectl --help
    ```
  </Tab>

  <Tab title="npm">
    Install with:

    ```shell theme={null}
    npm install --global @restatedev/restatectl@latest
    ```

    Then run:

    ```shell theme={null}
    restatectl --help
    ```
  </Tab>

  <Tab title="Docker">
    The server image contains the `restatectl` tool. To run `restatectl`, use the following command:

    ```shell theme={null}
    docker run -it --network=host --entrypoint restatectl docker.restate.dev/restatedev/restate:latest nodes ls
    ```

    You can also execute `restatectl` in a running server container using the following command:

    ```shell theme={null}
    docker exec restate_dev restatectl nodes ls
    ```

    Replace `restate_dev` with the name of a running container, and `nodes ls` with the subcommand you want to run.
  </Tab>
</Tabs>

<Info>
  `restatectl` requires direct access to nodes via their advertised addresses (default port: 5122). Ensure that restatectl commands can reach these addresses.
</Info>


# Quickstart
Source: https://docs.restate.dev/quickstart

Develop and run your first Restate service

This guide takes you through your first steps with Restate.

We will run a simple Restate Greeter service that listens on port `9080` and responds with `You said hi to <name>!` to a `greet` request.

<img alt="Quickstart" />

Select your SDK:

<Tabs>
  <Tab title="TypeScript" icon={"/img/languages/typescript.svg"}>
    <iframe title="YouTube video player" />

    Select your favorite runtime:

    <Tabs>
      <Tab title="Node.js" icon={"/img/quickstart/nodejs.svg"}>
        <Info>
          **Prerequisites**:

          * [NodeJS](https://nodejs.org/en/) >= v20
        </Info>

        <Steps>
          <Step title="Install Restate Server & CLI">
            Restate is a single self-contained binary. No external dependencies needed.

            <Tabs>
              <Tab title="Homebrew">
                ```shell theme={null}
                brew install restatedev/tap/restate-server restatedev/tap/restate
                ```

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="Download binaries">
                Download prebuilt binaries from the [releases page](https://github.com/restatedev/restate/releases/latest):

                <CodeGroup>
                  ```shell MacOS-x64 theme={null}
                  BIN=/usr/local/bin && RESTATE_PLATFORM=x86_64-apple-darwin && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  sudo mv restate $BIN && \
                  sudo mv restate-server $BIN
                  ```

                  ```shell MacOS-arm64 theme={null}
                  BIN=/usr/local/bin && RESTATE_PLATFORM=aarch64-apple-darwin && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  sudo mv restate $BIN && \
                  sudo mv restate-server $BIN
                  ```

                  ```shell Linux-x64 theme={null}
                  BIN=$HOME/.local/bin && RESTATE_PLATFORM=x86_64-unknown-linux-musl && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  mv restate $BIN && \
                  mv restate-server $BIN
                  ```

                  ```shell Linux-arm64 theme={null}
                  BIN=$HOME/.local/bin && RESTATE_PLATFORM=aarch64-unknown-linux-musl && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  mv restate $BIN && \
                  mv restate-server $BIN
                  ```
                </CodeGroup>

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="npm">
                ```shell theme={null}
                npm install --global @restatedev/restate-server@latest @restatedev/restate@latest
                ```

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="Docker">
                Run the Restate Server:

                ```shell theme={null}
                docker run --name restate_dev --rm \
                -p 8080:8080 -p 9070:9070 -p 9071:9071 \
                --add-host=host.docker.internal:host-gateway \
                docker.restate.dev/restatedev/restate:latest
                ```

                Run CLI commands:

                ```shell theme={null}
                docker run -it --network=host \
                docker.restate.dev/restatedev/restate-cli:latest \
                invocations ls
                ```

                Replace `invocations ls` with any CLI subcommand.
              </Tab>
            </Tabs>

            You can find the Restate UI running on port 9070 (`http://localhost:9070`) after starting the Restate Server.
          </Step>

          <Step title="Get the Greeter service template">
            <CodeGroup>
              ```shell CLI theme={null}
              restate example typescript-hello-world &&
              cd typescript-hello-world &&
              npm install
              ```

              ```shell npx theme={null}
              npx -y @restatedev/create-app@latest && cd restate-node-template &&
              npm install
              ```
            </CodeGroup>

            <GitHubLink />
          </Step>

          <Step title="Run the Greeter service">
            ```shell theme={null}
            npm run dev
            ```
          </Step>

          <Step title="Register the service">
            Tell Restate where the service is running, so Restate can discover and register the services and handlers behind this endpoint.
            You can do this via the UI (`http://localhost:9070`) or via:

            <CodeGroup>
              ```shell CLI theme={null}
              restate deployments register http://localhost:9080
              ```

              ```shell curl theme={null}
              curl localhost:9070/deployments --json '{"uri": "http://localhost:9080"}'
              ```
            </CodeGroup>

            <Expandable title="Output">
              <CodeGroup>
                ```shell CLI theme={null}
                ❯ SERVICES THAT WILL BE ADDED:
                - Greeter
                Type: Service
                HANDLER  INPUT                                     OUTPUT
                greet    value of content-type 'application/json'  value of content-type 'application/json'


                ✔ Are you sure you want to apply those changes? · yes
                ✅ DEPLOYMENT:
                SERVICE  REV
                Greeter  1
                ```

                ```shell curl theme={null}
                {
                    "id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                    "services": [
                        {
                            "name": "Greeter",
                            "handlers": [
                                {
                                    "name": "greet",
                                    "ty": "Shared",
                                    "input_description": "one of [\"none\", \"value of content-type 'application/json'\"]",
                                    "output_description": "value of content-type 'application/json'"
                                }
                            ],
                            "ty": "Service",
                            "deployment_id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                            "revision": 1,
                            "public": true,
                            "idempotency_retention": "1day"
                        }
                    ]
                }
                ```
              </CodeGroup>
            </Expandable>

            If you run Restate with Docker, register `http://host.docker.internal:9080` instead of `http://localhost:9080`.

            <Accordion title="Restate Cloud">
              When using [Restate Cloud](https://restate.dev/cloud), your service must be accessible over the public internet so Restate can invoke it.
              If you want to develop with a local service, you can expose it using our [tunnel](/deploy/server/cloud/#registering-restate-services-with-your-environment) feature.
            </Accordion>
          </Step>

          <Step title="Send a request to the Greeter service">
            Invoke the service via the Restate UI playground: go to `http://localhost:9070`,  click on your service and then on playground.

            <img alt="Restate UI Playground" />

            Or invoke via `curl`:

            ```shell theme={null}
            curl localhost:8080/Greeter/greet --json '{"name": "Sarah"}'
            ```

            Expected output: `You said hi to Sarah!`
          </Step>

          <Step title="Congratulations, you just ran Durable Execution!">
            The invocation you just sent used Durable Execution to make sure the request ran till completion.
            For each request, it sent a notification, slept for a second, and then sent a reminder.

            ```ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/templates/node/src/app.ts?collapse_prequel"}  theme={null}
            const greeter = restate.service({
              name: "Greeter",
              handlers: {
                greet: restate.createServiceHandler(
                  { input: restate.serde.schema(Greeting), output: restate.serde.schema(GreetingResponse) },
                  async (ctx: restate.Context, { name }) => {
                    // Durably execute a set of steps; resilient against failures
                    const greetingId = ctx.rand.uuidv4();
                    await ctx.run("Notification", () => sendNotification(greetingId, name));
                    await ctx.sleep({ seconds: 1 });
                    await ctx.run("Reminder", () => sendReminder(greetingId, name));

                    // Respond to caller
                    return { result: `You said hi to ${name}!` };
                  },
                ),
              },
            });

            restate.serve({
              services: [greeter],
              port: 9080,
            });
            ```

            <GitHubLink />

            <Accordion title="See how a failing request is retried">
              Send a request for `Alice` to see how the service behaves when it occasionally fails to send the reminder and notification:

              ```shell theme={null}
              curl localhost:8080/Greeter/greet --json '{"name": "Alice"}'
              ```

              Expected output:

              ```shell theme={null}
              You said hi to Alice!
              ```

              You can see in the service logs and in the Restate UI how the request gets retried.
              On a retry, it skipped the steps that already succeeded.

              <img alt="Restate UI Durable Execution" />

              Even the sleep is durable and tracked by Restate.
              If you kill/restart the service halfway through, the sleep will only last for what remained.

              <Tip title="Durable Execution">
                Restate persists the progress of the handler.
                Letting you write code that is resilient to failures out of the box.
                Have a look at the [Durable Execution page](/foundations/key-concepts#durable-execution) to learn more.
              </Tip>
            </Accordion>
          </Step>
        </Steps>
      </Tab>

      <Tab title="Bun" icon={"/img/quickstart/bun.svg"}>
        <Info>
          **Prerequisites**:

          * [Bun](https://bun.sh/docs/installation)
          * [npm CLI](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm) >= 9.6.7
        </Info>

        <Steps>
          <Step title="Install Restate Server & CLI">
            Restate is a single self-contained binary. No external dependencies needed.

            <Tabs>
              <Tab title="Homebrew">
                ```shell theme={null}
                brew install restatedev/tap/restate-server restatedev/tap/restate
                ```

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="Download binaries">
                Download prebuilt binaries from the [releases page](https://github.com/restatedev/restate/releases/latest):

                <CodeGroup>
                  ```shell MacOS-x64 theme={null}
                  BIN=/usr/local/bin && RESTATE_PLATFORM=x86_64-apple-darwin && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  sudo mv restate $BIN && \
                  sudo mv restate-server $BIN
                  ```

                  ```shell MacOS-arm64 theme={null}
                  BIN=/usr/local/bin && RESTATE_PLATFORM=aarch64-apple-darwin && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  sudo mv restate $BIN && \
                  sudo mv restate-server $BIN
                  ```

                  ```shell Linux-x64 theme={null}
                  BIN=$HOME/.local/bin && RESTATE_PLATFORM=x86_64-unknown-linux-musl && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  mv restate $BIN && \
                  mv restate-server $BIN
                  ```

                  ```shell Linux-arm64 theme={null}
                  BIN=$HOME/.local/bin && RESTATE_PLATFORM=aarch64-unknown-linux-musl && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  mv restate $BIN && \
                  mv restate-server $BIN
                  ```
                </CodeGroup>

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="npm">
                ```shell theme={null}
                npm install --global @restatedev/restate-server@latest @restatedev/restate@latest
                ```

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="Docker">
                Run the Restate Server:

                ```shell theme={null}
                docker run --name restate_dev --rm \
                -p 8080:8080 -p 9070:9070 -p 9071:9071 \
                --add-host=host.docker.internal:host-gateway \
                docker.restate.dev/restatedev/restate:latest
                ```

                Run CLI commands:

                ```shell theme={null}
                docker run -it --network=host \
                docker.restate.dev/restatedev/restate-cli:latest \
                invocations ls
                ```

                Replace `invocations ls` with any CLI subcommand.
              </Tab>
            </Tabs>

            You can find the Restate UI running on port 9070 (`http://localhost:9070`) after starting the Restate Server.
          </Step>

          <Step title="Get the Greeter service template">
            ```shell theme={null}
            restate example typescript-hello-world-bun &&
            cd typescript-hello-world-bun &&
            npm install
            ```

            <GitHubLink />
          </Step>

          <Step title="Run the Greeter service">
            ```shell theme={null}
            npm run dev
            ```
          </Step>

          <Step title="Register the service">
            Tell Restate where the service is running, so Restate can discover and register the services and handlers behind this endpoint.
            You can do this via the UI (`http://localhost:9070`) or via:

            <CodeGroup>
              ```shell CLI theme={null}
              restate deployments register http://localhost:9080
              ```

              ```shell curl theme={null}
              curl localhost:9070/deployments --json '{"uri": "http://localhost:9080"}'
              ```
            </CodeGroup>

            <Expandable title="Output">
              <CodeGroup>
                ```shell CLI theme={null}
                ❯ SERVICES THAT WILL BE ADDED:
                - Greeter
                Type: Service
                HANDLER  INPUT                                     OUTPUT
                greet    value of content-type 'application/json'  value of content-type 'application/json'


                ✔ Are you sure you want to apply those changes? · yes
                ✅ DEPLOYMENT:
                SERVICE  REV
                Greeter  1
                ```

                ```shell curl theme={null}
                {
                    "id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                    "services": [
                        {
                            "name": "Greeter",
                            "handlers": [
                                {
                                    "name": "greet",
                                    "ty": "Shared",
                                    "input_description": "one of [\"none\", \"value of content-type 'application/json'\"]",
                                    "output_description": "value of content-type 'application/json'"
                                }
                            ],
                            "ty": "Service",
                            "deployment_id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                            "revision": 1,
                            "public": true,
                            "idempotency_retention": "1day"
                        }
                    ]
                }
                ```
              </CodeGroup>
            </Expandable>

            If you run Restate with Docker, register `http://host.docker.internal:9080` instead of `http://localhost:9080`.

            <Accordion title="Restate Cloud">
              When using [Restate Cloud](https://restate.dev/cloud), your service must be accessible over the public internet so Restate can invoke it.
              If you want to develop with a local service, you can expose it using our [tunnel](/deploy/server/cloud/#registering-restate-services-with-your-environment) feature.
            </Accordion>
          </Step>

          <Step title="Send a request to the Greeter service">
            Invoke the service via the Restate UI playground: go to `http://localhost:9070`,  click on your service and then on playground.

            <img alt="Restate UI Playground" />

            Or invoke via `curl`:

            ```shell theme={null}
            curl localhost:8080/Greeter/greet --json '{"name": "Sarah"}'
            ```

            Expected output: `You said hi to Sarah!`
          </Step>

          <Step title="Congratulations, you just ran Durable Execution!">
            The invocation you just sent used Durable Execution to make sure the request ran till completion.
            For each request, it sent a notification, slept for a second, and then sent a reminder.

            ```ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/templates/bun/src/index.ts?collapse_prequel"}  theme={null}
            const greeter = restate.service({
              name: "Greeter",
              handlers: {
                greet: restate.createServiceHandler(
                  { input: restate.serde.schema(Greeting), output: restate.serde.schema(GreetingResponse) },
                  async (ctx: restate.Context, { name }) => {
                    // Durably execute a set of steps; resilient against failures
                    const greetingId = ctx.rand.uuidv4();
                    await ctx.run("Notification", () => sendNotification(greetingId, name));
                    await ctx.sleep({ seconds: 1 });
                    await ctx.run("Reminder", () => sendReminder(greetingId, name));

                    // Respond to caller
                    return { result: `You said hi to ${name}!` };
                  },
                ),
              },
            });

            restate.serve({
              services: [greeter],
              port: 9080,
            });
            ```

            <GitHubLink />

            Send a request for `Alice` to see how the service behaves when it occasionally fails to send the reminder and notification:

            ```shell theme={null}
            curl localhost:8080/Greeter/greet --json '{"name": "Alice"}'
            ```

            Expected output:

            ```shell theme={null}
            You said hi to Alice!
            ```

            You can see in the service logs and in the Restate UI how the request gets retried.
            On a retry, it skipped the steps that already succeeded.

            <img alt="Restate UI Durable Execution" />

            Even the sleep is durable and tracked by Restate.
            If you kill/restart the service halfway through, the sleep will only last for what remained.

            <Tip title="Durable Execution">
              Restate persists the progress of the handler.
              Letting you write code that is resilient to failures out of the box.
              Have a look at the [Durable Execution page](/foundations/key-concepts#durable-execution) to learn more.
            </Tip>
          </Step>
        </Steps>
      </Tab>

      <Tab title="Deno" icon={"/img/quickstart/deno.svg"}>
        <Info>
          **Prerequisites**:

          * [Deno](https://deno.land/#installation)
          * [npm CLI](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm) >= 9.6.7
        </Info>

        <Steps>
          <Step title="Install Restate Server & CLI">
            Restate is a single self-contained binary. No external dependencies needed.

            <Tabs>
              <Tab title="Homebrew">
                ```shell theme={null}
                brew install restatedev/tap/restate-server restatedev/tap/restate
                ```

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="Download binaries">
                Download prebuilt binaries from the [releases page](https://github.com/restatedev/restate/releases/latest):

                <CodeGroup>
                  ```shell MacOS-x64 theme={null}
                  BIN=/usr/local/bin && RESTATE_PLATFORM=x86_64-apple-darwin && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  sudo mv restate $BIN && \
                  sudo mv restate-server $BIN
                  ```

                  ```shell MacOS-arm64 theme={null}
                  BIN=/usr/local/bin && RESTATE_PLATFORM=aarch64-apple-darwin && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  sudo mv restate $BIN && \
                  sudo mv restate-server $BIN
                  ```

                  ```shell Linux-x64 theme={null}
                  BIN=$HOME/.local/bin && RESTATE_PLATFORM=x86_64-unknown-linux-musl && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  mv restate $BIN && \
                  mv restate-server $BIN
                  ```

                  ```shell Linux-arm64 theme={null}
                  BIN=$HOME/.local/bin && RESTATE_PLATFORM=aarch64-unknown-linux-musl && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  mv restate $BIN && \
                  mv restate-server $BIN
                  ```
                </CodeGroup>

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="npm">
                ```shell theme={null}
                npm install --global @restatedev/restate-server@latest @restatedev/restate@latest
                ```

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="Docker">
                Run the Restate Server:

                ```shell theme={null}
                docker run --name restate_dev --rm \
                -p 8080:8080 -p 9070:9070 -p 9071:9071 \
                --add-host=host.docker.internal:host-gateway \
                docker.restate.dev/restatedev/restate:latest
                ```

                Run CLI commands:

                ```shell theme={null}
                docker run -it --network=host \
                docker.restate.dev/restatedev/restate-cli:latest \
                invocations ls
                ```

                Replace `invocations ls` with any CLI subcommand.
              </Tab>
            </Tabs>

            You can find the Restate UI running on port 9070 (`http://localhost:9070`) after starting the Restate Server.
          </Step>

          <Step title="Get the Greeter service template">
            ```shell theme={null}
            restate example typescript-hello-world-deno &&
            cd typescript-hello-world-deno
            ```

            <GitHubLink />
          </Step>

          <Step title="Run the Greeter service">
            ```shell theme={null}
            deno task dev
            ```
          </Step>

          <Step title="Register the service">
            Tell Restate where the service is running, so Restate can discover and register the services and handlers behind this endpoint.
            You can do this via the UI (`http://localhost:9070`) or via:

            <CodeGroup>
              ```shell CLI theme={null}
              restate deployments register http://localhost:9080
              ```

              ```shell curl theme={null}
              curl localhost:9070/deployments --json '{"uri": "http://localhost:9080"}'
              ```
            </CodeGroup>

            <Expandable title="Output">
              <CodeGroup>
                ```shell CLI theme={null}
                ❯ SERVICES THAT WILL BE ADDED:
                - Greeter
                Type: Service
                HANDLER  INPUT                                     OUTPUT
                greet    value of content-type 'application/json'  value of content-type 'application/json'


                ✔ Are you sure you want to apply those changes? · yes
                ✅ DEPLOYMENT:
                SERVICE  REV
                Greeter  1
                ```

                ```shell curl theme={null}
                {
                    "id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                    "services": [
                        {
                            "name": "Greeter",
                            "handlers": [
                                {
                                    "name": "greet",
                                    "ty": "Shared",
                                    "input_description": "one of [\"none\", \"value of content-type 'application/json'\"]",
                                    "output_description": "value of content-type 'application/json'"
                                }
                            ],
                            "ty": "Service",
                            "deployment_id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                            "revision": 1,
                            "public": true,
                            "idempotency_retention": "1day"
                        }
                    ]
                }
                ```
              </CodeGroup>
            </Expandable>

            If you run Restate with Docker, register `http://host.docker.internal:9080` instead of `http://localhost:9080`.

            <Accordion title="Restate Cloud">
              When using [Restate Cloud](https://restate.dev/cloud), your service must be accessible over the public internet so Restate can invoke it.
              If you want to develop with a local service, you can expose it using our [tunnel](/deploy/server/cloud/#registering-restate-services-with-your-environment) feature.
            </Accordion>
          </Step>

          <Step title="Send a request to the Greeter service">
            Invoke the service via the Restate UI playground: go to `http://localhost:9070`,  click on your service and then on playground.

            <img alt="Restate UI Playground" />

            Or invoke via `curl`:

            ```shell theme={null}
            curl localhost:8080/Greeter/greet --json '{"name": "Sarah"}'
            ```

            Expected output: `You said hi to Sarah!`
          </Step>

          <Step title="Congratulations, you just ran Durable Execution!">
            The invocation you just sent used Durable Execution to make sure the request ran till completion.
            For each request, it sent a notification, slept for a second, and then sent a reminder.

            ```ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/templates/deno/main.ts?collapse_prequel"}  theme={null}
            export const greeter = restate.service({
              name: "Greeter",
              handlers: {
                greet: restate.createServiceHandler(
                  { input: restate.serde.schema(Greeting), output: restate.serde.schema(GreetingResponse) },
                  async (ctx: restate.Context, { name }) => {
                    // Durably execute a set of steps; resilient against failures
                    const greetingId = ctx.rand.uuidv4();
                    await ctx.run("Notification", () => sendNotification(greetingId, name));
                    await ctx.sleep({ seconds: 1 });
                    await ctx.run("Reminder", () => sendReminder(greetingId, name));

                    // Respond to caller
                    return { result: `You said hi to ${name}!` };
                  },
                ),
              },
            });

            const handler = restate.createEndpointHandler({
              services: [greeter],
              bidirectional: true,
            });

            Deno.serve({ port: 9080 }, handler);
            ```

            <GitHubLink />

            Send a request for `Alice` to see how the service behaves when it occasionally fails to send the reminder and notification:

            ```shell theme={null}
            curl localhost:8080/Greeter/greet --json '{"name": "Alice"}'
            ```

            Expected output:

            ```shell theme={null}
            You said hi to Alice!
            ```

            You can see in the service logs and in the Restate UI how the request gets retried.
            On a retry, it skipped the steps that already succeeded.

            <img alt="Restate UI Durable Execution" />

            Even the sleep is durable and tracked by Restate.
            If you kill/restart the service halfway through, the sleep will only last for what remained.

            <Tip title="Durable Execution">
              Restate persists the progress of the handler.
              Letting you write code that is resilient to failures out of the box.
              Have a look at the [Durable Execution page](/foundations/key-concepts#durable-execution) to learn more.
            </Tip>
          </Step>
        </Steps>
      </Tab>

      <Tab title="Cloudflare Workers" icon={"/img/quickstart/cloudflare_workers.svg"}>
        <Info>
          **Prerequisites**:

          * [NodeJS](https://nodejs.org/en/) >= v20
        </Info>

        <Steps>
          <Step title="Install Restate Server & CLI">
            Restate is a single self-contained binary. No external dependencies needed.

            <Tabs>
              <Tab title="Homebrew">
                ```shell theme={null}
                brew install restatedev/tap/restate-server restatedev/tap/restate
                ```

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="Download binaries">
                Download prebuilt binaries from the [releases page](https://github.com/restatedev/restate/releases/latest):

                <CodeGroup>
                  ```shell MacOS-x64 theme={null}
                  BIN=/usr/local/bin && RESTATE_PLATFORM=x86_64-apple-darwin && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  sudo mv restate $BIN && \
                  sudo mv restate-server $BIN
                  ```

                  ```shell MacOS-arm64 theme={null}
                  BIN=/usr/local/bin && RESTATE_PLATFORM=aarch64-apple-darwin && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  sudo mv restate $BIN && \
                  sudo mv restate-server $BIN
                  ```

                  ```shell Linux-x64 theme={null}
                  BIN=$HOME/.local/bin && RESTATE_PLATFORM=x86_64-unknown-linux-musl && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  mv restate $BIN && \
                  mv restate-server $BIN
                  ```

                  ```shell Linux-arm64 theme={null}
                  BIN=$HOME/.local/bin && RESTATE_PLATFORM=aarch64-unknown-linux-musl && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  mv restate $BIN && \
                  mv restate-server $BIN
                  ```
                </CodeGroup>

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="npm">
                ```shell theme={null}
                npm install --global @restatedev/restate-server@latest @restatedev/restate@latest
                ```

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="Docker">
                Run the Restate Server:

                ```shell theme={null}
                docker run --name restate_dev --rm \
                -p 8080:8080 -p 9070:9070 -p 9071:9071 \
                --add-host=host.docker.internal:host-gateway \
                docker.restate.dev/restatedev/restate:latest
                ```

                Run CLI commands:

                ```shell theme={null}
                docker run -it --network=host \
                docker.restate.dev/restatedev/restate-cli:latest \
                invocations ls
                ```

                Replace `invocations ls` with any CLI subcommand.
              </Tab>
            </Tabs>

            You can find the Restate UI running on port 9070 (`http://localhost:9070`) after starting the Restate Server.
          </Step>

          <Step title="Get the Greeter service template">
            ```shell theme={null}
            restate example typescript-hello-world-cloudflare-worker &&
            cd typescript-hello-world-cloudflare-worker &&
            npm install
            ```

            <GitHubLink />
          </Step>

          <Step title="Run the Greeter service">
            ```shell theme={null}
            npm run dev
            ```
          </Step>

          <Step title="Register the service">
            Tell Restate where the service is running, so Restate can discover and register the services and handlers behind this endpoint.
            You can do this via the UI (`http://localhost:9070`) or via:

            <CodeGroup>
              ```shell CLI theme={null}
              restate deployments register http://localhost:9080 --use-http1.1
              ```

              ```shell curl theme={null}
              curl localhost:9070/deployments --json '{"uri": "http://localhost:9080", "use_http_11": true}'
              ```
            </CodeGroup>

            Expected output:

            <CodeGroup>
              ```shell CLI theme={null}
              ❯ SERVICES THAT WILL BE ADDED:
              - Greeter
              Type: Service
              HANDLER  INPUT                                     OUTPUT
              greet    value of content-type 'application/json'  value of content-type 'application/json'


              ✔ Are you sure you want to apply those changes? · yes
              ✅ DEPLOYMENT:
              SERVICE  REV
              Greeter  1
              ```

              ```shell curl theme={null}
              {
              "id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                  "services": [
                      {
                          "name": "Greeter",
                          "handlers": [
                              {
                                  "name": "greet",
                                  "ty": "Shared",
                                  "input_description": "one of [\"none\", \"value of content-type 'application/json'\"]",
                                  "output_description": "value of content-type 'application/json'"
                              }
                          ],
                          "ty": "Service",
                          "deployment_id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                          "revision": 1,
                          "public": true,
                          "idempotency_retention": "1day"
                      }
                  ]
              }
              ```
            </CodeGroup>

            &#x20;does not support HTTP2, so we need to tell Restate to use HTTP1.1.

            If you run Restate with Docker, use `http://host.docker.internal:9080` instead of `http://localhost:9080`.

            <Info title="Restate Cloud">
              When using [Restate Cloud](https://restate.dev/cloud), your service must be accessible over the public internet so Restate can invoke it.
              If you want to develop with a local service, you can expose it using our [tunnel](/deploy/server/cloud/#registering-restate-services-with-your-environment) feature.
            </Info>
          </Step>

          <Step title="Send a request to the Greeter service">
            Invoke the service via the Restate UI playground: go to `http://localhost:9070`,  click on your service and then on playground.

            <img alt="Restate UI Playground" />

            Or invoke via `curl`:

            ```shell theme={null}
            curl localhost:8080/Greeter/greet --json '{"name": "Sarah"}'
            ```

            Expected output: `You said hi to Sarah!`
          </Step>

          <Step title="Congratulations, you just ran Durable Execution!">
            The invocation you just sent used Durable Execution to make sure the request ran till completion.
            For each request, it sent a notification, slept for a second, and then sent a reminder.

            ```ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/templates/cloudflare-worker/src/index.ts?collapse_prequel"}  theme={null}
            const greeter = restate.service({
              name: "Greeter",
              handlers: {
                greet: restate.createServiceHandler(
                  { input: restate.serde.schema(Greeting), output: restate.serde.schema(GreetingResponse) },
                  async (ctx: restate.Context, { name }) => {
                    // Durably execute a set of steps; resilient against failures
                    const greetingId = ctx.rand.uuidv4();
                    await ctx.run("Notification", () => sendNotification(greetingId, name));
                    await ctx.sleep({ seconds: 1 });
                    await ctx.run("Reminder", () => sendReminder(greetingId, name));

                    // Respond to caller
                    return { result: `You said hi to ${name}!` };
                  },
                ),
              },
            });

            export default {
                fetch: restate.createEndpointHandler({ services: [greeter] })
            };
            ```

            <GitHubLink />

            Send a request for `Alice` to see how the service behaves when it occasionally fails to send the reminder and notification:

            ```shell theme={null}
            curl localhost:8080/Greeter/greet --json '{"name": "Alice"}'
            ```

            Expected output:

            ```shell theme={null}
            You said hi to Alice!
            ```

            You can see in the service logs and in the Restate UI how the request gets retried.
            On a retry, it skipped the steps that already succeeded.

            <img alt="Restate UI Durable Execution" />

            Even the sleep is durable and tracked by Restate.
            If you kill/restart the service halfway through, the sleep will only last for what remained.

            <Tip title="Durable Execution">
              Restate persists the progress of the handler.
              Letting you write code that is resilient to failures out of the box.
              Have a look at the [Durable Execution page](/foundations/key-concepts#durable-execution) to learn more.
            </Tip>
          </Step>
        </Steps>
      </Tab>

      <Tab title="Next.js/Vercel" icon={"/img/quickstart/nextjs.svg"}>
        <Info>
          **Prerequisites**:

          * [NodeJS](https://nodejs.org/en/) >= v20
        </Info>

        <Steps>
          <Step title="Install Restate Server & CLI">
            Restate is a single self-contained binary. No external dependencies needed.

            <Tabs>
              <Tab title="Homebrew">
                ```shell theme={null}
                brew install restatedev/tap/restate-server restatedev/tap/restate
                ```

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="Download binaries">
                Download prebuilt binaries from the [releases page](https://github.com/restatedev/restate/releases/latest):

                <CodeGroup>
                  ```shell MacOS-x64 theme={null}
                  BIN=/usr/local/bin && RESTATE_PLATFORM=x86_64-apple-darwin && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  sudo mv restate $BIN && \
                  sudo mv restate-server $BIN
                  ```

                  ```shell MacOS-arm64 theme={null}
                  BIN=/usr/local/bin && RESTATE_PLATFORM=aarch64-apple-darwin && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  sudo mv restate $BIN && \
                  sudo mv restate-server $BIN
                  ```

                  ```shell Linux-x64 theme={null}
                  BIN=$HOME/.local/bin && RESTATE_PLATFORM=x86_64-unknown-linux-musl && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  mv restate $BIN && \
                  mv restate-server $BIN
                  ```

                  ```shell Linux-arm64 theme={null}
                  BIN=$HOME/.local/bin && RESTATE_PLATFORM=aarch64-unknown-linux-musl && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  mv restate $BIN && \
                  mv restate-server $BIN
                  ```
                </CodeGroup>

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="npm">
                ```shell theme={null}
                npm install --global @restatedev/restate-server@latest @restatedev/restate@latest
                ```

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="Docker">
                Run the Restate Server:

                ```shell theme={null}
                docker run --name restate_dev --rm \
                -p 8080:8080 -p 9070:9070 -p 9071:9071 \
                --add-host=host.docker.internal:host-gateway \
                docker.restate.dev/restatedev/restate:latest
                ```

                Run CLI commands:

                ```shell theme={null}
                docker run -it --network=host \
                docker.restate.dev/restatedev/restate-cli:latest \
                invocations ls
                ```

                Replace `invocations ls` with any CLI subcommand.
              </Tab>
            </Tabs>

            You can find the Restate UI running on port 9070 (`http://localhost:9070`) after starting the Restate Server.
          </Step>

          <Step title="Get the Greeter service template">
            ```shell theme={null}
            restate example typescript-hello-world-vercel &&
            cd typescript-hello-world-vercel &&
            npm install
            ```

            <GitHubLink />
          </Step>

          <Step title="Run the Greeter service">
            ```shell theme={null}
            npm run dev
            ```
          </Step>

          <Step title="Register the service">
            <CodeGroup>
              ```shell CLI theme={null}
              restate deployments register http://localhost:3000/restate --use-http1.1
              ```

              ```shell curl theme={null}
              curl localhost:9070/deployments --json '{"uri": "http://localhost:3000/restate", "use_http_11": true}'
              ```
            </CodeGroup>

            <Expandable title="Output">
              <CodeGroup>
                ```shell CLI theme={null}
                ❯ SERVICES THAT WILL BE ADDED:
                - Greeter
                Type: Service
                HANDLER  INPUT                                     OUTPUT
                greet    value of content-type 'application/json'  value of content-type 'application/json'


                ✔ Are you sure you want to apply those changes? · yes
                ✅ DEPLOYMENT:
                SERVICE  REV
                Greeter  1
                ```

                ```shell curl theme={null}
                {
                    "id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                    "services": [
                        {
                            "name": "Greeter",
                            "handlers": [
                        {
                            "name": "greet",
                            "ty": "Shared",
                            "input_description": "one of [\"none\", \"value of content-type 'application/json'\"]",
                            "output_description": "value of content-type 'application/json'"
                        }
                            ],
                            "ty": "Service",
                            "deployment_id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                            "revision": 1,
                            "public": true,
                            "idempotency_retention": "1day"
                        }
                    ]
                }
                ```
              </CodeGroup>
            </Expandable>

            If you run Restate with Docker, use `http://host.docker.internal:3000/restate` instead of `http://localhost:3000/restate`.

            <Accordion title="Restate Cloud">
              When using [Restate Cloud](https://restate.dev/cloud), your service must be accessible over the public internet so Restate can invoke it.
              If you want to develop with a local service, you can expose it using our [tunnel](/cloud/connecting-services#connecting-services-in-private-environments) feature.
            </Accordion>
          </Step>

          <Step title="Send a request to the Greeter service">
            Invoke the service via the Restate UI playground: go to `http://localhost:9070`,  click on your service and then on playground.

            <img alt="Restate UI Playground" />

            Or invoke via `curl`:

            ```shell theme={null}
            curl localhost:8080/Greeter/greet --json '{"name": "Sarah"}'
            ```

            Expected output: `You said hi to Sarah!`
          </Step>

          <Step title="Congratulations, you just ran Durable Execution!">
            The invocation you just sent used Durable Execution to make sure the request ran till completion.
            For each request, it sent a notification, slept for a second, and then sent a reminder.

            ```ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/templates/vercel/src/restate/greeter.ts?collapse_prequel"}  theme={null}
            export const greeter = restate.service({
              name: "Greeter",
              handlers: {
                greet: restate.createServiceHandler(
                  { input: restate.serde.schema(Greeting), output: restate.serde.schema(GreetingResponse) },
                  async (ctx: restate.Context, { name }) => {
                    // Durably execute a set of steps; resilient against failures
                    const greetingId = ctx.rand.uuidv4();
                    await ctx.run("Notification", () => sendNotification(greetingId, name));
                    await ctx.sleep({ seconds: 1 });
                    await ctx.run("Reminder", () => sendReminder(greetingId, name));

                    // Respond to caller
                    return { result: `You said hi to ${name}!` };
                  },
                ),
              },
            });

            export type Greeter = typeof greeter;
            ```

            <GitHubLink />

            Send a request for `Alice` to see how the service behaves when it occasionally fails to send the reminder and notification:

            ```shell theme={null}
            curl localhost:8080/Greeter/greet --json '{"name": "Alice"}'
            ```

            Expected output:

            ```shell theme={null}
            You said hi to Alice!
            ```

            You can see in the service logs and in the Restate UI how the request gets retried.
            On a retry, it skipped the steps that already succeeded.

            <img alt="Restate UI Durable Execution" />

            Even the sleep is durable and tracked by Restate.
            If you kill/restart the service halfway through, the sleep will only last for what remained.

            <Tip title="Durable Execution">
              Restate persists the progress of the handler.
              Letting you write code that is resilient to failures out of the box.
              Have a look at the [Durable Execution page](/foundations/key-concepts#durable-execution) to learn more.
            </Tip>
          </Step>
        </Steps>
      </Tab>
    </Tabs>
  </Tab>

  <Tab title="Java" icon={"/img/languages/java.svg"}>
    Select your favorite build tool and framework:

    <Tabs>
      <Tab title="Maven + Spring Boot" icon={"/img/quickstart/spring.svg"}>
        <Info>
          **Prerequisites**:

          * [JDK](https://whichjdk.com/) >= 17
        </Info>

        <Steps>
          <Step title="Install Restate Server & CLI">
            Restate is a single self-contained binary. No external dependencies needed.

            <Tabs>
              <Tab title="Homebrew">
                ```shell theme={null}
                brew install restatedev/tap/restate-server restatedev/tap/restate
                ```

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="Download binaries">
                Download prebuilt binaries from the [releases page](https://github.com/restatedev/restate/releases/latest):

                <CodeGroup>
                  ```shell MacOS-x64 theme={null}
                  BIN=/usr/local/bin && RESTATE_PLATFORM=x86_64-apple-darwin && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  sudo mv restate $BIN && \
                  sudo mv restate-server $BIN
                  ```

                  ```shell MacOS-arm64 theme={null}
                  BIN=/usr/local/bin && RESTATE_PLATFORM=aarch64-apple-darwin && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  sudo mv restate $BIN && \
                  sudo mv restate-server $BIN
                  ```

                  ```shell Linux-x64 theme={null}
                  BIN=$HOME/.local/bin && RESTATE_PLATFORM=x86_64-unknown-linux-musl && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  mv restate $BIN && \
                  mv restate-server $BIN
                  ```

                  ```shell Linux-arm64 theme={null}
                  BIN=$HOME/.local/bin && RESTATE_PLATFORM=aarch64-unknown-linux-musl && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  mv restate $BIN && \
                  mv restate-server $BIN
                  ```
                </CodeGroup>

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="npm">
                ```shell theme={null}
                npm install --global @restatedev/restate-server@latest @restatedev/restate@latest
                ```

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="Docker">
                Run the Restate Server:

                ```shell theme={null}
                docker run --name restate_dev --rm \
                -p 8080:8080 -p 9070:9070 -p 9071:9071 \
                --add-host=host.docker.internal:host-gateway \
                docker.restate.dev/restatedev/restate:latest
                ```

                Run CLI commands:

                ```shell theme={null}
                docker run -it --network=host \
                docker.restate.dev/restatedev/restate-cli:latest \
                invocations ls
                ```

                Replace `invocations ls` with any CLI subcommand.
              </Tab>
            </Tabs>

            You can find the Restate UI running on port 9070 (`http://localhost:9070`) after starting the Restate Server.
          </Step>

          <Step title="Get the Greeter service template">
            <CodeGroup>
              ```shell CLI theme={null}
              restate example java-hello-world-maven-spring-boot &&
              cd java-hello-world-maven-spring-boot
              ```

              ```shell wget theme={null}
              wget https://github.com/restatedev/examples/releases/latest/download/java-hello-world-maven-spring-boot.zip &&
              unzip java-hello-world-maven-spring-boot.zip -d java-hello-world-maven-spring-boot &&
              rm java-hello-world-maven-spring-boot.zip && cd java-hello-world-maven-spring-boot
              ```
            </CodeGroup>

            <GitHubLink />
          </Step>

          <Step title="Run the Greeter service">
            You are all set to start developing your service.
            Open the project in an IDE, run your service and let it listen on port `9080` for requests:

            ```shell theme={null}
            mvn compile spring-boot:run
            ```
          </Step>

          <Step title="Register the service">
            Tell Restate where the service is running, so Restate can discover and register the services and handlers behind this endpoint.
            You can do this via the UI (`http://localhost:9070`) or via:

            <CodeGroup>
              ```shell CLI theme={null}
              restate deployments register http://localhost:9080
              ```

              ```shell curl theme={null}
              curl localhost:9070/deployments --json '{"uri": "http://localhost:9080"}'
              ```
            </CodeGroup>

            <Expandable title="Output">
              <CodeGroup>
                ```shell CLI theme={null}
                ❯ SERVICES THAT WILL BE ADDED:
                - Greeter
                Type: Service
                HANDLER  INPUT                                     OUTPUT
                greet    value of content-type 'application/json'  value of content-type 'application/json'


                ✔ Are you sure you want to apply those changes? · yes
                ✅ DEPLOYMENT:
                SERVICE  REV
                Greeter  1
                ```

                ```shell curl theme={null}
                {
                    "id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                    "services": [
                        {
                            "name": "Greeter",
                            "handlers": [
                                {
                                    "name": "greet",
                                    "ty": "Shared",
                                    "input_description": "one of [\"none\", \"value of content-type 'application/json'\"]",
                                    "output_description": "value of content-type 'application/json'"
                                }
                            ],
                            "ty": "Service",
                            "deployment_id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                            "revision": 1,
                            "public": true,
                            "idempotency_retention": "1day"
                        }
                    ]
                }
                ```
              </CodeGroup>
            </Expandable>

            If you run Restate with Docker, register `http://host.docker.internal:9080` instead of `http://localhost:9080`.

            <Accordion title="Restate Cloud">
              When using [Restate Cloud](https://restate.dev/cloud), your service must be accessible over the public internet so Restate can invoke it.
              If you want to develop with a local service, you can expose it using our [tunnel](/deploy/server/cloud/#registering-restate-services-with-your-environment) feature.
            </Accordion>
          </Step>

          <Step title="Send a request to the Greeter service">
            Invoke the service via the Restate UI playground: go to `http://localhost:9070`,  click on your service and then on playground.

            <img alt="Restate UI Playground" />

            Or invoke via `curl`:

            ```shell theme={null}
            curl localhost:8080/Greeter/greet --json '{"name": "Sarah"}'
            ```

            Expected output: `You said hi to Sarah!`
          </Step>

          <Step title="Congratulations, you just ran Durable Execution!">
            The invocation you just sent used Durable Execution to make sure the request ran till completion.
            For each request, it sent a notification, slept for a second, and then sent a reminder.

            ```java {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/templates/java-maven-spring-boot/src/main/java/com/example/restatestarter/Greeter.java?collapse_prequel"}  theme={null}
            @RestateService
            public class Greeter {

              @Value("${greetingPrefix}")
              private String greetingPrefix;

              public record Greeting(String name) {}
              public record GreetingResponse(String message) {}

              @Handler
              public GreetingResponse greet(Context ctx, Greeting req) {
                // Durably execute a set of steps; resilient against failures
                String greetingId = ctx.random().nextUUID().toString();
                ctx.run("Notification", () -> sendNotification(greetingId, req.name));
                ctx.sleep(Duration.ofSeconds(1));
                ctx.run("Reminder", () -> sendReminder(greetingId, req.name));

                // Respond to caller
                return new GreetingResponse("You said " + greetingPrefix + " to " + req.name + "!");
              }
            }
            ```

            <GitHubLink />

            Send a request for `Alice` to see how the service behaves when it occasionally fails to send the reminder and notification:

            ```shell theme={null}
            curl localhost:8080/Greeter/greet --json '{"name": "Alice"}'
            ```

            Expected output:

            ```shell theme={null}
            You said hi to Alice!
            ```

            You can see in the service logs and in the Restate UI how the request gets retried.
            On a retry, it skipped the steps that already succeeded.

            <img alt="Restate UI Durable Execution" />

            Even the sleep is durable and tracked by Restate.
            If you kill/restart the service halfway through, the sleep will only last for what remained.

            <Tip title="Durable Execution">
              Restate persists the progress of the handler.
              Letting you write code that is resilient to failures out of the box.
              Have a look at the [Durable Execution page](/foundations/key-concepts#durable-execution) to learn more.
            </Tip>
          </Step>
        </Steps>
      </Tab>

      <Tab title="Maven + Quarkus" icon={"/img/quickstart/quarkus.svg"}>
        <Info>
          **Prerequisites**:

          * [JDK](https://whichjdk.com/) >= 17
        </Info>

        <Steps>
          <Step title="Install Restate Server & CLI">
            Restate is a single self-contained binary. No external dependencies needed.

            <Tabs>
              <Tab title="Homebrew">
                ```shell theme={null}
                brew install restatedev/tap/restate-server restatedev/tap/restate
                ```

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="Download binaries">
                Download prebuilt binaries from the [releases page](https://github.com/restatedev/restate/releases/latest):

                <CodeGroup>
                  ```shell MacOS-x64 theme={null}
                  BIN=/usr/local/bin && RESTATE_PLATFORM=x86_64-apple-darwin && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  sudo mv restate $BIN && \
                  sudo mv restate-server $BIN
                  ```

                  ```shell MacOS-arm64 theme={null}
                  BIN=/usr/local/bin && RESTATE_PLATFORM=aarch64-apple-darwin && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  sudo mv restate $BIN && \
                  sudo mv restate-server $BIN
                  ```

                  ```shell Linux-x64 theme={null}
                  BIN=$HOME/.local/bin && RESTATE_PLATFORM=x86_64-unknown-linux-musl && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  mv restate $BIN && \
                  mv restate-server $BIN
                  ```

                  ```shell Linux-arm64 theme={null}
                  BIN=$HOME/.local/bin && RESTATE_PLATFORM=aarch64-unknown-linux-musl && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  mv restate $BIN && \
                  mv restate-server $BIN
                  ```
                </CodeGroup>

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="npm">
                ```shell theme={null}
                npm install --global @restatedev/restate-server@latest @restatedev/restate@latest
                ```

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="Docker">
                Run the Restate Server:

                ```shell theme={null}
                docker run --name restate_dev --rm \
                -p 8080:8080 -p 9070:9070 -p 9071:9071 \
                --add-host=host.docker.internal:host-gateway \
                docker.restate.dev/restatedev/restate:latest
                ```

                Run CLI commands:

                ```shell theme={null}
                docker run -it --network=host \
                docker.restate.dev/restatedev/restate-cli:latest \
                invocations ls
                ```

                Replace `invocations ls` with any CLI subcommand.
              </Tab>
            </Tabs>

            You can find the Restate UI running on port 9070 (`http://localhost:9070`) after starting the Restate Server.
          </Step>

          <Step title="Get the Greeter service template">
            <CodeGroup>
              ```shell CLI theme={null}
              restate example java-hello-world-maven-quarkus &&
              cd java-hello-world-maven-quarkus
              ```

              ```shell wget theme={null}
              wget https://github.com/restatedev/examples/releases/latest/download/java-hello-world-maven-quarkus.zip &&
              unzip java-hello-world-maven-quarkus.zip -d java-hello-world-maven-quarkus &&
              rm java-hello-world-maven-quarkus.zip && cd java-hello-world-maven-quarkus
              ```
            </CodeGroup>

            <GitHubLink />
          </Step>

          <Step title="Run the Greeter service">
            You are all set to start developing your service.
            Open the project in an IDE, run your service and let it listen on port `9080` for requests:

            ```shell theme={null}
            quarkus dev
            ```
          </Step>

          <Step title="Register the service">
            Tell Restate where the service is running, so Restate can discover and register the services and handlers behind this endpoint.
            You can do this via the UI (`http://localhost:9070`) or via:

            <CodeGroup>
              ```shell CLI theme={null}
              restate deployments register http://localhost:9080
              ```

              ```shell curl theme={null}
              curl localhost:9070/deployments --json '{"uri": "http://localhost:9080"}'
              ```
            </CodeGroup>

            <Expandable title="Output">
              <CodeGroup>
                ```shell CLI theme={null}
                ❯ SERVICES THAT WILL BE ADDED:
                - Greeter
                Type: Service
                HANDLER  INPUT                                     OUTPUT
                greet    value of content-type 'application/json'  value of content-type 'application/json'


                ✔ Are you sure you want to apply those changes? · yes
                ✅ DEPLOYMENT:
                SERVICE  REV
                Greeter  1
                ```

                ```shell curl theme={null}
                {
                    "id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                    "services": [
                        {
                            "name": "Greeter",
                            "handlers": [
                                {
                                    "name": "greet",
                                    "ty": "Shared",
                                    "input_description": "one of [\"none\", \"value of content-type 'application/json'\"]",
                                    "output_description": "value of content-type 'application/json'"
                                }
                            ],
                            "ty": "Service",
                            "deployment_id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                            "revision": 1,
                            "public": true,
                            "idempotency_retention": "1day"
                        }
                    ]
                }
                ```
              </CodeGroup>
            </Expandable>

            If you run Restate with Docker, register `http://host.docker.internal:9080` instead of `http://localhost:9080`.

            <Accordion title="Restate Cloud">
              When using [Restate Cloud](https://restate.dev/cloud), your service must be accessible over the public internet so Restate can invoke it.
              If you want to develop with a local service, you can expose it using our [tunnel](/deploy/server/cloud/#registering-restate-services-with-your-environment) feature.
            </Accordion>
          </Step>

          <Step title="Send a request to the Greeter service">
            Invoke the service via the Restate UI playground: go to `http://localhost:9070`,  click on your service and then on playground.

            <img alt="Restate UI Playground" />

            Or invoke via `curl`:

            ```shell theme={null}
            curl localhost:8080/Greeter/greet --json '{"name": "Sarah"}'
            ```

            Expected output: `You said hi to Sarah!`
          </Step>

          <Step title="Congratulations, you just ran Durable Execution!">
            The invocation you just sent used Durable Execution to make sure the request ran till completion.
            For each request, it sent a notification, slept for a second, and then sent a reminder.

            ```java {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/templates/java-maven-quarkus/src/main/java/org/acme/Greeter.java?collapse_prequel"}  theme={null}
            @Service
            public class Greeter {

              @ConfigProperty(name = "greetingPrefix") String greetingPrefix;

              public record Greeting(String name) {}
              public record GreetingResponse(String message) {}

              @Handler
              public GreetingResponse greet(Context ctx, Greeting req) {
                // Durably execute a set of steps; resilient against failures
                String greetingId = ctx.random().nextUUID().toString();
                ctx.run("Notification", () -> sendNotification(greetingId, req.name));
                ctx.sleep(Duration.ofSeconds(1));
                ctx.run("Reminder", () -> sendReminder(greetingId, req.name));

                // Respond to caller
                return new GreetingResponse("You said hi to " + req.name + "!");
              }
            }
            ```

            <GitHubLink />

            Send a request for `Alice` to see how the service behaves when it occasionally fails to send the reminder and notification:

            ```shell theme={null}
            curl localhost:8080/Greeter/greet --json '{"name": "Alice"}'
            ```

            Expected output:

            ```shell theme={null}
            You said hi to Alice!
            ```

            You can see in the service logs and in the Restate UI how the request gets retried.
            On a retry, it skipped the steps that already succeeded.

            <img alt="Restate UI Durable Execution" />

            Even the sleep is durable and tracked by Restate.
            If you kill/restart the service halfway through, the sleep will only last for what remained.

            <Tip title="Durable Execution">
              Restate persists the progress of the handler.
              Letting you write code that is resilient to failures out of the box.
              Have a look at the [Durable Execution page](/foundations/key-concepts#durable-execution) to learn more.
            </Tip>
          </Step>
        </Steps>
      </Tab>

      <Tab title="Maven" icon={"/img/quickstart/maven.svg"}>
        <Info>
          **Prerequisites**:

          * [JDK](https://whichjdk.com/) >= 17
        </Info>

        <Steps>
          <Step title="Install Restate Server & CLI">
            Restate is a single self-contained binary. No external dependencies needed.

            <Tabs>
              <Tab title="Homebrew">
                ```shell theme={null}
                brew install restatedev/tap/restate-server restatedev/tap/restate
                ```

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="Download binaries">
                Download prebuilt binaries from the [releases page](https://github.com/restatedev/restate/releases/latest):

                <CodeGroup>
                  ```shell MacOS-x64 theme={null}
                  BIN=/usr/local/bin && RESTATE_PLATFORM=x86_64-apple-darwin && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  sudo mv restate $BIN && \
                  sudo mv restate-server $BIN
                  ```

                  ```shell MacOS-arm64 theme={null}
                  BIN=/usr/local/bin && RESTATE_PLATFORM=aarch64-apple-darwin && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  sudo mv restate $BIN && \
                  sudo mv restate-server $BIN
                  ```

                  ```shell Linux-x64 theme={null}
                  BIN=$HOME/.local/bin && RESTATE_PLATFORM=x86_64-unknown-linux-musl && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  mv restate $BIN && \
                  mv restate-server $BIN
                  ```

                  ```shell Linux-arm64 theme={null}
                  BIN=$HOME/.local/bin && RESTATE_PLATFORM=aarch64-unknown-linux-musl && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  mv restate $BIN && \
                  mv restate-server $BIN
                  ```
                </CodeGroup>

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="npm">
                ```shell theme={null}
                npm install --global @restatedev/restate-server@latest @restatedev/restate@latest
                ```

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="Docker">
                Run the Restate Server:

                ```shell theme={null}
                docker run --name restate_dev --rm \
                -p 8080:8080 -p 9070:9070 -p 9071:9071 \
                --add-host=host.docker.internal:host-gateway \
                docker.restate.dev/restatedev/restate:latest
                ```

                Run CLI commands:

                ```shell theme={null}
                docker run -it --network=host \
                docker.restate.dev/restatedev/restate-cli:latest \
                invocations ls
                ```

                Replace `invocations ls` with any CLI subcommand.
              </Tab>
            </Tabs>

            You can find the Restate UI running on port 9070 (`http://localhost:9070`) after starting the Restate Server.
          </Step>

          <Step title="Get the Greeter service template">
            <CodeGroup>
              ```shell CLI theme={null}
              restate example java-hello-world-maven &&
              cd java-hello-world-maven
              ```

              ```shell wget theme={null}
              wget https://github.com/restatedev/examples/releases/latest/download/java-hello-world-maven.zip &&
              unzip java-hello-world-maven.zip -d java-hello-world-maven &&
              rm java-hello-world-maven.zip && cd java-hello-world-maven
              ```
            </CodeGroup>

            <GitHubLink />
          </Step>

          <Step title="Run the Greeter service">
            You are all set to start developing your service.
            Open the project in an IDE, run your service and let it listen on port `9080` for requests:

            ```shell theme={null}
            mvn compile exec:java
            ```
          </Step>

          <Step title="Register the service">
            Tell Restate where the service is running, so Restate can discover and register the services and handlers behind this endpoint.
            You can do this via the UI (`http://localhost:9070`) or via:

            <CodeGroup>
              ```shell CLI theme={null}
              restate deployments register http://localhost:9080
              ```

              ```shell curl theme={null}
              curl localhost:9070/deployments --json '{"uri": "http://localhost:9080"}'
              ```
            </CodeGroup>

            <Expandable title="Output">
              <CodeGroup>
                ```shell CLI theme={null}
                ❯ SERVICES THAT WILL BE ADDED:
                - Greeter
                Type: Service
                HANDLER  INPUT                                     OUTPUT
                greet    value of content-type 'application/json'  value of content-type 'application/json'


                ✔ Are you sure you want to apply those changes? · yes
                ✅ DEPLOYMENT:
                SERVICE  REV
                Greeter  1
                ```

                ```shell curl theme={null}
                {
                    "id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                    "services": [
                        {
                            "name": "Greeter",
                            "handlers": [
                                {
                                    "name": "greet",
                                    "ty": "Shared",
                                    "input_description": "one of [\"none\", \"value of content-type 'application/json'\"]",
                                    "output_description": "value of content-type 'application/json'"
                                }
                            ],
                            "ty": "Service",
                            "deployment_id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                            "revision": 1,
                            "public": true,
                            "idempotency_retention": "1day"
                        }
                    ]
                }
                ```
              </CodeGroup>
            </Expandable>

            If you run Restate with Docker, register `http://host.docker.internal:9080` instead of `http://localhost:9080`.

            <Accordion title="Restate Cloud">
              When using [Restate Cloud](https://restate.dev/cloud), your service must be accessible over the public internet so Restate can invoke it.
              If you want to develop with a local service, you can expose it using our [tunnel](/deploy/server/cloud/#registering-restate-services-with-your-environment) feature.
            </Accordion>
          </Step>

          <Step title="Send a request to the Greeter service">
            Invoke the service via the Restate UI playground: go to `http://localhost:9070`,  click on your service and then on playground.

            <img alt="Restate UI Playground" />

            Or invoke via `curl`:

            ```shell theme={null}
            curl localhost:8080/Greeter/greet --json '{"name": "Sarah"}'
            ```

            Expected output: `You said hi to Sarah!`
          </Step>

          <Step title="Congratulations, you just ran Durable Execution!">
            The invocation you just sent used Durable Execution to make sure the request ran till completion.
            For each request, it sent a notification, slept for a second, and then sent a reminder.

            ```java {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/templates/java-maven/src/main/java/my/example/Greeter.java?collapse_prequel"}  theme={null}
            @Service
            public class Greeter {

              public record Greeting(String name) {}
              public record GreetingResponse(String message) {}

              @Handler
              public GreetingResponse greet(Context ctx, Greeting req) {
                // Durably execute a set of steps; resilient against failures
                String greetingId = ctx.random().nextUUID().toString();
                ctx.run("Notification", () -> sendNotification(greetingId, req.name));
                ctx.sleep(Duration.ofSeconds(1));
                ctx.run("Reminder", () -> sendReminder(greetingId, req.name));

                // Respond to caller
                return new GreetingResponse("You said hi to " + req.name + "!");
              }

              public static void main(String[] args) {
                RestateHttpServer.listen(Endpoint.bind(new Greeter()));
              }
            }
            ```

            <GitHubLink />

            Send a request for `Alice` to see how the service behaves when it occasionally fails to send the reminder and notification:

            ```shell theme={null}
            curl localhost:8080/Greeter/greet --json '{"name": "Alice"}'
            ```

            Expected output:

            ```shell theme={null}
            You said hi to Alice!
            ```

            You can see in the service logs and in the Restate UI how the request gets retried.
            On a retry, it skipped the steps that already succeeded.

            <img alt="Restate UI Durable Execution" />

            Even the sleep is durable and tracked by Restate.
            If you kill/restart the service halfway through, the sleep will only last for what remained.

            <Tip title="Durable Execution">
              Restate persists the progress of the handler.
              Letting you write code that is resilient to failures out of the box.
              Have a look at the [Durable Execution page](/foundations/key-concepts#durable-execution) to learn more.
            </Tip>
          </Step>
        </Steps>
      </Tab>

      <Tab title="Gradle" icon={"/img/quickstart/gradle.svg"}>
        <Info>
          **Prerequisites**:

          * [JDK](https://whichjdk.com/) >= 17
        </Info>

        <Steps>
          <Step title="Install Restate Server & CLI">
            Restate is a single self-contained binary. No external dependencies needed.

            <Tabs>
              <Tab title="Homebrew">
                ```shell theme={null}
                brew install restatedev/tap/restate-server restatedev/tap/restate
                ```

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="Download binaries">
                Download prebuilt binaries from the [releases page](https://github.com/restatedev/restate/releases/latest):

                <CodeGroup>
                  ```shell MacOS-x64 theme={null}
                  BIN=/usr/local/bin && RESTATE_PLATFORM=x86_64-apple-darwin && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  sudo mv restate $BIN && \
                  sudo mv restate-server $BIN
                  ```

                  ```shell MacOS-arm64 theme={null}
                  BIN=/usr/local/bin && RESTATE_PLATFORM=aarch64-apple-darwin && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  sudo mv restate $BIN && \
                  sudo mv restate-server $BIN
                  ```

                  ```shell Linux-x64 theme={null}
                  BIN=$HOME/.local/bin && RESTATE_PLATFORM=x86_64-unknown-linux-musl && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  mv restate $BIN && \
                  mv restate-server $BIN
                  ```

                  ```shell Linux-arm64 theme={null}
                  BIN=$HOME/.local/bin && RESTATE_PLATFORM=aarch64-unknown-linux-musl && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  mv restate $BIN && \
                  mv restate-server $BIN
                  ```
                </CodeGroup>

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="npm">
                ```shell theme={null}
                npm install --global @restatedev/restate-server@latest @restatedev/restate@latest
                ```

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="Docker">
                Run the Restate Server:

                ```shell theme={null}
                docker run --name restate_dev --rm \
                -p 8080:8080 -p 9070:9070 -p 9071:9071 \
                --add-host=host.docker.internal:host-gateway \
                docker.restate.dev/restatedev/restate:latest
                ```

                Run CLI commands:

                ```shell theme={null}
                docker run -it --network=host \
                docker.restate.dev/restatedev/restate-cli:latest \
                invocations ls
                ```

                Replace `invocations ls` with any CLI subcommand.
              </Tab>
            </Tabs>

            You can find the Restate UI running on port 9070 (`http://localhost:9070`) after starting the Restate Server.
          </Step>

          <Step title="Get the Greeter service template">
            <CodeGroup>
              ```shell CLI theme={null}
              restate example java-hello-world-gradle &&
              cd java-hello-world-gradle
              ```

              ```shell wget theme={null}
              wget https://github.com/restatedev/examples/releases/latest/download/java-hello-world-gradle.zip &&
              unzip java-hello-world-gradle.zip -d java-hello-world-gradle &&
              rm java-hello-world-gradle.zip && cd java-hello-world-gradle
              ```
            </CodeGroup>

            <GitHubLink />
          </Step>

          <Step title="Run the Greeter service">
            You are all set to start developing your service.
            Open the project in an IDE, run your service and let it listen on port `9080` for requests:

            ```shell theme={null}
            ./gradlew run
            ```
          </Step>

          <Step title="Register the service">
            Tell Restate where the service is running, so Restate can discover and register the services and handlers behind this endpoint.
            You can do this via the UI (`http://localhost:9070`) or via:

            <CodeGroup>
              ```shell CLI theme={null}
              restate deployments register http://localhost:9080
              ```

              ```shell curl theme={null}
              curl localhost:9070/deployments --json '{"uri": "http://localhost:9080"}'
              ```
            </CodeGroup>

            <Expandable title="Output">
              <CodeGroup>
                ```shell CLI theme={null}
                ❯ SERVICES THAT WILL BE ADDED:
                - Greeter
                Type: Service
                HANDLER  INPUT                                     OUTPUT
                greet    value of content-type 'application/json'  value of content-type 'application/json'


                ✔ Are you sure you want to apply those changes? · yes
                ✅ DEPLOYMENT:
                SERVICE  REV
                Greeter  1
                ```

                ```shell curl theme={null}
                {
                    "id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                    "services": [
                        {
                            "name": "Greeter",
                            "handlers": [
                                {
                                    "name": "greet",
                                    "ty": "Shared",
                                    "input_description": "one of [\"none\", \"value of content-type 'application/json'\"]",
                                    "output_description": "value of content-type 'application/json'"
                                }
                            ],
                            "ty": "Service",
                            "deployment_id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                            "revision": 1,
                            "public": true,
                            "idempotency_retention": "1day"
                        }
                    ]
                }
                ```
              </CodeGroup>
            </Expandable>

            If you run Restate with Docker, register `http://host.docker.internal:9080` instead of `http://localhost:9080`.

            <Accordion title="Restate Cloud">
              When using [Restate Cloud](https://restate.dev/cloud), your service must be accessible over the public internet so Restate can invoke it.
              If you want to develop with a local service, you can expose it using our [tunnel](/deploy/server/cloud/#registering-restate-services-with-your-environment) feature.
            </Accordion>
          </Step>

          <Step title="Send a request to the Greeter service">
            Invoke the service via the Restate UI playground: go to `http://localhost:9070`,  click on your service and then on playground.

            <img alt="Restate UI Playground" />

            Or invoke via `curl`:

            ```shell theme={null}
            curl localhost:8080/Greeter/greet --json '{"name": "Sarah"}'
            ```

            Expected output: `You said hi to Sarah!`
          </Step>

          <Step title="Congratulations, you just ran Durable Execution!">
            The invocation you just sent used Durable Execution to make sure the request ran till completion.
            For each request, it sent a notification, slept for a second, and then sent a reminder.

            ```java {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/templates/java-gradle/src/main/java/my/example/Greeter.java?collapse_prequel"}  theme={null}
            @Service
            public class Greeter {

                public record Greeting(String name) {}
                public record GreetingResponse(String message) {}

                @Handler
                public GreetingResponse greet(Context ctx, Greeting req) {
                    // Durably execute a set of steps; resilient against failures
                    String greetingId = ctx.random().nextUUID().toString();
                    ctx.run("Notification", () -> sendNotification(greetingId, req.name));
                    ctx.sleep(Duration.ofSeconds(1));
                    ctx.run("Reminder", () -> sendReminder(greetingId, req.name));

                    // Respond to caller
                    return new GreetingResponse("You said hi to " + req.name + "!");
                }

                public static void main(String[] args) {
                    RestateHttpServer.listen(Endpoint.bind(new Greeter()));
                }
            }
            ```

            <GitHubLink />

            Send a request for `Alice` to see how the service behaves when it occasionally fails to send the reminder and notification:

            ```shell theme={null}
            curl localhost:8080/Greeter/greet --json '{"name": "Alice"}'
            ```

            Expected output:

            ```shell theme={null}
            You said hi to Alice!
            ```

            You can see in the service logs and in the Restate UI how the request gets retried.
            On a retry, it skipped the steps that already succeeded.

            <img alt="Restate UI Durable Execution" />

            Even the sleep is durable and tracked by Restate.
            If you kill/restart the service halfway through, the sleep will only last for what remained.

            <Tip title="Durable Execution">
              Restate persists the progress of the handler.
              Letting you write code that is resilient to failures out of the box.
              Have a look at the [Durable Execution page](/foundations/key-concepts#durable-execution) to learn more.
            </Tip>
          </Step>
        </Steps>
      </Tab>
    </Tabs>
  </Tab>

  <Tab title="Kotlin" icon={"/img/languages/kotlin.svg"}>
    Select your favorite framework:

    <Tabs>
      <Tab title="Gradle + Spring Boot" icon={"/img/quickstart/spring.svg"}>
        <Info>
          **Prerequisites**:

          * [JDK](https://whichjdk.com/) >= 17
        </Info>

        <Steps>
          <Step title="Install Restate Server & CLI">
            Restate is a single self-contained binary. No external dependencies needed.

            <Tabs>
              <Tab title="Homebrew">
                ```shell theme={null}
                brew install restatedev/tap/restate-server restatedev/tap/restate
                ```

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="Download binaries">
                Download prebuilt binaries from the [releases page](https://github.com/restatedev/restate/releases/latest):

                <CodeGroup>
                  ```shell MacOS-x64 theme={null}
                  BIN=/usr/local/bin && RESTATE_PLATFORM=x86_64-apple-darwin && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  sudo mv restate $BIN && \
                  sudo mv restate-server $BIN
                  ```

                  ```shell MacOS-arm64 theme={null}
                  BIN=/usr/local/bin && RESTATE_PLATFORM=aarch64-apple-darwin && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  sudo mv restate $BIN && \
                  sudo mv restate-server $BIN
                  ```

                  ```shell Linux-x64 theme={null}
                  BIN=$HOME/.local/bin && RESTATE_PLATFORM=x86_64-unknown-linux-musl && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  mv restate $BIN && \
                  mv restate-server $BIN
                  ```

                  ```shell Linux-arm64 theme={null}
                  BIN=$HOME/.local/bin && RESTATE_PLATFORM=aarch64-unknown-linux-musl && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  mv restate $BIN && \
                  mv restate-server $BIN
                  ```
                </CodeGroup>

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="npm">
                ```shell theme={null}
                npm install --global @restatedev/restate-server@latest @restatedev/restate@latest
                ```

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="Docker">
                Run the Restate Server:

                ```shell theme={null}
                docker run --name restate_dev --rm \
                -p 8080:8080 -p 9070:9070 -p 9071:9071 \
                --add-host=host.docker.internal:host-gateway \
                docker.restate.dev/restatedev/restate:latest
                ```

                Run CLI commands:

                ```shell theme={null}
                docker run -it --network=host \
                docker.restate.dev/restatedev/restate-cli:latest \
                invocations ls
                ```

                Replace `invocations ls` with any CLI subcommand.
              </Tab>
            </Tabs>

            You can find the Restate UI running on port 9070 (`http://localhost:9070`) after starting the Restate Server.
          </Step>

          <Step title="Get the Greeter service template">
            <CodeGroup>
              ```shell CLI theme={null}
              restate example kotlin-hello-world-gradle-spring-boot &&
              cd kotlin-hello-world-gradle-spring-boot
              ```

              ```shell wget theme={null}
              wget https://github.com/restatedev/examples/releases/latest/download/kotlin-hello-world-gradle-spring-boot.zip &&
              unzip kotlin-hello-world-gradle-spring-boot.zip -d kotlin-hello-world-gradle-spring-boot &&
              rm kotlin-hello-world-gradle-spring-boot.zip && cd kotlin-hello-world-gradle-spring-boot
              ```
            </CodeGroup>

            <GitHubLink />
          </Step>

          <Step title="Run the Greeter service">
            You are all set to start developing your service.
            Open the project in an IDE and configure it to build with Gradle.
            Run your service and let it listen on port `9080` for requests:

            ```shell theme={null}
            ./gradlew bootRun
            ```
          </Step>

          <Step title="Register the service">
            Tell Restate where the service is running, so Restate can discover and register the services and handlers behind this endpoint.
            You can do this via the UI (`http://localhost:9070`) or via:

            <CodeGroup>
              ```shell CLI theme={null}
              restate deployments register http://localhost:9080
              ```

              ```shell curl theme={null}
              curl localhost:9070/deployments --json '{"uri": "http://localhost:9080"}'
              ```
            </CodeGroup>

            <Expandable title="Output">
              <CodeGroup>
                ```shell CLI theme={null}
                ❯ SERVICES THAT WILL BE ADDED:
                - Greeter
                Type: Service
                HANDLER  INPUT                                     OUTPUT
                greet    value of content-type 'application/json'  value of content-type 'application/json'


                ✔ Are you sure you want to apply those changes? · yes
                ✅ DEPLOYMENT:
                SERVICE  REV
                Greeter  1
                ```

                ```shell curl theme={null}
                {
                    "id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                    "services": [
                        {
                            "name": "Greeter",
                            "handlers": [
                                {
                                    "name": "greet",
                                    "ty": "Shared",
                                    "input_description": "one of [\"none\", \"value of content-type 'application/json'\"]",
                                    "output_description": "value of content-type 'application/json'"
                                }
                            ],
                            "ty": "Service",
                            "deployment_id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                            "revision": 1,
                            "public": true,
                            "idempotency_retention": "1day"
                        }
                    ]
                }
                ```
              </CodeGroup>
            </Expandable>

            If you run Restate with Docker, register `http://host.docker.internal:9080` instead of `http://localhost:9080`.

            <Accordion title="Restate Cloud">
              When using [Restate Cloud](https://restate.dev/cloud), your service must be accessible over the public internet so Restate can invoke it.
              If you want to develop with a local service, you can expose it using our [tunnel](/deploy/server/cloud/#registering-restate-services-with-your-environment) feature.
            </Accordion>
          </Step>

          <Step title="Send a request to the Greeter service">
            Invoke the service via the Restate UI playground: go to `http://localhost:9070`,  click on your service and then on playground.

            <img alt="Restate UI Playground" />

            Or invoke via `curl`:

            ```shell theme={null}
            curl localhost:8080/Greeter/greet --json '{"name": "Sarah"}'
            ```

            Expected output: `You said hi to Sarah!`
          </Step>

          <Step title="Congratulations, you just ran Durable Execution!">
            The invocation you just sent used Durable Execution to make sure the request ran till completion.
            For each request, it sent a notification, slept for a second, and then sent a reminder.

            ```kotlin {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/kotlin/templates/kotlin-gradle-spring-boot/src/main/kotlin/com/example/restatestarter/Greeter.kt?collapse_prequel"}  theme={null}
            @RestateService
            class Greeter {

              @Value("\${greetingPrefix}")
              lateinit var greetingPrefix: String

              @Serializable
              data class Greeting(val name: String)
              @Serializable
              data class GreetingResponse(val message: String)

              @Handler
              suspend fun greet(ctx: Context, req: Greeting): GreetingResponse {
                // Durably execute a set of steps; resilient against failures
                val greetingId = ctx.random().nextUUID().toString()
                ctx.runBlock("Notification") { sendNotification(greetingId, req.name) }
                ctx.sleep(1.seconds)
                ctx.runBlock("Reminder") { sendReminder(greetingId, req.name) }

                // Respond to caller
                return GreetingResponse("You said $greetingPrefix to ${req.name}!")
              }
            }
            ```

            <GitHubLink />

            Send a request for `Alice` to see how the service behaves when it occasionally fails to send the reminder and notification:

            ```shell theme={null}
            curl localhost:8080/Greeter/greet --json '{"name": "Alice"}'
            ```

            Expected output:

            ```shell theme={null}
            You said hi to Alice!
            ```

            You can see in the service logs and in the Restate UI how the request gets retried.
            On a retry, it skipped the steps that already succeeded.

            <img alt="Restate UI Durable Execution" />

            Even the sleep is durable and tracked by Restate.
            If you kill/restart the service halfway through, the sleep will only last for what remained.

            <Tip title="Durable Execution">
              Restate persists the progress of the handler.
              Letting you write code that is resilient to failures out of the box.
              Have a look at the [Durable Execution page](/foundations/key-concepts#durable-execution) to learn more.
            </Tip>
          </Step>
        </Steps>
      </Tab>

      <Tab title="Gradle" icon={"/img/quickstart/gradle.svg"}>
        <Info>
          **Prerequisites**:

          * [JDK](https://whichjdk.com/) >= 17
        </Info>

        <Steps>
          <Step title="Install Restate Server & CLI">
            Restate is a single self-contained binary. No external dependencies needed.

            <Tabs>
              <Tab title="Homebrew">
                ```shell theme={null}
                brew install restatedev/tap/restate-server restatedev/tap/restate
                ```

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="Download binaries">
                Download prebuilt binaries from the [releases page](https://github.com/restatedev/restate/releases/latest):

                <CodeGroup>
                  ```shell MacOS-x64 theme={null}
                  BIN=/usr/local/bin && RESTATE_PLATFORM=x86_64-apple-darwin && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  sudo mv restate $BIN && \
                  sudo mv restate-server $BIN
                  ```

                  ```shell MacOS-arm64 theme={null}
                  BIN=/usr/local/bin && RESTATE_PLATFORM=aarch64-apple-darwin && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  sudo mv restate $BIN && \
                  sudo mv restate-server $BIN
                  ```

                  ```shell Linux-x64 theme={null}
                  BIN=$HOME/.local/bin && RESTATE_PLATFORM=x86_64-unknown-linux-musl && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  mv restate $BIN && \
                  mv restate-server $BIN
                  ```

                  ```shell Linux-arm64 theme={null}
                  BIN=$HOME/.local/bin && RESTATE_PLATFORM=aarch64-unknown-linux-musl && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  mv restate $BIN && \
                  mv restate-server $BIN
                  ```
                </CodeGroup>

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="npm">
                ```shell theme={null}
                npm install --global @restatedev/restate-server@latest @restatedev/restate@latest
                ```

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="Docker">
                Run the Restate Server:

                ```shell theme={null}
                docker run --name restate_dev --rm \
                -p 8080:8080 -p 9070:9070 -p 9071:9071 \
                --add-host=host.docker.internal:host-gateway \
                docker.restate.dev/restatedev/restate:latest
                ```

                Run CLI commands:

                ```shell theme={null}
                docker run -it --network=host \
                docker.restate.dev/restatedev/restate-cli:latest \
                invocations ls
                ```

                Replace `invocations ls` with any CLI subcommand.
              </Tab>
            </Tabs>

            You can find the Restate UI running on port 9070 (`http://localhost:9070`) after starting the Restate Server.
          </Step>

          <Step title="Get the Greeter service template">
            <CodeGroup>
              ```shell CLI theme={null}
              restate example kotlin-hello-world-gradle &&
              cd kotlin-hello-world-gradle
              ```

              ```shell wget theme={null}
              wget https://github.com/restatedev/examples/releases/latest/download/kotlin-hello-world-gradle.zip &&
              unzip kotlin-hello-world-gradle.zip -d kotlin-hello-world-gradle &&
              rm kotlin-hello-world-gradle.zip && cd kotlin-hello-world-gradle
              ```
            </CodeGroup>

            <GitHubLink />
          </Step>

          <Step title="Run the Greeter service">
            You are all set to start developing your service.
            Open the project in an IDE and configure it to build with Gradle.
            Run your service and let it listen on port `9080` for requests:

            ```shell theme={null}
            ./gradlew run
            ```
          </Step>

          <Step title="Register the service">
            Tell Restate where the service is running, so Restate can discover and register the services and handlers behind this endpoint.
            You can do this via the UI (`http://localhost:9070`) or via:

            <CodeGroup>
              ```shell CLI theme={null}
              restate deployments register http://localhost:9080
              ```

              ```shell curl theme={null}
              curl localhost:9070/deployments --json '{"uri": "http://localhost:9080"}'
              ```
            </CodeGroup>

            <Expandable title="Output">
              <CodeGroup>
                ```shell CLI theme={null}
                ❯ SERVICES THAT WILL BE ADDED:
                - Greeter
                Type: Service
                HANDLER  INPUT                                     OUTPUT
                greet    value of content-type 'application/json'  value of content-type 'application/json'


                ✔ Are you sure you want to apply those changes? · yes
                ✅ DEPLOYMENT:
                SERVICE  REV
                Greeter  1
                ```

                ```shell curl theme={null}
                {
                    "id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                    "services": [
                        {
                            "name": "Greeter",
                            "handlers": [
                                {
                                    "name": "greet",
                                    "ty": "Shared",
                                    "input_description": "one of [\"none\", \"value of content-type 'application/json'\"]",
                                    "output_description": "value of content-type 'application/json'"
                                }
                            ],
                            "ty": "Service",
                            "deployment_id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                            "revision": 1,
                            "public": true,
                            "idempotency_retention": "1day"
                        }
                    ]
                }
                ```
              </CodeGroup>
            </Expandable>

            If you run Restate with Docker, register `http://host.docker.internal:9080` instead of `http://localhost:9080`.

            <Accordion title="Restate Cloud">
              When using [Restate Cloud](https://restate.dev/cloud), your service must be accessible over the public internet so Restate can invoke it.
              If you want to develop with a local service, you can expose it using our [tunnel](/deploy/server/cloud/#registering-restate-services-with-your-environment) feature.
            </Accordion>
          </Step>

          <Step title="Send a request to the Greeter service">
            Invoke the service via the Restate UI playground: go to `http://localhost:9070`,  click on your service and then on playground.

            <img alt="Restate UI Playground" />

            Or invoke via `curl`:

            ```shell theme={null}
            curl localhost:8080/Greeter/greet --json '{"name": "Sarah"}'
            ```

            Expected output: `You said hi to Sarah!`
          </Step>

          <Step title="Congratulations, you just ran Durable Execution!">
            The invocation you just sent used Durable Execution to make sure the request ran till completion.
            For each request, it sent a notification, slept for a second, and then sent a reminder.

            ```kotlin {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/kotlin/templates/kotlin-gradle/src/main/kotlin/my/example/Greeter.kt?collapse_prequel"}  theme={null}
            @Service
            class Greeter {

              @Serializable
              data class Greeting(val name: String)
              @Serializable
              data class GreetingResponse(val message: String)

              @Handler
              suspend fun greet(ctx: Context, req: Greeting): GreetingResponse {
                // Durably execute a set of steps; resilient against failures
                val greetingId = ctx.random().nextUUID().toString()
                ctx.runBlock("Notification") { sendNotification(greetingId, req.name) }
                ctx.sleep(1.seconds)
                ctx.runBlock("Reminder") { sendReminder(greetingId, req.name) }

                // Respond to caller
                return GreetingResponse("You said hi to ${req.name}!")
              }
            }

            fun main() {
              RestateHttpServer.listen(endpoint {
                bind(Greeter())
              })
            }
            ```

            <GitHubLink />

            Send a request for `Alice` to see how the service behaves when it occasionally fails to send the reminder and notification:

            ```shell theme={null}
            curl localhost:8080/Greeter/greet --json '{"name": "Alice"}'
            ```

            Expected output:

            ```shell theme={null}
            You said hi to Alice!
            ```

            You can see in the service logs and in the Restate UI how the request gets retried.
            On a retry, it skipped the steps that already succeeded.

            <img alt="Restate UI Durable Execution" />

            Even the sleep is durable and tracked by Restate.
            If you kill/restart the service halfway through, the sleep will only last for what remained.

            <Tip title="Durable Execution">
              Restate persists the progress of the handler.
              Letting you write code that is resilient to failures out of the box.
              Have a look at the [Durable Execution page](/foundations/key-concepts#durable-execution) to learn more.
            </Tip>
          </Step>
        </Steps>
      </Tab>
    </Tabs>
  </Tab>

  <Tab title="Go" icon={"/img/languages/go.svg"}>
    <Info>
      **Prerequisites**:

      * Go: >= 1.21.0
    </Info>

    <Steps>
      <Step title="Install Restate Server & CLI">
        Restate is a single self-contained binary. No external dependencies needed.

        <Tabs>
          <Tab title="Homebrew">
            ```shell theme={null}
            brew install restatedev/tap/restate-server restatedev/tap/restate
            ```

            Start the server:

            ```shell theme={null}
            restate-server
            ```
          </Tab>

          <Tab title="Download binaries">
            Download prebuilt binaries from the [releases page](https://github.com/restatedev/restate/releases/latest):

            <CodeGroup>
              ```shell MacOS-x64 theme={null}
              BIN=/usr/local/bin && RESTATE_PLATFORM=x86_64-apple-darwin && \
              curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
              tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
              tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
              chmod +x restate restate-server && \
              sudo mv restate $BIN && \
              sudo mv restate-server $BIN
              ```

              ```shell MacOS-arm64 theme={null}
              BIN=/usr/local/bin && RESTATE_PLATFORM=aarch64-apple-darwin && \
              curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
              tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
              tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
              chmod +x restate restate-server && \
              sudo mv restate $BIN && \
              sudo mv restate-server $BIN
              ```

              ```shell Linux-x64 theme={null}
              BIN=$HOME/.local/bin && RESTATE_PLATFORM=x86_64-unknown-linux-musl && \
              curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
              tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
              tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
              chmod +x restate restate-server && \
              mv restate $BIN && \
              mv restate-server $BIN
              ```

              ```shell Linux-arm64 theme={null}
              BIN=$HOME/.local/bin && RESTATE_PLATFORM=aarch64-unknown-linux-musl && \
              curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
              tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
              tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
              chmod +x restate restate-server && \
              mv restate $BIN && \
              mv restate-server $BIN
              ```
            </CodeGroup>

            Start the server:

            ```shell theme={null}
            restate-server
            ```
          </Tab>

          <Tab title="npm">
            ```shell theme={null}
            npm install --global @restatedev/restate-server@latest @restatedev/restate@latest
            ```

            Start the server:

            ```shell theme={null}
            restate-server
            ```
          </Tab>

          <Tab title="Docker">
            Run the Restate Server:

            ```shell theme={null}
            docker run --name restate_dev --rm \
            -p 8080:8080 -p 9070:9070 -p 9071:9071 \
            --add-host=host.docker.internal:host-gateway \
            docker.restate.dev/restatedev/restate:latest
            ```

            Run CLI commands:

            ```shell theme={null}
            docker run -it --network=host \
            docker.restate.dev/restatedev/restate-cli:latest \
            invocations ls
            ```

            Replace `invocations ls` with any CLI subcommand.
          </Tab>
        </Tabs>

        You can find the Restate UI running on port 9070 (`http://localhost:9070`) after starting the Restate Server.
      </Step>

      <Step title="Get the Greeter service template">
        <CodeGroup>
          ```shell CLI theme={null}
          restate example go-hello-world &&
          cd go-hello-world
          ```

          ```shell wget theme={null}
          wget https://github.com/restatedev/examples/releases/latest/download/go-hello-world.zip &&
          unzip go-hello-world.zip -d go-hello-world &&
          rm go-hello-world.zip && cd go-hello-world
          ```
        </CodeGroup>

        <GitHubLink />
      </Step>

      <Step title="Run the Greeter service">
        Now, start developing your service in `greeter.go`. Run it with:

        ```shell theme={null}
        go run .
        ```

        it will listen on port 9080 for requests.
      </Step>

      <Step title="Register the service">
        Tell Restate where the service is running, so Restate can discover and register the services and handlers behind this endpoint.
        You can do this via the UI (`http://localhost:9070`) or via:

        <CodeGroup>
          ```shell CLI theme={null}
          restate deployments register http://localhost:9080
          ```

          ```shell curl theme={null}
          curl localhost:9070/deployments --json '{"uri": "http://localhost:9080"}'
          ```
        </CodeGroup>

        <Expandable title="Output">
          <CodeGroup>
            ```shell CLI theme={null}
            ❯ SERVICES THAT WILL BE ADDED:
            - Greeter
            Type: Service
            HANDLER  INPUT                                     OUTPUT
            Greet    value of content-type 'application/json'  value of content-type 'application/json'


            ✔ Are you sure you want to apply those changes? · yes
            ✅ DEPLOYMENT:
            SERVICE  REV
            Greeter  1
            ```

            ```shell curl theme={null}
            {
                "id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                "services": [
            {
                "name": "Greeter",
                "handlers": [
            {
                "name": "Greet",
                "ty": "Shared",
                "input_description": "one of [\"none\", \"value of content-type 'application/json'\"]",
                "output_description": "value of content-type 'application/json'"
            }
                ],
                "ty": "Service",
                "deployment_id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                "revision": 1,
                "public": true,
                "idempotency_retention": "1day"
            }
                ]
            }
            ```
          </CodeGroup>
        </Expandable>

        If you run Restate with Docker, use `http://host.docker.internal:9080` instead of `http://localhost:9080`.
      </Step>

      <Step title="Send a request to the Greeter service">
        Invoke the service via the Restate UI playground: go to `http://localhost:9070`, click on your service and then on playground.

        <img alt="Restate UI Playground" />

        Or invoke via `curl`:

        ```shell theme={null}
        curl localhost:8080/Greeter/Greet --json '"Sarah"'
        ```

        Expected output:

        ```
        You said hi to Sarah!
        ```
      </Step>

      <Step title="Congratulations, you just ran Durable Execution!">
        The invocation you just sent used Durable Execution to make sure the request ran till completion.
        For each request, it sent a notification, slept for a second, and then sent a reminder.

        ```go {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/templates/go/greeter.go?collapse_prequel"}  theme={null}
        type Greeter struct{}

        func (Greeter) Greet(ctx restate.Context, name string) (string, error) {
          //  Durably execute a set of steps; resilient against failures
          greetingId := restate.UUID(ctx).String()

          if _, err := restate.Run(ctx,
            func(ctx restate.RunContext) (restate.Void, error) {
              return SendNotification(greetingId, name)
            },
            restate.WithName("Notification"),
          ); err != nil {
            return "", err
          }

          if err := restate.Sleep(ctx, 1*time.Second); err != nil {
            return "", err
          }

          if _, err := restate.Run(ctx,
            func(ctx restate.RunContext) (restate.Void, error) {
              return SendReminder(greetingId, name)
            },
            restate.WithName("Reminder"),
          ); err != nil {
            return "", err
          }

          // Respond to caller
          return "You said hi to " + name + "!", nil
        }
        ```

        <GitHubLink />

        Send a request for `Alice` to see how the service behaves when it occasionally fails to send the reminder and notification:

        ```shell theme={null}
        curl localhost:8080/Greeter/Greet --json '"Alice"'
        ```

        Expected output:

        ```
        You said hi to Alice!
        ```

        You can see in the service logs and in the Restate UI how the request gets retried.
        On a retry, it skipped the steps that already succeeded.

        <img alt="Restate UI Durable Execution" />

        Even the sleep is durable and tracked by Restate.
        If you kill/restart the service halfway through, the sleep will only last for what remained.

        <Tip title="Durable Execution">
          Restate persists the progress of the handler.
          Letting you write code that is resilient to failures out of the box.
          Have a look at the [Durable Execution page](/foundations/key-concepts#durable-execution) to learn more.
        </Tip>
      </Step>
    </Steps>
  </Tab>

  <Tab title="Python" icon={"/img/languages/python.svg"}>
    <Info>
      **Prerequisites**:

      * Python >= v3.11
      * [uv](https://docs.astral.sh/uv/getting-started/installation/)
    </Info>

    <Steps>
      <Step title="Install Restate Server & CLI">
        Restate is a single self-contained binary. No external dependencies needed.

        <Tabs>
          <Tab title="Homebrew">
            ```shell theme={null}
            brew install restatedev/tap/restate-server restatedev/tap/restate
            ```

            Start the server:

            ```shell theme={null}
            restate-server
            ```
          </Tab>

          <Tab title="Download binaries">
            Download prebuilt binaries from the [releases page](https://github.com/restatedev/restate/releases/latest):

            <CodeGroup>
              ```shell MacOS-x64 theme={null}
              BIN=/usr/local/bin && RESTATE_PLATFORM=x86_64-apple-darwin && \
              curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
              tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
              tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
              chmod +x restate restate-server && \
              sudo mv restate $BIN && \
              sudo mv restate-server $BIN
              ```

              ```shell MacOS-arm64 theme={null}
              BIN=/usr/local/bin && RESTATE_PLATFORM=aarch64-apple-darwin && \
              curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
              tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
              tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
              chmod +x restate restate-server && \
              sudo mv restate $BIN && \
              sudo mv restate-server $BIN
              ```

              ```shell Linux-x64 theme={null}
              BIN=$HOME/.local/bin && RESTATE_PLATFORM=x86_64-unknown-linux-musl && \
              curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
              tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
              tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
              chmod +x restate restate-server && \
              mv restate $BIN && \
              mv restate-server $BIN
              ```

              ```shell Linux-arm64 theme={null}
              BIN=$HOME/.local/bin && RESTATE_PLATFORM=aarch64-unknown-linux-musl && \
              curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
              tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
              tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
              chmod +x restate restate-server && \
              mv restate $BIN && \
              mv restate-server $BIN
              ```
            </CodeGroup>

            Start the server:

            ```shell theme={null}
            restate-server
            ```
          </Tab>

          <Tab title="npm">
            ```shell theme={null}
            npm install --global @restatedev/restate-server@latest @restatedev/restate@latest
            ```

            Start the server:

            ```shell theme={null}
            restate-server
            ```
          </Tab>

          <Tab title="Docker">
            Run the Restate Server:

            ```shell theme={null}
            docker run --name restate_dev --rm \
            -p 8080:8080 -p 9070:9070 -p 9071:9071 \
            --add-host=host.docker.internal:host-gateway \
            docker.restate.dev/restatedev/restate:latest
            ```

            Run CLI commands:

            ```shell theme={null}
            docker run -it --network=host \
            docker.restate.dev/restatedev/restate-cli:latest \
            invocations ls
            ```

            Replace `invocations ls` with any CLI subcommand.
          </Tab>
        </Tabs>

        You can find the Restate UI running on port 9070 (`http://localhost:9070`) after starting the Restate Server.
      </Step>

      <Step title="Get the Greeter service template">
        <CodeGroup>
          ```shell CLI theme={null}
          restate example python-hello-world &&
          cd python-hello-world
          ```

          ```shell wget theme={null}
          wget https://github.com/restatedev/examples/releases/latest/download/python-hello-world.zip &&
          unzip python-hello-world.zip -d python-hello-world &&
          rm python-hello-world.zip && cd python-hello-world
          ```
        </CodeGroup>

        <GitHubLink />
      </Step>

      <Step title="Run the Greeter service">
        Run it and let it listen on port 9080 for requests:

        ```shell theme={null}
        uv run .
        ```
      </Step>

      <Step title="Register the service">
        Tell Restate where the service is running, so Restate can discover and register the services and handlers behind this endpoint.
        You can do this via the UI (`http://localhost:9070`) or via:

        <CodeGroup>
          ```shell CLI theme={null}
          restate deployments register http://localhost:9080
          ```

          ```shell curl theme={null}
          curl localhost:9070/deployments --json '{"uri": "http://localhost:9080"}'
          ```
        </CodeGroup>

        <Expandable title="Output">
          <CodeGroup>
            ```shell CLI theme={null}
            ❯ SERVICES THAT WILL BE ADDED:
            - Greeter
            Type: Service
            HANDLER  INPUT                                     OUTPUT
            greet    value of content-type 'application/json'  value of content-type 'application/json'


            ✔ Are you sure you want to apply those changes? · yes
            ✅ DEPLOYMENT:
            SERVICE  REV
            Greeter  1
            ```

            ```shell curl theme={null}
            {
                "id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                "services": [
                    {
                        "name": "Greeter",
                        "handlers": [
                            {
                                "name": "greet",
                                "ty": "Shared",
                                "input_description": "one of [\"none\", \"value of content-type 'application/json'\"]",
                                "output_description": "value of content-type 'application/json'"
                            }
                        ],
                        "ty": "Service",
                        "deployment_id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                        "revision": 1,
                        "public": true,
                        "idempotency_retention": "1day"
                    }
                ]
            }
            ```
          </CodeGroup>
        </Expandable>

        If you run Restate with Docker, register `http://host.docker.internal:9080` instead of `http://localhost:9080`.

        <Accordion title="Restate Cloud">
          When using [Restate Cloud](https://restate.dev/cloud), your service must be accessible over the public internet so Restate can invoke it.
          If you want to develop with a local service, you can expose it using our [tunnel](/deploy/server/cloud/#registering-restate-services-with-your-environment) feature.
        </Accordion>
      </Step>

      <Step title="Send a request to the Greeter service">
        Invoke the service via the Restate UI playground: go to `http://localhost:9070`,  click on your service and then on playground.

        <img alt="Restate UI Playground" />

        Or invoke via `curl`:

        ```shell theme={null}
        curl localhost:8080/Greeter/greet --json '{"name": "Sarah"}'
        ```

        Expected output: `You said hi to Sarah!`
      </Step>

      <Step title="Congratulations, you just ran Durable Execution!">
        The invocation you just sent used Durable Execution to make sure the request ran till completion.
        For each request, it sent a notification, slept for a second, and then sent a reminder.

        ```python {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/python/templates/python/app/greeter.py?collapse_prequel"}  theme={null}
        greeter = restate.Service("Greeter")


        @greeter.handler()
        async def greet(ctx: restate.Context, req: GreetingRequest) -> Greeting:
            # Durably execute a set of steps; resilient against failures
            greeting_id = str(ctx.uuid())
            await ctx.run_typed("notification", send_notification, greeting_id=greeting_id, name=req.name)
            await ctx.sleep(timedelta(seconds=1))
            await ctx.run_typed("reminder", send_reminder, greeting_id=greeting_id, name=req.name)

            # Respond to caller
            return Greeting(message=f"You said hi to {req.name}!")
        ```

        <GitHubLink />

        Send a request for `Alice` to see how the service behaves when it occasionally fails to send the reminder and notification:

        ```shell theme={null}
        curl localhost:8080/Greeter/greet --json '{"name": "Alice"}'
        ```

        Expected output:

        ```shell theme={null}
        You said hi to Alice!
        ```

        You can see in the service logs and in the Restate UI how the request gets retried.
        On a retry, it skipped the steps that already succeeded.

        <img alt="Restate UI Durable Execution" />

        Even the sleep is durable and tracked by Restate.
        If you kill/restart the service halfway through, the sleep will only last for what remained.

        <Tip title="Durable Execution">
          Restate persists the progress of the handler.
          Letting you write code that is resilient to failures out of the box.
          Have a look at the [Durable Execution page](/foundations/key-concepts#durable-execution) to learn more.
        </Tip>
      </Step>
    </Steps>
  </Tab>

  <Tab title="Rust" icon={"/img/languages/rust.svg"}>
    Select your favorite runtime:

    <Tabs>
      <Tab title="Tokio" icon={"/img/quickstart/tokio.svg"}>
        <Info>
          **Prerequisites**:

          * [Rust](https://rustup.rs/)
        </Info>

        <Steps>
          <Step title="Install Restate Server & CLI">
            Restate is a single self-contained binary. No external dependencies needed.

            <Tabs>
              <Tab title="Homebrew">
                ```shell theme={null}
                brew install restatedev/tap/restate-server restatedev/tap/restate
                ```

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="Download binaries">
                Download prebuilt binaries from the [releases page](https://github.com/restatedev/restate/releases/latest):

                <CodeGroup>
                  ```shell MacOS-x64 theme={null}
                  BIN=/usr/local/bin && RESTATE_PLATFORM=x86_64-apple-darwin && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  sudo mv restate $BIN && \
                  sudo mv restate-server $BIN
                  ```

                  ```shell MacOS-arm64 theme={null}
                  BIN=/usr/local/bin && RESTATE_PLATFORM=aarch64-apple-darwin && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  sudo mv restate $BIN && \
                  sudo mv restate-server $BIN
                  ```

                  ```shell Linux-x64 theme={null}
                  BIN=$HOME/.local/bin && RESTATE_PLATFORM=x86_64-unknown-linux-musl && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  mv restate $BIN && \
                  mv restate-server $BIN
                  ```

                  ```shell Linux-arm64 theme={null}
                  BIN=$HOME/.local/bin && RESTATE_PLATFORM=aarch64-unknown-linux-musl && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  mv restate $BIN && \
                  mv restate-server $BIN
                  ```
                </CodeGroup>

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="npm">
                ```shell theme={null}
                npm install --global @restatedev/restate-server@latest @restatedev/restate@latest
                ```

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="Docker">
                Run the Restate Server:

                ```shell theme={null}
                docker run --name restate_dev --rm \
                -p 8080:8080 -p 9070:9070 -p 9071:9071 \
                --add-host=host.docker.internal:host-gateway \
                docker.restate.dev/restatedev/restate:latest
                ```

                Run CLI commands:

                ```shell theme={null}
                docker run -it --network=host \
                docker.restate.dev/restatedev/restate-cli:latest \
                invocations ls
                ```

                Replace `invocations ls` with any CLI subcommand.
              </Tab>
            </Tabs>

            You can find the Restate UI running on port 9070 (`http://localhost:9070`) after starting the Restate Server.
          </Step>

          <Step title="Get the Greeter service template">
            <CodeGroup>
              ```shell CLI theme={null}
              restate example rust-hello-world &&
              cd rust-hello-world
              ```

              ```shell wget theme={null}
              wget https://github.com/restatedev/examples/releases/latest/download/rust-hello-world.zip &&
              unzip rust-hello-world.zip -d rust-hello-world &&
              rm rust-hello-world.zip && cd rust-hello-world
              ```
            </CodeGroup>

            <GitHubLink />
          </Step>

          <Step title="Run the Greeter service">
            ```shell theme={null}
            cargo run
            ```
          </Step>

          <Step title="Register the service">
            Tell Restate where the service is running, so Restate can discover and register the services and handlers behind this endpoint.
            You can do this via the UI (`http://localhost:9070`) or via:

            <CodeGroup>
              ```shell CLI theme={null}
              restate deployments register http://localhost:9080
              ```

              ```shell curl theme={null}
              curl localhost:9070/deployments --json '{"uri": "http://localhost:9080"}'
              ```
            </CodeGroup>

            <Expandable title="Output">
              <CodeGroup>
                ```shell CLI theme={null}
                ❯ SERVICES THAT WILL BE ADDED:
                - Greeter
                Type: Service
                HANDLER  INPUT                                     OUTPUT
                greet    value of content-type 'application/json'  value of content-type 'application/json'


                ✔ Are you sure you want to apply those changes? · yes
                ✅ DEPLOYMENT:
                SERVICE  REV
                Greeter  1
                ```

                ```shell curl theme={null}
                {
                    "id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                    "services": [
                        {
                            "name": "Greeter",
                            "handlers": [
                                {
                                    "name": "greet",
                                    "ty": "Shared",
                                    "input_description": "one of [\"none\", \"value of content-type 'application/json'\"]",
                                    "output_description": "value of content-type 'application/json'"
                                }
                            ],
                            "ty": "Service",
                            "deployment_id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                            "revision": 1,
                            "public": true,
                            "idempotency_retention": "1day"
                        }
                    ]
                }
                ```
              </CodeGroup>
            </Expandable>

            If you run Restate with Docker, register `http://host.docker.internal:9080` instead of `http://localhost:9080`.

            <Accordion title="Restate Cloud">
              When using [Restate Cloud](https://restate.dev/cloud), your service must be accessible over the public internet so Restate can invoke it.
              If you want to develop with a local service, you can expose it using our [tunnel](/deploy/server/cloud/#registering-restate-services-with-your-environment) feature.
            </Accordion>
          </Step>

          <Step title="Send a request to the Greeter service">
            Invoke the service via the Restate UI playground: go to `http://localhost:9070`, click on your service and then on playground.

            <img alt="Restate UI Playground" />

            Or invoke via `curl`:

            ```shell theme={null}
            curl localhost:8080/Greeter/greet --json '"Sarah"'
            ```

            Expected output:

            ```log theme={null}
            You said hi to Sarah!
            ```
          </Step>

          <Step title="Congratulations, you just ran Durable Execution!">
            The invocation you just sent used Durable Execution to make sure the request ran till completion.
            For each request, it sent a notification, slept for a second, and then sent a reminder.

            ```rust {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/rust/templates/rust/src/main.rs?collapse_prequel"}  theme={null}
            #[restate_sdk::service]
            trait Greeter {
                async fn greet(name: String) -> Result<String, HandlerError>;
            }

            struct GreeterImpl;

            impl Greeter for GreeterImpl {
                async fn greet(&self, mut ctx: Context<'_>, name: String) -> Result<String, HandlerError> {
                    // Durably execute a set of steps; resilient against failures
                    let greeting_id = ctx.rand_uuid().to_string();
                    ctx.run(|| send_notification(&greeting_id, &name))
                        .name("notification")
                        .await?;
                    ctx.sleep(Duration::from_secs(1)).await?;
                    ctx.run(|| send_reminder(&greeting_id, &name))
                        .name("reminder")
                        .await?;

                    // Respond to caller
                    Ok(format!("You said hi to {name}"))
                }
            }

            #[tokio::main]
            async fn main() {
                // To enable logging
                tracing_subscriber::fmt::init();

                HttpServer::new(Endpoint::builder().bind(GreeterImpl.serve()).build())
                    .listen_and_serve("0.0.0.0:9080".parse().unwrap())
                    .await;
            }
            ```

            <GitHubLink />

            Send a request for `Alice` to see how the service behaves when it occasionally fails to send the reminder and notification:

            ```shell theme={null}
            curl localhost:8080/Greeter/greet --json '"Alice"'
            ```

            Example output:

            ```shell theme={null}
            You said hi to Alice!
            ```

            You can see in the service logs and in the Restate UI how the request gets retried.
            On a retry, it skipped the steps that already succeeded.

            <img alt="Restate UI Durable Execution" />

            Even the sleep is durable and tracked by Restate.
            If you kill/restart the service halfway through, the sleep will only last for what remained.

            <Tip title="Durable Execution">
              Restate persists the progress of the handler.
              Letting you write code that is resilient to failures out of the box.
              Have a look at the [Durable Execution page](/foundations/key-concepts#durable-execution) to learn more.
            </Tip>
          </Step>
        </Steps>
      </Tab>

      <Tab title="Shuttle" icon={"/img/quickstart/shuttle-rust.png"}>
        <Info>
          **Prerequisites**:

          * [Rust](https://rustup.rs/)
          * [Shuttle](https://docs.shuttle.dev/getting-started/installation)
        </Info>

        <Steps>
          <Step title="Install Restate Server & CLI">
            Restate is a single self-contained binary. No external dependencies needed.

            <Tabs>
              <Tab title="Homebrew">
                ```shell theme={null}
                brew install restatedev/tap/restate-server restatedev/tap/restate
                ```

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="Download binaries">
                Download prebuilt binaries from the [releases page](https://github.com/restatedev/restate/releases/latest):

                <CodeGroup>
                  ```shell MacOS-x64 theme={null}
                  BIN=/usr/local/bin && RESTATE_PLATFORM=x86_64-apple-darwin && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  sudo mv restate $BIN && \
                  sudo mv restate-server $BIN
                  ```

                  ```shell MacOS-arm64 theme={null}
                  BIN=/usr/local/bin && RESTATE_PLATFORM=aarch64-apple-darwin && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  sudo mv restate $BIN && \
                  sudo mv restate-server $BIN
                  ```

                  ```shell Linux-x64 theme={null}
                  BIN=$HOME/.local/bin && RESTATE_PLATFORM=x86_64-unknown-linux-musl && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  mv restate $BIN && \
                  mv restate-server $BIN
                  ```

                  ```shell Linux-arm64 theme={null}
                  BIN=$HOME/.local/bin && RESTATE_PLATFORM=aarch64-unknown-linux-musl && \
                  curl -L --remote-name-all https://restate.gateway.scarf.sh/latest/restate-{server,cli}-$RESTATE_PLATFORM.tar.xz && \
                  tar -xvf restate-server-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-server-$RESTATE_PLATFORM/restate-server && \
                  tar -xvf restate-cli-$RESTATE_PLATFORM.tar.xz --strip-components=1 restate-cli-$RESTATE_PLATFORM/restate && \
                  chmod +x restate restate-server && \
                  mv restate $BIN && \
                  mv restate-server $BIN
                  ```
                </CodeGroup>

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="npm">
                ```shell theme={null}
                npm install --global @restatedev/restate-server@latest @restatedev/restate@latest
                ```

                Start the server:

                ```shell theme={null}
                restate-server
                ```
              </Tab>

              <Tab title="Docker">
                Run the Restate Server:

                ```shell theme={null}
                docker run --name restate_dev --rm \
                -p 8080:8080 -p 9070:9070 -p 9071:9071 \
                --add-host=host.docker.internal:host-gateway \
                docker.restate.dev/restatedev/restate:latest
                ```

                Run CLI commands:

                ```shell theme={null}
                docker run -it --network=host \
                docker.restate.dev/restatedev/restate-cli:latest \
                invocations ls
                ```

                Replace `invocations ls` with any CLI subcommand.
              </Tab>
            </Tabs>

            You can find the Restate UI running on port 9070 (`http://localhost:9070`) after starting the Restate Server.
          </Step>

          <Step title="Get the Greeter service template">
            <CodeGroup>
              ```shell CLI theme={null}
              restate example rust-hello-world-shuttle &&
              cd rust-hello-world-shuttle
              ```

              ```shell wget theme={null}
              wget https://github.com/restatedev/examples/releases/latest/download/rust-hello-world-shuttle.zip &&
              unzip rust-hello-world-shuttle.zip -d rust-hello-world-shuttle &&
              rm rust-hello-world-shuttle.zip && cd rust-hello-world-shuttle
              ```
            </CodeGroup>

            <GitHubLink />
          </Step>

          <Step title="Run the Greeter service">
            ```shell theme={null}
            cargo shuttle run --port 9080
            ```
          </Step>

          <Step title="Register the service">
            Tell Restate where the service is running, so Restate can discover and register the services and handlers behind this endpoint.
            You can do this via the UI (`http://localhost:9070`) or via:

            <CodeGroup>
              ```shell CLI theme={null}
              restate deployments register http://localhost:9080 --use-http1.1
              ```

              ```shell curl theme={null}
              curl localhost:9070/deployments --json '{"uri": "http://localhost:9080", "use_http_11": true}'
              ```
            </CodeGroup>

            Expected output:

            <CodeGroup>
              ```shell CLI theme={null}
              ❯ SERVICES THAT WILL BE ADDED:
              - Greeter
              Type: Service
              HANDLER  INPUT                                     OUTPUT
              greet    value of content-type 'application/json'  value of content-type 'application/json'


              ✔ Are you sure you want to apply those changes? · yes
              ✅ DEPLOYMENT:
              SERVICE  REV
              Greeter  1
              ```

              ```shell curl theme={null}
              {
              "id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                  "services": [
                      {
                          "name": "Greeter",
                          "handlers": [
                              {
                                  "name": "greet",
                                  "ty": "Shared",
                                  "input_description": "one of [\"none\", \"value of content-type 'application/json'\"]",
                                  "output_description": "value of content-type 'application/json'"
                              }
                          ],
                          "ty": "Service",
                          "deployment_id": "dp_17sztQp4gnEC1L0OCFM9aEh",
                          "revision": 1,
                          "public": true,
                          "idempotency_retention": "1day"
                      }
                  ]
              }
              ```
            </CodeGroup>

            &#x20;does not support HTTP2, so we need to tell Restate to use HTTP1.1.

            If you run Restate with Docker, use `http://host.docker.internal:9080` instead of `http://localhost:9080`.

            <Info title="Restate Cloud">
              When using [Restate Cloud](https://restate.dev/cloud), your service must be accessible over the public internet so Restate can invoke it.
              If you want to develop with a local service, you can expose it using our [tunnel](/deploy/server/cloud/#registering-restate-services-with-your-environment) feature.
            </Info>
          </Step>

          <Step title="Send a request to the Greeter service">
            Invoke the service via the Restate UI playground: go to `http://localhost:9070`,  click on your service and then on playground.

            <img alt="Restate UI Playground" />

            Or invoke via `curl`:

            ```shell theme={null}
            curl localhost:8080/Greeter/greet --json '{"name": "Sarah"}'
            ```

            Expected output: `You said hi to Sarah!`
          </Step>

          <Step title="Congratulations, you just ran Durable Execution!">
            The invocation you just sent used Durable Execution to make sure the request ran till completion.
            For each request, it sent a notification, slept for a second, and then sent a reminder.

            ```rust {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/rust/templates/rust-shuttle/src/main.rs?collapse_prequel"}  theme={null}
            #[restate_sdk::service]
            trait Greeter {
                async fn greet(name: String) -> Result<String, HandlerError>;
            }

            struct GreeterImpl;

            impl Greeter for GreeterImpl {
                async fn greet(&self, mut ctx: Context<'_>, name: String) -> Result<String, HandlerError> {
                    // Durably execute a set of steps; resilient against failures
                    let greeting_id = ctx.rand_uuid().to_string();
                    ctx.run(|| send_notification(&greeting_id, &name))
                        .name("notification")
                        .await?;
                    ctx.sleep(Duration::from_secs(1)).await?;
                    ctx.run(|| send_reminder(&greeting_id, &name))
                        .name("reminder")
                        .await?;

                    // Respond to caller
                    Ok(format!("You said hi to {name}"))
                }
            }

            #[shuttle_runtime::main]
            async fn main() -> Result<RestateShuttleEndpoint, shuttle_runtime::Error> {
                Ok(RestateShuttleEndpoint::new(
                    Endpoint::builder().bind(GreeterImpl.serve()).build(),
                ))
            }
            ```

            <GitHubLink />

            Send a request for `Alice` to see how the service behaves when it occasionally fails to send the reminder and notification:

            ```shell theme={null}
            curl localhost:8080/Greeter/greet --json '"Alice"'
            ```

            Example output:

            ```shell theme={null}
            You said hi to Alice!
            ```

            You can see in the service logs and in the Restate UI how the request gets retried.
            On a retry, it skipped the steps that already succeeded.

            <img alt="Restate UI Durable Execution" />

            Even the sleep is durable and tracked by Restate.
            If you kill/restart the service halfway through, the sleep will only last for what remained.

            <Tip title="Durable Execution">
              Restate persists the progress of the handler.
              Letting you write code that is resilient to failures out of the box.
              Have a look at the [Durable Execution page](/foundations/key-concepts#durable-execution) to learn more.
            </Tip>
          </Step>
        </Steps>
      </Tab>
    </Tabs>
  </Tab>
</Tabs>

<Tip>
  [Once you are done with the quickstart, check out how to write Restate apps with AI assistants.](/develop/ai-assistant)
</Tip>


# Restate Architecture
Source: https://docs.restate.dev/references/architecture

Restate architecture and distributed deployment concepts

Restate is designed to be extremely simple to get started with by delivering all the functionality in a single binary with minimal upfront configuration needs. In particular, when starting out by running Restate on a single node, you don't need to understand its internal architecture in a great level of detail.

As you begin to plan for more complex deployment scenarios, you will benefit from having a deeper understanding of the various components and how they fit together to support scalable and resilient clusters. The goal of this section is to introduce the terminology we use throughout the server documentation and inform the choices involved in configuring Restate clusters.

## Overview

This section explains how Restate is built today: a partitioned, log-centric runtime where events are synchronously replicated between Restate server nodes, and partition-processor state is periodically snapshotted to S3 for bounded, fast recovery. The description focuses on the current implementation and omits optional storage trade-offs.

<Frame>
  <img alt="Restate Cluster Overview" />
</Frame>

## Components

At a high level, Restate interposes a log-first runtime between clients and your handlers, with clear separation of ingress, durability, execution, and control.

<Frame>
  <img alt={"Restate Node internal architecture"} />
</Frame>

### Ingress

Ingress is the front door for client and internal calls, identifies the target service/handler and partition (via workflow ID, virtual-object key, or idempotency key), and forwards the call to the current leader of that partition. Ingress is leader-aware; when leadership changes due to failover or rebalancing, routing updates automatically without client involvement.

### Durable Log (“Bifrost”)

The log is the primary durability layer. Each partition has a single sequencer/leader that orders events and replicates them to peer replicas on other Restate nodes. A write is committed when a quorum of replicas acknowledges the append. Restate uses a segmented virtual log: the active segment receives appends; reconfiguration seals the active segment and atomically publishes a new segment as the head. Segmentation enables clean and fast leadership changes, placement updates, and other reconfiguration without copying data.

### Partition Processor

Every partition has one processor leader (and optional followers). The processor tails the log, invokes your handler code via a bidirectional stream, and maintains a materialized state cache in an embedded RocksDB. This cache holds journals, idempotency metadata, key-scoped state for virtual objects, and timer indices—everything required for low-latency execution. Followers track the same state and can become leader quickly on failure.

### Control Plane

The control plane holds cluster metadata (configurations, partition placement, epochs, segment descriptors) behind a consensus  interface (built-in Raft). A cluster controller manages log nodes and processors and initiates failover when health checks fail. Strong consensus is confined to this metadata layer, and the data path “borrows” consensus in the form of a leader/epoch configuration which the control plane revokes and re-assignes upon failover, rebalancing, and other reconfigurations.

## Durability and storage model

The system treats the replicated log as ground truth and uses S3-backed snapshots to bound recovery time without introducing a second source of truth.

<Frame>
  <img alt="Durability and storage with RocksDB" />
</Frame>

### Hot-path durability

An operation “happens” when the partition leader appends its record to the log and receives quorum acks. That commit point defines the durable order of invocations, steps, state updates, messages, timers, and completions.

### Materializing state for fast access

The processor leader maintains a full cache of the partition’s materialized state in RocksDB for fast random reads and updates during execution. This cache is derivative—it can always be rebuilt from the log—and is not a second source of truth. In typical configurations, partitions also have processor followers, which maintain a materialization of the partition state, for fast failover.

### Recovery bounded by snapshots

Processors create periodic snapshots of their RocksDB partition store and upload them to S3. On restart or takeover, a fresh partition processor can download the latest snapshot and replay only the log suffix since the snapshot's sequence number. Once the partition store state is considered safely persisted (based on the configured [durability mode](/server/snapshots#understanding-durability-modes)), the corresponding log entries can be trimmed to cap storage and replay cost.

Partition processors fetch snapshots only when they were not previously a leader or follower and need to bootstrap a new copy of the partition state, or when they fall behind the log's trim point.

<Frame>
  <img alt={"Restate Server event loop"} />
</Frame>

## Partitioned scale-out and addressability

Restate scales by sharding orchestration and state by key, keeping hot paths partition-local and cross-partition work explicit.

<Frame>
  <img alt={"Restate invocation flow"} />
</Frame>

### Keyed routing

Workflow IDs, virtual-object keys, or idempotency keys deterministically hash to a partition. Non-keyed invocations are placed for locality by the ingress. The invocation ID encodes the partition, enabling efficient subsequent routing and lookups.

### Co-sharding of orchestration and state

Each partition owns both orchestration (invocation lifecycle, journaling, timers) and the state cache for its keys. Steps, state updates, and timers for a key execute entirely within one partition—no cross-partition coordination is needed for the hot path.

### Cross-partition actions

When a handler targets a different key (e.g., sends a message or performs an RPC to another service keyed on a different partition), the event is recorded in the origin partition’s log and delivered exactly once to the destination partition via an internal shuffler. Delivery is addressed and deduplicated by sequence numbers; the receiving partition treats it as a normal log-first operation.

### Elastic operations

While the number of partitions is configured at cluster creation today, the addressing scheme and segment abstraction are designed so the system can migrate partitions or split key ranges in future versions without violating ordering or idempotency guarantees.

## Write path and step lifecycle

To make the mechanics concrete, the flow below traces an end-to-end invocation of processPayment keyed by idempotency key K.

<Steps>
  <Step title="Client → Ingress (enqueue)">
    A client invokes processPayment with key K. The ingress hashes K to select the target partition and enqueues the invocation to that partition’s log.
  </Step>

  <Step title="Processor leader claims and starts execution">
    The partition’s processor leader consumes the enqueue event, checks its local idempotency state for K, and sees that K is not present for processPayment. It atomically records K (idempotency) and transitions the invocation to RUNNING, then opens a bidirectional stream to the target service endpoint and pushes the initial invoke journal entry to the handler.
  </Step>

  <Step title="Durable step commit (ctx.run)">
    As the service executes, it streams back a step result event (e.g., from `ctx.run`). The processor appends this step journal entry to the log. The moment this append is replicated to quorum defines “the step happened.” From then on, the step will be recovered on retries and won’t be re-executed.
  </Step>

  <Step title="Materialization and ack">
    When the processor subsequently reads the committed step event back from the log (confirming it still holds leadership), it adds the entry to the invocation’s journal state (and any relevant indexes) and acks the handler so the user code can proceed with the next step.
  </Step>

  <Step title="Other durable actions (state/timers/RPC/promises)">
    The same pattern applies to state updates, timers, inter-service RPC/messages, or durable promises/futures: each action is added to the log first, and upon reading the committed record, the processor applies it (e.g., updates key-scoped state, registers/fires a timer, routes an RPC to another partition via the internal shuffler, or resolves a promise).
  </Step>

  <Step title="Completion">
    When the handler finishes, the processor appends a Result event to the log. After reading that event back, it marks the invocation COMPLETE and returns the result to the client (or emits the completion signal for async flows).
  </Step>

  <Step title="Failures and retries (epoch fencing)">
    If execution fails at any point (process crash, stream loss, user exception), the processor dispatches a new attempt and attaches the full journal so far. Attempts carry monotonically increasing epochs; the processor rejects any events from superseded epochs (late messages from older attempts), preventing split-brain effects. This bookkeeping is partition-local and efficient because invocations are sticky to a single partition with a strong leader.
  </Step>
</Steps>

Note: Duplicate client calls with the same idempotency key K resolve to the same invocation: the processor returns the already-committed result from the journal without re-executing steps.

## Failover and reconfiguration

Leader loss or placement changes trigger a sealed-segment handover and epoch fencing that keep ordering and exactly-once properties intact.

### Detection and triggering

The cluster controller heartbeats sequencers and processors. Partition processors and log servers additionally gossip their detection of peer failures for faster failover handling.

### Log failover

On reconfiguration, the controller seals the active segment (a quorum of replicas refuses further appends), determines the authoritative tail of the log segment, elects a new sequencer/replica set, and performs a metadata CAS to publish the new segment as head. Appenders and readers consult metadata and continue on the new head.

<Frame>
  <img alt="Log failover" />
</Frame>

### Processor failover

A follower (or a restarted processor) is promoted, obtains a new epoch, appends an epoch-bump record. The new leader ensures it can resume the log’s event stream (possibly restoring the latest S3 snapshot, if it was not previously a follower). Any late appends from superseded leaders or stale handler attempts (carrying lower epochs) are fenced at the epoch boundary and ignored.

### Routing continuity

Ingress consults the updated control-plane metadata and routes directly to the new leaders. Clients do not need to reconnect to different endpoints or re-negotiate sessions; failover is transparent at the API boundary.

## Nodes and roles

You'll see many mentions of the terms server and node throughout this documentation. Generally, we use the term "server" to refer to a running instance of the `restate-server` binary. This binary can host multiple functions. When you start a single-node Restate server, for example when doing some local development or testing, you are hosting all the essential features in a single process. These include accepting incoming requests, durably recording events, processing work (delegating invocations to services, handling key-value operations), as well as maintaining metadata used internally by the system.

At its simplest, running a cluster is not that different - multiple nodes cooperate to share the responsibilities we mentioned earlier. This is accomplished by having multiple copies of the server process running on separate machines, although it is possible to create test clusters on a single machine. **Nodes** are therefore distinct instances of the Restate server within a cluster.

Restate clusters are designed to scale out in support of large deployments. As you add more machines, it becomes wasteful to replicate all the functionality across all the machines in a cluster, since not all features need to scale out at the same rate. **Roles** control which features run on any given node, enabling specialization within the cluster.

Here is an overview of the different roles that can run on a node:

<Frame>
  <img alt="Restate Node internal architecture" />
</Frame>

* Metadata server: the source of truth for cluster-wide information
* Ingress: the entry point for external requests
* Log server: responsible for durably persisting the log
* Worker: houses the partition processors

### Metadata store

The Restate metadata store is part of the control plane and is the internal source of truth for node membership and responsibilities. It is essential to the correctness of the overall system: In a cluster this service enables distributed consensus about other components' configuration. All nodes in a Restate cluster must be able to access the metadata store, though not all members of the cluster need to be part of hosting it. Restate includes a built-in Raft-based metadata store which is hosted on all nodes running the `metadata-server` role.

The metadata store is designed to support relatively low volumes of read and write operations (at least compared to other parts of Restate), with the highest level of integrity and availability.

### Ingress

External requests enter the Restate cluster via the HTTP ingress component, which runs on nodes assigned the `http-ingress` role. Compared to other roles, the HTTP ingress role does not involve long-lived state and it can move around relatively freely, since it only handles ongoing client connections.

### Log servers

Log server nodes running the `log-server` role are responsible for durably persisting the log. If the log is the equivalent of a WAL, then partition stores are the materializations that enable efficient reads of the events (invocation journals, key-value data) that have been recorded. Depending on the configured **log replication** requirements, Restate will replicate log records to multiple log servers to persist a given log, and this will change over time to support maintenance and resizing of the cluster.

### Workers

Nodes assigned the `worker` role run the partition processors, which are the Restate components responsible for maintaining the partition store.
Partition processors can operate in either leader or follower mode.
Only a single leader for a given partition can be active at a time, and this is the sole processor that handles invocations to deployed services.
Followers keep up with the log without taking action, and are ready to take over in the event that the partition's leader becomes unavailable.
The overall number of processors per partition is configurable via the **partition replication** configuration option.

<Frame>
  <img alt="Restate invocation flow" />
</Frame>

Partition processors replicate their state by following and applying the log for their partition.
If a processor needs to stop, for example for scheduled maintenance, it will typically catch up on the records it missed by reading them from the cluster's log servers once it comes back online.
Occasionally, a worker node might lose a disk - or you might need to grow your cluster by adding fresh nodes to it.
In these cases, it's far more efficient to obtain a **snapshot** of the partition state from a recent point in time than to replay all the missing log events.
Restate clusters can be configured to use an external **object store** as the snapshot repository, allowing partition processors to skip ahead in the log.
This also enables us to **trim logs** which might otherwise grow unboundedly.

## Other reading material

* [Blog post: Building a modern Durable Execution Engine from First Principles](https://restate.dev/blog/building-a-modern-durable-execution-engine-from-first-principles/)
* [Blog post: Distributed Restate - a first look](https://restate.dev/blog/distributed-restate-a-first-look/)
* [Blog post: Every System is a Log](https://restate.dev/blog/every-system-is-a-log-avoiding-coordination-in-distributed-applications/)


# CLI Configuration
Source: https://docs.restate.dev/references/cli-config

Restate CLI configuration options.

You can use the CLI to interact with Restate, and manage your services, deployments and invocations.

Have a look at the [CLI installation docs](/installation).

There are 2 ways to configure the CLI: via environment variables or via a configuration file.

## Using environment variables

You can specify the following environment variables:

* `RESTATE_HOST`: The hostname/IP address of the server. Default is `localhost`.
* `RESTATE_HOST_SCHEME`: Default is `http`.
* `RESTATE_ADMIN_URL`: To specify the full URL of the admin server (scheme+host+port).
* `RESTATE_AUTH_TOKEN`: Set if authentication is required.

For example, to specify the hostname `myhost` and the host scheme `https`, pass environment variables as follows:

```shell theme={null}
RESTATE_HOST=myhost RESTATE_HOST_SCHEME=https restate <command>
```

<Info>
  You can find the full list of configuration variables in the [CLI GitHub repo](https://github.com/restatedev/restate/blob/main/cli/src/cli_env.rs).
</Info>

You can also specify the configuration in a `.env` file. The CLI will look for a `.env` file in its current directory.

For example, to connect to a Restate admin server running at `http://myhost:9070`, save the following in a `.env` file:

```shell .env theme={null}
RESTATE_ADMIN_URL=http://myhost:9070
```

## Using the config file

By default, the CLI will treat `$HOME/.config/restate` as its config directory.
This is configurable with `$RESTATE_CLI_CONFIG_HOME`. Restate will look for file
named `config.toml` inside this directory. You can edit this file with
`restate config edit`, or view its contents with `restate config view`.

The config file has a native concept of 'enviroments' which are sets of
configuration intended to specify different instances of Restate. You can
list your configured environments:

```bash theme={null}
> restate config list-environments
CURRENT  NAME   ADMIN_BASE_URL
*        local  http://localhost:9070/
```

By default, the CLI uses the `local` environment which is configured to point
at a Restate instance running on your local machine. Additional environments
can be specified as new blocks in `config.toml`:

```toml config.toml theme={null}
[myhost]
ingress_base_url = "http://myhost:8080"
admin_base_url = "http://myhost:9070"
bearer_token = "..."
```

The title of the block is the name of the environment. You can switch
between environments in various ways:

1. With an argument: `restate -e myhost whoami`
2. With an environment variable: `RESTATE_ENVIRONMENT=myhost restate whoami`
3. With the command `restate config use-environment myhost`. This will write
   the name of the environment to the CLI config directory, so that it's used by
   default for all CLI commands. You can switch back to `local` with
   `restate config use-environment local`.


# Error Codes
Source: https://docs.restate.dev/references/errors

Descriptions of error codes emitted by Restate components

This page contains the list of error codes emitted by Restate components.

## META0003

Cannot reach the service endpoint to execute discovery. Make sure:

* The provided `URI`/`ARN` is correct
* The deployment is up and running
* Restate can reach the deployment through the configured `URI`/`ARN`
* If additional authentication is required, make sure it's configured through `additional_headers`

## META0004

Cannot register the provided deployment, because it conflicts with the uri of an already registered deployment.

In Restate deployments have a unique uri/arn and are immutable, thus it's not possible to discover the same deployment twice.
Make sure, when updating a deployment, to assign it a new uri/arn.

You can force the override using the `"force": true` field in the discover request, but beware that this can lead in-flight invocations to an unrecoverable error state.

See the [versioning documentation](https://docs.restate.dev/operate/versioning) for more information.

## META0005

Cannot propagate deployment/service metadata to Restate services. If you see this error when starting Restate, this might indicate a corrupted Meta storage.

We recommend wiping the Meta storage and recreating it by registering deployments in the same order they were registered before.

## META0006

Cannot register the newly discovered service revision in the provided deployment, because it conflicts with an already existing service revision.

When implementing a new service revision, make sure that:

* The service type is the same as the previous revision.
* The new revision contains at least all the handlers of the previous revision.

See the [versioning documentation](https://docs.restate.dev/operate/versioning) for more information.

## META0009

The provided subscription is invalid. Subscriptions should have:

* A `source` field in the format of `kafka://<CLUSTER_NAME>/<TOPIC_NAME>`. When registering, the Kafka cluster should be configured in the Restate configuration.
* A `sink` field in the format of `service://<service_NAME>/<HANDLER_NAME>`. When registering, service and handler should be available already in the registry, meaning they have been previously registered.
* Additional constraints may apply depending on the sink service type.

Please look at the Kafka documentation (for [TypeScript](https://docs.restate.dev/develop/ts/kafka) and [Java](https://docs.restate.dev/develop/java/kafka)) for more details on subscriptions and event handlers.

## META0010

Trying to open meta storage directory, configured via `meta.storage_path`, which contains incompatible data. This indicates that your data was written with a different Restate version than you are running right now.

Suggestions:

* Up/Downgrade your Restate server to the requested version.
* Migrate your data to the requested version by running the migration scripts.
* Wipe your meta storage directory to start afresh via `rm -rf <BASE_DIR>/<NODE_NAME>/local-metadata-store`.
* Configure a different meta storage directory via `meta.storage_path`.

## META0011

Non-empty meta storage directory, configured via `meta.storage_path`, is missing the version file. This indicates data corruption or that the data has been written with an incompatible Restate version \< 0.8.

Suggestions:

* Wipe your meta storage directory to start afresh via `rm -rf <BASE_DIR>/<NODE_NAME>/local-metadata-store`.
* Configure a different meta storage directory via `meta.storage_path`.
* Downgrade your Restate server to  0.7.

## META0012

Trying to register a service endpoint whose supported service protocol versions is incompatible with the server. This indicates that you have to upgrade your server to make it work together with the deployed SDK.

Suggestions:

* Check the compatibility matrix between SDK and server versions
  * Try upgrading to a server version which is compatible with your SDK
  * Try using an SDK version which is compatible with your server

## META0013

Received a bad service discovery response from the specified service endpoint. This indicates that you are trying to register a service endpoint with an incompatible server.

Suggestions:

* Check the compatibility matrix between SDK and server versions
  * Either deploy a server version which is compatible with your SDK
  * Or use an SDK version which is compatible with your server

## META0014

Service discovery response failed, and the server may have responded in HTTP1.1.
This can happen when discovering locally running dev servers from Faas platforms
eg `wrangler dev`. FaaS platforms in generally will support HTTP2, however, so
this is only a local development concern.

You can try to discover the endpoint with `--use-http1.1` when working
with these local dev servers. This should not be needed in production.

## META0015

The service discovery response suggested that the SDK is serving in
bidirectional protocol mode, but discovery is going over a protocol that does
not support it (currently only Lambda).

Lambda endpoints do not support the bidirectional protocol mode and should be
configured to announce themselves as being in request-response mode upon
discovery.

## META0016

Cannot update the provided deployment with the discovered metadata, because the new metadata is insufficiently similar to the old.

When updating a deployment, make sure that:

* All services have the same type as they did in the previous deployment.
* All services contain at least all the handlers that they did in the previous deployment.
* The updated deployment contains at least all the services that it previously did.
* The updated deployment has exactly the same supported protocol versions, which generally means you want to use the same SDK minor version.

See the [versioning documentation](https://docs.restate.dev/operate/versioning) for more information.

## META0017

Cannot force-add the provided URI/ARN as a new deployment, because two or more existing deployments use this URI.

Generally Restate enforces that there is only one deployment for a given destination (a HTTP URI or Lambda ARN). This means
that rediscovering the same destination requires a `force` flag. Adding a deployment in force mode instructs Restate to replace the existing deployment
with that destination, with the discovered metadata. This relies on there being an unambiguous deployment to replace.

When using the `PUT /deployments/{deployment_id}` API to update a deployment in place, it is possible to create two deployments that have the same destination.
This is intended to be a temporary measure to fix failing invocations on a draining deployment. While in this state, it is not possible to force deploy that same destination.
Instead, one of the two deployments must be deleted (`DELETE /deployments/{deployment_id}`) so that there is an unambiguous deployment to replace.

See the [versioning documentation](https://docs.restate.dev/operate/versioning) for more information.

## RT0001

The invocation response stream was aborted due to the timeout configured in `worker.invoker.abort_timeout`.
This timeout is fired when Restate has an open invocation, and it's waiting only for response messages, but no message is seen for the configured time.

Suggestions:

* Check for bugs in your code. Most likely no message was sent to Restate because your code is blocked and/or reached a deadlock.
* If your code is supposed to not send any message to Restate for longer than the configured timeout, because for example is doing a blocking operation that takes a long time, change the configuration accordingly.

## RT0002

Cannot start Restate because the configuration cannot be parsed. Check the configuration file and the environment variables provided.

For a complete list of configuration options, and a sample configuration, check [https://docs.restate.dev/operate/configuration](https://docs.restate.dev/operate/configuration)

## RT0003

The invocation failed because Restate received a message from a service larger than the `worker.invoker.message_size_limit`.

Suggestions:

* Check in your code whether there is a case where a very large message can be generated, such as a state entry being too large, a request payload being too large, etc.
* Increase the limit by tuning the `worker.invoker.message_size_limit` config entry, eventually tuning the memory of your operating system/machine where Restate is running.

## RT0004

Failed starting process because it could not bind to configured address.
This happens usually if another process has already bound to this address.

Suggestions:

* Select an address that is free.
* Stop the process that has bound to the specified address.
* Make sure you have the permissions to bind to the configured port. Some operating systems require admin/root privileges to bind to ports lower than 1024.

## RT0005

Failed opening RocksDB, because the db file is currently locked.\
This happens usually if another process still holds the lock.

Suggestions:

* Check no other Restate process is running and using the same db file.
* Configure a different RocksDB storage directory via `worker.storage_rocksdb.path`.

## RT0006

A generic error occurred while invoking the service.
We suggest checking the service/deployment logs as well to get any hint on the error cause.

## RT0007

A retry-able error was received from the handler while processing the invocation.
Restate will soon retry executing the invocation, replaying from the point where it left.

Suggestions:

* Check the service logs to get more info about the error cause, like the stacktrace.
* Look at the error handling docs for more info about error handling in services (e.g. [https://docs.restate.dev/develop/ts/error-handling](https://docs.restate.dev/develop/ts/error-handling) or [https://docs.restate.dev/develop/java/error-handling](https://docs.restate.dev/develop/java/error-handling)).

## RT0009

Trying to open worker storage directory, configured via `worker.storage_rocksdb.path`, which contains no storage format version information. This indicates data corruption or that the data has been written with an incompatible Restate version \< 0.8.

Suggestions:

* Wipe your meta storage directory to start afresh via `rm -rf <BASE_DIR>/<NODE_NAME>/db`.
* Configure a different worker storage directory via `worker.storage_rocksdb.path`.
* Downgrade your Restate server to \< 0.8.

## RT0010

Network error when interacting with the service endpoint.
This can be caused by a variety of reasons including:

* The service is (temporarily) down
* The service is (temporarily) not reachable over the network
* Your network security setup blocks Restate from reaching the service
* A config error where the registered service endpoint and the actually deployed service endpoint differ

## RT0011

No deployment found for the given service.
This might indicate that the service and/or the associated deployment was removed from the schema registry before starting to process the invocation.

Check whether the deployment still exists using `restate deployments list` or by looking in the UI.

## RT0012

Protocol violation error.
This can be caused by an incompatible runtime and SDK version, or by an SDK bug.

If the error persists, please file a bug report here: [https://github.com/restatedev/restate/issues](https://github.com/restatedev/restate/issues).

## RT0013

The service endpoint does not support any of the supported service protocol versions of the server.
Please make sure that the service endpoint's SDK and the Restate server are compatible.

Suggestions:

* Register a service endpoint which uses an SDK which is compatible with the used server
* Upgrade the server to a version which is compatible with the used SDK

## RT0014

The server cannot resume an in-flight invocation which has been started with a now incompatible service protocol version.
Restate does not support upgrading service protocols yet.

Suggestions:

* Downgrade the server to a version which is compatible with the used service protocol version
* Kill the affected invocation via the CLI.

## RT0015

The server can't establish an invocation stream because the SDK does not support the service protocol version negotiated during discovery.
This indicates that the SDK was updated to a new version that dropped support for old service protocol versions, but no re-registration was performed.

Suggestions:

* For in-flight invocations, downgrade the SDK version back to the previous version.
* For new invocations, register a new deployment with a new endpoint as described here: [https://docs.restate.dev/operate/versioning#deploying-new-service-versions](https://docs.restate.dev/operate/versioning#deploying-new-service-versions).
* Make sure the new SDK is compatible with this runtime version, for more info check out [https://docs.restate.dev/operate/upgrading#service-compatibility](https://docs.restate.dev/operate/upgrading#service-compatibility).

## RT0016

Journal mismatch detected when replaying the invocation: the handler generated a sequence of journal entries (thus context operations) that doesn't exactly match the recorded journal.
This indicates that either the service code was changed (e.g. the service container image updated) without registering a new version of the service deployment, or some code within the handler is non-deterministic.

To fix it:

* Update the deployment in place, fixing the bug, as described in [https://docs.restate.dev/operate/versioning/#updating-deployments-in-place](https://docs.restate.dev/operate/versioning/#updating-deployments-in-place)
* If you can afford to lose partial progress, then kill the invocation as described in [https://docs.restate.dev/operate/invocation/#killing-invocations](https://docs.restate.dev/operate/invocation/#killing-invocations)

Some common mistakes that lead to non-deterministic errors are:

* Branch the execution flow based on some non-deterministic information, such as the elapsed time between now and another timestamp, or the result of an HTTP request that was not recorded using the `ctx.run` feature.
* A parameter passed to a `Context` operation is non-deterministic, for example setting a state key using a random value or the current date-time.
* Execute a sequence of `Context` operations, such as calling other services, while iterating over a data structure with non-deterministic iteration order (such as sets/maps/dictionaries).

For more info about determinism and journaling of non-deterministic operations, check out [https://docs.restate.dev/get\_started/tour/#journaling-actions](https://docs.restate.dev/get_started/tour/#journaling-actions).

## RT0017

The entry cannot be processed due to a failed precondition.
This entry, and all the subsequent received entries, have been discarded and Restate will retry executing the invocation from the last recorded entry.

Entry preconditions are usually checked by the SDK.
If the error persists, please file a bug report here: [https://github.com/restatedev/restate/issues](https://github.com/restatedev/restate/issues).

## RT0018

The request submitted through the `Context` API to the given service handler cannot be processed, because the service handler doesn't exist.

Make sure the service/handler is registered, by using `restate svc ls` or through the UI:

* If the service/handler is correctly registered, you can ignore this error as it's a transient error, due to internal propagation of the cluster metadata.
* If the service/handler is not registered, you must register it in order for this invocation to progress.

## RT0019

The service replied with Content Too Large/Payload Too Large.
If you deploy on a serverless platform like Vercel, this might indicate that you're hitting the payload size limit imposed by service providers.

Suggestions:

* Check in your code whether there is a case where a very large message can be generated, such as a state being too large, a `ctx.run` result being too large, etc.
* Ask the service provider to increase the limits, if possible.

## RT0020

The requested service is exposed by a deployment using a deprecated service protocol version. New requests won't be accepted.
Upgrade the SDK in your service code and register it as a new deployment to continue accepting requests to the requested service.

## RT0021

The `inactivity_timeout` option is ignored when the deployment serving the services/handlers is using Request/Response mode, such as when deploying on Lambda or using HTTP/1.1

## RT0022

The configured `journal_retention` valued was clamped, because it's higher than the maximum value configured for the server.


# Go API
Source: https://docs.restate.dev/references/gopkg





# Javadocs
Source: https://docs.restate.dev/references/javadocs





# KotlinDocs
Source: https://docs.restate.dev/references/ktdocs





# Restate Server Configuration
Source: https://docs.restate.dev/references/server-config

Reference of the configuration options for Restate Server.

## Default configuration

The following is the default configuration. It does not include all possible configuration options, since some can be conflicting. Take a look at the configuration reference below for a full list of options.

Note that configuration defaults might change across server releases, if you want to make sure you use stable values, use an explicit configuration file and pass the path via `--config-path=<PATH>` as described above.

**Important changes in recent versions:**

* Restate now listens on both TCP and Unix sockets by default (`listen-mode = "all"`). Unix sockets are created under `restate-data/*.sock`.
* Advertised addresses are automatically detected based on your network configuration. You no longer need to explicitly set `advertised-address` for most deployments.
* The `metadata-client.addresses` field is now optional for single-node setups.

```toml restate.toml expandable {"CODE_LOAD::../docs/schemas/restate.toml"}  theme={null}
roles = [
    "http-ingress",
    "admin",
    "worker",
    "log-server",
    "metadata-server",
]
cluster-name = "localcluster"
auto-provision = true
default-num-partitions = 24
default-replication = 1
shutdown-timeout = "1m"
tracing-filter = "info"
log-filter = "warn,restate=info"
log-format = "pretty"
log-disable-ansi-codes = false
connect-timeout = "10s"
request-compression-threshold = "4.0 MiB"
disable-prometheus = false
rocksdb-total-memory-size = "2.0 GiB"
rocksdb-total-memtables-ratio = 0.85
rocksdb-high-priority-bg-threads = 2
rocksdb-perf-level = "enable-count"
metadata-update-interval = "10s"
metadata-fetch-from-peer-timeout = "3s"
initialization-timeout = "5m"
disable-telemetry = false
gossip-tick-interval = "100ms"
gossip-failure-threshold = 10
gossip-num-peers = 2
gossip-fd-stability-threshold = 3
gossip-suspect-interval = "5s"
gossip-loneliness-threshold = 30
gossip-extras-exchange-frequency = 10
gossip-time-skew-threshold = "1s"
hlc-max-drift = "5s"
experimental-kafka-batch-ingestion = false
experimental-shuffler-batch-ingestion = false
default-journal-retention = "1d"

[metadata-client]
type = "replicated"
addresses = []
connect-timeout = "3s"
keep-alive-interval = "5s"
keep-alive-timeout = "5s"

[metadata-client.backoff-policy]
type = "exponential"
initial-interval = "100ms"
factor = 1.4
max-attempts = 10
max-interval = "1s"

[http-keep-alive-options]
interval = "40s"
timeout = "20s"

[network-error-retry-policy]
type = "exponential"
initial-interval = "10ms"
factor = 2.0
max-attempts = 15
max-interval = "5s"

[default-retry-policy]
initial-interval = "500ms"
exponentiation-factor = 2.0
max-attempts = 70
on-max-attempts = "pause"
max-interval = "1m"

[worker]
internal-queue-length = 1000
cleanup-interval = "1h"
max-command-batch-size = 32

[worker.storage]
rocksdb-disable-wal = true
rocksdb-memory-ratio = 0.49

[worker.invoker]
inactivity-timeout = "1m"
abort-timeout = "10m"
message-size-warning = "10.0 MiB"
in-memory-queue-length-limit = 66049
concurrent-invocations-limit = 1000

[worker.snapshots]
enable-cleanup = true

[worker.snapshots.object-store-retry-policy]
type = "exponential"
initial-interval = "100ms"
factor = 2.0
max-attempts = 10
max-interval = "10s"

[worker.shuffle]
inflight-memory-budget = "10.0 MiB"
request-batch-size = "50.0 KiB"

[worker.shuffle.connection-retry-policy]
type = "exponential"
initial-interval = "10ms"
factor = 2.0
max-interval = "1s"

[admin]
heartbeat-interval = "1s 500ms"
disable-web-ui = false
disable-cluster-controller = false

[admin.query-engine]
memory-size = "1.0 GiB"

[ingress]
kafka-clusters = []

[ingress.ingestion]
inflight-memory-budget = "1.0 MiB"
request-batch-size = "50.0 KiB"

[ingress.ingestion.connection-retry-policy]
type = "exponential"
initial-interval = "10ms"
factor = 2.0
max-interval = "2s"

[bifrost]
default-provider = "replicated"
seal-retry-interval = "2s"
auto-recovery-interval = "15s"
append-retry-min-interval = "10ms"
append-retry-max-interval = "1s"
record-cache-memory-size = "250.0 MiB"
disable-auto-improvement = false

[bifrost.local]
rocksdb-disable-wal = false
rocksdb-memory-ratio = 0.5
rocksdb-disable-wal-fsync = false
writer-batch-commit-count = 5000
writer-batch-commit-duration = "0s"

[bifrost.replicated-loglet]
maximum-inflight-records = 1000
sequencer-inactivity-timeout = "15s"
log-server-rpc-timeout = "2s"
readahead-records = 20
read-batch-size = "32.0 KiB"
readahead-trigger-ratio = 0.5

[bifrost.replicated-loglet.sequencer-retry-policy]
type = "exponential"
initial-interval = "250ms"
factor = 2.0
max-interval = "5s"

[bifrost.replicated-loglet.log-server-retry-policy]
type = "exponential"
initial-interval = "250ms"
factor = 2.0
max-attempts = 3
max-interval = "2s"

[bifrost.read-retry-policy]
type = "exponential"
initial-interval = "50ms"
factor = 2.0
max-attempts = 50
max-interval = "1s"

[metadata-server]
request-queue-length = 32
rocksdb-memory-ratio = 0.01
rocksdb-disable-wal = false
raft-election-tick = 10
raft-heartbeat-tick = 2
raft-tick-interval = "100ms"
status-update-interval = "5s"
log-trim-threshold = 1000
auto-join = true

[networking]
connect-timeout = "3s"
handshake-timeout = "3s"
http2-keep-alive-interval = "1s"
http2-keep-alive-timeout = "3s"
http2-adaptive-window = true
disable-compression = false
data-stream-window-size = "2.0 MiB"

[networking.connect-retry-policy]
type = "exponential"
initial-interval = "250ms"
factor = 2.0
max-attempts = 10
max-interval = "3s"

[log-server]
rocksdb-disable-wal = false
rocksdb-memory-ratio = 0.5
rocksdb-disable-wal-fsync = false
rocksdb-max-sub-compactions = 0
writer-batch-commit-count = 5000
incoming-network-queue-length = 1000
```

## Configuration Reference

<ResponseField name="additional-request-headers" type="object | null">
  Additional request headers: Headers that should be applied to all outgoing requests (HTTP and Lambda).
  Defaults to `x-restate-cluster-name: &lt;cluster name&gt;`.

  Proxy type to implement HashMap\<HeaderName, HeaderValue> ser/de
  Use it directly or with `#[serde(with = \"serde_with::As::&lt;serde_with::FromInto&lt;restate_serde_util::SerdeableHeaderMap&gt;&gt;\")]`.
</ResponseField>

<ResponseField name="admin" type="object">
  Admin server options:

  <Expandable title="Properties">
    <ResponseField name="advertised-address" type="string | null">
      Address that other nodes will use to connect to this service.

      The full prefix that will be used to advertise this service publicly.
      For example, if this is set to `https://my-host` then others will use this
      as base URL to connect to this service.

      If unset, the advertised address will be inferred from public address of this node
      or it'll use the value supplied in `advertised-host` if set.

      advertised address: An externally accessible URI address for admin-api-server. This can be set to unix:restate-data/admin.sock to advertise the automatically created unix-socket instead of using tcp if needed

      Examples:
      "http//127.0.0.1:9070/" or "[https://my-host/](https://my-host/)" or "unix:/data/restate-data/admin.sock"
    </ResponseField>

    <ResponseField name="advertised-admin-endpoint" type="string | null">
      Advertised Admin endpoint: Optional advertised Admin API endpoint.
      \[Deprecated] Use `advertised-address` instead.

      advertised address: An externally accessible URI address for admin-api-server. This can be set to unix:restate-data/admin.sock to advertise the automatically created unix-socket instead of using tcp if needed

      Examples:
      "http//127.0.0.1:9070/" or "[https://my-host/](https://my-host/)" or "unix:/data/restate-data/admin.sock"
    </ResponseField>

    <ResponseField name="advertised-host" type="string | null">
      Hostname to advertise for this service
    </ResponseField>

    <ResponseField name="bind-address" type="string | null">
      The combination of `bind-ip` and `bind-port` that will be used to bind

      This has precedence over `bind-ip` and `bind-port`

      Bind address: The local network address to bind on for admin-api-server. This service uses default port 9070 and will create a unix-socket file at the data directory under the name `admin.sock`

      Examples:
      "0.0.0.0:9070" or "127.0.0.1:9070"
    </ResponseField>

    <ResponseField name="bind-ip" type="string | null">
      Local interface IP address to listen on
    </ResponseField>

    <ResponseField name="bind-port" type="integer | null">
      Network port to listen on
    </ResponseField>

    <ResponseField name="concurrent-api-requests-limit" type="integer | null">
      Concurrency limit for the Admin APIs. Default is unlimited.
    </ResponseField>

    <ResponseField name="deployment-routing-headers" type="array">
      Deployment routing headers: List of header names considered routing headers.

      These will be used during deployment creation to distinguish between an already existing deployment and a new deployment.

      <Expandable title="Array Items">
        <ResponseField name="item" type="string" />
      </Expandable>
    </ResponseField>

    <ResponseField name="disable-cluster-controller" type="boolean" />

    <ResponseField name="disable-web-ui" type="boolean">
      Disable serving the Restate Web UI on the admin port. Default is `false`.
    </ResponseField>

    <ResponseField name="heartbeat-interval" type="string">
      Controller heartbeats: Controls the interval at which cluster controller polls nodes of the cluster.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
    </ResponseField>

    <ResponseField name="listen-mode" type="oneOf | null">
      Listen on unix-sockets, TCP sockets, or both.

      The default is to listen on both.

      * `"unix"` : Exclusively listen on unix domain sockets

      If set, all services will listen exclusively on unix sockets, each service
      will create a socket file under the data directory.

      * `"tcp"` : Exclusively listen on TCP sockets
      * `"all"` : \[default] Listen on both Unix and TCP sockets
    </ResponseField>

    <ResponseField name="query-engine" type="object">
      Storage query engine options:

      <Expandable title="Properties">
        <ResponseField name="memory-size" type="string">
          Memory size limit: The total memory in bytes that can be used to preform sql queries
        </ResponseField>

        <ResponseField name="query-parallelism" type="integer | null">
          Default query parallelism: The degree of parallelism to use for query execution (Defaults to the number of available cores).
        </ResponseField>

        <ResponseField name="tmp-dir" type="string | null">
          Temp folder to use for spill: The path to spill to
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="use-random-ports" type="boolean | null">
      Use random ports instead of the default port
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="advertised-address" type="string | null">
  Address that other nodes will use to connect to this service.

  The full prefix that will be used to advertise this service publicly.
  For example, if this is set to `https://my-host` then others will use this
  as base URL to connect to this service.

  If unset, the advertised address will be inferred from public address of this node
  or it'll use the value supplied in `advertised-host` if set.

  advertised address: An externally accessible URI address for tokio-console-server. This can be set to unix:restate-data/tokio.sock to advertise the automatically created unix-socket instead of using tcp if needed

  Examples:
  "http//127.0.0.1:6669/" or "[https://my-host/](https://my-host/)" or "unix:/data/restate-data/tokio.sock"
</ResponseField>

<ResponseField name="advertised-host" type="string | null">
  Hostname to advertise for this service
</ResponseField>

<ResponseField name="auto-provision" type="boolean">
  Auto cluster provisioning: If true, then this node is allowed to automatically provision as a new cluster.
  This node *must* have an admin role and a new nodes configuration will be created that includes this node.

  auto-provision is allowed by default in development mode and is disabled if restate-server runs with `--production` flag
  to prevent cluster nodes from forming their own clusters, rather than forming a single cluster.

  Use `restatectl` to provision the cluster/node if automatic provisioning is disabled.

  This can also be explicitly disabled by setting this value to false.

  Default: true
</ResponseField>

<ResponseField name="aws-assume-role-external-id" type="string | null">
  AssumeRole external ID: An external ID to apply to any AssumeRole operations taken by this client.
  [https://docs.aws.amazon.com/IAM/latest/UserGuide/id\_roles\_create\_for-user\_externalid.html](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html)
  Can be overridden by the `AWS_EXTERNAL_ID` environment variable.
</ResponseField>

<ResponseField name="aws-profile" type="string | null">
  AWS Profile: Name of the AWS profile to select. Defaults to 'AWS\_PROFILE' env var, or otherwise
  the `default` profile.
</ResponseField>

<ResponseField name="base-dir" type="string | null">
  The working directory which this Restate node should use for relative paths. The default is
  `restate-data` under the current working directory.
</ResponseField>

<ResponseField name="bifrost" type="object">
  Bifrost options:

  <Expandable title="Properties">
    <ResponseField name="append-retry-max-interval" type="string">
      Append retry maximum interval: Maximum retry duration used by the exponential backoff mechanism for bifrost appends.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
    </ResponseField>

    <ResponseField name="append-retry-min-interval" type="string">
      Append retry minimum interval: Minimum retry duration used by the exponential backoff mechanism for bifrost appends.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
    </ResponseField>

    <ResponseField name="auto-recovery-interval" type="string">
      Auto recovery threshold: Time interval after which bifrost's auto-recovery mechanism will kick in. This
      is triggered in scenarios where the control plane took too long to complete loglet
      reconfigurations.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
    </ResponseField>

    <ResponseField name="default-provider">
      The default kind of loglet to be used: Default: Replicated

      * `"local"` : A local rocksdb-backed loglet.
      * `"replicated"` : Replicated loglets are restate's native log replication system. This requires
        `log-server` role to run on enough nodes in the cluster.
    </ResponseField>

    <ResponseField name="disable-auto-improvement" type="string">
      Disable Automatic Improvement: When enabled, automatic improvement periodically checks with the loglet provider
      if the loglet configuration can be improved by performing a reconfiguration.

      This allows the log to pick up replication property changes, apply better placement
      of replicas, or for other reasons.
    </ResponseField>

    <ResponseField name="local" type="string">
      Configuration of local loglet provider
    </ResponseField>

    <ResponseField name="read-retry-policy">
      Read retry policy: Retry policy to use when bifrost waits for reconfiguration to complete during
      read operations

      <Expandable title="Option 1: None">
        No retry strategy.

        <ResponseField name="type" type="string" />
      </Expandable>

      <Expandable title="Option 2: Fixed delay">
        Retry with a fixed delay strategy.

        <ResponseField name="interval" type="string">
          Interval between retries.

          Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

          Examples:
          "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
        </ResponseField>

        <ResponseField name="max-attempts" type="integer | null">
          Max attempts: Number of maximum attempts before giving up. Infinite retries if unset.
        </ResponseField>

        <ResponseField name="type" type="string" />
      </Expandable>

      <Expandable title="Option 3: Exponential">
        Retry with an exponential strategy. The next retry is computed as `min(last_retry_interval * factor, max_interval)`.

        <ResponseField name="factor" type="number">
          Factor: The factor to use to compute the next retry attempt.
        </ResponseField>

        <ResponseField name="initial-interval" type="string">
          Initial Interval: Initial interval for the first retry attempt.

          Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

          Examples:
          "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
        </ResponseField>

        <ResponseField name="max-attempts" type="integer | null">
          Max attempts: Number of maximum attempts before giving up. Infinite retries if unset.
        </ResponseField>

        <ResponseField name="max-interval" type="string | null">
          Max interval: Maximum interval between retries.

          Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

          Human-readable duration: Duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

          Examples:
          "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
        </ResponseField>

        <ResponseField name="type" type="string" />
      </Expandable>
    </ResponseField>

    <ResponseField name="record-cache-memory-size" type="string">
      In-memory RecordCache memory limit: Optional size of record cache in bytes.
      If set to 0, record cache will be disabled.
      Defaults: 250MB
    </ResponseField>

    <ResponseField name="replicated-loglet" type="object">
      Configuration of replicated loglet provider

      <Expandable title="Properties">
        <ResponseField name="log-server-retry-policy">
          Log Server RPC retry policy

          Retry policy for log server RPCs

          <Expandable title="Option 1: None">
            No retry strategy.

            <ResponseField name="type" type="string" />
          </Expandable>

          <Expandable title="Option 2: Fixed delay">
            Retry with a fixed delay strategy.

            <ResponseField name="interval" type="string">
              Interval between retries.

              Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

              Examples:
              "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
            </ResponseField>

            <ResponseField name="max-attempts" type="integer | null">
              Max attempts: Number of maximum attempts before giving up. Infinite retries if unset.
            </ResponseField>

            <ResponseField name="type" type="string" />
          </Expandable>

          <Expandable title="Option 3: Exponential">
            Retry with an exponential strategy. The next retry is computed as `min(last_retry_interval * factor, max_interval)`.

            <ResponseField name="factor" type="number">
              Factor: The factor to use to compute the next retry attempt.
            </ResponseField>

            <ResponseField name="initial-interval" type="string">
              Initial Interval: Initial interval for the first retry attempt.

              Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

              Examples:
              "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
            </ResponseField>

            <ResponseField name="max-attempts" type="integer | null">
              Max attempts: Number of maximum attempts before giving up. Infinite retries if unset.
            </ResponseField>

            <ResponseField name="max-interval" type="string | null">
              Max interval: Maximum interval between retries.

              Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

              Human-readable duration: Duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

              Examples:
              "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
            </ResponseField>

            <ResponseField name="type" type="string" />
          </Expandable>
        </ResponseField>

        <ResponseField name="log-server-rpc-timeout" type="string">
          Non-zero human-readable duration: Log Server RPC timeout

          Timeout waiting on log server response

          Examples:
          "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
        </ResponseField>

        <ResponseField name="maximum-inflight-records" type="integer">
          Maximum number of inflight records sequencer can accept

          Once this maximum is hit, sequencer will induce back pressure
          on clients. This controls the total number of records regardless of how many batches.

          Note that this will be increased to fit the biggest batch of records being enqueued.
        </ResponseField>

        <ResponseField name="read-batch-size" type="string">
          Limits memory per remote batch read: When reading from a log-server, the server stops reading if the next record will tip over the
          total number of bytes allowed in this configuration option.

          Note the limit is not strict and the server will always allow at least a single record to be
          read even if that record exceeds the stated budget.
        </ResponseField>

        <ResponseField name="readahead-records" type="integer">
          Maximum number of records to prefetch from log servers

          The number of records bifrost will attempt to prefetch from replicated loglet's log-servers
          for every loglet reader (e.g. partition processor). Note that this mainly impacts readers
          that are not co-located with the loglet sequencer (i.e. partition processor followers).
        </ResponseField>

        <ResponseField name="readahead-trigger-ratio" type="number">
          Trigger to prefetch more records

          When read-ahead is used (readahead-records), this value (percentage in float) will determine when
          readers should trigger a prefetch for another batch to fill up the buffer. For instance, if
          this value is 0.3, then bifrost will trigger a prefetch when 30% or more of the read-ahead
          slots become available (e.g. partition processor consumed records and freed up enough slots).

          The higher the value is, the longer bifrost will wait before it triggers the next fetch, potentially
          fetching more records as a result.

          To illustrate, if readahead-records is set to 100 and readahead-trigger-ratio is 1.0. Then
          bifrost will prefetch up to 100 records from log-servers and will not trigger the next
          prefetch unless the consumer consumes 100% of this buffer. This means that bifrost will
          read in batches but will not do while the consumer is still reading the previous batch.

          Value must be between 0 and 1. It will be clamped at `1.0`.
        </ResponseField>

        <ResponseField name="sequencer-inactivity-timeout" type="string">
          Non-zero human-readable duration: Sequencer inactivity timeout

          The sequencer is allowed to consider itself quiescent if it did not commit records for this period of time.
          It may use this to sends pre-emptive release/seal check requests to log-servers.

          The sequencer is also allowed to use this value as interval to send seal/release checks even if it's not quiescent.

          Examples:
          "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
        </ResponseField>

        <ResponseField name="sequencer-retry-policy">
          Retry policy: Sequencer retry policy

          Backoff introduced when sequencer fail to find a suitable spread of log servers

          <Expandable title="Option 1: None">
            No retry strategy.

            <ResponseField name="type" type="string" />
          </Expandable>

          <Expandable title="Option 2: Fixed delay">
            Retry with a fixed delay strategy.

            <ResponseField name="interval" type="string">
              Interval between retries.

              Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

              Examples:
              "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
            </ResponseField>

            <ResponseField name="max-attempts" type="integer | null">
              Max attempts: Number of maximum attempts before giving up. Infinite retries if unset.
            </ResponseField>

            <ResponseField name="type" type="string" />
          </Expandable>

          <Expandable title="Option 3: Exponential">
            Retry with an exponential strategy. The next retry is computed as `min(last_retry_interval * factor, max_interval)`.

            <ResponseField name="factor" type="number">
              Factor: The factor to use to compute the next retry attempt.
            </ResponseField>

            <ResponseField name="initial-interval" type="string">
              Initial Interval: Initial interval for the first retry attempt.

              Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

              Examples:
              "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
            </ResponseField>

            <ResponseField name="max-attempts" type="integer | null">
              Max attempts: Number of maximum attempts before giving up. Infinite retries if unset.
            </ResponseField>

            <ResponseField name="max-interval" type="string | null">
              Max interval: Maximum interval between retries.

              Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

              Human-readable duration: Duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

              Examples:
              "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
            </ResponseField>

            <ResponseField name="type" type="string" />
          </Expandable>
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="seal-retry-interval" type="string">
      Seal retry interval: Interval to wait between retries of loglet seal failures

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="bind-address" type="string | null">
  The combination of `bind-ip` and `bind-port` that will be used to bind

  This has precedence over `bind-ip` and `bind-port`

  Bind address: The local network address to bind on for tokio-console-server. This service uses default port 6669 and will create a unix-socket file at the data directory under the name `tokio.sock`

  Examples:
  "0.0.0.0:6669" or "127.0.0.1:6669"
</ResponseField>

<ResponseField name="bind-ip" type="string | null">
  Local interface IP address to listen on
</ResponseField>

<ResponseField name="bind-port" type="integer | null">
  Network port to listen on
</ResponseField>

<ResponseField name="cluster-name" type="string">
  Cluster name: A unique identifier for the cluster. All nodes in the same cluster should
  have the same.
</ResponseField>

<ResponseField name="connect-timeout" type="string">
  Connect timeout: How long to wait for a TCP connection to be established before considering
  it a failed attempt.

  Examples:
  "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
</ResponseField>

<ResponseField name="default-journal-retention" type="string">
  Human-readable duration: Duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

  Examples:
  "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
</ResponseField>

<ResponseField name="default-num-partitions" type="integer">
  Partitions: Number of partitions that will be provisioned during initial cluster provisioning.
  partitions are the logical shards used to process messages.

  Cannot be higher than `65535` (You should almost never need as many partitions anyway)

  NOTE 1: This config entry only impacts the initial number of partitions, the
  value of this entry is ignored for provisioned nodes/clusters.

  NOTE 2: This will be renamed to `default-num-partitions` by default as of v1.3+

  Default: 24
</ResponseField>

<ResponseField name="default-replication" type="string">
  Default replication factor: Configures the global default replication factor to be used by the the system.

  Note that this value only impacts the cluster initial provisioning and will not be respected after
  the cluster has been provisioned.

  To update existing clusters use the `restatectl` utility.
</ResponseField>

<ResponseField name="default-retry-policy" type="object">
  Default retry policy: The default retry policy to use for invocations.

  The retry policy can be customized on a service/handler basis, using the respective SDK APIs.
  Check [https://docs.restate.dev/services/configuration#retries](https://docs.restate.dev/services/configuration#retries) for more details.

  <Expandable title="Properties">
    <ResponseField name="exponentiation-factor" type="number">
      Factor: The factor to use to compute the next retry attempt. Default: `2.0`.
    </ResponseField>

    <ResponseField name="initial-interval" type="string">
      Initial Interval: Initial interval for the first retry attempt.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
    </ResponseField>

    <ResponseField name="max-attempts" type="anyOf">
      Max attempts: Number of maximum attempts (including the initial) before giving up.
      No retries if set to 1.

      * `"unlimited"` : Unlimited retries.
      * `Option 2: Bounded number of retries.` : Bounded number of retries.
    </ResponseField>

    <ResponseField name="max-interval" type="string">
      Max interval: Maximum interval between retries.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
    </ResponseField>

    <ResponseField name="on-max-attempts">
      On max attempts: Behavior when max attempts are reached.

      Set to `pause` to pause invocations when max attempts are reached.
      Set to `kill` to kill the invocation when max attempts are reached.

      For more details about the invocation lifecycle, check [https://docs.restate.dev/services/invocation/managing-invocations](https://docs.restate.dev/services/invocation/managing-invocations)

      * `"pause"` : Pause the invocation when max attempts are reached.
      * `"kill"` : Kill the invocation when max attempts are reached.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="default-thread-pool-size" type="integer | null">
  Default async runtime thread pool: Size of the default thread pool used to perform internal tasks.
  If not set, it defaults to the number of CPU cores.
</ResponseField>

<ResponseField name="disable-prometheus" type="boolean">
  Disable prometheus metric recording and reporting. Default is `false`.
</ResponseField>

<ResponseField name="disable-telemetry" type="boolean">
  Disable telemetry: Restate uses Scarf to collect anonymous usage data to help us understand how the software is being used.
  You can set this flag to true to disable this collection. It can also be set with the environment variable DO\_NOT\_TRACK=1.
</ResponseField>

<ResponseField name="experimental-kafka-batch-ingestion" type="boolean">
  Experimental Kafka batch ingestion: Use the new experimental kafka ingestion path which leverages batching
  for a faster kafka ingestion.

  Set to `true` to enable the experimental ingestion mechanism.

  The legacy path will be removed in v1.7.

  Defaults to `false` in v1.6.
</ResponseField>

<ResponseField name="experimental-shuffler-batch-ingestion" type="boolean">
  Experimental Shuffler batch ingestion: Use the new experimental batch ingestion path.

  Set to `true` to enable the experimental ingestion mechanism.

  The legacy path will be removed in v1.7.

  Defaults to `false` in v1.6.
</ResponseField>

<ResponseField name="force-node-id" type="integer | null">
  If set, the node insists on acquiring this node ID.
</ResponseField>

<ResponseField name="gossip-extras-exchange-frequency" type="integer">
  Gossip extras exchange frequency: In addition to basic health/liveness information, the gossip protocol is used to exchange
  extra information about the roles hosted by this node. For instance, which partitions are
  currently running, their configuration versions, and the durable LSN of the corresponding
  partition databases. This information is sent every Nth gossip message. This setting
  controls the frequency of this exchange. For instance, `10` means that every 10th gossip
  message will contain the extra information about.
</ResponseField>

<ResponseField name="gossip-failure-threshold" type="integer">
  Gossip failure threshold: Specifies how many gossip intervals of inactivity need to pass before
  considering a node as dead.
</ResponseField>

<ResponseField name="gossip-fd-stability-threshold" type="integer">
  Gossips before failure detector is stable:
</ResponseField>

<ResponseField name="gossip-loneliness-threshold" type="integer">
  Gossip loneliness threshold: How many intervals need to pass without receiving any gossip messages before considering
  this node as potentially isolated/dead. This threshold is used in the case where the node
  can still send gossip messages but did not receive any. This can rarely happen in
  asymmetric network partitions.

  In this case, the node will advertise itself as dead in the gossip messages it sends out.

  Note: this threshold does not apply to a cluster that's configured with a single node.
</ResponseField>

<ResponseField name="gossip-num-peers" type="integer">
  Number of peers to gossip: On every gossip interval, how many peers each node attempts to gossip with. The default is
  optimized for small clusters (less than 5 nodes). On larger clusters, if gossip overhead is noticeable,
  consider reducing this value to 1.
</ResponseField>

<ResponseField name="gossip-suspect-interval" type="string">
  Human-readable duration: Duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

  Examples:
  "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
</ResponseField>

<ResponseField name="gossip-tick-interval" type="string">
  Gossip tick interval: The interval at which the failure detector will tick. Decrease this value for faster reaction
  to node failures. Note, that every tick comes with an overhead.

  Examples:
  "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
</ResponseField>

<ResponseField name="gossip-time-skew-threshold" type="string">
  Gossips time skew threshold: The time skew is the maximum acceptable time difference between the local node and the time
  reported by peers via gossip messages. The time skew is also used to ignore gossip messages
  that are too old.

  Examples:
  "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
</ResponseField>

<ResponseField name="hlc-max-drift" type="string">
  Human-readable duration: Duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

  Examples:
  "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
</ResponseField>

<ResponseField name="http-keep-alive-options" type="object">
  HTTP/2 Keep-alive: Configuration for the HTTP/2 keep-alive mechanism, using PING frames.
  If unset, HTTP/2 keep-alive are disabled.

  <Expandable title="Properties">
    <ResponseField name="interval" type="string">
      HTTP/2 Keep-alive interval: Sets an interval for HTTP/2 PING frames should be sent to keep a
      connection alive.

      You should set this timeout with a value lower than the `abort_timeout`.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
    </ResponseField>

    <ResponseField name="timeout" type="string">
      Timeout: Sets a timeout for receiving an acknowledgement of the keep-alive ping.

      If the ping is not acknowledged within the timeout, the connection will
      be closed.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="http-proxy" type="string | null">
  Proxy URI: A URI, such as `http://127.0.0.1:10001`, of a server to which all invocations should be sent, with the `Host` header set to the deployment URI.
  HTTPS proxy URIs are supported, but only HTTP endpoint traffic will be proxied currently.
  Can be overridden by the `HTTP_PROXY` environment variable.
</ResponseField>

<ResponseField name="ingress" type="object">
  Ingress options:

  <Expandable title="Properties">
    <ResponseField name="advertised-address" type="string | null">
      Address that other nodes will use to connect to this service.

      The full prefix that will be used to advertise this service publicly.
      For example, if this is set to `https://my-host` then others will use this
      as base URL to connect to this service.

      If unset, the advertised address will be inferred from public address of this node
      or it'll use the value supplied in `advertised-host` if set.

      advertised address: An externally accessible URI address for http-ingress-server. This can be set to unix:restate-data/ingress.sock to advertise the automatically created unix-socket instead of using tcp if needed

      Examples:
      "http//127.0.0.1:8080/" or "[https://my-host/](https://my-host/)" or "unix:/data/restate-data/ingress.sock"
    </ResponseField>

    <ResponseField name="advertised-host" type="string | null">
      Hostname to advertise for this service
    </ResponseField>

    <ResponseField name="advertised-ingress-endpoint" type="string | null">
      \[Deprecated] Use `advertised-address` instead.
      Ingress endpoint that the Web UI should use to interact with.

      advertised address: An externally accessible URI address for http-ingress-server. This can be set to unix:restate-data/ingress.sock to advertise the automatically created unix-socket instead of using tcp if needed

      Examples:
      "http//127.0.0.1:8080/" or "[https://my-host/](https://my-host/)" or "unix:/data/restate-data/ingress.sock"
    </ResponseField>

    <ResponseField name="bind-address" type="string | null">
      The combination of `bind-ip` and `bind-port` that will be used to bind

      This has precedence over `bind-ip` and `bind-port`

      Bind address: The local network address to bind on for http-ingress-server. This service uses default port 8080 and will create a unix-socket file at the data directory under the name `ingress.sock`

      Examples:
      "0.0.0.0:8080" or "127.0.0.1:8080"
    </ResponseField>

    <ResponseField name="bind-ip" type="string | null">
      Local interface IP address to listen on
    </ResponseField>

    <ResponseField name="bind-port" type="integer | null">
      Network port to listen on
    </ResponseField>

    <ResponseField name="concurrent-api-requests-limit" type="integer | null">
      Concurrency limit: Local concurrency limit to use to limit the amount of concurrent requests. If exceeded,
      the ingress will reply immediately with an appropriate status code. Default is unlimited.
    </ResponseField>

    <ResponseField name="ingestion" type="object">
      Ingestion Options: Settings for the ingestion client
      Currently only used by the Kafka ingress and the admin API.

      <Expandable title="Properties">
        <ResponseField name="connection-retry-policy">
          Connection retry policy: Retry policy for the ingestion client. It must allow unlimited
          retries; if configured with a cap, the client falls back to
          retrying every 2 seconds.

          <Expandable title="Option 1: None">
            No retry strategy.

            <ResponseField name="type" type="string" />
          </Expandable>

          <Expandable title="Option 2: Fixed delay">
            Retry with a fixed delay strategy.

            <ResponseField name="interval" type="string">
              Interval between retries.

              Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

              Examples:
              "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
            </ResponseField>

            <ResponseField name="max-attempts" type="integer | null">
              Max attempts: Number of maximum attempts before giving up. Infinite retries if unset.
            </ResponseField>

            <ResponseField name="type" type="string" />
          </Expandable>

          <Expandable title="Option 3: Exponential">
            Retry with an exponential strategy. The next retry is computed as `min(last_retry_interval * factor, max_interval)`.

            <ResponseField name="factor" type="number">
              Factor: The factor to use to compute the next retry attempt.
            </ResponseField>

            <ResponseField name="initial-interval" type="string">
              Initial Interval: Initial interval for the first retry attempt.

              Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

              Examples:
              "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
            </ResponseField>

            <ResponseField name="max-attempts" type="integer | null">
              Max attempts: Number of maximum attempts before giving up. Infinite retries if unset.
            </ResponseField>

            <ResponseField name="max-interval" type="string | null">
              Max interval: Maximum interval between retries.

              Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

              Human-readable duration: Duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

              Examples:
              "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
            </ResponseField>

            <ResponseField name="type" type="string" />
          </Expandable>
        </ResponseField>

        <ResponseField name="inflight-memory-budget" type="string">
          Inflight Memory Budget: Maximum total size of in-flight ingestion requests in bytes.
          Tune this to your workload so there are enough unpersisted
          requests for efficient batching without exhausting memory.

          Defaults to 1 MiB.
        </ResponseField>

        <ResponseField name="request-batch-size" type="string">
          Request Batch Size: Maximum size of a single ingestion request batch.
          Tune to keep enough requests per batch for
          throughput; overly large batches can increase tail latency.

          Defaults to 50 KiB.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="kafka-clusters" type="array">
      <Expandable title="Array Items">
        <ResponseField name="item" type="object">
          Kafka cluster options: Configuration options to connect to a Kafka cluster.

          <Expandable title="Properties">
            <ResponseField name="brokers" type="array">
              Servers: Initial list of brokers (host or host:port).

              <Expandable title="Array Items">
                <ResponseField name="item" type="string" />
              </Expandable>
            </ResponseField>

            <ResponseField name="name" type="string">
              Cluster name (Used to identify subscriptions).
            </ResponseField>
          </Expandable>
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="listen-mode" type="oneOf | null">
      Listen on unix-sockets, TCP sockets, or both.

      The default is to listen on both.

      * `"unix"` : Exclusively listen on unix domain sockets

      If set, all services will listen exclusively on unix sockets, each service
      will create a socket file under the data directory.

      * `"tcp"` : Exclusively listen on TCP sockets
      * `"all"` : \[default] Listen on both Unix and TCP sockets
    </ResponseField>

    <ResponseField name="use-random-ports" type="boolean | null">
      Use random ports instead of the default port
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="initial-max-send-streams" type="integer | null">
  Initial Max Send Streams: Sets the initial maximum of locally initiated (send) streams.

  This value will be overwritten by the value included in the initial
  SETTINGS frame received from the peer as part of a \[connection preface].

  Default: None

  **NOTE**: Setting this value to None (default) users the default
  recommended value from HTTP2 specs
</ResponseField>

<ResponseField name="initialization-timeout" type="string">
  Initialization timeout: The timeout until the node gives up joining a cluster and initializing itself.

  Examples:
  "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
</ResponseField>

<ResponseField name="listen-mode" type="oneOf | null">
  Listen on unix-sockets, TCP sockets, or both.

  The default is to listen on both.

  * `"unix"` : Exclusively listen on unix domain sockets

  If set, all services will listen exclusively on unix sockets, each service
  will create a socket file under the data directory.

  * `"tcp"` : Exclusively listen on TCP sockets
  * `"all"` : \[default] Listen on both Unix and TCP sockets
</ResponseField>

<ResponseField name="location" type="string">
  Node Location: Setting the location allows Restate to form a tree-like cluster topology.
  The value is written in the format of "region\[.zone]" to assign this node
  to a specific region, or to a zone within a region.

  The value of region and zone is arbitrary but whitespace and `.` are disallowed.

  NOTE: It's *strongly* recommended to not change the node's location string after
  its initial registration. Changing the location may result in data loss or data
  inconsistency if `log-server` is enabled on this node.

  When this value is not set, the node is considered to be in the *default* location.
  The *default* location means that the node is not assigned to any specific region or zone.

  Examples

  * `us-west` -- the node is in the `us-west` region.
  * `us-west.a1` -- the node is in the `us-west` region and in the `a1` zone.
  * \`\` -- \[default] the node is in the default location
</ResponseField>

<ResponseField name="log-disable-ansi-codes" type="boolean">
  Disable ANSI in log output: Disable ANSI terminal codes for logs. This is useful when the log collector doesn't support processing ANSI terminal codes.
</ResponseField>

<ResponseField name="log-filter" type="string">
  Logging Filter: Log filter configuration. Can be overridden by the `RUST_LOG` environment variable.
  Check the [`RUST_LOG` documentation](https://docs.rs/tracing-subscriber/latest/tracing_subscriber/filter/struct.EnvFilter.html) for more details how to configure it.
</ResponseField>

<ResponseField name="log-format">
  Logging format: Format to use when logging.

  * `Option 1: Pretty` : Enables verbose logging. Not recommended in production.
  * `Option 2: Compact` : Enables compact logging.
  * `Option 3: Json` : Enables json logging. You can use a json log collector to ingest these logs and further process them.
</ResponseField>

<ResponseField name="log-server" type="object">
  Log server options: Configuration is only used on nodes running with `log-server` role.

  <Expandable title="Properties">
    <ResponseField name="incoming-network-queue-length" type="integer">
      The number of messages that can queue up on input network stream while request processor is busy.
    </ResponseField>

    <ResponseField name="rocksdb-block-size" type="string | null">
      RocksDB block size: Uncompressed block size

      Default: 64KiB

      Non-zero human-readable bytes
    </ResponseField>

    <ResponseField name="rocksdb-compaction-readahead-size" type="string | null">
      RocksDB compaction readahead size in bytes: If non-zero, we perform bigger reads when doing compaction. If you're
      running RocksDB on spinning disks, you should set this to at least 2MB.
      That way RocksDB's compaction is doing sequential instead of random reads.

      Non-zero human-readable bytes
    </ResponseField>

    <ResponseField name="rocksdb-disable-direct-io-for-flush-and-compactions" type="boolean | null">
      Disable Direct IO for flush and compactions: Use O\_DIRECT for writes in background flush and compactions.
    </ResponseField>

    <ResponseField name="rocksdb-disable-direct-io-for-reads" type="boolean | null">
      Disable Direct IO for reads: Files will be opened in "direct I/O" mode
      which means that data r/w from the disk will not be cached or
      buffered. The hardware buffer of the devices may however still
      be used. Memory mapped files are not impacted by these parameters.
    </ResponseField>

    <ResponseField name="rocksdb-disable-statistics" type="boolean | null">
      Disable rocksdb statistics collection

      Default: False (statistics enabled)
    </ResponseField>

    <ResponseField name="rocksdb-disable-wal" type="boolean | null">
      Disable WAL: The default depends on the different rocksdb use-cases at Restate.

      Supports hot-reloading (Partial / Bifrost only)
    </ResponseField>

    <ResponseField name="rocksdb-disable-wal-fsync" type="boolean">
      Disable fsync of WAL on every batch
    </ResponseField>

    <ResponseField name="rocksdb-log-keep-file-num" type="integer | null">
      RocksDB log keep file num: Number of info LOG files to keep

      Default: 1
    </ResponseField>

    <ResponseField name="rocksdb-log-level" type="string | null">
      RocksDB log level: Verbosity of the LOG.

      Default: "error"

      Verbosity of the LOG.
    </ResponseField>

    <ResponseField name="rocksdb-log-max-file-size" type="string | null">
      RocksDB log max file size: Max size of info LOG file

      Default: 64MB

      Non-zero human-readable bytes
    </ResponseField>

    <ResponseField name="rocksdb-max-background-jobs" type="integer | null">
      RocksDB max background jobs (flushes and compactions): Default: the number of CPU cores on this node.
    </ResponseField>

    <ResponseField name="rocksdb-max-sub-compactions" type="integer">
      The maximum number of subcompactions to run in parallel.

      Setting this to 1 means no sub-compactions are allowed (i.e. only 1 thread will do the compaction).

      Default is 0 which maps to floor(number of CPU cores / 2)
    </ResponseField>

    <ResponseField name="rocksdb-max-wal-size" type="string">
      Human-readable bytes: The size limit of all WAL files

      Use this to limit the size of WAL files. If the size of all WAL files exceeds this limit,
      the oldest WAL file will be deleted and if needed, memtable flush will be triggered.

      Note: RocksDB internally counts the uncompressed bytes to determine the WAL size, and since the WAL
      is compressed, the actual size on disk will be significantly smaller than this value (\~1/4
      depending on the compression ratio). For instance, if this is set to "1 MiB", then rocksdb
      might decide to flush if the total WAL (on disk) reached \~260 KiB (compressed).

      Default is `0` which translates into 6 times the memory allocated for membtables for this
      database.
    </ResponseField>

    <ResponseField name="rocksdb-memory-budget" type="string | null">
      The memory budget for rocksdb memtables in bytes

      If this value is set, it overrides the ratio defined in `rocksdb-memory-ratio`.

      Non-zero human-readable bytes
    </ResponseField>

    <ResponseField name="rocksdb-memory-ratio" type="number">
      The memory budget for rocksdb memtables as ratio

      This defines the total memory for rocksdb as a ratio of all memory available to the
      log-server.

      (See `rocksdb-total-memtables-ratio` in common).
    </ResponseField>

    <ResponseField name="rocksdb-statistics-level" type="oneOf | null">
      RocksDB statistics level: StatsLevel can be used to reduce statistics overhead by skipping certain
      types of stats in the stats collection process.

      Default: "except-detailed-timers"

      * `"disable-all"` : Disable all metrics
      * `"except-histogram-or-timers"` : Disable timer stats, and skip histogram stats
      * `"except-timers"` : Skip timer stats
      * `"except-detailed-timers"` : Collect all stats except time inside mutex lock AND time spent on
        compression.
      * `"except-time-for-mutex"` : Collect all stats except the counters requiring to get time inside the
        mutex lock.
      * `"all"` : Collect all stats, including measuring duration of mutex operations.
        If getting time is expensive on the platform to run, it can
        reduce scalability to more threads, especially for writes.
    </ResponseField>

    <ResponseField name="writer-batch-commit-count" type="integer">
      Trigger a commit when the batch size exceeds this threshold.

      Set to 0 or 1 to commit the write batch on every command.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="max-journal-retention" type="string | null">
  Maximum journal retention duration that can be configured.
  When discovering a service deployment, or when modifying the journal retention using the Admin API, the given value will be clamped.

  Unset means no limit.

  Human-readable duration: Duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

  Examples:
  "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
</ResponseField>

<ResponseField name="max-retry-policy-max-attempts" type="integer | null">
  Max configurable value for retry policy max attempts: Maximum max attempts configurable in an invocation retry policy.
  When discovering a service deployment with configured retry policies, or when modifying the invocation retry policy using the Admin API, the given value will be clamped.

  `None` means no limit, that is infinite retries is enabled.
</ResponseField>

<ResponseField name="metadata-client" type="object">
  Metadata client options: The metadata client type to store metadata

  <Expandable title="Option 1: Store metadata on the replicated metadata store that runs on nodes with the metadata-server role.">
    <ResponseField name="addresses" type="array">
      Restate metadata server address list:

      <Expandable title="Array Items">
        <ResponseField name="item" type="string" />
      </Expandable>
    </ResponseField>

    <ResponseField name="type" type="string" />

    <ResponseField name="backoff-policy">
      Backoff policy used by the metadata client when it encounters concurrent modifications.

      <Expandable title="Option 1: None">
        No retry strategy.

        <ResponseField name="type" type="string" />
      </Expandable>

      <Expandable title="Option 2: Fixed delay">
        Retry with a fixed delay strategy.

        <ResponseField name="interval" type="string">
          Interval between retries.

          Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

          Examples:
          "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
        </ResponseField>

        <ResponseField name="max-attempts" type="integer | null">
          Max attempts: Number of maximum attempts before giving up. Infinite retries if unset.
        </ResponseField>

        <ResponseField name="type" type="string" />
      </Expandable>

      <Expandable title="Option 3: Exponential">
        Retry with an exponential strategy. The next retry is computed as `min(last_retry_interval * factor, max_interval)`.

        <ResponseField name="factor" type="number">
          Factor: The factor to use to compute the next retry attempt.
        </ResponseField>

        <ResponseField name="initial-interval" type="string">
          Initial Interval: Initial interval for the first retry attempt.

          Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

          Examples:
          "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
        </ResponseField>

        <ResponseField name="max-attempts" type="integer | null">
          Max attempts: Number of maximum attempts before giving up. Infinite retries if unset.
        </ResponseField>

        <ResponseField name="max-interval" type="string | null">
          Max interval: Maximum interval between retries.

          Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

          Human-readable duration: Duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

          Examples:
          "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
        </ResponseField>

        <ResponseField name="type" type="string" />
      </Expandable>
    </ResponseField>

    <ResponseField name="connect-timeout" type="string">
      Connect timeout: TCP connection timeout for connecting to the metadata store.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
    </ResponseField>

    <ResponseField name="keep-alive-interval" type="string">
      Metadata Store Keep Alive Interval: Non-zero duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
    </ResponseField>

    <ResponseField name="keep-alive-timeout" type="string">
      Metadata Store Keep Alive Timeout: Non-zero duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
    </ResponseField>

    <ResponseField name="message-size-limit" type="string | null">
      Metadata Network Message Size: Maximum size of network messages that metadata client can receive from a metadata server.

      If unset, defaults to `networking.message-size-limit`. If set, it will be clamped at
      the value of `networking.message-size-limit` since larger messages cannot be transmitted
      over the cluster internal network.

      Non-zero human-readable bytes
    </ResponseField>
  </Expandable>

  <Expandable
    title="Option 2: Store metadata on an external etcd cluster.

The addresses are formatted as `host:port`"
  >
    <ResponseField name="addresses" type="string">
      Etcd cluster node address list:
    </ResponseField>

    <ResponseField name="type" type="string" />

    <ResponseField name="backoff-policy">
      Backoff policy used by the metadata client when it encounters concurrent modifications.

      <Expandable title="Option 1: None">
        No retry strategy.

        <ResponseField name="type" type="string" />
      </Expandable>

      <Expandable title="Option 2: Fixed delay">
        Retry with a fixed delay strategy.

        <ResponseField name="interval" type="string">
          Interval between retries.

          Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

          Examples:
          "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
        </ResponseField>

        <ResponseField name="max-attempts" type="integer | null">
          Max attempts: Number of maximum attempts before giving up. Infinite retries if unset.
        </ResponseField>

        <ResponseField name="type" type="string" />
      </Expandable>

      <Expandable title="Option 3: Exponential">
        Retry with an exponential strategy. The next retry is computed as `min(last_retry_interval * factor, max_interval)`.

        <ResponseField name="factor" type="number">
          Factor: The factor to use to compute the next retry attempt.
        </ResponseField>

        <ResponseField name="initial-interval" type="string">
          Initial Interval: Initial interval for the first retry attempt.

          Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

          Examples:
          "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
        </ResponseField>

        <ResponseField name="max-attempts" type="integer | null">
          Max attempts: Number of maximum attempts before giving up. Infinite retries if unset.
        </ResponseField>

        <ResponseField name="max-interval" type="string | null">
          Max interval: Maximum interval between retries.

          Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

          Human-readable duration: Duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

          Examples:
          "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
        </ResponseField>

        <ResponseField name="type" type="string" />
      </Expandable>
    </ResponseField>

    <ResponseField name="connect-timeout" type="string">
      Connect timeout: TCP connection timeout for connecting to the metadata store.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
    </ResponseField>

    <ResponseField name="keep-alive-interval" type="string">
      Metadata Store Keep Alive Interval: Non-zero duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
    </ResponseField>

    <ResponseField name="keep-alive-timeout" type="string">
      Metadata Store Keep Alive Timeout: Non-zero duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
    </ResponseField>

    <ResponseField name="message-size-limit" type="string | null">
      Metadata Network Message Size: Maximum size of network messages that metadata client can receive from a metadata server.

      If unset, defaults to `networking.message-size-limit`. If set, it will be clamped at
      the value of `networking.message-size-limit` since larger messages cannot be transmitted
      over the cluster internal network.

      Non-zero human-readable bytes
    </ResponseField>
  </Expandable>

  <Expandable title="Option 3: Store metadata on an external object store.">
    <ResponseField name="aws-access-key-id" type="string | null">
      AWS access key: Username for Minio, or consult the service documentation for other S3-compatible stores.
    </ResponseField>

    <ResponseField name="aws-allow-http" type="boolean | null">
      Allow insecure HTTP: Allow plain HTTP to be used with the object store endpoint. Required when the endpoint URL
      that isn't using HTTPS.
    </ResponseField>

    <ResponseField name="aws-endpoint-url" type="string | null">
      Object store API endpoint URL override: When you use Amazon S3, this is typically inferred from the region and there is no need to
      set it. With other object stores, you will have to provide an appropriate HTTP(S) endpoint.
      If *not* using HTTPS, also set `aws-allow-http` to `true`.
    </ResponseField>

    <ResponseField name="aws-profile" type="string | null">
      AWS profile: The AWS configuration profile to use for S3 object store destinations. If you use
      named profiles in your AWS configuration, you can replace all the other settings with
      a single profile reference. See the \[AWS documentation on profiles]
      ([https://docs.aws.amazon.com/sdkref/latest/guide/file-format.html](https://docs.aws.amazon.com/sdkref/latest/guide/file-format.html)) for more.
    </ResponseField>

    <ResponseField name="aws-region" type="string | null">
      AWS region to use with S3 object store destinations. This may be inferred from the
      environment, for example the current region when running in EC2. Because of the
      request signing algorithm this must have a value. For Minio, you can generally
      set this to any string, such as `us-east-1`.
    </ResponseField>

    <ResponseField name="aws-secret-access-key" type="string | null">
      AWS secret key: Password for Minio, or consult the service documentation for other S3-compatible stores.
    </ResponseField>

    <ResponseField name="aws-session-token" type="string | null">
      AWS session token: This is only needed with short-term STS session credentials.
    </ResponseField>

    <ResponseField name="object-store-retry-policy">
      Error retry policy: Definition of a retry policy

      <Expandable title="Option 1: None">
        No retry strategy.

        <ResponseField name="type" type="string" />
      </Expandable>

      <Expandable title="Option 2: Fixed delay">
        Retry with a fixed delay strategy.

        <ResponseField name="interval" type="string">
          Interval between retries.

          Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

          Examples:
          "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
        </ResponseField>

        <ResponseField name="max-attempts" type="integer | null">
          Max attempts: Number of maximum attempts before giving up. Infinite retries if unset.
        </ResponseField>

        <ResponseField name="type" type="string" />
      </Expandable>

      <Expandable title="Option 3: Exponential">
        Retry with an exponential strategy. The next retry is computed as `min(last_retry_interval * factor, max_interval)`.

        <ResponseField name="factor" type="number">
          Factor: The factor to use to compute the next retry attempt.
        </ResponseField>

        <ResponseField name="initial-interval" type="string">
          Initial Interval: Initial interval for the first retry attempt.

          Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

          Examples:
          "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
        </ResponseField>

        <ResponseField name="max-attempts" type="integer | null">
          Max attempts: Number of maximum attempts before giving up. Infinite retries if unset.
        </ResponseField>

        <ResponseField name="max-interval" type="string | null">
          Max interval: Maximum interval between retries.

          Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

          Human-readable duration: Duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

          Examples:
          "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
        </ResponseField>

        <ResponseField name="type" type="string" />
      </Expandable>
    </ResponseField>

    <ResponseField name="path" type="string">
      Object store path for metadata storage: This location will be used to persist cluster metadata. Takes the form of a URL
      with `s3://` as the protocol and bucket name as the authority, plus an optional
      prefix specified as the path component.

      Example: `s3://bucket/prefix`
    </ResponseField>

    <ResponseField name="type" type="string" />

    <ResponseField name="backoff-policy">
      Backoff policy used by the metadata client when it encounters concurrent modifications.

      <Expandable title="Option 1: None">
        No retry strategy.

        <ResponseField name="type" type="string" />
      </Expandable>

      <Expandable title="Option 2: Fixed delay">
        Retry with a fixed delay strategy.

        <ResponseField name="interval" type="string">
          Interval between retries.

          Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

          Examples:
          "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
        </ResponseField>

        <ResponseField name="max-attempts" type="integer | null">
          Max attempts: Number of maximum attempts before giving up. Infinite retries if unset.
        </ResponseField>

        <ResponseField name="type" type="string" />
      </Expandable>

      <Expandable title="Option 3: Exponential">
        Retry with an exponential strategy. The next retry is computed as `min(last_retry_interval * factor, max_interval)`.

        <ResponseField name="factor" type="number">
          Factor: The factor to use to compute the next retry attempt.
        </ResponseField>

        <ResponseField name="initial-interval" type="string">
          Initial Interval: Initial interval for the first retry attempt.

          Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

          Examples:
          "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
        </ResponseField>

        <ResponseField name="max-attempts" type="integer | null">
          Max attempts: Number of maximum attempts before giving up. Infinite retries if unset.
        </ResponseField>

        <ResponseField name="max-interval" type="string | null">
          Max interval: Maximum interval between retries.

          Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

          Human-readable duration: Duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

          Examples:
          "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
        </ResponseField>

        <ResponseField name="type" type="string" />
      </Expandable>
    </ResponseField>

    <ResponseField name="connect-timeout" type="string">
      Connect timeout: TCP connection timeout for connecting to the metadata store.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
    </ResponseField>

    <ResponseField name="keep-alive-interval" type="string">
      Metadata Store Keep Alive Interval: Non-zero duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
    </ResponseField>

    <ResponseField name="keep-alive-timeout" type="string">
      Metadata Store Keep Alive Timeout: Non-zero duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
    </ResponseField>

    <ResponseField name="message-size-limit" type="string | null">
      Metadata Network Message Size: Maximum size of network messages that metadata client can receive from a metadata server.

      If unset, defaults to `networking.message-size-limit`. If set, it will be clamped at
      the value of `networking.message-size-limit` since larger messages cannot be transmitted
      over the cluster internal network.

      Non-zero human-readable bytes
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="metadata-fetch-from-peer-timeout" type="string">
  Timeout for metadata peer-to-peer fetching: When a node detects that a new metadata version exists, it'll attempt to fetch it from
  its peers. After this timeout duration has passed, the node will attempt to fetch the
  metadata from metadata store as well. This is to ensure that the nodes converge quickly
  while reducing the load on the metadata store.

  Examples:
  "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
</ResponseField>

<ResponseField name="metadata-server" type="object">
  Metadata store options:

  <Expandable title="Properties">
    <ResponseField name="auto-join" type="boolean">
      Auto join the metadata cluster when being started

      Defines whether this node should auto join the metadata store cluster when being started
      for the first time.
    </ResponseField>

    <ResponseField name="log-trim-threshold" type="integer | null">
      The raft log trim threshold: The threshold for trimming the raft log. The log will be trimmed if the number of apply entries
      exceeds this threshold. The default value is `1000`.
    </ResponseField>

    <ResponseField name="raft-election-tick" type="integer">
      The number of ticks before triggering an election

      The number of ticks before triggering an election. The value must be larger than
      `raft_heartbeat_tick`. It's recommended to set `raft_election_tick = 10 * raft_heartbeat_tick`.
      Decrease this value if you want to react faster to failed leaders. Note, decreasing this
      value too much can lead to cluster instabilities due to falsely detecting dead leaders.
    </ResponseField>

    <ResponseField name="raft-heartbeat-tick" type="integer">
      The number of ticks before sending a heartbeat

      A leader sends heartbeat messages to maintain its leadership every heartbeat ticks.
      Decrease this value to send heartbeats more often.
    </ResponseField>

    <ResponseField name="raft-tick-interval" type="string">
      Non-zero human-readable duration: The raft tick interval

      The interval at which the raft node will tick. Decrease this value in order to let the Raft
      node react more quickly to changes. Note, that every tick comes with an overhead. Moreover,
      the tick interval directly affects the election timeout. If the election timeout becomes too
      small, then this can cause cluster instabilities due to frequent leader changes.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
    </ResponseField>

    <ResponseField name="request-queue-length" type="integer">
      Limit number of in-flight requests

      Number of in-flight metadata store requests.
    </ResponseField>

    <ResponseField name="rocksdb-block-size" type="string | null">
      RocksDB block size: Uncompressed block size

      Default: 64KiB

      Non-zero human-readable bytes
    </ResponseField>

    <ResponseField name="rocksdb-compaction-readahead-size" type="string | null">
      RocksDB compaction readahead size in bytes: If non-zero, we perform bigger reads when doing compaction. If you're
      running RocksDB on spinning disks, you should set this to at least 2MB.
      That way RocksDB's compaction is doing sequential instead of random reads.

      Non-zero human-readable bytes
    </ResponseField>

    <ResponseField name="rocksdb-disable-direct-io-for-flush-and-compactions" type="boolean | null">
      Disable Direct IO for flush and compactions: Use O\_DIRECT for writes in background flush and compactions.
    </ResponseField>

    <ResponseField name="rocksdb-disable-direct-io-for-reads" type="boolean | null">
      Disable Direct IO for reads: Files will be opened in "direct I/O" mode
      which means that data r/w from the disk will not be cached or
      buffered. The hardware buffer of the devices may however still
      be used. Memory mapped files are not impacted by these parameters.
    </ResponseField>

    <ResponseField name="rocksdb-disable-statistics" type="boolean | null">
      Disable rocksdb statistics collection

      Default: False (statistics enabled)
    </ResponseField>

    <ResponseField name="rocksdb-disable-wal" type="boolean | null">
      Disable WAL: The default depends on the different rocksdb use-cases at Restate.

      Supports hot-reloading (Partial / Bifrost only)
    </ResponseField>

    <ResponseField name="rocksdb-log-keep-file-num" type="integer | null">
      RocksDB log keep file num: Number of info LOG files to keep

      Default: 1
    </ResponseField>

    <ResponseField name="rocksdb-log-level" type="string | null">
      RocksDB log level: Verbosity of the LOG.

      Default: "error"

      Verbosity of the LOG.
    </ResponseField>

    <ResponseField name="rocksdb-log-max-file-size" type="string | null">
      RocksDB log max file size: Max size of info LOG file

      Default: 64MB

      Non-zero human-readable bytes
    </ResponseField>

    <ResponseField name="rocksdb-max-background-jobs" type="integer | null">
      RocksDB max background jobs (flushes and compactions): Default: the number of CPU cores on this node.
    </ResponseField>

    <ResponseField name="rocksdb-memory-budget" type="string | null">
      The memory budget for rocksdb memtables in bytes

      If this value is set, it overrides the ratio defined in `rocksdb-memory-ratio`.

      Non-zero human-readable bytes
    </ResponseField>

    <ResponseField name="rocksdb-memory-ratio" type="number">
      The memory budget for rocksdb memtables as ratio

      This defines the total memory for rocksdb as a ratio of all memory available to memtables
      (See `rocksdb-total-memtables-ratio` in common).
    </ResponseField>

    <ResponseField name="rocksdb-statistics-level" type="oneOf | null">
      RocksDB statistics level: StatsLevel can be used to reduce statistics overhead by skipping certain
      types of stats in the stats collection process.

      Default: "except-detailed-timers"

      * `"disable-all"` : Disable all metrics
      * `"except-histogram-or-timers"` : Disable timer stats, and skip histogram stats
      * `"except-timers"` : Skip timer stats
      * `"except-detailed-timers"` : Collect all stats except time inside mutex lock AND time spent on
        compression.
      * `"except-time-for-mutex"` : Collect all stats except the counters requiring to get time inside the
        mutex lock.
      * `"all"` : Collect all stats, including measuring duration of mutex operations.
        If getting time is expensive on the platform to run, it can
        reduce scalability to more threads, especially for writes.
    </ResponseField>

    <ResponseField name="status-update-interval" type="string">
      Non-zero human-readable duration: The status update interval

      The interval at which the raft node will update its status. Decrease this value in order to
      see more recent status updates.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="metadata-update-interval" type="string">
  Metadata update interval: The idle time after which the node will check for metadata updates from metadata store.
  This helps the node detect if it has been operating with stale metadata for extended period
  of time, primarily because it didn't interact with other peers in the cluster during that
  period.

  Examples:
  "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
</ResponseField>

<ResponseField name="network-error-retry-policy">
  Network error retry policy: The retry policy for network related errors

  <Expandable title="Option 1: None">
    No retry strategy.

    <ResponseField name="type" type="string" />
  </Expandable>

  <Expandable title="Option 2: Fixed delay">
    Retry with a fixed delay strategy.

    <ResponseField name="interval" type="string">
      Interval between retries.

      Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
    </ResponseField>

    <ResponseField name="max-attempts" type="integer | null">
      Max attempts: Number of maximum attempts before giving up. Infinite retries if unset.
    </ResponseField>

    <ResponseField name="type" type="string" />
  </Expandable>

  <Expandable title="Option 3: Exponential">
    Retry with an exponential strategy. The next retry is computed as `min(last_retry_interval * factor, max_interval)`.

    <ResponseField name="factor" type="number">
      Factor: The factor to use to compute the next retry attempt.
    </ResponseField>

    <ResponseField name="initial-interval" type="string">
      Initial Interval: Initial interval for the first retry attempt.

      Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
    </ResponseField>

    <ResponseField name="max-attempts" type="integer | null">
      Max attempts: Number of maximum attempts before giving up. Infinite retries if unset.
    </ResponseField>

    <ResponseField name="max-interval" type="string | null">
      Max interval: Maximum interval between retries.

      Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

      Human-readable duration: Duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
    </ResponseField>

    <ResponseField name="type" type="string" />
  </Expandable>
</ResponseField>

<ResponseField name="networking" type="object">
  Networking options: Common network configuration options for communicating with Restate cluster nodes. Note that
  similar keys are present in other config sections, such as in Service Client options.

  <Expandable title="Properties">
    <ResponseField name="connect-retry-policy">
      Connect retry policy: Retry policy to use for internal node-to-node networking.

      <Expandable title="Option 1: None">
        No retry strategy.

        <ResponseField name="type" type="string" />
      </Expandable>

      <Expandable title="Option 2: Fixed delay">
        Retry with a fixed delay strategy.

        <ResponseField name="interval" type="string">
          Interval between retries.

          Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

          Examples:
          "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
        </ResponseField>

        <ResponseField name="max-attempts" type="integer | null">
          Max attempts: Number of maximum attempts before giving up. Infinite retries if unset.
        </ResponseField>

        <ResponseField name="type" type="string" />
      </Expandable>

      <Expandable title="Option 3: Exponential">
        Retry with an exponential strategy. The next retry is computed as `min(last_retry_interval * factor, max_interval)`.

        <ResponseField name="factor" type="number">
          Factor: The factor to use to compute the next retry attempt.
        </ResponseField>

        <ResponseField name="initial-interval" type="string">
          Initial Interval: Initial interval for the first retry attempt.

          Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

          Examples:
          "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
        </ResponseField>

        <ResponseField name="max-attempts" type="integer | null">
          Max attempts: Number of maximum attempts before giving up. Infinite retries if unset.
        </ResponseField>

        <ResponseField name="max-interval" type="string | null">
          Max interval: Maximum interval between retries.

          Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

          Human-readable duration: Duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

          Examples:
          "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
        </ResponseField>

        <ResponseField name="type" type="string" />
      </Expandable>
    </ResponseField>

    <ResponseField name="connect-timeout" type="string">
      Connect timeout: TCP connection timeout for Restate cluster node-to-node network connections.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
    </ResponseField>

    <ResponseField name="data-stream-window-size" type="string">
      Non-zero human-readable bytes
    </ResponseField>

    <ResponseField name="disable-compression" type="boolean">
      Disable Compression: Disables Zstd compression for internal gRPC network connections
    </ResponseField>

    <ResponseField name="handshake-timeout" type="string">
      Handshake timeout: Timeout for receiving a handshake response from Restate cluster peers.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
    </ResponseField>

    <ResponseField name="http2-adaptive-window" type="boolean">
      HTTP/2 Adaptive Window:
    </ResponseField>

    <ResponseField name="http2-keep-alive-interval" type="string">
      HTTP/2 Keep Alive Interval: Non-zero duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
    </ResponseField>

    <ResponseField name="http2-keep-alive-timeout" type="string">
      HTTP/2 Keep Alive Timeout: Non-zero duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
    </ResponseField>

    <ResponseField name="message-size-limit" type="string">
      Non-zero human-readable bytes
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="no-proxy" type="string | null">
  No proxy: IP subnets, addresses, and domain names eg `localhost,restate.dev,127.0.0.1,::1,192.168.1.0/24` that should not be proxied by the http\_proxy.
  IP addresses must not have ports, and IPv6 addresses must not be wrapped in '\[]'.
  Subdomains are also matched. An entry “\*” matches all hostnames.
  Can be overridden by the `NO_PROXY` environment variable, which supports comma separated values.
</ResponseField>

<ResponseField name="node-name" type="string | null">
  Node Name: Unique name for this node in the cluster. The node must not change unless
  it's started with empty local store. It defaults to the node's hostname.
</ResponseField>

<ResponseField name="request-compression-threshold" type="string | null">
  Request Compression threshold: Request minimum size to enable compression.
  The request size includes the total of the journal replay and its framing using Restate service protocol, without accounting for the json envelope and the base 64 encoding.

  Default: 4MB (The default AWS Lambda Limit is 6MB, 4MB roughly accounts for +33% of Base64 and the json envelope).

  Human-readable bytes
</ResponseField>

<ResponseField name="request-identity-private-key-pem-file" type="string | null">
  Request identity private key PEM file: A path to a file, such as "/var/secrets/key.pem", which contains exactly one ed25519 private
  key in PEM format. Such a file can be generated with `openssl genpkey -algorithm ed25519`.
  If provided, this key will be used to attach JWTs to requests from this client which
  SDKs may optionally verify, proving that the caller is a particular Restate instance.

  This file is currently only read on client creation, but this may change in future.
  Parsed public keys will be logged at INFO level in the same format that SDKs expect.
</ResponseField>

<ResponseField name="rocksdb-bg-threads" type="integer | null">
  Rocksdb Background Threads: The number of threads to reserve to Rocksdb background tasks. Defaults to the number of
  cores on the machine.
</ResponseField>

<ResponseField name="rocksdb-block-size" type="string | null">
  RocksDB block size: Uncompressed block size

  Default: 64KiB

  Non-zero human-readable bytes
</ResponseField>

<ResponseField name="rocksdb-compaction-readahead-size" type="string | null">
  RocksDB compaction readahead size in bytes: If non-zero, we perform bigger reads when doing compaction. If you're
  running RocksDB on spinning disks, you should set this to at least 2MB.
  That way RocksDB's compaction is doing sequential instead of random reads.

  Non-zero human-readable bytes
</ResponseField>

<ResponseField name="rocksdb-disable-direct-io-for-flush-and-compactions" type="boolean | null">
  Disable Direct IO for flush and compactions: Use O\_DIRECT for writes in background flush and compactions.
</ResponseField>

<ResponseField name="rocksdb-disable-direct-io-for-reads" type="boolean | null">
  Disable Direct IO for reads: Files will be opened in "direct I/O" mode
  which means that data r/w from the disk will not be cached or
  buffered. The hardware buffer of the devices may however still
  be used. Memory mapped files are not impacted by these parameters.
</ResponseField>

<ResponseField name="rocksdb-disable-statistics" type="boolean | null">
  Disable rocksdb statistics collection

  Default: False (statistics enabled)
</ResponseField>

<ResponseField name="rocksdb-disable-wal" type="boolean | null">
  Disable WAL: The default depends on the different rocksdb use-cases at Restate.

  Supports hot-reloading (Partial / Bifrost only)
</ResponseField>

<ResponseField name="rocksdb-high-priority-bg-threads" type="integer">
  Rocksdb High Priority Background Threads: The number of threads to reserve to high priority Rocksdb background tasks.
</ResponseField>

<ResponseField name="rocksdb-log-keep-file-num" type="integer | null">
  RocksDB log keep file num: Number of info LOG files to keep

  Default: 1
</ResponseField>

<ResponseField name="rocksdb-log-level" type="string | null">
  RocksDB log level: Verbosity of the LOG.

  Default: "error"

  Verbosity of the LOG.
</ResponseField>

<ResponseField name="rocksdb-log-max-file-size" type="string | null">
  RocksDB log max file size: Max size of info LOG file

  Default: 64MB

  Non-zero human-readable bytes
</ResponseField>

<ResponseField name="rocksdb-max-background-jobs" type="integer | null">
  RocksDB max background jobs (flushes and compactions): Default: the number of CPU cores on this node.
</ResponseField>

<ResponseField name="rocksdb-perf-level">
  Rocksdb performance statistics level: Defines the level of PerfContext used internally by rocksdb. Default is `enable-count`
  which should be sufficient for most users. Note that higher levels incur a CPU cost and
  might slow down the critical path.

  * `"disable"` : Disable perf stats
  * `"enable-count"` : Enables only count stats
  * `"enable-time-except-for-mutex"` : Count stats and enable time stats except for mutexes
  * `"enable-time-and-c-p-u-time-except-for-mutex"` : Other than time, also measure CPU time counters. Still don't measure
    time (neither wall time nor CPU time) for mutexes
  * `"enable-time"` : Enables count and time stats
</ResponseField>

<ResponseField name="rocksdb-statistics-level" type="oneOf | null">
  RocksDB statistics level: StatsLevel can be used to reduce statistics overhead by skipping certain
  types of stats in the stats collection process.

  Default: "except-detailed-timers"

  * `"disable-all"` : Disable all metrics
  * `"except-histogram-or-timers"` : Disable timer stats, and skip histogram stats
  * `"except-timers"` : Skip timer stats
  * `"except-detailed-timers"` : Collect all stats except time inside mutex lock AND time spent on
    compression.
  * `"except-time-for-mutex"` : Collect all stats except the counters requiring to get time inside the
    mutex lock.
  * `"all"` : Collect all stats, including measuring duration of mutex operations.
    If getting time is expensive on the platform to run, it can
    reduce scalability to more threads, especially for writes.
</ResponseField>

<ResponseField name="rocksdb-total-memory-size" type="string">
  Non-zero human-readable bytes
</ResponseField>

<ResponseField name="rocksdb-total-memtables-ratio" type="number">
  Rocksdb total memtable size ratio: The memory size used across all memtables (ratio between 0 to 1.0). This
  limits how much memory memtables can eat up from the value in rocksdb-total-memory-limit.
  When set to 0, memtables can take all available memory up to the value specified
  in rocksdb-total-memory-limit. This value will be sanitized to 1.0 if outside the valid bounds.
</ResponseField>

<ResponseField name="roles" type="array">
  Defines the roles which this Restate node should run, by default the node
  starts with all roles.

  <Expandable title="Array Items">
    <ResponseField name="item">
      * `"http-ingress"` : Serves HTTP ingress requests
      * `"admin"` : Admin runs cluster controller and user-facing admin APIs
      * `"worker"` : A worker runs partition processor (journal, state, and drives invocations)
      * `"log-server"` : Serves a log-server for replicated loglets
      * `"metadata-server"` : Serves the metadata store
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="shutdown-timeout" type="string">
  Shutdown grace timeout: This timeout is used when shutting down the various Restate components to drain all the internal queues.

  Examples:
  "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
</ResponseField>

<ResponseField name="storage-high-priority-bg-threads" type="integer | null">
  Storage high priority thread pool

  This configures the restate-managed storage thread pool for performing
  high-priority or latency-sensitive storage tasks when the IO operation cannot
  be performed on in-memory caches.
</ResponseField>

<ResponseField name="storage-low-priority-bg-threads" type="integer | null">
  Storage low priority thread pool

  This configures the restate-managed storage thread pool for performing
  low-priority or latency-insensitive storage tasks.
</ResponseField>

<ResponseField name="tokio-console-bind-address" type="string">
  Address to bind for the tokio-console tracing subscriber. If unset and restate-server is
  built with tokio-console support, it'll listen on `0.0.0.0:6669`.
</ResponseField>

<ResponseField name="tracing-endpoint" type="string | null">
  Tracing Endpoint: This is a shortcut to set both \[`Self::tracing_runtime_endpoint`], and \[`Self::tracing_services_endpoint`].

  Specify the tracing endpoint to send runtime traces to.
  Traces will be exported using [OTLP gRPC](https://opentelemetry.io/docs/specs/otlp/#otlpgrpc)
  through [opentelemetry\_otlp](https://docs.rs/opentelemetry-otlp/0.12.0/opentelemetry_otlp/).

  To configure the sampling, please refer to the [opentelemetry autoconfigure docs](https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md#sampler).
</ResponseField>

<ResponseField name="tracing-filter" type="string">
  Tracing Filter: Distributed tracing exporter filter.
  Check the [`RUST_LOG` documentation](https://docs.rs/tracing-subscriber/latest/tracing_subscriber/filter/struct.EnvFilter.html) for more details how to configure it.
</ResponseField>

<ResponseField name="tracing-headers" type="object">
  Additional tracing headers: Proxy type to implement HashMap\<HeaderName, HeaderValue> ser/de
  Use it directly or with `#[serde(with = \"serde_with::As::&lt;serde_with::FromInto&lt;restate_serde_util::SerdeableHeaderMap&gt;&gt;\")]`.
</ResponseField>

<ResponseField name="tracing-json-path" type="string | null">
  Distributed Tracing JSON Export Path: If set, an exporter will be configured to write traces to files using the Jaeger JSON format.
  Each trace file will start with the `trace` prefix.

  If unset, no traces will be written to file.

  It can be used to export traces in a structured format without configuring a Jaeger agent.

  To inspect the traces, open the Jaeger UI and use the Upload JSON feature to load and inspect them.
</ResponseField>

<ResponseField name="tracing-runtime-endpoint" type="string | null">
  Runtime Tracing Endpoint: Overrides \[`Self::tracing_endpoint`] for runtime traces

  Specify the tracing endpoint to send runtime traces to.
  Traces will be exported using [OTLP gRPC](https://opentelemetry.io/docs/specs/otlp/#otlpgrpc)
  through [opentelemetry\_otlp](https://docs.rs/opentelemetry-otlp/0.12.0/opentelemetry_otlp/).

  To configure the sampling, please refer to the [opentelemetry autoconfigure docs](https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md#sampler).
</ResponseField>

<ResponseField name="tracing-services-endpoint" type="string | null">
  Services Tracing Endpoint: Overrides \[`Self::tracing_endpoint`] for services traces

  Specify the tracing endpoint to send services traces to.
  Traces will be exported using [OTLP gRPC](https://opentelemetry.io/docs/specs/otlp/#otlpgrpc)
  through [opentelemetry\_otlp](https://docs.rs/opentelemetry-otlp/0.12.0/opentelemetry_otlp/).

  To configure the sampling, please refer to the [opentelemetry autoconfigure docs](https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md#sampler).
</ResponseField>

<ResponseField name="use-random-ports" type="boolean | null">
  Use random ports instead of the default port
</ResponseField>

<ResponseField name="worker" type="object">
  Worker options:

  <Expandable title="Properties">
    <ResponseField name="cleanup-interval" type="string">
      Cleanup interval: In order to clean up completed invocations, that is invocations invoked with an idempotency id, or workflows,
      Restate periodically scans among the completed invocations to check whether they need to be removed or not.
      This interval sets the scan interval of the cleanup procedure. Default: 1 hour.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D"
    </ResponseField>

    <ResponseField name="durability-mode" type="oneOf | null">
      Durability mode: Every partition store is backed up by a durable log that is used to recover the state of
      the partition on restart or failover. The durability mode defines the criteria used
      to determine whether a partition is considered fully durable or not at a given point in the
      log history. Once a partition is fully durable, its backing log is allowed to be trimmed to
      the durability point.

      This helps keeping the log's disk usage under control but it forces nodes that need to restore
      the state of the partition to fetch a snapshot of that partition that covers the changes up to
      and including the "durability point".

      Since v1.4.2 (not compatible with earlier versions)

      * `"none"` : This disables durability tracking and trimming completely.

      Trims and snapshots are still possible if performed manually or by an external
      component.

      * `"snapshot-and-replica-set"` : In this mode, a partition is considered durable when its state can be restored from
        any of members of the replica-set as well as the latest snapshot.

      In other words, do not trim unless **all** replicas cover this Lsn, **and** the snapshot.

      \[requires snapshot repository]
      DurabilityPoint = Min(Min(ReplicaSetDurablePoints), SnapshotDurablePoint)

      * `"balanced"` : In this mode, a partition is considered durable when its state can be restored from
        the snapshot and at least a single replica.

      Do not trim unless the Lsn is covered (durably) by *any* of the replicas **and** by
      the snapshot. Gives weight to snapshots over the durability of the replica-set but
      without ignoring the replica-set completely.

      In practice, this means that after a snapshot has been created on the leader, the
      system will wait for the nearest memtable flush that cover this Lsn before considering
      this Lsn for trimming. If the leader crashes before the memtable flush, we are confident
      that the leader will be able to replay the log without any trim-gaps. This is under the
      condition that the leader didn't move to another node. In the latter case, the system will
      fetch the snapshot as usual.

      \[requires snapshot repository]
      \[default] if snapshot repository configured
      DurabilityPoint = Min(Max(ReplicaSetDurablePoints), SnapshotDurablePoint)

      * `"replica-set-only"` : A partition is considered durable once all nodes in the replica-set are durable, regardless
        of the state of snapshots.

      Do not trim unless all replicas durably include this Lsn.

      default in standalone-mode with no snapshot repository configured

      \[default] if snapshot repository is not configured
      DurabilityPoint = Min(ReplicaSetDurablePoints)

      * `"snapshot-only"` : A partition is durable ONLY after a snapshot has been created.
        \[requires snapshot repository]

      Do not trim unless the Lsn is covered by the snapshot with no regard to the
      state of durability of the replica-set members.

      DurabilityPoint = SnapshotDurablePoint
    </ResponseField>

    <ResponseField name="internal-queue-length" type="integer">
      Internal queue for partition processor communication:
    </ResponseField>

    <ResponseField name="invoker" type="object">
      Invoker options:

      <Expandable title="Properties">
        <ResponseField name="abort-timeout" type="string">
          Human-readable duration: Duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

          Examples:
          "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
        </ResponseField>

        <ResponseField name="action-throttling" type="object | null">
          Action throttling: Configures rate limiting for service actions at the node level.
          This throttling mechanism uses a token bucket algorithm to control the rate
          at which actions can be processed, helping to prevent resource exhaustion
          and maintain system stability under high load.

          The throttling limit is shared across all partitions running on this node,
          providing a global rate limit for the entire node rather than per-partition limits.
          When `unset`, no throttling is applied and actions are processed
          without throttling.

          Throttling options per invoker.

          <Expandable title="Properties">
            <ResponseField name="capacity" type="integer | null">
              Burst capacity: The maximum number of tokens the bucket can hold.
              Default to the rate value if not specified.
            </ResponseField>

            <ResponseField name="rate" type="string">
              Refill rate: The rate at which the tokens are replenished.

              Syntax: `&lt;rate&gt;/&lt;unit&gt;` where `&lt;unit&gt;` is `s|sec|second`, `m|min|minute`, or `h|hr|hour`.
              unit defaults to per second if not specified.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="concurrent-invocations-limit" type="integer | null">
          Limit number of concurrent invocations from this node: Number of concurrent invocations that can be processed by the invoker.
        </ResponseField>

        <ResponseField name="in-memory-queue-length-limit" type="integer">
          Spill invocations to disk: Defines the threshold after which queues invocations will spill to disk at
          the path defined in `tmp-dir`. In other words, this is the number of invocations
          that can be kept in memory before spilling to disk. This is a per-partition limit.
        </ResponseField>

        <ResponseField name="inactivity-timeout" type="string">
          Human-readable duration: Duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

          Examples:
          "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
        </ResponseField>

        <ResponseField name="invocation-throttling" type="object | null">
          Invocation throttling: Configures throttling for service invocations at the node level.
          This throttling mechanism uses a token bucket algorithm to control the rate
          at which invocations can be processed, helping to prevent resource exhaustion
          and maintain system stability under high load.

          The throttling limit is shared across all partitions running on this node,
          providing a global rate limit for the entire node rather than per-partition limits.
          When `unset`, no throttling is applied and invocations are processed
          without throttling.

          Throttling options per invoker.

          <Expandable title="Properties">
            <ResponseField name="capacity" type="integer | null">
              Burst capacity: The maximum number of tokens the bucket can hold.
              Default to the rate value if not specified.
            </ResponseField>

            <ResponseField name="rate" type="string">
              Refill rate: The rate at which the tokens are replenished.

              Syntax: `&lt;rate&gt;/&lt;unit&gt;` where `&lt;unit&gt;` is `s|sec|second`, `m|min|minute`, or `h|hr|hour`.
              unit defaults to per second if not specified.
            </ResponseField>
          </Expandable>
        </ResponseField>

        <ResponseField name="message-size-limit" type="string | null">
          Message size limit: Maximum size of journal messages that can be received from a service. If a service sends a message
          larger than this limit, the invocation will fail.

          If unset, defaults to `networking.message-size-limit`. If set, it will be clamped at
          the value of `networking.message-size-limit` since larger messages cannot be transmitted
          over the cluster internal network.

          Non-zero human-readable bytes
        </ResponseField>

        <ResponseField name="message-size-warning" type="string">
          Non-zero human-readable bytes
        </ResponseField>

        <ResponseField name="tmp-dir" type="string | null">
          Temporary directory to use for the invoker temporary files.
          If empty, the system temporary directory will be used instead.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="max-command-batch-size" type="integer">
      Maximum command batch size for partition processors: The maximum number of commands a partition processor will apply in a batch. The larger this
      value is, the higher the throughput and latency are.
    </ResponseField>

    <ResponseField name="num-timers-in-memory-limit" type="integer | null">
      Num timers in memory limit: The number of timers in memory limit is used to bound the amount of timers loaded in memory. If this limit is set, when exceeding it, the timers farther in the future will be spilled to disk.
    </ResponseField>

    <ResponseField name="shuffle" type="object">
      Worker Shuffle Options: Settings for the shared ingestion client used by all workers to
      manage record ingestion across partitions (shuffle).

      <Expandable title="Properties">
        <ResponseField name="connection-retry-policy">
          Connection retry policy: Retry policy for the ingestion client. It must allow unlimited
          retries; if configured with a cap, the client falls back to
          retrying every 2 seconds.

          <Expandable title="Option 1: None">
            No retry strategy.

            <ResponseField name="type" type="string" />
          </Expandable>

          <Expandable title="Option 2: Fixed delay">
            Retry with a fixed delay strategy.

            <ResponseField name="interval" type="string">
              Interval between retries.

              Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

              Examples:
              "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
            </ResponseField>

            <ResponseField name="max-attempts" type="integer | null">
              Max attempts: Number of maximum attempts before giving up. Infinite retries if unset.
            </ResponseField>

            <ResponseField name="type" type="string" />
          </Expandable>

          <Expandable title="Option 3: Exponential">
            Retry with an exponential strategy. The next retry is computed as `min(last_retry_interval * factor, max_interval)`.

            <ResponseField name="factor" type="number">
              Factor: The factor to use to compute the next retry attempt.
            </ResponseField>

            <ResponseField name="initial-interval" type="string">
              Initial Interval: Initial interval for the first retry attempt.

              Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

              Examples:
              "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
            </ResponseField>

            <ResponseField name="max-attempts" type="integer | null">
              Max attempts: Number of maximum attempts before giving up. Infinite retries if unset.
            </ResponseField>

            <ResponseField name="max-interval" type="string | null">
              Max interval: Maximum interval between retries.

              Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

              Human-readable duration: Duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

              Examples:
              "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
            </ResponseField>

            <ResponseField name="type" type="string" />
          </Expandable>
        </ResponseField>

        <ResponseField name="inflight-memory-budget" type="string">
          Inflight Memory Budget: Maximum total size of in-flight ingestion requests in bytes.
          Tune this to your workload so there are enough unpersisted
          requests for efficient batching without exhausting memory.

          Defaults to 1 MiB.
        </ResponseField>

        <ResponseField name="request-batch-size" type="string">
          Request Batch Size: Maximum size of a single ingestion request batch.
          Tune to keep enough requests per batch for
          throughput; overly large batches can increase tail latency.

          Defaults to 50 KiB.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="snapshots" type="object">
      Snapshots provide a mechanism for safely trimming the log and efficient bootstrapping of new
      worker nodes.

      <Expandable title="Properties">
        <ResponseField name="aws-access-key-id" type="string | null">
          AWS access key: Username for Minio, or consult the service documentation for other S3-compatible stores.
        </ResponseField>

        <ResponseField name="aws-allow-http" type="boolean | null">
          Allow insecure HTTP: Allow plain HTTP to be used with the object store endpoint. Required when the endpoint URL
          that isn't using HTTPS.
        </ResponseField>

        <ResponseField name="aws-endpoint-url" type="string | null">
          Object store API endpoint URL override: When you use Amazon S3, this is typically inferred from the region and there is no need to
          set it. With other object stores, you will have to provide an appropriate HTTP(S) endpoint.
          If *not* using HTTPS, also set `aws-allow-http` to `true`.
        </ResponseField>

        <ResponseField name="aws-profile" type="string | null">
          AWS profile: The AWS configuration profile to use for S3 object store destinations. If you use
          named profiles in your AWS configuration, you can replace all the other settings with
          a single profile reference. See the \[AWS documentation on profiles]
          ([https://docs.aws.amazon.com/sdkref/latest/guide/file-format.html](https://docs.aws.amazon.com/sdkref/latest/guide/file-format.html)) for more.
        </ResponseField>

        <ResponseField name="aws-region" type="string | null">
          AWS region to use with S3 object store destinations. This may be inferred from the
          environment, for example the current region when running in EC2. Because of the
          request signing algorithm this must have a value. For Minio, you can generally
          set this to any string, such as `us-east-1`.
        </ResponseField>

        <ResponseField name="aws-secret-access-key" type="string | null">
          AWS secret key: Password for Minio, or consult the service documentation for other S3-compatible stores.
        </ResponseField>

        <ResponseField name="aws-session-token" type="string | null">
          AWS session token: This is only needed with short-term STS session credentials.
        </ResponseField>

        <ResponseField name="destination" type="string | null">
          Snapshot destination URL: Base URL for cluster snapshots. Currently only supports the `s3://` protocol scheme.
          S3-compatible object stores must support ETag-based conditional writes.

          Default: `None`
        </ResponseField>

        <ResponseField name="enable-cleanup" type="boolean" />

        <ResponseField name="object-store-retry-policy">
          Error retry policy: A retry policy for dealing with retryable object store errors.

          <Expandable title="Option 1: None">
            No retry strategy.

            <ResponseField name="type" type="string" />
          </Expandable>

          <Expandable title="Option 2: Fixed delay">
            Retry with a fixed delay strategy.

            <ResponseField name="interval" type="string">
              Interval between retries.

              Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

              Examples:
              "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
            </ResponseField>

            <ResponseField name="max-attempts" type="integer | null">
              Max attempts: Number of maximum attempts before giving up. Infinite retries if unset.
            </ResponseField>

            <ResponseField name="type" type="string" />
          </Expandable>

          <Expandable title="Option 3: Exponential">
            Retry with an exponential strategy. The next retry is computed as `min(last_retry_interval * factor, max_interval)`.

            <ResponseField name="factor" type="number">
              Factor: The factor to use to compute the next retry attempt.
            </ResponseField>

            <ResponseField name="initial-interval" type="string">
              Initial Interval: Initial interval for the first retry attempt.

              Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

              Examples:
              "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
            </ResponseField>

            <ResponseField name="max-attempts" type="integer | null">
              Max attempts: Number of maximum attempts before giving up. Infinite retries if unset.
            </ResponseField>

            <ResponseField name="max-interval" type="string | null">
              Max interval: Maximum interval between retries.

              Can be configured using the [`jiff::fmt::friendly`](https://docs.rs/jiff/latest/jiff/fmt/friendly/index.html) format or ISO8601, for example `5 hours`.

              Human-readable duration: Duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

              Examples:
              "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
            </ResponseField>

            <ResponseField name="type" type="string" />
          </Expandable>
        </ResponseField>

        <ResponseField name="snapshot-interval" type="string | null">
          Automatic snapshot time interval: A time interval at which partition snapshots will be created. If
          `snapshot-interval-num-records` is also set, it will be treated as an additional requirement
          before a snapshot is taken. Use both time-based and record-based intervals to reduce the
          number of snapshots created during times of low activity.

          Snapshot intervals are calculated based on the wall clock timestamps reported by cluster
          nodes, assuming a basic level of clock synchronization within the cluster.

          This setting does not influence explicitly requested snapshots triggered using `restatectl`.

          Default: `None` - automatic snapshots are disabled

          Human-readable duration: Duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

          Examples:
          "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
        </ResponseField>

        <ResponseField name="snapshot-interval-num-records" type="integer | null">
          Automatic snapshot minimum records: Number of log records that trigger a snapshot to be created.

          As snapshots are created asynchronously, the actual number of new records that will trigger
          a snapshot will vary. The counter for the subsequent snapshot begins from the LSN at which
          the previous snapshot export was initiated.

          This setting does not influence explicitly requested snapshots triggered using `restatectl`.

          Default: `None` - automatic snapshots are disabled
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="storage" type="object">
      Storage options:

      <Expandable title="Properties">
        <ResponseField name="rocksdb-block-size" type="string | null">
          RocksDB block size: Uncompressed block size

          Default: 64KiB

          Non-zero human-readable bytes
        </ResponseField>

        <ResponseField name="rocksdb-compaction-readahead-size" type="string | null">
          RocksDB compaction readahead size in bytes: If non-zero, we perform bigger reads when doing compaction. If you're
          running RocksDB on spinning disks, you should set this to at least 2MB.
          That way RocksDB's compaction is doing sequential instead of random reads.

          Non-zero human-readable bytes
        </ResponseField>

        <ResponseField name="rocksdb-disable-compact-on-deletion" type="boolean">
          Disable compact-on-deletion collector: When set to `true`, disables RocksDB's CompactOnDeletionCollector for partition stores.
          The collector automatically triggers compaction when SST files accumulate a high density
          of tombstones (deletion markers), helping reclaim disk space after bulk deletions.

          This helps control space amplification when invocation journal retention expires and
          the cleaner purges completed invocations.

          Consider disabling this if you observe frequent unnecessary compactions triggered by
          the collector causing performance issues.
        </ResponseField>

        <ResponseField name="rocksdb-disable-direct-io-for-flush-and-compactions" type="boolean | null">
          Disable Direct IO for flush and compactions: Use O\_DIRECT for writes in background flush and compactions.
        </ResponseField>

        <ResponseField name="rocksdb-disable-direct-io-for-reads" type="boolean | null">
          Disable Direct IO for reads: Files will be opened in "direct I/O" mode
          which means that data r/w from the disk will not be cached or
          buffered. The hardware buffer of the devices may however still
          be used. Memory mapped files are not impacted by these parameters.
        </ResponseField>

        <ResponseField name="rocksdb-disable-statistics" type="boolean | null">
          Disable rocksdb statistics collection

          Default: False (statistics enabled)
        </ResponseField>

        <ResponseField name="rocksdb-disable-wal" type="boolean | null">
          Disable WAL: The default depends on the different rocksdb use-cases at Restate.

          Supports hot-reloading (Partial / Bifrost only)
        </ResponseField>

        <ResponseField name="rocksdb-log-keep-file-num" type="integer | null">
          RocksDB log keep file num: Number of info LOG files to keep

          Default: 1
        </ResponseField>

        <ResponseField name="rocksdb-log-level" type="string | null">
          RocksDB log level: Verbosity of the LOG.

          Default: "error"

          Verbosity of the LOG.
        </ResponseField>

        <ResponseField name="rocksdb-log-max-file-size" type="string | null">
          RocksDB log max file size: Max size of info LOG file

          Default: 64MB

          Non-zero human-readable bytes
        </ResponseField>

        <ResponseField name="rocksdb-max-background-jobs" type="integer | null">
          RocksDB max background jobs (flushes and compactions): Default: the number of CPU cores on this node.
        </ResponseField>

        <ResponseField name="rocksdb-memory-budget" type="string | null">
          The memory budget for rocksdb memtables in bytes

          The total is divided evenly across partitions. The server will rebalance the memory budget
          periodically depending on the number of running partitions on this node.

          If this value is set, it overrides the ratio defined in `rocksdb-memory-ratio`.

          Non-zero human-readable bytes
        </ResponseField>

        <ResponseField name="rocksdb-memory-ratio" type="number">
          The memory budget for rocksdb memtables as ratio

          This defines the total memory for rocksdb as a ratio of all memory available to memtables
          (See `rocksdb-total-memtables-ratio` in common). The budget is then divided evenly across
          partitions.
        </ResponseField>

        <ResponseField name="rocksdb-statistics-level" type="oneOf | null">
          RocksDB statistics level: StatsLevel can be used to reduce statistics overhead by skipping certain
          types of stats in the stats collection process.

          Default: "except-detailed-timers"

          * `"disable-all"` : Disable all metrics
          * `"except-histogram-or-timers"` : Disable timer stats, and skip histogram stats
          * `"except-timers"` : Skip timer stats
          * `"except-detailed-timers"` : Collect all stats except time inside mutex lock AND time spent on
            compression.
          * `"except-time-for-mutex"` : Collect all stats except the counters requiring to get time inside the
            mutex lock.
          * `"all"` : Collect all stats, including measuring duration of mutex operations.
            If getting time is expensive on the platform to run, it can
            reduce scalability to more threads, especially for writes.
        </ResponseField>
      </Expandable>
    </ResponseField>

    <ResponseField name="trim-delay-interval" type="string">
      Human-readable duration: Duration string in either jiff human friendly or ISO8601 format. Check [https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing](https://docs.rs/jiff/latest/jiff/struct.Span.html#parsing-and-printing) for more details.

      Examples:
      "10 hours" or "5 days" or "5d" or "1h 4m" or "P40D" or "0"
    </ResponseField>
  </Expandable>
</ResponseField>


# SQL Introspection API
Source: https://docs.restate.dev/references/sql-introspection

API reference for inspecting the invocation status and service state.

# SQL Introspection API

This page contains the reference of the introspection tables.
To learn how to access the introspection interface, check out the [introspection documentation](/services/introspection).

## Table: `state`

| Column name     | Type     | Description                                                                                                                                                                                             |
| --------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `partition_key` | `UInt64` | Internal column that is used for partitioning the services invocations. Can be ignored.                                                                                                                 |
| `service_name`  | `Utf8`   | The name of the invoked service.                                                                                                                                                                        |
| `service_key`   | `Utf8`   | The key of the Virtual Object.                                                                                                                                                                          |
| `key`           | `Utf8`   | The `utf8` state key.                                                                                                                                                                                   |
| `key_length`    | `UInt64` | The byte length of the state key. If you are writing a query that only needs to know the length, reading this field will be much more efficient than reading octet\_length(key).                        |
| `value_utf8`    | `Utf8`   | Only contains meaningful values when a service stores state as `utf8`. This is the case for services that serialize state using JSON (default for Typescript SDK, Java/Kotlin SDK if using JsonSerdes). |
| `value`         | `Binary` | A binary, uninterpreted representation of the value. You can use the more specific column `value_utf8` if the value is a string.                                                                        |
| `value_length`  | `UInt64` | The byte length of the value. If you are writing a query that only needs to know the length, reading this field will be much more efficient than reading length(value).                                 |

## Table: `sys_journal`

| Column name       | Type                   | Description                                                                                                                                                                                                                                                              |
| ----------------- | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `partition_key`   | `UInt64`               | Internal column that is used for partitioning the services invocations. Can be ignored.                                                                                                                                                                                  |
| `id`              | `Utf8`                 | [Invocation ID](/operate/invocation#invocation-identifier).                                                                                                                                                                                                              |
| `index`           | `UInt32`               | The index of this journal entry.                                                                                                                                                                                                                                         |
| `entry_type`      | `Utf8`                 | The entry type. You can check all the available entry types in [`entries.rs`](https://github.com/restatedev/restate/blob/main/crates/types/src/journal/entries.rs).                                                                                                      |
| `name`            | `Utf8`                 | The name of the entry supplied by the user, if any.                                                                                                                                                                                                                      |
| `completed`       | `Boolean`              | Indicates whether this journal entry has been completed; this is only valid for some entry types.                                                                                                                                                                        |
| `invoked_id`      | `Utf8`                 | If this entry represents an outbound invocation, indicates the ID of that invocation.                                                                                                                                                                                    |
| `invoked_target`  | `Utf8`                 | If this entry represents an outbound invocation, indicates the invocation Target. Format for plain services: `ServiceName/HandlerName`, e.g. `Greeter/greet`. Format for virtual objects/workflows: `VirtualObjectName/Key/HandlerName`, e.g. `Greeter/Francesco/greet`. |
| `sleep_wakeup_at` | `TimestampMillisecond` | If this entry represents a sleep, indicates wakeup time.                                                                                                                                                                                                                 |
| `promise_name`    | `Utf8`                 | If this entry is a promise related entry (GetPromise, PeekPromise, CompletePromise), indicates the promise name.                                                                                                                                                         |
| `raw`             | `Binary`               | Raw binary representation of the entry. Check the [service protocol](https://github.com/restatedev/service-protocol) for more details to decode it.                                                                                                                      |
| `version`         | `UInt32`               | The journal version.                                                                                                                                                                                                                                                     |
| `entry_json`      | `Utf8`                 | The entry serialized as a JSON string. Filled only if journal version is 2.                                                                                                                                                                                              |
| `entry_lite_json` | `Utf8`                 | The EntryLite projection serialized as a JSON string. Filled only if journal version is 2.                                                                                                                                                                               |
| `appended_at`     | `TimestampMillisecond` | When the entry was appended to the journal. Filled only if journal version is 2.                                                                                                                                                                                         |

## Table: `sys_journal_events`

| Column name                 | Type                   | Description                                                                                                                        |
| --------------------------- | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| `partition_key`             | `UInt64`               | Internal column that is used for partitioning the services invocations. Can be ignored.                                            |
| `id`                        | `Utf8`                 | [Invocation ID](/operate/invocation#invocation-identifier).                                                                        |
| `after_journal_entry_index` | `UInt32`               | The journal index after which this event happened. This can be used to establish a total order between events and journal entries. |
| `appended_at`               | `TimestampMillisecond` | When the entry was appended to the journal.                                                                                        |
| `event_type`                | `Utf8`                 | The event type.                                                                                                                    |
| `event_json`                | `Utf8`                 | The event serialized as a JSON string.                                                                                             |

## Table: `sys_keyed_service_status`

| Column name     | Type     | Description                                                                             |
| --------------- | -------- | --------------------------------------------------------------------------------------- |
| `partition_key` | `UInt64` | Internal column that is used for partitioning the services invocations. Can be ignored. |
| `service_name`  | `Utf8`   | The name of the invoked virtual object/workflow.                                        |
| `service_key`   | `Utf8`   | The key of the virtual object/workflow.                                                 |
| `invocation_id` | `Utf8`   | [Invocation ID](/operate/invocation#invocation-identifier).                             |

## Table: `sys_inbox`

| Column name       | Type                   | Description                                                                                                                                   |
| ----------------- | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `partition_key`   | `UInt64`               | Internal column that is used for partitioning the services invocations. Can be ignored.                                                       |
| `service_name`    | `Utf8`                 | The name of the invoked virtual object/workflow.                                                                                              |
| `service_key`     | `Utf8`                 | The key of the virtual object/workflow.                                                                                                       |
| `id`              | `Utf8`                 | [Invocation ID](/operate/invocation#invocation-identifier).                                                                                   |
| `sequence_number` | `UInt64`               | Sequence number in the inbox.                                                                                                                 |
| `created_at`      | `TimestampMillisecond` | Timestamp indicating the start of this invocation. DEPRECATED: you should not use this field anymore, but join with the sys\_invocation table |

## Table: `sys_idempotency`

| Column name       | Type     | Description                                                                             |
| ----------------- | -------- | --------------------------------------------------------------------------------------- |
| `partition_key`   | `UInt64` | Internal column that is used for partitioning the services invocations. Can be ignored. |
| `service_name`    | `Utf8`   | The name of the invoked service.                                                        |
| `service_key`     | `Utf8`   | The key of the virtual object or the workflow ID. Null for regular services.            |
| `service_handler` | `Utf8`   | The invoked handler.                                                                    |
| `idempotency_key` | `Utf8`   | The user provided idempotency key.                                                      |
| `invocation_id`   | `Utf8`   | [Invocation ID](/operate/invocation#invocation-identifier).                             |

## Table: `sys_promise`

| Column name                     | Type      | Description                                                                             |
| ------------------------------- | --------- | --------------------------------------------------------------------------------------- |
| `partition_key`                 | `UInt64`  | Internal column that is used for partitioning the services invocations. Can be ignored. |
| `service_name`                  | `Utf8`    | The name of the workflow service.                                                       |
| `service_key`                   | `Utf8`    | The workflow ID.                                                                        |
| `key`                           | `Utf8`    | The promise key.                                                                        |
| `completed`                     | `Boolean` | True if the promise was completed.                                                      |
| `completion_success_value`      | `Binary`  | The completion success, if any.                                                         |
| `completion_success_value_utf8` | `Utf8`    | The completion success as UTF-8 string, if any.                                         |
| `completion_failure`            | `Utf8`    | The completion failure, if any.                                                         |

## Table: `sys_service`

| Column name     | Type      | Description                                                            |
| --------------- | --------- | ---------------------------------------------------------------------- |
| `name`          | `Utf8`    | The name of the registered user service.                               |
| `revision`      | `UInt64`  | The latest deployed revision.                                          |
| `public`        | `Boolean` | Whether the service is accessible through the ingress endpoint or not. |
| `ty`            | `Utf8`    | The service type. Either `service` or `virtual_object` or `workflow`.  |
| `deployment_id` | `Utf8`    | The ID of the latest deployment                                        |

## Table: `sys_deployment`

| Column name                    | Type                   | Description                                                 |
| ------------------------------ | ---------------------- | ----------------------------------------------------------- |
| `id`                           | `Utf8`                 | The ID of the service deployment.                           |
| `ty`                           | `Utf8`                 | The type of the endpoint. Either `http` or `lambda`.        |
| `endpoint`                     | `Utf8`                 | The address of the endpoint. Either HTTP URL or Lambda ARN. |
| `created_at`                   | `TimestampMillisecond` | Timestamp indicating the deployment registration time.      |
| `min_service_protocol_version` | `UInt32`               | Minimum supported protocol version.                         |
| `max_service_protocol_version` | `UInt32`               | Maximum supported protocol version.                         |
| `services`                     | `Utf8 List`            | List of service names registered by this deployment.        |

## Table: `sys_invocation`

| Column name                          | Type                   | Description                                                                                                                                                                                                                                                                                                              |
| ------------------------------------ | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `id`                                 | `Utf8`                 | [Invocation ID](/operate/invocation#invocation-identifier).                                                                                                                                                                                                                                                              |
| `target`                             | `Utf8`                 | Invocation Target. Format for plain services: `ServiceName/HandlerName`, e.g. `Greeter/greet`. Format for virtual objects/workflows: `VirtualObjectName/Key/HandlerName`, e.g. `Greeter/Francesco/greet`.                                                                                                                |
| `target_service_name`                | `Utf8`                 | The name of the invoked service.                                                                                                                                                                                                                                                                                         |
| `target_service_key`                 | `Utf8`                 | The key of the virtual object or the workflow ID. Null for regular services.                                                                                                                                                                                                                                             |
| `target_handler_name`                | `Utf8`                 | The invoked handler.                                                                                                                                                                                                                                                                                                     |
| `target_service_ty`                  | `Utf8`                 | The service type. Either `service` or `virtual_object` or `workflow`.                                                                                                                                                                                                                                                    |
| `idempotency_key`                    | `Utf8`                 | Idempotency key, if any.                                                                                                                                                                                                                                                                                                 |
| `invoked_by`                         | `Utf8`                 | Either: \* `ingress` if the invocation was created externally. \* `service` if the invocation was created by another Restate service. \* `subscription` if the invocation was created by a subscription (e.g. Kafka). \* `restart_as_new` if the invocation was created by restarting an old invocation as new.          |
| `invoked_by_service_name`            | `Utf8`                 | The name of caller service if `invoked_by = 'service'`.                                                                                                                                                                                                                                                                  |
| `invoked_by_id`                      | `Utf8`                 | The caller [Invocation ID](/operate/invocation#invocation-identifier) if `invoked_by = 'service'`.                                                                                                                                                                                                                       |
| `invoked_by_subscription_id`         | `Utf8`                 | The subscription id if `invoked_by = 'subscription'`.                                                                                                                                                                                                                                                                    |
| `invoked_by_target`                  | `Utf8`                 | The caller invocation target if `invoked_by = 'service'`.                                                                                                                                                                                                                                                                |
| `restarted_from`                     | `Utf8`                 | The original invocation id if `invoked_by = 'restart_as_new'`.                                                                                                                                                                                                                                                           |
| `pinned_deployment_id`               | `Utf8`                 | The ID of the service deployment that started processing this invocation, and will continue to do so (e.g. for retries). This gets set after the first journal entry has been stored for this invocation.                                                                                                                |
| `pinned_service_protocol_version`    | `UInt32`               | The negotiated protocol version used for this invocation. This gets set after the first journal entry has been stored for this invocation.                                                                                                                                                                               |
| `trace_id`                           | `Utf8`                 | The ID of the trace that is assigned to this invocation. Only relevant when tracing is enabled.                                                                                                                                                                                                                          |
| `journal_size`                       | `UInt32`               | The number of journal entries durably logged for this invocation.                                                                                                                                                                                                                                                        |
| `journal_commands_size`              | `UInt32`               | The number of commands generated by this invocation, stored in the journal. Only relevant when pinned\_service\_protocol\_version >= 4.                                                                                                                                                                                  |
| `created_at`                         | `TimestampMillisecond` | Timestamp indicating the start of this invocation.                                                                                                                                                                                                                                                                       |
| `created_using_restate_version`      | `Utf8`                 | restate-server version in use when this invocation was created.                                                                                                                                                                                                                                                          |
| `modified_at`                        | `TimestampMillisecond` | Timestamp indicating the last invocation status transition. For example, last time the status changed from `invoked` to `suspended`.                                                                                                                                                                                     |
| `inboxed_at`                         | `TimestampMillisecond` | Timestamp indicating when the invocation was inboxed, if ever.                                                                                                                                                                                                                                                           |
| `scheduled_at`                       | `TimestampMillisecond` | Timestamp indicating when the invocation was scheduled, if ever.                                                                                                                                                                                                                                                         |
| `scheduled_start_at`                 | `TimestampMillisecond` | If the invocation was scheduled, indicates the timestamp when the invocation should start.                                                                                                                                                                                                                               |
| `running_at`                         | `TimestampMillisecond` | Timestamp indicating when the invocation first transitioned to running, if ever.                                                                                                                                                                                                                                         |
| `completed_at`                       | `TimestampMillisecond` | Timestamp indicating when the invocation was completed, if ever.                                                                                                                                                                                                                                                         |
| `completion_retention`               | `DurationMillisecond`  | For how long the metadata of this invocation, including its result, is retained after completion.                                                                                                                                                                                                                        |
| `journal_retention`                  | `DurationMillisecond`  | For how long the journal is retained after completion.                                                                                                                                                                                                                                                                   |
| `suspended_waiting_for_completions`  | `UInt32 List`          | List of completion ids the invocation is awaiting on, if `status = suspended`.                                                                                                                                                                                                                                           |
| `suspended_waiting_for_signals`      | `UInt32 List`          | List of signals the invocation is awaiting on, if `status = suspended`.                                                                                                                                                                                                                                                  |
| `retry_count`                        | `UInt64`               | The number of invocation attempts since the current leader started executing it. Increments on start, so a value greater than 1 means a failure occurred. Note: the value is not a global attempt counter across invocation suspensions and leadership changes.                                                          |
| `last_start_at`                      | `TimestampMillisecond` | Timestamp indicating the start of the most recent attempt of this invocation.                                                                                                                                                                                                                                            |
| `next_retry_at`                      | `TimestampMillisecond` | Timestamp indicating the start of the next attempt of this invocation.                                                                                                                                                                                                                                                   |
| `last_attempt_deployment_id`         | `Utf8`                 | The ID of the service deployment that executed the most recent attempt of this invocation; this is set before a journal entry is stored, but can change later.                                                                                                                                                           |
| `last_attempt_server`                | `Utf8`                 | Server/SDK version, e.g. `restate-sdk-java/1.0.1`                                                                                                                                                                                                                                                                        |
| `last_failure`                       | `Utf8`                 | An error message describing the most recent failed attempt of this invocation, if any.                                                                                                                                                                                                                                   |
| `last_failure_error_code`            | `Utf8`                 | The error code of the most recent failed attempt of this invocation, if any.                                                                                                                                                                                                                                             |
| `last_failure_related_entry_index`   | `UInt64`               | The index of the journal entry that caused the failure, if any. It may be out-of-bound of the currently stored entries in `sys_journal`. DEPRECATED: you should not use this field anymore, but last\_failure\_related\_command\_index instead.                                                                          |
| `last_failure_related_entry_name`    | `Utf8`                 | The name of the journal entry that caused the failure, if any. DEPRECATED: you should not use this field anymore, but last\_failure\_related\_command\_name instead.                                                                                                                                                     |
| `last_failure_related_entry_type`    | `Utf8`                 | The type of the journal entry that caused the failure, if any. You can check all the available entry types in [`entries.rs`](https://github.com/restatedev/restate/blob/main/crates/types/src/journal/entries.rs). DEPRECATED: you should not use this field anymore, but last\_failure\_related\_command\_type instead. |
| `last_failure_related_command_index` | `UInt64`               | The index of the command in the journal that caused the failure, if any. It may be out-of-bound of the currently stored commands in `sys_journal`.                                                                                                                                                                       |
| `last_failure_related_command_name`  | `Utf8`                 | The name of the command that caused the failure, if any.                                                                                                                                                                                                                                                                 |
| `last_failure_related_command_type`  | `Utf8`                 | The type of the command that caused the failure, if any. You can check all the available command types in [`entries.rs`](https://github.com/restatedev/restate/blob/main/crates/types/src/journal_v2/command.rs).                                                                                                        |
| `status`                             | `Utf8`                 | Either `pending` or `scheduled` or `ready` or `running` or `paused` or `backing-off` or `suspended` or `completed`.                                                                                                                                                                                                      |
| `completion_result`                  | `Utf8`                 | If `status = 'completed'`, this contains either `success` or `failure`                                                                                                                                                                                                                                                   |
| `completion_failure`                 | `Utf8`                 | If `status = 'completed' AND completion_result = 'failure'`, this contains the error cause                                                                                                                                                                                                                               |


# Typescript API
Source: https://docs.restate.dev/references/tsdocs





# Highly-Available Clusters
Source: https://docs.restate.dev/server/clusters

Deploying and operating Restate clusters

This page helps with deploying and operating [Restate clusters](/deploy/server/cluster).

<Info>
  To understand the terminology used on this page, it might be helpful to read through the [architecture reference](/references/architecture).
</Info>

<Info>
  For details on configuring network ports, addresses, and listeners, see the [networking guide](/server/networking).
</Info>

<Tip>
  To migrate an existing single-node deployment into a multi-node deployment without losing data, check out [this guide](/guides/local-to-replicated).
</Tip>

## Understanding Partition and Log Replication

Restate uses two complementary replication mechanisms to ensure both availability and durability in a distributed cluster:

### Partition Replication

**Partition replication** determines how many partition processors apply the log for each partition. Each partition handles service invocations for a specific key range. Partition processors run in a leader-follower model where one is the leader (actively processing invocations) while followers replay the log for redundancy.

**Key benefits:**

* **Fast failover**: Followers are already caught up with the log and can quickly take over as leader without reconstructing state from snapshots, reducing Mean Time To Recovery (MTTR) when a node fails

**Trade-offs:**

* Each partition replica consumes disk space and I/O resources to apply log records locally
* Does not improve data durability (that's the role of log replication)

**What it controls:**

* How many nodes simultaneously run partition processors that apply the log for each partition

### Log Replication

**Log replication** determines how many log-server nodes store copies of the durable write-ahead log (Bifrost). This is the foundation of data durability in Restate — the log stores the replicated data, not the partition processors.

**Key characteristics:**

* **Durability guarantee**: The log is the source of truth for all partition state changes. Your data is safe as long as you don't lose `n` log-server nodes, where `n` is the replication property value
* **Quorum-based writes**: Log records are replicated to multiple nodes before being acknowledged
* **Enables partition recovery**: Partitions can be restored on any node by replaying the log or loading from snapshots

**Trade-offs:**

* **Write latency**: Higher replication factors increase susceptibility to straggler delays, as writes must wait for slower nodes in the quorum
* **Storage costs**: Each log-server replica stores a full copy of the log segments

**What it controls:**

* How many log-server nodes store copies of each log segment

### Replication Properties

Both partition and log replication use the same **replication property** format, which can be either:

1. **Simple node count**: `2` means 2 replicas across any nodes
2. **Location-aware replication**: `{zone: 2, node: 3}` means 3 total replicas distributed across at least 2 zones (requires setting the `location` configuration option on nodes)

The `default-replication` setting in your configuration controls both partition and log replication by default, though they can be configured independently using `restatectl config set`.

<Info>
  We recommend setting partition replication to at most 2 unless you have strict Mean Time To Recovery (MTTR) requirements. Higher values consume more resources without improving durability.
</Info>

### Location-Aware Replication

Restate supports location-aware placement to improve fault tolerance across failure domains like zones and regions.

**Configuration:**
Each node can declare its location in the cluster configuration:

```toml restate.toml theme={null}
# Specify the location of this node
location = "us-west.a1"  # format: "region[.zone]"
```

**Examples:**

* `"us-east"` — Region only
* `"eu-central.zone-1"` — Region and zone
* `""` (empty) — Default location (no specific assignment)

**Replication property examples:**

* `{node: 3}` — 3 replicas, distributed across any 3 nodes
* `{zone: 2, node: 3}` — 3 replicas, distributed across at least 2 different zones
* `{region: 2, zone: 3, node: 5}` — 5 replicas, across at least 2 regions and 3 zones

<Warning>
  Node locations should not change after initial node registration, especially for nodes running the `log-server` role, as this can affect replication guarantees.
</Warning>

### Relationship Between the Two

**Partition replication** and **log replication** work together to provide complete availability and durability:

1. **Log replication** ensures your data is durable and survives node failures—the log stores the actual replicated data across log-server nodes
2. **Partition replication** ensures your application stays available by maintaining hot standby partition processors that apply the log
3. Partition processors don't store the replicated data; they apply log records to their local state. The log is the authoritative source
4. You must set log replication high enough to tolerate node failures and maintain durability

**Example configuration:**

```toml restate.toml theme={null}
# Replicate log data to 2 nodes for durability
# Run 2 partition processors for fast failover
default-replication = 2
```

This cluster requires 2 nodes to become operational. With 3 nodes, it can tolerate 1 node failure while maintaining both availability of the system and durability of the data.

## Deploying Restate clusters

To deploy a distributed Restate cluster without external dependencies, you need to configure the following settings in your [server configuration](/server/configuration):

```toml restate.toml theme={null}
# Every node needs to have a unique node name; by default this is the hostname (or podname in Kubernetes)
node-name = "UNIQUE_NODE_NAME"
# All nodes need to have the same cluster name
cluster-name = "CLUSTER_NAME"
# Optional: Specify the location of this node for location-aware replication
# Format: "region[.zone]" (e.g., "us-west.a1" or "eu-central")
location = "REGION[.ZONE]"
# At most one node can be configured with auto-provision = true
auto-provision = false
# Default replication factor for both the logs and the partitions.
#
# Replicate the data to a minimum of 2 nodes. This requires that the cluster has at least 2 nodes to
# become operational. If the cluster has at least 3 nodes, then it can tolerate 1 node failure.
#
# This also controls the default partition replication. A value of 2 means each partition
# will be running on up to 2 nodes whenever possible, ensuring fast fail-over without needing to
# reconstruct partition state if the partition leader becomes unavailable.
#
# For location-aware replication, you can use: {zone: 2, node: 3} or {region: 2, zone: 3, node: 5}
default-replication = 2

# Optional: Override the auto-detected advertised address if needed
# advertised-address = "http://NODE_PUBLIC_IP:5122/"
# Or just set the hostname and let Restate derive the full URL:
# advertised-host = "NODE_PUBLIC_HOSTNAME"

# Make sure it does not conflict with the other nodes when running multiple nodes on same host
# [default] 5122
bind-port = FABRIC_BIND_PORT

[metadata-client]
# List the advertised addresses of at least one node that runs the metadata-server role
# This node will auto-include itself if it runs the metadata-server role
addresses = ["PEER_ADVERTISED_FABRIC_ADDRESS"]

[admin]
# Make sure it does not conflict with the other nodes when running multiple nodes on same host
bind-port = ADMIN_BIND_PORT

[ingress]
# Make sure it does not conflict with other nodes when running multiple nodes on same host
bind-port = INGRESS_BIND_PORT
```

<Info>
  For detailed information about metadata storage options including etcd, and object store alternatives, see [Metadata Storage](/server/metadata).
</Info>

It is important that every Restate node you start has a **unique `node-name`** specified.
The node name defaults to the hostname. Restate uses a subdirectory inside `restate-data` named after the node name to store local data. You will not be able to change the node name once it is in use without removing the node from the cluster, or else risk data loss.
Nodes can have identical configurations aside from node name.

<Info>
  As of Restate v1.6, the server automatically detects a publicly routable IP address for the advertised address. You only need to explicitly set `advertised-address` or `advertised-host` if the auto-detection doesn't work for your environment (e.g., behind certain NAT configurations or load balancers).
</Info>

All nodes that are part of the cluster need to have the same `cluster-name` specified.

At most one node can be configured with `auto-provision = true`.
It is important to avoid having more than one node enabled to auto-provision a cluster as this can lead to provisioning multiple clusters if not all nodes can see each other during startup.
If this happens, you will need to abandon the redundantly provisioned cluster(s), as it is not possible to merge clusters.
If no node is allowed to auto provision, then you have to manually provision the cluster.
Refer to the [Cluster provisioning](#cluster-provisioning) section for more information.

<Tip>
  When running multiple nodes that run on the same host, make sure that ports do not conflict.
</Tip>

### Sizing Clusters

<Info>
  See [Understanding Partition and Log Replication](#understanding-partition-and-log-replication) for details on how replication works.
</Info>

The replicated log provider must be used with `default-provider = "replicated"` (this is the default).
The `default-replication` determines the minimum number of nodes the data must be replicated to for both log and partition replication.
If you run at least `2 * default-replication - 1` nodes, then the cluster can tolerate `default-replication - 1` node failures.
Nodes running the `log-server` role store segments of replicated logs.

Metadata availability is crucial for cluster availability, and the default metadata server type `replicated` can tolerate node failures.
Every node that runs the `metadata-server` role will join the metadata store cluster.
To tolerate `n` metadata node failures, you need to run at least `2 * n + 1` Restate nodes with the `metadata-server` role configured.

The `metadata-client` should be configured with at least one advertised address of a node that runs the `metadata-server` role. Nodes automatically include themselves if they run the metadata-server role, so for single-node setups, this field can be left empty.

Restate nodes running the `worker` role host partitions and handle service invocations, journaling execution, and data storage and queries. The `default-replication` property sets the total number of replicas that the cluster will schedule for any given partition. Only one of these partitions is designated as a leader and actively processes invocations; additional partition replicas are followers and serve as hot standby in case they need to take over processing. Running additional partition replicas does not increase durability, which is determined by log replication and object store snapshots, however it can increase system availability by ensuring a new partition leader can be quickly promoted.

<Info>
  Advanced users can configure different log and partition replication requirements via `restatectl config set --log-replication` / `--partition-replication`.
</Info>

Finally, nodes that run the `http-ingress` role will accept external invocation requests and route them to the appropriate partitions.

<Info>
  [Snapshots](/server/snapshots) are essential to support safe log trimming and also allow you to set partition replication to a subset of all cluster nodes, while still allowing for fast partition fail-over to any live node. Snapshots are also necessary to add more nodes in the future.
</Info>

<Info title="Growing the cluster in the future">
  If you plan to scale your cluster over time, we strongly recommend enabling [snapshotting](/server/snapshots#configuring-automatic-snapshotting).
  Without it, newly added nodes may not be fully utilized by the system.
</Info>

## Cluster provisioning

Once you start the node that is configured with `auto-provision = true`, it will provision the cluster so that other nodes can join.
The provision step initializes the metadata store and writes the initial `NodesConfiguration` with the initial cluster configuration to the metadata store.
In case none of the nodes is allowed to auto-provision, then you need to provision the cluster manually via `restatectl`.

```shell theme={null}
restatectl provision --address <SERVER_TO_PROVISION> --yes
```

This provisions the cluster with default settings specified in the server configuration. Replication settings can be updated after initial provisioning using the `restatectl config set` command, e.g. if you add more nodes to the cluster in the future.

## Controlling clusters with `restatectl`

Restate includes a command line utility tool to connect to and control running Restate servers called `restatectl`.
This tool is specifically designed for system operators to manage Restate servers and is particularly useful in a cluster environment.

<Info>
  Follow the [installation instructions](/installation#advanced%3A-installing-restatectl) to get `restatectl` set up on your machine.
</Info>

The `restatectl` tool communicates with Restate at the advertised address specified in the [server configuration](/references/server-config) - by default TCP port 5122.

<AccordionGroup>
  <Accordion title="View the cluster's current state">
    ```shell theme={null}
    restatectl status
    ```

    Optionally, specify the addresses via `--addresses http://localhost:5122/`:
  </Accordion>

  <Accordion title="Provisioning clusters">
    Check out the [cluster deployment documentation](/server/clusters)
  </Accordion>

  <Accordion title="List all nodes and their assigned roles">
    ```shell theme={null}
    restatectl nodes list
    ```

    Output:

    ```shell theme={null}
    Node Configuration (v21)
    NODE  GEN  NAME    ADDRESS                 ROLES
    N1    6    node-1  http://127.0.0.1:5122/  admin | log-server | metadata-server | worker
    N2    4    node-2  http://127.0.0.1:6122/  admin | log-server | metadata-server | worker
    N3    6    node-3  http://127.0.0.1:7122/  log-server | metadata-server | worker
    ```
  </Accordion>

  <Accordion title="View log configuration">
    View the current log configuration (provider, replication, and nodeset) and the effective logs per partition:

    ```shell theme={null}
    restatectl logs list
    ```

    ```shell theme={null}
    restatectl logs describe
    ```

    Output:

    ```shell theme={null}
    Log chain v8
    └ Logs Provider: replicated
    ├ Log replication: {node: 2}
    └ Nodeset size: 0
    L-ID  FROM-LSN  KIND        LOGLET-ID  REPLICATION  SEQUENCER  NODESET
    0     61        Replicated  0_5        {node: 2}    N1:6       [N1, N2, N3]
    1     4         Replicated  1_4        {node: 2}    N1:6       [N1, N2, N3]
    2     4         Replicated  2_4        {node: 2}    N1:6       [N1, N2, N3]
    3     5         Replicated  3_5        {node: 2}    N1:6       [N1, N2, N3]
    4     4         Replicated  4_4        {node: 2}    N1:6       [N1, N2, N3]
    5     5         Replicated  5_5        {node: 2}    N1:6       [N1, N2, N3]
    6     6         Replicated  6_6        {node: 2}    N1:6       [N1, N2, N3]
    7     4         Replicated  7_4        {node: 2}    N1:6       [N1, N2, N3]
    8     4         Replicated  8_4        {node: 2}    N1:6       [N1, N2, N3]
    ```
  </Accordion>

  <Accordion title="Lists all partition, their current state, and any dead nodes">
    ```shell theme={null}
    restatectl partitions list
    ```

    Output:

    ```shell theme={null}
    Alive partition processors (nodes config v21, partition table v21)
    ID   NODE  MODE      STATUS  EPOCH  APPLIED  DURABLE  ARCHIVED  LSN-LAG  UPDATED
    0    N1:6  Leader    Active  N1:6   61       -        -         0        1 second and 170 ms ago
    0    N2:4  Follower  Active  N1:6   61       -        -         0        1 second and 64 ms ago
    1    N1:6  Leader    Active  N1:6   4        -        -         0        801 ms ago
    1    N2:4  Follower  Active  N1:6   4        -        -         0        779 ms ago
    2    N1:6  Leader    Active  N1:6   4        -        -         0        600 ms ago
    2    N2:4  Follower  Active  N1:6   4        -        -         0        1 second and 108 ms ago
    3    N1:6  Leader    Active  N1:6   5        -        -         0        1 second and 369 ms ago
    3    N2:4  Follower  Active  N1:6   5        -        -         0        1 second and 306 ms ago
    4    N1:6  Leader    Active  N1:6   4        -        -         0        651 ms ago
    4    N2:4  Follower  Active  N1:6   4        -        -         0        1 second and 169 ms ago
    5    N1:6  Leader    Active  N1:6   5        -        -         0        567 ms ago
    5    N2:4  Follower  Active  N1:6   5        -        -         0        1 second and 382 ms ago
    6    N1:6  Leader    Active  N1:6   6        -        -         0        804 ms ago
    6    N2:4  Follower  Active  N1:6   6        -        -         0        1 second and 145 ms ago
    7    N1:6  Leader    Active  N1:6   4        -        -         0        1 second and 79 ms ago
    7    N2:4  Follower  Active  N1:6   4        -        -         0        974 ms ago
    8    N1:6  Leader    Active  N1:6   4        -        -         0        1 second and 71 ms ago
    8    N2:4  Follower  Active  N1:6   4        -        -         0        717 ms ago


    ☠️ Dead nodes
    NODE  LAST-SEEN
    N3    11 minutes, 40 seconds and 995 ms ago
    ```
  </Accordion>

  <Accordion title="View the cluster settings">
    ```shell theme={null}
    restatectl config get
    ```

    Output:

    ```shell theme={null}
    ⚙️ Cluster Configuration
    ├ Number of partitions: 24
    ├ Partition replication: {node: 1}
    └ Logs Provider: replicated
     ├ Log replication: {node: 1}
     └ Nodeset size: 0
    ```
  </Accordion>

  <Accordion title="Update the cluster settings">
    ```shell theme={null}
    restatectl config set --help # check options
    restatectl config set --replication 2 # increases replication
    ```

    Output:

    ```shell theme={null}
    ⚙️ Cluster Configuration
    ├ Number of partitions: 24
    -├ Partition replication: {node: 1}
    +├ Partition replication: {node: 2}
    └ Logs Provider: replicated
    - ├ Log replication: {node: 1}
    + ├ Log replication: {node: 2}
     └ Nodeset size: 0

    ? Apply changes? (y/n) › yes
    ```

    You can also configure partition and log replication independently:

    ```shell theme={null}
    # Set different partition and log replication
    restatectl config set --partition-replication 3 --log-replication 2

    # Use location-aware replication (requires nodes to have configured locations)
    restatectl config set --replication '{zone: 2, node: 3}'
    ```
  </Accordion>
</AccordionGroup>

## Growing clusters

You can expand an existing cluster by adding new nodes after it has been started.

<Steps>
  <Step title="Starting point: single node">
    A Restate cluster can initially be started with a single node.
    Follow the [cluster deployment instructions](/server/clusters) and ensure that:

    * It uses the replicated loglet. If you use local loglet, check [this migration guide](/guides/local-to-replicated).
    * `default-replication` is set to 1
    * [Snapshotting is enabled](/server/snapshots), to ensure that newly added nodes are fully utilized by the system.

    ```toml theme={null}
    # Replicating data to one node: cluster cannot tolerate node failures
    default-replication = 1
    ```
  </Step>

  <Step title="Launch new nodes">
    Launch a new node with the same `cluster-name` and specify at least one existing node's address in `metadata-client.addresses`.
    This allows the new node to discover the metadata servers and join the cluster.
  </Step>

  <Step title="Modify cluster configuration">
    Update the cluster’s replication settings to take advantage of the additional nodes and improve fault tolerance.

    Increase log replication to your desired number. For example, to replicate to two nodes:

    ```shell theme={null}
    restatectl config set --replication 2
    ```

    <Accordion title="Output">
      ```shell theme={null}
      ⚙️ Cluster Configuration
      ├ Number of partitions: 24
      -├ Partition replication: {node: 1}
      +├ Partition replication: {node: 2}
      └ Logs Provider: replicated
      - ├ Log replication: {node: 1}
      + ├ Log replication: {node: 2}
       └ Nodeset size: 0

      ? Apply changes? (y/n) › yes
      ```
    </Accordion>

    Then list the logs:

    ```shell theme={null}
    restatectl logs list
    ```

    <Accordion title="Output">
      ```shell theme={null}
      Log chain v5
      └ Logs Provider: replicated
      ├ Log replication: {node: 2}
      └ Nodeset size: 0
      L-ID  FROM-LSN  KIND        LOGLET-ID  REPLICATION  SEQUENCER  NODESET
      0     2         Replicated  0_2        {node: 2}    N1:2       [N1, N2, N3]
      1     2         Replicated  1_2        {node: 2}    N1:2       [N1, N2, N3]
      2     2         Replicated  2_2        {node: 2}    N1:2       [N1, N2, N3]
      3     2         Replicated  3_2        {node: 2}    N1:2       [N1, N2, N3]
      4     2         Replicated  4_2        {node: 2}    N1:2       [N1, N2, N3]
      5     2         Replicated  5_2        {node: 2}    N1:2       [N1, N2, N3]
      6     2         Replicated  6_2        {node: 2}    N1:2       [N1, N2, N3]
      7     2         Replicated  7_2        {node: 2}    N1:2       [N1, N2, N3]
      ```
    </Accordion>

    You might need to re-run the command a few times until all logs reflect the updated replication setting.
    If the update takes longer than expected, check the node logs for errors or warnings.
  </Step>
</Steps>

## Managing Replicated Logs

Restate relies on a [distributed write-ahead log](/references/architecture#durable-log-“bifrost”) to durably record data flowing through the cluster. The replicated loglet provider is the default backend for both single- and multi-node cluster logs however, in a distributed deployment, understanding how it works becomes more important for safe operation.

You can manage the replicated loglet provider via:

```shell theme={null}
restatectl replicated-loglet
```

The Restate [control plane](/references/architecture#control-plane) selects nodes on which to replicate the log according to the specified log replication.
Each [`log-server` node](/references/architecture#log-servers) in the cluster has a storage state which determines how the control plane may use this node. The `set-storage-state` tool allows you to manually override this state as operational needs dictate.
New log servers come up in the `provisioning` state and will automatically transition to `read-write`. The `read-write` state means that the node is considered both healthy to read from and accept writes, that is it may be selected as a nodeset member for new loglet segments.

### View storage state of log server

You can view the current storage state of log servers in your cluster using the `list-servers` sub-command.

```shell theme={null}
restatectl replicated-loglet list-servers
```

<Accordion title="Output">
  ```shell theme={null}
  Node configuration v12
  Log chain v3
  NODE  GEN   STORAGE-STATE  HISTORICAL LOGLETS  ACTIVE LOGLETS
  N1    N1:3  read-write     4                   2
  N2    N2:2  read-write     4                   2
  N3    N3:2  read-write     4                   2
  ```
</Accordion>

Other valid storage include `data-loss`, `read-only`, and `disabled`. Nodes may transition to `data-loss` if they detect that some previously written data is not available. This does not necessarily imply corruption, only that such nodes may not participate in some quorum checks. Such nodes may transition back to `read-write` if they can be repaired.

The `read-only` and `disabled` states are of particular interest to operators. Log servers in `read-only` storage state may continue to serve both reads and writes, but will no longer be selected as participants in new segments' nodesets. The control plane will reconfigure existing logs to move away from such nodes.

### Manually update the log server state

<Warning>
  **Danger of data loss**:

  `set-storage-state` is a low-level command that allows you to directly set log servers' storage-state. Changing this can lead to cluster unavailability or data loss.
</Warning>

Use the `set-storage-state` sub-command to manually update the log server state, for example to prevent log servers from being included in new nodesets. Consider the following example:

```shell theme={null}
restatectl replicated-loglet set-storage-state --node N.. --storage-state read-only
```

<Accordion title="Output">
  ```shell theme={null}
  Node N.. storage-state updated from read-write to read-only
  ```
</Accordion>

The cluster controller reconfigures the log nodeset to exclude the specified node `N..`. Depending on the configured log replication level, you may see a warning about compromised availability or, if insufficient log servers are available to achieve the minimum required replication, the log will stop accepting writes altogether.
The `restatectl` tool checks whether it is possible to create new nodesets after marking a given node or set of nodes as read-only.
Examine the logs using `restatectl logs describe`.

## Removing Nodes & Shrinking Clusters

You may need to permanently remove nodes from a cluster, whether to replace failing hardware or downsize capacity. This procedure ensures safe node removal without data loss or service interruption. If you are only replacing failing or failed hardware, and intend to keep the cluster at the original size, add replacement nodes first and do not adjust the replication settings. Use these steps only to permanently remove nodes from a cluster; nodes that are only temporarily down should not be removed from the cluster.

<Warning>
  Before removing nodes, ensure that the reduced cluster will meet your capacity, availability and durability requirements. See the section on [Cluster Sizing](#sizing-clusters) for more.
</Warning>

<Steps>
  <Step title="Review cluster nodes and settings">
    Review the cluster replication settings and nodes in use (as well as their current state). If Restate has detected that log data has become corrupted, you might notice that the storage state has already been automatically set to `data-loss`.

    ```shell theme={null}
    restatectl config get
    restatectl nodes list --extra
    ```
  </Step>

  <Step title="Adjust replication settings (if downsizing)">
    Reduce the replication factor to match your target cluster size. The new replication value should be appropriate for the number of nodes that will remain.

    For example, to shrink a 5-node cluster with `{node: 3}` replication) to a 3-node cluster:

    ```shell theme={null}
    restatectl config set --replication 2
    ```
  </Step>

  <Step title="Prepare nodes for removal">
    For nodes running the `log-server` or `worker` roles, set the storage state to `read-only` and worker state to `draining` on each node you plan to remove. This tells the cluster to reconfigure and move log nodesets and partition replicas to other nodes.

    ```shell theme={null}
    restatectl nodes set-storage-state --nodes N..,[N..] --storage-state read-only
    restatectl nodes set-worker-state --nodes N..,[N..] --worker-state draining
    ```

    Replace `N..,[N..]` with the comma-separated list of nodes you wish to remove.
  </Step>

  <Step title="Remove nodes from metadata server">
    If the nodes you are removing run the `metadata-server` role, remove them from the metadata Raft group:

    ```shell theme={null}
    restatectl metadata-server remove-node N..,[N..]
    ```
  </Step>

  <Step title="Verify nodes are not in use">
    Confirm that the nodes to be removed are in the correct state:

    ```shell theme={null}
    restatectl nodes list --extra
    restatectl metadata-server list-servers
    ```

    Verify that:

    * Nodes show `read-only` storage state
    * Nodes show `draining` worker state
    * Nodes are no longer metadata service members
  </Step>

  <Step title="Create partition snapshots">
    Create partition snapshots which will enable the cluster to trim older log segments. This will ensure that no historic nodesets reference nodes about to be removed.

    ```shell theme={null}
    restatectl snapshots create
    ```

    This step requires that you have configured an S3-compatible snapshot destination in your nodes' configurations (see [Configuring Snapshots](/server/snapshots) for more).
  </Step>

  <Step title="Confirm migration">
    Ensure the nodes are no longer running any partitions, nor participating in log nodesets:

    ```shell theme={null}
    restatectl partitions list
    restatectl logs describe
    ```

    Check that the nodes to be removed do not appear in the output. Wait for the cluster to fully migrate partitions and reconfigure logs before proceeding. In particular, ensure that historic nodesets do not reference nodes you intend to remove.
  </Step>

  <Step title="Stop the node processes">
    Stop the Restate processes on the nodes you are removing, e.g. by scaling down the Restate cluster via the Kubernetes Operator. Confirm that the cluster and any applications that depend on it remain operational:

    ```shell theme={null}
    restatectl status
    ```
  </Step>

  <Step title="Remove node entries from cluster">
    Once you have confirmed the cluster is healthy and the removed nodes are no longer needed, remove their entries from the cluster configuration:

    ```shell theme={null}
    restatectl nodes remove --nodes N..
    ```

    Removed nodes will not rejoin the cluster if restarted.
  </Step>
</Steps>

## Troubleshooting Clusters

<AccordionGroup>
  <Accordion title="Node id misconfiguration puts log server in data-loss state">
    If a misconfigured Restate node with the log server role attempts to join a cluster where the node id is already in use, you will observe that the newly started node aborts with an error:

    ```log wrap theme={null}
    ERROR restate_core::task_center: Shutting down: task 4 failed with: Node cannot start a log-server on N3, it has detected that it has lost its data. storage-state is `data-loss`
    ```

    Restarting the existing node that previously owned this id will also cause it to stop with the same message. Follow these steps to return the initial log server into service without losing its stored log segments.

    First, prevent the misconfigured node from starting again until the configuration has been corrected. If this was a brand new node, there should be no data stored on it, and you may delete it altogether.

    The reused node id has been marked as having `data-loss`. This precaution that tells the Restate control plane to avoid selecting this node as member of new log nodesets. You can view the current status using the [`restatectl replicated-loglet` tool](#managing-the-replicated-loglet):

    ```shell theme={null}
    restatectl replicated-loglet servers
    ```

    <Accordion title="Output">
      ```log theme={null}
      Node configuration v21
      Log chain v6
      NODE  GEN   STORAGE-STATE  HISTORICAL LOGLETS  ACTIVE LOGLETS
      N1    N1:5  read-write     8                   2
      N2    N2:4  read-write     8                   2
      N3    N3:6  data-loss      6                   0
      ```
    </Accordion>

    You should also observe that the control plane is now avoiding using this node for log storage. This will result in reduced fault tolerance or even unavailability, depending on the configured minimum log replication:

    ```shell theme={null}
    restatectl logs list
    ```

    <Accordion title="Output">
      ```log theme={null}
      Logs v3
      └ Logs Provider: replicated
      ├ Log replication: {node: 2}
      └ Nodeset size: 0
      L-ID  FROM-LSN  KIND        LOGLET-ID  REPLICATION  SEQUENCER  NODESET
      0     2         Replicated  0_1        {node: 2}    N2:1       [N1, N2]
      1     2         Replicated  1_1        {node: 2}    N2:1       [N1, N2]
      ```
    </Accordion>

    To restore the original node's ability to accept writes, we can update its metadata using `set-storage-state` subcommand.

    <Warning>
      Only proceed if you are confident that you understand the reason why the node is in this state, and are certain that its locally stored data is still intact. Since Restate cannot automatically validate that it safe to put this node back into service, we must use the `--force` flag to override the default state transition rules.
    </Warning>

    ```shell theme={null}
    restatectl replicated-loglet set-storage-state --node N.. --storage-state 'read-write' --force
    ```

    <Accordion title="Output">
      ```text theme={null}
      Node N.. storage-state updated from data-loss to read-write
      ```
    </Accordion>

    You can validate that the server is once again being used for log storage using `logs list` and `replicated-loglet servers` subcommands.
  </Accordion>

  <Accordion title="Handling missing snapshots">
    You are observing a partition processor repeatedly crash-looping with a `TrimGapEncountered` error, or see one of the following errors in the Restate server logs:

    * `A log trim gap was encountered, but no snapshot repository is configured!`
    * `A log trim gap was encountered, but no snapshot is available for this partition!`
    * `The latest available snapshot is from an LSN before the target LSN!`

    You are observing a situation where the local state available on a given worker node does not allow it to resume from the log's trim point - either because it is brand new, or because its applied partition state is behind the trim point of the partition log. If you are attempting to migrate from a single-node Restate to a cluster deployment, you can also refer to the [migration guide](/guides/local-to-replicated).

    To recover from this situation, you need to make available a snapshot of the partition state from another worker, which is up to date with the log. This situation can arise if you have manually trimmed the log, the node is missing a snapshot repository configuration, or the snapshot repository is otherwise inaccessible. See [Log Trimming and Durability](/server/snapshots#log-trimming-and-durability) for more context about how logs, partitions, and snapshots are related.

    #### Recovery procedure

    ##### 1. Identify whether a snapshot repository is configured and accessible

    If a snapshot repository is set up on other nodes in the cluster, and simply not configured on the node where you are seeing the partition processor startup errors, correct the configuration on the new node - refer to [Configuring Snapshots](/server/snapshots#configuring-automatic-snapshotting). If you have not yet set up a snapshot repository, please do so now. If it is impossible to use an object store to host the snapshots repository, you can export snapshots to a local filesystem and manually transfer them to other nodes - skip to step `2b`.

    In your server configuration, you should have a snapshot path specified as follows:

    ```toml theme={null}
    [worker.snapshots]
    destination = "s3://snapshots/prefix"
    ```

    Confirm that this is consistent with other nodes in the cluster.

    Check the server logs for any access errors; does the node have the necessary credentials and are those credentials authorized to access the snapshots destination?

    ##### 2. Publish a snapshot to the repository

    Snapshots are produced periodically by partition processors on certain triggers, such as a number of records being appended to the log. If you are seeing the following error, check that snapshot are being written to the object store destination you have configured.

    Verify that this partition has an active node:

    ```shell theme={null}
    restatectl partitions list
    ```

    If you have lost all nodes which previously hosted this partition, you have permanent data loss - the partition state can not be fully recovered. Get in touch with us to assist in re-starting the partition accepting the data loss.

    Request a snapshot for this partition:

    ```shell theme={null}
    restatectl snapshots create-snapshot {partition_id}
    ```

    You can manually confirm that the snapshot was published to the expected destination. Within the specified snapshot bucket and prefix, you will find a partition-based tree structure. Navigate to the bucket path `{prefix}/{partition_id}` - you should see an entry for the new snapshot id matching the output of the create snapshot command.

    ##### 2b. Alternative: Manually transfer snapshot from another node

    If you are running a cluster but are unable to setup a snapshot repository in a shared object store destination, you can still recover node state by publishing a snapshot from a healthy node ot the local filesystem and manually transferring it to the new node.

    <Info>
      **Experimenting with snapshots without an object store**:
      Note that shared filesystems are not a supported target for cluster snapshots, and have known correctness risks. The `file://` protocol does not support conditional updates, which makes it unsuitable for potentially contended operation.
    </Info>

    Identify an up-to-date node which is running the partition by running:

    ```shell theme={null}
    restatectl partitions list
    ```

    On this node, configure a local destination for the partition snapshot repository - make sure this already exists:

    ```toml theme={null}
    [worker.snapshots]
    destination = "file:///mnt/restate-data/snapshots-repository"
    ```

    Restart the node. If you have multiple nodes which may assume leadership for this partition, you will need to either repeat this on all of them, or temporarily shut them down. Create snapshot(s) for the affected partition(s):

    ```shell theme={null}
    restatectl snapshots create-snapshot {partition_id}
    ```

    Copy the contents of the snapshots repository to the node experiencing issues, and configure it to point to the snapshot repository. If you have multiple snapshots produced by multiple peer nodes, you can merge them all in the same location - each partition's snapshots will be written to dedicated sub-directory for that partition.

    ##### 3. Confirm that the affected node starts up and bootstraps its partition store from a snapshot

    Once you have confirmed that a snapshot for the partition is available at the configured location, the configured repository access credentials have the necessary permissions, and the local node configuration is correct, you should see the partition processor start up and join the partition. If you have updated the Restate server configuration in the process, you should restart the server process to ensure that the latest changes are picked up.
  </Accordion>
</AccordionGroup>


# Restate Server Configuration
Source: https://docs.restate.dev/server/configuration

Configure the Restate Server.

The Restate Server has a wide range of configuration options to tune it according to your needs.

<Info title="Configuration Reference">
  To learn about the configuration options, have a look at the [full configuration reference](/references/server-config).
</Info>

<Info title="Networking Configuration">
  For details on configuring network ports, addresses, and listeners, see the [networking guide](/server/networking).
</Info>

## Configuration file

The Restate Server accepts a [TOML](https://toml.io/en/) configuration file that can be specified either providing the command-line option `--config-file=<PATH>` or setting the environment variable `RESTATE_CONFIG=<PATH>`. If not set, the [default configuration](/references/server-config#default-configuration) will be applied.

## Overrides

Restate server accepts a sub-set of the configuration through command-line arguments, you can see all available options by adding `--help` to `restate-server`.

The order of applying configuration layers follows the following order:

1. Built-in defaults
2. Configuration file (`--config-file` or via `RESTATE_CONFIG`)
3. Environment variables
4. Command line arguments (`--cluster-name=<VALUE>`)

Every layer overrides the previous. For instance, command-line arguments will override a configuration key supplied through environment variable (if set).

### Environment variables

You can override any configuration entry with an environment variable, this overrides values loaded from the configuration file. To do that, the following rule applies:

* Prefix the configuration entry key with `RESTATE_`
* Separate every nested struct with `__` (double underscore) and all hyphens `-` with a `_` (single underscore).

For example, to override the `admin.bind-address`, the corresponding environment variable is `RESTATE_ADMIN__BIND_ADDRESS`.

## Configuration introspection

If you want to generate a configuration file that includes values loaded from your environment variables or overrides applied to `restate-server` command-line, you can add `--dump-config` to dump the default TOML config with overrides applied:

```shell theme={null}
restate-server --cluster-name=mycluster --dump-config
```

Example output:

```toml theme={null}
roles = [
    "worker",
    "admin",
    "metadata-server",
    "log-server",
    "http-ingress",
]
cluster-name = "mycluster"
...
```

At any time, you ask restate daemon to print the loaded configuration to the log by sending a `SIGUSR1` to the server process. This prints a dump of the live configuration to standard error.

For instance on Mac/Linux, you can find the PID of restate-server by running:

```shell theme={null}
pgrep restate-server
994921
```

Then send the signal to the process ID returned from `pgrep`'s output:

```shell theme={null}
kill -USR1 994921
```

Observe the output of the server for a dump of the configuration file contents.


# Deploying Restate on AWS
Source: https://docs.restate.dev/server/deploy/aws

Deploy the Restate Server on Amazon Web Services using EKS, Fargate, and S3.

We recommend deploying Restate on AWS using **Amazon EKS (Elastic Kubernetes Service)** with the [Restate Operator](/server/deploy/kubernetes). This approach provides the best balance of operational simplicity, scalability, and reliability.

## Recommended Architecture

For AWS deployments, we recommend:

* **Compute**: EKS with Fargate for serverless container management
* **Storage**: EBS for persistent volumes, S3 for snapshots (and optionally metadata)

## EKS with Restate Operator

We recommend an EKS cluster using a Fargate profile for compute nodes for the simplest operational experience.

Install the Restate Operator on your EKS cluster.
And create a Restate cluster with EBS storage and snapshots to S3 ([see instructions](/server/deploy/kubernetes)).

The Restate operator has support for [EKS Pod Identities](https://github.com/restatedev/restate-operator/tree/main?tab=readme-ov-file#eks-pod-identity) and [EKS security groups](https://github.com/restatedev/restate-operator/tree/main?tab=readme-ov-file#eks-security-groups-for-pods).

## EC2 and ECS

Running Restate on EKS with managed EC2 nodegroups is another good alternative, though in that case you will need to manage node patching yourself. ECS is not a suitable platform for deploying Restate Clusters as ECS lacks support for persistent volume attachments. ECS is a great target for hosting Restate services, however.

For the highest performance on AWS, expert operators can also consider long-lived EC2 instances with local SSD storage, however this is also going to be the most difficult configuration to manage. You will need to carefully plan your cluster size, node placement, and replication settings to ensure that you can tolerate the loss of a number of nodes without leading to permanent data loss at the Restate cluster level.

## EBS Persistent Volumes

For persistent storage, we generally recommend a [General Purpose SSD (gp3) EBS volume](https://docs.aws.amazon.com/ebs/latest/userguide/general-purpose.html#gp3-ebs-volume-type).

For single-node deployments, EBS volume snapshots can be used as [backups for point-in-time recovery](/server/snapshots#data-backups-2).

We do not recommend using [EBS volume snapshots](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-snapshots.html) for restoring multi-node Restate clusters, since coordinating simultaneous backups across multiple nodes presents significant timing precision requirements, as described in the [backups documentation](/server/snapshots#multi-node-backup-challenges).

## S3 Snapshots

We recommend using Amazon S3 for storing Restate snapshots, and optionally also as metadata store.

Consult [snapshot configuration documentation](/server/snapshots#configuring-automatic-snapshotting) for more information on how to configure this.

If you are using EKS with the Restate Operator, then also check the [Restate Operator documentation](/server/deploy/kubernetes#replicated-cluster-deployment).

## CDK Deployment

AWS CDK users can take advantage of the Restate CDK library provided service deployer construct for managing AWS Lambda-backed services.

<Card title="Restate CDK GitHub" href="https://github.com/restatedev/cdk" icon="github" />


# Deploying Restate on Azure
Source: https://docs.restate.dev/server/deploy/azure

Deploy the Restate Server on Microsoft Azure using AKS and persistent storage.

## Compute

The easiest way to manage Restate is via the [Kubernetes operator](/server/deploy/kubernetes).
You can deploy Restate on [Azure Kubernetes Service (AKS)](https://azure.microsoft.com/en-us/products/kubernetes-service) using the Restate Operator.

## Persistent volumes

For single node deployments, we recommend using Azure Managed Disk as the persistent volume backing Restate's state store.

For clusters, using persistent volumes is still recommended to speed up failover, but not strictly required.

## Object storage for snapshots

Running a cluster without object storage snapshots is not safe. Partition processors scheduled to move between nodes might take excessively long to catch up to the log, and trimming the log might result in errors that prevent them from starting up altogether.

Restate supports Azure Blob Storage natively as a snapshot repository. Configure your snapshot destination using the `az://` URL scheme:

```toml theme={null}
[worker.snapshots]
destination = "az://container/prefix"
```

Credentials are discovered automatically via the [Azure credential chain](https://docs.rs/object_store/latest/object_store/azure/index.html). You can also set `AZURE_STORAGE_ACCOUNT_NAME` and `AZURE_STORAGE_ACCESS_KEY` for explicit credentials.

For local development, you can use [Azurite](https://github.com/Azure/Azurite) as an emulator.

See the [Snapshots documentation](/server/snapshots) for more details on configuring snapshot intervals, retention, and log trimming.


# null
Source: https://docs.restate.dev/server/deploy/docker

Run Restate Server in Docker and Docker Compose.

We recommend using Docker primarily for local development and testing.

While you can deploy Restate with Docker in production, ensure that the data directory is stored on durable storage. In Kubernetes, this is typically handled by deploying Restate as a StatefulSet with persistent volumes (for example, EBS volumes), allowing pods to restart with the same data. Kubernetes also makes it easier to run distributed Restate clusters.

If your Docker setup can provide equivalent durability, such as guaranteeing the same volume is reused across restarts, then running Restate via Docker is perfectly fine.

## Single-node Restate Server

To run a single-node Restate server using Docker, you can use the following command.

```shell theme={null}
docker run -d \
  --name restate \
  --rm \  # Remove the named container on stop
  -p 8080:8080 \
  -p 9070:9070 \
  -p 5122:5122 \
  -v ./restate_data:/restate-data \  # Persist data in the current directory on the host machine
  --add-host host.docker.internal:host-gateway \  # Only when running locally
  docker.restate.dev/restatedev/restate:latest \
  --node-name="restate-1" # Makes sure to restore from restate_data/restate-1 on restart
```

This command starts a Restate server container named `restate` in detached mode.

It maps the necessary ports for ingress, admin, and node-to-node communication.

It also mounts a host volume to persist Restate data across restarts.
To run Restate durably, use a persistent volume for the `restate-data` directory as described in the [server overview](/server/overview#persistent-volumes).

The `--node-name` flag ensures that the server uses a consistent node name, which is essential for data restoration.
When Restate starts, it looks in the `restate-data` directory for a folder matching its node name and uses that data.
If a node starts with a different name (for example, `node-a` finds data for `node-b`), this mismatch leads to data loss.

## Multi-node Restate Cluster with Docker Compose

You can run a Restate Cluster using [Docker](https://docs.docker.com/get-started/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/).

To deploy a 3-node distributed Restate cluster, create a file `docker-compose.yml` and run `docker compose up`.

```yaml expandable lines docker-compose.yml theme={null}
x-environment: &common-env
  RESTATE_CLUSTER_NAME: "restate-cluster"
  # For more on logging, see: https://docs.restate.dev/operate/monitoring/logging
  RESTATE_LOG_FILTER: "restate=info"
  RESTATE_DEFAULT_REPLICATION: 2  # We require minimum of 2 nodes to accept writes
  # The addresses where nodes can reach each other over the "internal" Docker Compose network
  RESTATE_METADATA_CLIENT__ADDRESSES: '["http://restate-1:5122","http://restate-2:5122","http://restate-3:5122"]'
  # Partition snapshotting, see: https://docs.restate.dev/operate/snapshots
  RESTATE_WORKER__SNAPSHOTS__DESTINATION: "s3://restate/snapshots"
  RESTATE_WORKER__SNAPSHOTS__SNAPSHOT_INTERVAL_NUM_RECORDS: "1000"
  RESTATE_WORKER__SNAPSHOTS__AWS_REGION: "local"
  RESTATE_WORKER__SNAPSHOTS__AWS_ENDPOINT_URL: "http://minio:9000"
  RESTATE_WORKER__SNAPSHOTS__AWS_ALLOW_HTTP: true
  RESTATE_WORKER__SNAPSHOTS__AWS_ACCESS_KEY_ID: "minioadmin"
  RESTATE_WORKER__SNAPSHOTS__AWS_SECRET_ACCESS_KEY: "minioadmin"

x-defaults: &defaults
  image: docker.restate.dev/restatedev/restate:latest
  extra_hosts:
    - "host.docker.internal:host-gateway"
  volumes:
    - restate-data:/restate-data

services:
  restate-1:
    <<: *defaults
    ports:
      - "8080:8080"  # Ingress
      - "9070:9070"  # Admin
      - "5122:5122"  # Node-to-node communication
    environment:
      <<: *common-env
      RESTATE_NODE_NAME: restate-1
      RESTATE_FORCE_NODE_ID: 1
      RESTATE_ADVERTISED_ADDRESS: "http://restate-1:5122"  # Other Restate nodes must be able to reach us using this address
      RESTATE_AUTO_PROVISION: "true"                       # Only the first node provisions the cluster

  restate-2:
    <<: *defaults
    ports:
      - "25122:5122"
      - "29070:9070"
      - "28080:8080"
    environment:
      <<: *common-env
      RESTATE_NODE_NAME: restate-2
      RESTATE_FORCE_NODE_ID: 2
      RESTATE_ADVERTISED_ADDRESS: "http://restate-2:5122"
      RESTATE_AUTO_PROVISION: "false"

  restate-3:
    <<: *defaults
    ports:
      - "35122:5122"
      - "39070:9070"
      - "38080:8080"
    environment:
      <<: *common-env
      RESTATE_NODE_NAME: restate-3
      RESTATE_FORCE_NODE_ID: 3
      RESTATE_ADVERTISED_ADDRESS: "http://restate-3:5122"
      RESTATE_AUTO_PROVISION: "false"

  minio:
    image: quay.io/minio/minio
    entrypoint: "/bin/sh"
    # Ensure a bucket called "restate" exists on startup:
    command: "-c 'mkdir -p /data/restate && /usr/bin/minio server --quiet /data'"
    ports:
      - "9000:9000"

# We create a volume to persist data across container starts; delete it via `docker volume rm restate-data` if you want to start a fresh cluster
volumes:
  restate-data:
```

The cluster uses the `replicated` Bifrost provider and replicates log writes to a minimum of two nodes.
Since we are running with 3 nodes, the cluster can tolerate one node failure without becoming unavailable.
By default, partition state is replicated to all workers (though each partition has only one acting leader at a time).

The `replicated` metadata cluster consists of all nodes since they all run the `metadata-server` role.
Since the `replicated` metadata cluster requires a majority quorum to operate, the cluster can tolerate one node failure without becoming unavailable.

Take a look at the [cluster deployment documentation](/server/clusters) for more information on how to configure and deploy a distributed Restate cluster.

In this example we also deployed a Minio server to host the cluster snapshots bucket. Visit [Snapshots](/server/snapshots) to learn more about whis is strongly recommended for all clusters.

<Info> See the [guide on running Restate with Docker Compose](/guides/cluster) for a step-by-step tutorial. </Info>


# Deploying Restate on GCP
Source: https://docs.restate.dev/server/deploy/gcp

Deploy the Restate Server on Google Cloud Platform using GKE and persistent storage.

## Compute

The easiest way to manage Restate is via the [Kubernetes operator](/server/deploy/kubernetes).
You can deploy Restate on [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) using the Restate Operator.

## Persistent volumes

For single node deployments, we recommend using GKE with persistent volumes backing Restate's state store.

For clusters, using persistent volumes is still recommended to speed up failover, but not strictly required.

## Object storage for snapshots

Running a cluster without object storage snapshots is highly discouraged. Partition processors that move to other nodes might not be able to rebuild state and will get stuck.

Restate supports Google Cloud Storage natively as a snapshot repository. Configure your snapshot destination using the `gs://` URL scheme:

```toml theme={null}
[worker.snapshots]
destination = "gs://bucket/prefix"
```

Credentials are discovered automatically via [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials). You can also set `GOOGLE_SERVICE_ACCOUNT_PATH` to use a specific service account JSON file.

For local development, you can use [fake-gcs-server](https://github.com/fsouza/fake-gcs-server) as an emulator.

See the [Snapshots documentation](/server/snapshots) for more details on configuring snapshot intervals, retention, and log trimming.


# null
Source: https://docs.restate.dev/server/deploy/kubernetes

Deploy the Restate Server on Kubernetes.

There are two main ways to deploy the Restate Server on [Kubernetes](https://kubernetes.io/):

1. Using the [Restate Operator](#restate-kubernetes-operator) (recommended): a Kubernetes operator that simplifies deploying and managing Restate clusters and services, with advanced features like automatic service versioning and cloud environment integration.
2. Using the [Helm Chart](#helm-chart): a more bare-bone deployment method that requires more manual operational tasks.

## Restate Kubernetes Operator

We recommend running Restate with the Restate Operator.
The operator simplifies deploying and managing Restate clusters and services on Kubernetes.

**We recommend using the Restate Operator in combination with a managed Kubernetes provider, such as AWS EKS, Azure's AKS, or Google's GKE.**
This eases the management of compute instances and persistent volumes. See the deployment guides for [AWS](/server/deploy/aws), [GCP](/server/deploy/gcp), and [Azure](/server/deploy/azure).

<Card title="Restate Operator GitHub" href="https://github.com/restatedev/restate-operator" icon="github" />

### Features

* Resource types to deploy, manage, and configure Restate clusters and services
* Connect Restate deployments running on Kubernetes to Restate Cloud Environments
* [Automatic service versioning](/services/versioning#automatic-versioning-with-kubernetes-operator) and scaling down old versions
* Set up network security via `NetworkPolicy`
* Sign requests using private keys from Secrets or [CSI Secret Store](https://secrets-store-csi-driver.sigs.k8s.io/)
* Online volume expansion: dynamically increase the size of persistent volumes used by Restate clusters
* For AWS EKS: Manage credentials using [EKS Pod Identity](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html) and security groups using [Security Groups for Pods](https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html)

### Custom Resource Definitions

The Restate Operator introduces Custom Resource Definitions (CRDs) to Kubernetes to deploy Restate clusters and services.

It includes the following CRDs:

* `RestateCluster`: defines a Restate cluster deployment, including storage, networking, and configuration options.
* `RestateDeployment`: defines a Restate service deployment, including container image, resource requirements, and registration options ([learn more](/services/deploy/kubernetes)).
* `RestateCloudEnvironment`: defines a connection between a Kubernetes service deployment and a Restate Cloud Environment ([learn more](/cloud/connecting-services)).

### Operator deployment

Install the Restate Operator via Helm:

```bash theme={null}
helm install restate-operator \
  oci://ghcr.io/restatedev/restate-operator-helm \
  --namespace restate-operator \
  --create-namespace
```

To install the operator, you need to be able to create namespaces and CRDs.

### Single-node deployment

The `RestateCluster` CRD defines a Restate cluster. The operator watches for these objects and creates the necessary Kubernetes resources, such as `StatefulSet`, `Service`, and `NetworkPolicy` objects in a new namespace that matches the `RestateCluster` name.

```yaml restate-server.yaml theme={null}
apiVersion: restate.dev/v1
kind: RestateCluster
metadata:
  name: restate-test
spec:
  compute:
    image: restatedev/restate:1.5
  storage:
    storageRequestBytes: 2147483648 # 2 GiB
```

Apply the manifest via `kubectl`:

```bash theme={null}
kubectl apply -f restate-server.yaml
```

This will deploy the Restate cluster in its own namespace, called `restate-test`.

To learn more about the `RestateCluster` spec options, see the [`RestateCluster` pkl definition](https://github.com/restatedev/restate-operator/blob/main/crd/RestateCluster.pkl) (or the less-readable [yaml version](https://github.com/restatedev/restate-operator/blob/main/crd/restateclusters.yaml)).

<Info>
  Follow [the guide](/guides/restate-on-kind-with-operator) to experiment with deploying a Restate cluster and services on a local kind cluster.
</Info>

### Replicated cluster deployment

Deploy a multi-node Restate cluster by:

1. Setting the `replicas` field in the `compute` section
2. Providing [Restate cluster configuration](/server/clusters) via the `config` field

```yaml restate-cluster.yaml theme={null}
apiVersion: restate.dev/v1
kind: RestateCluster
metadata:
  name: restate-test
spec:
  compute:
    replicas: 3
    image: restatedev/restate:1.5
  storage:
    storageRequestBytes: 2147483648 # 2 GiB
  security:
    # this kind of annotation can be used to give your cluster an IAM role in EKS
    serviceAccountAnnotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/my-role-that-can-read-write-to-the-bucket
  config: |
    roles = [
        "worker",
        "admin",
        "log-server",
        "http-ingress",
    ]
    auto-provision = true
    default-num-partitions = 128
    default-replication = 2

    [metadata-client]
    type = "object-store"
    path = "s3://some-bucket/metadata"
    # the same aws-* parameters as below are supported here

    [bifrost]
    default-provider = "replicated"

    [worker.snapshots]
    destination = "s3://some-bucket/snapshots"
    snapshot-interval-num-records = 10000
    # you can also provide parameters here for non-S3 stores eg:
    # aws-region = "local"
    # aws-access-key-id = "minioadmin"
    # aws-secret-access-key = "minioadmin"
    # aws-endpoint-url = "http://localhost:9000"
    # aws-allow-http = true
```

This example deploys a 3-node Restate cluster with snapshots and metadata stored in S3.

Restate uses RocksDB to store state.
In the Restate Server settings, set the RocksDB memory limit to align with pod memory request and limit. Typically, the RocksDB memory limit should be 75% of pod requests. For example, add to your `restate-config.toml` the following line: `rocksdb-total-memory-size = "3 GiB"`.

S3 is the only supported object store for metadata storage. Other S3-compatible stores (such as MinIO) can be used for snapshots, but not for metadata storage.
You can find more information about configuring MinIO with your Restate cluster [here](https://github.com/restatedev/restate-operator/blob/main/docs/minio.md).

<Warning>
  Running a distributed cluster without snapshots is not recommended for production use.
</Warning>

Apply the manifest:

```bash theme={null}
kubectl apply -f restate-cluster.yaml
```

<Tip>
  Have a look at the [cluster documentation](/server/clusters) to learn how to operate your clusters.
</Tip>

## Helm Chart

For a more bare-bone deployment of a Restate cluster, you can use the Helm chart.
This will require you to do more manual operational tasks, such as registration and versioning of services.

<Card title="Restate Helm Chart GitHub" href="https://github.com/restatedev/restate/tree/main/charts/restate-helm" icon="github" />

The Helm chart does not require anything besides standard Kubernetes functionality: the ability to deploy containers and attach volumes to them.
Any changes to the deployment need to be done manually by updating the Helm chart values and redeploying.

### Single-node deployment

By default, the Helm chart deploys Restate as a single-replica StatefulSet:

```shell theme={null}
helm install restate oci://ghcr.io/restatedev/restate-helm --namespace restate --create-namespace
```

[View the default values.yaml](https://github.com/restatedev/restate/blob/main/charts/restate-helm/values.yaml)

<Info>
  Follow [the guide](/guides/restate-on-kind-with-helm) to experiment with deploying Restate with Helm on a local kind cluster.
</Info>

### Replicated cluster deployment

For a multi-node cluster, use the provided replicated configuration:

```shell theme={null}
helm install restate oci://ghcr.io/restatedev/restate-helm \
  --namespace restate \
  --create-namespace \
  -f https://raw.githubusercontent.com/restatedev/restate/main/charts/restate-helm/replicated-values.yaml
```

[View the replicated-values.yaml](https://github.com/restatedev/restate/blob/main/charts/restate-helm/replicated-values.yaml)

After deployment, you must manually provision the cluster:

```shell theme={null}
kubectl exec -n restate -it restate-0 -- restatectl provision --replication 2 --yes
```

<Warning>
  Make sure to use the `replicated-values.yaml` file that matches your Helm chart version.
</Warning>

### Configuration options

Key values you can customize in your own `values.yaml`:

* **Resources**: Configure CPU/memory limits and requests
  ```yaml values.yaml theme={null}
  resources:
    limits:
      cpu: 1
      memory: 4Gi
    requests:
      cpu: 500m
      memory: 1Gi
  ```

In the Restate Server settings, set the RocksDB memory limit to align with pod memory request and limit. Typically, the RocksDB memory limit should be 75% of pod requests. For example, add to your `restate-config.toml` the following line: `rocksdb-total-memory-size = "3 GiB"`.

* **Storage**: Set persistent volume size and storage class
  ```yaml values.yaml theme={null}
  storage:
    size: 64Gi
    storageClassName: fast-ssd
  ```

* **Replica count**: Number of nodes in the cluster
  ```yaml replicated-values.yaml theme={null}
  replicaCount: 3
  ```

## MinIO object storage

To configure a RestateCluster to send snapshots to self-hosted S3-compatible object store like MinIO, you can point the server to your MinIO instance. As a security best practice, create a dedicated role with access scoped only to the buckets Restate needs. You can assign this role to a dedicated service account or use an AWS-specific feature like [EKS Pod Identity](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html).

To deploy MinIO on your Kubernetes cluster, follow the instructions in the [MinIO documentation](https://docs.min.io/enterprise/aistor-object-store/installation/kubernetes/).

<Warning>
  MinIO should **only** be used for snapshots, not for metadata storage. MinIO's consistency model may corrupt metadata when read quorum is lost, breaking your Restate cluster. Always use the default Raft metadata store for cluster metadata when using MinIO.
</Warning>

Run your MinIO instance with at least 4 replicas to avoid data loss, as [recommended by MinIO](https://docs.min.io/enterprise/aistor-object-store/operations/core-concepts/#minio-architecture).

For more information on configuring MinIO with your Restate cluster, see [here](https://github.com/restatedev/restate-operator/blob/main/docs/minio.md).

## Load balancing

When Restate is deployed as a multi-node cluster, requests can be sent to any node running the [`http-ingress` role](/references/architecture#ingress-2). Restate automatically forwards each request to the node responsible for the target partition.

Although a load balancer is not strictly required, having a single entry point is recommended. The simplest option is a DNS record with multiple A entries for round-robin resolution across all `http-ingress` nodes.

On Kubernetes, you can create a `Service` object for this purpose. The Restate operator creates these services automatically for you.

Adding a load balancer can further improve your setup by performing health checks and routing traffic only to ready nodes.

The specific Kubernetes provider to which you deploy will likely support binding to specific native load-balancer services, for example, AWS ELB in AWS EKS.


# Memory Configuration
Source: https://docs.restate.dev/server/memory

Understanding and configuring Restate Server memory usage.

Memory is a critical resource for Restate, and the server is designed to manage it efficiently while giving you straightforward control over usage. Restate provides top-level configuration options for each major memory pool, and internally handles the complexity of distributing and rebalancing memory as your cluster scales and during failover events.

## Overview

Restate allocates memory from several distinct pools, each with its own configuration:

| Memory Pool              | Default | Configuration                      |
| ------------------------ | ------- | ---------------------------------- |
| **RocksDB**              | 2 GiB   | `rocksdb-total-memory-size`        |
| **Query Engine**         | 1 GiB   | `admin.query-engine.memory-size`   |
| **Bifrost Record Cache** | 250 MiB | `bifrost.record-cache-memory-size` |

The total memory footprint is approximately the sum of these pools plus overhead for the runtime, network buffers, and other internal structures. You should configure each pool based on your available resources and workload requirements.

<Info title="Default values are not minimum requirements">
  The default memory values are sized for production workloads. Restate runs well with significantly less memory for development, testing, or smaller deployments.

  For example, a minimal configuration suitable for development or low-traffic environments:

  ```toml restate.toml theme={null}
  # Minimal memory configuration (~1.5 GiB total)
  rocksdb-total-memory-size = "1 GiB"

  [admin.query-engine]
  memory-size = "256 MiB"

  [bifrost]
  record-cache-memory-size = "128 MiB"
  ```

  Or via environment variables:

  ```bash theme={null}
  export RESTATE_ROCKSDB_TOTAL_MEMORY_SIZE="1 GiB"
  export RESTATE_ADMIN__QUERY_ENGINE__MEMORY_SIZE="256 MiB"
  export RESTATE_BIFROST__RECORD_CACHE_MEMORY_SIZE="128 MiB"
  ```
</Info>

## RocksDB Memory

The RocksDB memory pool is typically the largest. Restate uses [RocksDB](https://rocksdb.org/) as its storage engine for durable state, journal entries, and log data.

```toml restate.toml theme={null}
# Total memory limit for RocksDB caches and memtables (default: 2 GiB)
rocksdb-total-memory-size = "2 GiB"
```

**Environment variable**: `RESTATE_ROCKSDB_TOTAL_MEMORY_SIZE`

This value can be specified using human-readable byte units such as `"4 GiB"`, `"8192 MiB"`, or as a raw byte count.

Restate creates a shared block cache and write buffer manager used by all RocksDB databases on the node. The `rocksdb-total-memory-size` parameter sets the total budget that Restate internally manages and distributes across the partition store, log server, metadata server, and local loglet databases.

### Automatic Memory Rebalancing

In a cluster environment, the number of active partitions on each node can dynamically change as nodes come up, go down, and during failover events. To ensure efficient use of the memory budget regardless of how many partitions are currently active, Restate includes a memory controller that automatically rebalances memory budgets across partitions.

```mermaid theme={null}
flowchart LR
    subgraph Before["Before Failover"]
        direction TB
        N1A["Node 1<br/>4 partitions<br/>250 MiB each"]
        N2A["Node 2<br/>4 partitions<br/>250 MiB each"]
    end

    subgraph After["After Node 2 Failure"]
        direction TB
        N1B["Node 1<br/>8 partitions<br/>125 MiB each"]
        N2B["Node 2<br/>(down)"]
    end

    Before --> |"Failover<br/>+ Rebalance"| After
```

Every few seconds, the controller monitors memory usage across all open partition databases, redistributes the total memory budget evenly across active partitions, and triggers memtable flushes for partitions that exceed their allocated budget. The rebalancing also responds to configuration changes, so you can adjust `rocksdb-total-memory-size` at runtime and the system will adapt.

### Expected Memory Behavior

<Info title="Memory stays high after load decreases">
  It is completely normal and expected to see the RocksDB memory budget remain fully consumed even after load on the cluster goes down. This is because the block cache operates as an LRU (Least Recently Used) cache.

  The cache does not proactively release memory when load decreases. Instead, it retains cached data until new data needs to be cached, at which point older entries are evicted to make room. This behavior is intentional and beneficial: keeping the cache warm means that if similar queries or operations occur again, they can be served from memory rather than requiring disk I/O.

  In other words, high memory usage by the block cache is a sign that the system is working efficiently, not a problem to be solved.
</Info>

## Query Engine Memory

The query engine (used for SQL introspection queries) has its own separate memory budget:

```toml restate.toml theme={null}
[admin.query-engine]
# Memory budget for SQL query execution (default: 1 GiB)
memory-size = "1 GiB"
```

**Environment variable**: `RESTATE_ADMIN__QUERY_ENGINE__MEMORY_SIZE`

This memory is used for query processing, including sorting, aggregation, and join operations. Complex queries on large datasets may require more memory. If queries fail with out-of-memory errors, consider increasing this value.

## Bifrost Record Cache

The Bifrost log subsystem maintains an in-memory cache for recently written log records:

```toml restate.toml theme={null}
[bifrost]
# In-memory record cache size (default: 250 MiB)
record-cache-memory-size = "250 MiB"
```

**Environment variable**: `RESTATE_BIFROST__RECORD_CACHE_MEMORY_SIZE`

This cache improves read performance for the replicated log by keeping recently written records in memory. Increasing this value can improve performance for workloads that frequently read from the log, such as partition replay during failover.

## Recommendations

### General Guidelines

1. **Consider allocating 20-50% of available process memory to RocksDB** as a starting point. This leaves room for the query engine, Bifrost record cache, runtime, network buffers, and other components. You may need to adjust this based on your specific workload and usage patterns.

2. **On Linux, Restate automatically detects cgroup memory limits** (both v1 and v2) and will emit warnings if the configured `rocksdb-total-memory-size` is too close to the process memory limit.

3. **Monitor memory usage** using the metrics exposed by Restate to ensure your configuration is appropriate for your workload.

### Container and Kubernetes Deployments

When running Restate in containers or Kubernetes, consider setting memory pools to fit within your container's memory limit. For example, with a 4 GiB container:

```toml restate.toml theme={null}
rocksdb-total-memory-size = "2 GiB"

[admin.query-engine]
memory-size = "1 GiB"
```

Or via environment variables in your container spec:

```yaml theme={null}
env:
  - name: RESTATE_ROCKSDB_TOTAL_MEMORY_SIZE
    value: "2 GiB"
  - name: RESTATE_ADMIN__QUERY_ENGINE__MEMORY_SIZE
    value: "1 GiB"
```

<Warning>
  If `rocksdb-total-memory-size` exceeds 90% of the process memory limit, Restate will log an error warning of potential OOM conditions. If it exceeds 100%, OOM under load is guaranteed.
</Warning>

### Using the Restate Operator

When deploying Restate on Kubernetes using the [Restate Operator](https://github.com/restatedev/restate-operator), configure memory through the `RestateCluster` resource:

```yaml theme={null}
apiVersion: restate.dev/v1
kind: RestateCluster
metadata:
  name: my-cluster
spec:
  compute:
    replicas: 3
    image: restatedev/restate:latest
    resources:
      requests:
        memory: "8Gi"
      limits:
        memory: "8Gi"
  storage:
    storageRequestBytes: 10737418240  # 10 GiB
  config: |
    rocksdb-total-memory-size = "2 GiB"
    
    [admin.query-engine]
    memory-size = "1 GiB"
```

The operator passes the configuration through to the Restate server via the `spec.config` field.

## Monitoring Memory Usage

Restate exposes memory usage metrics via Prometheus at the `/metrics` endpoint on the node port (default: 5122). Key metrics to monitor include:

| Metric                                                       | Description                                                                                                  |
| ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------ |
| `restate_rocksdb_memory_write_buffer_manager_capacity_bytes` | Total capacity of the write buffer manager, which manages combined memory usage across all RocksDB databases |
| `restate_rocksdb_memory_write_buffer_manager_usage_bytes`    | Current usage of the write buffer manager (combined across all RocksDB databases)                            |
| `restate_rocksdb_memory_approx_memtable_bytes`               | Approximate total memory used by memtables                                                                   |
| `restate_rocksdb_memory_approx_memtable_unflushed_bytes`     | Approximate memory used by unflushed memtables                                                               |
| `restate_rocksdb_block_cache_capacity_bytes`                 | Total block cache capacity                                                                                   |
| `restate_rocksdb_block_cache_usage_bytes`                    | Current block cache usage                                                                                    |

Additional per-column-family metrics are available for deeper analysis:

| Metric                                            | Description                                |
| ------------------------------------------------- | ------------------------------------------ |
| `restate_rocksdb_cur_size_active_mem_table_bytes` | Size of the active memtable                |
| `restate_rocksdb_cur_size_all_mem_tables_bytes`   | Size of all memtables (active + immutable) |
| `restate_rocksdb_mem_table_flush_pending_count`   | Whether a memtable flush is pending        |

Example of scraping metrics:

```bash theme={null}
curl http://localhost:5122/metrics | grep restate_rocksdb
```

You can also view memory statistics by sending `SIGUSR1` to the Restate process, which will dump the current configuration including memory settings to standard error.

## Troubleshooting

### OOM Errors

If the Restate process is being killed by the OOM killer:

1. **Reduce `rocksdb-total-memory-size`** to leave more headroom
2. **Try reducing `admin.query-engine.memory-size`** if running complex queries
3. **Ensure container memory limits match your configuration**
4. **Review `bifrost.record-cache-memory-size`** and reduce if necessary

## Advanced Configuration

For most deployments, the top-level memory settings described above are sufficient. This section covers advanced options for fine-tuning how RocksDB memory is distributed internally.

### RocksDB Memory Architecture

RocksDB uses memory for two primary purposes:

1. **Block Cache**: An LRU cache that stores uncompressed data blocks read from disk. This speeds up read operations by avoiding disk I/O for frequently accessed data.

2. **Memtables**: In-memory write buffers where data is accumulated before being flushed to disk as sorted string tables (SST files).

```mermaid theme={null}
flowchart TB
    subgraph RocksDB["RocksDB Total Memory Budget"]
        direction TB
        BC["Block Cache<br/>(15% default)"]
        MT["Memtables<br/>(85% default)"]
    end

    subgraph Databases["Distributed Across Databases"]
        PS["Partition Store<br/>(49% of memtables)"]
        LS["Log Server<br/>(50% of memtables)"]
        MS["Metadata Server<br/>(1% of memtables)"]
    end

    BC --> Databases
    MT --> PS
    MT --> LS
    MT --> MS
```

### Memtable Ratio

By default, 85% of the total RocksDB memory is allocated to memtables, with the remaining 15% for the block cache. You can adjust this ratio:

```toml restate.toml theme={null}
# Ratio of total memory allocated to memtables (default: 0.85)
rocksdb-total-memtables-ratio = 0.85
```

**Environment variable**: `RESTATE_ROCKSDB_TOTAL_MEMTABLES_RATIO`

A higher ratio allocates more memory to write buffers, which can improve write throughput but reduces the block cache size available for reads.

### Per-Component Memory Allocation

Within the total memtable budget, each component (partition store, log server, etc.) receives a portion based on its configured ratio. These are the defaults:

| Component       | Default Ratio | Description                              |
| --------------- | ------------- | ---------------------------------------- |
| Partition Store | 49%           | Stores service state and journal entries |
| Log Server      | 50%           | Stores replicated log segments           |
| Metadata Server | 1%            | Stores cluster metadata                  |

You can override the default ratios by setting a different ratio or an explicit memory budget for each component:

```toml restate.toml theme={null}
[worker.storage]
# Override partition store memtable ratio (default: 0.49)
rocksdb-memory-ratio = 0.6

# Or set an explicit budget which takes precedence over the ratio
rocksdb-memory-budget = "2 GiB"

[log-server]
# Override log server memtable ratio (default: 0.5)
rocksdb-memory-ratio = 0.3

# Or set an explicit budget which takes precedence over the ratio
rocksdb-memory-budget = "1 GiB"

[metadata-server]
# Override metadata server memtable ratio (default: 0.01)
rocksdb-memory-ratio = 0.02
```

**Environment variables**:

* `RESTATE_WORKER__STORAGE__ROCKSDB_MEMORY_RATIO`
* `RESTATE_WORKER__STORAGE__ROCKSDB_MEMORY_BUDGET`
* `RESTATE_LOG_SERVER__ROCKSDB_MEMORY_RATIO`
* `RESTATE_LOG_SERVER__ROCKSDB_MEMORY_BUDGET`
* `RESTATE_METADATA_SERVER__ROCKSDB_MEMORY_RATIO`
* `RESTATE_METADATA_SERVER__ROCKSDB_MEMORY_BUDGET`

<Info>
  When both `rocksdb-memory-ratio` and `rocksdb-memory-budget` are set for a component, the explicit budget takes precedence over the ratio.
</Info>

## Configuration Reference

For the complete list of memory-related configuration options, see the [Server Configuration Reference](/references/server-config).

| Option                             | Default   | Environment Variable                        | Description                                        |
| ---------------------------------- | --------- | ------------------------------------------- | -------------------------------------------------- |
| `rocksdb-total-memory-size`        | `2 GiB`   | `RESTATE_ROCKSDB_TOTAL_MEMORY_SIZE`         | Total memory for RocksDB block cache and memtables |
| `rocksdb-total-memtables-ratio`    | `0.85`    | `RESTATE_ROCKSDB_TOTAL_MEMTABLES_RATIO`     | Ratio of total memory allocated to memtables       |
| `admin.query-engine.memory-size`   | `1 GiB`   | `RESTATE_ADMIN__QUERY_ENGINE__MEMORY_SIZE`  | Memory budget for SQL query execution              |
| `bifrost.record-cache-memory-size` | `250 MiB` | `RESTATE_BIFROST__RECORD_CACHE_MEMORY_SIZE` | In-memory cache for log records                    |


# Metadata Storage
Source: https://docs.restate.dev/server/metadata

Understanding and configuring metadata storage providers for Restate

Restate stores cluster configuration data in a dedicated metadata store. This page explains the available metadata backend options, and how to choose and configure the right one for your deployment.

<Info title="Architectural overview">
  To understand the terminology used on this page, it might be helpful to read through the [architecture reference](/references/architecture#metadata-store).
</Info>

## What is Metadata?

Metadata in Restate includes critical cluster coordination information:

* **Cluster membership**: Which nodes are part of the cluster and their roles
* **Partition assignments**: How partitions are distributed across worker nodes
* **Service deployments**: Registered service endpoints and their versions
* **Log configuration**: How the replicated log is configured and which nodes participate

The metadata store is essential for cluster operation. While metadata is updated relatively infrequently, its availability is crucial for cluster health.

<Warning>
  Choose your metadata provider during initial deployment based on your infrastructure and requirements. While migration between providers is possible from replicated to external stores, this operation requires a maintenance window (downtime) and careful execution.
</Warning>

## Metadata Storage Providers

Restate supports four metadata storage providers, each suited to different deployment scenarios:

1. **Replicated** (Default) - Built-in Raft-based consensus metadata server
2. **Etcd** - External etcd cluster integration
3. **Amazon S3** - We strongly recommend against using other S3-compatible stores for metadata

## When to Use Each Provider

### Replicated (Default)

The replicated metadata server uses Raft consensus to provide a highly-available metadata store built directly into Restate nodes.

**Use cases:**

* Standard production deployments
* On-premises or multi-cloud environments
* Deployments where you want minimal external dependencies
* Environments where you already run stateful services

**Requirements:**

* Nodes must run the `metadata-server` role
* Persistent storage for metadata server nodes
* Odd number of metadata-server nodes for quorum (typically 3 or 5)

**Trade-offs:**

* ✅ No external dependencies - everything runs within Restate
* ✅ Battle-tested Raft consensus for strong consistency
* ✅ Works in any environment (cloud, on-prem, hybrid)
* ⚠️ Needs careful attention to node placement for fault tolerance
* ⚠️ Requires manual provisioning after majority of nodes are up to avoid split brain (other backends safely support auto-provisioning in a multi-node cluster)

### Etcd

Use an external etcd cluster as the metadata store. This is ideal if you already operate etcd infrastructure or prefer to centralize metadata management.

**Use cases:**

* Organizations with existing etcd infrastructure
* Environments with dedicated etcd operations expertise
* Multi-tenant deployments with centralized control

**Requirements:**

* External etcd cluster (v3.x) must be available
* Network connectivity between Restate nodes and etcd
* etcd cluster properly configured for production use

**Trade-offs:**

* ✅ Leverage existing etcd infrastructure
* ✅ Separate metadata concerns from Restate node management
* ✅ Allows sharing metadata across clusters
* ⚠️ Adds external dependency that must be separately managed
* ⚠️ Network latency to etcd can affect cluster operations
* ⚠️ Requires etcd operations expertise

### Object Store (Amazon S3)

Use Amazon S3 as a metadata store. This provides a lightweight alternative to running metadata servers while keeping all data in object storage.

**Use cases:**

* AWS deployments with existing S3 infrastructure
* Minimizing stateful components
* Environments optimizing for S3 as primary storage

**Requirements:**

* Amazon S3 bucket (not S3-compatible alternatives)
* IAM permissions for S3 operations
* Very low-latency access to S3 (same region recommended)

**Trade-offs:**

* ✅ No metadata servers to manage
* ✅ Integrates with existing S3 buckets
* ✅ Simple backup and recovery
* ⚠️ **Only Amazon S3 is tested and supported** (notably, MinIO is not supported for metadata)
* ⚠️ Tied to a single region!
* ⚠️ Network latency to S3 directly affects cluster operations
* ⚠️ Must ensure very low-latency access (typically same region)

<Warning>
  For object store metadata: Only Amazon S3 is currently tested and supported. Using S3-compatible alternatives is not supported and may lead to correctness issues. MinIO in particular is known to be incompatible as a metadata backend.
</Warning>

## Configuration Examples

### Replicated (Default)

This is the default configuration for multi-node clusters. Each node needs to know about all metadata server addresses.

```toml restate.toml theme={null}
# Node configuration - each node needs these settings
node-name = "node-1"
cluster-name = "my-cluster"

# Enable the metadata-server role on this node
roles = [
    "metadata-server",
    "worker",
    "log-server",
    "admin",
    "http-ingress"
]

# List all metadata server nodes (all nodes need this)
[metadata-client]
type = "replicated"
addresses = [
    "http://node-1:5122/",
    "http://node-2:5122/",
    "http://node-3:5122/"
]
```

<Info>
  For fault tolerance, run metadata-server on an odd number of nodes (typically 3 or 5). To tolerate `n` node failures, you need at least `2n + 1` metadata server nodes.
</Info>

### Etcd

Configure Restate to use an external etcd cluster:

```toml restate.toml theme={null}
# No metadata-server role needed
roles = [
    "worker",
    "log-server",
    "admin",
    "http-ingress"
]

[metadata-client]
type = "etcd"
# Etcd cluster addresses (host:port format)
addresses = [
    "etcd-1.example.com:2379",
    "etcd-2.example.com:2379",
    "etcd-3.example.com:2379"
]

# Optional: Adjust timeouts if needed
connect-timeout = "5s"
keep-alive-interval = "10s"
```

### Object Store (Amazon S3)

Configure Restate to use Amazon S3 for metadata storage:

```toml restate.toml theme={null}
# No metadata-server role needed
roles = [
    "worker",
    "log-server",
    "admin",
    "http-ingress"
]

[metadata-client]
type = "object-store"
# S3 bucket and optional prefix
path = "s3://restate-metadata-bucket/cluster-prod"

# Authentication options
aws-profile = "my-profile"
# Or use explicit credentials:
# aws-access-key-id = "..."
# aws-secret-access-key = "..."
# aws-session-token = "..."

aws-region = "us-east-1"

# Optional: For testing with local S3-compatible storage
# aws-endpoint-url = "http://localhost:9000"
# aws-allow-http = true
```

<Warning>
  Object store metadata requires low-latency access to S3. Deploy your Restate cluster in the same AWS region as your S3 bucket to minimize latency impact on cluster operations.
</Warning>

## Migrating Between Providers

Restate supports migrating metadata from the replicated provider to external providers (etcd, or object-store) using the `restatectl metadata migrate` command. This allows you to change metadata providers without losing cluster state.

<Warning>
  **Important limitations:**

  * Migration only supports moving FROM replicated TO external providers (not the reverse)
  * Migration requires all cluster nodes to be restarted in migration mode
  * Plan for a maintenance window as the cluster will not process invocations during migration
</Warning>

### Migration Process

#### Prerequisites

Before starting the migration:

1. **Target metadata store must be accessible and ready:**
   * Etcd: Cluster running and accessible
   * Object Store: S3 bucket created and accessible

2. **Prepare target configuration:**
   * Have your target `[metadata-client]` configuration ready in TOML format
   * Test connectivity from Restate nodes to the target store

3. **Plan for downtime:**
   * All nodes will need to be restarted in migration mode
   * Invocation processing will be paused during migration

#### Step-by-Step Migration

<Steps>
  <Step title="Prepare target metadata store">
    For Etcd:

    ```bash theme={null}
    # Ensure your etcd cluster is running and accessible
    etcdctl endpoint health
    ```

    For Object Store:

    ```bash theme={null}
    # Ensure S3 bucket exists and is accessible
    aws s3 ls s3://restate-metadata-bucket/
    ```
  </Step>

  <Step title="Restart all nodes in migration mode">
    Stop all cluster nodes and restart them with the `--metadata-migration-mode` flag:

    ```bash theme={null}
    restate --metadata-migration-mode --config-file restate.toml
    ```

    <Info>
      In migration mode, only the Admin and MetadataServer roles are active. The cluster will not process invocations or serve requests until migration is complete and nodes are restarted normally.
    </Info>
  </Step>

  <Step title="Create target configuration file">
    Create a TOML file (e.g., `target-metadata.toml`) with your target metadata client configuration:

    **For Etcd:**

    ```toml target-metadata.toml theme={null}
    [metadata-client]
    type = "etcd"
    addresses = [
        "etcd-1.example.com:2379",
        "etcd-2.example.com:2379",
        "etcd-3.example.com:2379"
    ]
    ```

    **For Object Store:**

    ```toml target-metadata.toml theme={null}
    [metadata-client]
    type = "object-store"
    path = "s3://restate-metadata-bucket/cluster-prod"
    aws-profile = "my-profile"
    aws-region = "us-east-1"
    ```
  </Step>

  <Step title="Run migration command">
    Execute the migration using `restatectl`:

    ```bash theme={null}
    restatectl metadata migrate --toml target-metadata.toml
    ```

    By default, the migration will fail if any keys already exist in the target store. To override existing keys:

    ```bash theme={null}
    restatectl metadata migrate --toml target-metadata.toml --force
    ```

    <Warning>
      Use `--force` with caution. It will overwrite any existing metadata in the target store, which could cause issues if the target is shared with other clusters.
    </Warning>

    The migration command will:

    1. Read all metadata from the replicated store
    2. Write it to the target store
    3. Verify the migration succeeded
    4. Display next steps

    Expected output:

    ```
    ✅ Metadata migration completed

    1. Please make sure to update all nodes config with
       [metadata-client]
       type = "etcd"
       addresses = [
           "etcd-1.example.com:2379",
           "etcd-2.example.com:2379",
           "etcd-3.example.com:2379"
       ]

    2. Restart all nodes without the --metadata-migration-mode
    ```
  </Step>

  <Step title="Update node configurations">
    Update the `[metadata-client]` section in all node configuration files with the target configuration:

    ```toml restate.toml theme={null}
    # Remove or comment out metadata-server role if present
    roles = [
        "worker",
        "log-server",
        "admin",
        "http-ingress"
    ]

    # Replace [metadata-client] section with target configuration
    [metadata-client]
    type = "etcd"
    addresses = [
        "etcd-1.example.com:2379",
        "etcd-2.example.com:2379",
        "etcd-3.example.com:2379"
    ]
    ```

    Also remove or comment out the `[metadata-server]` section if present, as it's no longer needed with external metadata providers.
  </Step>

  <Step title="Restart all nodes normally">
    Stop all nodes running in migration mode and restart them with their updated configurations:

    ```bash theme={null}
    restate --config-file restate.toml
    ```

    Verify the cluster is healthy:

    ```bash theme={null}
    restatectl status
    restatectl nodes list
    ```
  </Step>
</Steps>

### Troubleshooting Migration

**Migration fails with "target keys exist":**

* The target store already contains metadata keys
* Either use `--force` to override, or clean the target store first
* Verify you're not accidentally sharing a metadata store with another cluster

**Migration command times out:**

* Check that nodes are actually running in migration mode
* Verify network connectivity to both source and target stores
* Check Restate server logs for errors

**Cluster won't start after migration:**

* Verify all nodes have the updated `[metadata-client]` configuration
* Verify etcd cluster is accessible if using etcd
* Check server logs for connection errors

## Related Documentation

* [Cluster Configuration](/server/clusters): Understanding Restate cluster deployments
* [Architecture Reference](/references/architecture#metadata-store): Deep dive into metadata architecture
* [Server Configuration](/server/configuration): Complete server configuration guide
* [Configuration Reference](/references/server-config#metadata-client): Detailed metadata client options
* [AWS Deployment Guide](/server/deploy/aws): Deploying Restate on AWS
* [Kubernetes Deployment](/server/deploy/kubernetes): Deploying Restate on Kubernetes


# Logging
Source: https://docs.restate.dev/server/monitoring/logging

Configure logging for Restate Server.

By default, Restate logs INFO, WARN and ERROR events using a pretty format.

## Log filter

You can modify the filter used for log events setting the [configuration entry](/server/configuration) `log-filter`, or alternatively setting the `RUST_LOG` environment variable. Check the [`RUST_LOG` documentation](https://docs.rs/tracing-subscriber/latest/tracing_subscriber/filter/struct.EnvFilter.html) for more details about the filter format. For example to enable debug logs on all Restate components, and info on other logs, set the filter as `info,restate=debug`. See the paragraph below [components and log events](#components-and-log-event-context-fields) for more details about filter targets.

## Log format

You can modify the log format by setting the [configuration entry](/server/configuration) `log-format` (or via `restate-server --log-format=<FORMAT>`) as follows:

* `pretty`: Very verbose pretty format with rendered error descriptions, when available
* `compact`: Compact single line format
* `json`: Newline delimited json format

## Components and log event context fields

The following components are producing relevant logs:

* `restate_ingress_http`: The Restate component ingesting HTTP requests
* `restate_admin`: The component responsible for holding the metadata information and executing service discovery
* `restate_invoker`: The component interacting with deployed service deployments
* `restate_worker::partition::state_machine`: The state machine effects
* `restate_partition_store`: Restate partition storage layer
* `restate_bifrost`: Restate durable log layer
* `hyper`: The HTTP library

Most log events generated by Restate will also have the following attached attributes:

* `rpc.service`: The related service
* `rpc.method`: The related method
* `restate.invocation.id`: The [invocation identifier](/services/invocation/http#invocation-identifier)

## Recommendations

When testing Restate locally, we recommend keeping the default configuration.

When deploying in production, we recommend setting the log level to `info` and enabling the `json` format in conjunction with a log collector, so you can later inspect and filter logs based on its event fields.

We recommend to set up logging to a more verbose filter, and use the `pretty` format. Some example filters:

* `info,restate_ingress_http=trace,restate_invoker=trace,restate=debug,hyper=debug` for network related issues
* `info,restate_worker::partition::effects=debug` to get insights on the state machine effects
* `info,restate_admin=trace` to check service discovery and registration


# Metrics
Source: https://docs.restate.dev/server/monitoring/metrics

Expose Restate Server Prometheus metrics.

Restate servers expose operational metrics in [Prometheus exposition format](https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md) via the NodeCtl(port `5122`) endpoint, i.e. `localhost:5122/metrics`. For instance, configure Prometheus to scrape this endpoint every 30 seconds by adding this section to Prometheus configuration (assuming Restate server's IP address is `10.10.10.1` and accessible by Prometheus:

```yml theme={null}
scrape_configs:
- job_name: restate_server_1
  metrics_path: "/metrics"
  static_configs:
  - targets:
    - 10.10.10.1:5122
```

Note that some metrics are dependent on the value of `rocksdb-statistics-level` in the configuration file. In most cases, the default value will be sufficient for production deployment monitoring.

## Grafana Dashboards

Restate provides two pre-built Grafana dashboards for monitoring your cluster. You can import them directly from [Grafana.com](https://grafana.com/grafana/dashboards/):

| Dashboard              | Grafana ID                                            | Description                                          |
| ---------------------- | ----------------------------------------------------- | ---------------------------------------------------- |
| **Restate: Overview**  | [24747](https://grafana.com/grafana/dashboards/24747) | High-level cluster health, resources, and throughput |
| **Restate: Internals** | [24748](https://grafana.com/grafana/dashboards/24748) | Deep-dive into Bifrost, Invoker, RocksDB, and more   |

To import a dashboard:

1. Open Grafana and go to **Dashboards** > **Import**
2. Enter the dashboard ID (24747 or 24748) and click **Load**
3. Select your Prometheus datasource
4. Click **Import**

Import both dashboards to enable navigation links between them.

### Overview Dashboard

<img />

### Internals Dashboard

<img />

## Example Metrics

This is a non-exhaustive list of metrics that can be used to measure system performance:

* `restate_ingress_requests_total` (counter) - Number of ingress requests in different states (admitted, completed, throttled, etc.)
* `restate_ingress_request_duration_seconds` (summary) - Total latency of Ingress request processing in seconds
* `restate_rocksdb_estimate_live_data_size_bytes` (Gauge) - Size of the live data in RocksDb databases in bytes
* `restate_invoker_invocation_task_total` (counter) - The number of invocation tasks to user handlers

For example, we can use the following Prometheus queries to visualize throughput (ops/s) of HTTP ingress requests with an overlay of P99 latency:

```javascript theme={null}
rate(restate_ingress_requests_total{cluster_name="localcluster"}[$__rate_interval])
```

```javascript theme={null}
restate_ingress_request_duration_seconds{cluster_name="localcluster", quantile="0.99"}
```

<img />


# Tracing
Source: https://docs.restate.dev/server/monitoring/tracing

Export OTEL traces of your invocations.

Restate supports the following tracing features:

* Runtime execution tracing per invocation
* Exporting traces to OTLP-compatible systems (e.g. Jaeger)
* Correlating parent traces of incoming HTTP requests, using the [W3C TraceContext](https://github.com/w3c/trace-context) specification.

## Setting up OTLP exporter

Set up the [OTLP exporter](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/otlpexporter/README.md) by pointing the configuration entry `tracing-endpoint` to your trace collector.

By default, a `tracing-endpoint` using the `http://` or `https://` scheme will emit trace data in the OTLP/gRPC format.
Restate also supports the `otlp+http://` and `otlp+https://` schemes, to emit trace data in the OTLP/HTTP format.
Note that when using OTLP/HTTP, the `tracing-endpoint` URI usually needs to include an e.g. `/v1/traces` path.

### Exporting traces to Jaeger

[Jaeger](https://www.jaegertracing.io/docs/2.4/deployment/) accepts OTLP trace data on port `4317` (gRPC) and `4318` (HTTP).
Start Jaeger locally with Docker, for example:

```shell theme={null}
docker run -d --name jaeger \
    -p 4317:4317 -p 4318:4318 -p 16686:16686 \
    jaegertracing/jaeger:2.4.0
```

Configure the tracing endpoint in Restate as a valid URL:

```shell theme={null}
restate-server --tracing-endpoint http://localhost:4317 # for gRPC
restate-server --tracing-endpoint otlp+http://localhost:4318/v1/traces # for HTTP (note /v1/traces)
```

If you run Restate in Docker, then instead add the environment variable `-e RESTATE_TRACING_ENDPOINT=http://host.docker.internal:4317`.

If you now spin up your services and send requests to them, you will see the traces appear in the Jaeger UI at [http://localhost:16686](http://localhost:16686)

<img />

<AccordionGroup>
  <Accordion title="Specifying additional tracing headers">
    You can specify additional headers to be sent with the trace data by setting the `tracing-headers` configuration entry.
    For example, to specify an `authorization` header add the following snippet to the [configuration file](/server/configuration/#configuration-file):

    ```toml theme={null}
    [tracing-headers]
    authorization = "Bearer some-auth-token"
    ```
  </Accordion>

  <Accordion title="Filtering traces">
    You can configure a span/event filter in a similar fashion to the [log filter](/server/monitoring/logging#log-filter) setting the `tracing-filter` configuration entry.
  </Accordion>

  <Accordion title="Setting up Jaeger file exporter">
    If you can't configure a Jaeger agent, you can export traces by writing them to files, using the Jaeger JSON format.
    To do so, set up the configuration entry `tracing-json-path` pointing towards the path where trace files should be written.

    You can import the trace files using the Jaeger UI:

    <img />
  </Accordion>
</AccordionGroup>

## Understanding traces

The traces contain detailed information about the context calls that were done during the invocation (e.g. sleep, one-way calls, interaction with state):

<img />

The initial `ingress_invoke` spans show when the HTTP request was received by Restate. The `invoke` span beneath it shows when Restate invoked the service deployment to process the request.

The tags of the spans contain the metadata of the context calls (e.g. call arguments, invocation id).

<Info title="One-way call traces are detached from the parent">
  When a service invokes another service, the child invocation is linked automatically to the parent invocation, as you can see in the image.
  Note that the spans of one-way calls are shown as separate traces. The parent invocation only shows that the one-way call was scheduled, not its entire tracing span.
  To see this information, search for the trace of the one-way call by filtering on the invocation id tag `restate.invocation.id="inv_19maBIcE9uRD0gIu30mu6eqhZ4pQT"`.
</Info>

## Searching traces

Traces export attributes and tags that correlate the trace with the service and/or invocation. For example, in the Jaeger UI, you can filter on the invocation id (`restate.invocation.id`) or any other tag:

<img />


# Networking
Source: https://docs.restate.dev/server/networking

Configure network ports, addresses, and listeners for Restate Server

This page covers how to configure network listeners, ports, and advertised addresses for Restate Server. For guidance on securing these ports in production, see [Server Security](/server/security).

## Overview

Restate Server exposes several network services:

| Service     | Default Port | Description                                             |
| ----------- | ------------ | ------------------------------------------------------- |
| **Ingress** | 8080         | HTTP API for invoking services and managing invocations |
| **Admin**   | 9070         | Admin API for registering deployments and introspection |
| **Fabric**  | 5122         | Node-to-node communication for clustered deployments    |

By default, Restate listens on both TCP sockets and Unix domain sockets for each service.

## Listen Modes

Restate supports three listen modes controlled by the `listen-mode` configuration option:

| Mode   | Description                                        |
| ------ | -------------------------------------------------- |
| `all`  | **\[Default]** Listen on both TCP and Unix sockets |
| `tcp`  | Only listen on TCP sockets                         |
| `unix` | Only listen on Unix domain sockets                 |

```toml restate.toml theme={null}
# Listen on TCP only (useful if Unix sockets are not needed)
listen-mode = "tcp"
```

Or via environment variable:

```bash theme={null}
RESTATE_LISTEN_MODE=tcp
```

### Unix Domain Sockets

When Unix sockets are enabled (the default), Restate creates socket files under the data directory:

| Service | Socket Path                             |
| ------- | --------------------------------------- |
| Ingress | `restate-data/<node-name>/ingress.sock` |
| Admin   | `restate-data/<node-name>/admin.sock`   |
| Fabric  | `restate-data/<node-name>/fabric.sock`  |

Unix sockets are automatically cleaned up on shutdown. They are useful for:

* Local development without port conflicts
* Tools like `curl` that support `--unix-socket`
* Secure local communication without exposing network ports

**Example using Unix sockets with curl:**

```bash theme={null}
curl --unix-socket restate-data/node-1/ingress.sock http://local/MyService/myHandler --json '{}'
```

**Example using Unix sockets with restatectl:**

```bash theme={null}
restatectl -s unix:restate-data/node-1/admin.sock status
```

## Cascading Configuration

Networking configuration follows a cascading model where global options provide defaults that can be overridden per-service. The resolution order is:

1. **Global options** (top-level in config file) apply to all services
2. **Per-service options** (in `[admin]`, `[ingress]` sections) override global options for that specific service
3. **Environment variables** override config file values
4. **Command-line arguments** have the highest precedence

For example, if you set `bind-ip = "192.168.1.10"` at the top level, all services will bind to that IP unless a specific service overrides it with its own `bind-address`.

The following options cascade from global to per-service:

* `use-random-ports`
* `listen-mode`
* `bind-ip`
* `advertised-host`

The following options have per-service defaults and can be overridden individually:

* `bind-port` (each service has its own default port)
* `advertised-address` (auto-generated per service, but can be explicitly set for full control)

Additionally, `networking.message-size-limit` cascades to component-specific limits. See [Message Size Limits](#message-size-limits) for details.

## Configuring Ports and Addresses

### Global Options

These options apply to all services unless overridden per-service:

| Option             | Default       | Description                                    |
| ------------------ | ------------- | ---------------------------------------------- |
| `bind-ip`          | `0.0.0.0`     | Local interface IP to bind on                  |
| `bind-port`        | `5122`        | Port for the fabric service                    |
| `use-random-ports` | `false`       | Use random available ports instead of defaults |
| `advertised-host`  | Auto-detected | Hostname to advertise to peers                 |

```toml restate.toml theme={null}
# Bind all services to a specific interface
bind-ip = "192.168.1.10"

# Or use random ports (useful for testing)
use-random-ports = true
```

### Per-Service Configuration

Each service can override the global settings. Use `bind-port` to change just the port while keeping the default bind IP (`0.0.0.0`):

```toml restate.toml theme={null}
# Fabric service (node-to-node) - uses global bind-ip with custom port
bind-port = 5122

[admin]
# Override just the port
bind-port = 9070

[ingress]
# Override just the port
bind-port = 8080
```

Or use `bind-address` for full control over both IP and port:

```toml restate.toml theme={null}
[admin]
bind-address = "192.168.1.10:9070"

[ingress]
bind-address = "192.168.1.10:8080"
```

### Environment Variable Overrides

All networking options can be set via environment variables:

| Config Option        | Environment Variable         |
| -------------------- | ---------------------------- |
| `listen-mode`        | `RESTATE_LISTEN_MODE`        |
| `use-random-ports`   | `RESTATE_USE_RANDOM_PORTS`   |
| `bind-ip`            | `RESTATE_BIND_IP`            |
| `bind-port`          | `RESTATE_BIND_PORT`          |
| `bind-address`       | `RESTATE_BIND_ADDRESS`       |
| `advertised-host`    | `RESTATE_ADVERTISED_HOST`    |
| `advertised-address` | `RESTATE_ADVERTISED_ADDRESS` |

Per-service overrides use double underscores:

```bash theme={null}
RESTATE_ADMIN__BIND_PORT=9070
RESTATE_INGRESS__BIND_PORT=8080
```

## Advertised Addresses

Advertised addresses are the URLs that Restate publishes for others to connect to its services. Each service has its own advertised address that is **automatically generated** based on the bound address and detected network configuration. You can optionally set these explicitly if you need full control.

<Note>
  Restate does **not** validate the reachability of advertised addresses. It trusts that the configured addresses are correct and reachable by the intended clients or peers.
</Note>

### Fabric Advertised Address

The **fabric advertised address** (configured at the top level) is the network address that cluster nodes use to communicate with each other. This is the most critical advertised address and requires careful consideration:

* **Must be resolvable and reachable** from all other nodes in the cluster
* **Should not be behind a proxy** to avoid performance issues or connectivity problems
* **Should be unique** within the cluster

<Warning>
  We recommend using **IP addresses** rather than DNS names for the fabric advertised address. This avoids DNS resolution latency, potential caching issues, and DNS-related errors during node-to-node communication. DNS names are supported but not recommended for production clusters.
</Warning>

```toml restate.toml theme={null}
# Recommended: Use IP address for cluster communication
advertised-address = "http://10.0.1.15:5122/"
```

### Ingress Advertised Address

The **ingress advertised address** is shared with:

* The **Admin UI** (Web UI service playground)
* The **Restate CLI** (`restate` command)

This address needs to be accessible by users who invoke services through Restate. For example, when users run `restate whoami`, this is the address they will see for the ingress endpoint.

Unlike the fabric address, it's perfectly fine to expose the ingress behind an HTTPS proxy or load balancer on a different host:

```toml restate.toml theme={null}
[ingress]
# Ingress exposed behind an HTTPS proxy
advertised-address = "https://my-ingress.example.com/"
```

### Admin Advertised Address

The **admin advertised address** is used by:

* The **Restate CLI** for administrative operations
* The **Admin UI** for deployment management

This is the address shown when users run `restate whoami` for the admin endpoint. Like the ingress address, it can be behind a proxy if needed:

```toml restate.toml theme={null}
[admin]
# Admin API behind an HTTPS proxy
advertised-address = "https://admin.example.com/"
```

### Auto-Detection

Restate automatically generates advertised addresses for all services based on:

1. The listen mode (TCP vs Unix)
2. The bound address
3. The publicly routable IP address of the host (auto-detected)

For most deployments, especially single-node setups, you don't need to configure advertised addresses explicitly.

### When to Set Explicitly

You may want to explicitly set advertised addresses when:

* Running ingress or admin behind a load balancer or reverse proxy
* In containerized environments where auto-detection doesn't work correctly
* Using NAT with specific port mappings
* You want users to connect via a specific hostname or domain

**Using `advertised-host` (recommended for simple cases):**

```toml restate.toml theme={null}
# Restate will construct full URLs using this hostname with the correct ports
advertised-host = "restate.mycompany.internal"
```

This sets the hostname for all services. Restate will construct URLs like:

* Fabric: `http://restate.mycompany.internal:5122/`
* Admin: `http://restate.mycompany.internal:9070/`
* Ingress: `http://restate.mycompany.internal:8080/`

**Using per-service `advertised-address` for full control:**

```toml restate.toml theme={null}
# Cluster communication - use IP address, no proxy
advertised-address = "http://10.0.1.15:5122/"

[admin]
# Admin API behind HTTPS proxy
advertised-address = "https://admin.example.com/"

[ingress]
# Ingress behind HTTPS proxy on different host
advertised-address = "https://api.example.com/"
```

### Docker and Kubernetes

In Docker Compose or Kubernetes, set the fabric advertised address to the internal hostname that other containers/pods can resolve:

```yaml theme={null}
environment:
  # Use the container/service name for cluster communication
  RESTATE_ADVERTISED_ADDRESS: "http://restate-node-1:5122"
```

This ensures nodes can reach each other using internal DNS names provided by Docker or Kubernetes. For ingress and admin, you may want to set separate advertised addresses pointing to your external load balancer or ingress controller.

## Random Ports

For testing or running multiple instances on the same host, enable random port selection:

```bash theme={null}
restate-server --use-random-ports=true
```

Or via environment variable:

```bash theme={null}
RESTATE_USE_RANDOM_PORTS=true restate-server
```

When using random ports:

* Restate prints the actual bound addresses on startup
* Unix sockets remain at predictable paths, so tools can connect via Unix socket while ports are dynamic
* Use `restate-server --no-logo` to have the addresses printed first on stdout (useful for scripting)

## Common Configurations

### Local Development (Default)

No configuration needed. Restate listens on:

* TCP: `0.0.0.0:8080` (ingress), `0.0.0.0:9070` (admin), `0.0.0.0:5122` (fabric)
* Unix: `restate-data/<node-name>/*.sock`

### Multi-Node Cluster

```toml restate.toml theme={null}
node-name = "node-1"
cluster-name = "production"
# We don't want this node to provision its own cluster
auto-provision = false

# Use IP address for reliable cluster communication
# You don't need to specify this if this IP address is the public routable IP address of the host (it'll be auto-detected)
advertised-address = "http://10.0.1.15:5122/"

[metadata-client]
# Point to at least one peer running metadata-server role
addresses = ["http://10.0.1.16:5122"]
```

### Running Multiple Instances on Same Host

When running multiple Restate instances on the same host, each instance needs unique ports and a separate data directory:

```toml restate-1.toml theme={null}
# Instance 1
node-name = "node-1"
cluster-name = "test-cluster"
```

```toml restate-2.toml theme={null}
# Instance 2
node-name = "node-2"
cluster-name = "test-cluster"
# We don't want this node to provision its own cluster
auto-provision = false

bind-port = 5123

[admin]
bind-port = 9071

[ingress]
bind-port = 8081


# Important to join the same cluster as node-1. Connect to node-1's metadata-server to join the cluster.
[metadata-client]
addresses = ["http://127.0.0.1:5122"]
```

Start each instance with its config file. It's best to run each instance in its own terminal window to avoid confusion:

First node:

```bash theme={null}
restate-server -c restate-1.toml
```

The second node:

```bash theme={null}
restate-server -c restate-2.toml &
```

### TCP Only (No Unix Sockets)

```toml restate.toml theme={null}
listen-mode = "tcp"
```

## CLI Options

Networking options can also be set via command-line flags:

```bash theme={null}
restate-server \
  --listen-mode=tcp \
  --bind-ip=0.0.0.0 \
  --bind-port=5122 \
  --advertised-host=my-host.example.com
```

Use `restate-server --help` for the full list of CLI options.

## Message Size Limits

Restate enforces message size limits to prevent oversized messages from causing issues during network transmission and replication. The limits follow a cascading configuration model similar to other networking options.

### Global Limit

The primary configuration option is `networking.message-size-limit`, which controls the maximum size of messages transmitted over the cluster internal network:

| Option                          | Default  | Description                                            |
| ------------------------------- | -------- | ------------------------------------------------------ |
| `networking.message-size-limit` | `32 MiB` | Maximum size of network messages between cluster nodes |

```toml restate.toml theme={null}
[networking]
# Increase the message size limit to 64 MiB
message-size-limit = "64 MiB"
```

Or via environment variable:

```bash theme={null}
RESTATE_NETWORKING__MESSAGE_SIZE_LIMIT="64 MiB"
```

### Cascading to Component Limits

Several components have their own message size limits that cascade from the global `networking.message-size-limit`:

| Component           | Config Option                        | Description                                    |
| ------------------- | ------------------------------------ | ---------------------------------------------- |
| **Invoker**         | `worker.invoker.message-size-limit`  | Maximum size of journal messages from services |
| **Metadata Client** | `metadata-client.message-size-limit` | Maximum size of metadata messages              |

These component-specific limits follow cascading rules:

* **If not set**: Defaults to `networking.message-size-limit`
* **If set explicitly**: Clamped to not exceed `networking.message-size-limit`

This ensures that no component can send messages larger than what the network layer can handle.

### Example: Increasing Limits

To allow larger Virtual Object state values or journal entries, increase the global limit:

```toml restate.toml theme={null}
[networking]
# Allow messages up to 64 MiB
message-size-limit = "64 MiB"
```

The invoker and other components will automatically inherit this limit. If you want a component to use a smaller limit than the global setting, you can set it explicitly:

```toml restate.toml theme={null}
[networking]
message-size-limit = "64 MiB"

[worker.invoker]
# Invoker uses a smaller limit than global
message-size-limit = "32 MiB"
```

<Note>
  Increasing message size limits affects memory usage and network bandwidth. Only increase these limits if your workload requires larger payloads, and ensure your infrastructure can handle the increased resource requirements.
</Note>

## Deprecated Options

The following options are deprecated and will be removed in a future release:

| Deprecated Option                       | Replacement                    |
| --------------------------------------- | ------------------------------ |
| `[admin] advertised-admin-endpoint`     | `[admin] advertised-address`   |
| `[ingress] advertised-ingress-endpoint` | `[ingress] advertised-address` |


# Self-hosted Restate Overview
Source: https://docs.restate.dev/server/overview

Learn how to deploy and operate a self-hosted Restate server.

Restate is distributed as a single binary that implements all features required to run a single- or multi-node cluster, making it easy to run it on a variety of platforms, from your local machine, to a VM, to Kubernetes.

This page gives an overview of the different deployment strategies and requirements.

<Card icon={"cloud"} href={"/cloud/getting-started"} title="The easiest way to run Restate is with Restate Cloud.">
  Keep deploying your code the same way you do now.
  Restate Cloud handles resilience, observability, and scalability for your code, with no additional infrastructure to manage beyond your services.
  Pair Restate Cloud with a serverless compute provider for the simplest operational experience.
</Card>

## Pre-requisites

You can deploy Restate on any platform that meets the following requirements:

1. **Compute**: a runtime environment where you can run the Restate Docker container or binary.
2. A **persistent volume** attached to store Restate's state. Recommended for cluster deployments. Required for single-node deployments.
3. (For clustered setups) An **object store** to store snapshots.

Restate can run as a single node or as a cluster. See [the trade-offs below](/server/overview#single-node-vs-cluster-deployments).

## Compute

The easiest way to deploy and manage a Restate cluster and containerized Restate services is via the [Kubernetes operator](/server/deploy/kubernetes), as it provides the simplest experience for deploying Restate clusters, deploying services while managing versioning, and more.
Therefore, **we recommend using the Restate Kubernetes Operator in combination with a managed Kubernetes provider such as AWS EKS, Azure's AKS, or Google's GKE.**

This approach is great even for single-node Restate deployments because it makes managing the Restate node much simpler, handling node restarts or Restate version updates with ease. Kubernetes is responsible for keeping the Restate container running and, crucially, makes sure that the persistent volume attachment follows it around if it moves between nodes.

## Persistent volumes

Restate is a stateful system that persists data to disk via its embedded state store.
Restate stores its state by default in the `restate-data` directory in the location where the Restate binary gets executed.

To ensure durability, it is essential to use a persistent volume to back this directory.

If you run Restate as a single node, then the durability of the disk defines how durable Restate is.
Therefore, having a persistent volume is required for single-node deployments.
Additionally, you might want to take periodic backups of the persistent volume (see [data backups](/server/snapshots#data-backups-2)).

In clustered deployments, Restate nodes can failover to other nodes and restore from snapshots stored in an object store.
But each Restate node still benefits from having a persistent volume to store its local state during normal operation.
This makes restarts and failover much faster, without the need to reconstruct partition state from object store or other nodes.

<Accordion title="What does Restate store?">
  Restate has a layered architecture with the following stateful components:

  <Frame>
    <img alt={"Restate Node internal architecture"} />
  </Frame>

  * [The metadata store](/references/architecture#metadata-store): Restate's control plane includes a built-in Raft-based metadata store. This store keeps track of the cluster configuration, partition assignments, and other critical metadata. It is essential to the correctness of the overall system.
  * [The durable log](/references/architecture#durability-and-storage-model): The log is the primary durability layer. New events (invocations, journal entries, state updates, ...) are first persisted in the log, and then move to the partition processors. The log is stored on disk and replicated. Durability is critical here.
  * [The partition processor stores](/references/architecture#materializing-state-for-fast-access): The partition processor leader maintains a full cache of the partition’s materialized state in RocksDB for fast reads and updates during execution. This cache can always be rebuilt from the log, so durability is not critical here. In cluster deployments, the content of RocksDB should be periodically snapshotted to an object store, for faster failover.

  For more information, check out the [architecture reference](/references/architecture).
</Accordion>

## Object storage for snapshots

Snapshotting Restate partition processors is necessary when running multi-node clusters. Partition snapshots are required to enable log trimming and avoid long catch-up delays when handing over partitions between nodes.

Restate supports snapshotting to object stores because they offer great scalability, durability, and cost and are available in all cloud providers and most on-prem setups.

Performance requirements for snapshot storage are minimal. Snapshots typically occur only a few times per hour and are read rarely, so the snapshot destination's performance doesn't affect cluster performance.

See the [snapshot documentation](/server/snapshots#snapshots-2) for more information.

<Note>
  It is also possible to use an object store for metadata storage, removing the need to run the stateful metadata-server role on some or all cluster nodes. However, this feature is only tested and supported on Amazon S3 at present, and it requires nodes to have low-latency access to the object storage service. For detailed information about choosing and configuring metadata providers, see [Metadata Storage](/server/metadata).
</Note>

### Supported Object Stores

Currently we support the following object stores:

* [AWS S3](/server/deploy/aws#using-s3)
* [MinIO](/server/deploy/kubernetes#minio-object-store) (only for snapshots, not metadata)
* Coming soon: Google Cloud Storage and Azure Blob Storage (Contact us for more info: [Discord](https://discord.restate.dev)/[Slack](https://slack.restate.dev))

## Single-Node vs. Cluster Deployments

Restate can be deployed as a single-node setup or as a distributed cluster, depending on your requirements for availability and scalability.

| Requirement            | Single-Node                    | Cluster |
| ---------------------- | ------------------------------ | ------- |
| **Durability**         | Good (when using durable disk) | Better  |
| **High Availability**  | ❌                              | ✅       |
| **Geo-replication**    | ❌                              | ✅       |
| **Horizontal Scaling** | ❌                              | ✅       |
| **Vertical Scaling**   | ✅                              | ✅       |

### When to Choose Single-Node

Single-node Restate runs as a single binary that persists data to disk. To ensure durability, it is essential to use a persistent volume for storing all Restate data.

Despite being a single-node setup, this setup does not sacrifice durability. Restate fsyncs committed data to disk before acknowledging network messages, providing the **highest level of durability**.
The durability of the underlying disk storage determines how durable Restate is.

In case of a node or Restate process crash, you won't lose data, but you will experience a short window of unavailability while the process restarts.

**Use this for**: development, testing, or production workloads which can tolerate short downtimes and do not require high throughput (horizontal scaling).

### When to use Cluster Deployments

Restate cluster deployments serve multiple purposes:

* **High-availability**: In case of a crash, other nodes of the cluster can take over immediately, rather than waiting on the failed node to get rescheduled.
* **Geo-replication**: To reduce the risk of downtime even further, Restate clusters can run in a geo-replicated setup, with nodes deployed across multiple availability zones or even regions. Depending on your architecture, such a deployment allows you to remain available through outages affecting entire zones and even regions.
* **Scalability** by spreading load over multiple nodes.
* Even **better durability**, by persisting copies of the data across multiple zones / regions.

**Use this for**: use cases requiring high availability, very high throughput, or geo-replication.

## Platform-Specific Deployment Guides

Follow the links below for platform-specific deployment instructions:

* [Kubernetes](/server/deploy/kubernetes): Recommended for production deployments, in combination with a managed Kubernetes provider from your cloud provider.
* Cloud Providers:
  * [AWS](/server/deploy/aws)
  * [GCP](/server/deploy/gcp)
  * [Azure](/server/deploy/azure)
* [Docker](/server/deploy/docker): For local development and testing. More involved to run in production.


# Security
Source: https://docs.restate.dev/server/security

Securing your self-hosted Restate Server deployment

This page covers the security model for self-hosted Restate Server deployments and the operational measures you should take to protect your infrastructure.

For securing communication between Restate and your service deployments, see [Service Security](/services/security).

<Info>
  [Restate Cloud](/cloud/getting-started) provides built-in authentication, access control, and header filtering out of the box. The guidance on this page applies to self-hosted deployments where you manage these layers yourself.
</Info>

## Security model

Restate Server exposes three network services, each with different security considerations:

| Service     | Default Port | Audience                                            | Security posture                                              |
| ----------- | ------------ | --------------------------------------------------- | ------------------------------------------------------------- |
| **Ingress** | 8080         | External callers invoking services                  | Should sit behind a reverse proxy that handles authentication |
| **Admin**   | 9070         | Operators managing deployments and inspecting state | Must be restricted to trusted operators via network controls  |
| **Fabric**  | 5122         | Cluster-internal node-to-node communication         | Must not be exposed outside the cluster                       |

Restate Server is an infrastructure component — like etcd, Consul, or a database. It does not include application-level authentication on its management interfaces, by design. You are expected to secure access to these ports using the network and proxy layers available in your deployment environment.

## Securing the ingress port

The ingress port (default 8080) is the entry point for service invocations. In production, place a reverse proxy or API gateway in front of it to handle:

* **Authentication** — validate caller identity (bearer tokens, API keys, mTLS)
* **Header filtering** — strip infrastructure auth headers before forwarding to Restate

<Warning>
  HTTP headers that reach the ingress port are persisted in Restate's invocation journal for the configured retention period. This is by design — Restate needs the complete original request to replay invocations after failures. Headers that pass through your proxy will be stored and forwarded to the target service deployment.

  Strip sensitive infrastructure headers (e.g., proxy auth tokens) at your reverse proxy **before** they reach Restate. Only headers intended for the downstream service should be allowed through.
</Warning>

### Example: nginx reverse proxy

```nginx nginx.conf theme={null}
server {
    listen 443 ssl;
    server_name api.example.com;

    location / {
        # Authenticate the caller
        auth_request /auth;

        # Strip infrastructure auth headers before forwarding to Restate
        proxy_set_header Authorization "";
        proxy_set_header X-Internal-Token "";

        proxy_pass http://restate:8080;
    }
}
```

## Securing the admin port

The admin port (default 9070) provides full control over the Restate instance: registering deployments, managing invocations, and querying internal state via the [SQL introspection API](/references/sql-introspection).

**Do not expose the admin port to untrusted networks.** Restrict access using:

* **Network policies** — in Kubernetes, use NetworkPolicy resources to limit which pods can reach port 9070
* **Security groups** — in cloud environments, restrict ingress rules to management IPs or bastion hosts
* **Bind address** — bind the admin port to a private interface rather than `0.0.0.0`

```toml restate.toml theme={null}
[admin]
# Bind admin API to localhost only — accessible via SSH tunnel or sidecar
bind-address = "127.0.0.1:9070"
```

Or use Unix domain sockets for local-only admin access:

```toml restate.toml theme={null}
[admin]
listen-mode = "unix"
```

See [Networking](/server/networking) for more configuration options.

## Securing the fabric port

The fabric port (default 5122) carries cluster-internal replication and coordination traffic. Do not expose it outside the cluster — restrict access to cluster members and monitoring infrastructure only.

## Deployment registration

When you register an HTTP service deployment via the admin API, Restate makes an outbound HTTP connection to the provided URI for service discovery. Ensure that:

* Only trusted operators can access the admin API (see [above](#securing-the-admin-port))
* Network-level controls prevent the Restate process from reaching unintended internal endpoints if your environment requires it (e.g., cloud metadata services)

## Request identity

Restate can cryptographically sign requests to your service deployments, allowing your services to verify that requests originate from your Restate instance. This is important when your services are reachable from networks beyond just Restate.

See [Service Security — Locking down service access](/services/security#locking-down-service-access) for setup instructions.

## Summary checklist

* Place a reverse proxy in front of the ingress port that authenticates callers and strips infrastructure auth headers
* Restrict admin port access to trusted operators via network controls or bind to localhost
* Do not expose the fabric port outside the cluster
* Configure [request identity](/services/security#locking-down-service-access) to sign requests to service deployments
* Consider [client-side journal encryption](/services/security#client-side-journal-encryption) for sensitive workloads


# Snapshots & Backups
Source: https://docs.restate.dev/server/snapshots

Understanding and configuring snapshots and data backups in Restate clusters

Restate provides two different mechanisms for data persistence and recovery, each serving distinct purposes: snapshots and backups.

<Info title="Architectural overview">
  To understand the terminology used on this page, it might be helpful to read through the [architecture reference](/references/architecture).
</Info>

## Overview

A Restate cluster maintains three essential types of state:

* **Metadata**: cluster membership as well as log and partition configuration
* **Logs**: The Bifrost log disseminates all events and state changes to partition workers
* **Partition store**: Stores ongoing invocations and their journals, persisted state, timers, deployments and more, for each partition

A disaster recovery and backup strategy must address all three plus the configuration of the cluster nodes.

### Snapshots

Internal mechanism for cluster operations and state sharing between nodes:

* **Goal**: Enable fast bootstrap of new nodes and support [log trimming](/server/snapshots#log-trimming-and-durability) in clusters
* **Scope**: A snapshot of the most recent state of a specific partition, produced by a fully caught up partition processor
* **When**: Essential for multi-node clusters; optional for single-node deployments

### Data Backups

Full copies of all data stored by Restate for disaster recovery:

* **Goal**: Restore a Restate Server to a previous point in time
* **Scope**: Complete copy of the `restate-data` directory or storage volumes
* **When**: Currently only for single-node deployments due to timing coordination challenges

<Accordion title="Multi-node backup challenges">
  Coordinating simultaneous backups across multiple nodes presents significant timing precision requirements. Even millisecond differences in backup timing can result in one node capturing state that's progressed further than another, creating inconsistent snapshots across the cluster. This timing skew leads to data inconsistencies that prevent successful cluster restoration from backup.

  While atomically snapshotting restate-data at the volume level is still very useful as part of a broader disaster recovery and backup strategy, some manual repair work may be required to restore from such backups. There will also be some expected data loss between the latest LSN/point in time captured by the snapshot and the latest accepted/processed transaction by the cluster before it lost availability.

  Since tooling for automated cluster restoration is not yet available, cluster-wide full node snapshots would currently require manual intervention to repair the system back into a workable state.
</Accordion>

## When to Use Each

### Use Snapshots When:

* Operating a multi-node cluster (required)
* Adding or removing nodes from a cluster
* Enabling [log trimming](/server/snapshots#log-trimming-and-durability) to manage storage
* Supporting fast partition processor failover (having warm standbys ready for near-instant takeover)
* Growing the cluster or replacing completely failed nodes (newly added nodes can bootstrap from snapshots)

### Use Backups When:

* Doing point-in-time recovery of a single-node deployment

## Snapshots

[Snapshots](operate/data-backup#snapshots) are essential for multi-node cluster operations, enabling efficient state sharing and log management.

Snapshots are essential to support safe [log trimming](/server/snapshots#log-trimming-and-durability) and fast partition fail-over to a different cluster node. Snapshots are optional for single-node deployments and required for multi-node clusters.

Restate Partition Processors can be configured to periodically publish snapshots of their partition state to a shared object store.

Snapshots serve to allow nodes that do not have an up-to-date local copy of a partition's state to quickly start a processor for the given partition.
Without snapshots, trimming the log could lead to data loss if all the nodes replicating a particular partition are lost. Additionally, starting new partition processors would require the full replay of that partition's log which might take a long time.

When partition processors successfully publish a snapshot, this is reflected in the archived log sequence number (LSN). This value is the safe point up to which Restate can trim the Bifrost log.

### Configuring Automatic Snapshotting

Restate clusters should always be configured with a snapshot repository to allow nodes to efficiently share partition state, and for new nodes to be added to the cluster in the future.
Restate supports Amazon S3 (or S3-compatible stores), Google Cloud Storage, and Azure Blob Storage as shared snapshot repositories.
To set up a snapshot destination, update your [server configuration](/server/configuration) as follows:

```toml theme={null}
[worker.snapshots]
destination = "s3://snapshots-bucket/cluster-prefix"
snapshot-interval-num-records = 100000
snapshot-interval = "60m"
```

When both options are set, snapshots are triggered when *both* conditions are met: the specified time has elapsed *and* at least the minimum number of records have been applied. You can also set only one:

* `snapshot-interval` alone triggers time-based snapshots unconditionally
* `snapshot-interval-num-records` alone triggers snapshots based on record count only

Records correspond to persisted actions in Restate such as receiving an ingress request, appending to an invocation journal, or storing a state key-value pair. As these are variable in size, the exact amount of data accumulated between snapshots will vary depending on the workload. You should tune the snapshot frequency based on the observed partition processor catch-up times and your availability goals.

You can also trigger snapshots manually using [`restatectl`](/installation#advanced%3A-operating-clusters-with-restatectl):

```shell theme={null}
restatectl snapshots create-snapshot --partition-id <PARTITION_ID>
```

We recommend testing the snapshot configuration by requesting a snapshot and examining the contents of the bucket.
You should see a new prefix with each partition's id, and a `latest.json` file pointing to the most recent snapshot.

No additional configuration is required to enable restoring snapshots.
When partition processors first start up, and no local partition state is found, the processor will attempt to restore the latest snapshot from the repository.
This allows for efficient bootstrapping of additional partition workers.

<Info title="Local development with Azurite or fake-gcs-server">
  For local development without cloud access, you can use emulators like [Azurite](https://github.com/Azure/Azurite) for Azure or [fake-gcs-server](https://github.com/fsouza/fake-gcs-server) for GCS. See the Minio example below for S3-compatible local development.
</Info>

### Object Store endpoint and access credentials

Restate supports Amazon S3 (and S3-compatible stores), Google Cloud Storage, and Azure Blob Storage. Object store locations are specified as a URL where the scheme indicates the provider and the authority is the bucket name:

| Provider             | URL Scheme | Example                 |
| -------------------- | ---------- | ----------------------- |
| Amazon S3            | `s3://`    | `s3://bucket/prefix`    |
| Google Cloud Storage | `gs://`    | `gs://bucket/prefix`    |
| Azure Blob Storage   | `az://`    | `az://container/prefix` |

Optionally, you may supply an additional path within the bucket, which will be used as a common prefix for all operations.

In typical cloud deployments, credentials are automatically discovered from the environment (instance metadata, workload identity, managed identity, etc.):

* **Amazon S3**: Uses [AWS SDK credential discovery](https://docs.aws.amazon.com/sdkref/latest/guide/file-format.html). For S3-compatible stores, override the endpoint with `aws-endpoint-url`.
* **Google Cloud Storage**: Uses [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials). Set `GOOGLE_SERVICE_ACCOUNT_PATH` to use a specific service account JSON file.
* **Azure Blob Storage**: Uses the [Azure credential chain](https://docs.rs/object_store/latest/object_store/azure/index.html). Set `AZURE_STORAGE_ACCOUNT_NAME` and `AZURE_STORAGE_ACCESS_KEY` for explicit credentials.

#### Local development with Minio

Minio is a common target while developing locally. You can configure it as follows:

```toml theme={null}
[worker.snapshots]
destination = "s3://bucket/cluster-name"
snapshot-interval-num-records = 1000

aws-region = "local"
aws-access-key-id = "minioadmin"
aws-secret-access-key = "minioadmin"
aws-endpoint-url = "http://localhost:9000"
aws-allow-http = true
```

#### Local development with S3

Assuming you have a profile set up to assume a specific role granted access to your bucket, you can work with S3 directly using a configuration like:

```toml theme={null}
[worker.snapshots]
destination = "s3://bucket/cluster-name"
snapshot-interval-num-records = 1000
aws-profile = "restate-dev"
```

This assumes that in your `~/.aws/config` you have a profile similar to:

```
[profile restate-dev]
source_profile = ...
region = us-east-1
role_arn = arn:aws:iam::123456789012:role/restate-local-dev-role
```

### Log Trimming and Durability

In a distributed environment, the Bifrost log is the mechanism for replicating partition state among nodes. Partition processors apply records from Bifrost to build and maintain their local **partition store** (a materialized view of the partition state in RocksDB). This partition store enables fast reads and efficient processing, but it's derived from the log and can always be rebuilt by replaying log records.

The challenge is that all cluster members need access to the relevant log records, including newly added nodes that will join the cluster in the future. This requirement is at odds with an immutable log growing unboundedly. **Log trimming** is the process of removing older segments of the log that are no longer needed.

#### Why Snapshots Matter for Log Trimming

When the log is trimmed, any partition processor that hasn't yet applied those records loses the ability to catch up by replaying the log. Instead, it must:

1. **Fetch a snapshot** from the object store that covers the trimmed records
2. **Replay only the remaining log records** after the snapshot's LSN

This is why configuring a snapshot repository is essential for multi-node clusters: without snapshots, trimmed log records are permanently lost, and any node that falls behind cannot recover.

#### Understanding Durability Modes

The **durability mode** defines the criteria that tell Restate when the partition store's state has been durably persisted elsewhere, making it safe to trim the corresponding log records. In other words, it controls when Restate considers the materialized view "backed up" enough that the original log records can be discarded.

<Accordion title="Available durability modes">
  | Mode                       | Description                                                                                                                                                                                                          |
  | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
  | `balanced`                 | Partition store is durable when covered by a snapshot **and** at least one replica has flushed to local storage. This is the **default when a snapshot repository is configured**.                                   |
  | `snapshot-only`            | Partition store is durable only after a snapshot has been created, regardless of local replica state.                                                                                                                |
  | `snapshot-and-replica-set` | Partition store is durable when **all** replicas have flushed locally **and** a snapshot exists.                                                                                                                     |
  | `replica-set-only`         | Partition store is durable when all replicas have flushed locally, regardless of snapshot state. This is the **default when no snapshot repository is configured**. Often used in Single-node setups or for testing. |
  | `none`                     | Disables automatic durability tracking and trimming entirely.                                                                                                                                                        |
</Accordion>

You can configure the durability mode in your server configuration:

```toml theme={null}
[worker]
# Controls when partition store state is considered durable enough to trim the log
# Values: "balanced", "snapshot-only", "snapshot-and-replica-set", "replica-set-only", "none"
durability-mode = "balanced"
```

<Warning>
  The `replica-set-only` mode should only be used for single-node deployments or testing. Without snapshots, if the entire cluster fails or a new node needs to bootstrap, there is no way to recover partition state for the trimmed portion of the log. This can result in permanent data loss.
</Warning>

#### Delayed Log Trimming

In some scenarios, you may want to delay log trimming even after the durability condition is met. This is particularly useful for geo-replicated deployments where snapshots need time to replicate across regions (e.g., S3 Cross-Region Replication).

```toml theme={null}
[worker]
# Delay trimming by 5 minutes after durability condition is met
trim-delay-interval = "5m"
```

This gives the snapshot repository time to replicate snapshots to other regions before the log segments they depend on are trimmed. Check your object store's cross-region replication SLA to determine an appropriate delay.

#### How Trimming Works

Each partition leader runs a **durability tracker** that monitors:

* **Durable LSN**: The log position that has been flushed to local storage on each replica (partition store flush)
* **Archived LSN**: The log position of the latest published snapshot in the object store (or the oldest retained snapshot if `worker.snapshots.experimental-num-retained` is configured)

Based on the configured durability mode, the tracker calculates the **durability point**: the LSN up to which the partition store state is considered safely persisted. Once determined:

1. The partition reports its durability point
2. If `trim-delay-interval` is configured, the actual trim is delayed by that duration
3. Log records up to the durability point are trimmed

The presence of unreachable nodes in a cluster does not affect trimming, as long as the remaining nodes continue to produce snapshots. However, active partition processors that are behind the archived LSN will cause trimming to be delayed to allow them to catch up.

Nodes that are temporarily down when the log is trimmed will fetch snapshots from the object store to fast-forward their local partition store state when they come back.

<Info title="Handling log trim gap errors">
  If you observe repeated `Shutting partition processor down because it encountered a trim gap in the log.` errors in the Restate server log, it indicates that a processor cannot start because log records it needs have been trimmed. This happens when the processor's local partition store is behind the log's trim point and no snapshot is available to bridge the gap.

  To recover, ensure a snapshot repository is correctly configured and accessible from the node reporting errors. You can still recover even if no snapshots were taken previously, as long as there is at least one healthy node with a copy of the partition data. In that case, first configure the existing node(s) to publish snapshots for the affected partition(s) to a shared destination. See the [Handling missing snapshots](/server/clusters#handling-missing-snapshots) section for detailed recovery steps.
</Info>

### Observing processor persisted state

You can use [`restatectl`](/server/clusters#controlling-clusters-with-restatectl) to see the progress of partition processors with the `list` subcommand:

```shell theme={null}
restatectl partitions list
```

This will produce output similar to the below:

```
Alive partition processors (nodes config v6, partition table v2)
 ID  NODE  MODE    STATUS  EPOCH  APPLIED  DURABLE  ARCHIVED  LSN-LAG  UPDATED
 0   N1:4  Leader  Active  e4     121428   121343   115779    0        268 ms ago
 1   N1:4  Leader  Active  e4     120778   120735   116216    0        376 ms ago
 2   N1:4  Leader  Active  e4     121348   121303   117677    0        394 ms ago
 3   N1:4  Leader  Active  e4     120328   120328   117303    0        259 ms ago
 4   N1:4  Leader  Active  e4     121108   120989   119359    0        909 ms ago
 5   N1:4  Leader  Active  e4     121543   121481   119818    0        467 ms ago
 6   N1:4  Leader  Active  e4     121253   121194   119568    0        254 ms ago
 7   N1:4  Leader  Active  e4     120598   120550   118923    0        387 ms ago
```

There are three notable persistence-related attributes in `restatectl`'s partition list output:

* **Applied LSN** - the latest log record record applied by this processor
* **Durable LSN** - the log position of the latest partition store flushed to local node storage; by default processors optimize performance by relying on Bifrost for durability and only periodically flush partition store to disk
* **Archived LSN** - if a snapshot repository is configured, this LSN represents the latest published snapshot (or the oldest retained snapshot if `worker.snapshots.experimental-num-retained` is configured); this determines the log safe trim point in multi-node clusters

### Snapshot retention

By default, Restate adds new snapshots without removing old ones. You can configure automatic pruning using the experimental `experimental-num-retained` option:

```toml theme={null}
[worker.snapshots]
experimental-num-retained = 1
```

This keeps only the most recent snapshot and automatically deletes older ones.

<Info>
  This feature is only available in Restate v1.6 and newer. Only newly uploaded snapshots after the experimental feature was activated will be pruned. Existing snapshots predating the configuration change will not be affected.
</Info>

<Warning>
  When `experimental-num-retained` is greater than 1, the archived LSN advances to the *oldest* retained snapshot rather than the latest. This delays log trimming and increases storage usage on log servers. For most deployments, `experimental-num-retained = 1` is recommended unless you need the ability to fall back to older snapshots.
</Warning>

## Data Backups

Data backups are primarily used for single-node Restate deployments.

### What does a backup contain?

The Restate server persists both metadata (such as the details of deployed services, in-flight invocations) and data (e.g., virtual object and workflow state keys) in its data store, which is located in its base directory (by default, the `restate-data` path relative to the startup working directory). Restate is configured to perform write-ahead logging with fsync enabled to ensure that effects are fully persisted before being acknowledged to participating services.

Backing up the full contents of the Restate base directory will ensure that you can recover this state in the event of a server failure. We recommend placing the data directory on fast block storage that supports atomic snapshots, such as Amazon EBS volume snapshots. Alternatively, you can stop the restate-server process, archive the base directory contents, and then restart the process. This ensures that the backup contains an atomic view of the persisted state.

In addition to the data store, you should also make sure you have a back up of the effective Restate server configuration. Be aware that this may be spread across command line arguments, environment variables, and the server configuration file.

### Restoring Backups

To restore from backup, ensure the following:

* Use a Restate server release that is compatible with the version that produced the data store snapshot. See the [Upgrading](/server/upgrading) section.
* Use an equivalent [Restate server configuration](/server/configuration). In particular, ensure that the `cluster-name` and `node-name` attributes match those of the previous Restate server operating on this data store.
* Exclusive access to a data store restored from the most recent atomic snapshot of the previous Restate installation.

<Warning title="Prevent multiple instances of the same node">
  Restate cannot guarantee that it is the only instance of the given node. You must ensure that only one instance of any given Restate node is running when restoring the data store from a backup. Running multiple instances could lead to a "split-brain" scenario where different servers process invocations for the same set of services, causing state divergence.
</Warning>


# Upgrading Restate
Source: https://docs.restate.dev/server/upgrading

Version upgrades of Restate Server, compatibility policy, and rollback strategy

Restate follows [Semantic Versioning](https://semver.org/). The server persists compatibility markers which enable it to detect incompatible data versions. However, you should be careful to follow supported version migration paths and perform [data backups](/server/snapshots#data-backups) when performing software upgrades.

## Version compatibility

<Info>
  Consult the release notes for the specific details of any new version when planning upgrades. As with all critical infrastructure changes, we recommend that you always verify the upgrade path in an isolated test environment first.
</Info>

Upgrading to the latest patch version should always be possible and is recommended to benefit from the latest bug fixes and enhancements.

Incremental minor version upgrades will retain functional compatibility with the immediate prior version. That is, for any minor version update, you will be able to upgrade from `x.y` to `x.(y+1)` while retaining all persisted data and metadata. You must not skip minor version upgrades, i.e. go directly from `x.y` to `x.(y+2)`, as it may bypass necessary data store migrations required for preserving forward compatibility.

Since later versions may introduce new functionality that is on by default, it's crucial that you baseline your configuration on the release from which you will be upgrading. If you haven't done so already, be sure to capture the [effective runtime configuration](/server/configuration/#configuration-introspection) of your existing Restate cluster on the original version, and use this configuration initially when you upgrade.

If you encounter any issues with a new version, you can downgrade a Restate installation to the latest patch level of the previous minor version. For example, you can safely rollback the Restate server version from `x.y.0` to `x.(y-1).z` if you encounter any issues. However, rollback is only supported as long as you have not used any new features exclusive to the newer version. You should enable new features in the server configuration only after you have verified the overall system behaviour on the new version. Going back to more than one minor version behind to the most recent version used with the data store is not supported.

## Service compatibility

Registered Restate services must use an SDK compatible with the service protocol version(s) of the running Restate server. Note that Restate SDKs follow independent versioning from the server. You can find the latest SDK compatibility matrix in the respective SDK version documentation.

* [Java SDK](https://github.com/restatedev/sdk-java#versions)
* [TypeScript SDK](https://github.com/restatedev/sdk-typescript#versions)
* [Go SDK](https://github.com/restatedev/sdk-go#versions)
* [Python SDK](https://github.com/restatedev/sdk-python#versions)
* [Rust SDK](https://github.com/restatedev/sdk-rust#versions)


# Service Configuration
Source: https://docs.restate.dev/services/configuration

Configure service-level behavior like retries, timeouts, retention, and privacy.

Restate services support several configuration options to adapt their behavior for different use cases.
These include timeout settings, retry policies, retention policies, and privacy controls.

## Retries

Restate retries failed invocations according to a retry policy.

The retry policy always uses exponential back-off. You can tune the initial interval, the exponentiation factor and the maximum interval.

The retry policy can set the maximum number of attempts, which can be either limited or unlimited.
When attempts are exhausted, you can configure what Restate should do with the invocation:

* **Pause** it, requiring the user to [**manually resume it**](/services/invocation/managing-invocations#resume).
* **Kill** it, which automatically fails the invocation and responds to the caller with a terminal error. Note: **compensation logic will not be executed**, so this may leave your service in an inconsistent state. For more details, check [killing the invocation](/services/invocation/managing-invocations#kill).

**Default:**

```toml restate.toml theme={null}
[invocation.default-retry-policy]
initial-interval = "50ms"
exponentiation-factor = 2.0
max-attempts = 70
max-interval = "60s"
on-max-attempts = "pause"  # Use "pause" (default) or "kill"
```

**When to adjust:** To bound the maximum number of attempts when running on FaaS, avoiding costs for unnecessary retries. For example, when an invocation constantly fails on the same bug. Use longer retry intervals when you want to reduce the load on downstream services, or shorter intervals when you want snappier retries.

<Info>
  Learn how to configure this at the [global level](/guides/error-handling#configure-restate-server-defaults), [service level](/services/configuration#service-level-configuration), or [handler level](/services/configuration#handler-level-configuration).
</Info>

## Retention of completed invocations

You can tune the retention period of the metadata and journal of completed invocations.
Decrease the retention period to save disk space. Increase it to preserve a longer history of your invocations.

<AccordionGroup>
  <Accordion title="Idempotency retention">
    How long Restate retains idempotency keys to prevent duplicate request processing.

    You can see the status of the completed invocation in the UI.

    **Default:** 24 hours

    **When to adjust:** For services that need longer idempotency windows, such as financial transactions or critical business operations.
  </Accordion>

  <Accordion title="Workflow retention">
    The workflow retention describes for how long you will be able to call the shared handlers on a workflow after the run handler has finished.
    The retention time starts counting down from the moment the run handler completes.

    After this time, the workflow's K/V state is cleared, you cannot call any shared handlers anymore, and any [Durable Promises](/develop/ts/external-events#durable-promises) are discarded.

    **Default:** 24 hours

    **When to adjust:** For workflows that need longer retention windows, to retrieve state or the result of a [Durable Promise](/develop/ts/external-events#durable-promises).
  </Accordion>

  <Accordion title="Journal retention">
    How long Restate keeps invocation journals after completion for debugging and auditing.

    Increase to see journals of completed invocations in the log.

    For workflow executions or invocations with idempotency keys:

    * the journal retention is capped by the workflow retention or idempotency retention time
    * If the journal retention is set to 0, you can see completion status but not journal entries.

    **Default:** 24 hours (tunable in self-hosted Restate as [`default-journal-retention`](/references/server-config#param-default-journal-retention))

    **When to adjust:** For workflows, AI agents, or any service where you need to inspect the execution history after completion.
  </Accordion>
</AccordionGroup>

<Info>
  [Learn how to configure these](#how-to-configure)
</Info>

## Timeouts

There are two types of timeouts describing the behavior between Restate and the service deployment.
You should tune these values if you have long-running operations in your code.

<AccordionGroup>
  <Accordion title="Inactivity timeout">
    The maximum time Restate waits for new journal entries from a service before Restate considers it stalled.
    After this timeout, Restate will ask the service to suspend.

    **Default**: 1 minute (tunable in self-hosted Restate as [`worker.invoker.inactivity-timeout`](/references/server-config#param-inactivity-timeout))

    **When to adjust:** For long-running operations like LLM calls, long-running tasks, or external API calls that take longer than the default timeout.
  </Accordion>

  <Accordion title="Abort timeout">
    Once the inactivity timeout is reached, Restate will wait for the abort timeout before interrupting the user code.
    So this is the maximum time Restate will wait for a service to suspend before it forcibly aborts the execution.

    **Default**: 10 minutes (tunable in self-hosted Restate as [`worker.invoker.abort-timeout`](/references/server-config#param-abort-timeout))

    **When to adjust:** For long-running operations like LLM calls, long-running tasks, or external API calls that take longer than the default timeout.
  </Accordion>
</AccordionGroup>

<Info>
  [Learn how to configure these](#how-to-configure)
</Info>

## Private services

Services can receive requests from HTTP, Kafka and via other services.
You can mark services as private to prevent receiving requests from HTTP and Kafka, while allowing internal service-to-service communication.

**Default**: Public

**When to adjust:** When you want to forbid requests from 3rd party services through the Restate [HTTP API](/services/invocation/http), e.g. for an internal utility service that shouldn't be exposed externally, or for a backend service that only other Restate services should access.

<Info>
  [Learn how to configure this](#how-to-configure)
</Info>

## State

Virtual Objects and Workflow allow to store [persistent state](/foundations/key-concepts#consistent-state). You can tune the state access behavior in your services and handlers.

<AccordionGroup>
  <Accordion title="Eager/Lazy state access">
    To access the Virtual object/Workflow embedded K/V store, there are two strategies:

    * Eager: A full snapshot of all K/V entries of the invoked Virtual Object/Workflow instance is sent when an invocation is invoked.
    * Lazy: Each K/V entry is individually loaded when read through `ctx.get`. On Lambda and other FaaS platforms that doesn't support bidirectional streaming, this requires the invocation to suspend and replay.

    **Default**: Eager

    **When to adjust:** Enable lazy state when you have potentially large state entries, but your handler doesn't get all of them. Enable eager state if your state is small, or you're on AWS Lambda and the cost of replay is too high.
  </Accordion>
</AccordionGroup>

<Info>
  [Learn how to configure this](#how-to-configure)
</Info>

## How to configure

### Service-level configuration

To configure your services:

<CodeGroup>
  ```ts TypeScript {"CODE_LOAD::ts/src/services/configuration/my_advanced_service.ts#options"}  theme={null}
  // Add service options to the service definition
  const myWorkflow = restate.workflow({
    name: "MyWorkflow",
    handlers: {
      run: async (ctx: restate.Context) => {},
    },
    options: {
      retryPolicy: {
        initialInterval: { seconds: 1 },
        maxInterval: { seconds: 30 },
        maxAttempts: 10,
        onMaxAttempts: "pause",
      },
      abortTimeout: { minutes: 15 },
      inactivityTimeout: { minutes: 15 },
      idempotencyRetention: { days: 3 },
      workflowRetention: { days: 3 }, // only for workflows
      journalRetention: { days: 7 },
      ingressPrivate: true,
      enableLazyState: true, // only for Virtual Objects and Workflows
    },
  });
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/services/configuration/MyWorkflow.java#options"}  theme={null}
  // Specify service options when binding them to an endpoint
  RestateHttpServer.listen(
      Endpoint.bind(
          new MyWorkflow(),
          conf ->
              conf.invocationRetryPolicy(
                      InvocationRetryPolicy.builder()
                          .initialInterval(Duration.ofSeconds(1))
                          .maxInterval(Duration.ofSeconds(1))
                          .maxAttempts(10)
                          .onMaxAttempts(InvocationRetryPolicy.OnMaxAttempts.PAUSE))
                  .abortTimeout(Duration.ofMinutes(15))
                  .inactivityTimeout(Duration.ofMinutes(15))
                  .idempotencyRetention(Duration.ofDays(3))
                  .workflowRetention(Duration.ofDays(10)) // Only for workflows
                  .journalRetention(Duration.ofDays(7))
                  .ingressPrivate(true)
                  .enableLazyState(true)));
  ```

  ```python Python {"CODE_LOAD::python/src/services/configuration.py#options"}  theme={null}
  # Specify service options when you create them
  my_workflow = restate.Workflow(
      "MyWorkflow",
      inactivity_timeout=timedelta(minutes=15),
      abort_timeout=timedelta(minutes=15),
      idempotency_retention=timedelta(days=3),
      journal_retention=timedelta(days=7),
      ingress_private=True,
      enable_lazy_state=True,  # only for Objects/Workflows
      invocation_retry_policy=restate.InvocationRetryPolicy(
          initial_interval=timedelta(seconds=1),
          max_interval=timedelta(seconds=30),
          max_attempts=10,
          on_max_attempts="pause"
      )
  )
  ```

  ```go Go {"CODE_LOAD::go/services/configuration.go#options"}  theme={null}
  // Specify service options when binding them to an endpoint
  if err := server.NewRestate().
    Bind(
      restate.Reflect(
        MyWorkflow{},
        restate.WithInvocationRetryPolicy(
          restate.WithInitialInterval(time.Second),
          restate.WithMaxInterval(30*time.Second),
          restate.WithMaxAttempts(10),
          restate.PauseOnMaxAttempts()),
        restate.WithInactivityTimeout(15*time.Minute),
        restate.WithAbortTimeout(15*time.Minute),
        restate.WithIdempotencyRetention(3*24*time.Hour),
        restate.WithJournalRetention(7*24*time.Hour),
        restate.WithIngressPrivate(true),
        restate.WithEnableLazyState(true),
        restate.WithWorkflowRetention(10*24*time.Hour), // Only for workflows
      ),
    ).
    Start(context.Background(), "0.0.0.0:9080"); err != nil {
    log.Fatal(err)
  }
  ```
</CodeGroup>

### Handler-level configuration

To set configuration at the handler level:

<CodeGroup>
  ```ts TypeScript {"CODE_LOAD::ts/src/services/configuration/my_advanced_service.ts#handleropts"}  theme={null}
  // Add handler options to the handler definition
  // For services:
  const myService = restate.service({
    name: "MyService",
    handlers: {
      myHandler: restate.handlers.handler(
        {
          retryPolicy: {
            initialInterval: { seconds: 1 },
            maxInterval: { seconds: 30 },
            maxAttempts: 10,
            onMaxAttempts: "pause",
          },
          abortTimeout: { minutes: 15 },
          inactivityTimeout: { minutes: 15 },
          idempotencyRetention: { days: 3 },
          journalRetention: { days: 7 },
          ingressPrivate: true,
        },
        async (ctx: restate.Context) => {}
      ),
    },
  });

  // For Virtual Objects:
  const myObject = restate.object({
    name: "MyObject",
    handlers: {
      myHandler: restate.handlers.object.exclusive(
        {
          retryPolicy: {
            initialInterval: { seconds: 1 },
            maxInterval: { seconds: 30 },
            maxAttempts: 10,
            onMaxAttempts: "pause",
          },
          abortTimeout: { minutes: 15 },
          inactivityTimeout: { minutes: 15 },
          idempotencyRetention: { days: 3 },
          journalRetention: { days: 7 },
          ingressPrivate: true,
          enableLazyState: true,
        },
        async (ctx: restate.ObjectContext) => {}
      ),
      mySharedHandler: restate.handlers.object.shared(
        {
          /*... my options ...*/
        },
        async (ctx: restate.ObjectSharedContext) => {}
      ),
    },
  });

  // For Workflows:
  const myWf = restate.workflow({
    name: "MyWf",
    handlers: {
      run: restate.handlers.workflow.workflow(
        {
          retryPolicy: {
            initialInterval: { seconds: 1 },
            maxInterval: { seconds: 30 },
            maxAttempts: 10,
            onMaxAttempts: "pause",
          },
          abortTimeout: { minutes: 15 },
          inactivityTimeout: { minutes: 15 },
          journalRetention: { days: 7 },
          ingressPrivate: true,
          enableLazyState: true,
        },
        async (ctx: restate.WorkflowContext) => {}
      ),
      signal: restate.handlers.workflow.shared(
        {
          /*... my options ...*/
        },
        async (ctx: restate.WorkflowSharedContext) => {}
      ),
    },
  });
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/services/configuration/MyWorkflow.java#handleropts"}  theme={null}
  // Or specify handler options when binding their service to an endpoint
  RestateHttpServer.listen(
      Endpoint.bind(
          new MyWorkflow(),
          conf ->
              conf.configureHandler(
                  "run",
                  handlerConf ->
                      handlerConf
                          .invocationRetryPolicy(
                              InvocationRetryPolicy.builder()
                                  .initialInterval(Duration.ofSeconds(1))
                                  .maxInterval(Duration.ofSeconds(1))
                                  .maxAttempts(10)
                                  .onMaxAttempts(InvocationRetryPolicy.OnMaxAttempts.PAUSE))
                          .abortTimeout(Duration.ofMinutes(15))
                          .inactivityTimeout(Duration.ofMinutes(15))
                          .journalRetention(Duration.ofDays(7))
                          .ingressPrivate(true)
                          .enableLazyState(true))));
  ```

  ```python Python {"CODE_LOAD::python/src/services/configuration.py#handleropts"}  theme={null}
  # Specify handler options via the decorator
  @my_workflow.main(
      inactivity_timeout=timedelta(minutes=15),
      abort_timeout=timedelta(minutes=15),
      workflow_retention=timedelta(days=3),
      # -> or idempotency_retention for Services/Objects
      journal_retention=timedelta(days=7),
      ingress_private=True,
      enable_lazy_state=True,  # only for Objects/Workflows
      invocation_retry_policy=restate.InvocationRetryPolicy(
          initial_interval=timedelta(seconds=1),
          max_interval=timedelta(seconds=30),
          max_attempts=10,
          on_max_attempts="pause"
      )
  )
  async def run(ctx: restate.WorkflowContext, req: str) -> str:
      # ... implement workflow logic here ---
      return "success"
  ```

  ```go Go {"CODE_LOAD::go/services/configuration.go#handleropts"}  theme={null}
  // Use ConfigureHandler to customize a handler configuration
  if err := server.NewRestate().
    Bind(
      restate.Reflect(MyWorkflow{}).
        ConfigureHandler("MyHandler",
          restate.WithInvocationRetryPolicy(
            restate.WithInitialInterval(time.Second),
            restate.WithMaxInterval(30*time.Second),
            restate.WithMaxAttempts(10),
            restate.PauseOnMaxAttempts()),
          restate.WithInactivityTimeout(15*time.Minute),
          restate.WithAbortTimeout(15*time.Minute),
          restate.WithIdempotencyRetention(3*24*time.Hour),
          restate.WithJournalRetention(7*24*time.Hour),
          restate.WithIngressPrivate(true),
          restate.WithEnableLazyState(true),
          restate.WithWorkflowRetention(10*24*time.Hour), // Only for workflows
        ),
    ).
    Start(context.Background(), "0.0.0.0:9080"); err != nil {
    log.Fatal(err)
  }
  ```
</CodeGroup>

### Override configuration

You can override the service options configured in your code via the UI or CLI.
These options apply until the next revision of the service is registered.

<AccordionGroup>
  <Accordion title="UI">
    Use the [Restate UI](/installation#restate-ui) to configure services interactively:

    * Navigate to your registered deployment
    * Click on the service you want to configure
    * Modify settings through the web interface
  </Accordion>

  <Accordion title="CLI">
    Use the Restate CLI to edit the configuration:

    ```bash theme={null}
    restate services config edit <SERVICE_NAME>
    ```
  </Accordion>
</AccordionGroup>


# Cloudflare Workers
Source: https://docs.restate.dev/services/deploy/cloudflare-workers

Run your TypeScript services on Cloudflare Workers

Deploy your Restate services on Cloudflare Workers.
This guide covers project configuration, service registration, and security setup.

<Tip>
  Follow the [quickstart](/quickstart#cloudflare-workers) to try Cloudflare Workers and Restate locally.
</Tip>

## Set up your project

<Tabs>
  <Tab title="New project">
    Start with the [cloudflare-workers-template](https://github.com/restatedev/cloudflare-workers-template) and follow its README.

    <Card title="Create your Restate + CloudFlare workers repository" icon="github" href="https://github.com/new?template_name=cloudflare-workers-template&template_owner=restatedev" />
  </Tab>

  <Tab title="Existing Cloudflare Workers project">
    Import the Restate SDK with the `fetch` component:

    ```typescript theme={null}
    import * as restate from "@restatedev/restate-sdk-cloudflare-workers/fetch";

    // Export the Restate handler
    export default {
        fetch: restate.createEndpointHandler({ services: [greeter] })
    };
    ```

    Make sure your `wrangler.toml` sets the `nodejs_compat` flag and enables preview urls:

    ```toml {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/templates/cloudflare-worker/wrangler.toml"}  theme={null}
    # The name of your Cloudflare Worker project
    name = "restate-cloudflare-worker"

    # Entrypoint
    main = "./src/index.ts"

    # Preview URLs are used to register to Restate
    preview_urls = true

    # Enable NodeJS compatibility (used by the SDK for the Node Buffer API)
    # and Node's process.env (for logging)
    compatibility_date = "2025-09-23"
    compatibility_flags = [ "nodejs_compat", "nodejs_compat_populate_process_env" ]
    ```
  </Tab>
</Tabs>

## Local development

A Workers dev server can be started on port 9080 using:

```shell theme={null}
wrangler dev --port 9080
```

Register the service with Restate:

```shell theme={null}
npx @restatedev/restate deployments register --use-http1.1 http://localhost:9080
```

<Note>
  `wrangler` only supports HTTP/1.1 when running locally, so the `--use-http1.1` flag is required. This flag is not needed when deploying to Cloudflare Workers.
</Note>

## Register the service to Restate

After deploying to Cloudflare Workers, [register the service](/services/versioning) with Restate CLI or UI providing [Preview URLs](https://developers.cloudflare.com/workers/configuration/previews/) to target specific service versions:

```shell theme={null}
npx @restatedev/restate deployments register \
  https://<VERSION_PREFIX OR ALIAS>-<WORKER_NAME>.<SUBDOMAIN>.workers.dev
```

You're set up. Head over to the **Overview page > Greeter > Playground** and start sending requests to your service.

## Restate identity keys (for Restate Cloud)

To allow only a specific Restate Cloud environment to send requests to your Cloudflare Workers deployment,
head over to your Restate Cloud Dashboard to set up Restate identity keys.

<Card title="Restate Cloud > Developers > Security" icon="lock" href="https://cloud.restate.dev/to/developers/integration#http-endpoints">
  Set up Restate identity keys
</Card>

## CI/CD Automation

To automatically deploy to Cloudflare Workers on each git commit to `main` and automatically register new [Restate service versions](/services/versioning):

```yml theme={null}
name: Deploy Worker
on:
  push:
    branches:
      - main
jobs:
  deploy:
    runs-on: ubuntu-latest
    timeout-minutes: 60
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v5
      - run: npm ci

      - name: Build & Deploy Worker
        uses: cloudflare/wrangler-action@v3
        id: deploy
        with:
          # Check https://developers.cloudflare.com/workers/ci-cd/external-cicd/github-actions/
          apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
          accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
          # Add a message to the deployment with the commit
          command: versions upload --message "Deployed via GitHub Actions - commit ${{ github.sha }}"
      - name: Get deployment URL
        id: get-url
        env:
          WRANGLER_OUTPUT: ${{ steps.deploy.outputs.command-output }}
        run: |
          URL=$(echo "$WRANGLER_OUTPUT" | awk '/Version Preview URL:/ {gsub(/.*Version Preview URL: /, ""); print}')
          echo "deployment-url=$URL" >> $GITHUB_OUTPUT

      - name: Register Restate deployment
        env:
          RESTATE_ADMIN_URL: ${{ secrets.RESTATE_ADMIN_URL }}
          RESTATE_AUTH_TOKEN: ${{ secrets.RESTATE_AUTH_TOKEN }}
        run: npx -y @restatedev/restate deployment register -y ${{ steps.get-url.outputs.deployment-url }}
```

<Note>
  Make sure your Cloudflare Worker project exists. If it doesn't, create it with: `npx wrangler deploy`
</Note>

For this workflow to execute, you need to add the following [**GitHub Actions repository secrets**](https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/use-secrets):

* `CLOUDFLARE_ACCOUNT_ID`: Your Cloudflare Account. To get your account id, check [https://developers.cloudflare.com/workers/ci-cd/external-cicd/github-actions/#cloudflare-account-id](https://developers.cloudflare.com/workers/ci-cd/external-cicd/github-actions/#cloudflare-account-id)
* `CLOUDFLARE_API_TOKEN`: Your Cloudflare API token. To get a token, check [https://developers.cloudflare.com/workers/ci-cd/external-cicd/github-actions/](https://developers.cloudflare.com/workers/ci-cd/external-cicd/github-actions/)
* `RESTATE_ADMIN_URL`: The Admin URL. You can find it in [Developers > Admin URL](https://cloud.restate.dev/to/developers/integration#admin)
* `RESTATE_AUTH_TOKEN`: Your Restate Cloud auth token. To get one, go to [Developers > API Keys > Create API Key](https://cloud.restate.dev?createApiKey=true\&createApiKeyDescription=deployment-key\&createApiKeyRole=rst:role::AdminAccess), and make sure to select **Admin** for role

<img alt="Token setup" />

<AccordionGroup>
  <Accordion title="Self-hosted Restate">
    You can use this workflow with Self-hosted Restate as well,
    just make sure to correctly set up `RESTATE_AUTH_TOKEN` and `RESTATE_ADMIN_URL` to reach your Restate cluster.
  </Accordion>
</AccordionGroup>


# Deno Deploy
Source: https://docs.restate.dev/services/deploy/deno-deploy

Run your TypeScript services on Deno Deploy

Deploy your Restate services on Deno Deploy.
This guide covers project configuration, service registration, and security setup.

<Tip>
  Follow the [quickstart](/quickstart#deno) to try Deno and Restate locally.
</Tip>

## Set up your project

<Tabs>
  <Tab title="New project">
    Start with the [deno-template](https://github.com/restatedev/examples/tree/main/typescript/templates/deno) and follow its README.

    [![Deploy on Deno](https://deno.com/button)](https://console.deno.com/new?clone=https://github.com/restatedev/deno-template)
  </Tab>

  <Tab title="Existing Deno project">
    Import the Restate SDK with the `fetch` component:

    ```typescript theme={null}
    import * as restate from "@restatedev/restate-sdk/fetch";

    // Create the Restate endpoint
    // Here you need to register your services
    const handler = restate.createEndpointHandler({
        services: [greeter],
        bidirectional: true,
    });

    // Serve using Deno
    Deno.serve({ port: 9080 }, handler);
    ```
  </Tab>
</Tabs>

## Register the service to Restate

Once deployed on Deno Deploy EA, [register the service](/services/versioning) with Restate using the CLI or UI. Use **Preview URLs** so that Restate can target specific Deno deployments:

```shell theme={null}
npx @restatedev/restate deployments register \
  https://your-project-name-<deployment_id>.deno.dev
```

You're set up. Head over to the **Overview page > Greeter > Playground** and start sending requests to your service.

## Restate identity keys (for Restate Cloud)

In order to make sure only a specific Restate Cloud environment can push requests to your Deno Deploy deployment,
head over to your Restate Cloud Dashboard to set up Restate identity keys.

<Card title="Restate Cloud > Developers > Security" icon="lock" href="https://cloud.restate.dev/to/developers/integration#http-endpoints">
  Set up Restate identity keys
</Card>

## CI/CD Automation

You can set up automation to register new [Restate service versions](/services/versioning) every time you deploy to Deno Deploy:

```yml theme={null}
name: Register to Restate

on:
  repository_dispatch:
    types: [deno_deploy.build.routed] # Listen for successful builds

jobs:
  notify:
    runs-on: ubuntu-latest
    steps:
      - name: Register Restate deployment
        env:
          RESTATE_ADMIN_URL: ${{ secrets.RESTATE_ADMIN_URL }}
          RESTATE_AUTH_TOKEN: ${{ secrets.RESTATE_AUTH_TOKEN }}
        run: npx -y @restatedev/restate deployment register -y ${{ github.event.client_payload.revision.preview_url }}
```

For this workflow to execute, you need to add the following [**GitHub Actions repository secrets**](https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/use-secrets):

* `RESTATE_ADMIN_URL`: The Admin URL. You can find it in [Developers > Admin URL](https://cloud.restate.dev/to/developers/integration#admin)
* `RESTATE_AUTH_TOKEN`: Your Restate Cloud auth token. To get one, go to [Developers > API Keys > Create API Key](https://cloud.restate.dev/to/developers/integration?createApiKey=true\&createApiKeyDescription=deployment-key\&createApiKeyRole=rst:role::AdminAccess), and make sure to select **Admin** for the role

<img alt="Token setup" />

<AccordionGroup>
  <Accordion title="Deno Deploy Classic">
    For [Deno Deploy Classic](https://docs.deno.com/deploy/classic/), we suggest the following CI/CD setup that deploys to Deno and then registers the deployment with Restate:

    ```yml theme={null}
    name: Deploy Deno

    on:
      push:
        branches:
          - main

    env:
      # Create the Deno project going to https://dash.deno.com/new_project linking this repository.
      # When creating the project, check **Just link the repo, I’ll set up GitHub Actions myself**.
      # Make sure this project name matches the deno project.
      DENO_PROJECT_NAME: ${{ vars.DENO_PROJECT_NAME }}

    jobs:
      deploy:
        permissions:
          id-token: write # required
          contents: read
        runs-on: ubuntu-latest
        timeout-minutes: 60
        steps:
          - uses: actions/checkout@v4
          - uses: denoland/setup-deno@v2
            with:
              deno-version: v2.x

          # Deploy using deployctl
          - name: Deploy to Deno Deploy
            uses: denoland/deployctl@v1
            id: deploy
            with:
              project: ${{ env.DENO_PROJECT_NAME }}
              entrypoint: main.ts

          - name: Register Restate deployment
            env:
              RESTATE_ADMIN_URL: ${{ secrets.RESTATE_ADMIN_URL }}
              RESTATE_AUTH_TOKEN: ${{ secrets.RESTATE_AUTH_TOKEN }}
            # Revision URL https://docs.deno.com/deploy/classic/deployments/#production-vs.-preview-deployments
            run: npx -y @restatedev/restate deployment register -y https://${{ env.DENO_PROJECT_NAME }}-${{ steps.deploy.outputs.deployment-id }}.deno.dev
    ```
  </Accordion>

  <Accordion title="Self-hosted Restate">
    You can use this workflow with Self-hosted Restate as well,
    just make sure to correctly set up `RESTATE_AUTH_TOKEN` and `RESTATE_ADMIN_URL` to reach your Restate cluster.
  </Accordion>
</AccordionGroup>


# Kubernetes
Source: https://docs.restate.dev/services/deploy/kubernetes

Learn how to run Restate applications on Kubernetes.

Kubernetes is a common choice for running Restate services in production environments.
This page explains how to deploy Restate applications on Kubernetes using several approaches:

* **Restate Operator**: the recommended way to manage deployments, handle service registration, automate versioning, and connect to Restate Cloud.
* **Direct Kubernetes Deployments**: for teams who prefer to manage their own Deployments and Services without the operator.
* **Knative Deployments**: for environments that benefit from autoscaling and scale-to-zero capabilities.

## Deployment with Restate Operator

When deploying Restate services to Kubernetes, we recommend using the Restate Operator.

<Card title="Restate Operator GitHub" href="https://github.com/restatedev/restate-operator" icon="github" />

The Restate Operator helps you manage Restate service deployments, including registration with your Restate cluster and versioning of your services.
It also helps with connecting your services to Restate Cloud.

### Operator deployment

Install the Restate Operator via Helm:

```bash theme={null}
helm install restate-operator \
  oci://ghcr.io/restatedev/restate-operator-helm \
  --namespace restate-operator \
  --create-namespace
```

To install the operator, you need to be able to create namespaces and CRDs.

### RestateDeployment CRD

The Restate operator allows for the use of a `RestateDeployment` CRD to deploy your services:

```yaml theme={null}
apiVersion: restate.dev/v1beta1
kind: RestateDeployment
metadata:
  name: service
spec:
  replicas: 1
  restate:
    register:
      # set this if you want to register against your RestateCluster named 'restate'
      # alternatively, you can set a url or a Service reference to register against
      cluster: restate
  selector:
    matchLabels:
      app: service
  template:
    metadata:
      labels:
        app: service
    spec:
      containers:
        - name: service
          image: path.to/yourrepo:yourtag
          env:
            - name: PORT
              value: "9080"
          ports:
            - containerPort: 9080
              name: restate
```

The CRD is an extension of a native `Deployment` and `Service` object, but automatically manages registration and [versioning](/services/deploy/kubernetes#automatic-service-versioning) for you.

View the full spec of the `RestateDeployment` CRD as [pkl file](https://github.com/restatedev/restate-operator/blob/main/crd/RestateDeployment.pkl) or less-readable [yaml file](https://github.com/restatedev/restate-operator/blob/main/crd/restatedeployments.yaml).

<Info>
  Try running a local kind cluster with a Restate deployment by following this [guide](/guides/restate-on-kind-with-operator).
</Info>

### Connecting to Restate Cloud

The Restate Operator can help you connect your services to Restate Cloud by managing authentication and secure communication.

<Info>
  Try connecting your Kubernetes services to Restate Cloud by following this [guide](/guides/connecting-k8s-services-to-cloud).
  Or check out the [Restate Cloud documentation](/cloud/connecting-services#connecting-kubernetes-services) for more information.
</Info>

### Automatic Service Versioning

Versioning gets handled automatically by the operator by keeping old ReplicaSets around with an associated Service object so that in-flight invocations can drain against the old code versions.

Have a look at the [versioning documentation](/services/versioning#automatic-versioning-with-kubernetes-operator) to learn more.

## Direct Kubernetes Deployments

If you prefer not to use the Restate Operator, you can deploy your Restate services directly using standard Kubernetes `Deployment` and `Service` resources.

A Kubernetes `Deployment` of more than one replica is generally appropriate,
and a Kubernetes `Service` is used to provide a stable DNS name and IP for the pods.

Here is an example manifest with a single pod in Kubernetes:

```yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: service
  template:
    metadata:
      labels:
        app: service
    spec:
      containers:
        - name: service
          image: path.to/yourrepo:yourtag
          env:
            - name: PORT
              value: "9080"
          ports:
            - containerPort: 9080
              name: restate
---
apiVersion: v1
kind: Service
metadata:
  name: service
spec:
  selector:
    app: service
  ports:
    - port: 9080
      name: restate
  type: ClusterIP
```

Once you have applied the manifest, [register the service](/services/versioning#registering-a-deployment) at `http://<service>.<namespace>:9080`.

⚠️ Note that this setup will not account for keeping around old code versions, so updating your code can break in-flight invocations.
Check the [versioning documentation](/services/versioning) for more information.

## Horizontal scaling and load balancing

Restate allows scaling your services horizontally by running multiple replicas of your pods.
To distribute traffic across these pods, you need to set up load balancing in between the Restate server and your service pods.
This way, Restate gets a stable endpoint to send requests to, and the load balancer distributes the requests across the available pods.

In a standard Kubernetes setup, you would typically use a `Service` of type `ClusterIP` or `LoadBalancer` to distribute traffic across your pods.
Kubernetes Services use a round-robin algorithm to balance the load among the pods.

If your services are running over HTTP2 (the default), each Restate partition will generally have only one TCP connection to a single destination pod.
However, because there are many partitions (by default 24, typically more in a larger cluster) this should get your pods a reasonably
even distribution of traffic.

For more advanced load balancing options, you can use an L7 load balancer like Istio or Cilium.

Note that there is no need for routing to be sticky, because each invocation is self-contained and has all the context it needs to execute (the journal and any K/V state).
No context is assumed to be sticky on the node.
However, if you wish to do so, you can do sticky routing via the invocation ID in your request headers (`x-restate-invocation-id`).

## Knative Deployment

Restate supports Knative services. Knative allows scaling to zero when there are no in-flight invocations and automatically configures an L7 load balancer. There are no special requirements to deploy a service deployment container with Knative:

```shell theme={null}
kn service create service-name --port h2c:9080 --image path.to/yourrepo:yourtag
```

Or using the YAML manifest:

```yaml theme={null}
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: service-name
spec:
  template:
    spec:
      containers:
        - image: path.to/yourrepo:yourtag
          ports:
            - name: h2c
              containerPort: 9080
```

The service will be accessible at `http://<service-name>.<namespace>` but to handle [versioning](/services/configuration), it is preferable to register the new revision url like `http://<service-name>-0001.<namespace>` as part of your deployment workflow.

By default Knative exposes the service through the Ingress. This is not required by Restate, and you can disable this behavior adding the argument `--cluster-local` to the aforementioned creation command.

<Info>
  Learn more about deploying Restate services with Knative by reading this [blog post](https://www.restate.dev/blog/building-stateful-serverless-applications-with-knative-and-restate) and the [Golang example](https://github.com/restatedev/examples/tree/main/go/integrations/knative-go).
</Info>


# AWS Lambda
Source: https://docs.restate.dev/services/deploy/lambda

Run your Restate services on AWS Lambda

Deploy your Restate services as serverless functions on AWS Lambda.
This guide covers project setup, packaging, and service registration.

<Tip>
  For infrastructure as code, check out the [Restate CDK construct library](https://github.com/restatedev/cdk).
</Tip>

## Set up your project

<Tabs>
  <Tab title="TypeScript">
    <Card title="Create your Restate + Typescript + Lambda repository" icon="github" href="https://github.com/new?template_name=lambda-typescript-template&template_owner=restatedev" />

    You can check out the template here: [https://github.com/restatedev/lambda-typescript-template](https://github.com/restatedev/lambda-typescript-template)

    See the [TypeScript serving docs](/develop/ts/serving#creating-a-lambda-handler) for more details.
  </Tab>

  <Tab title="Python">
    <Card title="Create your Restate + Python + Lambda repository" icon="github" href="https://github.com/new?template_name=lambda-python-template&template_owner=restatedev" />

    You can check out the template here: [https://github.com/restatedev/lambda-python-template](https://github.com/restatedev/lambda-python-template)
  </Tab>

  <Tab title="Java">
    See the [Java serving docs](/develop/java/serving#creating-a-lambda-handler) to setup a lambda handler.
  </Tab>

  <Tab title="Kotlin">
    See the [Java serving docs](/develop/java/serving#creating-a-lambda-handler) to setup a lambda handler.
  </Tab>

  <Tab title="Go">
    See the [Go serving docs](/develop/go/serving#creating-a-lambda-handler) to setup a lambda handler.
  </Tab>
</Tabs>

## Package and deploy

Build a deployment package containing your application code and dependencies.

<Tabs>
  <Tab title="TypeScript">
    Package your application as a zip file for Lambda:

    ```shell theme={null}
    npm run bundle
    ```

    The Lambda handler to setup is `app.handler`.
  </Tab>

  <Tab title="Python">
    Using `uv`, prepare a directory containing all the source and the dependencies, then zip it:

    ```shell theme={null}
    mkdir -p dist
    # Install project and dependencies directly into dist using uv pip
    uv pip install --python 3.13 --target dist .
    # Copy source code
    cp *.py dist/
    # Zip the dist content
    cd dist && zip ../dist.zip -r * && cd ..
    ```

    The lambda handler to setup is `handler.app`.

    <Warning>
      When installing the dependencies, make sure the python version and your machine architecture match the one configured in the Lambda runtime.
      If you get an error like `cannot import restate._internal`, most likely you have a python version and/or architecture mismatch between the Lambda runtime and the machine where the packaging happens.
    </Warning>
  </Tab>

  <Tab title="Java">
    Build an uber jar with all dependencies, using gradle:

    ```shell theme={null}
    ./gradlew shadowJar
    ```

    Or using Maven:

    ```shell theme={null}
    mvn package
    ```

    Upload the jar file from `build/libs/` or `target/` to Lambda.
    Set the handler to your class name, e.g., `com.example.MyLambdaHandler`.
  </Tab>

  <Tab title="Kotlin">
    Build an uber jar with all dependencies:

    ```shell theme={null}
    ./gradlew shadowJar
    ```

    Upload the jar file from `build/libs/` to Lambda.
    Set the handler to your class name, e.g., `com.example.MyLambdaHandler`.
  </Tab>

  <Tab title="Go">
    Build the Go binary for Lambda:

    ```shell theme={null}
    GOOS=linux GOARCH=amd64 go build -o bootstrap main.go
    zip function.zip bootstrap
    ```

    Upload the zip file to Lambda.
    The handler is automatically set to the binary name `bootstrap`.
  </Tab>
</Tabs>

Head over to your AWS dashboard to create the Lambda and upload the zip. For more details, follow the [AWS Lambda documentation](https://docs.aws.amazon.com/lambda/latest/dg/configuration-function-zip.html).

## Role to invoke the Lambda

When using Cloud, you'll need to create an IAM Role for Restate Cloud to invoke your Lambda.

To set up the role, visit your Restate Cloud Dashboard at [Developers > Security > AWS Lambda](https://cloud.restate.dev/to/developers/integration#lambda).

## Register the service to Restate

In order for Restate to push requests to your Lambda function, you need to [register the service to Restate](/services/versioning) using the CLI or UI:

```shell theme={null}
restate deployments register \
  arn:aws:lambda:region:account-id:function:function-name:version \
  --assume-role-arn <INVOKE_ROLE>
```

<Info>
  Always register a specific Lambda version (not `$LATEST`) to ensure Restate routes requests to a stable deployment.
  Check the [versioning documentation](/services/versioning) for more info.
</Info>

Once your service is registered, you can start sending requests to it.

## CI/CD Automation

You can set up automation to upload a new Lambda version and register new [Restate service versions](/services/versioning).

<Tabs>
  <Tab title="TypeScript">
    ```yml {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/templates/lambda/.github/workflows/deploy.yml"}  theme={null}
    name: Deploy to AWS Lambda

    on:
      push:
        branches:
          - main

    permissions:
      id-token: write   # This is required for OIDC authentication
      contents: read    # This is required to checkout the repository

    jobs:
      deploy:
        name: Deploy
        runs-on: ubuntu-latest
        environment: production
        env:
          AWS_REGION: us-east-1

        steps:
          - name: Checkout
            uses: actions/checkout@v5
          - uses: actions/setup-node@v5

          - name: Configure AWS credentials
            uses: aws-actions/configure-aws-credentials@v4
            with:
              role-to-assume: ${{ secrets.AWS_DEPLOY_ROLE_TO_ASSUME }}
              aws-region: ${{ env.AWS_REGION }}

          - name: Install dependencies
            run: npm ci
          - name: Build
            run: npm run build

          - name: Deploy Lambda Function
            uses: aws-actions/aws-lambda-deploy@v1.1.0
            id: deploy
            with:
              function-name: my-greeter
              code-artifacts-dir: dist
              handler: app.handler
              runtime: nodejs22.x
              publish: true

          - name: Register Restate deployment
            env:
              # Admin URL of your Cloud environment. You can find it in Developers > Admin URL: https://cloud.restate.dev/to/developers/integration#admin
              RESTATE_ADMIN_URL: ${{ secrets.RESTATE_ADMIN_URL }}
              # Auth token with Admin Role. To get one, go to Developers > API Keys > Create API Key: https://cloud.restate.dev?createApiKey=true&createApiKeyDescription=deployment-key&createApiKeyRole=rst:role::AdminAccess
              RESTATE_AUTH_TOKEN: ${{ secrets.RESTATE_AUTH_TOKEN }}
            run: npx -y @restatedev/restate deployment register -y ${{ steps.deploy.outputs.function-arn }} --assume-role-arn ${{ secrets.AWS_INVOKE_ROLE_TO_ASSUME }}
    ```
  </Tab>

  <Tab title="Python">
    ```yml {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/python/templates/lambda/.github/workflows/deploy.yml"}  theme={null}
    name: Deploy to AWS Lambda

    on:
      push:
        branches:
          - main

    permissions:
      id-token: write   # This is required for OIDC authentication
      contents: read    # This is required to checkout the repository

    jobs:
      deploy:
        name: Deploy
        runs-on: ubuntu-latest
        environment: production
        env:
          AWS_REGION: us-east-1
          PYTHON_VERSION: 3.13
        steps:
          - name: Checkout
            uses: actions/checkout@v5
          
          - name: Configure AWS credentials
            uses: aws-actions/configure-aws-credentials@v4
            with:
              role-to-assume: ${{ secrets.AWS_DEPLOY_ROLE_TO_ASSUME }}
              aws-region: ${{ env.AWS_REGION }}

          - name: Install uv
            uses: astral-sh/setup-uv@v5
            with:
              enable-cache: true
          
          - name: Set up Python
            # Keep this python version in sync with the Lambda runtime
            run: uv python install ${{ env.PYTHON_VERSION }}

          - name: Build Lambda package
            run: |
              mkdir -p dist
              # Install project and dependencies directly into dist using uv pip
              uv pip install --python ${{ env.PYTHON_VERSION }} --target dist .
              # Copy source code
              cp *.py dist/

          - name: Deploy Lambda Function
            uses: aws-actions/aws-lambda-deploy@v1.1.0
            id: deploy
            with:
              function-name: my-greeter
              code-artifacts-dir: dist
              handler: handler.app
              runtime: python${{ env.PYTHON_VERSION }}
              publish: true

          - name: Register Restate deployment
            env:
              # Admin URL of your Cloud environment. You can find it in Developers > Admin URL: https://cloud.restate.dev/to/developers/integration#admin
              RESTATE_ADMIN_URL: ${{ secrets.RESTATE_ADMIN_URL }}
              # Auth token with Admin Role. To get one, go to Developers > API Keys > Create API Key: https://cloud.restate.dev?createApiKey=true&createApiKeyDescription=deployment-key&createApiKeyRole=rst:role::AdminAccess
              RESTATE_AUTH_TOKEN: ${{ secrets.RESTATE_AUTH_TOKEN }}
            run: npx -y @restatedev/restate deployment register -y ${{ steps.deploy.outputs.function-arn }} --assume-role-arn ${{ secrets.AWS_INVOKE_ROLE_TO_ASSUME }}
    ```
  </Tab>
</Tabs>

For this workflow to execute, you need to configure your [AWS account for the Github OIDC provider](https://github.com/aws-actions/configure-aws-credentials/tree/main?tab=readme-ov-file#configuring-iam-to-trust-github).

Then, add the following [**GitHub Actions repository secrets**](https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/use-secrets):

* `RESTATE_ADMIN_URL`: The Admin URL. You can find it in [Developers > Admin URL](https://cloud.restate.dev/to/developers/integration#admin)
* `RESTATE_AUTH_TOKEN`: Your Restate Cloud auth token. To get one, go to [Developers > API Keys > Create API Key](https://cloud.restate.dev/to/developers/integration?createApiKey=true\&createApiKeyDescription=deployment-key\&createApiKeyRole=rst:role::AdminAccess), and make sure to select **Admin** for the role
* `AWS_INVOKE_ROLE_TO_ASSUME`: The role created in the [above paragraph](#role-to-invoke-the-lambda) to invoke the function
* `AWS_DEPLOY_ROLE_TO_ASSUME`: The role to deploy the function, see below

<AccordionGroup>
  <Accordion title="Github OIDC setup">
    To configure your account for the Github OIDC provider, run:

    ```shell theme={null}
    aws iam create-open-id-connect-provider \
        --url https://token.actions.githubusercontent.com \
        --client-id-list sts.amazonaws.com
    ```
  </Accordion>

  <Accordion title="Deploy role set up">
    To create the deploy role, head over to the AWS IAM console, and create a new role.

    The role should have the following Trust policy:

    ```json theme={null}
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Federated": "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/token.actions.githubusercontent.com"
          },
          "Action": "sts:AssumeRoleWithWebIdentity",
          "Condition": {
            "StringEquals": {
              "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
            },
            "StringLike": {
              "token.actions.githubusercontent.com:sub": "repo:<GITHUB_ORG>/<GITHUB_REPO>:*"
            }
          }
        }
      ]
    }
    ```

    And the following permissions:

    ```json theme={null}
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "LambdaDeployPermissions",
          "Effect": "Allow",
          "Action": [
            "lambda:GetFunctionConfiguration",
            "lambda:CreateFunction",
            "lambda:UpdateFunctionCode",
            "lambda:UpdateFunctionConfiguration",
            "lambda:PublishVersion"
          ],
          "Resource": "arn:aws:lambda:<REGION>:<ACCOUNT_ID>:function:<FUNCTION_NAME>"
        },
        {
          "Sid":"PassRolesDefinition",
          "Effect":"Allow",
          "Action":[
            "iam:PassRole"
          ],
          "Resource":[
            "<AWS_INVOKE_ROLE_TO_ASSUME>"
          ]
        }
      ]
    }
    ```
  </Accordion>

  <Accordion title="Self-hosted Restate">
    You can use this workflow with Self-hosted Restate as well,
    just make sure to correctly set up `RESTATE_AUTH_TOKEN` and `RESTATE_ADMIN_URL` to reach your Restate cluster.
  </Accordion>
</AccordionGroup>


# Standalone
Source: https://docs.restate.dev/services/deploy/standalone

Learn how to run standalone Restate services.

You can deploy your Restate service on any standalone machine. Restate services run as a separate process started either via the appropriate runtime (e.g. Node.js) or a standalone compiled binary (e.g. Rust) that accepts HTTP connections on a configured port, conventionally 9080.

## Docker

You can run your Restate service in a Docker container.

Most of the Restate service templates come with a `Dockerfile` that you can use to build a Docker image for your service.

## Running services behind a load balancer

To spread load across multiple instances of services and higher availability, we recommend using a load balancer. The Restate server does not currently support multiple endpoints for a single deployment.

When running an L7 load balancer such AWS Application Load Balancer, be sure to configure it to support HTTP/2 as this enables Restate to use the more efficient bi-directional service invocation protocol.


# Vercel
Source: https://docs.restate.dev/services/deploy/vercel

Run your TypeScript services on Vercel

Deploy your Restate services on Vercel.
This guide covers project configuration, service registration, and security setup.

<Tip>
  Follow the [quickstart](/quickstart) to try Next.js and Restate locally.
</Tip>

## Set up your project

<Tabs>
  <Tab title="New project">
    Start from the [vercel-template](https://github.com/restatedev/vercel-template).

    <Card title="Create your Restate + Vercel repository" icon="github" href="https://github.com/new?template_name=vercel-template&template_owner=restatedev" />
  </Tab>

  <Tab title="Existing Next.js project">
    Create a `fetch` endpoint, providing the services:

    ```typescript {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/templates/vercel/src/restate/serve.ts"}  theme={null}
    import * as restate from "@restatedev/restate-sdk/fetch";
    import { greeter } from "@/restate/greeter";

    // Create the Restate endpoint. 
    // Here you need to register your services
    const endpoint = restate.createEndpointHandler({ services: [greeter] });

    // Adapt it to Next.js route handlers
    export const serve = () => {
      return {
        POST: (req: Request) => endpoint(req),
        GET: (req: Request) => endpoint(req),
      };
    };
    ```

    And expose the endpoint in a Next.js parameterized route, e.g. `/restate/[[...services]]`:

    ```typescript {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/templates/vercel/src/app/restate/%5B%5B...services%5D%5D/route.ts"}  theme={null}
    import { serve } from "@/restate/serve";

    // This is the route that exposes the Restate services.
    // You can register it to Restate using /restate subpath (check the README)
    // To call it, open Restate > Overview > Greeter > Playground
    export const { GET, POST } = serve();
    ```
  </Tab>
</Tabs>

## Configure Vercel project

With the GitHub repository set up, go over to [the Vercel dashboard](https://vercel.com/new) to create the project.

For Restate to push requests to the Vercel project, you need Vercel Generated URLs to be accessible by Restate.
You have two alternatives:

* Disable [Vercel Authentication](https://vercel.com/docs/security/deployment-protection/methods-to-protect-deployments/vercel-authentication) for the project, making its URLs publicly accessible.
  <img alt="Vercel Authentication" />
* Enable [Protection Bypass for Automation](https://vercel.com/docs/deployment-protection/methods-to-bypass-deployment-protection/protection-bypass-automation). This makes the URLs accessible only if the `x-vercel-protection-bypass` header is provided with the right value.

## Register the service to Restate

In order for Restate to push requests to your services, you need to [register the service to Restate](/services/versioning) using the CLI or UI.
Make sure to register the [Commit URL](https://vercel.com/docs/deployments/generated-urls#generated-from-git) so that Restate can address specific Vercel deployments:

```shell theme={null}
npx @restatedev/restate deployments register \
  --use-http1.1 \
  https://<project-name>-<unique-hash>-<scope-slug>.vercel.app/restate
```

<Tip>
  `--use-http1.1` is required to interact with Next.js projects.
</Tip>

If you set up **Protection Bypass for Automation** as described above, pass the additional header as an argument:

```shell theme={null}
npx @restatedev/restate deployments register \
    --use-http1.1 \
    --extra-header x-vercel-protection-bypass=<token> \
    https://<project-name>-<unique-hash>-<scope-slug>.vercel.app/restate
```

You're set up. Head over to the **Overview page > Greeter > Playground** and start sending requests to your service.

## Restate identity keys (for Restate Cloud)

In order to make sure only a specific Restate Cloud environment can push requests to your Vercel deployment,
head over to your Restate Cloud Dashboard to set up Restate identity keys.

<Card title="Restate Cloud > Developers > Security" icon="lock" href="https://cloud.restate.dev/to/developers/integration#http-endpoints">
  Set up Restate identity keys
</Card>

## CI/CD Automation

You can set up automation to automatically register new [Restate service versions](/services/versioning) every time a Vercel deployment gets promoted:

```yml theme={null}
name: Register to Restate

# React to vercel.deployment.promoted event
on:
  repository_dispatch:
    types:
      - 'vercel.deployment.promoted'
jobs:
  deploy-to-restate:
    if: github.event_name == 'repository_dispatch'
    runs-on: ubuntu-latest
    steps:
      - name: Register Restate deployment
        env:
          # In your Restate Cloud dashboard, go to Developers > API Keys > Create API Key, and make sure to select **Admin** for the role
          RESTATE_ADMIN_URL: ${{ secrets.RESTATE_ADMIN_URL }}
          # You can find this out in Developers > Admin URL.
          RESTATE_AUTH_TOKEN: ${{ secrets.RESTATE_AUTH_TOKEN }}

        # Run service registration
        run: |
          npx -y @restatedev/restate deployment register -y \
            --use-http1.1 \
            ${{ secrets.VERCEL_PROTECTION_BYPASS_TOKEN && format('--extra-header x-vercel-protection-bypass={0}', secrets.VERCEL_PROTECTION_BYPASS_TOKEN) }} \
            ${{ github.event.client_payload.url }}/restate
```

For this workflow to execute, you need to add the following [**GitHub Actions repository secrets**](https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/use-secrets):

* `RESTATE_ADMIN_URL`: The Admin URL. You can find it in [Developers > Admin URL](https://cloud.restate.dev/to/developers/integration#admin)
* `RESTATE_AUTH_TOKEN`: Your Restate Cloud auth token. To get one, go to [Developers > API Keys > Create API Key](https://cloud.restate.dev/to/developers/integration?createApiKey=true\&createApiKeyDescription=deployment-key\&createApiKeyRole=rst:role::AdminAccess), and make sure to select **Admin** for the role

<img alt="Token setup" />

* If you set up the Protection Bypass for Automation as described above, add the secret `VERCEL_PROTECTION_BYPASS_TOKEN` with the token value

<Accordion title="Self-hosted Restate">
  You can use this workflow with Self-hosted Restate as well,
  just make sure to correctly set up `RESTATE_AUTH_TOKEN` and `RESTATE_ADMIN_URL` to reach your Restate cluster.
</Accordion>


# Introspection
Source: https://docs.restate.dev/services/introspection

Inspect the status of invocations/services.

Restate exposes information on invocations and application state via its CLI and [Introspection SQL API](/references/sql-introspection). You can use this to gain insight into the status of invocations and the service state that is stored.

This can be useful for troubleshooting. For example, a Virtual Object might be blocked and you want to kill the invocation that is blocking it, but you don't know the invocation ID. Or you want to check what is currently stored in the state of a service.

You can inspect what is stored in Restate via the UI, via the [CLI](/installation#running-restate-server--cli-locally) (via commands or SQL queries), and via curl.

<Tip title="Restate UI for understanding your applications">
  You can use the [UI](/installation#restate-ui) to debug your applications.
  Have a look at the [UI announcement blog post](https://restate.dev/blog/announcing-restate-ui/) to get some inspiration on how you can use the UI for debugging and understanding your applications.
</Tip>

## SQL over the data in Restate

Restate exposes the following SQL tables:

* `sys_invocation`: to inspect invocations
* `sys_inbox`: to inspect queue of pending invocations
* `sys_keyed_service_status`: to inspect the status of a Virtual Object
* `sys_journal`: to inspect the invocations' journal
* `sys_service`: to inspect the registered services
* `sys_deployment`: to inspect service deployments
* `sys_idempotency`: to inspect idempotency keys
* `state`: to inspect application state

You can find the schema of each of the tables in the [references](/references/sql-introspection).

The Restate Introspection SQL API has been implemented based on [DataFusion](https://arrow.apache.org/datafusion/) and supports standard SQL syntax.

## Inspecting invocations

You can execute SQL queries via the CLI or over HTTP.

<AccordionGroup>
  <Accordion title="Listing ongoing invocations">
    For each of the queries we will show the CLI command and the equivalent SQL query that you can execute via the CLI or over HTTP.

    <CodeGroup>
      ```shell CLI theme={null}
      restate invocations list
      ```

      ```shell CLI-SQL theme={null}
      restate sql --json "select * from sys_invocation;"
      ```

      ```shell curl theme={null}
      curl localhost:9070/query --json '{ "query" : "select * from sys_invocation" }'
      ```
    </CodeGroup>

    Restate only retains the entries for active invocations, workflows or invocations that were invoked with an idempotency key.
    Active invocations are invocations that haven't completed yet and are either invoked or suspended.
    For workflows and invocations that were invoked with an idempotency key, the entries are retained for their specified retention time.

    The CLI command only shows the active invocations, not the completed ones. Use `--all` to see completed ones as well.
  </Accordion>

  <Accordion title="Retrieving the status of an invocation">
    <CodeGroup>
      ```shell CLI theme={null}
      restate invocations describe my_invocation_id
      ```

      ```shell CLI-SQL theme={null}
      restate sql --json "select * from sys_invocation where id = 'my_invocation_id';"
      ```

      ```shell curl theme={null}
      curl localhost:9070/query --json '{ "query" : "select * from sys_invocation where id = 'my_invocation_id';" }'
      ```
    </CodeGroup>

    The status is either:

    * `pending`: enqueued waiting for its turn
    * `ready`: ready to be processed, but not yet running
    * `running`: actively processing
    * `backing-off`: retrying due to a failure
    * `suspended`: waiting on some external input (e.g. request-response call, awakeable, sleep, ...)
    * `completed`: completed (this is shown only for idempotent invocations)
  </Accordion>

  <Accordion title="Inspecting the invocation journal">
    <CodeGroup>
      ```shell CLI theme={null}
      restate invocations describe my_invocation_id
      ```

      ```shell CLI-SQL theme={null}
      restate sql --json "select * from sys_journal where id = 'my_invocation_id';"
      ```

      ```shell curl theme={null}
      curl localhost:9070/query --json '{ "query" : "select * from sys_journal where id = 'my_invocation_id';" }'
      ```
    </CodeGroup>

    You see the journal printed in the output.
  </Accordion>

  <Accordion title="Inspecting invocation retries">
    To have a look at the invocations that are currently in a retry loop, you can execute:

    <CodeGroup>
      ```shell CLI theme={null}
      restate invocations list --status backing-off
      ```

      ```shell CLI-SQL theme={null}
      restate sql --json "select * from sys_invocation where retry_count > 1;"
      ```

      ```shell curl theme={null}
      curl localhost:9070/query --json '{ "query" : "select * from sys_invocation where retry_count > 1;" }'
      ```
    </CodeGroup>
  </Accordion>

  <Accordion title="Listing invocations that are blocking a Virtual Object">
    You can retrieve the invocation ID that is currently blocking a Virtual Object via:

    <CodeGroup>
      ```shell CLI theme={null}
      restate invocations list --service MyService --key myKey
      ```

      ```shell CLI-SQL theme={null}
      restate sql --json "select invocation_id from sys_keyed_service_status where service_name = 'test.MyServiceName' and service_key = 'myKey';"
      ```

      ```shell curl theme={null}
      curl localhost:9070/query --json '{ "query" : "select invocation_id from sys_keyed_service_status where service_name = 'test.MyServiceName' and service_key = 'myKey';" }'
      ```
    </CodeGroup>

    With the CLI, you can also drill down and list only invocations that are blocking any Virtual Object:

    ```shell theme={null}
    restate invocations list --virtual-objects-only
    ```

    Add `--key my-key` to list only invocations that are blocking a specific Virtual Object.

    You can then use the invocation ID to [cancel the invocation](/services/invocation/managing-invocations#cancel).
  </Accordion>

  <Accordion title="Checking the last time an invocation was modified">
    <CodeGroup>
      ```shell CLI theme={null}
      restate invocations describe my_invocation_id
      ```

      ```shell CLI-SQL theme={null}
      restate sql --json "select modified_at from sys_invocation where id = 'my_invocation_id';"
      ```

      ```shell curl theme={null}
      curl localhost:9070/query --json '{ "query" : "select modified_at from sys_invocation where id = 'my_invocation_id';" }'
      ```
    </CodeGroup>

    This includes any modification to the invocation "data", for example when the service last switched its status from `invoked` to `suspended`, or when the last journal entry was added.
  </Accordion>

  <Accordion title="Checking how an invocation was triggered">
    To find out if an invocation was triggered via the ingress or by another service:

    <CodeGroup>
      ```shell CLI theme={null}
      restate invocations describe my_invocation_id
      ```

      ```shell CLI-SQL theme={null}
      restate sql --json "select invoked_by, invoked_by_service_name, invoked_by_id from sys_invocation where id = 'my_invocation_id';"
      ```

      ```shell curl theme={null}
      curl localhost:9070/query --json '{ "query" : "select invoked_by, invoked_by_service_name, invoked_by_id from sys_invocation where id = 'my_invocation_id';" }'
      ```
    </CodeGroup>

    With the CLI, you see the caller at the root of the tree in the invocation progress:

    ```shell theme={null}
    🚂 Invocation Progress:
    ―――――――――――――――――――――――
    [Ingress]
    └──(this)─> Greeter/greet
    ```

    For the SQL queries, the `invoked_by` field contains either `ingress` or `service`.
    If the invocation was triggered by another service, then the fields `invoked_by_service_name` and `invoked_by_id` will supply more information about the invoking service.
  </Accordion>

  <Accordion title="Retrieving the trace ID of an invocation">
    <CodeGroup>
      ```shell CLI theme={null}
      restate invocations describe my_invocation_id
      ```

      ```shell CLI-SQL theme={null}
      restate sql --json "select trace_id from sys_invocation where id = 'my_invocation_id';"
      ```

      ```shell curl theme={null}
      curl localhost:9070/query --json '{ "query" : "select trace_id from sys_invocation where id = 'my_invocation_id';" }'
      ```
    </CodeGroup>

    Afterwards, you can use this trace ID to [search for spans in Jaeger](/server/monitoring/tracing#searching-traces).
  </Accordion>

  <Accordion title="Listing inactive invocations">
    To list the oldest invocations that are not making progress:

    <CodeGroup>
      ```shell CLI theme={null}
      restate invocations list --oldest-first --status pending,backing-off,suspended
      ```

      ```shell CLI-SQL theme={null}
      restate sql --json "select * from sys_invocation where to_timestamp(modified_at) <= now() - interval '1' hour;"
      ```

      ```shell curl theme={null}
      curl localhost:9070/query --json '{ "query" : "select * from sys_invocation where to_timestamp(modified_at) <= now() - interval '1' hour;" }'
      ```
    </CodeGroup>
  </Accordion>

  <Accordion title="Listing zombie invocations">
    Zombie invocations are invocations that are pinned to a specific deployment but that deployment was forcefully removed. You can list them by executing:

    <CodeGroup>
      ```shell CLI theme={null}
      restate invocations list --zombie
      ```

      ```shell CLI-SQL theme={null}
      restate sql --json "select * from sys_invocation where pinned_deployment_id = 'my_deployment_id';"
      ```

      ```shell curl theme={null}
      curl localhost:9070/query --json '{ "query" : "select * from sys_invocation where pinned_deployment_id = 'my_deployment_id';" }'
      ```
    </CodeGroup>

    For the SQL queries, you need to know the deployment ID of the deployment that was forcefully removed.
  </Accordion>
</AccordionGroup>

## Inspecting application state

### With the CLI

To retrieve the state of a specific service and service key, do:

```shell theme={null}
restate kv get <SERVICE_NAME> <SERVICE_KEY>
```

Example output:

```log theme={null}
🤖 State:
―――――――――

service  counter
key      bob

KEY   VAL
seen  8
```

If the values are not JSON-encoded UTF-8 strings, then it is also possible to use the `--binary` flag,
and get the value as base64 encoded string.

### With SQL queries

You can query the application state via the `state` table.

<CodeGroup>
  ```shell CLI-SQL theme={null}
  restate sql --json "select * from state where service_name = 'test.MyServiceName' and service_key = 'myKey';"
  ```

  ```shell curl theme={null}
  curl localhost:9070/query --json '{ "query" : "select * from state where service_name = 'test.MyServiceName' and service_key = 'myKey';" }'
  ```
</CodeGroup>

If your state value is a regular string, then you can access its content in the column `value_utf8`.

To retrieve the state of a specific service name, service key and state key, do:

<CodeGroup>
  ```shell CLI-SQL theme={null}
  restate sql --json "select * from state where service_name = 'MyServiceName' and service_key = 'myKey' and key = 'myStateKey';"
  ```

  ```shell curl theme={null}
  curl localhost:9070/query --json '{ "query" : "select * from state where service_name = 'MyServiceName' and service_key = 'myKey' and key = 'myStateKey';" }'
  ```
</CodeGroup>

The state key is the name you used to store the state with the SDK. For example, the code snippet `ctx.set("count", 1)` stores `1` under the key `count`.

<Accordion title="Joining the sys_invocation and state table">
  To join the `sys_invocation` and `state` table:

  <CodeGroup>
    ```shell CLI-SQL theme={null}
    restate sql --json "select * from sys_invocation JOIN state on sys_invocation.target_service_name = state.service_name and sys_invocation.target_service_key = state.service_key;"
    ```

    ```shell curl theme={null}
    curl localhost:9070/query --json '{ "query" : "select * from sys_invocation JOIN state on sys_invocation.target_service_name = state.service_name and sys_invocation.target_service_key = state.service_key;" }'
    ```
  </CodeGroup>
</Accordion>

### Edit application state

You can edit the application state either via the state tab of the UI or via the CLI:

```shell theme={null}
restate kv edit <SERVICE_NAME> <SERVICE_KEY>
```

This command opens your default editor (as configured in the `cli env`).
It sends the new state values back to the runtime to be applied.

Use `--binary` if the values are not JSON-encoded UTF-8 strings.
In this case, you need to decode the base64-encoded string, and encode it back to base64 after editing.

Use `--plain` to retrieve the state as a JSON object.
This can be useful in combination with tools like `jq` for example:

```shell theme={null}
restate kv get counter bob --plain | jq '.seen'
```

<Warning title="Editing state during ongoing invocations">
  If during the editing of the state with the CLI, an invocation changed the state as well, then the edit of the CLI will not take affect.
  If you want the CLI state edit to be applied even if the state has changed in the meantime, then use the `--force` flag.
</Warning>

An example on how to edit the K/V state of the service `counter` for the key `bob`:

<Terminal>
  ```shell !command theme={null}
  restate kv edit counter bob
  ```

  ```log !output theme={null}
  ℹ️  About to write the following state :
  ―――――――――――――――――――――――――――――――――――――――


  service  counter
  key      bob
  force?   false
  binary?  false

  KEY   VAL
  seen  8

  ✔ Are you sure? · yes


  Enqueued successfully for processing
  ```
</Terminal>


# Go SDK Clients
Source: https://docs.restate.dev/services/invocation/clients/go-sdk

Invoke services from any Go code.

An invocation is a request to execute a handler.
The Go SDK client library lets you invoke Restate handlers from anywhere in your application.
Use this only in non-Restate services without access to the Restate Context.

<Info>
  Each invocation has its own unique ID and lifecycle.
  Have a look at [managing invocations](/services/invocation/managing-invocations) to learn about the lifecycle, kill or cancel it.
</Info>

<Info>
  Always [invoke handlers via the context](/develop/ts/service-communication), if you have access to it.
  Restate then attaches information about the invocation to the parent invocation.
</Info>

## Installation

First, add the dependency to your project `github.com/restatedev/sdk-go/ingress`.

Then, [register the service](/services/versioning) you want to invoke.

Finally, connect to Restate and invoke the handler with your preferred semantics.

## Request-response invocations

To wait on a response from the handler:

```go {"CODE_LOAD::go/develop/ingressclients.go#rpc"}  theme={null}
restateClient := restateingress.NewClient("http://localhost:8080")

// To call a service
svcResponse, err := restateingress.Service[*MyInput, *MyOutput](
  restateClient, "MyService", "MyHandler").
  Request(context.Background(), &input)

// To call a virtual object
objResponse, err := restateingress.Object[*MyInput, *MyOutput](
  restateClient, "MyObject", "Mary", "MyHandler").
  Request(context.Background(), &input)

// To Run a workflow
wfResponse, err := restateingress.Workflow[*MyInput, *MyOutput](
  restateClient, "MyWorkflow", "Mary", "Run").
  Request(context.Background(), &input)

// To interact with a workflow
status, err := restateingress.Object[*MyInput, *MyOutput](
  restateClient, "MyWorkflow", "Mary", "interactWithWorkflow").
  Request(context.Background(), &input)
```

## One-way invocations

To send a message without waiting for a response:

```go {"CODE_LOAD::go/develop/ingressclients.go#one_way_call"}  theme={null}
restateClient := restateingress.NewClient("http://localhost:8080")

// To message a service
restateingress.ServiceSend[*MyInput](
  restateClient, "MyService", "MyHandler").
  Send(context.Background(), &input)

// To message a virtual object
restateingress.ObjectSend[*MyInput](
  restateClient, "MyObject", "Mary", "MyHandler").
  Send(context.Background(), &input)

// To Run a workflow without waiting for the result
restateingress.WorkflowSend[*MyInput](
  restateClient, "MyWorkflow", "Mary", "Run").
  Send(context.Background(), &input)
```

## Delayed invocations

To schedule an invocation for a later point in time:

```go {"CODE_LOAD::go/develop/ingressclients.go#delayed_call"}  theme={null}
restateClient := restateingress.NewClient("http://localhost:8080")

// To message a service with a delay
restateingress.ServiceSend[*MyInput](
  restateClient, "MyService", "MyHandler").
  Send(context.Background(), &input, restate.WithDelay(5*24*time.Hour))

// To message a virtual object with a delay
restateingress.ObjectSend[*MyInput](
  restateClient, "MyObject", "Mary", "MyHandler").
  Send(context.Background(), &input, restate.WithDelay(5*24*time.Hour))

// To Run a workflow without waiting for the result
restateingress.WorkflowSend[*MyInput](
  restateClient, "MyWorkflow", "Mary", "Run").
  Send(context.Background(), &input, restate.WithDelay(5*24*time.Hour))
```

## Invoke a handler idempotently

By using Restate and an idempotency key, you can make any service call idempotent, without any extra code or setup. This is a very powerful feature to ensure that your system stays consistent and doesn’t perform the same operation multiple times.

To make a service call idempotent, you can use the idempotency key feature.
Add the idempotency key [to the header](https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/) via:

```go {"CODE_LOAD::go/develop/ingressclients.go#service_idempotent"}  theme={null}
restateClient := restateingress.NewClient("http://localhost:8080")

restateingress.ServiceSend[*MyInput](
  restateClient, "MyService", "MyHandler").
  Send(context.Background(), &input, restate.WithIdempotencyKey("abc"))
```

After the invocation completes, Restate persists the response for a retention period of one day (24 hours).
If you re-invoke the service with the same idempotency key within 24 hours, Restate sends back the same response and doesn't re-execute the request to the service.

The call options, with which we set the idempotency key, also let you add other headers to the request.

<Info>
  Check out the [service configuration docs](/services/configuration) to tune the retention time.
</Info>

## Retrieve result of invocations and workflows

You can use the client library to retrieve the results of invocations **with an idempotency key** or workflows.

### Attach to an invocation with an idempotency key

For invocations with an idempotency key, you can attach to the invocation and wait for it to finish:

```go {"CODE_LOAD::go/develop/ingressclients.go#service_attach"}  theme={null}
restateClient := restateingress.NewClient("http://localhost:8080")

// The call to which we want to attach later
handle, err := restateingress.ServiceSend[*MyInput](
  restateClient, "MyService", "MyHandler").
  Send(context.Background(), &input, restate.WithIdempotencyKey("my-idempotency-key"))
if err != nil {
  return err
}

// ... do something else ...

// ---------------------------------
// OPTION 1: With the invocation Id
invocationId := handle.Id()
result1, err := restateingress.InvocationById[*MyOutput](
  restateClient, invocationId).
  Attach(context.Background())

// ---------------------------------
// OPTION 2: With the idempotency key
result2, err := restateingress.ServiceInvocationByIdempotencyKey[*MyOutput](
  restateClient, "MyService", "MyHandler", "my-idempotency-key").
  Attach(context.Background())
```

### Attach/peek at a workflow execution

For workflows, you can attach to the workflow execution and wait for it to finish or peek at the output:

```go {"CODE_LOAD::go/develop/ingressclients.go#workflow_attach"}  theme={null}
restateClient := restateingress.NewClient("http://localhost:8080")

// The workflow to which we want to attach later
wfHandle, err := restateingress.WorkflowSend[*MyInput](
  restateClient, "MyWorkflow", "Mary", "Run").
  Send(context.Background(), &input)
if err != nil {
  return err
}

// ... do something else ...

// ---------------------------------
// OPTION 1: With the handle returned by the workflow submission
result, err := restateingress.InvocationById[*MyOutput](
  restateClient, wfHandle.Id()).
  Attach(context.Background())

// ---------------------------------
// OPTION 2: With the workflow ID
wfHandle2, err := restateingress.WorkflowHandle[*MyOutput](
  restateClient, "MyWorkflow", "wf-id").
  Attach(context.Background())
```


# Java/Kotlin SDK Clients
Source: https://docs.restate.dev/services/invocation/clients/java-sdk

Invoke services from any Java/Kotlin code.

An invocation is a request to execute a handler.
The Restate SDK client library lets you invoke Restate handlers from anywhere in your application.
Use this only in non-Restate services without access to the Restate Context.

<Info>
  Each invocation has its own unique ID and lifecycle.
  Have a look at [managing invocations](/services/invocation/managing-invocations) to learn how to manage the lifecycle of an invocation.
</Info>

<Info>
  Always [invoke handlers via the context](/develop/ts/service-communication), if you have access to it.
  Restate then attaches information about the invocation to the parent invocation.
</Info>

## Installation

First, add the dependency to your project

* For Java:&#x20;
* For Kotlin:&#x20;

Then, [register the service](/services/versioning) you want to invoke.

Finally, connect to Restate and invoke the handler with your preferred semantics.

## Request-response invocations

To wait on a response from the handler:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/IngressClient.java#rpc"}  theme={null}
  Client restateClient = Client.connect("http://localhost:8080");

  // To call a service
  String svcResponse = MyServiceClient.fromClient(restateClient).myHandler("Hi");

  // To call a virtual object
  String objResponse = MyObjectClient.fromClient(restateClient, "Mary").myHandler("Hi");

  // To submit a workflow
  String wfResponse =
      MyWorkflowClient.fromClient(restateClient, "Mary").submit("Hi").attach().response();
  // To interact with a workflow
  String status =
      MyWorkflowClient.fromClient(restateClient, "Mary").interactWithWorkflow("my signal");
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/IngressClient.kt#rpc"}  theme={null}
  val restateClient = Client.connect("http://localhost:8080")

  // To call a service
  val svcResponse = MyServiceClient.fromClient(restateClient).myHandler("Hi")

  // To call a virtual object
  val objResponse = MyObjectClient.fromClient(restateClient, "Mary").myHandler("Hi")

  // To submit a workflow
  val wfResponse =
      MyWorkflowClient.fromClient(restateClient, "Mary").submit("Hi").attach().response()
  // To interact with a workflow
  val status =
      MyWorkflowClient.fromClient(restateClient, "Mary").interactWithWorkflow("my signal")
  ```
</CodeGroup>

## One-way invocations

To send a message without waiting for a response:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/IngressClient.java#one_way_call"}  theme={null}
  Client restateClient = Client.connect("http://localhost:8080");

  // To message a service
  MyServiceClient.fromClient(restateClient).send().myHandler("Hi");

  // To message a virtual object
  MyObjectClient.fromClient(restateClient, "Mary").send().myHandler("Hi");

  // To submit a workflow without waiting for the result
  MyWorkflowClient.fromClient(restateClient, "Mary").submit("Hi");
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/IngressClient.kt#one_way_call"}  theme={null}
  val restateClient = Client.connect("http://localhost:8080")

  // To message a service
  MyServiceClient.fromClient(restateClient).send().myHandler("Hi")

  // To message a virtual object
  MyObjectClient.fromClient(restateClient, "Mary").send().myHandler("Hi")

  // To submit a workflow without waiting for the result
  MyWorkflowClient.fromClient(restateClient, "Mary").submit("Hi")
  ```
</CodeGroup>

## Delayed invocations

To schedule an invocation for a later point in time:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/IngressClient.java#delayed_call"}  theme={null}
  Client restateClient = Client.connect("http://localhost:8080");

  // To message a service with a delay
  MyServiceClient.fromClient(restateClient).send().myHandler("Hi", Duration.ofDays(5));

  // To message a virtual object with a delay
  MyObjectClient.fromClient(restateClient, "Mary").send().myHandler("Hi", Duration.ofDays(5));

  // To submit a workflow with a delay
  MyWorkflowClient.fromClient(restateClient, "Mary").submit("Hi", Duration.ofDays(5));
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/IngressClient.kt#delayed_call"}  theme={null}
  val restateClient = Client.connect("http://localhost:8080")

  // To message a service with a delay
  MyServiceClient.fromClient(restateClient).send().myHandler("Hi", 5.days)

  // To message a virtual object with a delay
  MyObjectClient.fromClient(restateClient, "Mary").send().myHandler("Hi", 5.days)
  ```
</CodeGroup>

## Invoke a handler idempotently

By using Restate and an idempotency key, you can make any service call idempotent, without any extra code or setup. This is a very powerful feature to ensure that your system stays consistent and doesn’t perform the same operation multiple times.

To make a service call idempotent, you can use the idempotency key feature.
Add the idempotency key [to the header](https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/) via:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/IngressClient.java#service_idempotent"}  theme={null}
  Client restateClient = Client.connect("http://localhost:8080");
  MyObjectClient.fromClient(restateClient, "Mary")
      .send()
      .myHandler("Hi", opt -> opt.idempotencyKey("abc"));
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/IngressClient.kt#service_idempotent"}  theme={null}
  val restateClient = Client.connect("http://localhost:8080")
  MyServiceClient.fromClient(restateClient).send().myHandler("Hi") { idempotencyKey = "abc" }
  ```
</CodeGroup>

After the invocation completes, Restate persists the response for a retention period of one day (24 hours).
If you re-invoke the service with the same idempotency key within 24 hours, Restate sends back the same response and doesn't re-execute the request to the service.

The call options, with which we set the idempotency key, also let you add other headers to the request.

<Info>
  Check out the [service configuration docs](/services/configuration) to tune the retention time.
</Info>

## Retrieve result of invocations and workflows

You can use the client library to retrieve the results of invocations **with an idempotency key** or workflows.

### Attach to an invocation with an idempotency key

For invocations with an idempotency key, you can attach to the invocation and wait for it to finish:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/IngressClient.java#service_attach"}  theme={null}
  Client restateClient = Client.connect("http://localhost:8080");

  // The call to which we want to attach later
  var handle =
      MyServiceClient.fromClient(restateClient)
          .send()
          .myHandler("Hi", opt -> opt.idempotencyKey("my-idempotency-key"));

  // ... do something else ...

  // ---------------------------------
  // OPTION 1: With the handle returned by the call
  // - Attach
  String result1 = handle.attach().response();
  // - Peek
  Output<String> output = handle.getOutput().response();
  if (output.isReady()) {
    String result2 = output.getValue();
  }

  // ---------------------------------
  // OPTION 2: With the Invocation ID
  // Retrieve the invocation ID from the handle and send it to another process
  String invocationId = handle.invocationId();

  // Attach/peek later from the other process
  var handle2 = restateClient.invocationHandle(invocationId, String.class);
  // use it to attach or peek (see above)

  // ---------------------------------
  // OPTION 3: With the idempotency key
  var myService = Target.service("MyService", "myHandler");
  var handle3 =
      restateClient.idempotentInvocationHandle(myService, "my-idempotency-key", String.class);
  // use it to attach or peek (see above)
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/IngressClient.kt#service_attach"}  theme={null}
  val restateClient = Client.connect("http://localhost:8080")

  // The call to which we want to attach later
  val handle =
      MyServiceClient.fromClient(restateClient).send().myHandler("Hi") {
        idempotencyKey = "my-idempotency-key"
      }

  // ... do something else ...

  // ---------------------------------
  // OPTION 1: With the handle returned by the call
  // - Attach
  val result1 = handle.attach().response()
  // - Peek
  val output = handle.getOutput().response()
  if (output.isReady()) {
    val result2 = output.getValue()
  }

  // ---------------------------------
  // OPTION 2: With the Invocation ID
  // Retrieve the invocation ID from the handle and send it to another process
  val invocationId = handle.invocationId()

  // Attach/peek later from the other process
  val handle2 = restateClient.invocationHandle(invocationId, typeTag<String>())
  // use it to attach or peek (see above)

  // ---------------------------------
  // OPTION 3: With the idempotency key
  val target = Target.service("MyService", "myHandler")
  val handle3 =
      restateClient.idempotentInvocationHandle(target, "my-idempotency-key", typeTag<String>())
  // use it to attach or peek (see above)
  ```
</CodeGroup>

### Attach/peek at a workflow execution

For workflows, you can attach to the workflow execution and wait for it to finish or peek at the output:

<CodeGroup>
  ```java Java {"CODE_LOAD::java/src/main/java/develop/IngressClient.java#workflow_attach"}  theme={null}
  Client restateClient = Client.connect("http://localhost:8080");

  // The workflow to which we want to attach later
  var wfHandle = MyWorkflowClient.fromClient(restateClient, "Mary").submit("Hi");

  // ... do something else ...

  // ---------------------------------
  // OPTION 1: With the handle returned by the workflow submission
  // - Attach
  String result = wfHandle.attach().response();
  // - Peek
  Output<String> output = wfHandle.getOutput().response();
  if (output.isReady()) {
    String result2 = output.getValue();
  }

  // ---------------------------------
  // OPTION 2: With the workflow ID
  var wfHandle2 = restateClient.workflowHandle("MyWorkflow", "wf-id", String.class);
  // use it to attach or peek (see above)
  ```

  ```kotlin Kotlin {"CODE_LOAD::kotlin/src/main/kotlin/develop/IngressClient.kt#workflow_attach"}  theme={null}
  val restateClient = Client.connect("http://localhost:8080")

  // The workflow to which we want to attach later
  val wfHandle = MyWorkflowClient.fromClient(restateClient, "Mary").submit("Hi")

  // ... do something else ...

  // ---------------------------------
  // OPTION 1: With the handle returned by the workflow submission
  // - Attach
  val result = wfHandle.attach().response()
  // - Peek
  val output = wfHandle.getOutput().response()
  if (output.isReady()) {
    val result2 = output.getValue()
  }

  // ---------------------------------
  // OPTION 2: With the workflow ID
  val wfHandle2 = restateClient.workflowHandle("MyWorkflow", "wf-id", typeTag<String>())
  // use it to attach or peek (see above)
  ```
</CodeGroup>


# TypeScript SDK Clients
Source: https://docs.restate.dev/services/invocation/clients/typescript-sdk

Invoke services from any TypeScript code.

An invocation is a request to execute a handler.
The Restate SDK client library lets you invoke Restate handlers from anywhere in your application.
Use this only in non-Restate services without access to the Restate Context.

<Info>
  Each invocation has its own unique ID and lifecycle.
  Have a look at [managing invocations](/services/invocation/managing-invocations) to learn how to manage the lifecycle of an invocation.
</Info>

<Info>
  Always [invoke handlers via the context](/develop/ts/service-communication), if you have access to it.
  Restate then attaches information about the invocation to the parent invocation.
</Info>

## Installation

First, add the dependency to your project

```shell theme={null}
npm install @restatedev/restate-sdk-clients
```

Then, [register the service](/services/versioning) you want to invoke.

Finally, connect to Restate and invoke the handler with your preferred semantics.

## Request-response invocations

To wait on a response from the handler:

```ts {"CODE_LOAD::ts/src/develop/clients/ingress.ts#rpc_call_node"}  theme={null}
// import * as clients from "@restatedev/restate-sdk-clients";
const restateClient = clients.connect({ url: "http://localhost:8080" });

// To call a service
const greet = await restateClient
  .serviceClient<MyService>({ name: "MyService" })
  .greet({ greeting: "Hi" });

// To call an object
const count = await restateClient
  .objectClient<MyObject>({ name: "MyObject" }, "Mary")
  .greet({ greeting: "Hi" });

// To call a workflow
const handle = await restateClient
  .workflowClient<MyWorkflow>({ name: "MyWorkflow" }, "someone")
  .workflowSubmit({ greeting: "Hi" });
const result = await restateClient.result(handle);

const status = await restateClient
  .workflowClient<MyWorkflow>({ name: "MyWorkflow" }, "someone")
  .myOtherHandler();
```

## One-way invocations

To send a message without waiting for a response:

```ts {"CODE_LOAD::ts/src/develop/clients/ingress.ts#one_way_call_node"}  theme={null}
// import * as clients from "@restatedev/restateClient-sdk-clients";
const restateClient = clients.connect({ url: "http://localhost:8080" });

// To send a message to a service
await restateClient
  .serviceSendClient<MyService>({ name: "MyService" })
  .greet({ greeting: "Hi" });

// To send a message to an object
await restateClient
  .objectSendClient<MyObject>({ name: "MyObject" }, "Mary")
  .greet({ greeting: "Hi" });

// To send a message to a workflow
const handle = await restateClient
  .workflowClient<MyWorkflow>({ name: "MyWorkflow" }, "someone")
  .workflowSubmit({ greeting: "Hi" });
// You cannot send a message to a shared handler in a workflow
```

## Delayed invocations

To schedule an invocation for a later point in time:

```ts {"CODE_LOAD::ts/src/develop/clients/ingress.ts#delayed_call_node"}  theme={null}
// import * as clients from "@restatedev/restate-sdk-clients";
const restateClient = clients.connect({ url: "http://localhost:8080" });

// To send a delayed message to a service
await restateClient
  .serviceSendClient<MyService>({ name: "MyService" })
  .greet({ greeting: "Hi" }, clients.rpc.sendOpts({ delay: { seconds: 1 } }));

// To send a delayed message to an object
await restateClient
  .objectSendClient<MyObject>({ name: "MyObject" }, "Mary")
  .greet({ greeting: "Hi" }, clients.rpc.sendOpts({ delay: { seconds: 1 } }));

// To send a delayed message to a workflow
const handle = await restateClient
  .workflowClient<MyWorkflow>({ name: "MyWorkflow" }, "someone")
  .workflowSubmit(
    { greeting: "Hi" },
    clients.rpc.sendOpts({ delay: { seconds: 1 } })
  );
// You cannot send a delayed message to a shared handler in a workflow
```

## Invoke a handler idempotently

By using Restate and an idempotency key, you can make any service call idempotent, without any extra code or setup. This is a very powerful feature to ensure that your system stays consistent and doesn’t perform the same operation multiple times.

To make a service call idempotent, you can use the idempotency key feature.
Add the idempotency key [to the header](https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/) via:

```typescript {"CODE_LOAD::ts/src/develop/clients/ingress.ts#service_idempotent"}  theme={null}
await restateClient
  .serviceSendClient<MyService>({ name: "MyService" })
  .greet(request, clients.rpc.sendOpts({ idempotencyKey: "abcde" }));
```

After the invocation completes, Restate persists the response for a retention period of one day (24 hours).
If you re-invoke the service with the same idempotency key within 24 hours, Restate sends back the same response and doesn't re-execute the request to the service.

The call options, with which we set the idempotency key, also let you add other headers to the request.

<Info>
  Check out the [service configuration docs](/services/configuration) to tune the retention time.
</Info>

## Retrieve result of invocations and workflows

You can use the client library to retrieve the results of invocations **with an idempotency key** or workflows.

### Attach to an invocation with an idempotency key

For invocations with an idempotency key, you can attach to the invocation and wait for it to finish:

```typescript {"CODE_LOAD::ts/src/develop/clients/ingress.ts#service_attach"}  theme={null}
// import * as clients from "@restatedev/restate-sdk-clients";
const restateClient = clients.connect({ url: "http://localhost:8080" });
// To send a message
const handle = await restateClient
  .serviceSendClient<MyService>({ name: "MyService" })
  .greet(request, clients.rpc.sendOpts({ idempotencyKey: "abcde" }));

// ... do something else ...

// Attach later to retrieve the result
const response = await restateClient.result(handle);
```

### Attach/peek at a workflow execution

For workflows, you can attach to the workflow execution and wait for it to finish or peek at the output:

```typescript {"CODE_LOAD::ts/src/develop/clients/ingress.ts#workflow_attach"}  theme={null}
// import * as clients from "@restatedev/restate-sdk-clients";
const restateClient = clients.connect({ url: "http://localhost:8080" });

// Option 1: attach and wait for result with workflow ID
const result = await restateClient
  .workflowClient<MyWorkflow>({ name: "MyWorkflow" }, "someone")
  .workflowAttach();

// Option 2: peek to check if ready with workflow ID
const peekOutput = await restateClient
  .workflowClient<MyWorkflow>({ name: "MyWorkflow" }, "someone")
  .workflowOutput();
if (peekOutput.ready) {
  const result2 = peekOutput.result;
}
```

## Advanced: sharing service type definitions

Have a look at the options listed in the [Service Communication docs](/develop/ts/service-communication#advanced%3A-sharing-service-type-definitions). These are also valid for the SDK clients.


# HTTP
Source: https://docs.restate.dev/services/invocation/http

Learn how to invoke Restate services over HTTP.

An invocation is a request to execute a handler.
You can invoke handlers over HTTP with or without waiting for a response, and with or without an idempotency key.

Make sure to first [register the handler](/services/versioning) you want to invoke.

The [UI](/installation#restate-ui) helps you with invoking your services.
Open the UI at port 9070, register your service, click on the service, open the playground, and invoke your handlers from there.

<Info>
  Each invocation has its own unique ID and lifecycle.
  Have a look at [managing invocations](/services/invocation/managing-invocations) to learn how to manage the lifecycle of an invocation.
</Info>

## Request-response calls

You can invoke services over HTTP 1.1 or higher.
Request/response bodies should be encoded as JSON.

Invoking `myHandler` of `myService` as follows:

```shell theme={null}
curl localhost:8080/MyService/myHandler \
  --json '{"name": "Mary", "age": 25}'
```

Invoke `myHandler` of `myVirtualObject` for `myKey` as follows:

```shell theme={null}
curl localhost:8080/MyVirtualObject/myObjectKey/myHandler \
  --json '{"name": "Mary", "age": 25}'
```

Call the `run` handler of the `MyWorkflow` as follows:

```shell theme={null}
curl localhost:8080/MyWorkflow/myWorkflowId/run \
  --json '{"name": "Mary", "age": 25}'
```

A workflow can be submitted only once. Resubmission of the same workflow will fail with "Previously accepted". The invocation ID can be found in the request header `x-restate-id`.

Follow the same pattern for calling the other handlers of the workflow.

<Note title="Restate as proxy">
  Note that all invocations go first via the Restate Server. The server then forwards the request to the appropriate service.
  Therefore, `localhost:8080` refers to ingress port of the Restate Server, not the service instance.
</Note>

## Sending messages

If you do not want to wait for the response, you can also send a message by adding `/send` to the URL path:

```shell theme={null}
curl localhost:8080/MyService/myHandler/send \
  --json '{"name": "Mary", "age": 25}'
```

Example output:

```json theme={null}
{"invocationId":"inv_1aiqX0vFEFNH1Umgre58JiCLgHfTtztYK5","status":"Accepted"}
```

The response contains the [Invocation ID](#invocation-identifier).
You can use this identifier [to cancel](#cancelling-invocations) or [kill the invocation](#killing-invocations).

## Delayed messages

You can **delay the message** by adding a delay request parameter in ISO8601 notation or using [humantime format](https://docs.rs/humantime/latest/humantime/):

<CodeGroup>
  ```shell humantime theme={null}
  curl "localhost:8080/MyService/myHandler/send?delay=10s" \
      --json '{"name": "Mary", "age": 25}'
  ```

  ```shell ISO8601 theme={null}
  curl "localhost:8080/MyService/myHandler/send?delay=PT10S" \
      --json '{"name": "Mary", "age": 25}'
  ```
</CodeGroup>

<Note title="Not supported for workflows">
  You cannot yet use this feature for workflows.
  Workflows can only be scheduled with a delay from within another Restate handler ([TS](/develop/ts/service-communication#delayed-calls)/[Java/Kotlin](/develop/java/service-communication#delayed-calls)).
</Note>

## Using an idempotency key

You can send requests to Restate providing an idempotency key, through the [`Idempotency-Key` header](https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/):

```shell theme={null}
curl localhost:8080/MyService/myHandler \
  -H 'idempotency-key: ad5472esg4dsg525dssdfa5loi' \
  --json '{"name": "Mary", "age": 25}'
```

After the invocation completes, Restate persists the response for a retention period of one day (24 hours).
If you re-invoke the service with the same idempotency key within 24 hours, Restate sends back the same response and doesn't re-execute the request to the service.

Check out the [service configuration docs](/services/configuration) to tune the retention time.

<Tip title="Make any service call idempotent with Restate">
  With Restate and an idempotency key, you can make any service call idempotent, without any extra code or setup.
  This is a very powerful feature to ensure that your system stays consistent and doesn't perform the same operation multiple times.
</Tip>

## Cancel

You can cancel an invocation as follows:

```shell curl theme={null}
curl -X PATCH http://localhost:9070/invocations/inv_1gdJBtdVEcM942bjcDmb1c1khoaJe11Hbz/cancel
```

<Info>
  [Read the detailed docs on cancelling invocations](/services/invocation/managing-invocations#cancel)
</Info>

## Attach to an invocation

Restate allows you to retrieve the result of workflows and invocations with an idempotency key.
There are two options:

* To **attach** to an invocation or workflow and wait for it to finish, use `/attach`.
* To **peek at the output** of an invocation or workflow, use `/output`. This will return:
  * `{"message":"not ready"}` for ongoing workflows
  * The result for finished workflows
  * `{"message":"not found"}` for non-existing workflows

You can attach to a service/object invocation only if the invocation used an idempotency key:

```shell theme={null}
# Via invocation ID
curl localhost:8080/restate/invocation/myInvocationId/attach
curl localhost:8080/restate/invocation/myInvocationId/output

# For Services, via idempotency key
curl localhost:8080/restate/invocation/MyService/myHandler/myIdempotencyKey/attach
curl localhost:8080/restate/invocation/MyService/myHandler/myIdempotencyKey/output

# For Virtual Objects, via idempotency key
curl localhost:8080/restate/invocation/myObject/myKey/myHandler/myIdempotencyKey/attach
curl localhost:8080/restate/invocation/myObject/myKey/myHandler/myIdempotencyKey/output

# For Workflows, with the Workflow ID
curl localhost:8080/restate/workflow/MyWorkflow/myWorkflowId/attach
curl localhost:8080/restate/workflow/MyWorkflow/myWorkflowId/output
```

## OpenAPI support

Restate exposes for every service an OpenAPI 3.1 definition, to get it:

```shell theme={null}
curl localhost:9070/services/MyService/openapi > MyService_openapi.json
```

You can use this definition with any OpenAPI 3.1 compliant tool to generate clients for your service, such as [openapi-generator](https://openapi-generator.tech/).

Depending on the SDKs, the rich input/output JSON schemas are included as well. At the moment, rich schemas are supported for:

* [TypeScript SDK with Zod schemas](/develop/ts/serialization#zod-schemas)
* [Python SDK with Pydantic](/develop/python/serialization#pydantic) models
* Java and Kotlin SDK


# null
Source: https://docs.restate.dev/services/invocation/kafka

Invoke handlers via Kafka events.

Connect your Restate handlers to Kafka topics.
Restate takes care of Kafka consumer management and pushes events to your Restate handlers.

You get zero-overhead consumer management with automatic retries, durable execution, and stateful processing capabilities.

<Info>
  Each event leads to an invocation, meaning a request to execute a handler. Each invocation has its own unique ID and lifecycle.
  Have a look at [managing invocations](/services/invocation/managing-invocations) to learn how to manage the lifecycle of an invocation.
</Info>

## Invoking Handlers via Kafka Events

You can invoke handlers via Kafka events, by doing the following:

<Steps>
  <Step title="Develop and register an event handler">
    You can invoke any handler via Kafka events.
    The event payload will be (de)serialized as JSON.

    * When invoking **Virtual Object** or **Workflow** handlers via Kafka, the key of the Kafka record will be used to determine the Virtual Object/Workflow key.
      The key needs to be a valid UTF-8 string.
      The events are delivered to the subscribed handler in the order in which they arrived on the topic partition.
    * When invoking **Virtual Object** or **Workflow** *shared* handlers via Kafka, the key of the Kafka record will be used to determine the Virtual Object/Workflow key.
      The key needs to be a valid UTF-8 string.
      The events are delivered to the subscribed handler in parallel without ordering guarantees.
    * When invoking **Service** handlers over Kafka, events are delivered in parallel without ordering guarantees.

    Since you can invoke any handler via Kafka events, a single handler can be invoked both by RPC and via Kafka.
  </Step>

  <Step title="Configure Restate to connect to a Kafka cluster">
    Define the Kafka cluster that Restate needs to connect to in the [Restate configuration file](/server/configuration#configuration-file):

    ```toml restate.toml theme={null}
    [[ingress.kafka-clusters]]
    name = "my-cluster"
    brokers = ["PLAINTEXT://broker:9092"]
    ```

    And make sure the Restate Server uses it via `restate-server --config-file restate.toml`.

    Check the [configuration docs](/server/configuration) for more details.

    <AccordionGroup>
      <Accordion title="Using SASL/SSL (e.g. Confluent Kafka)">
        To connect to a Kafka cluster that requires SASL/SSL authentication (e.g., Confluent Kafka), you can specify the necessary parameters in the `restate.toml` file:

        ```toml restate.toml theme={null}
        [[ingress.kafka-clusters]]
        name = "my-kafka"
        brokers = [ "my-kafka:9092" ]
        "security.protocol" = "SASL_SSL"
        "sasl.mechanisms" = "PLAIN"
        "sasl.username" = "user"
        "sasl.password" = "pass"

        "client.id" = "client-id"
        ```

        Note the quotation marks around the configuration keys.

        For Confluent Cloud, the rest of the configuration can be copied from the Confluent Cloud "Rust client" configuration.
      </Accordion>

      <Accordion title="Using SASL OAuth2.0 / OpenID Connect">
        The Kafka ingress supports SASL OAUTHBEARER authentication, enabling OAuth 2.0/OpenID Connect (OIDC) token-based connections to managed Kafka services.

        Configure SASL OAUTHBEARER via the `additional_options` field in your Kafka cluster configuration. These options are passed directly to librdkafka.

        **Example for Confluent Cloud:**

        ```toml theme={null}
        [[ingress.kafka-clusters]]
        name = "my-cluster"
        brokers = ["SASL_SSL://pkc-xxxxxx.eu-central-1.aws.confluent.cloud:9092"]
        "security.protocol"="SASL_SSL"
        "sasl.mechanisms"="OAUTHBEARER"
        "sasl.oauthbearer.method"="oidc"
        "sasl.oauthbearer.client.id"="<your-client-id>"
        "sasl.oauthbearer.client.secret"="<your-client-secret>"
        "sasl.oauthbearer.token.endpoint.url"="<your-token-endpoint>"
        "sasl.oauthbearer.scope"="kafka"
        ```

        **Common OAUTHBEARER options:**

        | Option                                | Description                                  |
        | ------------------------------------- | -------------------------------------------- |
        | `security.protocol`                   | Set to `SASL_SSL` for encrypted connections  |
        | `sasl.mechanism`                      | Set to `OAUTHBEARER`                         |
        | `sasl.oauthbearer.method`             | Set to `oidc` for OIDC-based token retrieval |
        | `sasl.oauthbearer.client.id`          | OAuth client ID                              |
        | `sasl.oauthbearer.client.secret`      | OAuth client secret                          |
        | `sasl.oauthbearer.token.endpoint.url` | OAuth token endpoint URL                     |
        | `sasl.oauthbearer.scope`              | OAuth scope (if required by provider)        |

        For the full list of available options, see the [librdkafka CONFIGURATION.md](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md).
      </Accordion>

      <Accordion title="Configuring Kafka clusters via environment variables">
        You can also configure the Kafka clusters via the `RESTATE_INGRESS__KAFKA_CLUSTERS` environment variable:

        ```bash theme={null}
        RESTATE_INGRESS__KAFKA_CLUSTERS=[{name="my-cluster",brokers=["PLAINTEXT://broker:9092"]}]
        ```
      </Accordion>

      <Accordion title="Experimental: Improved Kafka batch ingestion">
        In certain scenarios, such as consuming from a Kafka topic with few partitions, the new batch ingestion can significantly improve ingestion throughput compared to the legacy implementation.

        **This feature is disabled by default and should be used with caution.**

        All nodes in the cluster must be running Restate v1.6 before enabling this feature.

        **Once enabled and data has been ingested, you cannot roll back to a version prior to Restatev1.6.**

        To enable the experimental Kafka batch ingestion, set the following environment variable on all nodes:

        ```bash theme={null}
        RESTATE_EXPERIMENTAL_KAFKA_BATCH_INGESTION=true
        ```

        Or in your configuration file:

        ```toml theme={null}
        experimental-kafka-batch-ingestion = true
        ```

        Contact us on [Discord](https://discord.restate.dev) or [Slack](https://slack.restate.dev) to test it together with us.
      </Accordion>
    </AccordionGroup>
  </Step>

  <Step title="Register the service you want to invoke." />

  <Step title="Subscribe the event handler to the Kafka topic">
    Let Restate forward events from the Kafka topic to the event handler by creating a subscription:

    ```bash theme={null}
    curl localhost:9070/subscriptions --json '{
        "source": "kafka://my-cluster/my-topic",
        "sink": "service://MyService/handle",
        "options": {"auto.offset.reset": "earliest"}
    }'
    ```

    Once you've created a subscription, Restate immediately starts consuming events from Kafka.
    The handler will be invoked for each event received from Kafka.

    The `options` field is optional and accepts any configuration parameter from [librdkafka configuration](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md).
  </Step>
</Steps>

<AccordionGroup>
  <Accordion title="Kafka connection configuration">
    You can pass arbitrary Kafka cluster options in the `restate.toml`, and those options will be applied for all the subscriptions to that cluster, for example:

    ```toml restate.toml theme={null}
    [[ingress.kafka-clusters]]
    name = "my-cluster"
    brokers = ["PLAINTEXT://broker:9092"]
    "sasl.username" = "me"
    "sasl.password" = "pass"
    ```

    For the full list of options, check [librdkafka configuration](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md).
  </Accordion>

  <Accordion title="Multiple Kafka clusters support">
    You can configure multiple kafka clusters in the `restate.toml` file:

    ```toml restate.toml theme={null}
    [[ingress.kafka-clusters]]
    name = "my-cluster-1"
    brokers = ["PLAINTEXT://localhost:9092"]

    [[ingress.kafka-clusters]]
    name = "my-cluster-2"
    brokers = ["PLAINTEXT://localhost:9093"]
    ```

    And then, when creating the subscriptions, you refer to the specific cluster by `name`:

    ```bash theme={null}
    # Subscription to my-cluster-1
    curl localhost:9070/subscriptions --json '{
        "source": "kafka://my-cluster-1/topic-1",
        "sink": "service://MyService/handleCluster1"
    }'

    # Subscription to my-cluster-2
    curl localhost:9070/subscriptions --json '{
        "source": "kafka://my-cluster-2/topic-2",
        "sink": "service://MyService/handleCluster2"
    }'
    ```
  </Accordion>

  <Accordion title="Event metadata">
    You can access the event metadata in the handler by getting the request headers map:

    <CodeGroup>
      ```ts TypeScript {"CODE_LOAD::ts/src/develop/kafka.ts#headers"}  theme={null}
      ctx.request().headers,
      ```

      ```java Java {"CODE_LOAD::java/src/main/java/develop/MyKafkaVirtualObject.java#headers"}  theme={null}
      ctx.request().headers();
      ```

      ```go Go {"CODE_LOAD::go/develop/kafka.go#headers"}  theme={null}
      ctx.Request().Headers
      ```

      ```python Python {"CODE_LOAD::python/src/develop/kafka.py#headers"}  theme={null}
      ctx.request().headers
      ```
    </CodeGroup>

    Each event carries within this map the following entries:

    * `restate.subscription.id`: The subscription identifier, as shown by the Admin API.
    * `kafka.offset`: The record offset.
    * `kafka.partition`: The record partition.
    * `kafka.timestamp`: The record timestamp.
  </Accordion>

  <Accordion title="Raw event support">
    Check out the serialization documentation of your SDK to learn how to receive raw events in your handler.
  </Accordion>
</AccordionGroup>

## Managing Kafka Subscriptions

Restate can trigger handlers via Kafka events.

### Create Subscriptions

Subscribe a handler to a Kafka topic:

```bash theme={null}
curl localhost:9070/subscriptions --json '{
    "source": "kafka://my-cluster/my-topic",
    "sink": "service://MyService/Handle",
    "options": {"auto.offset.reset": "earliest"}
}'
```

The `options` field is optional and accepts any [librdkafka configuration](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md) parameter.

### List Subscriptions

View current subscriptions:

```bash theme={null}
curl localhost:9070/subscriptions
```

**Example response:**

```json theme={null}
{
    "subscriptions": [
        {
            "id": "sub_11XHoawrCiWtv8kzhEyGtsR",
            "source": "kafka://my-cluster/my-topic",
            "sink": "service://Greeter/greet",
            "options": {
                "auto.offset.reset": "earliest",
                "client.id": "restate",
                "group.id": "sub_11XHoawrCiWtv8kzhEyGtsR"
            }
        }
    ]
}
```

### Delete Subscriptions

Remove a subscription using its ID (starts with `sub_`):

```bash theme={null}
curl -X DELETE localhost:9070/subscriptions/sub_11XHoawrCiWtv8kzhEyGtsR
```

When you delete a subscription, Restate stops the associated consumer group. Messages already enqueued by Restate will still be processed.


# Managing Invocations
Source: https://docs.restate.dev/services/invocation/managing-invocations

Understand the lifecycle of a Restate invocation and how to manage it.

An invocation is a request to execute a handler. Each invocation has its own unique ID and lifecycle.

<Frame>
  <img alt="Conversation State Management" />
</Frame>

## Invocation ID

*Invocations* have a unique **Invocation ID** starting with `inv_`. You can find this ID in every place where an invocation is mentioned:

* The invocations page of the [Restate UI](/installation#restate-ui)
* Logs and traces (`restate.invocation.id`), both in Restate and SDKs
* In CLI commands such as `restate invocation ls`

## Lifecycle

Invocations follow a well-defined lifecycle:

<img alt="Invocation lifecycle" />

This page describes how to manage invocations in different states.

## Cancel

You can cancel an invocation at any point in its lifecycle.

Cancellation has the following characteristics:

* Frees held resources
* Cooperates with your handler code to roll back any changes made so far
* Allows proper cleanup

Via the UI:

<Frame>
  <img alt="Restart from prefix" />
</Frame>

Via the command line:

<CodeGroup>
  ```shell CLI theme={null}
  restate invocations cancel inv_1gdJBtdVEcM942bjcDmb1c1khoaJe11Hbz

  # Or bulk cancel, e.g. per service or handler
  restate invocations cancel Greeter
  restate invocations cancel Greeter/greet
  restate invocations cancel CartObject/cart55/add
  ```

  ```shell curl theme={null}
  curl -X PATCH http://localhost:9070/invocations/inv_1gdJBtdVEcM942bjcDmb1c1khoaJe11Hbz/cancel
  ```
</CodeGroup>

Cancellation is non-blocking. The API call may return before cancellation completes. In rare cases, cancellation may not take effect - retry the operation if needed.

<Info>
  For proper rollback, handlers must include compensation logic to maintain service state consistency. Check the [sagas guide](/guides/sagas).
</Info>

<Accordion title="How Cancellation Works">
  ### Requirements

  For cancellation to work, the invocation must reach the deployment (the service endpoint must be reachable).

  If the deployment is not reachable, use [kill](#kill) instead.

  ### Cancellation Flow

  When an invocation is canceled:

  1. **User sends cancellation request** to the Restate runtime
  2. **Runtime forwards cancellation** to the SDK eagerly
  3. **SDK surfaces cancellation** at the next await point in your handler code

  The cancellation exception (`TerminalError` in TypeScript/Python, `TerminalException` in Java/Kotlin) is thrown at the next **await point** - operations that wait for a result, such as `ctx.run()`, call responses, sleeps, awakeable results, etc.

  <Note>
    One-way calls (`ctx.send()`) and delayed calls (`ctx.sendDelayed()`) are detached from the call tree. They will be scheduled and execute fully even if the originating invocation is cancelled.
  </Note>

  ### Example

  <CodeGroup>
    ```typescript TypeScript {"CODE_LOAD::ts/src/services/cancellation-example.ts#here"}  theme={null}
    async function processOrder(ctx: restate.ObjectContext, order: Order) {
      // If cancellation happened before this line, this still executes
      ctx.set("status", "processing");

      // If cancellation happened before this line, this still executes
      const paymentId = ctx.rand.uuidv4().toString();

      try {
        // If cancelled right before this line, ctx.run won't execute
        // If cancelled during run block execution,
        // then a terminal error gets thrown here once execution finishes
        await ctx.run(() => processPayment(paymentId, order));

        // If cancellation happened before this line, this still executes
        ctx.serviceSendClient(notificationService).notify("Payment processed");
      } catch (error) {
        if (error instanceof restate.TerminalError) {
          // Cancellation detected - run compensation
          await ctx.run(() => refundPayment(paymentId, order));
          throw error; // Re-throw to propagate cancellation
        }
      }
    }
    ```

    ```go Go {"CODE_LOAD::go/services/cancellation-example.go#here"}  theme={null}
    func (OrderService) processOrder(ctx restate.ObjectContext, order Order) error {
      // If cancellation happened right before this line, this still executes
      restate.Set(ctx, "status", "processing")

      // If cancellation happened right before this line, this still executes
      paymentId := restate.Rand(ctx).UUID().String()

      // If cancelled right before this line, Run won't execute
      // If cancelled during run block execution,
      // then a terminal error gets returned here once execution finishes
      _, err := restate.Run(ctx, func(ctx restate.RunContext) (Payment, error) {
        return processPayment(paymentId, order)
      })
      if err != nil {
        if restate.IsTerminalError(err) {
          // Cancellation detected - run compensation
          _, _ = restate.Run(ctx, func(ctx restate.RunContext) (any, error) {
            return nil, refundPayment(paymentId)
          })
        }
        return err
      }

      // If cancellation happened right before this line, this still executes
      restate.ServiceSend(ctx, "NotificationService", "Notify").
        Send("Payment processed")

      return nil
    }
    ```

    ```java Java {"CODE_LOAD::java/src/main/java/services/CancellationExample.java#here"}  theme={null}
    @Handler
    public void processOrder(ObjectContext ctx, Order order) {
      // If cancellation happened before this line, this still executes
      ctx.set(STATUS, "processing");

      // If cancellation happened before this line, this still executes
      String paymentId = ctx.random().nextUUID().toString();
      try {

        // If canceled right before this line, ctx.run won't execute
        // If canceled during run block execution,
        // then a terminal exception gets thrown here once execution finishes
        ctx.run(() -> processPayment(paymentId, order));

        // If cancellation happened right before this line, this still executes
        NotificationServiceClient.fromContext(ctx).send().sendNotification("Your order is processed");

      } catch (TerminalException e) {
        // Cancellation detected - run compensation
        ctx.run(() -> refundPayment(paymentId, order));
        throw e; // Re-throw to propagate cancellation
      }
    }
    ```

    ```python Python {"CODE_LOAD::python/src/services/cancellation_example.py#here"}  theme={null}
    @order_service.handler()
    async def process_order(ctx: restate.ObjectContext, order: Order):
        # If cancellation happened before this line, this still executes
        ctx.set("status", "processing")

        # If cancellation happened before this line, this still executes
        payment_id = str(ctx.uuid())
        try:
            # If cancelled before this await, ctx.run won't execute
            # If cancelled during run block execution,
            # then a terminal error gets raised here once execution finishes
            await ctx.run_typed("process-payment", process_payment, id=payment_id, order=order)

            # If cancellation happened before this line, this still executes
            ctx.service_send(notify, "Payment processed")

        except TerminalError as e:
            # Cancellation detected - run compensation
            await ctx.run_typed("refund-payment", refund_payment, id=payment_id)
            raise e # Re-throw to propagate cancellation
    ```
  </CodeGroup>

  ### Call Graph Propagation

  Cancellation propagates recursively through the call graph:

  1. **Propagate to leaves**: Cancellation first reaches the leaves of the call graph
  2. **Throw at await points**: Each handler receives the cancellation exception at its next await point
  3. **Run compensation**: Handlers execute their compensation logic (if defined)
  4. **Propagate upward**: Cancellation flows back up the call graph (unless handled differently), allowing each parent to compensate
</Accordion>

## Kill

Use kill when cancellation fails (e.g., when endpoints are permanently unavailable).

Killing immediately stops all calls in the invocation tree **without executing compensation logic**. This may leave your service in an inconsistent state. Only use as a last resort after trying other fixes.

Note that one-way calls and delayed calls are not killed because they're detached from the originating call tree.

Via the UI:

<Frame>
  <img alt="Restart from prefix" />
</Frame>

Or the command line:

<CodeGroup>
  ```shell CLI theme={null}
  restate invocations kill inv_1gdJBtdVEcM942bjcDmb1c1khoaJe11Hbz

  # Or bulk kill, e.g. per service
  restate invocations kill Greeter
  restate invocations kill Greeter/greet
  restate invocations kill CartObject/cart55/add
  ```

  ```shell curl theme={null}
  curl -X PATCH http://localhost:9070/invocations/inv_1gdJBtdVEcM942bjcDmb1c1khoaJe11Hbz/kill
  ```
</CodeGroup>

## Resume

If an invocation retries for too many times, Restate will pause it.

When an invocation is paused, you can resume it manually.

Via the UI:

<Frame>
  <img alt="Restart from prefix" />
</Frame>

Via the command line:

<CodeGroup>
  ```shell CLI theme={null}
  restate invocations resume inv_1gdJBtdVEcM942bjcDmb1c1khoaJe11Hbz

  # Or bulk resume, e.g. per service
  restate invocations resume Greeter
  ```

  ```shell curl theme={null}
  curl -X PATCH http://localhost:9070/invocations/inv_1gdJBtdVEcM942bjcDmb1c1khoaJe11Hbz/resume
  ```
</CodeGroup>

Check out the [service configuration docs](/services/configuration#retries) to tune the invocation retry policy,
including the retry interval and how many attempts to perform before pausing.

You can also resume an invocation that is waiting on a retry timer, instructing Restate to retry immediately.

<Accordion title="Resuming on a different deployment">
  When resuming, the invocation will run on the same deployment where it started.

  If the invocation failed due to some bug you fixed on a new/different deployment,
  you can override on which deployment the invocation should be resumed:

  <CodeGroup>
    ```shell CLI theme={null}
    # Resume on a different deployment
    restate invocations resume <invocation_id> --deployment latest
    restate invocations resume <invocation_id> --deployment <deployment_id>
    ```

    ```shell curl theme={null}
    curl -X PATCH http://localhost:9070/invocations/inv_1gdJBtdVEcM942bjcDmb1c1khoaJe11Hbz/resume\?deployment=dp_17sztQp4gnEC1L0OCFM9aEh
    ```
  </CodeGroup>

  Be aware that if the business logic/flow between the old and the new deployment code differ, once resumed the invocation will start fail with **non-determinism errors**!
</Accordion>

## Pause

You can manually pause an invocation too, for example, to resume it on a different endpoint as mentioned above.

<CodeGroup>
  ```shell CLI theme={null}
  restate invocations pause inv_1gdJBtdVEcM942bjcDmb1c1khoaJe11Hbz

  # Or bulk pause, e.g. per service
  restate invocations pause Greeter
  ```

  ```shell curl theme={null}
  curl -X PATCH http://localhost:9070/invocations/inv_1gdJBtdVEcM942bjcDmb1c1khoaJe11Hbz/pause
  ```
</CodeGroup>

## Purge

After an invocation completes, it will be retained by Restate for some time, in order to introspect it and, in case of idempotent requests, to perform deduplication.

Check out the [service configuration docs](/services/configuration#retention-of-completed-invocations) to tune the retention time.

You can also manually purge completed invocations if you need to free up disk space:

<CodeGroup>
  ```shell CLI theme={null}
  restate invocations purge inv_1gdJBtdVEcM942bjcDmb1c1khoaJe11Hbz

  # Or bulk purge, e.g. per service
  restate invocations purge Greeter
  ```

  ```shell curl theme={null}
  curl -X PATCH http://localhost:9070/invocations/inv_1gdJBtdVEcM942bjcDmb1c1khoaJe11Hbz/purge
  ```
</CodeGroup>

## Restart as new

Invocations, once completed, can be **restarted as new invocations**.

Via the UI:

<Frame>
  <img alt="Restart from prefix" />
</Frame>

Via the command line:

<CodeGroup>
  ```shell CLI theme={null}
  restate invocations restart-as-new inv_1gdJBtdVEcM942bjcDmb1c1khoaJe11Hbz

  # Or bulk restart as new, e.g. per service
  restate invocations restart-as-new Greeter
  ```

  ```shell curl theme={null}
  curl -X PATCH http://localhost:9070/invocations/inv_1gdJBtdVEcM942bjcDmb1c1khoaJe11Hbz/restart-as-new
  ```
</CodeGroup>

The new invocation will have a different invocation ID, but the input and headers of the original request.
The old invocation will be left untouched.

Keep in mind that when restarting an invocation as new, no partial progress will be kept, meaning all the operations the service executed will be executed again.

For workflows, this feature is only available in the UI.

## Restart from Prefix

You can restart an invocation while retaining a part of the journal of the previous execution.
This creates a new invocation which replays the retained part of the journal and then continues the execution from there on.

Via the UI:

<img alt="Restart from prefix" />


# Security
Source: https://docs.restate.dev/services/security

Restrict access to Restate services

This page covers securing communication between Restate and your service deployments. For securing the Restate Server itself (network ports, admin access, header handling), see [Server Security](/server/security).

## Private services

When registering an endpoint, every service is by default reachable via HTTP requests to the ingress.

You can configure a service as `private`, via the [service configuration](/services/configuration).

Note that private services can still be invoked by other handlers via the SDK.

## Locking down service access

Only Restate needs to be able to make requests to your services.
The Restate Server will proxy all requests for these services.

Therefore, it is advisable to ensure that only Restate can reach your service.
Unrestricted access to the services is dangerous. If you're working with multiple Restate instances, you also may want to check that requests are
coming from the right instance.

To make this easier, Restate has a native request identity feature which can be
used in the SDK to cryptographically verify that requests have come from a
particular Restate instance.

To get started, you need an ED25519 private key, which you can generate using
openssl as follows:

```bash theme={null}
openssl genpkey -algorithm ed25519 -outform pem -out private.pem
```

You can provide the path to the key to Restate on startup using an environment
variable: `RESTATE_REQUEST_IDENTITY_PRIVATE_KEY_PEM_FILE=private.pem`.

On start, Restate will log out the public key in a convenient compact
format:

```
INFO restate_service_client::request_identity::v1
  Loaded request identity key
    kid: "publickeyv1_w7YHemBctH5Ck2nQRQ47iBBqhNHy4FV7t2Usbye2A6f"
    path: private.pem
```

You can also obtain the public key in this format without running Restate using
the following script:

```shell generate.sh expandable theme={null}
#!/usr/bin/env bash
set -euo pipefail

# generate private key
openssl genpkey -algorithm ed25519 -outform pem -out private.pem
echo "Wrote private key to private.pem"

base58_chars="123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz"
# encode public key
encoded=$(openssl ec -in private.pem -inform pem -pubout -outform der -out /dev/stdout 2>/dev/null |
tail -c +13 |
basenc --base16 "${1:-/dev/stdin}" -w0 |
if
read
[[ $REPLY =~ ^((00)*)(([[:xdigit:]]{2})*) ]]
echo -n "${BASH_REMATCH[1]//00/1}" # leading 0s -> 1
(( ${#BASH_REMATCH[3]} > 0 ))
then
dc -e "16i0${BASH_REMATCH[3]^^} Ai[58~rd0<x]dsxx+f" | # hex bytes to indexes into the char string
while read -r
do echo -n "${base58_chars:REPLY:1}"
done
fi)

echo -n "publickeyv1_${encoded}" > public-key
echo "Wrote publickeyv1_${encoded} to public-key"
```

The string `publickeyv1_w7YHemBctH5Ck2nQRQ47iBBqhNHy4FV7t2Usbye2A6f` does not
need to be kept secret and can be safely included in your code when providing to the
SDK. To learn how to provide the public key to the SDK, see the serving docs of your SDK:
([TS](/develop/ts/serving#validating-request-identity) /
[Java](/develop/java/serving#validating-request-identity) /
[Go](/develop/go/serving#validating-request-identity) /
[Python](/develop/python/serving#validating-request-identity) /
[Rust](https://docs.rs/restate-sdk/latest/restate_sdk/http_server/index.html#validating-request-identity))

## Client side journal encryption

You can configure SDKs to encrypt invocation inputs and outputs, step results, and state on the client side, while preserving observability in the Restate UI.

This feature is currently available upon request. [Contact us](mailto:info@restate.dev?subject=Journal%20Encryption%20request) to learn more.


# Versioning
Source: https://docs.restate.dev/services/versioning

Understand deployments, registration, and versioning in Restate.

Deploying new versions of code without breaking ongoing work **is complicated in every stateful application**.
You need to make sure that requests start, retry, and complete on the same version of your "business logic", to avoid problems like inconsistent execution or state corruption.

Restate helps via the concept of **immutable deployments**. When you deploy a version of your code, you give it an immutable, unique endpoint and register it with Restate.
Restate then makes sure that requests start and end on the same version, by sending any retry attempts always to the same endpoint.
This eliminates mid-execution version mismatches: **no version compatibility logic is needed** in your code.

## Service deployments

A service deployment is a specific instance of your service code associated with an HTTP endpoint, a Lambda function, or other supported environment.

Deployments can be managed through the [Restate UI](/installation#restate-ui), the CLI, or the Admin REST API:

```shell theme={null}
# Register a deployment
restate deployments register http://localhost:9080
# Get detailed information about a specific deployment
restate deployment describe <deployment_id>
# Check the registered deployments
restate deployments list
```

<Note>
  Some deployment targets support only HTTP/1.1. Use `--use-http1.1` when registering these deployments.
</Note>

<Note>
  For AWS Lambda, use the function ARN instead of a URL. See [AWS Lambda deployment](/services/deploy/lambda).
</Note>

<Tip>
  Avoid long-running handlers (days or months) - otherwise you need to keep old deployments around until all invocations complete. Instead, break work into smaller chunks using delayed calls.
</Tip>

## Automatic versioning with FaaS platforms

Function-as-a-Service (FaaS) platforms automatically handle immutable versioning through version-specific URLs or ARNs.
This makes them ideal for Restate deployments as they eliminate the complexity of manual version management.

Have a look at the dedicated deployment docs to learn more:

* [Vercel](/services/deploy/vercel#register-the-service-to-restate): Register the Commit URL so that Restate can address specific Vercel deployments.
* [AWS Lambda](/services/deploy/lambda): When you [publish a Lambda function](https://docs.aws.amazon.com/lambda/latest/dg/configuration-versions.html), it automatically creates an immutable version with a unique ARN that never changes.
* [Deno Deploy](/services/deploy/deno-deploy#register-the-service-to-restate): Register the Preview URLs so that Restate can target specific Deno deployments.
* [Cloudflare Workers](/services/deploy/cloudflare-workers#register-the-service-to-restate): Register the Preview URLs so that Restate can target specific Cloudflare deployments.

## Automatic versioning with Kubernetes Operator

The [Restate Kubernetes operator](/services/deploy/kubernetes#deployment-with-restate-operator) provides a higher-level abstraction for managing service deployments and their versions automatically.

The operator handles the complete versioning lifecycle:

1. **Deploy new versions**: Create a new `RestateDeployment` with your updated container image
2. **Automatic registration**: The operator registers the new deployment with the Restate cluster
3. **Traffic routing**: New requests automatically route to the latest version
4. **Graceful draining**: The operator monitors older deployments for ongoing invocations
5. **Auto-scaling to zero**: Once drained, older versions automatically scale to zero

For complete examples and specifications, see [Restate Operator git repo](https://github.com/restatedev/restate-operator).

## Manual versioning

For non-FaaS deployments, you can use either the UI or the CLI to manage deployments. The typical update flow looks like this:

<Steps>
  <Step title="Deploy new version">
    Deploy your updated service code to a new endpoint (e.g., `http://greeter-v2/`).
  </Step>

  <Step title="Register the new deployment">
    Then register it with Restate:

    ```shell CLI theme={null}
    restate deployments register http://greeter-v2/
    ```

    Restate automatically routes new requests to the latest deployment. Existing requests continue on the original deployment.
  </Step>

  <Step title="Monitor deployment status">
    Check for in-flight invocations on deployments [via the UI](https://restate.dev/blog/announcing-restate-ui/#an-aid-for-versioning) or CLI.

    ```shell theme={null}
    # Get detailed information about a specific deployment
    # including the list of active invocations
    restate deployment describe <deployment_id> --extra
    ```
  </Step>

  <Step title="Remove old deployment">
    Once all invocations are complete, you can safely remove the old deployment via the UI or CLI.

    ```shell theme={null}
    restate deployments remove <deployment_id>
    ```

    If you need to force removal before the deployment is fully drained, use the `--force` flag in CLI.
  </Step>
</Steps>

<Note title="State compatibility">
  Virtual Object state persists across versions. Ensure your state schema remains backward compatible.
</Note>

## Local development

During local development, you're iterating quickly on your code and don't need immutable deployments.
You can safely re-register the same endpoint after code changes using the `--force` flag:

```shell theme={null}
restate deployments register --force localhost:9080
```

This overwrites the existing deployment registration, allowing Restate to discover your updated service definition.

<Note>
  In-flight invocations might keep failing with [non-determinism errors](/references/errors#rt0016), but this is typically fine during development.

  You can kill all the in-flight invocations to a service using either CLI or UI:

  ```shell CLI theme={null}
  restate invocations kill <SERVICE_NAME>
  ```
</Note>

## Advanced operations and troubleshooting

This section covers common scenarios you may encounter when managing deployments, and some of the troubleshooting best practices we recommend.

### Journal mismatch errors

Restate performs journal compatibility checks during replay to prevent corruption.
When you see a journal mismatch error ([RT0016](/references/errors#rt0016)), it means the code executed during replay has produced a different journal than the original execution.
This is typically caused by two scenarios:

**Non-deterministic code**: Code that produces different results on each execution, even with the same inputs.

Examples of non-deterministic operations:

* External operations such as HTTP requests
* Random value generation: `Math.random()`, `uuid.v4()`, etc.
* Getting the current time or date: `new Date()`
* Iterating over unordered collections, such as hash maps

To record non-deterministic operation results for replay, you need to record its results using `ctx.run`.
For more info, see the **durable steps** documentation: [TypeScript](/develop/ts/durable-steps), [Python](/develop/python/durable-steps), [Java/Kotlin](/develop/java/durable-steps), [Go](/develop/go/durable-steps).

**In-place code changes**: Modifying deployed code at the same endpoint. This violates the immutable deployment principle and can cause in-flight invocations to fail when they replay with the updated code.

Unsafe changes include:

* Reordering Restate SDK operations (`run`, state access, service calls, awakeables)
* Adding or removing SDK operations in the execution path
* Changing operation inputs (state keys, service call payloads)
* Modifying conditional logic that affects which operations execute

See the sections above for deployment best practices and the sections below for how to fix in-flight invocations.

### Fixing a bug by updating deployed code

If you have a bug in your deployment code, sometimes it is safe to fix it by updating the deployment in-place.

| Change                                                                 | Safe? |
| ---------------------------------------------------------------------- | ----- |
| Fixing a bug inside `ctx.run`                                          | ✅     |
| Fixing a bug that consistently reproduces (e.g. deserialization error) | ✅     |
| Changing the order of Restate operations, or adding/removing them      | ❌     |
| Changing operation inputs (state keys, call payloads, etc.)            | ❌     |

<Note>
  This approach doesn't work on FaaS platforms (e.g. Lambda) or with the Kubernetes Operator, since these use immutable deployments by design.
  In those cases, use [pause and resume](#pause-invocations-and-resume-on-a-new-deployment) instead.
</Note>

### Pause invocations and resume on a new deployment

This is the **recommended approach** for fixing bugs affecting in-flight invocations. It works on all platforms, including FaaS and Kubernetes.

When a bug affects in-flight invocations, they remain pinned to the original deployment.
Registering a new deployment with a fix only helps new invocations. To fix existing invocations:

<Steps>
  <Step title="Register a new deployment with the fix">
    Deploy your fixed code and register it with Restate:

    ```shell theme={null}
    restate deployments register http://greeter-v2/
    ```
  </Step>

  <Step title="Pause the affected invocations">
    ```shell theme={null}
    restate invocations pause <invocation_id>
    ```
  </Step>

  <Step title="Resume on the new deployment">
    ```shell theme={null}
    restate invocations resume <invocation_id> --deployment <new_deployment_id>
    ```
  </Step>
</Steps>

The fix must be compatible with already-executed journal entries so they can replay successfully.
If the business logic differs, the invocation will fail with [non-determinism errors](/references/errors#rt0016).

For more details, see [managing invocations](/services/invocation/managing-invocations#resume).

### Reassigning a deployment endpoint

<Warning>
  This is an advanced technique. In most cases, use [pause and resume](#pause-invocations-and-resume-on-a-new-deployment) instead.
</Warning>

You can use the Admin API to change which endpoint a deployment points to.
This is useful when you've deployed a fix to a new URL and want stuck invocations to use it:

```shell theme={null}
curl -X PUT localhost:9070/deployments/dp_14LsPzGz9HBxXIeBoH5wYUh \
  --json '{"uri": "http://greeter-patched/"}'
```

The same determinism rules apply: the fix must be compatible with already-executed journal entries.

### Updating a service interface

When you register a new deployment, Restate validates that it doesn't break existing service interfaces: adding handlers is allowed, but removing or renaming them is not.

For example, given you register a new deployment for the already existing service `Greeter`, renaming its handler from `greet` to `sayGreet` is a breaking change, thus registration will fail.

To allow breaking changes, use the `--breaking` flag:

```shell theme={null}
restate deployments register --breaking http://greeter-v2/
```

<Tip>
  Before using `--breaking`, make sure all callers have been updated to use the new service interface.
</Tip>

### Removing a service

In Restate to remove a service, you must remove the deployment that contains it.

To do this safely, follow the steps below:

1. Ensure no other handlers or services have business logic that calls the service you're removing.
2. If several services are bundled in the same deployment, you can't remove only one of them. You have to remove the whole deployment.
   So make sure that you first deploy the services you want to keep in a separate new deployment.
3. [Make the service private](/services/security#private-services) to avoid accepting new HTTP requests.
4. Check whether the service has pending invocations by filtering the invocations on deployment ID in the [UI](/installation#restate-ui) or via `restate services status`, and wait until the service is drained (i.e. no ongoing invocations).

**When all prerequisites are fulfilled**, you can remove the deployment containing the service via the [UI](/installation#restate-ui) or via CLI:

```shell theme={null}
restate deployments remove dp_14LsPzGz9HBxXIeBoH5wYUh
```

If the deployment isn't drained yet but you still want to remove it, use the `--force` flag in CLI.


# Tour of Restate for Agents with Google ADK
Source: https://docs.restate.dev/tour/google-adk

Build stateful, observable AI agents that recover from failures.

AI agents are long-running processes that combine LLMs with tools and external APIs to complete complex tasks. With Restate, you can build agents that are **resilient to failures**, **stateful across conversations**, and **observable** without managing complex retry logic or external state stores.

In this guide, you'll learn how to:

* Build durable AI agents that recover automatically from crashes and API failures
* Integrate Restate with the&#x20;
* Observe and debug agent executions with detailed traces
* Implement resilient human-in-the-loop workflows with approvals and timeouts
* Manage conversation history and state across multi-turn interactions
* Orchestrate multiple agents working together on complex tasks

## Getting Started

A Restate AI application has two main components:

* **Restate Server**: The core engine that takes care of the orchestration and resiliency of your agents
* **Agent Services**: Your agent or AI workflow logic using the Restate SDK for durability

<img alt="Application Structure" />

Restate works with how you already deploy your agents, whether that's in Docker, on Kubernetes, or via serverless platforms (Modal, AWS Lambda...). You don't need to run your agents in any special way.

Let's run an example locally to get a better feel for how it works.

### Run the agent

[Install Restate](/installation) and launch it:

```bash theme={null}
restate-server
```

Get the example:

```bash theme={null}
git clone git@github.com:restatedev/ai-examples.git
cd ai-examples/google-adk/tour-of-agents
```

<GitHubLink />

Export your [Google API key](https://aistudio.google.com/app/apikey) and run the agent:

```bash theme={null}
export GOOGLE_API_KEY=your-api-key
uv run .
```

Then, tell Restate where your agent is running via the UI (`http://localhost:9070`) or CLI:

```bash theme={null}
restate deployments register http://localhost:9080
```

This registers a set of agents that we will be covering in this tutorial.

To test your setup, invoke the weather agent, either via the UI playground by clicking on the `run` handler of the `WeatherAgent` in the overview:

<Frame>
  <img alt="Playground" />
</Frame>

Or via `curl`:

```bash theme={null}
curl localhost:8080/WeatherAgent/run \
  --json '{"message": "What is the weather like in San Francisco?", "user_id": "user-123", "session_id": "session-123"}'
```

You should see the weather information printed in the terminal.

Let's have a look at what happened under the hood to make your agents resilient.

## Durable Execution

AI agents make multiple LLM calls and tool executions that can fail due to rate limits, network issues, or service outages.
Restate uses Durable Execution to make your agents withstand failures without losing progress.

The Restate SDK records the steps the agent executes in a log and replays them if the process crashes or is restarted:

<img alt="Durable AI Agent Execution" />

Durable Execution is the basis of how Restate makes your agents resilient to failures.
Restate offers durable execution primitives via its SDK.

## Creating a Durable Agent

To implement a durable agent, you can use the Restate SDK in combination with the Google Agent Development Kit (ADK).

Here's the implementation of the durable weather agent you just invoked:

```python durable_agent.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/google-adk/tour-of-agents/app/durable_agent.py?collapse_imports"}  theme={null}
APP_NAME = "agents"


async def get_weather(city: str) -> WeatherResponse:
    """Get the current weather for a given city."""
    #  Do one or more durable steps using the Restate context
    return await restate_context().run_typed(
        f"Get weather {city}", call_weather_api, city=city
    )


# Specify your agent in the default ADK way
agent = Agent(
    model="gemini-2.5-flash",
    name="weather_agent",
    instruction="You are a helpful agent that provides weather updates.",
    tools=[get_weather],
)

app = App(name=APP_NAME, root_agent=agent, plugins=[RestatePlugin()])
session_service = InMemorySessionService()

agent_service = restate.Service("WeatherAgent")


@agent_service.handler()
async def run(_ctx: restate.Context, req: WeatherPrompt) -> str | None:
    await get_or_create_session(session_service, APP_NAME, req.user_id, req.session_id)
    runner = Runner(app=app, session_service=session_service)
    events = runner.run_async(
        user_id=req.user_id,
        session_id=req.session_id,
        new_message=Content(role="user", parts=[Part.from_text(text=req.message)]),
    )

    final_response = None
    async for event in events:
        if event.is_final_response() and event.content and event.content.parts:
            if event.content.parts[0].text:
                final_response = event.content.parts[0].text
    return final_response
```

<GitHubLink />

First, you implement your agent and its tools, similar to how you would do it with the Google ADK.

With these three additions, you make the agent durable:

1. **Restate Plugin**: Add the `RestatePlugin` to your Google ADK App. This enables Restate's durability features for model calls and tool executions within the agent.
2. **Resilient Tools**: Wrap tool logic in durable steps using the Restate Context. In the `get_weather` tool, the call to the weather API is wrapped in `restate_context().run_typed`, making it a durable step that Restate can retry and recover.

To serve the agent over HTTP with Restate, you create a Restate Service and define handlers.
Here, the agent logic is called from the `run` handler.
The endpoint that serves the agents of this tour over HTTP is defined in [`__main__.py`](https://github.com/restatedev/ai-examples/blob/main/google-adk/tour-of-agents/__main__.py).
The agent can now be called at `http://localhost:8080/WeatherAgent/run`.

<Accordion title="Try out Durable Execution" icon={"laptop"}>
  Ask for the weather in Denver:

  ```bash theme={null}
  curl localhost:8080/WeatherAgent/run \
  --json '{"message": "What is the weather like in Denver?", "user_id": "user-123", "session_id": "session-234"}'
  ```

  On the invocation page in the UI, click on the invocation ID of the failing invocation.
  You can see that your request is retrying because the weather API is down:

  <Frame>
    <img alt="Invocation overview" />
  </Frame>

  To fix the problem, remove the line `fail_on_denver` from the `fetch_weather` function in the `app/utils/utils.py` file:

  ```python utils/utils.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/google-adk/tour-of-agents/app/utils/utils.py#weather"}  theme={null}
  async def call_weather_api(city: str) -> WeatherResponse:
      fail_on_denver(city)
      weather_data = await fetch_weather(city)
      return parse_weather_data(weather_data)
  ```

  Once you restart the service, the workflow finishes successfully.
</Accordion>

## Observing your Agent

As you saw in the previous section, the Restate UI comes in handy when monitoring and debugging your agents.

The Invocations tab shows all agent executions with detailed traces of every LLM call, tool execution, and state change:

<Frame>
  <img alt="Invocation overview" />
</Frame>

<Accordion title="OpenTelemetry Integration">
  Restate supports OpenTelemetry for exporting traces to external systems like Langfuse, DataDog, or Jaeger:

  Have a look at the [tracing docs](/server/monitoring/tracing) to set this up.
</Accordion>

Now that you know how to build and debug an agent, let's look at more advanced patterns.

## Human-in-the-Loop Agent

Many AI agents need human oversight for high-risk decisions or gathering additional input. Restate makes it easy to pause agent execution and wait for human input.

**Benefits with Restate:**

* If the agent crashes while waiting for human input, Restate continues waiting and recovers the promise on another process.
* If the agent runs on function-as-a-service platforms, the Restate SDK lets the function suspend while it's waiting. Once the approval comes in, the Restate Server invokes the function again and lets it resume where it left off. This way, you don't pay for idle waiting time ([Learn more](https://www.restate.dev/blog/resilient-serverless-agents#cost-efficiency-scaling-to-zero-while-waiting)).

Here's a tool that asks for human approval for high-value claims:

```python human_approval_agent.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/google-adk/tour-of-agents/app/human_approval_agent.py#here"}  theme={null}
async def human_approval(claim: InsuranceClaim) -> str:
    """Ask for human approval for high-value claims."""
    # Create an awakeable for human approval
    approval_id, approval_promise = restate_context().awakeable(type_hint=str)

    # Request human review
    await restate_context().run_typed(
        "Request review",
        request_human_review,
        claim=claim,
        awakeable_id=approval_id,
    )

    # Wait for human approval
    return await approval_promise
```

<GitHubLink />

To implement human approval steps, you can use Restate's awakeables. An awakeable is a promise that can be resolved externally via an API call by providing its ID.
When you create the awakeable, you get back an ID and a promise. You can send the ID to the human approver, and then wait for the promise to be resolved.

<Info>
  You can also use awakeables outside of tools, for example, to implement human approval steps in between agent iterations.
</Info>

<Accordion title="Try out human approval" icon={"laptop"}>
  Start a request for a high-value claim that needs human approval.
  Use the playground or `curl` with `/send` to start the claim asynchronously, without waiting for the result.

  ```bash theme={null}
  curl localhost:8080/HumanClaimApprovalAgent/run/send \
  --json '{"message": "Process my hospital bill of 3000USD for a broken leg.", "user_id": "user-123", "session_id": "session-123"}'
  ```

  You can restart the service to see how Restate continues waiting for the approval.

  If you wait for more than a minute, the invocation will get suspended.

  <Frame>
    <img alt="Invocation overview" />
  </Frame>

  Simulate approving the claim by executing the **curl request that was printed in the service logs**, similar to:

  ```bash theme={null}
  curl localhost:8080/restate/awakeables/sign_1M28aqY6ZfuwBmRnmyP/resolve --json 'true'
  ```

  See in the UI how the workflow resumes and finishes after the approval.

  <Frame>
    <img alt="Invocation overview" />
  </Frame>
</Accordion>

<Accordion title="Timeouts and Escalation">
  Add timeouts to human approval steps to prevent workflows from hanging indefinitely.

  Restate persists the timer and the approval promise, so if the service crashes or is restarted, it will continue waiting with the correct remaining time:

  ```python human_approval_agent_with_timeout.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/google-adk/tour-of-agents/app/human_approval_agent_with_timeout.py#here"}  theme={null}
  # Wait for human approval for at most 3 hours to reach our SLA
  match await restate.select(
      approval=approval_promise,
      timeout=restate_context().sleep(timedelta(hours=3)),
  ):
      case ["approval", approved]:
          return "Approved" if approved else "Rejected"
      case _:
          return "Approval timed out - Evaluate with AI"
  ```

  <GitHubLink />

  Try it out by sending a request to the service:

  ```bash theme={null}
  curl localhost:8080/HumanClaimApprovalWithTimeoutsAgent/run/send \
  --json '{"message": "Process my hospital bill of 3000USD for a broken leg.", "user_id": "user-123", "session_id": "session-123"}'
  ```

  You restart the service and check in the UI how the process will block for the remaining time without starting over.

  You can also lower the timeout to a few seconds to see how the timeout path is taken.
</Accordion>

## Resilient workflows as tools

You can pull out complex parts of your tool logic into separate workflows.
This lets you break down complex agents into smaller, reusable components that can be developed, deployed, and scaled independently.

The Restate SDK gives you clients to call these workflows durably from your agent logic.
All calls are proxied via Restate. Restate persists the call and takes care of retries and recovery.

For example, let's implement the human approval tool as a separate service:

```python sub_workflow_agent.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/google-adk/tour-of-agents/app/sub_workflow_agent.py#wf"}  theme={null}
# Sub-workflow service for human approval
human_approval_workflow = restate.Service("HumanApprovalWorkflow")


@human_approval_workflow.handler()
async def review(ctx: restate.Context, claim: InsuranceClaim) -> str:
    """Request human approval for a claim and wait for response."""
    # Create an awakeable that can be resolved via HTTP
    approval_id, approval_promise = ctx.awakeable(type_hint=str)

    # Request human review
    await ctx.run_typed(
        "Request review", request_human_review, claim=claim, awakeable_id=approval_id
    )

    # Wait for human approval
    return await approval_promise
```

<GitHubLink />

This can now be called from the main agent via a service client:

```python sub_workflow_agent.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/google-adk/tour-of-agents/app/sub_workflow_agent.py#here"}  theme={null}
async def human_approval(claim: InsuranceClaim) -> str:
    """Ask for human approval for high-value claims."""
    return await restate_context().service_call(review, claim)
```

<GitHubLink />

These workflows have access to all Restate SDK features, including durable execution, state management, awakeables, and observability.
They can be developed, deployed, and scaled independently.

<Accordion title="Try out sub-workflows" icon={"laptop"}>
  Start a request for a high-value claim that needs human approval.
  Use `/send` to start the claim asynchronously, without waiting for the result.

  ```bash theme={null}
  curl localhost:8080/SubWorkflowClaimApprovalAgent/run/send \
  --json '{"message": "Process my hospital bill of 3000USD for a broken leg.", "user_id": "user-123", "session_id": "session-123"}'
  ```

  In the UI, you can see that the agent called the workflow service and is waiting for the response.
  You can see the trace of the sub-workflow in the timeline.

  Once you approve the claim, the workflow returns, and the agent continues.

  <Frame>
    <img alt="Invocation overview" />
  </Frame>
</Accordion>

<Info>
  Follow the [Tour of Workflows](/tour/workflows) to learn more about implementing resilient workflows with Restate.
</Info>

## Durable Sessions

The next ingredient we need to build AI agents is the ability to maintain context and memory across multiple interactions.

To implement stateful entities like chat sessions, or stateful agents, Restate provides a special service type called Virtual Objects.

When you send a message to a Virtual Object, you provide a unique key that identifies the object instance (for example, a chat session ID or user ID).

Each instance of a Virtual Object maintains isolated state. The handlers of the Virtual Object can read and modify the object's state via the Restate `ObjectContext`.

<img alt="Objects" />

### Virtual Objects for stateful agents

With Google ADK and Restate, you can create stateful agents that maintain conversation history across multiple interactions.

Here is an example of a chat agent represented as a Virtual Object that is keyed by user ID:

```python chat.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/google-adk/tour-of-agents/app/chat.py?collapse_imports"}  theme={null}
APP_NAME = "agents"

agent = Agent(
    model="gemini-2.5-flash",
    name="assistant",
    instruction="You are a helpful assistant. Be concise and helpful.",
)

# Enables retries and recovery for model calls and tool executions
app = App(name=APP_NAME, root_agent=agent, plugins=[RestatePlugin()])
runner = Runner(app=app, session_service=RestateSessionService())

chat = restate.VirtualObject("Chat")


@chat.handler()
async def message(ctx: restate.ObjectContext, req: ChatMessage) -> str | None:
    events = runner.run_async(
        user_id=ctx.key(),
        session_id=req.session_id,
        new_message=Content(role="user", parts=[Part.from_text(text=req.message)]),
    )
    final_response = None
    async for event in events:
        if event.is_final_response() and event.content and event.content.parts:
            if event.content.parts[0].text:
                final_response = event.content.parts[0].text
    return final_response
```

<GitHubLink />

This automatically persists the agent events (LLM calls, tool calls, etc.) and the conversation history in Restate.
It uses Restate as the session provider for the Google ADK Agent.

<Accordion title="Try out Virtual Objects" icon={"laptop"}>
  Ask the agent to do some task and provide a user ID as the object key:

  ```bash theme={null}
  curl localhost:8080/Chat/user123/message \
  --json '{"message": "Make a poem about durable execution.", "session_id": "session-123"}'
  ```

  Continue the conversation with the same user and session ID. The agent will remember previous context:

  ```bash theme={null}
  curl localhost:8080/Chat/user123/message \
  --json '{"message": "Shorten it to 2 lines.", "session_id": "session-123"}'
  ```

  Go to the state tab of the UI to view the conversation history.
</Accordion>

Virtual Objects are ideal for implementing stateful agents because they provide:

* **Long-lived state**: K/V state is stored permanently. It has no automatic expiry. Clear it via `ctx.clear()`.
* **Durable state changes**: State changes are logged with Durable Execution, so they survive failures and are consistent with code execution
* **State is queryable** via the state tab in the UI.

<Frame>
  <img alt="Conversation State Management" />
</Frame>

### Built-in concurrency control

Restate’s Virtual Objects have built-in queuing and consistency guarantees per object key.

You provide the unique key when invoking the Virtual Object, for example, the user ID.
When multiple requests come in for the same object key, Restate automatically queues them and ensures consistency.

<img alt="Queue" />

The semantics are as follows:

* Handlers either have read-write access (`ObjectContext`) or read-only access (shared object context).
* Only one handler with write access can run at a time per object key to prevent concurrent/lost writes or race conditions (for example `message()`).
* Handlers with read-only access can run concurrently to the write-access handlers (for example `get_history()`).

<Accordion title="Try out queuing" icon={"laptop"}>
  Let's send several messages concurrently to different users:

  ```bash theme={null}
  curl localhost:8080/Chat/user123/message/send --json '{"message": "make a poem about durable execution", "session_id": "session-123"}' &
  curl localhost:8080/Chat/user456/message/send --json '{"message": "what are the benefits of durable execution?", "session_id": "session-567"}' &
  curl localhost:8080/Chat/user789/message/send --json '{"message": "how does workflow orchestration work?", "session_id": "session-999"}' &
  curl localhost:8080/Chat/user123/message/send --json '{"message": "can you make it rhyme better?", "session_id": "session-123"}' &
  curl localhost:8080/Chat/user456/message/send --json '{"message": "what about fault tolerance in distributed systems?", "session_id": "session-567"}' &
  curl localhost:8080/Chat/user789/message/send --json '{"message": "give me a practical example", "session_id": "session-999"}' &
  curl localhost:8080/Chat/user101/message/send --json '{"message": "explain event sourcing in simple terms", "session_id": "session-123"}' &
  curl localhost:8080/Chat/user202/message/send --json '{"message": "what is the difference between async and sync processing?", "session_id": "session-123"}'
  ```

  The UI shows how Restate queues the requests per session to ensure consistency:

  <Frame>
    <img alt="Conversation State Management" />
  </Frame>
</Accordion>

<Accordion title="Stateful Serverless Agents">
  You can run Virtual Objects on serverless platforms like Modal, Render, or AWS Lambda.
  When the request comes in, Restate attaches the correct state to the request, so your handler can access it locally.

  This way, you can implement [stateful, serverless agents without managing any external state store](https://www.restate.dev/blog/resilient-serverless-agents#cost-efficiency-scaling-to-zero-while-waiting) and without worrying about concurrency issues.
</Accordion>

### Virtual Objects for storing context

You can store any context information in Virtual Objects, for example, user preferences or the last agent they interacted with.

Use `ctx.set` and `ctx.get` in your handler to store and retrieve state.

Have a look [here for more information](/ai/patterns/sessions-and-chat).

## Resilient multi-agent coordination

As your agents grow more complex, you may want to break them down into smaller, specialized agents that can delegate tasks to each other.

Similar to sub-workflows, you can break down complex agents into multiple specialized agents.

All agents can run in the same process or be deployed independently.

### Agents as tools/handoffs

If you want to share context between agents, run the agents in the same process and use handoffs or tools.

You don't need to do anything special to make this work with Restate:

```python multi_agent.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/google-adk/tour-of-agents/app/multi_agent.py?collapse_imports"}  theme={null}
APP_NAME = "agents"

# AGENTS
# Determine which specialist to use based on claim type
medical_agent = Agent(
    model="gemini-2.5-flash",
    name="medical_specialist",
    description="Reviews medical insurance claims for coverage and necessity.",
    instruction="Review medical claims for coverage and necessity. Approve/deny up to $50,000.",
)

car_agent = Agent(
    model="gemini-2.5-flash",
    name="car_specialist",
    description="Assesses car insurance claims for liability and damage.",
    instruction="Assess car claims for liability and damage. Approve/deny up to $25,000.",
)

agent = Agent(
    model="gemini-2.5-flash",
    name="intake_agent",
    instruction="Route insurance claims to the appropriate specialist",
    sub_agents=[car_agent, medical_agent],
)

# Enables retries and recovery for model calls and tool executions
app = App(name=APP_NAME, root_agent=agent, plugins=[RestatePlugin()])
runner = Runner(app=app, session_service=RestateSessionService())

agent_service = restate.VirtualObject("MultiAgentClaimApproval")


@agent_service.handler()
async def run(ctx: restate.ObjectContext, claim: InsuranceClaim) -> str | None:
    events = runner.run_async(
        user_id=ctx.key(),
        session_id=claim.session_id,
        new_message=Content(
            role="user",
            parts=[Part.from_text(text=f"Claim: {claim.model_dump_json()}")],
        ),
    )

    final_response = None
    async for event in events:
        if event.is_final_response() and event.content and event.content.parts:
            if event.content.parts[0].text:
                final_response = event.content.parts[0].text
    return final_response
```

<GitHubLink />

The execution trace in the Restate UI will allow you to see the full chain of calls between agents and their individual steps.

<Accordion title="Try out multi-agent systems" icon={"laptop"}>
  Start a request for a claim that needs to be analyzed by multiple agents.

  ```bash theme={null}
  curl localhost:8080/MultiAgentClaimApproval/user123/run --json '{
      "amount": 3000,
      "category": "orthopedic",
      "date": "2024-10-01",
      "placeOfService": "General Hospital",
      "reason": "hospital bill for a broken leg",
      "sessionId": "session-123"
  }'
  ```

  In the UI, you can see that the agent called the sub-agents (multiple LLM calls) and is waiting for their responses.
  You can see the trace of the sub-agents in the timeline.

  Once all sub-agents return, the main agent continues and makes a decision.

  <Frame>
    <img alt="Invocation overview" />
  </Frame>
</Accordion>

### Remote agents as tools

If you want to run agents independently, for example, to scale them separately, run them on different platforms, or let them get developed by different teams, then you can call them as tools via service calls.

Restate will proxy all calls, persist them, and will guarantee that they complete successfully.

Your main agent can suspend and save resources while waiting for the remote agent to finish.
Restate invokes your main agent again once the remote agent returns.

```python multi_agent_remote.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/google-adk/tour-of-agents/app/multi_agent_remote.py#here"}  theme={null}
# Durable service call to the fraud agent; persisted and retried by Restate
async def check_fraud(claim: InsuranceClaim) -> str:
    """Analyze the probability of fraud."""
    return await restate_context().service_call(run_fraud_agent, claim)


agent = Agent(
    model="gemini-2.5-flash",
    name="ClaimApprovalCoordinator",
    instruction="You are a claim approval engine. Analyze the claim and use your tools to decide whether to approve it.",
    tools=[check_fraud, check_eligibility],
)

app = App(name=APP_NAME, root_agent=agent, plugins=[RestatePlugin()])
runner = Runner(app=app, session_service=RestateSessionService())

agent_service = restate.VirtualObject("RemoteMultiAgentClaimApproval")


@agent_service.handler()
async def run(ctx: restate.ObjectContext, claim: InsuranceClaim) -> str | None:
    events = runner.run_async(
        user_id=ctx.key(),
        session_id=claim.session_id,
        new_message=Content(
            role="user",
            parts=[Part.from_text(text=f"Claim: {claim.model_dump_json()}")],
        ),
    )

    final_response = None
    async for event in events:
        if event.is_final_response() and event.content and event.content.parts:
            if event.content.parts[0].text:
                final_response = event.content.parts[0].text
    return final_response
```

<GitHubLink />

Note, any shared context between agents needs to be passed explicitly via the input.

The execution trace in the Restate UI will allow you to see the full chain of calls between agents and their individual steps.

<Accordion title="Try out calling a remote agent" icon={"laptop"}>
  Start a request for a claim that needs to be analyzed by multiple agents.

  ```bash theme={null}
  curl localhost:8080/RemoteMultiAgentClaimApproval/user123/run --json '{
      "amount": 3000,
      "category": "orthopedic",
      "date": "2024-10-01",
      "placeOfService": "General Hospital",
      "reason": "hospital bill for a broken leg",
      "sessionId": "session-123"
  }'
  ```

  In the UI, you can see that the agent called the sub-agents and is waiting for their responses.
  You can see the trace of the sub-agents in the timeline.

  Once all sub-agents return, the main agent continues and makes a decision.

  <Frame>
    <img alt="Invocation overview" />
  </Frame>
</Accordion>

<Warning>
  You cannot put both agents within the same Virtual Object, because this leads to deadlocks.
  The main agent would block on the call to the sub-agent, preventing the sub-agent from executing, cause only one handler can run at a time per object key.
</Warning>

## Parallel Work

Now that our agents are broken down into smaller parts, let's have a look at how to run different parts of our agent logic in parallel to speed up execution.

Restate provides primitives that allow you to run tasks concurrently while maintaining deterministic execution during replays.

Most actions on the Restate Context can be composed using `restate.gather` to gather their results or `restate.select` to wait for the first one to complete.

### Parallel Tool Steps

When using the Google ADK with Restate, tool calls are forced to be executed sequentially to ensure deterministic execution during replays.
When multiple tools execute in parallel and use the Restate Context, the order of operations might differ between the original execution and the replay, leading to inconsistencies.

Therefore, the only way to run multiple tool steps in parallel is to implement an orchestrator tool that uses durable execution to run multiple steps in parallel.

Here is an insurance claim agent tool that runs multiple analyses in parallel:

```python parallel_tools.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/google-adk/tour-of-agents/app/parallel_tools.py#here"}  theme={null}
async def calculate_metrics(claim: InsuranceClaim) -> List[str]:
    """Calculate claim metrics using parallel execution."""
    ctx = restate_object_context()

    # Run tools/steps in parallel with durable execution
    results_done = await restate.gather(
        ctx.run_typed("eligibility", check_eligibility, claim=claim),
        ctx.run_typed("cost", compare_to_standard_rates, claim=claim),
        ctx.run_typed("fraud", check_fraud, claim=claim),
    )
    return [await result for result in results_done]
```

<GitHubLink />

Restate makes sure that all parallel tasks are retried and recovered until they succeed.

<Accordion title="Try out parallel tool steps" icon={"laptop"}>
  Start a request for a claim that needs to be analyzed by multiple tools in parallel.

  ```bash theme={null}
  curl localhost:8080/ParallelToolClaimAgent/user123/run --json '{
      "amount": 3000,
      "category": "orthopedic",
      "date": "2024-10-01",
      "placeOfService": "General Hospital",
      "reason": "hospital bill for a broken leg",
      "sessionId": "session-123"
  }'
  ```

  In the UI, you can see that the agent ran the tool steps in parallel.
  Their traces all start at the same time.

  Once all tools return, the agent continues and makes a decision.

  <Frame>
    <img alt="Invocation overview" />
  </Frame>
</Accordion>

### Parallel Agents

You can use the same durable execution primitives to run multiple agents in parallel.

For example, to race agents against each other and [use the first result that returns, while cancelling the others](/ai/patterns/competitive-racing).

Or to let a main orchestrator agent combine the results of multiple specialized agents in parallel:

```python parallel_agents.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/google-adk/tour-of-agents/app/parallel_agents.py#here"}  theme={null}
@agent_service.handler()
async def run(ctx: restate.ObjectContext, claim: InsuranceClaim) -> str | None:

    # Start multiple agents in parallel with auto retries and recovery
    eligibility = ctx.service_call(run_eligibility_agent, claim)
    cost = ctx.service_call(run_rate_comparison_agent, claim)
    fraud = ctx.service_call(run_fraud_agent, claim)

    # Wait for all responses
    await restate.gather(eligibility, cost, fraud)

    # Get the results
    eligibility_result = await eligibility
    cost_result = await cost
    fraud_result = await fraud

    # Run decision agent on outputs
    prompt = f"""Decide about claim: {claim.model_dump_json()}. Assessments:
    Eligibility: {eligibility_result} Cost: {cost_result} Fraud: {fraud_result}"""

    events = runner.run_async(
        user_id=ctx.key(),
        session_id=claim.session_id,
        new_message=Content(role="user", parts=[Part.from_text(text=prompt)]),
    )

    final_response = None
    async for event in events:
        if event.is_final_response() and event.content and event.content.parts:
            if event.content.parts[0].text:
                final_response = event.content.parts[0].text
    return final_response
```

<GitHubLink />

<Accordion title="Try out parallel agents" icon={"laptop"}>
  Start a request for a claim that needs to be analyzed by multiple agents in parallel.

  ```bash theme={null}
  curl localhost:8080/ParallelAgentClaimApproval/user123/run --json '{
      "amount": 3000,
      "category": "orthopedic",
      "date": "2024-10-01",
      "placeOfService": "General Hospital",
      "reason": "hospital bill for a broken leg",
      "sessionId": "session-123"
  }'
  ```

  In the UI, you can see that the handler called the sub-agents in parallel.
  Once all sub-agents return, the main agent makes a decision.

  <Frame>
    <img alt="Invocation overview" />
  </Frame>
</Accordion>

## Error Handling

LLM calls are costly, so you can configure retry behavior in both Restate and your AI SDK to avoid infinite loops and high costs.

Restate distinguishes between two types of errors:

* **Transient errors**: Temporary issues like network failures or rate limits. Restate automatically retries these until they succeed or the retry policy is exhausted.
* **Terminal errors**: Permanent failures like invalid input or business rule violations. Restate does not retry these. The invocation fails permanently. You can catch these errors and handle them gracefully.

You can throw a terminal error via:

```python theme={null}
from restate import TerminalError

raise TerminalError("This tool is not allowed to run for this input.")
```

You can catch and handle terminal errors in your agent logic if needed.

Many AI SDKs also have their own retry behavior for LLM calls and tool executions, so let's look at how these interact.

### Retries of LLM calls

Configure the number of retries for LLM calls when activating the Restate plugin for your ADK App:

```python error_handling.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/google-adk/tour-of-agents/app/error_handling.py#retries"}  theme={null}
app = App(
    name=APP_NAME, root_agent=agent, plugins=[RestatePlugin(max_model_call_retries=3)]
)
```

<GitHubLink />

By default, the runner retries ten times with an initial interval of one second. Once Restate's retries are exhausted, the invocation fails with a `TerminalError` and won't be retried further.

### Tool execution errors

Restate retries tool executions by default until they succeed.
For errors which should not be retried, you can raise terminal errors from within your tool implementations.

You can catch these terminal errors in your handler and handle them accordingly.

```python error_handling.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/google-adk/tour-of-agents/app/error_handling.py#handle"}  theme={null}
@agent_service.handler()
async def run(_ctx: restate.Context, req: WeatherPrompt) -> str | None:
    try:
        await get_or_create_session(
            session_service, APP_NAME, req.user_id, req.session_id
        )
        runner = Runner(app=app, session_service=session_service)
        events = runner.run_async(
            user_id=req.user_id,
            session_id=req.session_id,
            new_message=Content(role="user", parts=[Part.from_text(text=req.message)]),
        )

        final_response = None
        async for event in events:
            if event.is_final_response() and event.content and event.content.parts:
                if event.content.parts[0].text:
                    final_response = event.content.parts[0].text
        return final_response
    except TerminalError as e:
        # Handle the error appropriately, e.g., log it or return a default response
        print(f"An error occurred: {e}")
        return "Sorry, I'm unable to process your request at the moment."
```

<Info>
  You can set [custom retry policies](/guides/error-handling#at-the-run-block-level) for `.run` actions in your tool executions.
</Info>

## Summary

Durable Execution, paired with your existing SDKs, gives your agents a powerful upgrade:

* **Durable Execution**: Automatic recovery from failures without losing progress
* **Persistent memory and context**: Persistent conversation history and context
* **Observability** by default across your agents and workflows
* **Human-in-the-Loop**: Seamless approval workflows with timeouts
* **Multi-Agent Coordination**: Reliable orchestration of specialized agents
* **Suspensions** to save costs on function-as-a-service platforms when agents need to wait
* **Advanced Patterns**: Real-time progress updates, interruptions, and long-running workflows

## Next Steps

* [AI Agent documentation](/ai): for advanced patterns and other integrations.
* [Restate AI examples repository](https://github.com/restatedev/ai-examples)
* Sign up for [Restate Cloud](https://restate.dev/cloud/) and start building agents without managing infrastructure


# Microservice Orchestration
Source: https://docs.restate.dev/tour/microservice-orchestration

Learn how to orchestrate microservices with durable execution, sagas, and async communication patterns.

Microservice orchestration is about coordinating multiple services to complete complex business workflows. Restate provides powerful primitives for building resilient, observable orchestration patterns.

In this guide, you'll learn how to:

* Build durable, fault-tolerant service orchestrations with automatic failure recovery
* Implement sagas for distributed transactions with resilient compensation
* Use durable timers and external events for complex async patterns
* Implement stateful entities with Virtual Objects

Select your SDK:

<GlobalTabs>
  <GlobalTab title="TypeScript" />

  <GlobalTab title="Java" />

  <GlobalTab title="Go" />

  <GlobalTab title="Python" />
</GlobalTabs>

## Getting Started

A Restate application is composed of two main components:

* **Restate Server**: The core engine that manages durable execution and orchestrates services. It acts as a message broker or reverse proxy in front of your services.
* **Your Services**: Your business logic, implemented as service handlers using the Restate SDK to perform durable operations.

<img alt="Application Structure" />

A basic subscription service orchestration looks like this:

<GlobalTabs>
  <GlobalTab title="TypeScript">
    ```ts src/getstarted/service.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/tutorials/tour-of-orchestration-typescript/src/getstarted/service.ts?collapse_prequel"}  theme={null}
    export const subscriptionService = restate.service({
      name: "SubscriptionService",
      handlers: {
        add: async (ctx: Context, req: SubscriptionRequest) => {
          const paymentId = ctx.rand.uuidv4();

          const payRef = await ctx.run("pay", () =>
            createRecurringPayment(req.creditCard, paymentId),
          );

          for (const subscription of req.subscriptions) {
            await ctx.run(`add-${subscription}`, () =>
              createSubscription(req.userId, subscription, payRef),
            );
          }
        },
      },
    });
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Java">
    ```java getstarted/SubscriptionService.java {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/tutorials/tour-of-orchestration-java/src/main/java/my/example/getstarted/SubscriptionService.java?collapse_prequel"}  theme={null}
    @Service
    public class SubscriptionService {

      @Handler
      public void add(Context ctx, SubscriptionRequest req) {
        var paymentId = ctx.random().nextUUID().toString();

        String payRef =
            ctx.run("pay", String.class, () -> createRecurringPayment(req.creditCard(), paymentId));

        for (String subscription : req.subscriptions()) {
          ctx.run("add-" + subscription, () -> createSubscription(req.userId(), subscription, payRef));
        }
      }
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Go">
    ```go getstarted.go {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/tutorials/tour-of-orchestration-go/examples/getstarted.go?collapse_prequel"}  theme={null}
    type SubscriptionService struct{}

    func (SubscriptionService) Add(ctx restate.Context, req SubscriptionRequest) error {
      paymentId := restate.UUID(ctx).String()

      payRef, err := restate.Run(ctx, func(ctx restate.RunContext) (string, error) {
        return CreateRecurringPayment(req.CreditCard, paymentId)
      }, restate.WithName("pay"))
      if err != nil {
        return err
      }

      for _, subscription := range req.Subscriptions {
        _, err := restate.Run(ctx, func(ctx restate.RunContext) (string, error) {
          return CreateSubscription(req.UserId, subscription, payRef)
        }, restate.WithName(fmt.Sprintf("add-%s", subscription)))
        if err != nil {
          return err
        }
      }

      return nil
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Python">
    ```python app/getstarted/service.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/python/tutorials/tour-of-orchestration-python/app/getstarted/service.py?collapse_prequel"}  theme={null}
    subscription_service = restate.Service("SubscriptionService")


    @subscription_service.handler()
    async def add(ctx: restate.Context, req: SubscriptionRequest) -> None:
        payment_id = str(ctx.uuid())

        pay_ref = await ctx.run_typed(
            "pay",
            create_recurring_payment,
            credit_card=req.credit_card,
            payment_id=payment_id,
        )

        for subscription in req.subscriptions:
            await ctx.run_typed(
                f"add-{subscription}",
                create_subscription,
                user_id=req.user_id,
                subscription=subscription,
                payment_ref=pay_ref,
            )
    ```

    <GitHubLink />
  </GlobalTab>
</GlobalTabs>

A service has handlers that can be called over HTTP. Each handler receives a `Context` object that provides durable execution primitives. Any action performed with the Context is automatically recorded and can survive failures.

You don't need to run your services in any special way. Restate works with how you already deploy your code, whether that's in Docker, on Kubernetes, or via AWS Lambda.

<GlobalTabs>
  <GlobalTab title="TypeScript">
    The endpoint that serves the services of this tour over HTTP is defined in `src/app.ts`.
  </GlobalTab>

  <GlobalTab title="Java">
    The endpoint that serves the services of this tour over HTTP is defined in `AppMain.java`.
  </GlobalTab>

  <GlobalTab title="Go">
    The endpoint that serves the services of this tour over HTTP is defined in `main.go`.
  </GlobalTab>

  <GlobalTab title="Python">
    The endpoint that serves the services of this tour over HTTP is defined in `__main__.py`.
  </GlobalTab>
</GlobalTabs>

### Run the example

[Install Restate](/installation) and launch it:

```bash theme={null}
restate-server
```

<GlobalTabs>
  <GlobalTab title="TypeScript">
    Get the example:

    ```bash theme={null}
    restate example typescript-tour-of-orchestration && cd typescript-tour-of-orchestration
    npm install
    ```

    <GitHubLink />

    Run the example:

    ```bash theme={null}
    npm run dev
    ```

    Then, tell Restate where your services are running via the UI (`http://localhost:9070`) or CLI:

    ```bash theme={null}
    restate deployments register http://localhost:9080
    ```

    This registers a set of services that we will be covering in this tutorial.

    To invoke a handler, send a request to `restate-ingress/MyServiceName/handlerName`:

    ```bash theme={null}
    curl localhost:8080/SubscriptionService/add \
    --json '{"userId": "user-123", "creditCard": "4111111111111111", "subscriptions": ["Hulu", "Prime"]}'
    ```
  </GlobalTab>

  <GlobalTab title="Java">
    Get the example:

    ```bash theme={null}
    restate example java-tour-of-orchestration && cd java-tour-of-orchestration
    ```

    <GitHubLink />

    Run the example:

    ```bash theme={null}
    ./gradlew run
    ```

    Then, tell Restate where your services are running via the UI (`http://localhost:9070`) or CLI:

    ```bash theme={null}
    restate deployments register http://localhost:9080
    ```

    This registers a set of services that we will be covering in this tutorial.

    To invoke a handler, send a request to `restate-ingress/MyServiceName/handlerName`:

    ```bash theme={null}
    curl localhost:8080/SubscriptionService/add \
    --json '{"userId": "user-123", "creditCard": "4111111111111111", "subscriptions": ["Hulu", "Prime"]}'
    ```
  </GlobalTab>

  <GlobalTab title="Go">
    Get the example:

    ```bash theme={null}
    restate example go-tour-of-orchestration && cd go-tour-of-orchestration
    ```

    <GitHubLink />

    Run the example:

    ```bash theme={null}
    go run .
    ```

    Then, tell Restate where your services are running via the UI (`http://localhost:9070`) or CLI:

    ```bash theme={null}
    restate deployments register http://localhost:9080
    ```

    This registers a set of services that we will be covering in this tutorial.

    To invoke a handler, send a request to `restate-ingress/MyServiceName/handlerName`:

    ```bash theme={null}
    curl localhost:8080/SubscriptionService/Add \
    --json '{"userId": "user-123", "creditCard": "4111111111111111", "subscriptions": ["Hulu", "Prime"]}'
    ```
  </GlobalTab>

  <GlobalTab title="Python">
    Get the example:

    ```bash theme={null}
    restate example python-tour-of-orchestration && cd python-tour-of-orchestration
    ```

    <GitHubLink />

    Run the example:

    ```bash theme={null}
    uv run .
    ```

    Then, tell Restate where your services are running via the UI (`http://localhost:9070`) or CLI:

    ```bash theme={null}
    restate deployments register http://localhost:9080
    ```

    This registers a set of services that we will be covering in this tutorial.

    To invoke a handler, send a request to `restate-ingress/MyServiceName/handlerName`:

    ```bash theme={null}
    curl localhost:8080/SubscriptionService/add \
    --json '{"userId": "user-123", "creditCard": "4111111111111111", "subscriptions": ["Hulu", "Prime"]}'
    ```
  </GlobalTab>
</GlobalTabs>

Click in the UI's invocations tab on the inovcation ID of your request to see the execution trace of your request.

<Frame>
  <img alt="Invocation overview" />
</Frame>

## Durable Execution

Restate uses Durable Execution to ensure your orchestration logic survives failures and restarts.
Whenever a handler executes an action with the Restate `Context`, this gets send over to the Restate Server and persisted in a log.

On a failure or a crash, the Restate Server sends a retry request that contains the log of the actions that were executed so far.
The service then replays the log to restore state and continues executing the remaining actions.
This process continues until the handler runs till completion.

<img alt="Context in Restate" />

**Key Benefits:**

* Context `run` actions make external calls or non-deterministic operations durable. They get replayed on failures.
* If the service crashes after payment creation, it resumes at the subscription step
* Deterministic IDs logged with the context ensure operations are idempotent
* Full execution traces for debugging and monitoring

<Accordion title="Try out Durable Execution" icon={"laptop"}>
  <GlobalTabs>
    <GlobalTab title="TypeScript">
      Try to add a subscription for Netflix:

      ```bash theme={null}
      curl localhost:8080/SubscriptionService/add \
      --json '{"userId": "user-123", "creditCard": "4111111111111111", "subscriptions": ["Hulu", "Prime", "Netflix"]}'
      ```

      On the invocation page in the UI, you can see that your request is retrying because the Netflix API is down:

      <Frame>
        <img alt="Invocation overview" />
      </Frame>

      To fix the problem, remove the line `failOnNetflix` from the `createSubscription` function in the `utils.ts` file:

      ```ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/tutorials/tour-of-orchestration-typescript/src/utils.ts#subscription"}  theme={null}
      export function createSubscription(
        userId: string,
        subscription: string,
        _paymentRef: string,
      ): string {
        failOnNetflix(subscription);
        terminalErrorOnDisney(subscription);
        console.log(`>>> Created subscription ${subscription} for user ${userId}`);
        return "SUCCESS";
      }
      ```
    </GlobalTab>

    <GlobalTab title="Java">
      Try to add a subscription for Netflix:

      ```bash theme={null}
      curl localhost:8080/SubscriptionService/add \
      --json '{"userId": "user-123", "creditCard": "4111111111111111", "subscriptions": ["Hulu", "Prime", "Netflix"]}'
      ```

      On the invocation page in the UI, you can see that your request is retrying because the Netflix API is down:

      <Frame>
        <img alt="Invocation overview" />
      </Frame>

      To fix the problem, remove the line `failOnNetflix` from the `createSubscription` function in the `auxiliary/clients/SubscriptionClient.java` file:

      ```java {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/tutorials/tour-of-orchestration-java/src/main/java/my/example/auxiliary/clients/SubscriptionClient.java#subscription"}  theme={null}
      public static String createSubscription(String userId, String subscription, String paymentRef) {
        failOnNetflix(subscription);
        terminalErrorOnDisney(subscription);
        System.out.println(">>> Created subscription " + subscription + " for user " + userId);
        return "SUCCESS";
      }
      ```
    </GlobalTab>

    <GlobalTab title="Go">
      Try to add a subscription for Netflix:

      ```bash theme={null}
      curl localhost:8080/SubscriptionService/Add \
      --json '{"userId": "user-123", "creditCard": "4111111111111111", "subscriptions": ["Hulu", "Prime", "Netflix"]}'
      ```

      On the invocation page in the UI, you can see that your request is retrying because the Netflix API is down:

      <Frame>
        <img alt="Invocation overview" />
      </Frame>

      To fix the problem, remove the line `failOnNetflix` from the `CreateSubscription` function in the `utils.go` file:

      ```go {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/tutorials/tour-of-orchestration-go/examples/utils.go#subscription"}  theme={null}
      func CreateSubscription(userId, subscription, paymentRef string) (string, error) {
        if err := failOnNetflix(subscription); err != nil {
          return "", err
        }
        if err := terminalErrorOnDisney(subscription); err != nil {
          return "", err
        }
        fmt.Printf(">>> Created subscription %s for user %s\n", subscription, userId)
        return "SUCCESS", nil
      }
      ```
    </GlobalTab>

    <GlobalTab title="Python">
      Try to add a subscription for Netflix:

      ```bash theme={null}
      curl localhost:8080/SubscriptionService/add \
      --json '{"userId": "user-123", "creditCard": "4111111111111111", "subscriptions": ["Hulu", "Prime", "Netflix"]}'
      ```

      On the invocation page in the UI, you can see that your request is retrying because the Netflix API is down:

      <Frame>
        <img alt="Invocation overview" />
      </Frame>

      To fix the problem, remove the line `fail_on_netflix` from the `create_subscription` function in the `utils.py` file:

      ```python {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/python/tutorials/tour-of-orchestration-python/app/utils.py#subscription"}  theme={null}
      def create_subscription(user_id: str, subscription: str, payment_ref: str) -> str:
          fail_on_netflix(subscription)
          terminal_error_on_disney(subscription)
          print(f">>> Created subscription {subscription} for user {user_id}")
          return "SUCCESS"
      ```
    </GlobalTab>
  </GlobalTabs>

  Once you restart the service, the workflow finishes successfully:

  <Frame>
    <img alt="Invocation overview" />
  </Frame>
</Accordion>

## Error Handling

By default, Restate retries failures infinitely with an exponential backoff strategy.
For some failures, you might not want to retry or only retry a limited number of times.

For these cases, Restate distinguishes between two types of errors:

* **Transient Errors**: These are temporary issues that can be retried, such as network timeouts or service unavailability. Restate automatically retries these errors.
* **Terminal Errors**: These indicate a failure that will not be retried, such as invalid input or business logic violations. Restate stops execution and allows you to handle these errors gracefully.

Throw a terminal error in your handler to indicate a terminal failure:

<GlobalTabs>
  <GlobalTab title="TypeScript">
    ```typescript {"CODE_LOAD::ts/src/tour/microservices/terminal_error.ts#terminal_error"}  theme={null}
    throw new TerminalError("Invalid credit card");
    ```
  </GlobalTab>

  <GlobalTab title="Java">
    ```java {"CODE_LOAD::java/src/main/java/tour/microservices/ErrorHandler.java#here"}  theme={null}
    throw new TerminalException("Invalid credit card");
    ```
  </GlobalTab>

  <GlobalTab title="Go">
    ```go {"CODE_LOAD::go/tour/microservices/errorhandling.go#here"}  theme={null}
    return restate.TerminalError(fmt.Errorf("invalid credit card"))
    ```
  </GlobalTab>

  <GlobalTab title="Python">
    ```python {"CODE_LOAD::python/src/tour/microservices/terminal_error.py#here"}  theme={null}
    from restate.exceptions import TerminalError

    raise TerminalError("Invalid credit card")
    ```
  </GlobalTab>
</GlobalTabs>

<Accordion title="Configuring Retry Behavior">
  Some actions let you configure their retry behavior, for example to limit the number of retries of a run block:

  <GlobalTabs>
    <GlobalTab title="TypeScript">
      ```ts {"CODE_LOAD::ts/src/tour/microservices/retries.ts#retries"}  theme={null}
      const retryPolicy = {
        maxRetryAttempts: 3,
        initialRetryIntervalMillis: 1000,
      };

      const payRef = await ctx.run(
        "pay",
        () => createRecurringPayment(req.creditCard, paymentId),
        retryPolicy
      );
      ```
    </GlobalTab>

    <GlobalTab title="Java">
      ```java {"CODE_LOAD::java/src/main/java/tour/microservices/Retries.java#here"}  theme={null}
      RetryPolicy myRunRetryPolicy =
          RetryPolicy.defaultPolicy().setInitialDelay(Duration.ofSeconds(1)).setMaxAttempts(3);
      String payRef =
          ctx.run(
              "pay",
              String.class,
              myRunRetryPolicy,
              () -> createRecurringPayment(req.creditCard(), paymentId));
      ```
    </GlobalTab>

    <GlobalTab title="Go">
      ```go {"CODE_LOAD::go/tour/microservices/retries.go#here"}  theme={null}
      result, err := restate.Run(ctx,
        func(ctx restate.RunContext) (string, error) {
          return createRecurringPayment(req.CreditCard, paymentId)
        },
        restate.WithInitialRetryInterval(time.Millisecond*100),
        restate.WithMaxRetryAttempts(3),
        restate.WithName("pay"),
      )
      if err != nil {
        return err
      }
      ```
    </GlobalTab>

    <GlobalTab title="Python">
      ```python {"CODE_LOAD::python/src/tour/microservices/retries.py#here"}  theme={null}
      pay_ref = await ctx.run_typed(
          "pay",
          lambda: create_recurring_payment(req["creditCard"], payment_id),
          restate.RunOptions(
              max_attempts=10, max_retry_duration=timedelta(seconds=30)
          ),
      )
      ```
    </GlobalTab>
  </GlobalTabs>

  When the retries are exhausted, the run block will throw a `TerminalError`, that you can handle in your handler logic.
</Accordion>

Learn more with the [Error Handling Guide](/guides/error-handling).

## Sagas and Rollback

On a terminal failure, Restate stops the execution of the handler.
You might, however, want to roll back the changes made by the workflow to keep your system in a consistent state.
This is where Sagas come in.

Sagas are a pattern for rolling back changes made by a handler when it fails.

In Restate, you can implement a saga by building a list of compensating actions for each step of the workflow.
On a terminal failure, you execute them in reverse order:

<GlobalTabs>
  <GlobalTab title="TypeScript">
    ```ts src/sagas/service.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/tutorials/tour-of-orchestration-typescript/src/sagas/service.ts?collapse_prequel"}  theme={null}
    export const subscriptionSaga = restate.service({
      name: "SubscriptionSaga",
      handlers: {
        add: async (ctx: Context, req: SubscriptionRequest) => {
          const compensations = [];

          try {
            const paymentId = ctx.rand.uuidv4();
            compensations.push(() =>
              ctx.run("undo-pay", () => removeRecurringPayment(paymentId)),
            );
            const payRef = await ctx.run("pay", () =>
              createRecurringPayment(req.creditCard, paymentId),
            );

            for (const subscription of req.subscriptions) {
              compensations.push(() =>
                ctx.run(`undo-${subscription}`, () =>
                  removeSubscription(req.userId, subscription),
                ),
              );
              await ctx.run(`add-${subscription}`, () =>
                createSubscription(req.userId, subscription, payRef),
              );
            }
          } catch (e) {
            if (e instanceof restate.TerminalError) {
              for (const compensation of compensations.reverse()) {
                await compensation();
              }
            }
            throw e;
          }
        },
      },
    });
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Java">
    ```java sagas/SubscriptionSaga.java {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/tutorials/tour-of-orchestration-java/src/main/java/my/example/sagas/SubscriptionSaga.java?collapse_prequel"}  theme={null}
    @Service
    public class SubscriptionSaga {

      @Handler
      public void add(Context ctx, SubscriptionRequest req) {
        List<Runnable> compensations = new ArrayList<>();
        try {
          var paymentId = ctx.random().nextUUID().toString();

          compensations.add(
              () -> ctx.run("undo-pay", () -> PaymentClient.removeRecurringPayment(paymentId)));
          String payRef =
              ctx.run(
                  "pay",
                  String.class,
                  () -> PaymentClient.createRecurringPayment(req.creditCard(), paymentId));

          for (String subscription : req.subscriptions()) {
            compensations.add(
                () ->
                    ctx.run(
                        "undo-" + subscription,
                        () -> SubscriptionClient.removeSubscription(req.userId(), subscription)));
            ctx.run(
                "add-" + subscription,
                () -> SubscriptionClient.createSubscription(req.userId(), subscription, payRef));
          }
        } catch (TerminalException e) {
          // Run compensations in reverse order
          Collections.reverse(compensations);
          for (Runnable compensation : compensations) {
            compensation.run();
          }
          throw e;
        }
      }
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Go">
    ```go sagas.go {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/tutorials/tour-of-orchestration-go/examples/sagas.go?collapse_prequel"}  theme={null}
    type SubscriptionSaga struct{}

    func (SubscriptionSaga) Add(ctx restate.Context, req SubscriptionRequest) (err error) {
      var compensations []func() error

      // Run compensations at the end if err != nil
      defer func() {
        if err != nil {
          for _, compensation := range slices.Backward(compensations) {
            if compErr := compensation(); compErr != nil {
              err = compErr
            }
          }
        }
      }()

      paymentId := restate.UUID(ctx).String()

      // Add compensation for payment
      compensations = append(compensations, func() error {
        _, err := restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
          return RemoveRecurringPayment(paymentId)
        }, restate.WithName("undo-pay"))
        return err
      })

      // Create payment
      payRef, err := restate.Run(ctx, func(ctx restate.RunContext) (string, error) {
        return CreateRecurringPayment(req.CreditCard, paymentId)
      }, restate.WithName("pay"))
      if err != nil {
        return err
      }

      // Process subscriptions
      for _, subscription := range req.Subscriptions {
        // Add compensation for this subscription
        sub := subscription // Capture loop variable
        compensations = append(compensations, func() error {
          _, err := restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
            return RemoveSubscription(req.UserId, sub)
          }, restate.WithName(fmt.Sprintf("undo-%s", sub)))
          return err
        })

        // Create subscription
        _, err := restate.Run(ctx, func(ctx restate.RunContext) (string, error) {
          return CreateSubscription(req.UserId, subscription, payRef)
        }, restate.WithName(fmt.Sprintf("add-%s", subscription)))
        if err != nil {
          return err
        }
      }

      return nil
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Python">
    ```python app/sagas/service.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/python/tutorials/tour-of-orchestration-python/app/sagas/service.py?collapse_prequel"}  theme={null}
    subscription_saga = restate.Service("SubscriptionSaga")


    @subscription_saga.handler()
    async def add(ctx: restate.Context, req: SubscriptionRequest) -> None:
        compensations = []

        try:
            payment_id = str(ctx.uuid())

            # Add compensation for payment
            compensations.append(
                lambda: ctx.run_typed(
                    "undo-pay", remove_recurring_payment, payment_id=payment_id
                )
            )

            # Create payment
            pay_ref = await ctx.run_typed(
                "pay",
                create_recurring_payment,
                credit_card=req.credit_card,
                payment_id=payment_id,
            )

            # Process subscriptions
            for subscription in req.subscriptions:
                # Add compensation for this subscription
                compensations.append(
                    lambda s=subscription: ctx.run_typed( # type: ignore
                        f"undo-{s}",
                        remove_subscription,
                        user_id=req.user_id,
                        subscription=s,
                    )
                )

                # Create subscription
                await ctx.run_typed(
                    f"add-{subscription}",
                    create_subscription,
                    user_id=req.user_id,
                    subscription=subscription,
                    payment_ref=pay_ref,
                )

        except restate.TerminalError as e:
            # Run compensations in reverse order
            for compensation in reversed(compensations):
                await compensation()
            raise e
    ```

    <GitHubLink />
  </GlobalTab>
</GlobalTabs>

**Benefits with Restate:**

* The list of compensations can be recovered after a crash, and Restate knows which compensations still need to be run.
* Sagas always run till completion (success or complete rollback)
* Full trace of all operations and compensations
* No complex state machines needed

<Accordion title="Try out sagas" icon={"laptop"}>
  Add a subscription for Disney:

  <GlobalTabs>
    <GlobalTab title="TypeScript">
      ```bash theme={null}
      curl localhost:8080/SubscriptionSaga/add \
      --json '{"userId": "user-123", "creditCard": "4111111111111111", "subscriptions": ["Hulu", "Prime", "Disney"]}'
      ```
    </GlobalTab>

    <GlobalTab title="Java">
      ```bash theme={null}
      curl localhost:8080/SubscriptionSaga/add \
      --json '{"userId": "user-123", "creditCard": "4111111111111111", "subscriptions": ["Hulu", "Prime", "Disney"]}'
      ```
    </GlobalTab>

    <GlobalTab title="Go">
      ```bash theme={null}
      curl localhost:8080/SubscriptionSaga/Add \
      --json '{"userId": "user-123", "creditCard": "4111111111111111", "subscriptions": ["Hulu", "Prime", "Disney"]}'
      ```
    </GlobalTab>

    <GlobalTab title="Python">
      ```bash theme={null}
      curl localhost:8080/SubscriptionSaga/add \
      --json '{"userId": "user-123", "creditCard": "4111111111111111", "subscriptions": ["Hulu", "Prime", "Disney"]}'
      ```
    </GlobalTab>
  </GlobalTabs>

  The Disney subscription is not available, so the handler will fail and run compensations:

  <Frame>
    <img alt="Sagas" />
  </Frame>
</Accordion>

Learn more with the [Sagas Guide](/guides/sagas).

## Virtual Objects

Until now, the services we looked at did not share any state between requests.

To implement stateful entities like shopping carts, user profiles, or AI agents, Restate provides **Virtual Objects**.

Each Virtual Object instance maintains isolated state and is identified by a unique key.

Here is an example of a Virtual Object that tracks user subscriptions:

<img alt="Objects" />

<GlobalTabs>
  <GlobalTab title="TypeScript">
    ```ts src/objects/service.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/tutorials/tour-of-orchestration-typescript/src/objects/service.ts?collapse_prequel"}  theme={null}
    export const userSubscriptions = restate.object({
      name: "UserSubscriptions",
      handlers: {
        add: async (ctx: ObjectContext, subscription: string) => {
          // Get current subscriptions
          const subscriptions = (await ctx.get<string[]>("subscriptions")) ?? [];

          // Add new subscription
          if (!subscriptions.includes(subscription)) {
            subscriptions.push(subscription);
          }
          ctx.set("subscriptions", subscriptions);

          // Update metrics
          ctx.set("lastUpdated", await ctx.date.toJSON());
        },

        getSubscriptions: restate.handlers.object.shared(
          async (ctx: ObjectSharedContext) => {
            return (await ctx.get<string[]>("subscriptions")) ?? [];
          },
        ),
      },
    });
    ```

    <GitHubLink />

    Virtual Objects are ideal for implementing any entity with mutable state:

    * **Long-lived state**: K/V state is stored permanently. It has no automatic expiry. Clear it via `ctx.clear()`.
    * **Durable state changes**: State changes are logged with Durable Execution, so they survive failures and are consistent with code execution
    * **State is queryable** via the state tab in the UI:

    <Frame>
      <img alt="State" />
    </Frame>

    * **Built-in concurrency control**: Restate’s Virtual Objects have built-in queuing and consistency guarantees per object key. Handlers either have read-write access (`ObjectContext`) or read-only access (shared object context).
    * Only one handler with write access can run at a time per object key to prevent concurrent/lost writes or race conditions.
    * Handlers with read-only access can run concurrently to the write-access handlers.

    <img alt="Queue" />
  </GlobalTab>

  <GlobalTab title="Java">
    ```java objects/UserSubscriptions.java {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/tutorials/tour-of-orchestration-java/src/main/java/my/example/objects/UserSubscriptions.java?collapse_prequel"}  theme={null}
    @VirtualObject
    public class UserSubscriptions {
      private static final StateKey<Set<String>> SUBSCRIPTIONS =
          StateKey.of("subscriptions", new TypeRef<>() {});
      private static final StateKey<String> LAST_UPDATED = StateKey.of("lastUpdated", String.class);

      @Handler
      public void add(ObjectContext ctx, String subscription) {
        // Get current subscriptions
        Set<String> subscriptions = ctx.get(SUBSCRIPTIONS).orElse(new HashSet<>());

        // Add new subscription
        subscriptions.add(subscription);
        ctx.set(SUBSCRIPTIONS, subscriptions);

        // Update metrics
        ctx.set(LAST_UPDATED, Instant.now().toString());
      }

      @Shared
      public Set<String> getSubscriptions(SharedObjectContext ctx) {
        return ctx.get(SUBSCRIPTIONS).orElse(Set.of());
      }
    }
    ```

    <GitHubLink />

    Virtual Objects are ideal for implementing any entity with mutable state:

    * **Long-lived state**: K/V state is stored permanently. It has no automatic expiry. Clear it via `ctx.clear()`.
    * **Durable state changes**: State changes are logged with Durable Execution, so they survive failures and are consistent with code execution
    * **State is queryable** via the state tab in the UI:

    <Frame>
      <img alt="State" />
    </Frame>

    * **Built-in concurrency control**: Restate’s Virtual Objects have built-in queuing and consistency guarantees per object key. Handlers either have read-write access (`ObjectContext`) or read-only access (shared object context).
    * Only one handler with write access can run at a time per object key to prevent concurrent/lost writes or race conditions.
    * Handlers with read-only access can run concurrently to the write-access handlers.

    <img alt="Queue" />
  </GlobalTab>

  <GlobalTab title="Go">
    ```go objects.go {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/tutorials/tour-of-orchestration-go/examples/objects.go?collapse_prequel"}  theme={null}
    type UserSubscriptions struct{}

    func (UserSubscriptions) Add(ctx restate.ObjectContext, subscription string) error {
      // Get current subscriptions
      subscriptions, err := restate.Get[[]string](ctx, "subscriptions")
      if err != nil {
        return err
      }
      if subscriptions == nil {
        subscriptions = []string{}
      }

      // Add new subscription if not already present
      found := false
      for _, sub := range subscriptions {
        if sub == subscription {
          found = true
          break
        }
      }
      if !found {
        subscriptions = append(subscriptions, subscription)
      }

      // Save subscriptions
      restate.Set(ctx, "subscriptions", subscriptions)

      // Update metrics
      restate.Set(ctx, "lastUpdated", time.Now().Format(time.RFC3339))

      return nil
    }

    func (UserSubscriptions) GetSubscriptions(ctx restate.ObjectSharedContext) ([]string, error) {
      return restate.Get[[]string](ctx, "subscriptions")
    }
    ```

    <GitHubLink />

    Virtual Objects are ideal for implementing any entity with mutable state:

    * **Long-lived state**: K/V state is stored permanently. It has no automatic expiry. Clear it via `restate.Clear(ctx, "my-key")`.
    * **Durable state changes**: State changes are logged with Durable Execution, so they survive failures and are consistent with code execution
    * **State is queryable** via the state tab in the UI:

    <Frame>
      <img alt="State" />
    </Frame>

    * **Built-in concurrency control**: Restate’s Virtual Objects have built-in queuing and consistency guarantees per object key. Handlers either have read-write access (`ObjectContext`) or read-only access (shared object context).
    * Only one handler with write access can run at a time per object key to prevent concurrent/lost writes or race conditions.
    * Handlers with read-only access can run concurrently to the write-access handlers.

    <img alt="Queue" />
  </GlobalTab>

  <GlobalTab title="Python">
    ```python app/objects/service.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/python/tutorials/tour-of-orchestration-python/app/objects/service.py?collapse_prequel"}  theme={null}
    user_subscriptions = restate.VirtualObject("UserSubscriptions")


    @user_subscriptions.handler()
    async def add(ctx: restate.ObjectContext, subscription: str) -> None:
        # Get current subscriptions
        subscriptions = await ctx.get("subscriptions", type_hint=List[str]) or []

        # Add new subscription
        if subscription not in subscriptions:
            subscriptions.append(subscription)

        ctx.set("subscriptions", subscriptions)

        # Update metrics
        ctx.set("lastUpdated", datetime.now().isoformat())


    @user_subscriptions.handler("getSubscriptions")
    async def get_subscriptions(ctx: restate.ObjectSharedContext) -> List[str]:
        return await ctx.get("subscriptions", type_hint=List[str]) or []
    ```

    <GitHubLink />

    Virtual Objects are ideal for implementing any entity with mutable state:

    * **Long-lived state**: K/V state is stored permanently. It has no automatic expiry. Clear it via `ctx.clear()`.
    * **Durable state changes**: State changes are logged with Durable Execution, so they survive failures and are consistent with code execution
    * **State is queryable** via the state tab in the UI:

    <Frame>
      <img alt="State" />
    </Frame>

    * **Built-in concurrency control**: Restate’s Virtual Objects have built-in queuing and consistency guarantees per object key. Handlers either have read-write access (`ObjectContext`) or read-only access (shared object context).
    * Only one handler with write access can run at a time per object key to prevent concurrent/lost writes or race conditions.
    * Handlers with read-only access can run concurrently to the write-access handlers.

    <img alt="Queue" />
  </GlobalTab>
</GlobalTabs>

<Accordion title="Try out Virtual Objects" icon={"laptop"}>
  Add a few subscriptions for some users.
  To call a Virtual Object, you specify the object key in the URL (here `user-123` and `user-456`):

  <GlobalTabs>
    <GlobalTab title="TypeScript">
      ```bash theme={null}
      curl localhost:8080/UserSubscriptions/user-123/add --json '"Hulu"'
      curl localhost:8080/UserSubscriptions/user-123/add --json '"Prime"'
      curl localhost:8080/UserSubscriptions/user-123/add --json '"Disney"'
      curl localhost:8080/UserSubscriptions/user-456/add --json '"Netflix"'
      ```

      Get the subscriptions for `user-123`:

      ```bash theme={null}
      curl localhost:8080/UserSubscriptions/user-123/getSubscriptions
      ```
    </GlobalTab>

    <GlobalTab title="Java">
      ```bash theme={null}
      curl localhost:8080/UserSubscriptions/user-123/add --json '"Hulu"'
      curl localhost:8080/UserSubscriptions/user-123/add --json '"Prime"'
      curl localhost:8080/UserSubscriptions/user-123/add --json '"Disney"'
      curl localhost:8080/UserSubscriptions/user-456/add --json '"Netflix"'
      ```

      Get the subscriptions for `user-123`:

      ```bash theme={null}
      curl localhost:8080/UserSubscriptions/user-123/getSubscriptions
      ```
    </GlobalTab>

    <GlobalTab title="Go">
      ```bash theme={null}
      curl localhost:8080/UserSubscriptions/user-123/Add --json '"Hulu"'
      curl localhost:8080/UserSubscriptions/user-123/Add --json '"Prime"'
      curl localhost:8080/UserSubscriptions/user-123/Add --json '"Disney"'
      curl localhost:8080/UserSubscriptions/user-456/Add --json '"Netflix"'
      ```

      Get the subscriptions for `user-123`:

      ```bash theme={null}
      curl localhost:8080/UserSubscriptions/user-123/GetSubscriptions
      ```
    </GlobalTab>

    <GlobalTab title="Python">
      ```bash theme={null}
      curl localhost:8080/UserSubscriptions/user-123/add --json '"Hulu"'
      curl localhost:8080/UserSubscriptions/user-123/add --json '"Prime"'
      curl localhost:8080/UserSubscriptions/user-123/add --json '"Disney"'
      curl localhost:8080/UserSubscriptions/user-456/add --json '"Netflix"'
      ```

      Get the subscriptions for `user-123`:

      ```bash theme={null}
      curl localhost:8080/UserSubscriptions/user-123/getSubscriptions
      ```
    </GlobalTab>
  </GlobalTabs>

  Or use the UI's state tab to explore the object state.
</Accordion>

## Resilient Communication

The Restate SDK includes clients to call other handlers reliably. You can call another handler in three ways:

* **Request-Response**: Wait for a response
* **One-Way Messages**: Fire-and-forget
* **Delayed Messages**: Schedule for later

When you call another handler, the Restate Server acts as a message broker.
All communication is proxied via the Restate Server where it gets durably logged and retried till completion.

Imagine a handler which processes a concert ticket purchase, and calls multiple services to handle payment, ticket delivery, and reminders:

<GlobalTabs>
  <GlobalTab title="TypeScript">
    ```ts src/communication/service.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/tutorials/tour-of-orchestration-typescript/src/communication/service.ts?collapse_prequel"}  theme={null}
    export const concertTicketingService = restate.service({
      name: "ConcertTicketingService",
      handlers: {
        buy: async (ctx: Context, req: PurchaseTicketRequest) => {
          // Request-response call - wait for payment to complete
          const payRef = await ctx.serviceClient(paymentService).charge(req);

          // One-way message - fire and forget ticket delivery
          ctx.serviceSendClient(emailService).emailTicket(req);

          // Delayed message - schedule reminder for day before concert
          ctx
            .serviceSendClient(emailService)
            .sendReminder(req, sendOpts({ delay: dayBefore(req.concertDate) }));

          return `Ticket purchased successfully with payment reference: ${payRef}`;
        },
      },
    });
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Java">
    ```java communication/ConcertTicketingService.java {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/tutorials/tour-of-orchestration-java/src/main/java/my/example/communication/ConcertTicketingService.java?collapse_prequel"}  theme={null}
    @Service
    public class ConcertTicketingService {

      @Handler
      public String buy(Context ctx, PurchaseTicketRequest req) {
        // Request-response call - wait for payment to complete
        String payRef = PaymentServiceClient.fromContext(ctx).charge(req).await();

        // One-way message - fire and forget ticket delivery
        EmailServiceClient.fromContext(ctx).send().emailTicket(req);

        // Delayed message - schedule reminder for day before concert
        EmailServiceClient.fromContext(ctx).send().sendReminder(req, req.dayBefore());

        return "Ticket purchased successfully with payment reference: " + payRef;
      }
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Go">
    ```go communication.go {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/tutorials/tour-of-orchestration-go/examples/communication.go?collapse_prequel"}  theme={null}
    type ConcertTicketingService struct{}

    func (ConcertTicketingService) Buy(ctx restate.Context, req PurchaseTicketRequest) (string, error) {
      // Request-response call - wait for payment to complete
      payRef, err := restate.Service[string](ctx, "PaymentService", "Charge").Request(req)
      if err != nil {
        return "", err
      }

      // One-way message - fire and forget ticket delivery
      restate.Service[restate.Void](ctx, "EmailService", "EmailTicket").Send(req)

      // Delayed message - schedule reminder for day before concert
      delay := DayBefore(req.ConcertDate)
      restate.Service[restate.Void](ctx, "EmailService", "SendReminder").
        Send(req, restate.WithDelay(delay))

      return fmt.Sprintf("Ticket purchased successfully with payment reference: %s", payRef), nil
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Python">
    ```python app/communication/service.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/python/tutorials/tour-of-orchestration-python/app/communication/service.py?collapse_prequel"}  theme={null}
    concert_ticketing_service = restate.Service("ConcertTicketingService")


    @concert_ticketing_service.handler()
    async def buy(ctx: restate.Context, req: PurchaseTicketRequest) -> str:
        # Request-response call - wait for payment to complete
        pay_ref = await ctx.service_call(charge, req)

        # One-way message - fire and forget ticket delivery
        ctx.service_send(email_ticket, req)

        # Delayed message - schedule reminder for day before concert
        ctx.service_send(send_reminder_email, req, send_delay=day_before(req.concert_date))

        return f"Ticket purchased successfully with payment reference: {pay_ref}"
    ```

    <GitHubLink />
  </GlobalTab>
</GlobalTabs>

Each of these calls gets persisted in Restate's log and will be retried upon failures.
The handler can finish execution without waiting for the ticket delivery or reminder to complete.

You can use Restate's communication primitives to implement microservices that communicate reliably and scale independently.

<Accordion title="Try out resilient communication" icon={"laptop"}>
  Buy a concert ticket:

  <GlobalTabs>
    <GlobalTab title="TypeScript">
      ```bash theme={null}
      curl localhost:8080/ConcertTicketingService/buy --json '{
      "ticketId": "ticket-789",
      "price": 100,
      "customerEmail": "me@mail.com",
      "concertDate": "2026-10-01T20:00:00Z"
      }'
      ```
    </GlobalTab>

    <GlobalTab title="Java">
      ```bash theme={null}
      curl localhost:8080/ConcertTicketingService/buy --json '{
      "ticketId": "ticket-789",
      "price": 100,
      "customerEmail": "me@mail.com",
      "concertDate": "2026-10-01T20:00:00Z"
      }'
      ```
    </GlobalTab>

    <GlobalTab title="Go">
      ```bash theme={null}
      curl localhost:8080/ConcertTicketingService/Buy --json '{
      "ticketId": "ticket-789",
      "price": 100,
      "customerEmail": "me@mail.com",
      "concertDate": "2026-10-01T20:00:00Z"
      }'
      ```
    </GlobalTab>

    <GlobalTab title="Python">
      ```bash theme={null}
      curl localhost:8080/ConcertTicketingService/buy --json '{
      "ticketId": "ticket-789",
      "price": 100,
      "customerEmail": "me@mail.com",
      "concertDate": "2026-10-01T20:00:00Z"
      }'
      ```
    </GlobalTab>
  </GlobalTabs>

  See in the UI how the first call had the response logged, while the ticket delivery happened asynchronously and the reminder was scheduled for in 406 days:

  <Frame>
    <img alt="Communication" />
  </Frame>
</Accordion>

## Request Idempotency

Restate allows adding an idempotency header to your requests. It will then deduplicate requests with the same idempotency key, ensuring that they only execute once.

This can help us prevent duplicate calls the concert ticketing service if the user accidentally clicks "buy" multiple times.

Add an idempotency header to your request:

<GlobalTabs>
  <GlobalTab title="TypeScript">
    ```shell theme={null}
    curl -X POST localhost:8080/ConcertTicketingService/buy \
    -H 'Idempotency-Key: unique-key-123' \
    --json '{"ticketId": "ticket-789", "price": 100, "customerEmail": "me@mail.com", "concertDate": "2023-10-01T20:00:00Z"}'
    ```
  </GlobalTab>

  <GlobalTab title="Java">
    ```shell theme={null}
    curl -X POST localhost:8080/ConcertTicketingService/buy \
    -H 'Idempotency-Key: unique-key-123' \
    --json '{"ticketId": "ticket-789", "price": 100, "customerEmail": "me@mail.com", "concertDate": "2023-10-01T20:00:00Z"}'
    ```
  </GlobalTab>

  <GlobalTab title="Go">
    ```shell theme={null}
    curl -X POST localhost:8080/ConcertTicketingService/Buy \
    -H 'Idempotency-Key: unique-key-123' \
    --json '{"ticketId": "ticket-789", "price": 100, "customerEmail": "me@mail.com", "concertDate": "2023-10-01T20:00:00Z"}'
    ```
  </GlobalTab>

  <GlobalTab title="Python">
    ```shell theme={null}
    curl -X POST localhost:8080/ConcertTicketingService/buy \
    -H 'Idempotency-Key: unique-key-123' \
    --json '{"ticketId": "ticket-789", "price": 100, "customerEmail": "me@mail.com", "concertDate": "2023-10-01T20:00:00Z"}'
    ```
  </GlobalTab>
</GlobalTabs>

Notice how doing the same request with the same idempotency key will print the same payment reference.
Instead of executing the handler again, Restate returns the result of the first execution.

## External Events

Until now we showed either synchronous API calls via `run` or calls to other Restate services.

Another common scenario is APIs that respond asynchronously via webhooks or callbacks.
For this, you can use Restate's awakeables.

For example, some payment providers like Stripe require you to initiate a payment and then wait for their webhook to confirm the transaction.

<GlobalTabs>
  <GlobalTab title="TypeScript">
    ```ts src/events/service.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/tutorials/tour-of-orchestration-typescript/src/events/service.ts?collapse_prequel"}  theme={null}
    export const payments = restate.service({
      name: "Payments",
      handlers: {
        process: async (ctx: Context, req: PaymentRequest) => {
          // Create awakeable to wait for webhook payment confirmation
          const confirmation = ctx.awakeable<PaymentResult>();

          // Initiate payment with external provider (Stripe, PayPal, etc.)
          const paymentId = ctx.rand.uuidv4();
          await ctx.run("pay", () => initPayment(req, paymentId, confirmation.id));

          // Wait for external payment provider to call our webhook
          return confirmation.promise;
        },

        // Webhook handler called by external payment provider
        confirm: async (
          ctx: Context,
          confirmation: { id: string; result: PaymentResult },
        ) => {
          // Resolve the awakeable to continue the payment flow
          ctx.resolveAwakeable(confirmation.id, confirmation.result);
        },
      },
    });
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Java">
    ```java events/Payments.java {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/tutorials/tour-of-orchestration-java/src/main/java/my/example/events/Payments.java?collapse_prequel"}  theme={null}
    @Service
    public class Payments {

      @Handler
      public PaymentResult process(Context ctx, PaymentRequest req) {
        // Create awakeable to wait for webhook payment confirmation
        var confirmation = ctx.awakeable(PaymentResult.class);

        // Initiate payment with external provider (Stripe, PayPal, etc.)
        var paymentId = ctx.random().nextUUID().toString();
        ctx.run(() -> initPayment(req, paymentId, confirmation.id()));

        // Wait for external payment provider to call our webhook
        return confirmation.await();
      }

      // Webhook handler called by external payment provider
      @Handler
      public void confirm(Context ctx, ConfirmationRequest confirmation) {
        // Resolve the awakeable to continue the payment flow
        ctx.awakeableHandle(confirmation.id()).resolve(PaymentResult.class, confirmation.result());
      }
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Go">
    ```go events.go {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/tutorials/tour-of-orchestration-go/examples/events.go?collapse_prequel"}  theme={null}
    type Payments struct{}

    func (Payments) Process(ctx restate.Context, req PaymentRequest) (PaymentResult, error) {
      // Create awakeable to wait for webhook payment confirmation
      confirmation := restate.Awakeable[PaymentResult](ctx)

      // Initiate payment with external provider (Stripe, PayPal, etc.)
      paymentId := restate.UUID(ctx).String()
      _, err := restate.Run(ctx, func(ctx restate.RunContext) (string, error) {
        return InitPayment(req, paymentId, confirmation.Id())
      }, restate.WithName("pay"))
      if err != nil {
        return PaymentResult{}, err
      }

      // Wait for external payment provider to call our webhook
      return confirmation.Result()
    }

    // Webhook handler called by external payment provider
    func (Payments) Confirm(ctx restate.Context, confirmation ConfirmationRequest) error {
      // Resolve the awakeable to continue the payment flow
      restate.ResolveAwakeable(ctx, confirmation.Id, confirmation.Result)
      return nil
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Python">
    ```python app/events/service.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/python/tutorials/tour-of-orchestration-python/app/events/service.py?collapse_prequel"}  theme={null}
    payments = restate.Service("Payments")


    @payments.handler()
    async def process(ctx: restate.Context, req: PaymentRequest) -> PaymentResult:

        # Create awakeable to wait for webhook payment confirmation
        confirmation_id, confirmation_promise = ctx.awakeable(type_hint=PaymentResult)

        # Initiate payment with external provider (Stripe, PayPal, etc.)
        payment_id = str(ctx.uuid())
        await ctx.run_typed(
            "pay",
            init_payment,
            req=req,
            payment_id=payment_id,
            confirmation_id=confirmation_id,
        )

        # Wait for external payment provider to call our webhook
        return await confirmation_promise


    @payments.handler()
    async def confirm(ctx: restate.Context, confirmation: ConfirmationRequest) -> None:
        # Resolve the awakeable to continue the payment flow
        ctx.resolve_awakeable(confirmation.id, confirmation.result)
    ```

    <GitHubLink />
  </GlobalTab>
</GlobalTabs>

Awakeables are like promises or futures that can be recovered after a crash.
Restate persists the awakeable in its log and can recover it on another process when needed.

There is no limit to how long you can persist an awakeable, so you can wait for external events that may take hours, or even months to arrive.

You can also use awakeables to implement human-in-the-loop interactions, such as waiting for user input or approvals.

<Accordion title="Try out external events" icon={"laptop"}>
  Initiate a payment by calling the `process` handler. Add `/send` to call the handler without waiting for the response:

  <GlobalTabs>
    <GlobalTab title="TypeScript">
      ```bash theme={null}
      curl localhost:8080/Payments/process/send \
      --json '{"amount": 100, "currency": "USD", "customerId": "cust-123", "orderId": "order-456"}'
      ```
    </GlobalTab>

    <GlobalTab title="Java">
      ```bash theme={null}
      curl localhost:8080/Payments/process/send \
      --json '{"amount": 100, "currency": "USD", "customerId": "cust-123", "orderId": "order-456"}'
      ```
    </GlobalTab>

    <GlobalTab title="Go">
      ```bash theme={null}
      curl localhost:8080/Payments/Process/send \
      --json '{"amount": 100, "currency": "USD", "customerId": "cust-123", "orderId": "order-456"}'
      ```
    </GlobalTab>

    <GlobalTab title="Python">
      ```bash theme={null}
      curl localhost:8080/Payments/process/send \
      --json '{"amount": 100, "currency": "USD", "customerId": "cust-123", "orderId": "order-456"}'
      ```
    </GlobalTab>
  </GlobalTabs>

  In the UI, you can see that the payment is waiting for confirmation.

  You can restart the service to see how Restate continues waiting for the payment confirmation.

  Simulate approving the payment by executing the **curl request that was printed in the service logs**, similar to:

  <GlobalTabs>
    <GlobalTab title="TypeScript">
      ```bash theme={null}
      curl localhost:8080/Payments/confirm \
      --json '{"id": "sign_1PrDkbECjgdsBmMfEUQyCnioCP-csLbd2AAAAEQ", "result": {"success": true, "transactionId": "txn-123"}}'
      ```
    </GlobalTab>

    <GlobalTab title="Java">
      ```bash theme={null}
      curl localhost:8080/Payments/confirm \
      --json '{"id": "sign_1PrDkbECjgdsBmMfEUQyCnioCP-csLbd2AAAAEQ", "result": {"success": true, "transactionId": "txn-123"}}'
      ```
    </GlobalTab>

    <GlobalTab title="Go">
      ```bash theme={null}
      curl localhost:8080/Payments/Confirm \
      --json '{"id": "sign_1PrDkbECjgdsBmMfEUQyCnioCP-csLbd2AAAAEQ", "result": {"success": true, "transactionId": "txn-123"}}'
      ```
    </GlobalTab>

    <GlobalTab title="Python">
      ```bash theme={null}
      curl localhost:8080/Payments/confirm \
      --json '{"id": "sign_1PrDkbECjgdsBmMfEUQyCnioCP-csLbd2AAAAEQ", "result": {"success": true, "transactionId": "txn-123"}}'
      ```
    </GlobalTab>
  </GlobalTabs>

  You can see in the UI that the payment was processed successfully and the awakeable was resolved:

  <Frame>
    <img alt="Awakeables" />
  </Frame>
</Accordion>

## Durable Timers

Waiting on external events might take a long time, and you might want to add timeouts to operations like this.

The Restate SDK offers durable timer implementations that you can use to limit waiting for an action.
Restate tracks these timers so they survive crashes and do not restart from the beginning.

Let's extend our payment service to automatically cancel payments that don't complete within a reasonable time:

<GlobalTabs>
  <GlobalTab title="TypeScript">
    ```ts src/timers/service.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/tutorials/tour-of-orchestration-typescript/src/timers/service.ts?collapse_prequel"}  theme={null}
    export const paymentsWithTimeout = restate.service({
      name: "PaymentsWithTimeout",
      handlers: {
        process: async (ctx: Context, req: PaymentRequest) => {
          const confirmation = ctx.awakeable<PaymentResult>();

          const paymentId = ctx.rand.uuidv4();
          const payRef = await ctx.run("pay", () =>
            initPayment(req, paymentId, confirmation.id),
          );

          // Race between payment confirmation and timeout
          try {
            return await confirmation.promise.orTimeout({ seconds: 30 });
          } catch (e) {
            if (e instanceof TimeoutError) {
              // Cancel the payment with external provider
              await ctx.run("cancel-payment", () => cancelPayment(payRef));
              return {
                success: false,
                errorMessage: "Payment timeout",
              };
            }
            throw e;
          }
        },

        confirm: async (
          ctx: Context,
          confirmation: { id: string; result: PaymentResult },
        ) => {
          ctx.resolveAwakeable(confirmation.id, confirmation.result);
        },
      },
    });
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Java">
    ```java timers/PaymentsWithTimeout.java {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/tutorials/tour-of-orchestration-java/src/main/java/my/example/timers/PaymentsWithTimeout.java?collapse_prequel"}  theme={null}
    @Service
    public class PaymentsWithTimeout {

      @Handler
      public PaymentResult process(Context ctx, PaymentRequest req) {
        var confirmation = ctx.awakeable(PaymentResult.class);

        var paymentId = ctx.random().nextUUID().toString();
        String payRef =
            ctx.run("pay", String.class, () -> initPayment(req, paymentId, confirmation.id()));

        try {
          return confirmation.await(Duration.ofSeconds(30));
        } catch (TimeoutException e) {
          ctx.run("cancel-payment", () -> cancelPayment(payRef));
          return new PaymentResult(false, null, "Payment timeout");
        }
      }

      @Handler
      public void confirm(Context ctx, ConfirmationRequest confirmation) {
        ctx.awakeableHandle(confirmation.id()).resolve(PaymentResult.class, confirmation.result());
      }
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Go">
    ```go timers.go {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/tutorials/tour-of-orchestration-go/examples/timers.go?collapse_prequel"}  theme={null}
    type PaymentsWithTimeout struct{}

    func (PaymentsWithTimeout) Process(ctx restate.Context, req PaymentRequest) (PaymentResult, error) {
      confirmation := restate.Awakeable[PaymentResult](ctx)

      paymentId := restate.UUID(ctx).String()
      payRef, err := restate.Run(ctx, func(ctx restate.RunContext) (string, error) {
        return InitPayment(req, paymentId, confirmation.Id())
      }, restate.WithName("pay"))
      if err != nil {
        return PaymentResult{}, err
      }

      // Race between payment confirmation and timeout
      timeout := restate.After(ctx, 30*time.Second)
      resFut, err := restate.WaitFirst(ctx, confirmation, timeout)
      if err != nil {
        return PaymentResult{}, err
      }

      switch resFut {
      case confirmation:
        return confirmation.Result()
      default:
        if err := timeout.Done(); err != nil {
          return PaymentResult{}, err
        }
        // Cancel the payment with external provider
        _, err := restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
          return CancelPayment(payRef)
        }, restate.WithName("cancel-payment"))
        if err != nil {
          return PaymentResult{}, err
        }

        return PaymentResult{
          Success:      false,
          ErrorMessage: "Payment timeout",
        }, nil
      }
    }

    func (PaymentsWithTimeout) Confirm(ctx restate.Context, confirmation ConfirmationRequest) error {
      restate.ResolveAwakeable(ctx, confirmation.Id, confirmation.Result)
      return nil
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Python">
    ```python app/timers/service.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/python/tutorials/tour-of-orchestration-python/app/timers/service.py?collapse_prequel"}  theme={null}
    payments_with_timeout = restate.Service("PaymentsWithTimeout")


    @payments_with_timeout.handler()
    async def process(ctx: restate.Context, req: PaymentRequest) -> PaymentResult:
        confirmation_id, confirmation_promise = ctx.awakeable(type_hint=PaymentResult)

        payment_id = str(ctx.uuid())
        pay_ref = await ctx.run_typed(
            "pay",
            init_payment,
            req=req,
            payment_id=payment_id,
            confirmation_id=confirmation_id,
        )

        # Race between payment confirmation and timeout
        match await restate.select(
            confirmation=confirmation_promise, timeout=ctx.sleep(timedelta(seconds=30))
        ):
            case ["confirmation", result]:
                return result
            case _:
                # Cancel the payment with external provider
                await ctx.run_typed("cancel-payment", cancel_payment, pay_ref=pay_ref)
                return PaymentResult(
                    success=False, transaction_id=None, error_message="Payment timeout"
                )


    @payments_with_timeout.handler()
    async def confirm(ctx: restate.Context, confirmation: ConfirmationRequest) -> None:
        ctx.resolve_awakeable(confirmation.id, confirmation.result)
    ```

    <GitHubLink />
  </GlobalTab>
</GlobalTabs>

You can also set timeouts for RPC calls or other asynchronous operations with the Restate SDK.

<Accordion title="Try out durable timers" icon={"laptop"}>
  Initiate a payment by calling the `process` handler. Add `/send` to call the handler without waiting for the response:

  <GlobalTabs>
    <GlobalTab title="TypeScript">
      ```bash theme={null}
      curl localhost:8080/PaymentsWithTimeout/process/send \
      --json '{"amount": 100, "currency": "USD", "customerId": "cust-123", "orderId": "order-456"}'
      ```
    </GlobalTab>

    <GlobalTab title="Java">
      ```bash theme={null}
      curl localhost:8080/PaymentsWithTimeout/process/send \
      --json '{"amount": 100, "currency": "USD", "customerId": "cust-123", "orderId": "order-456"}'
      ```
    </GlobalTab>

    <GlobalTab title="Go">
      ```bash theme={null}
      curl localhost:8080/PaymentsWithTimeout/Process/send \
      --json '{"amount": 100, "currency": "USD", "customerId": "cust-123", "orderId": "order-456"}'
      ```
    </GlobalTab>

    <GlobalTab title="Python">
      ```bash theme={null}
      curl localhost:8080/PaymentsWithTimeout/process/send \
      --json '{"amount": 100, "currency": "USD", "customerId": "cust-123", "orderId": "order-456"}'
      ```
    </GlobalTab>
  </GlobalTabs>

  Wait for 30 seconds without confirming the payment.

  Try restarting the service while the payment is waiting for confirmation to see how Restate continues waiting for the timer and the confirmation.

  In the UI, you can see that the payment times out and cancels the payment:

  <Frame>
    <img alt="Timers" />
  </Frame>
</Accordion>

## Concurrent Tasks

When you are waiting on an awakeable or a timer, you are effectively running concurrent tasks and waiting for one of them to complete.

Restate allows more advanced concurrency patterns to run tasks in parallel and wait for their results.

Let's extend our subscription service to process all subscriptions concurrently and handle failures gracefully:

<GlobalTabs>
  <GlobalTab title="TypeScript">
    ```ts src/concurrenttasks/service.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/tutorials/tour-of-orchestration-typescript/src/concurrenttasks/service.ts?collapse_prequel"}  theme={null}
    export const parallelSubscriptionService = restate.service({
      name: "ParallelSubscriptionService",
      handlers: {
        add: async (ctx: Context, req: SubscriptionRequest) => {
          const paymentId = ctx.rand.uuidv4();
          const payRef = await ctx.run("pay", () =>
            createRecurringPayment(req.creditCard, paymentId),
          );

          // Start all subscriptions in parallel
          const subscriptionPromises = [];
          for (const subscription of req.subscriptions) {
            subscriptionPromises.push(
              ctx.run(`add-${subscription}`, () =>
                createSubscription(req.userId, subscription, payRef),
              ),
            );
          }

          // Wait for all subscriptions to complete
          await RestatePromise.all(subscriptionPromises);

          return { success: true, paymentRef: payRef };
        },
      },
    });
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Java">
    ```java concurrenttasks/ParallelSubscriptionService.java {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/tutorials/tour-of-orchestration-java/src/main/java/my/example/concurrenttasks/ParallelSubscriptionService.java?collapse_prequel"}  theme={null}
    @Service
    public class ParallelSubscriptionService {

      @Handler
      public SubscriptionResult add(Context ctx, SubscriptionRequest req) {
        var paymentId = ctx.random().nextUUID().toString();
        var payRef =
            ctx.run("pay", String.class, () -> createRecurringPayment(req.creditCard(), paymentId));

        // Start all subscriptions in parallel
        List<DurableFuture<?>> subscriptionFutures = new ArrayList<>();
        for (String subscription : req.subscriptions()) {
          subscriptionFutures.add(
              ctx.runAsync(
                  "add-" + subscription, () -> createSubscription(req.userId(), subscription, payRef)));
        }

        // Wait for all subscriptions to complete
        DurableFuture.all(subscriptionFutures).await();

        return new SubscriptionResult(true, payRef);
      }
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Go">
    ```go concurrenttasks.go {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/tutorials/tour-of-orchestration-go/examples/concurrenttasks.go?collapse_prequel"}  theme={null}
    type ParallelSubscriptionService struct{}

    func (ParallelSubscriptionService) Add(ctx restate.Context, req SubscriptionRequest) (SubscriptionResult, error) {
      paymentId := restate.UUID(ctx).String()

      payRef, err := restate.Run(ctx, func(ctx restate.RunContext) (string, error) {
        return CreateRecurringPayment(req.CreditCard, paymentId)
      }, restate.WithName("pay"))
      if err != nil {
        return SubscriptionResult{}, err
      }

      // Process all subscriptions sequentially
      var subscriptionFutures []restate.Future
      for _, subscription := range req.Subscriptions {
        future := restate.RunAsync(ctx, func(ctx restate.RunContext) (string, error) {
          return CreateSubscription(req.UserId, subscription, payRef)
        }, restate.WithName(fmt.Sprintf("add-%s", subscription)))
        subscriptionFutures = append(subscriptionFutures, future)
      }

      for fut, err := range restate.Wait(ctx, subscriptionFutures...) {
        if err != nil {
          return SubscriptionResult{}, err
        }
        _, err := fut.(restate.RunAsyncFuture[string]).Result()
        if err != nil {
          return SubscriptionResult{}, err
        }
      }

      return SubscriptionResult{
        Success:    true,
        PaymentRef: payRef,
      }, nil
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Python">
    ```python app/concurrenttasks/service.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/python/tutorials/tour-of-orchestration-python/app/concurrenttasks/service.py?collapse_prequel"}  theme={null}
    parallel_subscription_service = restate.Service("ParallelSubscriptionService")


    @parallel_subscription_service.handler()
    async def add(ctx: restate.Context, req: SubscriptionRequest) -> SubscriptionResult:
        payment_id = str(ctx.uuid())
        pay_ref = await ctx.run_typed(
            "pay",
            create_recurring_payment,
            credit_card=req.credit_card,
            payment_id=payment_id,
        )

        # Start all subscriptions in parallel
        subscription_tasks = []
        for subscription in req.subscriptions:
            task = ctx.run_typed(
                f"add-{subscription}",
                create_subscription,
                user_id=req.user_id,
                subscription=subscription,
                payment_ref=pay_ref,
            )
            subscription_tasks.append(task)

        # Wait for all subscriptions to complete
        await restate.gather(*subscription_tasks)

        return SubscriptionResult(success=True, payment_ref=pay_ref)
    ```

    <GitHubLink />
  </GlobalTab>
</GlobalTabs>

Restate retries all parallel tasks until they all complete and can deterministically replay the order of completion.

<Accordion title="Try out concurrent tasks" icon={"laptop"}>
  Add a few subscriptions for some users.

  <GlobalTabs>
    <GlobalTab title="TypeScript">
      ```bash theme={null}
      curl localhost:8080/ParallelSubscriptionService/add \
      --json '{"userId": "user-123", "creditCard": "4111111111111111", "subscriptions": ["Hulu", "Prime", "YouTube"]}'
      ```
    </GlobalTab>

    <GlobalTab title="Java">
      ```bash theme={null}
      curl localhost:8080/ParallelSubscriptionService/add \
      --json '{"userId": "user-123", "creditCard": "4111111111111111", "subscriptions": ["Hulu", "Prime", "YouTube"]}'
      ```
    </GlobalTab>

    <GlobalTab title="Go">
      ```bash theme={null}
      curl localhost:8080/ParallelSubscriptionService/Add \
      --json '{"userId": "user-123", "creditCard": "4111111111111111", "subscriptions": ["Hulu", "Prime", "YouTube"]}'
      ```
    </GlobalTab>

    <GlobalTab title="Python">
      ```bash theme={null}
      curl localhost:8080/ParallelSubscriptionService/add \
      --json '{"userId": "user-123", "creditCard": "4111111111111111", "subscriptions": ["Hulu", "Prime", "YouTube"]}'
      ```
    </GlobalTab>
  </GlobalTabs>

  In the UI, you can see that all subscriptions are processed in parallel:

  <Frame>
    <img alt="Concurrent Tasks" />
  </Frame>
</Accordion>

You can extend this to include the saga pattern and run all compensations in parallel as well.

Have a look at the Concurrent Tasks docs for your SDK to learn more ([TS](/develop/ts/concurrent-tasks) / [Java / Kotlin](/develop/java/concurrent-tasks) / [Python](/develop/python/concurrent-tasks) / [Go](/develop/go/concurrent-tasks)).

## Summary

Restate simplifies microservice orchestration with:

* **Durable Execution**: Automatic failure recovery without complex retry logic
* **Sagas**: Distributed transactions with resilient compensation
* **Service Communication**: Reliable RPC and messaging between services
* **Stateful Processing**: Consistent state management without external stores
* **Advanced Patterns**: Fault-tolerant timers, awakeables, and parallel execution

Build resilient distributed systems without the typical complexity.


# Tour of Restate for Agents with OpenAI SDK
Source: https://docs.restate.dev/tour/openai-agents

Build stateful, observable AI agents that recover from failures.

AI agents are long-running processes that combine LLMs with tools and external APIs to complete complex tasks. With Restate, you can build agents that are **resilient to failures**, **stateful across conversations**, and **observable** without managing complex retry logic or external state stores.

In this guide, you'll learn how to:

* Build durable AI agents that recover automatically from crashes and API failures
* Integrate Restate with the&#x20;
* Observe and debug agent executions with detailed traces
* Implement resilient human-in-the-loop workflows with approvals and timeouts
* Manage conversation history and state across multi-turn interactions
* Orchestrate multiple agents working together on complex tasks

## Getting Started

A Restate AI application has two main components:

* **Restate Server**: The core engine that takes care of the orchestration and resiliency of your agents
* **Agent Services**: Your agent or AI workflow logic using the Restate SDK for durability

<img alt="Application Structure" />

Restate works with how you already deploy your agents, whether that's in Docker, on Kubernetes, or via serverless platforms (Modal, AWS Lambda...). You don't need to run your agents in any special way.

Let's run an example locally to get a better feel for how it works.

### Run the agent

[Install Restate](/installation) and launch it:

```bash theme={null}
restate-server
```

Get the example:

```bash theme={null}
git clone git@github.com:restatedev/ai-examples.git
cd ai-examples/openai-agents/tour-of-agents
```

<GitHubLink />

Export your [OpenAI API key](https://platform.openai.com/api-keys) and run the agent:

```bash theme={null}
export OPENAI_API_KEY=sk-...
uv run .
```

Then, tell Restate where your agent is running via the UI (`http://localhost:9070`) or CLI:

```bash theme={null}
restate deployments register http://localhost:9080
```

This registers a set of agents that we will be covering in this tutorial.

To test your setup, invoke the weather agent, either via the UI playground by clicking on the `run` handler of the `WeatherAgent` in the overview:

<Frame>
  <img alt="Playground" />
</Frame>

Or via `curl`:

```bash theme={null}
curl localhost:8080/WeatherAgent/run \
  --json '{"message": "What is the weather like in San Francisco?"}'
```

You should see the weather information printed in the terminal.

Let's have a look at what happened under the hood to make your agents resilient.

## Durable Execution

AI agents make multiple LLM calls and tool executions that can fail due to rate limits, network issues, or service outages.
Restate uses Durable Execution to make your agents withstand failures without losing progress.

The Restate SDK records the steps the agent executes in a log and replays them if the process crashes or is restarted:

<img alt="Durable AI Agent Execution" />

Durable Execution is the basis of how Restate makes your agents resilient to failures.
Restate offers durable execution primitives via its SDK.

## Creating a Durable Agent

To implement a durable agent, you can use the Restate SDK in combination with the OpenAI Agent SDK.

Here's the implementation of the durable weather agent you just invoked:

```python durable_agent.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/openai-agents/tour-of-agents/app/durable_agent.py?collapse_imports"}  theme={null}
@durable_function_tool
async def get_weather(city: WeatherRequest) -> WeatherResponse:
    """Get the current weather for a given city."""
    return await restate_context().run_typed("get weather", fetch_weather, req=city)


agent = Agent(
    name="WeatherAgent",
    instructions="You are a helpful agent that provides weather updates.",
    tools=[get_weather],
)

agent_service = restate.Service("WeatherAgent")


@agent_service.handler()
async def run(_ctx: restate.Context, req: WeatherPrompt) -> str:
    result = await DurableRunner.run(agent, req.message)
    return result.final_output
```

<GitHubLink />

First, you implement your agent and its tools, similar to how you would do it with the OpenAI Agent SDK.

The main difference compared to a standard OpenAI agent is the use of the Restate Context at key points throughout the agent logic.
Any action with the Context is automatically recorded by the Restate Server and survives failures.
We use this for:

1. **Persisting LLM responses**: We use the `DurableRunner`, so that every LLM response is saved in Restate Server and can be replayed during recovery. The `DurableRunner` is provided to us via the OpenAI extensions in the Restate SDK.
2. **Resilient tool execution**: Tools can make steps durable by annotating them with `@durable_function_tool` and using Restate Context actions (available via `restate_context()` or `restate_object_context()`). Whenever you do an action with the Restate Context, the result gets persisted in Restate and can be recovered after a failure. The example uses `restate_context().run_typed`, which runs the function you provide to it and persists its result (e.g. database interaction, API calls, non-deterministic actions).

To serve the agent over HTTP with Restate, you create a Restate Service and define handlers.
Here, the agent logic is called from the `run` handler.
The endpoint that serves the agents of this tour over HTTP is defined in [`__main__.py`](https://github.com/restatedev/ai-examples/blob/main/openai-agents/tour-of-agents/__main__.py).
The agent can now be called at `http://localhost:8080/WeatherAgent/run`.

<Accordion title="Try out Durable Execution" icon={"laptop"}>
  Ask for the weather in Denver:

  ```bash theme={null}
  curl localhost:8080/WeatherAgent/run \
  --json '{"message": "What is the weather like in Denver?"}'
  ```

  On the invocation page in the UI, click on the invocation ID of the failing invocation.
  You can see that your request is retrying because the weather API is down:

  <Frame>
    <img alt="Invocation overview" />
  </Frame>

  To fix the problem, remove the line `fail_on_denver` from the `fetch_weather` function in the `app/utils/utils.py` file:

  ```python utils/utils.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/openai-agents/tour-of-agents/app/utils/utils.py#weather"}  theme={null}
  async def fetch_weather(req: WeatherRequest) -> WeatherResponse:
      fail_on_denver(req.city)
      weather_data = await call_weather_api(req.city)
      return parse_weather_data(weather_data)
  ```

  Once you restart the service, the workflow finishes successfully.
</Accordion>

## Observing your Agent

As you saw in the previous section, the Restate UI comes in handy when monitoring and debugging your agents.

The Invocations tab shows all agent executions with detailed traces of every LLM call, tool execution, and state change:

<Frame>
  <img alt="Invocation overview" />
</Frame>

<Accordion title="OpenTelemetry Integration">
  Restate supports OpenTelemetry for exporting traces to external systems like Langfuse, DataDog, or Jaeger:

  Have a look at the [tracing docs](/server/monitoring/tracing) to set this up.
</Accordion>

Now that you know how to build and debug an agent, let's look at more advanced patterns.

## Human-in-the-Loop Agent

Many AI agents need human oversight for high-risk decisions or gathering additional input. Restate makes it easy to pause agent execution and wait for human input.

**Benefits with Restate:**

* If the agent crashes while waiting for human input, Restate continues waiting and recovers the promise on another process.
* If the agent runs on function-as-a-service platforms, the Restate SDK lets the function suspend while it's waiting. Once the approval comes in, the Restate Server invokes the function again and lets it resume where it left off. This way, you don't pay for idle waiting time ([Learn more](https://www.restate.dev/blog/resilient-serverless-agents#cost-efficiency-scaling-to-zero-while-waiting)).

Here's a tool that asks for human approval for high-value claims:

```python human_approval_agent.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/openai-agents/tour-of-agents/app/human_approval_agent.py#here"}  theme={null}
@durable_function_tool
async def human_approval(claim: InsuranceClaim) -> str:
    """Ask for human approval for high-value claims."""

    # Create an awakeable for human approval
    approval_id, approval_promise = restate_context().awakeable(type_hint=str)

    # Request human review
    await restate_context().run_typed(
        "Request review", request_human_review, claim=claim, awakeable_id=approval_id
    )

    # Wait for human approval
    return await approval_promise
```

<GitHubLink />

To implement human approval steps, you can use Restate's awakeables. An awakeable is a promise that can be resolved externally via an API call by providing its ID.
When you create the awakeable, you get back an ID and a promise. You can send the ID to the human approver, and then wait for the promise to be resolved.

<Info>
  You can also use awakeables outside of tools, for example, to implement human approval steps in between agent iterations.
</Info>

<Accordion title="Try out human approval" icon={"laptop"}>
  Start a request for a high-value claim that needs human approval.
  Use the playground or `curl` with `/send` to start the claim asynchronously, without waiting for the result.

  ```bash theme={null}
  curl localhost:8080/HumanClaimApprovalAgent/run/send \
  --json '{"message": "Process my hospital bill of 3000USD for a broken leg."}'
  ```

  You can restart the service to see how Restate continues waiting for the approval.

  If you wait for more than a minute, the invocation will get suspended.

  <Frame>
    <img alt="Invocation overview" />
  </Frame>

  Simulate approving the claim by executing the **curl request that was printed in the service logs**, similar to:

  ```bash theme={null}
  curl localhost:8080/restate/awakeables/sign_1M28aqY6ZfuwBmRnmyP/resolve --json 'true'
  ```

  See in the UI how the workflow resumes and finishes after the approval.

  <Frame>
    <img alt="Invocation overview" />
  </Frame>
</Accordion>

<Accordion title="Timeouts and Escalation">
  Add timeouts to human approval steps to prevent workflows from hanging indefinitely.

  Restate persists the timer and the approval promise, so if the service crashes or is restarted, it will continue waiting with the correct remaining time:

  ```python human_approval_agent_with_timeout.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/openai-agents/tour-of-agents/app/human_approval_agent_with_timeout.py#here"}  theme={null}
  # Wait for human approval for at most 3 hours to reach our SLA
  match await restate.select(
      approval=approval_promise,
      timeout=restate_context().sleep(timedelta(hours=3)),
  ):
      case ["approval", approved]:
          return "Approved" if approved else "Rejected"
      case _:
          return "Approval timed out - Evaluate with AI"
  ```

  <GitHubLink />

  Try it out by sending a request to the service:

  ```bash theme={null}
  curl localhost:8080/HumanClaimApprovalWithTimeoutsAgent/run/send \
  --json '{"message": "Process my hospital bill of 3000USD for a broken leg."}'
  ```

  You restart the service and check in the UI how the process will block for the remaining time without starting over.

  You can also lower the timeout to a few seconds to see how the timeout path is taken.
</Accordion>

## Resilient workflows as tools

You can pull out complex parts of your tool logic into separate workflows.
This lets you break down complex agents into smaller, reusable components that can be developed, deployed, and scaled independently.

The Restate SDK gives you clients to call these workflows durably from your agent logic.
All calls are proxied via Restate. Restate persists the call and takes care of retries and recovery.

For example, let's implement the human approval tool as a separate service:

```python sub_workflow_agent.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/openai-agents/tour-of-agents/app/sub_workflow_agent.py#wf"}  theme={null}
# Sub-workflow service for human approval
human_approval_workflow = restate.Service("HumanApprovalWorkflow")


@human_approval_workflow.handler()
async def review(ctx: restate.Context, claim: InsuranceClaim) -> str:
    """Request human approval for a claim and wait for response."""
    # Create an awakeable that can be resolved via HTTP
    approval_id, approval_promise = ctx.awakeable(type_hint=str)

    # Request human review
    await ctx.run_typed(
        "Request review", request_human_review, claim=claim, awakeable_id=approval_id
    )

    # Wait for human approval
    return await approval_promise
```

<GitHubLink />

This can now be called from the main agent via a service client:

```python sub_workflow_agent.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/openai-agents/tour-of-agents/app/sub_workflow_agent.py#here"}  theme={null}
@durable_function_tool
async def human_approval(claim: InsuranceClaim) -> str:
    """Ask for human approval for high-value claims."""
    return await restate_context().service_call(review, claim)
```

<GitHubLink />

These workflows have access to all Restate SDK features, including durable execution, state management, awakeables, and observability.
They can be developed, deployed, and scaled independently.

<Accordion title="Try out sub-workflows" icon={"laptop"}>
  Start a request for a high-value claim that needs human approval.
  Use `/send` to start the claim asynchronously, without waiting for the result.

  ```bash theme={null}
  curl localhost:8080/SubWorkflowClaimApprovalAgent/run/send \
  --json '{"message": "Process my hospital bill of 3000USD for a broken leg."}'
  ```

  In the UI, you can see that the agent called the workflow service and is waiting for the response.
  You can see the trace of the sub-workflow in the timeline.

  Once you approve the claim, the workflow returns, and the agent continues.

  <Frame>
    <img alt="Invocation overview" />
  </Frame>
</Accordion>

<Info>
  Follow the [Tour of Workflows](/tour/workflows) to learn more about implementing resilient workflows with Restate.
</Info>

## Durable Sessions

The next ingredient we need to build AI agents is the ability to maintain context and memory across multiple interactions.

To implement stateful entities like chat sessions, or stateful agents, Restate provides a special service type called Virtual Objects.

When you send a message to a Virtual Object, you provide a unique key that identifies the object instance (for example, a chat session ID or user ID).

Each instance of a Virtual Object maintains isolated state. The handlers of the Virtual Object can read and modify the object's state via the Restate `ObjectContext`.

<img alt="Objects" />

### Virtual Objects for stateful agents

Restate's DurableRunner includes a configuration setting to automatically store your agent's events and state in Restate.

Here is an example of a stateful, durable agent represented as a Virtual Object:

```python chat.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/openai-agents/tour-of-agents/app/chat.py?collapse_imports"}  theme={null}
chat = VirtualObject("Chat")


@chat.handler()
async def message(_ctx: ObjectContext, req: ChatMessage) -> dict:
    # Set use_restate_session=True to store the session in Restate's key-value store
    # Make sure you use a VirtualObject to enable this feature
    result = await DurableRunner.run(
        Agent(name="Assistant", instructions="You are a helpful assistant."),
        req.message,
        use_restate_session=True,
    )
    return result.final_output
```

<GitHubLink />

This automatically persists the agent events (LLM calls, tool calls, etc.) and the conversation history in Restate.
It uses Restate as the session provider for the OpenAI Agent SDK.

<Accordion title="Try out Virtual Objects" icon={"laptop"}>
  Ask the agent to do some task and provide a session ID as the object key:

  ```bash theme={null}
  curl localhost:8080/Chat/session123/message \
  --json '{"message": "Make a poem about durable execution."}'
  ```

  Continue the conversation with the same session ID. The agent will remember previous context:

  ```bash theme={null}
  curl localhost:8080/Chat/session123/message \
  --json '{"message": "Shorten it to 2 lines."}'
  ```

  Go to the state tab of the UI to view the conversation history.
</Accordion>

Virtual Objects are ideal for implementing stateful agents because they provide:

* **Long-lived state**: K/V state is stored permanently. It has no automatic expiry. Clear it via `ctx.clear()`.
* **Durable state changes**: State changes are logged with Durable Execution, so they survive failures and are consistent with code execution
* **State is queryable** via the state tab in the UI.

<Frame>
  <img alt="Conversation State Management" />
</Frame>

### Built-in concurrency control

Restate’s Virtual Objects have built-in queuing and consistency guarantees per object key.

You provide the unique key when invoking the Virtual Object, for example, the chat session ID or user ID.
When multiple requests come in for the same object key, Restate automatically queues them and ensures consistency.

<img alt="Queue" />

The semantics are as follows:

* Handlers either have read-write access (`ObjectContext`) or read-only access (shared object context).
* Only one handler with write access can run at a time per object key to prevent concurrent/lost writes or race conditions (for example `message()`).
* Handlers with read-only access can run concurrently to the write-access handlers (for example `get_history()`).

<Accordion title="Try out queuing" icon={"laptop"}>
  Let's send several messages concurrently to different chat sessions:

  ```bash theme={null}
  curl localhost:8080/Chat/session123/message/send --json '{"message": "make a poem about durable execution"}' &
  curl localhost:8080/Chat/session456/message/send --json '{"message": "what are the benefits of durable execution?"}' &
  curl localhost:8080/Chat/session789/message/send --json '{"message": "how does workflow orchestration work?"}' &
  curl localhost:8080/Chat/session123/message/send --json '{"message": "can you make it rhyme better?"}' &
  curl localhost:8080/Chat/session456/message/send --json '{"message": "what about fault tolerance in distributed systems?"}' &
  curl localhost:8080/Chat/session789/message/send --json '{"message": "give me a practical example"}' &
  curl localhost:8080/Chat/session101/message/send --json '{"message": "explain event sourcing in simple terms"}' &
  curl localhost:8080/Chat/session202/message/send --json '{"message": "what is the difference between async and sync processing?"}'
  ```

  The UI shows how Restate queues the requests per session to ensure consistency:

  <Frame>
    <img alt="Conversation State Management" />
  </Frame>
</Accordion>

<Accordion title="Stateful Serverless Agents">
  You can run Virtual Objects on serverless platforms like Modal, Render, or AWS Lambda.
  When the request comes in, Restate attaches the correct state to the request, so your handler can access it locally.

  This way, you can implement [stateful, serverless agents without managing any external state store](https://www.restate.dev/blog/resilient-serverless-agents#cost-efficiency-scaling-to-zero-while-waiting) and without worrying about concurrency issues.
</Accordion>

### Virtual Objects for storing context

You can store any context information in Virtual Objects, for example, user preferences or the last agent they interacted with.

Use `ctx.set` and `ctx.get` in your handler to store and retrieve state.

We will show an example of this in the next section when we orchestrate multiple agents.

## Resilient multi-agent coordination

As your agents grow more complex, you may want to break them down into smaller, specialized agents that can delegate tasks to each other.

Similar to sub-workflows, you can break down complex agents into multiple specialized agents.

All agents can run in the same process or be deployed independently.

### Agents as tools/handoffs

If you want to share context between agents, run the agents in the same process and use handoffs or tools.

You don't need to do anything special to make this work with Restate.

Use Virtual Object state to maintain context between runs. For example, store the last agent that was called in the object state, so the user can connect back seamlessly on the next interaction:

```python multi_agent.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/openai-agents/tour-of-agents/app/multi_agent.py?collapse_imports"}  theme={null}
medical_agent = Agent(
    name="MedicalSpecialist",
    handoff_description="I handle medical insurance claims from intake to final decision.",
    instructions="Review medical claims for coverage and necessity. Approve/deny up to $50,000.",
)

car_agent = Agent(
    name="CarSpecialist",
    handoff_description="I handle car insurance claims from intake to final decision.",
    instructions="Assess car claims for liability and damage. Approve/deny up to $25,000.",
)


intake_agent = Agent(
    name="IntakeAgent",
    instructions="Route insurance claims to the appropriate specialist",
    handoffs=[medical_agent, car_agent],
)

agent_dict = {
    "IntakeAgent": intake_agent,
    "MedicalSpecialist": medical_agent,
    "AutoSpecialist": car_agent,
}

agent_service = restate.VirtualObject("MultiAgentClaimApproval")


@agent_service.handler()
async def run(ctx: restate.ObjectContext, claim: InsuranceClaim) -> str:
    # Store context in Restate's key-value store
    last_agent_name = await ctx.get("last_agent_name", type_hint=str) or "IntakeAgent"
    last_agent = agent_dict.get(last_agent_name, intake_agent)

    result = await DurableRunner.run(
        last_agent, f"Claim: {claim.model_dump_json()}", session=RestateSession()
    )

    ctx.set("last_agent_name", result.last_agent.name)
    return result.final_output
```

<GitHubLink />

The execution trace in the Restate UI will allow you to see the full chain of calls between agents and their individual steps.

<Accordion title="Try out multi-agent systems" icon={"laptop"}>
  Start a request for a claim that needs to be analyzed by multiple agents.

  ```bash theme={null}
  curl localhost:8080/MultiAgentClaimApproval/session123/run --json '{
      "date":"2024-10-01",
      "category":"orthopedic",
      "reason":"hospital bill for a broken leg",
      "amount":3000,
      "placeOfService":"General Hospital"
  }'
  ```

  In the UI, you can see that the agent called the sub-agents and is waiting for their responses.
  You can see the trace of the sub-agents in the timeline.

  Once all sub-agents return, the main agent continues and makes a decision.

  <Frame>
    <img alt="Invocation overview" />
  </Frame>

  The state now contains the last agent that was called, so you can continue the conversation directly with the same agent:

  <Frame>
    <img alt="Invocation overview" />
  </Frame>
</Accordion>

### Remote agents as tools

If you want to run agents independently, for example, to scale them separately, run them on different platforms, or let them get developed by different teams, then you can call them as tools via service calls.

Restate will proxy all calls, persist them, and will guarantee that they complete successfully.

Your main agent can suspend and save resources while waiting for the remote agent to finish.
Restate invokes your main agent again once the remote agent returns.

```python multi_agent.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/openai-agents/tour-of-agents/app/multi_agent_remote.py#here"}  theme={null}
# Durable service call to the fraud agent; persisted and retried by Restate
@durable_function_tool
async def check_fraud(claim: InsuranceClaim) -> str:
    """Analyze the probability of fraud."""
    return await restate_context().service_call(run_fraud_agent, claim)


agent = Agent(
    name="ClaimApprovalCoordinator",
    instructions="You are a claim approval engine. Analyze the claim and use your tools to decide whether to approve it.",
    tools=[check_eligibility, check_fraud],
)

agent_service = restate.Service("RemoteMultiAgentClaimApproval")


@agent_service.handler()
async def run(_ctx: restate.Context, claim: InsuranceClaim) -> str:
    result = await DurableRunner.run(agent, f"Claim: {claim.model_dump_json()}")
    return result.final_output
```

<GitHubLink />

Note, any shared context between agents needs to be passed explicitly via the input.

The execution trace in the Restate UI will allow you to see the full chain of calls between agents and their individual steps.

<Accordion title="Try out multi-agent systems" icon={"laptop"}>
  Start a request for a claim that needs to be analyzed by multiple agents.

  ```bash theme={null}
  curl localhost:8080/RemoteMultiAgentClaimApproval/run --json '{
      "date":"2024-10-01",
      "category":"orthopedic",
      "reason":"hospital bill for a broken leg",
      "amount":3000,
      "placeOfService":"General Hospital"
  }'
  ```

  In the UI, you can see that the agent called the sub-agents and is waiting for their responses.
  You can see the trace of the sub-agents in the timeline.

  Once all sub-agents return, the main agent continues and makes a decision.

  <Frame>
    <img alt="Invocation overview" />
  </Frame>
</Accordion>

<Warning>
  You cannot put both agents within the same Virtual Object, because this leads to deadlocks.
  The main agent would block on the call to the sub-agent, preventing the sub-agent from executing, cause only one handler can run at a time per object key.
</Warning>

## Parallel Work

Now that our agents are broken down into smaller parts, let's have a look at how to run different parts of our agent logic in parallel to speed up execution.

Restate provides primitives that allow you to run tasks concurrently while maintaining deterministic execution during replays.

Most actions on the Restate Context can be composed using `restate.gather` to gather their results or `restate.select` to wait for the first one to complete.

### Parallel Tool Steps

When using the OpenAI SDK with Restate, tool calls are forced to be executed sequentially to ensure deterministic execution during replays.
When multiple tools execute in parallel and use the Restate Context, the order of operations might differ between the original execution and the replay, leading to inconsistencies.

Therefore, the only way to run multiple tool steps in parallel is to implement an orchestrator tool that uses durable execution to run multiple steps in parallel.

Here is an insurance claim agent tool that runs multiple analyses in parallel:

```python parallel_tools_agent.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/openai-agents/tour-of-agents/app/parallel_tools_agent.py#here"}  theme={null}
@durable_function_tool
async def calculate_metrics(claim: InsuranceClaim) -> list[str]:
    """Calculate claim metrics."""
    ctx = restate_context()

    # Run tools/steps in parallel with durable execution
    results_done = await restate.gather(
        ctx.run_typed("eligibility", check_eligibility, claim=claim),
        ctx.run_typed("cost", compare_to_standard_rates, claim=claim),
        ctx.run_typed("fraud", check_fraud, claim=claim),
    )
    return [await result for result in results_done]
```

<GitHubLink />

Restate makes sure that all parallel tasks are retried and recovered until they succeed.

<Accordion title="Try out parallel tool steps" icon={"laptop"}>
  Start a request for a claim that needs to be analyzed by multiple tools in parallel.

  ```bash theme={null}
  curl localhost:8080/ParallelToolClaimAgent/run --json '{
      "date":"2024-10-01",
      "category":"orthopedic",
      "reason":"hospital bill for a broken leg",
      "amount":3000,
      "placeOfService":"General Hospital"
  }'
  ```

  In the UI, you can see that the agent ran the tool steps in parallel.
  Their traces all start at the same time.

  Once all tools return, the agent continues and makes a decision.

  <Frame>
    <img alt="Invocation overview" />
  </Frame>
</Accordion>

### Parallel Agents

You can use the same durable execution primitives to run multiple agents in parallel.

For example, to race agents against each other and [use the first result that returns, while cancelling the others](/ai/patterns/competitive-racing).

Or to let a main orchestrator agent combine the results of multiple specialized agents in parallel:

```python parallel_agents.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/openai-agents/tour-of-agents/app/parallel_agents.py#here"}  theme={null}
@agent_service.handler()
async def run(restate_context: restate.Context, claim: InsuranceClaim) -> str:
    # Start multiple agents in parallel with auto retries and recovery
    eligibility = restate_context.service_call(run_eligibility_agent, claim)
    cost = restate_context.service_call(run_rate_comparison_agent, claim)
    fraud = restate_context.service_call(run_fraud_agent, claim)

    # Wait for all responses
    await restate.gather(eligibility, cost, fraud)

    # Run decision agent on outputs
    result = await DurableRunner.run(
        Agent(
            name="ClaimApprovalAgent", instructions="You are a claim decision engine."
        ),
        input=f"Decide about claim: {claim.model_dump_json()}. "
        "Base your decision on the following analyses:"
        f"Eligibility: {await eligibility} Cost {await cost} Fraud: {await fraud}",
    )
    return result.final_output
```

<GitHubLink />

<Accordion title="Try out parallel agents" icon={"laptop"}>
  Start a request for a claim that needs to be analyzed by multiple agents in parallel.

  ```bash theme={null}
  curl localhost:8080/ParallelAgentClaimApproval/run --json '{
      "date":"2024-10-01",
      "category":"orthopedic",
      "reason":"hospital bill for a broken leg",
      "amount":3000,
      "placeOfService":"General Hospital"
  }'
  ```

  In the UI, you can see that the handler called the sub-agents in parallel.
  Once all sub-agents return, the main agent makes a decision.

  <Frame>
    <img alt="Invocation overview" />
  </Frame>
</Accordion>

## Error Handling

LLM calls are costly, so you can configure retry behavior in both Restate and your AI SDK to avoid infinite loops and high costs.

Restate distinguishes between two types of errors:

* **Transient errors**: Temporary issues like network failures or rate limits. Restate automatically retries these until they succeed or the retry policy is exhausted.
* **Terminal errors**: Permanent failures like invalid input or business rule violations. Restate does not retry these. The invocation fails permanently. You can catch these errors and handle them gracefully.

You can throw a terminal error via:

```python theme={null}
from restate import TerminalError

raise TerminalError("This tool is not allowed to run for this input.")
```

You can catch and handle terminal errors in your agent logic if needed.

Many AI SDKs also have their own retry behavior for LLM calls and tool executions, so let's look at how these interact.

### Retries of LLM calls

Restate's `DurableRunner` lets you specify the retry behavior for LLM calls as follows:

```python error_handling.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/openai-agents/tour-of-agents/app/error_handling.py#handle"}  theme={null}
try:
    result = await DurableRunner.run(
        agent,
        req.message,
        llm_retry_opts=LlmRetryOpts(
            max_attempts=3, initial_retry_interval=timedelta(seconds=2)
        ),
    )
except restate.TerminalError as e:
    # Handle terminal errors gracefully
    return f"The agent couldn't complete the request: {e.message}"
```

<GitHubLink />

By default, the runner retries ten times with an initial interval of one second. Once Restate's retries are exhausted, the invocation fails with a `TerminalError` and won't be retried further.

### Tool execution errors

By default, the Restate OpenAI integration will raise any terminal errors in tool executions and will let you handle them in your handler, similar to what we did above for model calls.

If you use Restate Context actions in your tool execution, Restate retries any transient errors in these actions until they succeed.
So for all operations that might suffer from transient errors (like network calls, database interactions, etc.), you should use Context actions to make them resilient.

<Info>
  You can set [custom retry policies](/guides/error-handling#at-the-run-block-level) for `.run` actions in your tool executions.
</Info>

<Warning>
  The OpenAI Agent SDK also allows setting `failure_error_function` to `None`, which will rethrow any error in the agent execution as-is.
  Also for example invalid LLM responses (e.g. tool call with invalid arguments or to a tool that doesn't exist).
  The error will then lead to Restate retries. Restate will recover the invocation by replaying the journal entries.
  This can lead to infinite retries if the error is not transient.
  Therefore, be careful when using this option and handle errors appropriately in your agent logic.
  You also might want to set [a retry policy at the service or handler level](/services/configuration#how-to-configure) to avoid infinite retries.
</Warning>

## Summary

Durable Execution, paired with your existing SDKs, gives your agents a powerful upgrade:

* **Durable Execution**: Automatic recovery from failures without losing progress
* **Persistent memory and context**: Persistent conversation history and context
* **Observability** by default across your agents and workflows
* **Human-in-the-Loop**: Seamless approval workflows with timeouts
* **Multi-Agent Coordination**: Reliable orchestration of specialized agents
* **Suspensions** to save costs on function-as-a-service platforms when agents need to wait
* **Advanced Patterns**: Real-time progress updates, interruptions, and long-running workflows

## Next Steps

* [AI Agent documentation](/ai): for advanced patterns and other integrations.
* [Restate AI examples repository](https://github.com/restatedev/ai-examples)
* Sign up for [Restate Cloud](https://restate.dev/cloud/) and start building agents without managing infrastructure


# Tour of Restate for Agents with Vercel AI SDK
Source: https://docs.restate.dev/tour/vercel-ai-agents

Build stateful, observable AI agents that recover from failures.

AI agents are long-running processes that combine LLMs with tools and external APIs to complete complex tasks. With Restate, you can build agents that are **resilient to failures**, **stateful across conversations**, and **observable** without managing complex retry logic or external state stores.

In this guide, you'll learn how to:

* Build durable AI agents that recover automatically from crashes and API failures
* Integrate Restate with the&#x20;
* Observe and debug agent executions with detailed traces
* Implement resilient human-in-the-loop workflows with approvals and timeouts
* Manage conversation history and state across multi-turn interactions
* Orchestrate multiple agents working together on complex tasks

## Getting Started

A Restate AI application has two main components:

* **Restate Server**: The core engine that takes care of the orchestration and resiliency of your agents
* **Agent Services**: Your agent or AI workflow logic using the Restate SDK for durability

<img alt="Application Structure" />

Restate works with how you already deploy your agents, whether that's in Docker, on Kubernetes, or via serverless platforms (Modal, AWS Lambda...). You don't need to run your agents in any special way.

Let's run an example locally to get a better feel for how it works.

### Run the agent

[Install Restate](/installation) and launch it:

```bash theme={null}
npm install --global @restatedev/restate-server@latest @restatedev/restate@latest
restate-server
```

Get the example:

```bash theme={null}
git clone git@github.com:restatedev/ai-examples.git
cd ai-examples/vercel-ai/tour-of-agents
npm install
```

<GitHubLink />

Export your [OpenAI API key](https://platform.openai.com/api-keys) and run the agent:

```bash theme={null}
export OPENAI_API_KEY=sk-...
npm run dev
```

Then, tell Restate where your agent is running via the UI (`http://localhost:9070`) or CLI:

```bash theme={null}
restate deployments register http://localhost:9080
```

This registers a set of agents that we will be covering in this tutorial.

To test your setup, invoke the weather agent, either via the UI playground (by clicking on the service) or curl:

```bash theme={null}
curl localhost:8080/WeatherAgent/run \
  --json '{"prompt": "What is the weather like in San Francisco?"}'
```

You should see the weather information printed in the terminal.

Let's have a look at what happened under the hood to make your agents resilient.

## Durable Execution

AI agents make multiple LLM calls and tool executions that can fail due to rate limits, network issues, or service outages.
Restate uses Durable Execution to make your agents withstand failures without losing progress.

The Restate SDK records the steps the agent executes in a log and replays them if the process crashes or is restarted:

<img alt="Durable AI Agent Execution" />

Durable Execution is the basis of how Restate makes your agents resilient to failures.
Restate offers durable execution primitives via its SDK.

## Creating a Durable Agent

To implement a durable agent, you can use the Restate SDK in combination with existing AI frameworks like the Vercel AI SDK.

Here's the implementation of the durable weather agent you just invoked:

```typescript durableexecution/agent.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/vercel-ai/tour-of-agents/src/durableexecution/agent.ts?collapse_prequel"}  theme={null}
export default restate.service({
  name: "WeatherAgent",
  handlers: {
    run: async (ctx: restate.Context, { prompt }: { prompt: string }) => {
      const model = wrapLanguageModel({
        model: openai("gpt-4o"),
        middleware: durableCalls(ctx, { maxRetryAttempts: 3 }),
      });

      const { text } = await generateText({
        model,
        system: "You are a helpful agent that provides weather updates.",
        prompt,
        tools: {
          getWeather: tool({
            description: "Get the current weather for a given city.",
            inputSchema: z.object({ city: z.string() }),
            execute: async ({ city }) =>
              ctx.run("get weather", () => fetchWeather(city)),
          }),
        },
        stopWhen: [stepCountIs(5)],
        providerOptions: { openai: { parallelToolCalls: false } },
      });

      return text;
    },
  },
});
```

<GitHubLink />

The agent logic is implemented in a handler of a Restate service, here the `run` handler.

The endpoint that serves the agents of this tour over HTTP is defined in `src/app.ts`. The agent can now be called at `http://localhost:8080/WeatherAgent/run`.

The main difference compared to a standard Vercel AI agent is the use of the Restate Context at key points throughout the agent logic.
Any action with the Context is automatically recorded by the Restate Server and survives failures.
We use this for:

1. **Persisting LLM responses**: We wrap the model with the `durableCalls(ctx)` middleware, so that every LLM response is saved in Restate Server and can be replayed during recovery. The middleware is provided via the package [`@restatedev/vercel-ai-middleware`](https://github.com/restatedev/vercel-ai-middleware).
2. **Resilient tool execution**: Tools can make steps durable by using Context actions. Their outcome will then be persisted for recovery and retried until they succeed. `ctx.run` runs an action durably, retrying it until it succeeds and persisting the result in Restate (e.g. database interaction, API calls, non-deterministic actions).

<Accordion title="Try out Durable Execution" icon={"laptop"}>
  Ask for the weather in Denver:

  ```bash theme={null}
  curl localhost:8080/WeatherAgent/run \
  --json '{"prompt": "What is the weather like in Denver?"}'
  ```

  On the invocation page in the UI, click on the invocation ID of the failing invocation.
  You can see that your request is retrying because the weather API is down:

  <Frame>
    <img alt="Invocation overview" />
  </Frame>

  To fix the problem, remove the line `failOnDenver` from the `fetchWeather` function in the `utils.ts` file:

  ```ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/vercel-ai/tour-of-agents/src/utils.ts#weather"}  theme={null}
  export async function fetchWeather(city: string) {
    failOnDenver(city);
    const output = await fetchWeatherFromAPI(city);
    return parseWeatherResponse(output);
  }
  ```

  Once you restart the service, the workflow finishes successfully.
</Accordion>

## Observing your Agent

As you saw in the previous section, the Restate UI comes in handy when monitoring and debugging your agents.

The Invocations tab shows all agent executions with detailed traces of every LLM call, tool execution, and state change:

<Frame>
  <img alt="Invocation overview" />
</Frame>

<Accordion title="OpenTelemetry Integration">
  Restate supports OpenTelemetry for exporting traces to external systems like Langfuse, DataDog, or Jaeger:

  Have a look at the [tracing docs](/server/monitoring/tracing) to set this up.
</Accordion>

Now that you know how to build and debug an agent, let's look at more advanced patterns.

## Human-in-the-Loop Agent

Many AI agents need human oversight for high-risk decisions or gathering additional input. Restate makes it easy to pause agent execution and wait for human input.

**Benefits with Restate:**

* If the agent crashes while waiting for human input, Restate continues waiting and recovers the promise on another process.
* If the agent runs on function-as-a-service platforms, the Restate SDK lets the function suspend while it's waiting. Once the approval comes in, the Restate Server invokes the function again and lets it resume where it left off. This way, you don't pay for idle waiting time ([Learn more](https://www.restate.dev/blog/resilient-serverless-agents#cost-efficiency-scaling-to-zero-while-waiting)).

Here's an insurance claim agent that asks for human approval for high-value claims:

```typescript humanintheloop/agent.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/vercel-ai/tour-of-agents/src/humanintheloop/agent.ts#here"}  theme={null}
const { text } = await generateText({
  model,
  system:
    "You are an insurance claim evaluation agent. Use these rules: " +
    "* if the amount is more than 1000, ask for human approval, " +
    "* if the amount is less than 1000, decide by yourself",
  prompt,
  tools: {
    humanApproval: tool({
      description: "Ask for human approval for high-value claims.",
      inputSchema: InsuranceClaimSchema,
      execute: async (claim: InsuranceClaim): Promise<boolean> => {
        const approval = ctx.awakeable<boolean>();
        await ctx.run("request-review", () =>
          requestHumanReview(claim, approval.id),
        );
        return approval.promise;
      },
    }),
  },
  stopWhen: [stepCountIs(5)],
  providerOptions: { openai: { parallelToolCalls: false } },
});
```

<GitHubLink />

To implement human approval steps, you can use Restate's awakeables. An awakeable is a promise that can be resolved externally via an API call by providing its ID.
When you create the awakeable, you get back an ID and a promise. You can send the ID to the human approver, and then wait for the promise to be resolved.

<Info>
  You can also use awakeables outside of tools, for example, to implement human approval steps in between agent iterations.
</Info>

<Accordion title="Try out human approval" icon={"laptop"}>
  Start a request for a high-value claim that needs human approval, by clicking on the `run` handler of the `HumanClaimApprovalAgent`, and sending the default request via the playground.

  Or use `curl` with `/send` to start the claim asynchronously, without waiting for the result.

  ```bash theme={null}
  curl localhost:8080/HumanClaimApprovalAgent/run/send \
  --json '{"prompt": "Process my hospital bill of 3000USD for a broken leg."}'
  ```

  You can restart the service to see how Restate continues waiting for the approval.

  If you wait for more than a minute, the invocation will get suspended.

  <Frame>
    <img alt="Invocation overview" />
  </Frame>

  Simulate approving the claim by executing the **curl request that was printed in the service logs**, similar to:

  ```bash theme={null}
  curl localhost:8080/restate/awakeables/sign_1M28aqY6ZfuwBmRnmyP/resolve --json 'true'
  ```

  See in the UI how the workflow resumes and finishes after the approval.

  <Frame>
    <img alt="Invocation overview" />
  </Frame>
</Accordion>

<Accordion title="Timeouts and Escalation">
  Add timeouts to human approval steps to prevent workflows from hanging indefinitely.

  Restate persists the timer and the approval promise, so if the service crashes or is restarted, it will continue waiting with the correct remaining time:

  ```typescript humanintheloop/agent-with-timeout.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/vercel-ai/tour-of-agents/src/humanintheloop/agent-with-timeout.ts#here"}  theme={null}
  try {
    // At most 3 hours, to reach our SLA
    const approved = await approval.promise.orTimeout({ hours: 3 });
    return { approved };
  } catch (e) {
    if (e instanceof TimeoutError) {
      return {
        approved: false,
        reason: "Approval timed out - Evaluate with AI",
      };
    }
    throw e;
  }
  ```

  <GitHubLink />

  Try it out by sending a request to the service:

  ```bash theme={null}
  curl localhost:8080/HumanClaimApprovalWithTimeoutsAgent/run/send \
  --json '{"prompt": "Process my hospital bill of 3000USD for a broken leg."}'
  ```

  You restart the service and check in the UI how the process will block for the remaining time without starting over.

  You can also lower the timeout to a few seconds to see how the timeout path is taken.
</Accordion>

## Chat Agent with Memory

The next ingredient we need to build AI agents is the ability to maintain context and memory across multiple interactions.

To implement stateful entities like chat sessions, or stateful agents, Restate provides Virtual Objects.

Each Virtual Object instance maintains isolated state and is identified by a unique key.

Here is an example of a Virtual Object that represents chat sessions:

<img alt="Objects" />

```typescript chat/agent.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/vercel-ai/tour-of-agents/src/chat/agent.ts?collapse_prequel"}  theme={null}
export default restate.object({
  name: "Chat",
  handlers: {
    message: async (ctx: restate.ObjectContext, req: { message: string }) => {
      const model = wrapLanguageModel({
        model: openai("gpt-4o"),
        middleware: durableCalls(ctx, { maxRetryAttempts: 3 }),
      });

      const messages =
        (await ctx.get<ModelMessage[]>("messages", superJson)) ?? [];
      messages.push({ role: "user", content: req.message });

      const res = await generateText({
        model,
        system: "You are a helpful assistant.",
        messages,
      });

      ctx.set("messages", [...messages, ...res.response.messages], superJson);
      return { answer: res.text };
    },
    getHistory: shared(async (ctx: restate.ObjectSharedContext) =>
      ctx.get<ModelMessage[]>("messages", superJson),
    ),
  },
});
```

<GitHubLink />

Virtual Objects are ideal for implementing any entity with mutable state:

* **Long-lived state**: K/V state is stored permanently. It has no automatic expiry. Clear it via `ctx.clear()`.
* **Durable state changes**: State changes are logged with Durable Execution, so they survive failures and are consistent with code execution
* **State is queryable** via the state tab in the UI.

<Frame>
  <img alt="Conversation State Management" />
</Frame>

* **Built-in concurrency control**: Restate’s Virtual Objects have built-in queuing and consistency guarantees per object key. Handlers either have read-write access (`ObjectContext`) or read-only access (shared object context).
  * Only one handler with write access can run at a time per object key to prevent concurrent/lost writes or race conditions (for example `message()`).
  * Handlers with read-only access can run concurrently to the write-access handlers (for example `getHistory()`).

<img alt="Queue" />

<Accordion title="Try out Virtual Objects" icon={"laptop"}>
  **Stateful Chat Agent:**

  Ask the agent to do some task:

  ```bash theme={null}
  curl localhost:8080/Chat/session123/message \
  --json '{"message": "make a poem about durable execution"}'
  ```

  Continue the conversation - the agent remembers previous context:

  ```bash theme={null}
  curl localhost:8080/Chat/session123/message \
  --json '{"message": "shorten it to 2 lines"}'
  ```

  Get conversation history or view it in the UI:

  ```bash theme={null}
  curl localhost:8080/Chat/session123/getHistory
  ```

  **Seeing concurrency control in action:**

  In the chat service, the `message` handler is an exclusive handler, while the `getHistory` handler is a shared handler.

  Let's send some messages to a chat session:

  ```bash theme={null}
  curl localhost:8080/Chat/session123/message/send --json '{"message": "make a poem about durable execution"}' &
  curl localhost:8080/Chat/session456/message/send --json '{"message": "what are the benefits of durable execution?"}' &
  curl localhost:8080/Chat/session789/message/send --json '{"message": "how does workflow orchestration work?"}' &
  curl localhost:8080/Chat/session123/message/send --json '{"message": "can you make it rhyme better?"}' &
  curl localhost:8080/Chat/session456/message/send --json '{"message": "what about fault tolerance in distributed systems?"}' &
  curl localhost:8080/Chat/session789/message/send --json '{"message": "give me a practical example"}' &
  curl localhost:8080/Chat/session101/message/send --json '{"message": "explain event sourcing in simple terms"}' &
  curl localhost:8080/Chat/session202/message/send --json '{"message": "what is the difference between async and sync processing?"}'
  ```

  The UI shows how Restate queues the requests per session to ensure consistency:

  <Frame>
    <img alt="Conversation State Management" />
  </Frame>
</Accordion>

<Accordion title="Stateful Serverless Agents">
  You can run Virtual Objects on serverless platforms like Vercel, Modal, Cloudflare Workers, or AWS Lambda.
  When the request comes in, Restate attaches the correct state to the request, so your handler can access it locally.

  This way, you can implement [stateful, serverless agents without managing any external state store](https://www.restate.dev/blog/resilient-serverless-agents#cost-efficiency-scaling-to-zero-while-waiting) and without worrying about concurrency issues.
</Accordion>

## Agent Orchestration

As your agents grow more complex, you may want to break them down into smaller, specialized sub-workflows and sub-agents.
Each of these can then be developed, deployed, and scaled independently.

### Tools as sub-workflows

You can pull out complex parts of your tool logic into separate workflows.

The Restate SDK gives you clients to call other Restate services durably from your agent logic.
All calls are proxied via Restate. Restate persists the call and takes care of retries and recovery.

For example, let's implement the human approval tool as a separate service:

```typescript orchestration/sub-workflow-agent.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/vercel-ai/tour-of-agents/src/orchestration/sub-workflow-agent.ts#wf"}  theme={null}
export const humanApprovalWorfklow = restate.service({
  name: "HumanApprovalWorkflow",
  handlers: {
    requestApproval: async (ctx: restate.Context, claim: InsuranceClaim) => {
      const approval = ctx.awakeable<boolean>();
      await ctx.run("request-review", () =>
        requestHumanReview(claim, approval.id),
      );
      return approval.promise;
    },
  },
});
```

<GitHubLink />

This can now be called from the main agent via a service client:

```typescript orchestration/sub-workflow-agent.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/vercel-ai/tour-of-agents/src/orchestration/sub-workflow-agent.ts#here"}  theme={null}
humanApproval: tool({
  description: "Ask for human approval for high-value claims.",
  inputSchema: InsuranceClaimSchema,
  execute: async (claim: InsuranceClaim) =>
    ctx.serviceClient(humanApprovalWorfklow).requestApproval(claim),
}),
```

<GitHubLink />

These workflows have access to all Restate SDK features, including durable execution, state management, awakeables, and observability.
They can be developed, deployed, and scaled independently.

<Accordion title="Try out sub-workflows" icon={"laptop"}>
  Start a request for a high-value claim that needs human approval.
  Use `/send` to start the claim asynchronously, without waiting for the result.

  ```bash theme={null}
  curl localhost:8080/SubWorkflowClaimApprovalAgent/run/send \
  --json '{"prompt": "Process my hospital bill of 3000USD for a broken leg."}'
  ```

  In the UI, you can see that the agent called the workflow service and is waiting for the response.
  You can see the trace of the sub-workflow in the timeline.

  Once you approve the claim, the workflow returns, and the agent continues.

  <Frame>
    <img alt="Invocation overview" />
  </Frame>
</Accordion>

<Info>
  Follow the [Tour of Workflows](/tour/workflows) to learn more about implementing resilient workflows with Restate.
</Info>

### Multi-agent Systems

Similar to sub-workflows, you can break down complex agents into multiple specialized agents.

You can let your agent hand off tasks to other agents by calling them from tools:

```typescript orchestration/multi-agent.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/vercel-ai/tour-of-agents/src/orchestration/multi-agent.ts#here"}  theme={null}
const { text } = await generateText({
  model,
  prompt: `Claim: ${JSON.stringify(claim)}`,
  system:
    "You are a claim approval engine. Analyze the claim and use your tools to decide whether to approve.",
  tools: {
    analyzeEligibility: tool({
      description: "Analyze claim eligibility.",
      inputSchema: InsuranceClaimSchema,
      execute: async (claim: InsuranceClaim) =>
        ctx.serviceClient(eligibilityAgent).run(claim),
    }),
    analyzeFraud: tool({
      description: "Analyze probability of fraud.",
      inputSchema: InsuranceClaimSchema,
      execute: async (claim: InsuranceClaim) =>
        ctx.serviceClient(fraudCheckAgent).run(claim),
    }),
  },
  stopWhen: [stepCountIs(10)],
  providerOptions: { openai: { parallelToolCalls: false } },
});
```

<GitHubLink />

<Accordion title="Try out multi-agent systems" icon={"laptop"}>
  Start a request for a claim that needs to be analyzed by multiple agents.

  ```bash theme={null}
  curl localhost:8080/MultiAgentClaimApproval/run --json '{
      "date":"2024-10-01",
      "category":"orthopedic",
      "reason":"hospital bill for a broken leg",
      "amount":3000,
      "placeOfService":"General Hospital"
  }'
  ```

  In the UI, you can see that the agent called the sub-agents and is waiting for their responses.
  You can see the trace of the sub-agents in the timeline.

  Once all sub-agents return, the main agent continues and makes a decision.

  <Frame>
    <img alt="Invocation overview" />
  </Frame>
</Accordion>

## Parallel Work

Now that our agents are broken down into smaller parts, let's have a look at how to run different parts of our agent logic in parallel to speed up execution.

<Warning>
  You might have noticed that all example agents set `parallelToolCalls: false` in the OpenAI provider options.
  This is required to ensure deterministic execution during replays.
  When multiple tools execute in parallel and use the Context, the order of operations might differ between the original execution and the replay, leading to inconsistencies.
</Warning>

Restate provides primitives that allow you to run tasks concurrently while maintaining deterministic execution during replays.

Most actions on the Restate Context return a RestatePromise. These can be composed using `RestatePromise.all`, `RestatePromise.allSettled`, and `RestatePromise.race` to gather their results.

### Parallel Tool Steps

To parallelize tool steps, implement an orchestrator tool that uses `RestatePromise`  to run multiple steps in parallel.

Here is an insurance claim agent that runs multiple analyses in parallel:

```typescript parallelwork/parallel-tools-agent.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/vercel-ai/tour-of-agents/src/parallelwork/parallel-tools-agent.ts#here"}  theme={null}
const { text } = await generateText({
  model,
  prompt: `Analyze the claim ${JSON.stringify(claim)}. 
  Use your tools to calculate key metrics and decide whether to approve.`,
  tools: {
    calculateMetrics: tool({
      description: "Calculate claim metrics.",
      inputSchema: InsuranceClaimSchema,
      execute: async (claim: InsuranceClaim) => {
        // Execute each calculation as a parallel durable step
        return RestatePromise.all([
          ctx.run("eligibility", () => checkEligibility(claim)),
          ctx.run("cost", () => compareToStandardRates(claim)),
          ctx.run("fraud", () => checkFraud(claim)),
        ]);
      },
    }),
  },
  stopWhen: [stepCountIs(10)],
  providerOptions: { openai: { parallelToolCalls: false } },
});
```

<GitHubLink />

Restate makes sure that all parallel tasks are retried and recovered until they succeed.

<Info>
  If you want to allow the LLM to call multiple tools in parallel with `parallelToolCalls: true`, then you need to [manually implement the agent tool execution loop](#advanced-patterns) using `RestatePromise`.
</Info>

<Accordion title="Try out parallel tool steps" icon={"laptop"}>
  Start a request for a claim that needs to be analyzed by multiple tools in parallel.

  ```bash theme={null}
  curl localhost:8080/ParallelToolClaimAgent/run --json '{
      "date":"2024-10-01",
      "category":"orthopedic",
      "reason":"hospital bill for a broken leg",
      "amount":3000,
      "placeOfService":"General Hospital"
  }'
  ```

  In the UI, you can see that the agent ran the tool steps in parallel.
  Their traces all start at the same time.

  Once all tools return, the agent continues and makes a decision.

  <Frame>
    <img alt="Invocation overview" />
  </Frame>
</Accordion>

### Parallel Agents

You can use the same `RestatePromise` primitives to run multiple agents in parallel.

For example, to race agents against each other and [use the first result that returns, while cancelling the others](/ai/patterns/competitive-racing).

Or to let a main orchestrator agent combine the results of multiple specialized agents in parallel:

```typescript parallelwork/parallel-agents.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/vercel-ai/tour-of-agents/src/parallelwork/parallel-agents.ts?collapse_prequel"}  theme={null}
export default restate.service({
  name: "ParallelAgentClaimApproval",
  handlers: {
    run: async (ctx: restate.Context, claim: InsuranceClaim) => {
      const [eligibility, rateComparison, fraudCheck] =
        await RestatePromise.all([
          ctx.serviceClient(eligibilityAgent).run(claim),
          ctx.serviceClient(rateComparisonAgent).run(claim),
          ctx.serviceClient(fraudCheckAgent).run(claim),
        ]);

      const model = wrapLanguageModel({
        model: openai("gpt-4o"),
        middleware: durableCalls(ctx, { maxRetryAttempts: 3 }),
      });

      const { text } = await generateText({
        model,
        system: "You are a claim decision engine.",
        prompt: `Decide about claim ${JSON.stringify(claim)}. 
        Base your decision on the following analyses:
        Eligibility: ${eligibility}, Cost: ${rateComparison} Fraud: ${fraudCheck}`,
      });
      return text;
    },
  },
});
```

<GitHubLink />

<Accordion title="Try out parallel agents" icon={"laptop"}>
  Start a request for a claim that needs to be analyzed by multiple agents in parallel.

  ```bash theme={null}
  curl localhost:8080/ParallelAgentClaimApproval/run --json '{
      "date":"2024-10-01",
      "category":"orthopedic",
      "reason":"hospital bill for a broken leg",
      "amount":3000,
      "placeOfService":"General Hospital"
  }'
  ```

  In the UI, you can see that the handler called the sub-agents in parallel.
  Once all sub-agents return, the main agent makes a decision.

  <Frame>
    <img alt="Invocation overview" />
  </Frame>
</Accordion>

## Error Handling

LLM calls are costly, so you can configure retry behavior in both Restate and your AI SDK to avoid infinite loops and high costs.

Restate distinguishes between two types of errors:

* **Transient errors**: Temporary issues like network failures or rate limits. Restate automatically retries these until they succeed or the retry policy is exhausted.
* **Terminal errors**: Permanent failures like invalid input or business rule violations. Restate does not retry these. The invocation fails permanently. You can catch these errors and handle them gracefully.

You can throw a terminal error via:

```typescript {"CODE_LOAD::ts/src/tour/agents/terminal_error.ts#terminal_error"}  theme={null}
throw new TerminalError("This tool is not allowed to run for this input.");
```

You can catch and handle terminal errors in your agent logic if needed.

Many AI SDKs also have their own retry behavior for LLM calls and tool executions, so let's look at how these interact.

### Retries of LLM calls

In the Vercel AI SDK, set `maxRetries` on `generateText` (default: 2) to retry failed calls due to rate limits or transient errors.
After retries are exhausted, the agent throws an error.
Restate then retries the invocation with exponential backoff to handle longer outages or network issues.

You can limit Restate’s retries with the `maxRetryAttempts` option in `durableCalls` middleware:

```typescript errorhandling/fail-on-terminal-tool-agent.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/vercel-ai/tour-of-agents/src/errorhandling/fail-on-terminal-tool-agent.ts#max_attempts_example"}  theme={null}
const model = wrapLanguageModel({
  model: openai("gpt-4o"),
  middleware: durableCalls(ctx, { maxRetryAttempts: 3 }),
});
```

<GitHubLink />

Each Restate retry triggers up to `maxRetries` SDK attempts.
For example, with `maxRetryAttempts`: 3 and `maxRetries`: 2, a call may be attempted 6 times.
Once Restate’s retries are exhausted, the invocation fails with a `TerminalError` and won’t be retried further.

### Tool execution errors

By default, the Vercel AI SDK will convert any errors in tool executions into a message to the LLM, and the agent will decide how to proceed.
This is often desirable, as the LLM can decide to use a different tool or provide a fallback answer.

However, if you use Restate Context actions like `ctx.run` in your tool execution, Restate will retry any transient errors in these actions until they succeed.
So for all operations that might suffer from transient errors (like network calls, database interactions, etc.), you should use Context actions to make them resilient.

Here is a small practical example:

```typescript {"CODE_LOAD::ts/src/tour/agents/inline-tool-errors.ts#here"}  theme={null}
// Without ctx.run - error goes straight to agent
async function myTool() {
  const result = await fetch("/api/data"); // Might fail due to network
  // If this fails, agent gets the error immediately
}

// With ctx.run - Restate handles retries
async function myToolWithRestate(ctx: restate.Context) {
  const result = await ctx.run("fetch-data", () => fetch("/api/data"));
  // Network failures get retried automatically
  // Only terminal errors reach the AI
}
```

If your tool calls other Restate handlers or workflows (for example sub-workflow or sub-agent), then these are also Restate Context actions that get retried until they succeed.

Terminal errors thrown from Restate Context actions are not retried by Restate, and get processed by the Vercel AI SDK.
Also here, the Vercel AI SDK will convert the error into a message to the LLM, and the agent will decide how to proceed.

In some cases, you might want to treat terminal tool execution errors as permanent failures and stop the agent instead of letting the LLM decide how to proceed.
The Restate middleware provides two utilities to help with this:

<AccordionGroup>
  <Accordion title="Fail the agent on terminal tool errors">
    **To fail the agent on terminal tool errors**, rethrow the error in `onStepFinish`:

    ```typescript errorhandling/fail-on-terminal-tool-agent.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/vercel-ai/tour-of-agents/src/errorhandling/fail-on-terminal-tool-agent.ts#option2"}  theme={null}
    const { text } = await generateText({
      model,
      tools: {
        getWeather: tool({
          description: "Get the current weather for a given city.",
          inputSchema: z.object({ city: z.string() }),
          execute: async ({ city }) => {
            return await ctx.run("get weather", () => fetchWeather(city));
          },
        }),
      },
      stopWhen: [stepCountIs(5)],
      onStepFinish: rethrowTerminalToolError,
      system: "You are a helpful agent that provides weather updates.",
      messages: [{ role: "user", content: prompt }],
    });
    ```

    <GitHubLink />
  </Accordion>

  <Accordion title="Stop the agent on terminal tool errors">
    To stop the agent on terminal tool errors and handle it after the agent finishes, you can use `hasTerminalToolError` in `stopWhen` and then inspect the steps for errors:

    ```typescript errorhandling/stop-on-terminal-tool-agent.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/ai-examples/refs/heads/main/vercel-ai/tour-of-agents/src/errorhandling/stop-on-terminal-tool-agent.ts#option3"}  theme={null}
    const { steps, text } = await generateText({
      model,
      tools: {
        getWeather: tool({
          description: "Get the current weather for a given city.",
          inputSchema: z.object({ city: z.string() }),
          execute: async ({ city }) => {
            return await ctx.run("get weather", () => fetchWeather(city));
          },
        }),
      },
      stopWhen: [stepCountIs(5), hasTerminalToolError],
      system: "You are a helpful agent that provides weather updates.",
      messages: [{ role: "user", content: prompt }],
    });

    const terminalSteps = getTerminalToolSteps(steps);
    if (terminalSteps.length > 0) {
      // Do something with the terminal tool error steps
    }
    ```

    <GitHubLink />
  </Accordion>
</AccordionGroup>

You can catch and handle terminal errors in your agent logic if needed.
Have a look at the advanced patterns section for an example of rolling back previous tool executions on failure.

<Info>
  You can set [custom retry policies](/guides/error-handling#at-the-run-block-level) for `ctx.run` steps in your tool executions.
</Info>

## Summary

Durable Execution, paired with your existing SDKs, gives your agents a powerful upgrade:

* **Durable Execution**: Automatic recovery from failures without losing progress
* **Persistent memory and context**: Persistent conversation history and context
* **Observability** by default across your agents and workflows
* **Human-in-the-Loop**: Seamless approval workflows with timeouts
* **Multi-Agent Coordination**: Reliable orchestration of specialized agents
* **Suspensions** to save costs on function-as-a-service platforms when agents need to wait
* **Advanced Patterns**: Real-time progress updates, interruptions, and long-running workflows

## Next Steps

* [AI Agent documentation](/ai): for advanced patterns and other integrations.
* [Restate AI examples repository](https://github.com/restatedev/ai-examples)
* Sign up for [Restate Cloud](https://restate.dev/cloud/) and start building agents without managing infrastructure


# Workflows
Source: https://docs.restate.dev/tour/workflows

Build resilient workflows with familiar programming patterns.

Workflows orchestrate complex business processes that span multiple steps, services, and time periods. Restate workflows are written as regular functions in your programming language, with automatic durability, state management, and event handling built in.

In this guide, you'll learn how to:

* Write workflows as regular functions with automatic durability
* Handle long-running processes with state and event patterns
* Deploy steps inline or services that can scale independently
* Build resilient, observable workflows without external dependencies

Select your SDK:

<GlobalTabs>
  <GlobalTab title="TypeScript" />

  <GlobalTab title="Java" />

  <GlobalTab title="Go" />

  <GlobalTab title="Python" />
</GlobalTabs>

## Getting Started

A Restate application is composed of two main components:

* **Restate Server**: The core engine that manages durable execution and orchestrates services. It acts as a message broker or reverse proxy in front of your services.
* **Your Services**: Your workflows and business logic, implemented as service handlers using the Restate SDK to perform durable operations.

<img alt="Application Structure" />

A basic signup workflow looks like this:

<GlobalTabs>
  <GlobalTab title="TypeScript">
    ```typescript signup-workflow.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/tutorials/tour-of-workflows-typescript/src/workflows/signup-workflow.ts?collapse_prequel"}  theme={null}
    export const signupWorkflow = restate.workflow({
      name: "SignupWorkflow",
      handlers: {
        run: async (ctx: WorkflowContext, user: User) => {
          const userId = ctx.key; // workflow ID = user ID

          // Write to database
          const success = await ctx.run("create", () => createUser(userId, user));
          if (!success) return { success };

          // Call APIs
          await ctx.run("activate", () => activateUser(userId));
          await ctx.run("welcome", () => sendWelcomeEmail(user));
          return { success };
        },
      },
    });
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Java">
    ```java SignupWorkflow.java {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/tutorials/tour-of-workflows-java/src/main/java/my/example/workflows/SignupWorkflow.java?collapse_prequel"}  theme={null}
    @Workflow
    public class SignupWorkflow {

      @Workflow
      public boolean run(WorkflowContext ctx, User user) {
        String userId = ctx.key(); // workflow ID = user ID

        // Write to database
        boolean success = ctx.run("create", Boolean.class, () -> createUser(userId, user));
        if (!success) {
          return false;
        }

        // Call APIs
        ctx.run("activate", () -> activateUser(userId));
        ctx.run("welcome", () -> sendWelcomeEmail(user));

        return true;
      }
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Go">
    ```go getstarted.go {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/tutorials/tour-of-workflows-go/examples/getstarted.go?collapse_prequel"}  theme={null}
    type SignupWorkflow struct{}

    func (SignupWorkflow) Run(ctx restate.WorkflowContext, user User) (bool, error) {
      userID := restate.Key(ctx) // workflow ID = user ID

      // Write to database
      success, err := restate.Run(ctx, func(ctx restate.RunContext) (bool, error) {
        return CreateUser(userID, user)
      }, restate.WithName("create"))
      if err != nil || !success {
        return false, err
      }

      // Call APIs
      _, err = restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
        return ActivateUser(userID)
      }, restate.WithName("activate"))
      if err != nil {
        return false, err
      }

      _, err = restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
        return SendWelcomeEmail(user)
      }, restate.WithName("welcome"))
      if err != nil {
        return false, err
      }

      return true, nil
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Python">
    ```python signup_workflow.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/python/tutorials/tour-of-workflows-python/app/workflows/signup_workflow.py?collapse_prequel"}  theme={null}
    signup_workflow = restate.Workflow("SignupWorkflow")


    @signup_workflow.main()
    async def run(ctx: WorkflowContext, user: User) -> bool:
        user_id = ctx.key()  # workflow ID = user ID

        # Write to database
        success = await ctx.run_typed("create", create_user, user_id=user_id, user=user)
        if not success:
            return False

        # Call APIs
        await ctx.run_typed("activate", activate_user, user_id=user_id)
        await ctx.run_typed("welcome", send_welcome_email, user=user)
        return True
    ```

    <GitHubLink />
  </GlobalTab>
</GlobalTabs>

A workflow has handlers that can be called over HTTP.
The `run` handler is the main entry point that executes the workflow logic.

An execution of the workflow is identified by a unique key (in this case, the user ID)
and uses Restate's `WorkflowContext` to make steps durable.

You don't need to run your services in any special way. Restate works with how you already deploy your code, whether that's in Docker, on Kubernetes, or via AWS Lambda.

<GlobalTabs>
  <GlobalTab title="TypeScript">
    The endpoint that serves the workflows of this tour over HTTP is defined in `src/app.ts`.
  </GlobalTab>

  <GlobalTab title="Java">
    The endpoint that serves the workflows of this tour over HTTP is defined in `AppMain.java`.
  </GlobalTab>

  <GlobalTab title="Go">
    The endpoint that serves the workflows of this tour over HTTP is defined in `main.go`.
  </GlobalTab>

  <GlobalTab title="Python">
    The endpoint that serves the workflows of this tour over HTTP is defined in `__main__.py`.
  </GlobalTab>
</GlobalTabs>

### Run the example

[Install Restate](/installation) and launch it:

```bash theme={null}
restate-server
```

<GlobalTabs>
  <GlobalTab title="TypeScript">
    Get the example:

    ```bash theme={null}
    restate example typescript-tour-of-workflows && cd typescript-tour-of-workflows
    npm install
    ```

    <GitHubLink />

    Run the example:

    ```bash theme={null}
    npm run dev
    ```

    Then, tell Restate where your workflow is running via the UI (`http://localhost:9070`) or CLI:

    ```bash theme={null}
    restate deployments register http://localhost:9080
    ```
  </GlobalTab>

  <GlobalTab title="Java">
    Get the example:

    ```bash theme={null}
    restate example java-tour-of-workflows && cd java-tour-of-workflows
    ```

    <GitHubLink />

    Run the example:

    ```bash theme={null}
    ./gradlew run
    ```

    Then, tell Restate where your services are running via the UI (`http://localhost:9070`) or CLI:

    ```bash theme={null}
    restate deployments register http://localhost:9080
    ```
  </GlobalTab>

  <GlobalTab title="Go">
    Get the example:

    ```bash theme={null}
    restate example go-tour-of-workflows && cd go-tour-of-workflows
    ```

    <GitHubLink />

    Run the example:

    ```bash theme={null}
    go run .
    ```

    Then, tell Restate where your services are running via the UI (`http://localhost:9070`) or CLI:

    ```bash theme={null}
    restate deployments register http://localhost:9080
    ```
  </GlobalTab>

  <GlobalTab title="Python">
    Get the example:

    ```bash theme={null}
    restate example python-tour-of-workflows && cd python-tour-of-workflows
    ```

    <GitHubLink />

    Run the example:

    ```bash theme={null}
    uv run .
    ```

    Then, tell Restate where your services are running via the UI (`http://localhost:9070`) or CLI:

    ```bash theme={null}
    restate deployments register http://localhost:9080
    ```
  </GlobalTab>
</GlobalTabs>

This registers a set of workflows that we will be covering in this tutorial.

## Submitting Workflows

The workflow can be submitted over HTTP, Kafka, programmatically, or via the UI.

To submit the workflow via HTTP send the request to `restate-ingress/workflow-name/key/run`, in our case:

<GlobalTabs>
  <GlobalTab title="TypeScript">
    ```bash theme={null}
    curl localhost:8080/SignupWorkflow/johndoe/run \
    --json '{"name": "John Doe", "email": "john@mail.com"}'
    ```
  </GlobalTab>

  <GlobalTab title="Java">
    ```bash theme={null}
    curl localhost:8080/SignupWorkflow/johndoe/run \
    --json '{"name": "John Doe", "email": "john@mail.com"}'
    ```
  </GlobalTab>

  <GlobalTab title="Go">
    ```bash theme={null}
    curl localhost:8080/SignupWorkflow/johndoe/Run \
    --json '{"name": "John Doe", "email": "john@mail.com"}'
    ```
  </GlobalTab>

  <GlobalTab title="Python">
    ```bash theme={null}
    curl localhost:8080/SignupWorkflow/johndoe/run \
    --json '{"name": "John Doe", "email": "john@mail.com"}'
    ```
  </GlobalTab>
</GlobalTabs>

Restate deduplicates workflow executions on the key, here `johndoe`.

Resubmission of the same workflow will fail with "Previously accepted". The invocation ID can be found in the request header `x-restate-id` (add `-v` to your request).

To try out a workflow multiple times during the tour, use a different key.

<AccordionGroup>
  <Accordion title="Programmatic invocation">
    <GlobalTabs>
      <GlobalTab title="TypeScript">
        You can invoke a workflow programmatically with the Restate SDK:

        ```ts client.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/tutorials/tour-of-workflows-typescript/src/client.ts#submit"}  theme={null}
        const restateClient = clients.connect({ url: "http://localhost:8080" });

        const handle = await restateClient
          .workflowClient(signupWorkflow, id)
          .workflowSubmit({ name, email });
        const result = await restateClient.result(handle);
        ```

        The workflow gets submitted and afterwards you can retrieve the result by attaching to it.

        Run the client script via:

        ```bash theme={null}
        npm run client
        ```
      </GlobalTab>

      <GlobalTab title="Java">
        You can invoke a workflow programmatically with the Restate SDK:

        ```java WorkflowSubmitter.java {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/tutorials/tour-of-workflows-java/src/main/java/my/example/WorkflowSubmitter.java#submit"}  theme={null}
        Client restateClient = Client.connect("http://localhost:8080");

        boolean result =
            SignupWorkflowClient.fromClient(restateClient, "user-123").submit(user).attach().response();
        ```

        The workflow gets submitted and afterwards you can retrieve the result by attaching to it.

        Run the client script via:

        ```bash theme={null}
        ./gradlew -PmainClass=my.example.WorkflowSubmitter run
        ```
      </GlobalTab>

      <GlobalTab title="Go">
        You can invoke a workflow programmatically with the Restate SDK:

        ```go client.go {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/tutorials/tour-of-workflows-go/client/client.go#submit"}  theme={null}
        restateClient := restateingress.NewClient("http://localhost:8080")
        result, err := restateingress.Workflow[utils.User, bool](
          restateClient, "SignupWorkflow", "user-123", "Run").
          Request(context.Background(), user)
        ```

        The workflow gets submitted and afterwards you can retrieve the result by attaching to it.

        Run the client script via:

        ```bash theme={null}
        go run ./client
        ```
      </GlobalTab>

      <GlobalTab title="Python">
        You can invoke a workflow programmatically by sending an HTTP request:

        ```python client.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/python/tutorials/tour-of-workflows-python/client.py#submit"}  theme={null}
        key = "user-123"
        url = "http://127.0.0.1:8080/SignupWorkflow/" + key + "/run"

        payload = {"name": "John Doe", "email": "john@mail.com"}
        headers = {"Content-Type": "application/json", "Accept": "application/json"}

        response = httpx.post(url, json=payload, headers=headers)
        ```

        Run the client script via:

        ```bash theme={null}
        uv run client.py
        ```
      </GlobalTab>
    </GlobalTabs>
  </Accordion>

  <Accordion title="Scheduling for later">
    You can schedule a workflow to run at a later time by specifying a delay:

    <GlobalTabs>
      <GlobalTab title="TypeScript">
        ```bash theme={null}
        curl "localhost:8080/SignupWorkflow/petewhite/run/send?delay=5m" \
        --json '{"name": "Pete White", "email": "pete@mail.com"}'
        ```
      </GlobalTab>

      <GlobalTab title="Java">
        ```bash theme={null}
        curl "localhost:8080/SignupWorkflow/petewhite/run/send?delay=5m" \
        --json '{"name": "Pete White", "email": "pete@mail.com"}'
        ```
      </GlobalTab>

      <GlobalTab title="Go">
        ```bash theme={null}
        curl "localhost:8080/SignupWorkflow/petewhite/Run/send?delay=5m" \
        --json '{"name": "Pete White", "email": "pete@mail.com"}'
        ```
      </GlobalTab>

      <GlobalTab title="Python">
        ```bash theme={null}
        curl "localhost:8080/SignupWorkflow/petewhite/run/send?delay=5m" \
        --json '{"name": "Pete White", "email": "pete@mail.com"}'
        ```
      </GlobalTab>
    </GlobalTabs>

    There is no limit to how long you can delay a workflow (works for months, even years).

    Have a look at the SDK docs to learn how to schedule workflows programmatically ([TS](/services/invocation/clients/typescript-sdk) / [Java](/services/invocation/clients/java-sdk) / [Go](/services/invocation/clients/go-sdk)).
  </Accordion>

  <Accordion title="Attach to ongoing workflow">
    If a workflow is already ongoing, you can also attach to it to get the result once it finishes:

    ```bash theme={null}
    curl localhost:8080/restate/workflow/SignupWorkflow/johndoe/attach
    ```

    Have a look at the SDK docs to learn how to attach to workflows programmatically ([TS](/services/invocation/clients/typescript-sdk) / [Java](/services/invocation/clients/java-sdk) / [Go](/services/invocation/clients/go-sdk)).
  </Accordion>
</AccordionGroup>

## Durable Execution

Restate uses Durable Execution to ensure your business logic survives any failure and resumes exactly where it left off. Unlike traditional workflow systems that require separate orchestrator infrastructure and worker management, Restate lets you deploy your workflows the same way you deploy your application code.

You write a workflow as a regular function. You use the Restate SDK to persist the steps your workflow completes in the Restate Server.

If your workflow crashes or restarts, the execution replays from the journal to restore state and continue processing:

<img alt="Durable Workflow Execution" />

To persist a workflow step, you use the `WorkflowContext` actions:

* **Durable Steps**: Restate's run actions ensures non-deterministic operations like database writes or external API calls are persisted
* **Progress Recovery**: If the workflow crashes after user creation, it resumes at the email step
* **Observability**: Full execution traces for debugging and monitoring

<Accordion title="Try out Durable Execution" icon={"laptop"}>
  Send a request for Alice:

  <GlobalTabs>
    <GlobalTab title="TypeScript">
      ```bash theme={null}
      curl localhost:8080/SignupWorkflow/alicedoe/run \
      --json '{"name": "Alice", "email": "alice@mail.com"}'
      ```
    </GlobalTab>

    <GlobalTab title="Java">
      ```bash theme={null}
      curl localhost:8080/SignupWorkflow/alicedoe/run \
      --json '{"name": "Alice", "email": "alice@mail.com"}'
      ```
    </GlobalTab>

    <GlobalTab title="Go">
      ```bash theme={null}
      curl localhost:8080/SignupWorkflow/alicedoe/Run \
      --json '{"name": "Alice", "email": "alice@mail.com"}'
      ```
    </GlobalTab>

    <GlobalTab title="Python">
      ```bash theme={null}
      curl localhost:8080/SignupWorkflow/alicedoe/run \
      --json '{"name": "Alice", "email": "alice@mail.com"}'
      ```
    </GlobalTab>
  </GlobalTabs>

  Go to the UI at `http://localhost:9070`, on the invocations page, and click on the invocation ID of the retrying invocation:

  <Frame>
    <img alt="Workflow Retries" />
  </Frame>

  You see how the invocation went through the steps of the workflow, and how it is stuck on retrying to send the welcome email.

  <GlobalTabs>
    <GlobalTab title="TypeScript">
      To fix the problem, remove the line `failOnAlice` from the `sendWelcomeEmail` function in the `utils.ts` file:

      ```ts utils.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/tutorials/tour-of-workflows-typescript/src/utils.ts#here"}  theme={null}
      export function sendWelcomeEmail(user: User) {
        failOnAlice(user.name, "send welcome email");
        console.log(`Welcome email sent: ${user.email}`);
      }
      ```
    </GlobalTab>

    <GlobalTab title="Java">
      To fix the problem, remove the line `failOnAlice` from the `sendWelcomeEmail` function in the `Utils.java` file:

      ```java Utils.java {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/tutorials/tour-of-workflows-java/src/main/java/my/example/utils/Utils.java#here"}  theme={null}
      private static void terminalErrorOnAlice(String name, String action) {
        if ("Alice".equals(name)) {
          String message =
              "[👻 SIMULATED] Failed to " + action + " for " + name + ": not available in this country";
          System.err.println(message);
          throw new TerminalException(message);
        }
      }
      ```
    </GlobalTab>

    <GlobalTab title="Go">
      To fix the problem, remove the line `failOnAlice` from the `sendWelcomeEmail` function in the `utils.go` file:

      ```go utils.go {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/tutorials/tour-of-workflows-go/examples/utils.go#here"}  theme={null}
      func SendWelcomeEmail(user User) (restate.Void, error) {
        if err := failOnAlice(user.Name, "send welcome email"); err != nil {
          return restate.Void{}, err
        }
        fmt.Printf("Welcome email sent: %s\n", user.Email)
        return restate.Void{}, nil
      }
      ```
    </GlobalTab>

    <GlobalTab title="Python">
      To fix the problem, remove the line `fail_on_alice` from the `send_welcome_email` function in the `utils.py` file:

      ```python utils.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/python/tutorials/tour-of-workflows-python/app/utils.py#here"}  theme={null}
      def send_welcome_email(user: User):
          fail_on_alice(user.name, "send welcome email")
          print(f"Welcome email sent: {user.email}")
      ```
    </GlobalTab>
  </GlobalTabs>

  Once you restart the service, the workflow finishes successfully:

  <Frame>
    <img alt="Workflow Success" />
  </Frame>
</Accordion>

## In-line Steps vs. Separate Activities

Restate workflows can execute operations inline or delegate to separate services, giving you flexibility in how you structure your applications.

* **In-line Steps** - Execute directly in the workflow, for example a run block.
* **Separate Activities** - Call dedicated services for independent scaling, separation of concerns, or different concurrency requirements.

<GlobalTabs>
  <GlobalTab title="TypeScript">
    ```ts signup-with-activities.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/tutorials/tour-of-workflows-typescript/src/workflows/signup-with-activities.ts#activities"}  theme={null}
    // Move user DB interaction to dedicated service
    const success = await ctx
      .serviceClient(userService)
      .createUser({ userId, user });
    if (!success) return { success };

    // Execute other steps inline
    await ctx.run("activate", () => activateUser(userId));
    await ctx.run("welcome", () => sendWelcomeEmail(user));
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Java">
    ```java SignupWithActivitiesWorkflow.java {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/tutorials/tour-of-workflows-java/src/main/java/my/example/workflows/SignupWithActivitiesWorkflow.java#activities"}  theme={null}
    // Move user DB interaction to dedicated service
    boolean success = true;
    UserServiceClient.fromContext(ctx).createUser(new CreateUserRequest(userId, user)).await();

    if (!success) {
      return false;
    }

    // Execute other steps inline
    ctx.run("activate", () -> activateUser(userId));
    ctx.run("welcome", () -> sendWelcomeEmail(user));
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Go">
    ```go activities.go {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/tutorials/tour-of-workflows-go/examples/activities.go#activities"}  theme={null}
    // Move user DB interaction to dedicated service
    success, err := restate.Service[bool](ctx, "UserService", "CreateUser").
      Request(CreateUserRequest{UserID: userID, User: user})
    if err != nil || !success {
      return false, err
    }

    // Execute other steps inline
    _, err = restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
      return ActivateUser(userID)
    }, restate.WithName("activate"))
    if err != nil {
      return false, err
    }

    _, err = restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
      return SendWelcomeEmail(user)
    }, restate.WithName("welcome"))
    if err != nil {
      return false, err
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Python">
    ```python signup_with_activities.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/python/tutorials/tour-of-workflows-python/app/workflows/signup_with_activities.py#activities"}  theme={null}
    # Move user DB interaction to dedicated service
    success = await ctx.service_call(
        create_user_handler, arg=CreateUserRequest(user_id=user_id, user=user)
    )
    if not success:
        return False

    # Execute other steps inline
    await ctx.run_typed("activate", activate_user, user_id=user_id)
    await ctx.run_typed("welcome", send_welcome_email, user=user)
    ```

    <GitHubLink />
  </GlobalTab>
</GlobalTabs>

You can also use this to nest workflows. The main workflow can call other workflows as activities.

Workflows are just one of the service types Restate supports. The other service types are:

* [Services](/foundations/services): collections of independent handlers which get executed with Durable Execution.
* [Virtual Objects](/foundations/services): stateful services that can be used to manage state and concurrency across multiple invocations.

<Info>
  To learn more, follow at the [Microservice Orchestration Tour](/tour/microservice-orchestration).
</Info>

<Accordion title="Try out separate activities" icon={"laptop"}>
  Submit the workflow:

  <GlobalTabs>
    <GlobalTab title="TypeScript">
      ```bash theme={null}
      curl localhost:8080/SignupWithActivitiesWorkflow/carl/run \
      --json '{"name": "Carl", "email": "carl@mail.com"}'
      ```
    </GlobalTab>

    <GlobalTab title="Java">
      ```bash theme={null}
      curl localhost:8080/SignupWithActivitiesWorkflow/carl/run \
      --json '{"name": "Carl", "email": "carl@mail.com"}'
      ```
    </GlobalTab>

    <GlobalTab title="Go">
      ```bash theme={null}
      curl localhost:8080/SignupWithActivitiesWorkflow/carl/Run \
      --json '{"name": "Carl", "email": "carl@mail.com"}'
      ```
    </GlobalTab>

    <GlobalTab title="Python">
      ```bash theme={null}
      curl localhost:8080/SignupWithActivitiesWorkflow/carl/run \
      --json '{"name": "Carl", "email": "carl@mail.com"}'
      ```
    </GlobalTab>
  </GlobalTabs>

  In the UI, you can see how the invocation called another service called user service:

  <Frame>
    <img alt="Workflow Success" />
  </Frame>
</Accordion>

## Workflow Patterns

Restate provides powerful patterns for building complex workflows using familiar programming constructs.

### Querying Workflow State

Workflows can store state in Restate, which can be queried later by other handlers:

<GlobalTabs>
  <GlobalTab title="TypeScript">
    ```ts signup-with-queries.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/tutorials/tour-of-workflows-typescript/src/workflows/signup-with-queries.ts?collapse_prequel"}  theme={null}
    export const signupWithQueries = restate.workflow({
      name: "SignupWithQueriesWorkflow",
      handlers: {
        run: async (ctx: WorkflowContext, user: User) => {
          const userId = ctx.key;

          ctx.set("user", user);
          const success = await ctx.run("create", () => createUser(userId, user));
          if (!success) {
            ctx.set("status", "failed");
            return { success };
          }
          ctx.set("status", "created");

          await ctx.run("activate", () => activateUser(userId));
          await ctx.run("welcome", () => sendWelcomeEmail(user));
          return { success };
        },

        getStatus: async (ctx: WorkflowSharedContext) => {
          return {
            status: await ctx.get("status"),
            user: await ctx.get("user"),
          };
        },
      },
    });
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Java">
    ```java SignupWithQueriesWorkflow.java {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/tutorials/tour-of-workflows-java/src/main/java/my/example/workflows/SignupWithQueriesWorkflow.java?collapse_prequel"}  theme={null}
    @Workflow
    public class SignupWithQueriesWorkflow {

      private static final StateKey<User> USER = StateKey.of("user", User.class);
      private static final StateKey<String> STATUS = StateKey.of("status", String.class);

      @Workflow
      public boolean run(WorkflowContext ctx, User user) {
        String userId = ctx.key();

        ctx.set(USER, user);
        boolean success = ctx.run("create", Boolean.class, () -> createUser(userId, user));
        if (!success) {
          ctx.set(STATUS, "failed");
          return false;
        }
        ctx.set(STATUS, "created");

        ctx.run("activate", () -> activateUser(userId));
        ctx.run("welcome", () -> sendWelcomeEmail(user));

        return true;
      }

      @Shared
      public StatusResponse getStatus(SharedWorkflowContext ctx) {
        String status = ctx.get(STATUS).orElse("unknown");
        User user = ctx.get(USER).orElse(null);
        return new StatusResponse(status, user);
      }
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Go">
    ```go queries.go {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/tutorials/tour-of-workflows-go/examples/queries.go?collapse_prequel"}  theme={null}
    type SignupWithQueriesWorkflow struct{}

    func (SignupWithQueriesWorkflow) Run(ctx restate.WorkflowContext, user User) (bool, error) {
      userID := restate.Key(ctx)

      restate.Set(ctx, "user", user)
      success, err := restate.Run(ctx, func(ctx restate.RunContext) (bool, error) {
        return CreateUser(userID, user)
      }, restate.WithName("create"))
      if err != nil || !success {
        restate.Set(ctx, "status", "failed")
        return false, err
      }
      restate.Set(ctx, "status", "created")

      _, err = restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
        return ActivateUser(userID)
      }, restate.WithName("activate"))
      if err != nil {
        return false, err
      }

      _, err = restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
        return SendWelcomeEmail(user)
      }, restate.WithName("welcome"))
      if err != nil {
        return false, err
      }

      return success, nil
    }

    func (SignupWithQueriesWorkflow) GetStatus(ctx restate.WorkflowSharedContext) (StatusResponse, error) {
      status, err := restate.Get[string](ctx, "status")
      if err != nil {
        return StatusResponse{}, err
      }
      user, err := restate.Get[User](ctx, "user")
      if err != nil {
        return StatusResponse{}, err
      }
      return StatusResponse{Status: &status, User: &user}, nil
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Python">
    ```python signup_with_queries.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/python/tutorials/tour-of-workflows-python/app/workflows/signup_with_queries.py?collapse_prequel"}  theme={null}
    signup_with_queries = restate.Workflow("SignupWithQueriesWorkflow")


    @signup_with_queries.main()
    async def run(ctx: WorkflowContext, user: User) -> bool:
        user_id = ctx.key()

        ctx.set("user", user.model_dump())
        success = await ctx.run_typed("create", create_user, user_id=user_id, user=user)
        if not success:
            ctx.set("status", "failed")
            return False
        ctx.set("status", "created")

        await ctx.run_typed("activate", activate_user, user_id=user_id)
        await ctx.run_typed("welcome", send_welcome_email, user=user)
        return True


    @signup_with_queries.handler("getStatus")
    async def get_status(ctx: WorkflowSharedContext) -> StatusResponse:
        return StatusResponse(status=await ctx.get("status"), user=await ctx.get("user"))
    ```

    <GitHubLink />
  </GlobalTab>
</GlobalTabs>

Key characteristics:

* State is isolated per workflow execution.
* State lives up to the duration of the workflow retention ([default one day](/services/configuration)).
* State is queryable from other handlers or the Restate UI.

<Frame>
  <img alt="Workflow Queries" />
</Frame>

<Accordion title="Try it querying state" icon={"laptop"}>
  Submit the workflow:

  <GlobalTabs>
    <GlobalTab title="TypeScript">
      ```bash theme={null}
      curl localhost:8080/SignupWithQueriesWorkflow/janedoe/run \
      --json '{"name": "Jane Doe", "email": "jane@mail.com"}'
      ```
    </GlobalTab>

    <GlobalTab title="Java">
      ```bash theme={null}
      curl localhost:8080/SignupWithQueriesWorkflow/janedoe/run \
      --json '{"name": "Jane Doe", "email": "jane@mail.com"}'
      ```
    </GlobalTab>

    <GlobalTab title="Go">
      ```bash theme={null}
      curl localhost:8080/SignupWithQueriesWorkflow/janedoe/Run \
      --json '{"name": "Jane Doe", "email": "jane@mail.com"}'
      ```
    </GlobalTab>

    <GlobalTab title="Python">
      ```bash theme={null}
      curl localhost:8080/SignupWithQueriesWorkflow/janedoe/run \
      --json '{"name": "Jane Doe", "email": "jane@mail.com"}'
      ```
    </GlobalTab>
  </GlobalTabs>

  In the UI, look at the state tab and filter on the `SignupWithQueriesWorkflow`.
</Accordion>

### Signaling

Pause workflow execution waiting for external events using durable promises:

<GlobalTabs>
  <GlobalTab title="TypeScript">
    ```ts signup-with-signals.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/tutorials/tour-of-workflows-typescript/src/workflows/signup-with-signals.ts?collapse_prequel"}  theme={null}
    export const signupWithSignals = restate.workflow({
      name: "SignupWithSignalsWorkflow",
      handlers: {
        run: async (ctx: WorkflowContext, user: User) => {
          const userId = ctx.key;

          // Generate verification secret and send email
          const secret = ctx.rand.uuidv4();
          await ctx.run("verify", () =>
            sendVerificationEmail(userId, user, secret),
          );

          // Wait for user to click verification link
          const clickedSecret = await ctx.promise<string>("email-verified");
          return { success: clickedSecret === secret };
        },
        verifyEmail: async (
          ctx: WorkflowSharedContext,
          req: { secret: string },
        ) => {
          // Resolve the promise to continue the main workflow
          await ctx.promise<string>("email-verified").resolve(req.secret);
        },
      },
    });
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Java">
    ```java SignupWithSignalsWorkflow.java {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/tutorials/tour-of-workflows-java/src/main/java/my/example/workflows/SignupWithSignalsWorkflow.java?collapse_prequel"}  theme={null}
    @Workflow
    public class SignupWithSignalsWorkflow {

      private static final DurablePromiseKey<String> EMAIL_VERIFIED_PROMISE =
          DurablePromiseKey.of("email-verified", String.class);

      @Workflow
      public boolean run(WorkflowContext ctx, User user) {
        String userId = ctx.key();

        // Generate verification secret and send email
        String secret = ctx.random().nextUUID().toString();
        ctx.run("verify", () -> sendVerificationEmail(userId, user, secret));

        // Wait for user to click verification link
        String clickedSecret = ctx.promise(EMAIL_VERIFIED_PROMISE).future().await();
        return secret.equals(clickedSecret);
      }

      @Shared
      public void verifyEmail(SharedWorkflowContext ctx, VerifyEmailRequest req) {
        ctx.promiseHandle(EMAIL_VERIFIED_PROMISE).resolve(req.secret());
      }
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Go">
    ```go signals.go {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/tutorials/tour-of-workflows-go/examples/signals.go?collapse_prequel"}  theme={null}
    type SignupWithSignalsWorkflow struct{}

    func (SignupWithSignalsWorkflow) Run(ctx restate.WorkflowContext, user User) (bool, error) {
      userID := restate.Key(ctx)

      // Generate verification secret and send email
      secret := restate.UUID(ctx).String()
      _, err := restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
        return SendVerificationEmail(userID, user, secret)
      }, restate.WithName("verify"))
      if err != nil {
        return false, err
      }

      // Wait for user to click verification link
      clickedSecret, err := restate.Promise[string](ctx, "email-verified").Result()
      if err != nil {
        return false, err
      }

      return clickedSecret == secret, nil
    }

    func (SignupWithSignalsWorkflow) VerifyEmail(ctx restate.WorkflowSharedContext, req VerifyEmailRequest) error {
      // Resolve the promise to continue the main workflow
      return restate.Promise[string](ctx, "email-verified").Resolve(req.Secret)
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Python">
    ```python signup_with_signals.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/python/tutorials/tour-of-workflows-python/app/workflows/signup_with_signals.py?collapse_prequel"}  theme={null}
    signup_with_signals = restate.Workflow("SignupWithSignalsWorkflow")


    @signup_with_signals.main()
    async def run(ctx: WorkflowContext, user: User) -> bool:
        user_id = ctx.key()

        # Generate verification secret and send email
        secret = str(ctx.uuid())
        await ctx.run_typed(
            "verify",
            send_verification_email,
            user_id=user_id,
            user=user,
            verification_secret=secret,
        )

        # Wait for user to click verification link
        clicked_secret = await ctx.promise("email-verified", type_hint=str).value()
        return clicked_secret == secret


    @signup_with_signals.handler("verifyEmail")
    async def verify_email(ctx: WorkflowSharedContext, req: VerifyEmailRequest) -> None:
        # Resolve the promise to continue the main workflow
        await ctx.promise("email-verified", type_hint=str).resolve(req.secret)
    ```

    <GitHubLink />
  </GlobalTab>
</GlobalTabs>

The promise can survive restarts and crashes, and can be recovered on another process.

You can use Restate's Durable Promises to handle asynchronous events without complex message queues or external state management.

Promises can be resolved before the workflow waits for them, avoiding complex synchronization issues.

<Accordion title="Try out signaling" icon={"laptop"}>
  Submit the workflow asynchronously with `/send`:

  <GlobalTabs>
    <GlobalTab title="TypeScript">
      ```bash theme={null}
      curl localhost:8080/SignupWithSignalsWorkflow/johndoe/run/send \
      --json '{"name": "John Doe", "email": "john@mail.com"}'
      ```
    </GlobalTab>

    <GlobalTab title="Java">
      ```bash theme={null}
      curl localhost:8080/SignupWithSignalsWorkflow/johndoe/run/send \
      --json '{"name": "John Doe", "email": "john@mail.com"}'
      ```
    </GlobalTab>

    <GlobalTab title="Go">
      ```bash theme={null}
      curl localhost:8080/SignupWithSignalsWorkflow/johndoe/Run/send \
      --json '{"name": "John Doe", "email": "john@mail.com"}'
      ```
    </GlobalTab>

    <GlobalTab title="Python">
      ```bash theme={null}
      curl localhost:8080/SignupWithSignalsWorkflow/johndoe/run/send \
      --json '{"name": "John Doe", "email": "john@mail.com"}'
      ```
    </GlobalTab>
  </GlobalTabs>

  In the UI, you can see the workflow waiting for the `email-verified` promise to be resolved.

  <Frame>
    <img alt="Workflow Queries" />
  </Frame>

  Try killing the service and restarting it. The workflow will continue waiting for the promise to be resolved.

  To resolve the promise, **copy over the curl request from the service logs**, which looks like this:

  <GlobalTabs>
    <GlobalTab title="TypeScript">
      ```bash theme={null}
      curl localhost:8080/SignupWithSignalsWorkflow/johndoe/verifyEmail \
      --json '{"secret": "the-secret-from-email"}'
      ```
    </GlobalTab>

    <GlobalTab title="Java">
      ```bash theme={null}
      curl localhost:8080/SignupWithSignalsWorkflow/johndoe/verifyEmail \
      --json '{"secret": "the-secret-from-email"}'
      ```
    </GlobalTab>

    <GlobalTab title="Go">
      ```bash theme={null}
      curl localhost:8080/SignupWithSignalsWorkflow/johndoe/VerifyEmail \
      --json '{"secret": "the-secret-from-email"}'
      ```
    </GlobalTab>

    <GlobalTab title="Python">
      ```bash theme={null}
      curl localhost:8080/SignupWithSignalsWorkflow/johndoe/verifyEmail \
      --json '{"secret": "the-secret-from-email"}'
      ```
    </GlobalTab>
  </GlobalTabs>

  Now the UI will show the workflow completed successfully.
</Accordion>

### Workflow Events

You can also use promises the other way around: to send events from the workflow and wait on them in one of the other handlers:

<GlobalTabs>
  <GlobalTab title="TypeScript">
    ```ts signup-with-events.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/tutorials/tour-of-workflows-typescript/src/workflows/signup-with-events.ts?collapse_prequel"}  theme={null}
    export const signupWithEvents = restate.workflow({
      name: "SignupWithEventsWorkflow",
      handlers: {
        run: async (ctx: WorkflowContext, user: User) => {
          const userId = ctx.key;

          const success = await ctx.run("create", () => createUser(userId, user));
          if (!success) {
            await ctx.promise<string>("user-created").reject("Creation failed.");
            return { success };
          }
          await ctx.promise<string>("user-created").resolve("User created.");

          await ctx.run("activate", () => activateUser(userId));
          await ctx.run("welcome", () => sendWelcomeEmail(user));
          return { success };
        },

        waitForUserCreation: async (ctx: WorkflowSharedContext) => {
          return ctx.promise<string>("user-created");
        },
      },
    });
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Java">
    ```java SignupWithEventsWorkflow.java {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/tutorials/tour-of-workflows-java/src/main/java/my/example/workflows/SignupWithEventsWorkflow.java?collapse_prequel"}  theme={null}
    @Workflow
    public class SignupWithEventsWorkflow {

      private static final DurablePromiseKey<String> USER_CREATED_PROMISE =
          DurablePromiseKey.of("user-created", String.class);

      @Workflow
      public boolean run(WorkflowContext ctx, User user) {
        String userId = ctx.key();

        boolean success = ctx.run("create", Boolean.class, () -> createUser(userId, user));
        if (!success) {
          ctx.promiseHandle(USER_CREATED_PROMISE).reject("Creation failed.");
          return false;
        }

        ctx.promiseHandle(USER_CREATED_PROMISE).resolve("User created.");

        ctx.run("activate", () -> activateUser(userId));
        ctx.run("welcome", () -> sendWelcomeEmail(user));

        return true;
      }

      @Shared
      public String waitForUserCreation(SharedWorkflowContext ctx) {
        return ctx.promise(USER_CREATED_PROMISE).future().await();
      }
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Go">
    ```go events.go {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/tutorials/tour-of-workflows-go/examples/events.go?collapse_prequel"}  theme={null}
    type SignupWithEventsWorkflow struct{}

    func (SignupWithEventsWorkflow) Run(ctx restate.WorkflowContext, user User) (bool, error) {
      userID := restate.Key(ctx)

      success, err := restate.Run(ctx, func(ctx restate.RunContext) (bool, error) {
        return CreateUser(userID, user)
      }, restate.WithName("create"))
      if err != nil || !success {
        err = restate.Promise[string](ctx, "user-created").Reject(fmt.Errorf("creation failed"))
        return false, err
      }

      if err := restate.Promise[string](ctx, "user-created").Resolve("User created."); err != nil {
        return false, err
      }

      _, err = restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
        return ActivateUser(userID)
      }, restate.WithName("activate"))
      if err != nil {
        return false, err
      }

      _, err = restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
        return SendWelcomeEmail(user)
      }, restate.WithName("welcome"))
      if err != nil {
        return false, err
      }

      return true, nil
    }

    func (SignupWithEventsWorkflow) WaitForUserCreation(ctx restate.WorkflowSharedContext) (string, error) {
      return restate.Promise[string](ctx, "user-created").Result()
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Python">
    ```python signup_with_events.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/python/tutorials/tour-of-workflows-python/app/workflows/signup_with_events.py?collapse_prequel"}  theme={null}
    signup_with_events = restate.Workflow("SignupWithEventsWorkflow")


    @signup_with_events.main()
    async def run(ctx: WorkflowContext, user: User) -> bool:
        user_id = ctx.key()

        success = await ctx.run_typed("create", create_user, user_id=user_id, user=user)
        if not success:
            await ctx.promise("user-created").reject("Creation failed.")
            return False

        await ctx.promise("user-created", type_hint=str).resolve("User created.")

        await ctx.run_typed("activate", activate_user, user_id=user_id)
        await ctx.run_typed("welcome", send_welcome_email, user=user)
        return True


    @signup_with_events.handler("waitForUserCreation")
    async def wait_for_user_creation(ctx: WorkflowSharedContext) -> str:
        return await ctx.promise("user-created").value()
    ```

    <GitHubLink />
  </GlobalTab>
</GlobalTabs>

Here, external clients can wait for the user to be created in the database.

These handlers can be called up to the workflow's retention period ([default one day](/services/configuration#workflow-retention).

<Accordion title="Try out workflow events" icon={"laptop"}>
  Submit the workflow asynchronously with `/send`:

  <GlobalTabs>
    <GlobalTab title="TypeScript">
      ```bash theme={null}
      curl localhost:8080/SignupWithEventsWorkflow/johndoe/run/send \
      --json '{"name": "John Doe", "email": "john@mail.com"}'
      ```

      Then wait for the user creation event:

      ```bash theme={null}
      curl localhost:8080/SignupWithEventsWorkflow/johndoe/waitForUserCreation
      ```
    </GlobalTab>

    <GlobalTab title="Java">
      ```bash theme={null}
      curl localhost:8080/SignupWithEventsWorkflow/johndoe/run/send \
      --json '{"name": "John Doe", "email": "john@mail.com"}'
      ```

      Then wait for the user creation event:

      ```bash theme={null}
      curl localhost:8080/SignupWithEventsWorkflow/johndoe/waitForUserCreation
      ```
    </GlobalTab>

    <GlobalTab title="Go">
      ```bash theme={null}
      curl localhost:8080/SignupWithEventsWorkflow/johndoe/Run/send \
      --json '{"name": "John Doe", "email": "john@mail.com"}'
      ```

      Then wait for the user creation event:

      ```bash theme={null}
      curl localhost:8080/SignupWithEventsWorkflow/johndoe/WaitForUserCreation
      ```
    </GlobalTab>

    <GlobalTab title="Python">
      ```bash theme={null}
      curl localhost:8080/SignupWithEventsWorkflow/johndoe/run/send \
      --json '{"name": "John Doe", "email": "john@mail.com"}'
      ```

      Then wait for the user creation event:

      ```bash theme={null}
      curl localhost:8080/SignupWithEventsWorkflow/johndoe/waitForUserCreation
      ```
    </GlobalTab>
  </GlobalTabs>

  You will get a response like `"User created."`. If the promise hadn't been resolved yet, the request will wait until it is.
</Accordion>

### Timers and Scheduling

Use durable timers for long-running workflows with timeouts and retries:

<GlobalTabs>
  <GlobalTab title="TypeScript">
    ```ts signup-with-timers.ts expandable {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/tutorials/tour-of-workflows-typescript/src/workflows/signup-with-timers.ts?collapse_prequel"}  theme={null}
    export const signupWithTimers = restate.workflow({
      name: "SignupWithTimersWorkflow",
      handlers: {
        run: async (ctx: WorkflowContext, user: User) => {
          const userId = ctx.key;

          const secret = ctx.rand.uuidv4();
          await ctx.run("verify", () =>
            sendVerificationEmail(userId, user, secret),
          );

          const clickedPromise = ctx.promise<string>("email-verified").get();
          const verificationTimeout = ctx.sleep({ days: 1 });
          while (true) {
            const reminderTimer = ctx.sleep({ seconds: 15 });

            // Wait for email verification, reminder timer or timeout
            const result = await RestatePromise.race([
              clickedPromise.map(() => "verified"),
              reminderTimer.map(() => "reminder"),
              verificationTimeout.map(() => "timeout"),
            ]);

            switch (result) {
              case "verified":
                const clickedSecret = await clickedPromise;
                return { success: clickedSecret === secret };
              case "reminder":
                await ctx.run("send reminder", () =>
                  sendReminderEmail(userId, user, secret),
                );
                break;
              case "timeout":
                throw new TerminalError(
                  "Email verification timed out after 24 hours",
                );
            }
          }
        },
        verifyEmail: async (
          ctx: WorkflowSharedContext,
          req: { secret: string },
        ) => {
          await ctx.promise<string>("email-verified").resolve(req.secret);
        },
      },
    });
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Java">
    ```java SignupWithTimersWorkflow.java {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/tutorials/tour-of-workflows-java/src/main/java/my/example/workflows/SignupWithTimersWorkflow.java?collapse_prequel"}  theme={null}
    @Workflow
    public class SignupWithTimersWorkflow {

      private static final DurablePromiseKey<String> EMAIL_VERIFIED_PROMISE =
          DurablePromiseKey.of("email-verified", String.class);

      @Workflow
      public boolean run(WorkflowContext ctx, User user) {
        String userId = ctx.key();

        var confirmationFuture = ctx.promise(EMAIL_VERIFIED_PROMISE).future();
        var secret = ctx.random().nextUUID().toString();
        ctx.run("verify", () -> sendVerificationEmail(userId, user, secret));

        var verificationTimeout = ctx.timer(Duration.ofDays(1));

        while (true) {
          var reminderTimer = ctx.timer(Duration.ofSeconds(10));

          var selected =
              Select.<String>select()
                  .when(confirmationFuture, res -> "verified")
                  .when(reminderTimer, unused -> "reminder")
                  .when(verificationTimeout, unused -> "timeout")
                  .await();

          switch (selected) {
            case "verified":
              var clickedSecret = confirmationFuture.await();
              return secret.equals(clickedSecret);
            case "reminder":
              ctx.run("send reminder", () -> sendReminderEmail(userId, user, secret));
              break;
            case "timeout":
              throw new TerminalException("Verification timed out");
          }
        }
      }

      @Shared
      public void verifyEmail(SharedWorkflowContext ctx, VerifyEmailRequest req) {
        ctx.promiseHandle(EMAIL_VERIFIED_PROMISE).resolve(req.secret());
      }
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Go">
    ```go timers.go {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/tutorials/tour-of-workflows-go/examples/timers.go?collapse_prequel"}  theme={null}
    type SignupWithTimersWorkflow struct{}

    func (SignupWithTimersWorkflow) Run(ctx restate.WorkflowContext, user User) (bool, error) {
      userID := restate.Key(ctx)

      secret := restate.UUID(ctx).String()
      _, err := restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
        return SendVerificationEmail(userID, user, secret)
      }, restate.WithName("verify"))
      if err != nil {
        return false, err
      }

      clickedPromise := restate.Promise[string](ctx, "email-verified")
      verificationTimeoutFuture := restate.After(ctx, 24*time.Hour)

      for {
        reminderTimerFuture := restate.After(ctx, 15*time.Second)

        // Create futures for racing
        resFut, err := restate.WaitFirst(ctx,
          clickedPromise,
          reminderTimerFuture,
          verificationTimeoutFuture,
        )
        if err != nil {
          return false, err
        }

        switch resFut {
        case clickedPromise:
          clickedSecret, err := clickedPromise.Result()
          if err != nil {
            return false, err
          }
          return clickedSecret == secret, nil
        case reminderTimerFuture:
          _, err = restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
            return SendReminderEmail(userID, user, secret)
          }, restate.WithName("send reminder"))
          if err != nil {
            return false, err
          }
          break // Break out the switch to continue the main loop
        case verificationTimeoutFuture:
          return false, restate.TerminalError(fmt.Errorf("email verification timed out after 24 hours"))
        }
      }
    }

    func (SignupWithTimersWorkflow) VerifyEmail(ctx restate.WorkflowSharedContext, req VerifyEmailRequest) error {
      return restate.Promise[string](ctx, "email-verified").Resolve(req.Secret)
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Python">
    ```python signup_with_timers.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/python/tutorials/tour-of-workflows-python/app/workflows/signup_with_timers.py?collapse_prequel"}  theme={null}
    signup_with_timers = restate.Workflow("SignupWithTimersWorkflow")


    @signup_with_timers.main()
    async def run(ctx: WorkflowContext, user: User) -> bool:
        user_id = ctx.key()

        secret = str(ctx.uuid())
        await ctx.run_typed(
            "verify",
            send_verification_email,
            user_id=user_id,
            user=user,
            verification_secret=secret,
        )

        clicked_promise = ctx.promise("email-verified", type_hint=str)
        verification_timeout = ctx.sleep(timedelta(days=1))

        while True:
            reminder_timer = ctx.sleep(timedelta(seconds=15))

            # Wait for email verification, reminder timer or timeout
            result = await restate.select(
                verification=clicked_promise.value(),
                reminder=reminder_timer,
                timeout=verification_timeout,
            )

            match result:
                case ["verification", clicked_secret]:
                    return clicked_secret == secret
                case ["reminder", _]:
                    await ctx.run_typed(
                        "remind",
                        send_reminder_email,
                        user_id=user_id,
                        user=user,
                        verification_secret=secret,
                    )
                case ["timeout", _]:
                    raise TerminalError("Email verification timed out after 24 hours")


    @signup_with_timers.handler("verifyEmail")
    async def verify_email(ctx: WorkflowSharedContext, req: VerifyEmailRequest) -> None:
        await ctx.promise("email-verified", type_hint=str).resolve(req.secret)
    ```

    <GitHubLink />
  </GlobalTab>
</GlobalTabs>

Because Restate lets you write workflows as regular functions, you can use your language's native constructs like `while`/`for` loops, `if` statements, and `switch` cases to control flow.
This makes it easy to implement complex logic with timers, loops, and conditional execution.

<Accordion title="Try out timers" icon={"laptop"}>
  Submit the workflow asynchronously with `/send`:

  <GlobalTabs>
    <GlobalTab title="TypeScript">
      ```bash theme={null}
      curl localhost:8080/SignupWithTimersWorkflow/johndoe/run/send \
      --json '{"name": "John Doe", "email": "john@mail.com"}'
      ```
    </GlobalTab>

    <GlobalTab title="Java">
      ```bash theme={null}
      curl localhost:8080/SignupWithTimersWorkflow/johndoe/run/send \
      --json '{"name": "John Doe", "email": "john@mail.com"}'
      ```
    </GlobalTab>

    <GlobalTab title="Go">
      ```bash theme={null}
      curl localhost:8080/SignupWithTimersWorkflow/johndoe/Run/send \
      --json '{"name": "John Doe", "email": "john@mail.com"}'
      ```
    </GlobalTab>

    <GlobalTab title="Python">
      ```bash theme={null}
      curl localhost:8080/SignupWithTimersWorkflow/johndoe/run/send \
      --json '{"name": "John Doe", "email": "john@mail.com"}'
      ```
    </GlobalTab>
  </GlobalTabs>

  See in the UI how the workflow is waiting for the email verification to be resolved, and sends reminder emails every 15 seconds:

  <Frame>
    <img alt="Workflow Timers" />
  </Frame>

  Try killing the service and restarting it. The workflow will continue sending reminders as if it never stopped.

  To resolve the promise, **copy over the curl request from the service logs**, which looks like this:

  <GlobalTabs>
    <GlobalTab title="TypeScript">
      ```bash theme={null}
      curl localhost:8080/SignupWithTimersWorkflow/johndoe/verifyEmail \
      --json '{"secret": "the-secret-from-email"}'
      ```
    </GlobalTab>

    <GlobalTab title="Java">
      ```bash theme={null}
      curl localhost:8080/SignupWithTimersWorkflow/johndoe/verifyEmail \
      --json '{"secret": "the-secret-from-email"}'
      ```
    </GlobalTab>

    <GlobalTab title="Go">
      ```bash theme={null}
      curl localhost:8080/SignupWithTimersWorkflow/johndoe/verifyEmail \
      --json '{"secret": "the-secret-from-email"}'
      ```
    </GlobalTab>

    <GlobalTab title="Python">
      ```bash theme={null}
      curl localhost:8080/SignupWithTimersWorkflow/johndoe/verifyEmail \
      --json '{"secret": "the-secret-from-email"}'
      ```
    </GlobalTab>
  </GlobalTabs>

  Now the UI will show the workflow completed successfully.
</Accordion>

### Parallel Execution

The timers example ran three operations in parallel: two timers and awaiting a promise.

Restate supports different ways of waiting for parallel operations to complete and takes care of retries and recovery for you.

Have a look at the Concurrent Tasks docs for your SDK to learn more ([TS](/develop/ts/concurrent-tasks) / [Java / Kotlin](/develop/java/concurrent-tasks) / [Python](/develop/python/concurrent-tasks) / [Go](/develop/go/concurrent-tasks)).

## Error Handling

By default, Restate retries failures infinitely with an exponential backoff strategy.
For some failures, you might not want to retry or only retry a limited number of times.

For these cases, Restate distinguishes between two types of errors: transient errors and terminal errors.

### Transient vs Terminal Errors

* **Transient errors**: These are temporary issues that can be retried, such as network timeouts or service unavailability. Restate automatically retries these errors.
* **Terminal errors**: These indicate a failure that will not be retried, such as invalid input or business logic violations. Restate stops execution and allows you to handle these errors gracefully.

Throw a terminal error in your handler to indicate a terminal failure:

<GlobalTabs>
  <GlobalTab title="TypeScript">
    ```typescript {"CODE_LOAD::ts/src/tour/workflows/terminal_error.ts#terminal_error"}  theme={null}
    throw new TerminalError("Subscription plan not available");
    ```
  </GlobalTab>

  <GlobalTab title="Java">
    ```java {"CODE_LOAD::java/src/main/java/tour/workflows/WorkflowErrorHandler.java#here"}  theme={null}
    throw new TerminalException("Subscription plan not available");
    ```
  </GlobalTab>

  <GlobalTab title="Go">
    ```go {"CODE_LOAD::go/tour/workflows/errorhandling.go#here"}  theme={null}
    return restate.TerminalError(fmt.Errorf("subscription plan not available"))
    ```
  </GlobalTab>

  <GlobalTab title="Python">
    ```python {"CODE_LOAD::python/src/tour/workflows/terminal_error.py#here"}  theme={null}
    from restate.exceptions import TerminalError

    raise TerminalError("Invalid credit card")
    ```
  </GlobalTab>
</GlobalTabs>

### Configuring Retry Behavior

You can limit the number of retries of a run block:

<GlobalTabs>
  <GlobalTab title="TypeScript">
    ```ts signup-with-retries.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/tutorials/tour-of-workflows-typescript/src/workflows/signup-with-retries.ts#retries"}  theme={null}
    try {
      const retryPolicy = {
        maxRetryAttempts: 3,
        initialRetryInterval: { seconds: 1 },
      };
      await ctx.run("welcome", () => sendWelcomeEmail(user), retryPolicy);
    } catch (error) {
      // This gets hit on retry exhaustion with a terminal error
      // Log and continue; without letting the workflow fail
      console.error("Failed to send welcome email after retries:", error);
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Java">
    ```java SignupWithRetriesWorkflow.java {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/tutorials/tour-of-workflows-java/src/main/java/my/example/workflows/SignupWithRetriesWorkflow.java#retries"}  theme={null}
    try {
      RetryPolicy myRunRetryPolicy =
          RetryPolicy.defaultPolicy()
              .setInitialDelay(Duration.ofMillis(500))
              .setExponentiationFactor(2)
              .setMaxDelay(Duration.ofSeconds(10))
              .setMaxAttempts(3)
              .setMaxDuration(Duration.ofSeconds(30));

      ctx.run("welcome", myRunRetryPolicy, () -> sendWelcomeEmail(user));
    } catch (TerminalException error) {
      // This gets hit on retry exhaustion with a terminal error
      // Log and continue; without letting the workflow fail
      System.err.println("Failed to send welcome email after retries: " + error.getMessage());
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Go">
    ```go retries.go {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/tutorials/tour-of-workflows-go/examples/retries.go#retries"}  theme={null}
    _, err = restate.Run(ctx,
      func(ctx restate.RunContext) (restate.Void, error) {
        return SendWelcomeEmail(user)
      },
      restate.WithName("welcome"),
      restate.WithMaxRetryAttempts(3),
      restate.WithInitialRetryInterval(1000),
    )
    if err != nil {
      // This gets hit on retry exhaustion with a terminal error
      // Log and continue; without letting the workflow fail
      fmt.Printf("Couldn't send the email due to terminal error %s", err)
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Python">
    ```python signup_with_retries.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/python/tutorials/tour-of-workflows-python/app/workflows/signup_with_retries.py?collapse_prequel"}  theme={null}
    signup_with_retries = restate.Workflow("SignupWithRetriesWorkflow")


    @signup_with_retries.main()
    async def run(ctx: WorkflowContext, user: User) -> bool:
        user_id = ctx.key()

        success = await ctx.run_typed("create", create_user, user_id=user_id, user=user)
        if not success:
            return False

        await ctx.run_typed("activate", activate_user, user_id=user_id)

        # Configure retry policy
        try:
            await ctx.run_typed(
                "welcome",
                send_welcome_email,
                restate.RunOptions(
                    max_attempts=3, max_retry_duration=timedelta(seconds=30)
                ),
                user=user,
            )
        except TerminalError as error:
            # This gets hit on retry exhaustion with a terminal error
            # Log and continue; without letting the workflow fail
            print(f"Failed to send welcome email after retries: {error}")

        return True
    ```

    <GitHubLink />
  </GlobalTab>
</GlobalTabs>

When the retries are exhausted, the run block will throw a terminal error, that you can handle in your handler logic.

<Accordion title="Try out retry policies" icon={"laptop"}>
  Submit the workflow for Alice:

  <GlobalTabs>
    <GlobalTab title="TypeScript">
      ```bash theme={null}
      curl localhost:8080/SignupWithRetriesWorkflow/alice/run \
      --json '{"name": "Alice", "email": "alice@mail.com"}'
      ```
    </GlobalTab>

    <GlobalTab title="Java">
      ```bash theme={null}
      curl localhost:8080/SignupWithRetriesWorkflow/alice/run \
      --json '{"name": "Alice", "email": "alice@mail.com"}'
      ```
    </GlobalTab>

    <GlobalTab title="Go">
      ```bash theme={null}
      curl localhost:8080/SignupWithRetriesWorkflow/alice/Run \
      --json '{"name": "Alice", "email": "alice@mail.com"}'
      ```
    </GlobalTab>

    <GlobalTab title="Python">
      ```bash theme={null}
      curl localhost:8080/SignupWithRetriesWorkflow/alice/run \
      --json '{"name": "Alice", "email": "alice@mail.com"}'
      ```
    </GlobalTab>
  </GlobalTabs>

  In the UI, you can see how the workflow is waiting for the `welcome` step to finish, and how it retries sending the welcome email up to 3 times across three seconds.
  After three attempts, the workflow continues without failing:

  <Frame>
    <img alt="Workflow Retries" />
  </Frame>
</Accordion>

Learn more with the [Error Handling Guide](/guides/error-handling).

## Sagas and rollback

On a terminal failure, Restate stops the execution of the handler.
You might, however, want to roll back the changes made by the workflow to keep your system in a consistent state.
This is where Sagas come in.

Sagas are a pattern for rolling back changes made by a workflow when it fails.

In Restate, you can implement a saga by building a list of compensating actions for each step of the workflow.
On a terminal failure, you execute them in reverse order:

<GlobalTabs>
  <GlobalTab title="TypeScript">
    ```ts signup-with-sagas.ts {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/tutorials/tour-of-workflows-typescript/src/workflows/signup-with-sagas.ts?collapse_prequel"}  theme={null}
    export const signupWithSagas = restate.workflow({
      name: "SignupWithSagasWorkflow",
      handlers: {
        run: async (ctx: WorkflowContext, user: User) => {
          const userId = ctx.key;
          const compensations = [];

          try {
            compensations.push(() => ctx.run("delete", () => deleteUser(userId)));
            await ctx.run("create", () => createUser(userId, user));

            compensations.push(() =>
              ctx.run("deactivate", () => deactivateUser(userId)),
            );
            await ctx
              .run("activate", () => activateUser(userId))
              .orTimeout({ minutes: 5 });

            compensations.push(() =>
              ctx.run("unsubscribe", () => cancelSubscription(user)),
            );
            await ctx.run("subscribe", () => subscribeToPaidPlan(user));
          } catch (e) {
            if (e instanceof restate.TerminalError) {
              for (const compensation of compensations.reverse()) {
                await compensation();
              }
            }
            return { success: false };
          }
          return { success: true };
        },
      },
    });
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Java">
    ```java SignupWithSagasWorkflow.java {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/tutorials/tour-of-workflows-java/src/main/java/my/example/workflows/SignupWithSagasWorkflow.java?collapse_prequel"}  theme={null}
    @Workflow
    public class SignupWithSagasWorkflow {

      @Workflow
      public boolean run(WorkflowContext ctx, User user) {
        String userId = ctx.key();
        List<Runnable> compensations = new ArrayList<>();

        try {
          compensations.add(() -> ctx.run("delete", () -> deleteUser(userId)));
          ctx.run("create", () -> createUser(userId, user));

          compensations.add(() -> ctx.run("deactivate", () -> deactivateUser(userId)));
          ctx.run("activate", () -> activateUser(userId));

          compensations.add(() -> ctx.run("unsubscribe", () -> cancelSubscription(user)));
          ctx.run("subscribe", () -> subscribeToPaidPlan(user));

        } catch (TerminalException e) {
          // Run compensations in reverse order
          Collections.reverse(compensations);
          for (Runnable compensation : compensations) {
            compensation.run();
          }
          return false;
        }

        return true;
      }
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Go">
    ```go sagas.go {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/tutorials/tour-of-workflows-go/examples/sagas.go?collapse_prequel"}  theme={null}
    type SagasWorkflow struct{}

    func (SagasWorkflow) Run(ctx restate.WorkflowContext, user User) (res bool, err error) {
      userID := restate.Key(ctx)
      var compensations []func() error

      defer func() {
        // All errors that end up here are terminal errors, so run compensations
        // (Retry-able errors got returned by the SDK without ending up here)
        if err != nil {
          for _, compensation := range slices.Backward(compensations) {
            if compErr := compensation(); compErr != nil {
              err = compErr
            }
          }
        }
      }()

      // Add compensation for user creation
      compensations = append(compensations, func() error {
        _, err := restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
          return DeleteUser(userID)
        })
        return err
      })

      _, err = restate.Run(ctx, func(ctx restate.RunContext) (bool, error) {
        return CreateUser(userID, user)
      }, restate.WithName("create"))
      if err != nil {
        return false, err
      }

      // Add compensation for user activation
      compensations = append(compensations, func() error {
        _, err := restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
          return DeactivateUser(userID)
        })
        return err
      })

      _, err = restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
        return ActivateUser(userID)
      })
      if err != nil {
        return false, err
      }

      // Add compensation for subscription
      compensations = append(compensations, func() error {
        _, err := restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
          return CancelSubscription(user)
        })
        return err
      })

      _, err = restate.Run(ctx, func(ctx restate.RunContext) (bool, error) {
        return SubscribeToPaidPlan(user)
      })
      if err != nil {
        return false, err
      }

      return true, nil
    }
    ```

    <GitHubLink />
  </GlobalTab>

  <GlobalTab title="Python">
    ```python signup_with_sagas.py {"CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/python/tutorials/tour-of-workflows-python/app/workflows/signup_with_sagas.py?collapse_prequel"}  theme={null}
    signup_with_sagas = restate.Workflow("SignupWithSagasWorkflow")


    @signup_with_sagas.main()
    async def run(ctx: WorkflowContext, user: User) -> bool:
        user_id = ctx.key()
        compensations = []

        try:
            compensations.append(
                lambda: ctx.run_typed("delete", delete_user, user_id=user_id)
            )
            await ctx.run_typed("create", create_user, user_id=user_id, user=user)

            compensations.append(
                lambda: ctx.run_typed("deactivate", deactivate_user, user_id=user_id)
            )
            await ctx.run_typed("activate", activate_user, user_id=user_id)

            compensations.append(
                lambda: ctx.run_typed("unsubscribe", cancel_subscription, user=user)
            )
            await ctx.run_typed("subscribe", subscribe_to_paid_plan, user=user)
        except TerminalError:
            # Run compensations in reverse order
            for compensation in reversed(compensations):
                await compensation()
            return False

        return True
    ```

    <GitHubLink />
  </GlobalTab>
</GlobalTabs>

**Benefits with Restate:**

* The list of compensations can be recovered after a crash, and Restate knows which compensations still need to be run.
* Sagas always run till completion (success or complete rollback)
* Full trace of all operations and compensations
* No complex state machines needed

<Accordion title="Try out sagas" icon={"laptop"}>
  Submit the workflow for Alice:

  <GlobalTabs>
    <GlobalTab title="TypeScript">
      ```bash theme={null}
      curl localhost:8080/SignupWithSagasWorkflow/alice/run \
      --json '{"name": "Alice", "email": "alice@mail.com"}'
      ```
    </GlobalTab>

    <GlobalTab title="Java">
      ```bash theme={null}
      curl localhost:8080/SignupWithSagasWorkflow/alice/run \
      --json '{"name": "Alice", "email": "alice@mail.com"}'
      ```
    </GlobalTab>

    <GlobalTab title="Go">
      ```bash theme={null}
      curl localhost:8080/SignupWithSagasWorkflow/alice/Run \
      --json '{"name": "Alice", "email": "alice@mail.com"}'
      ```
    </GlobalTab>

    <GlobalTab title="Python">
      ```bash theme={null}
      curl localhost:8080/SignupWithSagasWorkflow/alice/run \
      --json '{"name": "Alice", "email": "alice@mail.com"}'
      ```
    </GlobalTab>
  </GlobalTabs>

  Alice is not able to get a subscription, so the workflow will fail and run compensations:

  <Frame>
    <img alt="Workflow Sagas" />
  </Frame>
</Accordion>

Learn more with the [Sagas Guide](/guides/sagas).

## Cancellation

You can cancel user signup workflows via HTTP, CLI, UI, or programmatically from other services.

When you cancel a workflow, Restate stops the execution by throwing a Terminal Error.
This allows your handler to run compensating actions or clean up resources.

First, the cancellation gets propagated to the leaf nodes of the call tree (in case the workflow called other services or workflows).
Then, the cancellation propagates back up the tree, allowing each handler to run its compensations.

<Accordion title="Try out cancellation" icon={"laptop"}>
  Start the workflow asynchronously with `/send`:

  <GlobalTabs>
    <GlobalTab title="TypeScript">
      ```bash theme={null}
      curl localhost:8080/SignupWithSignalsWorkflow/eve/run/send \
      --json '{"name": "Eve", "email": "eve@mail.com"}'
      ```
    </GlobalTab>

    <GlobalTab title="Java">
      ```bash theme={null}
      curl localhost:8080/SignupWithSignalsWorkflow/eve/run/send \
      --json '{"name": "Eve", "email": "eve@mail.com"}'
      ```
    </GlobalTab>

    <GlobalTab title="Go">
      ```bash theme={null}
      curl localhost:8080/SignupWithSignalsWorkflow/eve/Run/send \
      --json '{"name": "Eve", "email": "eve@mail.com"}'
      ```
    </GlobalTab>

    <GlobalTab title="Python">
      ```bash theme={null}
      curl localhost:8080/SignupWithSignalsWorkflow/eve/run/send \
      --json '{"name": "Eve", "email": "eve@mail.com"}'
      ```
    </GlobalTab>
  </GlobalTabs>

  This returns the invocation ID, which you can use to cancel the workflow later via the UI, CLI or HTTP:

  <CodeGroup>
    ```bash CLI theme={null}
    restate invocations cancel inv_1gdJBtdVEcM942bjcDmb1c1khoaJe11Hbz
    ```

    ```bash curl theme={null}
    curl -X PATCH http://localhost:9070/invocations/inv_1gdJBtdVEcM942bjcDmb1c1khoaJe11Hbz/cancel
    ```
  </CodeGroup>

  You can see the cancellation in the UI:

  <Frame>
    <img alt="Workflow Cancellation" />
  </Frame>
</Accordion>

Check out the SDK docs, to learn how to programmatically cancel a workflow from another service ([TS](/develop/ts/service-communication#cancel-an-invocation) / [Java / Kotlin](/develop/java/service-communication#cancel-an-invocation)  / [Python](/develop/python/service-communication#cancel-an-invocation)/ [Go](/develop/go/service-communication#cancel-an-invocation)).

## Serverless Deployment

Restate lets you run your workflows and services on serverless platforms like AWS Lambda or Google Cloud Run.

Restate automatically suspends workflows when they are waiting for events or timers, and resumes them when the event occurs or the timer expires.
This means you can run long-running workflows on function-as-a-service platforms without paying for the wait time.

Turning your signup workflow into a serverless function is as simple as adapting the endpoint:

<GlobalTabs>
  <GlobalTab title="TypeScript">
    ```typescript {"CODE_LOAD::ts/src/tour/workflows/serving_lambda.ts#lambda"}  theme={null}
    import * as restate from "@restatedev/restate-sdk/lambda";
    export const handler = restate.createEndpointHandler({
      services: [signupWorkflow],
    });
    ```

    Learn more from the [serving docs](/develop/ts/serving).
  </GlobalTab>

  <GlobalTab title="Java">
    ```java {"CODE_LOAD::java/src/main/java/tour/workflows/WorkflowServingLambda.java#here"}  theme={null}
    import dev.restate.sdk.endpoint.Endpoint;
    import dev.restate.sdk.lambda.BaseRestateLambdaHandler;

    class MyLambdaHandler extends BaseRestateLambdaHandler {
      @Override
      public void register(Endpoint.Builder builder) {
        builder.bind(new SignupWorkflow());
      }
    }
    ```

    Learn more from the [serving docs](/develop/java/serving).
  </GlobalTab>

  <GlobalTab title="Go">
    ```go {"CODE_LOAD::go/tour/workflows/servinglambda.go#here"}  theme={null}
    handler, err := server.NewRestate().
      Bind(restate.Reflect(SignupWorkflow{})).
      Bidirectional(false).
      LambdaHandler()
    if err != nil {
      log.Fatal(err)
    }
    lambda.Start(handler)
    ```

    Learn more from the [serving docs](/develop/go/serving).
  </GlobalTab>

  <GlobalTab title="Python">
    ```python {"CODE_LOAD::python/src/tour/workflows/lambda_handler.py#here"}  theme={null}
    handler = restate.app(services=[signup_workflow])
    ```

    Learn more from the [serving docs](/develop/python/serving).
  </GlobalTab>
</GlobalTabs>

## Summary

Restate workflows provide:

* **Natural Programming**: Write workflows as regular functions in your preferred language
* **Automatic Durability**: Built-in resilience without infrastructure complexity
* **Flexible Patterns**: State management, events, timers, and parallel execution
* **Modern Deployment**: Strong serverless support and simple single-binary deployment

With Restate, you can build complex, long-running workflows using familiar programming patterns while getting the durability you need.


# AI Agents
Source: https://docs.restate.dev/use-cases/ai-agents

Build resilient, observable AI agents that recover from failures and handle complex multi-step tasks.

## Durable Agents and Workflows

Restate automatically handles the reliability challenges of AI agents:

<img alt="Durable AI Agent Execution" />

* **Automatically retry transient errors** like rate limits and network failures
* **Persist steps** (LLM calls, tools) and recover previous progress after failures
* **Suspend long-running agents** when idle to save costs

## Plugs into Popular SDKs

Restate works independently of any SDK and specific AI stack, but its lightweight programming abstraction integrates easily into many popular SDKs. A few lines turn your agent into a durable agent.

```typescript {"CODE_LOAD::ts/src/usecases/agents/weather-agent.ts#here"}  theme={null}
const model = wrapLanguageModel({
  model: openai("gpt-4o"),
  middleware: durableCalls(restateContext, { maxRetryAttempts: 3 }),
});
```

Works with [Vercel AI SDK](/ai/sdk-integrations/vercel-ai-sdk), [OpenAI](/ai/sdk-integrations/openai-agents-sdk), and [others](/ai#llm-&-agent-sdk-integrations).

## Human-in-the-Loop and Workflow Patterns

Restate's workflows-as-code and building blocks make it easy to reliably implement:

<CardGroup>
  <Card title="Human Approval" icon="user-check" href="/ai/patterns/human-in-the-loop">
    Durable waiting for human decisions with crash-proof timeouts
  </Card>

  <Card title="Parallelization" icon="arrows-split-up-and-left" href="/ai/patterns/parallelization">
    Speed up multi-step workflows with recoverable parallel tasks
  </Card>

  <Card title="Sub-workflows" icon="sitemap" href="/ai/patterns/tools">
    Break complex agents into smaller, specialized workflows
  </Card>

  <Card title="Multi-agent Orchestration" icon="users-gear" href="/ai/patterns/multi-agent">
    Coordinate specialized agents with reliable communication
  </Card>

  <Card title="Compensation Patterns" icon="arrow-rotate-left" href="/ai/patterns/rollback">
    Automatically undo previous actions when later steps fail
  </Card>

  <Card title="And much more" icon="stars" href="/ai">
    Build agents that can be paused, modified, and resumed during execution
  </Card>
</CardGroup>

## Observability and Debugging

See all ongoing executions with detailed journals of agent steps:

<Frame>
  <img alt="AI Agent Execution Trace" />
</Frame>

* **Complete execution timeline**: Every LLM call and tool execution
* **Debug failed agents**: Inspect exactly where and why agents failed
* **Agent control**: Pause, resume, restart agents during development and production

## End-to-End Resilient Applications

Agents are just a part of your application. Restate covers the plumbing around your agents:

<Frame>
  <img alt="Application Structure" />
</Frame>

* **Queuing, state, session management**: Built-in primitives for reliable agent coordination
* **Deterministic workflows**: Complement agents with structured business logic
* **Reliable asynchronous tasks**: Handle background work and inter-service communication

## Flexible Deployments and Scalability

Restate's durable execution runtime lets you run your durable code where you want at the scale you want:

<img alt="Application Structure" />

* **Scale to millions** of concurrent agent executions
* **Deploy your agents** on FaaS or containers
* **You own the infrastructure**: Run on Restate Cloud or self-host

## Getting Started

<CardGroup>
  <Card title="Quickstart" icon="rocket" href="/quickstart">
    Set up Restate and run your first agent
  </Card>

  <Card title="AI recipes" icon="pen" href="/ai">
    Build durable agents, chatbots, and multi-agent systems
  </Card>

  <Card title="AI Examples" icon="code" href="https://github.com/restatedev/ai-examples">
    Explore templates, examples, and SDK integrations
  </Card>
</CardGroup>

<Info>
  Questions? Join our community on [Discord](https://discord.restate.dev) or [Slack](https://slack.restate.dev).
</Info>


# Event Processing
Source: https://docs.restate.dev/use-cases/event-processing

Build lightweight, transactional event handlers with built-in resiliency.

Kafka event processing requires managing Kafka consumers, handling retries, maintaining state stores, and coordinating complex workflows. Restate eliminates this complexity by providing **lightweight, transactional event processing** with zero consumer management and built-in state.

## Workflows from Kafka

Build event handlers with complex control flow, loops, timers, and transactional guarantees:

<CodeGroup>
  ```typescript TypeScript {"CODE_LOAD::ts/src/usecases/eventprocessing/user-feed.ts#here"}  theme={null}
  export default restate.object({
    name: "userFeed",
    handlers: {
      processPost: async (ctx: restate.ObjectContext, post: SocialMediaPost) => {
        const userId = ctx.key;

        // Durable side effect: persisted and replayed on retries
        const postId = await ctx.run(() => createPost(userId, post));

        // Wait for processing to complete with durable timers
        while ((await ctx.run(() => getPostStatus(postId))) === PENDING) {
          await ctx.sleep({ seconds: 5 });
        }

        await ctx.run(() => updateUserFeed(userId, postId));
      },
    },
  });
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/usecases/eventprocessing/eventtransactions/UserFeed.java#here"}  theme={null}
  @VirtualObject
  public class UserFeed {
    @Handler
    public void processPost(ObjectContext ctx, SocialMediaPost post) {
      String userId = ctx.key();

      String postId = ctx.run(String.class, () -> createPost(userId, post));

      while (ctx.run(String.class, () -> getPostStatus(postId)).equals("PENDING")) {
        ctx.sleep(Duration.ofSeconds(5));
      }

      ctx.run(() -> updateUserFeed(userId, postId));
    }
  }
  ```

  ```python Python {"CODE_LOAD::python/src/usecases/eventprocessing/user_feed.py#here"}  theme={null}
  user_feed = restate.VirtualObject("UserFeed")


  @user_feed.handler()
  async def process_post(ctx: restate.ObjectContext, post: SocialMediaPost):
      user_id = ctx.key()

      post_id = await ctx.run_typed(
          "create post", create_post, user_id=user_id, post=post
      )

      while (
          await ctx.run_typed("get status", get_post_status, post_id=post_id)
          == Status.PENDING
      ):
          await ctx.sleep(timedelta(seconds=5))

      await ctx.run_typed("update feed", update_user_feed, user=user_id, post_id=post_id)
  ```

  ```go Go {"CODE_LOAD::go/usecases/eventprocessing/userfeed.go#here"}  theme={null}
  type UserFeed struct{}

  func (UserFeed) ProcessPost(ctx restate.ObjectContext, post SocialMediaPost) error {
    var userId = restate.Key(ctx)

    postId, err := restate.Run(ctx, func(ctx restate.RunContext) (string, error) {
      return CreatePost(userId, post)
    })
    if err != nil {
      return err
    }

    for {
      status, err := restate.Run(ctx, func(ctx restate.RunContext) (string, error) {
        return GetPostStatus(postId), nil
      })
      if err != nil {
        return err
      }
      if status != PENDING {
        break
      }
      if err = restate.Sleep(ctx, 5*time.Second); err != nil {
        // </mark_2>
        return err
      }
    }

    if _, err := restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
      return UpdateUserFeed(userId, postId)
    }); err != nil {
      return err
    }

    return nil
  }
  ```
</CodeGroup>

**Key Benefits:**

* **Push-based delivery**: Events delivered directly to handlers with zero consumer management
* **Durable execution**: Failed handlers are retried with exponential backoff until they succeed. Handlers replay previously completed steps and resume exactly where they left off.
* **Parallel processing**: Events for different keys process concurrently, like a queue per object key
* **Timers and scheduling**: Timers and delays that survive crashes and restarts for **long-running workflows**

## Stateful Event Handlers

Maintain K/V state across events:

<CodeGroup>
  ```typescript TypeScript {"CODE_LOAD::ts/src/usecases/eventprocessing/delivery-tracker.ts#here"}  theme={null}
  export default restate.object({
    name: "delivery-tracker",
    handlers: {
      register: async (ctx: restate.ObjectContext, delivery: Delivery) =>
        ctx.set("delivery", delivery),

      setLocation: async (ctx: restate.ObjectContext, location: Location) => {
        const delivery = await ctx.get<Delivery>("delivery");
        if (!delivery) {
          throw new TerminalError(`Delivery not found`);
        }

        delivery.locations.push(location);
        ctx.set("delivery", delivery);
      },

      getDelivery: shared(async (ctx: restate.ObjectSharedContext) =>
        ctx.get<Delivery>("delivery")
      ),
    },
  });
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/usecases/eventprocessing/eventenrichment/DeliveryTracker.java#here"}  theme={null}
  @VirtualObject
  public class DeliveryTracker {
    private static final StateKey<Delivery> DELIVERY = StateKey.of("delivery", Delivery.class);

    @Handler
    public void register(ObjectContext ctx, Delivery packageInfo) {
      ctx.set(DELIVERY, packageInfo);
    }

    @Handler
    public void setLocation(ObjectContext ctx, Location location) {
      var delivery = ctx.get(DELIVERY).orElseThrow(() -> new TerminalException("Delivery not found"));

      delivery.addLocation(location);
      ctx.set(DELIVERY, delivery);
    }

    @Shared
    public Delivery getDelivery(SharedObjectContext ctx) {
      return ctx.get(DELIVERY).orElseThrow(() -> new TerminalException("Delivery not found"));
    }
  }
  ```

  ```python Python {"CODE_LOAD::python/src/usecases/eventprocessing/delivery_tracker.py#here"}  theme={null}
  delivery_tracker = restate.VirtualObject("DeliveryTracker")


  @delivery_tracker.handler()
  async def register(ctx: restate.ObjectContext, delivery: Delivery):
      ctx.set("delivery", delivery)


  @delivery_tracker.handler()
  async def set_location(ctx: restate.ObjectContext, location: Location):
      delivery = await ctx.get("delivery", type_hint=Delivery)
      if delivery is None:
          raise TerminalError(f"Delivery {ctx.key()} not found")

      delivery.locations.append(location)
      ctx.set("delivery", delivery)


  @delivery_tracker.handler(kind="shared")
  async def get_delivery(ctx: restate.ObjectSharedContext) -> Delivery:
      delivery = await ctx.get("delivery", type_hint=Delivery)
      if delivery is None:
          raise TerminalError(f"Delivery {ctx.key()} not found")
      return delivery
  ```

  ```go Go {"CODE_LOAD::go/usecases/eventprocessing/packagetracker.go#here"}  theme={null}
  type DeliveryTracker struct{}

  func (DeliveryTracker) Register(ctx restate.ObjectContext, delivery Delivery) error {
    restate.Set[Delivery](ctx, "delivery", delivery)
    return nil
  }

  func (DeliveryTracker) SetLocation(ctx restate.ObjectContext, location Location) error {
    packageInfo, err := restate.Get[*Delivery](ctx, "delivery")
    if err != nil {
      return err
    }
    if packageInfo == nil {
      return restate.TerminalError(errors.New("delivery not found"))
    }

    packageInfo.Locations = append(packageInfo.Locations, location)
    restate.Set[Delivery](ctx, "delivery", *packageInfo)
    return nil
  }

  func (DeliveryTracker) GetDelivery(ctx restate.ObjectSharedContext) (*Delivery, error) {
    return restate.Get[*Delivery](ctx, "delivery")
  }
  ```
</CodeGroup>

**Key Benefits**:

* **Persistent state**: Store and retrieve state directly in handlers without external stores
* **Built-in consistency**: State operations are always consistent with execution
* **Agents, actors, digital twins**: Model stateful entities that react to events

## When to Choose Restate

**✅ Choose Restate when you need:**

* **Kafka integration**: Process Kafka events with zero consumer management
* **Reliable processing**: Automatic retry and recovery for failed event handlers
* **Transactional processing**: Execute side effects with durable execution guarantees
* **Stateful event processing**: Maintain state across events without external stores
* **Event-driven workflows**: Build complex flows with loops, timers, and conditions

<Info>
  Processing events with Restate? Join our community on [Discord](https://discord.restate.dev) or [Slack](https://slack.restate.dev) to discuss your use case.
</Info>

## Comparison with Other Solutions

| Feature                 | Restate                                                                  | Traditional Kafka Processing                                        | Stream Processing Frameworks                                       |
| ----------------------- | ------------------------------------------------------------------------ | ------------------------------------------------------------------- | ------------------------------------------------------------------ |
| **Event Delivery**      | Push-based to durable handlers                                           | Polling-based consumer groups                                       | Built-in sources and sinks                                         |
| **Consumer Management** | Zero configuration                                                       | Manual offset and consumer group management                         | Zero configuration                                                 |
| **Failure Recovery**    | Fine-grained progress persistence with durable side effects              | Coarse-grained offset commits with potential reprocessing           | Coarse-grained checkpointing with potential duplicate side effects |
| **State Management**    | Built-in durable state                                                   | External state store required                                       | Built-in state store                                               |
| **Queuing Semantics**   | Queue-per-key with ordering guarantees                                   | Queue-per-partition with ordering guarantees                        | Queue-per-partition (inherited from Kafka)                         |
| **Complex Workflows**   | Unlimited control flow: loops, timers, conditions                        | Long-running logic blocks consumer loop; requires external services | DAG-based processing with limited control flow                     |
| **Best For**            | Event-driven state machines, transactional processing, complex workflows | Simple ETL and message processing                                   | High-throughput analytics, aggregations, joins                     |

## Getting Started

Ready to build event processing systems with Restate? Here are your next steps:

<CardGroup>
  <Card title="Quickstart" icon="rocket" href="/quickstart">
    Set up Restate and process your first events
  </Card>

  <Card title="Kafka Quickstart" icon="plug" href="/guides/kafka-quickstart">
    Follow the quickstart to implement your first durable event handler
  </Card>

  <Card title="Examples" icon="code" href="https://github.com/restatedev/examples">
    Explore templates, patterns, and end-to-end applications
  </Card>
</CardGroup>

<Info>
  Evaluating Restate and missing a feature? Contact us on [Discord](https://discord.restate.dev) or [Slack](https://slack.restate.dev).
</Info>


# Microservice Orchestration
Source: https://docs.restate.dev/use-cases/microservice-orchestration

Build resilient, distributed microservices with durable execution, sagas, and reliable service communication.

Restate provides **durable execution primitives** that make distributed systems resilient by default, without the operational overhead.

## Resilient Orchestration

Build microservices that automatically recover from failures without losing progress:

<CodeGroup>
  ```typescript TypeScript {"CODE_LOAD::ts/src/usecases/microservices/order-service.ts#here"}  theme={null}
  export const orderService = restate.service({
    name: "OrderService",
    handlers: {
      process: async (ctx: restate.Context, order: Order) => {
        // Each step is automatically durable and resumable
        const paymentId = ctx.rand.uuidv4();

        await ctx.run(() => chargePayment(order.creditCard, paymentId));

        for (const item of order.items) {
          await ctx.run(() => reserveInventory(item.id, item.quantity));
        }
        return { success: true, paymentId };
      },
    },
  });
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/usecases/microservices/OrderService.java#here"}  theme={null}
  @Service
  public class OrderService {

    @Handler
    public OrderResult process(Context ctx, Order order) {
      // Each step is automatically durable and resumable
      String paymentId = UUID.randomUUID().toString();

      ctx.run(() -> chargePayment(order.creditCard, paymentId));

      for (var item : order.items) {
        ctx.run(() -> reserveInventory(item.id, item.quantity));
      }

      return new OrderResult(true, paymentId);
    }
  }
  ```

  ```python Python {"CODE_LOAD::python/src/usecases/microservices/order_service.py#here"}  theme={null}
  order_service = restate.Service("OrderService")


  @order_service.handler()
  async def process(ctx: restate.Context, order: Order):
      # Each step is automatically durable and resumable
      payment_id = str(uuid.uuid4())

      await ctx.run_typed(
          "charge", charge_payment, credit_card=order.credit_card, payment_id=payment_id
      )

      for item in order.items:
          await ctx.run_typed(
              f"reserve_{item.id}", reserve, item_id=item.id, amount=item.amount
          )

      return {"success": True, "payment_id": payment_id}
  ```

  ```go Go {"CODE_LOAD::go/usecases/microservices/orderservice.go#here"}  theme={null}
  type OrderService struct{}

  func (OrderService) Process(ctx restate.Context, order Order) (OrderResult, error) {
    // Each step is automatically durable and resumable
    paymentID := restate.UUID(ctx).String()

    _, err := restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
      return ChargePayment(order.CreditCard, paymentID)
    })
    if err != nil {
      return OrderResult{}, err
    }

    for _, item := range order.Items {
      _, err := restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
        return ReserveInventory(item.ID, item.Quantity)
      })
      if err != nil {
        return OrderResult{}, err
      }
    }

    return OrderResult{Success: true, PaymentID: paymentID}, nil
  }
  ```
</CodeGroup>

* **Automatic recovery**: Code resumes exactly where it left off after failures
* **Standard development**: Write services like regular HTTP APIs
* **[Resilient Sagas](/guides/sagas)**: Implement complex multi-step transactions with resilient rollback

## Reliable Communication & Idempotency

Flexible communication patterns with strong delivery guarantees:

<img alt="Invocations" />

* **Zero message loss**: All service communication is durably logged
* **Built-in retries**: Automatic exponential backoff for transient failures
* **Scheduling**: Delay messages for future processing
* **Request deduplication**: Idempotency keys prevent duplicate processing

<Accordion title="Code example">
  <CodeGroup>
    ```typescript TypeScript {"CODE_LOAD::ts/src/usecases/microservices/service-actions.ts#communication"}  theme={null}
    // Request-response: Wait for result
    const payRef = await ctx.serviceClient(paymentService).charge(req);

    // Fire-and-forget: Guaranteed delivery without waiting
    ctx.serviceSendClient(emailService).emailTicket(req);

    // Delayed execution: Schedule for later
    ctx
      .serviceSendClient(emailService)
      .sendReminder(order, sendOpts({ delay: dayBefore(req.concertDate) }));
    ```

    ```java Java {"CODE_LOAD::java/src/main/java/usecases/microservices/ServiceActions.java#communication"}  theme={null}
    // Request-response: Wait for result
    var result = InventoryServiceClient.fromContext(ctx).checkStock(item);

    // Fire-and-forget: Guaranteed delivery without waiting
    EmailServiceClient.fromContext(ctx).send().emailTicket(order);

    // Delayed execution: Schedule for later
    EmailServiceClient.fromContext(ctx).send().sendReminder(order, Duration.ofDays(21));
    ```

    ```python Python {"CODE_LOAD::python/src/usecases/microservices/service_actions.py#communication"}  theme={null}
    # Request-response: Wait for result
    result = await ctx.service_call(charge_payment, req)

    # Fire-and-forget: Guaranteed delivery without waiting
    ctx.service_send(send_ticket_email, ticket)

    # Delayed execution: Schedule for later
    ctx.service_send(send_reminder, ticket, send_delay=timedelta(days=7))
    ```

    ```go Go {"CODE_LOAD::go/usecases/microservices/serviceactions.go#communication"}  theme={null}
    // Request-response: Wait for result
    result, err := restate.Service[StockResult](ctx, "PaymentService", "Charge").Request(req)
    if err != nil {
      return err
    }
    _ = result

    // Fire-and-forget: Guaranteed delivery without waiting
    restate.ServiceSend(ctx, "EmailService", "EmailTicket").Send(ticket)

    // Delayed execution: Schedule for later
    restate.ServiceSend(ctx, "EmailService", "SendReminder").Send(ticket, restate.WithDelay(7*24*time.Hour))
    ```
  </CodeGroup>
</Accordion>

## Durable Stateful Entities

Manage stateful entities without external databases or complex consistency mechanisms:

<img alt="Virtual Objects" />

* **Durable persistence**: Application state survives crashes and deployments
* **Simple concurrency model**: Single-writer semantics prevent consistency issues and race conditions
* **Horizontal scaling**: Each object has its own message queue. Different entity keys process independently
* **Built-in querying**: Access state via UI and APIs

<Accordion title="Code example">
  <CodeGroup>
    ```typescript TypeScript {"CODE_LOAD::ts/src/usecases/microservices/user-account.ts#here"}  theme={null}
    export default restate.object({
      name: "UserAccount",
      handlers: {
        updateBalance: async (ctx: restate.ObjectContext, amount: number) => {
          const balance = (await ctx.get<number>("balance")) ?? 0;
          const newBalance = balance + amount;

          if (newBalance < 0) {
            throw new TerminalError("Insufficient funds");
          }

          ctx.set("balance", newBalance);
          return newBalance;
        },

        getBalance: shared(async (ctx: restate.ObjectSharedContext) => {
          return (await ctx.get<number>("balance")) ?? 0;
        }),
      },
    });
    ```

    ```java Java {"CODE_LOAD::java/src/main/java/usecases/microservices/UserAccount.java#here"}  theme={null}
    @VirtualObject
    public class UserAccount {
      private static final StateKey<Double> BALANCE = StateKey.of("balance", Double.class);

      @Handler
      public double updateBalance(ObjectContext ctx, double amount) {
        double balance = ctx.get(BALANCE).orElse(0.0);
        double newBalance = balance + amount;

        if (newBalance < 0) {
          throw new TerminalException("Insufficient funds");
        }

        ctx.set(BALANCE, newBalance);
        return newBalance;
      }

      @Shared
      public double getBalance(SharedObjectContext ctx) {
        return ctx.get(BALANCE).orElse(0.0);
      }
    }
    ```

    ```python Python {"CODE_LOAD::python/src/usecases/microservices/user_account.py#here"}  theme={null}
    user_account = restate.VirtualObject("UserAccount")


    @user_account.handler()
    async def update_balance(ctx: restate.ObjectContext, amount: float):
        balance = await ctx.get("balance", type_hint=float) or 0.0
        new_balance = balance + amount

        if new_balance < 0.0:
            raise TerminalError("Insufficient funds")

        ctx.set("balance", new_balance)
        return new_balance


    @user_account.handler(kind="shared")
    async def get_balance(ctx: restate.ObjectSharedContext):
        return await ctx.get("balance", type_hint=float) or 0.0
    ```

    ```go Go {"CODE_LOAD::go/usecases/microservices/useraccount.go#here"}  theme={null}
    type UserAccount struct{}

    func (UserAccount) UpdateBalance(ctx restate.ObjectContext, amount float64) (float64, error) {
      balance, err := restate.Get[float64](ctx, "balance")
      if err != nil {
        return 0.0, err
      }

      newBalance := balance + amount
      if newBalance < 0.0 {
        return 0.0, restate.TerminalError(errors.New("insufficient funds"))
      }

      restate.Set(ctx, "balance", newBalance)
      return newBalance, nil
    }

    func (UserAccount) GetBalance(ctx restate.ObjectSharedContext) (float64, error) {
      return restate.Get[float64](ctx, "balance")
    }
    ```
  </CodeGroup>
</Accordion>

## Operational simplicity

Reduce infrastructure complexity (no need for queues + state stores + schedulers, etc.). A single binary including everything you need.

<img alt="Application Structure" />

## Key Orchestration Patterns

<CardGroup>
  <Card title="Sagas" icon="backward" href="/guides/sagas">
    Implement resilient rollback logic for non-transient failures
  </Card>

  <Card title="Database interaction" icon="database" href="/guides/databases">
    Use Durable Execution to make database operations resilient and consistent
  </Card>

  <Card title="Parallel Processing" icon="grip-lines" href="/guides/parallelizing-work">
    Execute independent operations concurrently while maintaining durability
  </Card>

  <Card title="Durable RPC, Idempotency & Concurrency" icon="phone" href="/tour/microservice-orchestration#resilient-communication">
    Call other services with guaranteed delivery, retries, and deduplication
  </Card>

  <Card title="Event-Driven Coordination" icon="messages" href="/tour/microservice-orchestration#external-events">
    Wait for external events and webhooks with promises that survive crashes
  </Card>

  <Card title="Durable State Machines" href="/tour/microservice-orchestration#virtual-objects" icon="diagram-project">
    Implement consistent state machines that survive crashes and restarts
  </Card>
</CardGroup>

## Comparison with Other Solutions

| Feature                    | Restate                         | Traditional Orchestration                         |
| -------------------------- | ------------------------------- | ------------------------------------------------- |
| **Infrastructure**         | Single binary deployment        | Message brokers + workflow engines + state stores |
| **Service Communication**  | Built-in reliable messaging     | External message queues required                  |
| **State Management**       | Integrated durable state        | External state stores + locks                     |
| **Failure Recovery**       | Automatic progress recovery     | Manual checkpoint/restart logic                   |
| **Deployment Model**       | Standard HTTP services          | Standard HTTP services                            |
| **Development Experience** | Regular code + IDE support      | Regular code + IDE support                        |
| **Observability**          | Built-in UI & execution tracing | Manual setup                                      |

## Getting Started

Ready to build resilient microservices with Restate? Here are your next steps:

<CardGroup>
  <Card title="Quickstart" icon="rocket" href="/quickstart">
    Run your first Restate service
  </Card>

  <Card title="Hands-on Tutorial" icon="play" href="/tour/microservice-orchestration">
    Learn orchestration patterns with interactive examples
  </Card>

  <Card title="Examples" icon="code" href="https://github.com/restatedev/examples">
    Explore templates, patterns, and end-to-end applications
  </Card>
</CardGroup>

<Info>
  Evaluating Restate and missing a feature? Contact us on [Discord](https://discord.restate.dev) or [Slack](https://slack.restate.dev).
</Info>


# Workflows
Source: https://docs.restate.dev/use-cases/workflows

Build resilient, low-latency workflows with code.

Restate lets you **write workflows as regular code** in your preferred programming language, with automatic resilience.

## Workflows as code

Write resilient workflows using familiar programming constructs:

* **Automatically retry transient errors** like infrastructure crashes and network failures
* Use **standard language constructs** (if/else, loops) and durable versions of familiar building blocks (e.g., timers, promises)
* **Handle errors naturally** with try/catch blocks and automatic retries
* **Test and debug** with your existing IDE and standard development tools

<CodeGroup>
  ```typescript TypeScript {"CODE_LOAD::ts/src/usecases/workflows/simple-signup.ts#here"}  theme={null}
  export const userSignup = restate.workflow({
    name: "user-signup",
    handlers: {
      run: async (ctx: WorkflowContext, user: User) => {
        const userId = ctx.key; // unique workflow key

        // Use regular if/else, loops, and functions
        const success = await ctx.run("create", () => createUser(userId, user));
        if (!success) return { success };

        // Execute durable steps
        await ctx.run("activate", () => activateUser(userId));
        await ctx.run("welcome", () => sendWelcomeEmail(user));

        return { success: true };
      },
    },
  });
  ```

  ```java Java {"CODE_LOAD::java/src/main/java/usecases/workflows/UserSignup.java#here"}  theme={null}
  @Workflow
  public class UserSignup {

    @Workflow
    public boolean run(WorkflowContext ctx, User user) {
      String userId = ctx.key(); // unique workflow key

      // Use regular if/else, loops, and functions
      boolean success = ctx.run("create", Boolean.class, () -> createUser(userId, user));
      if (!success) {
        return false;
      }

      // Execute durable steps
      ctx.run("activate", () -> activateUser(userId));
      ctx.run("welcome", () -> sendWelcomeEmail(user));

      return true;
    }
  }
  ```

  ```python Python {"CODE_LOAD::python/src/usecases/workflows/signup.py#here"}  theme={null}
  user_signup = restate.Workflow("user-signup")


  @user_signup.main()
  async def run(ctx: restate.WorkflowContext, user: User) -> Dict[str, bool]:
      # Unique workflow key
      user_id = ctx.key()

      # Use regular if/else, loops, and functions
      success = await ctx.run_typed("create", create_user, user_id=user_id, user=user)
      if not success:
          return {"success": False}

      # Execute durable steps
      await ctx.run_typed("activate", activate_user, user_id=user_id)
      await ctx.run_typed("welcome", send_welcome_email, user=user)

      return {"success": True}
  ```

  ```go Go {"CODE_LOAD::go/usecases/workflows/signup.go#here"}  theme={null}
  type UserSignup struct{}

  func (UserSignup) Run(ctx restate.WorkflowContext, user User) (bool, error) {
    // unique workflow key
    userID := restate.Key(ctx)

    // Use regular if/else, loops, and functions
    success, err := restate.Run(ctx, func(ctx restate.RunContext) (bool, error) {
      return CreateUser(userID, user)
    })
    if err != nil || !success {
      return false, err
    }

    // Execute durable steps
    _, err = restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
      return ActivateUser(userID)
    })
    if err != nil {
      return false, err
    }

    _, err = restate.Run(ctx, func(ctx restate.RunContext) (restate.Void, error) {
      return SendWelcomeEmail(user)
    })
    if err != nil {
      return false, err
    }

    return true, nil
  }
  ```
</CodeGroup>

## Low-Latency Workflows

Restate is built from the ground up for low-latency workflow execution. Restate workflows can be placed directly in the latency-sensitive path of user interactions:

* **Lightweight execution**: Workflows run like regular functions with minimal overhead
* **Event-driven foundation**: Built in Rust for high-performance, low-latency operations
* **No coordination delays**: Immediate workflow execution via a push-based model

<Info>
  [Benchmark results Restate v1.2](https://restate.dev/blog/building-a-modern-durable-execution-engine-from-first-principles/#some-performance-numbers)
</Info>

## Simple Deployment Model

<img alt="Application Structure" />

**Restate Server**: Restate is packaged as a single binary with built-in persistence and messaging. Run it as a single instance or in a high-availability cluster.

**Service Deployment**: Deploy your workflows using your existing deployment pipeline: containers, Kubernetes, serverless functions, or any HTTP-capable platform.

On FaaS, Restate suspends workflows while they are waiting (e.g. timer) to reduce costs.

## Key Workflow Patterns

<CardGroup>
  <Card title="Query Workflow State" icon="database" href={"/tour/workflows#querying-workflow-state"}>
    Store workflow state that survives crashes and can be queried from external systems
  </Card>

  <Card title="Event-Driven Coordination" icon="bolt" href={"/tour/workflows#signaling"}>
    Handle external events and signals without complex event sourcing infrastructure
  </Card>

  <Card title="Durable Timers and Scheduling" icon="clock" href={"/tour/workflows#timers-and-scheduling"}>
    Long-running processes with built-in timer management and timeout handling
  </Card>

  <Card title="Flexible Activity Execution" icon="code-branch" href={"/tour/workflows#in-line-steps-vs-separate-activities"}>
    Execute steps inline within the workflow or split them out into separate services
  </Card>

  <Card title="Parallelize Work" icon="arrows-split-up-and-left" href="/guides/parallelizing-work">
    Speed up multi-step workflows with recoverable parallel tasks
  </Card>

  <Card title="Resilient rollback" icon="backward" href="/guides/sagas">
    Automatically undo previous actions when later steps fail
  </Card>
</CardGroup>

## Comparison with Other Solutions

| Feature                | Restate                                          | Traditional Orchestrators           |
| ---------------------- | ------------------------------------------------ | ----------------------------------- |
| **Performance**        | Low-latency, lightweight execution               | High overhead, poll-for-work delays |
| **Language**           | Native code (TS, Python, Go, Java, Kotlin, Rust) | DSLs or YAML                        |
| **Development**        | Standard IDE, testing, debugging                 | Platform-specific tooling           |
| **Infrastructure**     | Single binary, no dependencies                   | Separate databases and queues       |
| **Service Deployment** | Any platform (containers, serverless, K8s)       | Worker-based deployment models      |
| **State Management**   | Built-in K/V state store                         | External state stores required      |

## Getting Started

Ready to build workflows with Restate? Here are your next steps:

<CardGroup>
  <Card title="Quickstart" icon="rocket" href="/quickstart">
    Run your first Restate service
  </Card>

  <Card title="Hands-on Tutorial" icon="pen" href="/tour/workflows">
    Explore the APIs to build workflows with Restate
  </Card>

  <Card title="Examples" icon="code" href="https://github.com/restatedev/examples">
    Explore templates, patterns, and end-to-end applications
  </Card>
</CardGroup>

<Info>
  Evaluating Restate and missing a feature? Contact us on [Discord](https://discord.restate.dev) or [Slack](https://slack.restate.dev).
</Info>