- Build durable AI agents that recover automatically from crashes and API failures
- Integrate Restate with the
- Observe and debug agent executions with detailed traces
- Implement resilient human-in-the-loop workflows with approvals and timeouts
- Manage conversation history and state across multi-turn interactions
- Orchestrate multiple agents working together on complex tasks
Getting Started
A Restate AI application has two main components:- Restate Server: The core engine that takes care of the orchestration and resiliency of your agents
- Agent Services: Your agent or AI workflow logic using the Restate SDK for durability
Restate works with how you already deploy your agents, whether that’s in Docker, on Kubernetes, or via serverless platforms (Modal, AWS Lambda…). You don’t need to run your agents in any special way.
Let’s run an example locally to get a better feel for how it works.
Run the agent
Install Restate and launch it:http://localhost:9070) or CLI:
run handler of the WeatherAgent in the overview:

curl:
Durable Execution
AI agents make multiple LLM calls and tool executions that can fail due to rate limits, network issues, or service outages. Restate uses Durable Execution to make your agents withstand failures without losing progress. The Restate SDK records the steps the agent executes in a log and replays them if the process crashes or is restarted:
Durable Execution is the basis of how Restate makes your agents resilient to failures.
Restate offers durable execution primitives via its SDK.
Creating a Durable Agent
To implement a durable agent, you can use the Restate SDK in combination with the Google Agent Development Kit (ADK). Here’s the implementation of the durable weather agent you just invoked:durable_agent.py
- Restate Plugin: Add the
RestatePluginto your Google ADK App. This enables Restate’s durability features for model calls and tool executions within the agent. - Resilient Tools: Wrap tool logic in durable steps using the Restate Context. In the
get_weathertool, the call to the weather API is wrapped inrestate_context().run_typed, making it a durable step that Restate can retry and recover.
run handler.
The endpoint that serves the agents of this tour over HTTP is defined in __main__.py.
The agent can now be called at http://localhost:8080/WeatherAgent/run.
Try out Durable Execution
Try out Durable Execution
Ask for the weather in Denver:On the invocation page in the UI, click on the invocation ID of the failing invocation.
You can see that your request is retrying because the weather API is down:
To fix the problem, remove the line Once you restart the service, the workflow finishes successfully.

fail_on_denver from the fetch_weather function in the app/utils/utils.py file:utils/utils.py
Observing your Agent
As you saw in the previous section, the Restate UI comes in handy when monitoring and debugging your agents. The Invocations tab shows all agent executions with detailed traces of every LLM call, tool execution, and state change:
OpenTelemetry Integration
OpenTelemetry Integration
Restate supports OpenTelemetry for exporting traces to external systems like Langfuse, DataDog, or Jaeger:Have a look at the tracing docs to set this up.
Human-in-the-Loop Agent
Many AI agents need human oversight for high-risk decisions or gathering additional input. Restate makes it easy to pause agent execution and wait for human input. Benefits with Restate:- If the agent crashes while waiting for human input, Restate continues waiting and recovers the promise on another process.
- If the agent runs on function-as-a-service platforms, the Restate SDK lets the function suspend while it’s waiting. Once the approval comes in, the Restate Server invokes the function again and lets it resume where it left off. This way, you don’t pay for idle waiting time (Learn more).
human_approval_agent.py
You can also use awakeables outside of tools, for example, to implement human approval steps in between agent iterations.
Try out human approval
Try out human approval
Start a request for a high-value claim that needs human approval.
Use the playground or You can restart the service to see how Restate continues waiting for the approval.If you wait for more than a minute, the invocation will get suspended.
Simulate approving the claim by executing the curl request that was printed in the service logs, similar to:See in the UI how the workflow resumes and finishes after the approval.
curl with /send to start the claim asynchronously, without waiting for the result.

Timeouts and Escalation
Timeouts and Escalation
Add timeouts to human approval steps to prevent workflows from hanging indefinitely.Restate persists the timer and the approval promise, so if the service crashes or is restarted, it will continue waiting with the correct remaining time:Try it out by sending a request to the service:You restart the service and check in the UI how the process will block for the remaining time without starting over.You can also lower the timeout to a few seconds to see how the timeout path is taken.
human_approval_agent_with_timeout.py
Resilient workflows as tools
You can pull out complex parts of your tool logic into separate workflows. This lets you break down complex agents into smaller, reusable components that can be developed, deployed, and scaled independently. The Restate SDK gives you clients to call these workflows durably from your agent logic. All calls are proxied via Restate. Restate persists the call and takes care of retries and recovery. For example, let’s implement the human approval tool as a separate service:sub_workflow_agent.py
sub_workflow_agent.py
Try out sub-workflows
Try out sub-workflows
Start a request for a high-value claim that needs human approval.
Use In the UI, you can see that the agent called the workflow service and is waiting for the response.
You can see the trace of the sub-workflow in the timeline.Once you approve the claim, the workflow returns, and the agent continues.
/send to start the claim asynchronously, without waiting for the result.
Follow the Tour of Workflows to learn more about implementing resilient workflows with Restate.
Durable Sessions
The next ingredient we need to build AI agents is the ability to maintain context and memory across multiple interactions. To implement stateful entities like chat sessions, or stateful agents, Restate provides a special service type called Virtual Objects. When you send a message to a Virtual Object, you provide a unique key that identifies the object instance (for example, a chat session ID or user ID). Each instance of a Virtual Object maintains isolated state. The handlers of the Virtual Object can read and modify the object’s state via the RestateObjectContext.

Virtual Objects for stateful agents
With Google ADK and Restate, you can create stateful agents that maintain conversation history across multiple interactions. Here is an example of a chat agent represented as a Virtual Object that is keyed by user ID:chat.py
Try out Virtual Objects
Try out Virtual Objects
Ask the agent to do some task and provide a user ID as the object key:Continue the conversation with the same user and session ID. The agent will remember previous context:Go to the state tab of the UI to view the conversation history.
- Long-lived state: K/V state is stored permanently. It has no automatic expiry. Clear it via
ctx.clear(). - Durable state changes: State changes are logged with Durable Execution, so they survive failures and are consistent with code execution
- State is queryable via the state tab in the UI.

Built-in concurrency control
Restate’s Virtual Objects have built-in queuing and consistency guarantees per object key. You provide the unique key when invoking the Virtual Object, for example, the user ID. When multiple requests come in for the same object key, Restate automatically queues them and ensures consistency.
- Handlers either have read-write access (
ObjectContext) or read-only access (shared object context). - Only one handler with write access can run at a time per object key to prevent concurrent/lost writes or race conditions (for example
message()). - Handlers with read-only access can run concurrently to the write-access handlers (for example
get_history()).
Try out queuing
Try out queuing
Let’s send several messages concurrently to different users:The UI shows how Restate queues the requests per session to ensure consistency:

Stateful Serverless Agents
Stateful Serverless Agents
You can run Virtual Objects on serverless platforms like Modal, Render, or AWS Lambda.
When the request comes in, Restate attaches the correct state to the request, so your handler can access it locally.This way, you can implement stateful, serverless agents without managing any external state store and without worrying about concurrency issues.
Virtual Objects for storing context
You can store any context information in Virtual Objects, for example, user preferences or the last agent they interacted with. Usectx.set and ctx.get in your handler to store and retrieve state.
Have a look here for more information.
Resilient multi-agent coordination
As your agents grow more complex, you may want to break them down into smaller, specialized agents that can delegate tasks to each other. Similar to sub-workflows, you can break down complex agents into multiple specialized agents. All agents can run in the same process or be deployed independently.Agents as tools/handoffs
If you want to share context between agents, run the agents in the same process and use handoffs or tools. You don’t need to do anything special to make this work with Restate:multi_agent.py
Try out multi-agent systems
Try out multi-agent systems
Start a request for a claim that needs to be analyzed by multiple agents.In the UI, you can see that the agent called the sub-agents (multiple LLM calls) and is waiting for their responses.
You can see the trace of the sub-agents in the timeline.Once all sub-agents return, the main agent continues and makes a decision.

Remote agents as tools
If you want to run agents independently, for example, to scale them separately, run them on different platforms, or let them get developed by different teams, then you can call them as tools via service calls. Restate will proxy all calls, persist them, and will guarantee that they complete successfully. Your main agent can suspend and save resources while waiting for the remote agent to finish. Restate invokes your main agent again once the remote agent returns.multi_agent_remote.py
Try out calling a remote agent
Try out calling a remote agent
Start a request for a claim that needs to be analyzed by multiple agents.In the UI, you can see that the agent called the sub-agents and is waiting for their responses.
You can see the trace of the sub-agents in the timeline.Once all sub-agents return, the main agent continues and makes a decision.

Parallel Work
Now that our agents are broken down into smaller parts, let’s have a look at how to run different parts of our agent logic in parallel to speed up execution. Restate provides primitives that allow you to run tasks concurrently while maintaining deterministic execution during replays. Most actions on the Restate Context can be composed usingrestate.gather to gather their results or restate.select to wait for the first one to complete.
Parallel Tool Steps
When using the Google ADK with Restate, tool calls are forced to be executed sequentially to ensure deterministic execution during replays. When multiple tools execute in parallel and use the Restate Context, the order of operations might differ between the original execution and the replay, leading to inconsistencies. Therefore, the only way to run multiple tool steps in parallel is to implement an orchestrator tool that uses durable execution to run multiple steps in parallel. Here is an insurance claim agent tool that runs multiple analyses in parallel:parallel_tools.py
Try out parallel tool steps
Try out parallel tool steps
Start a request for a claim that needs to be analyzed by multiple tools in parallel.In the UI, you can see that the agent ran the tool steps in parallel.
Their traces all start at the same time.Once all tools return, the agent continues and makes a decision.

Parallel Agents
You can use the same durable execution primitives to run multiple agents in parallel. For example, to race agents against each other and use the first result that returns, while cancelling the others. Or to let a main orchestrator agent combine the results of multiple specialized agents in parallel:parallel_agents.py
Try out parallel agents
Try out parallel agents
Start a request for a claim that needs to be analyzed by multiple agents in parallel.In the UI, you can see that the handler called the sub-agents in parallel.
Once all sub-agents return, the main agent makes a decision.

Error Handling
LLM calls are costly, so you can configure retry behavior in both Restate and your AI SDK to avoid infinite loops and high costs. Restate distinguishes between two types of errors:- Transient errors: Temporary issues like network failures or rate limits. Restate automatically retries these until they succeed or the retry policy is exhausted.
- Terminal errors: Permanent failures like invalid input or business rule violations. Restate does not retry these. The invocation fails permanently. You can catch these errors and handle them gracefully.
Retries of LLM calls
Configure the number of retries for LLM calls when activating the Restate plugin for your ADK App:error_handling.py
TerminalError and won’t be retried further.
Tool execution errors
Restate retries tool executions by default until they succeed. For errors which should not be retried, you can raise terminal errors from within your tool implementations. You can catch these terminal errors in your handler and handle them accordingly.error_handling.py
You can set custom retry policies for
.run actions in your tool executions.Advanced patterns
Control the Agent Loop
Control the Agent Loop
If you need more control over the agent loop, you can implement it manually using Restate’s durable primitives.This allows you to:
- Parallelize tool calls with
restate.selectandrestate.gather - Implement custom stopping conditions
- Implement custom logic between steps (e.g. human approval)
- Interact with external systems between steps
- Handle errors in a custom way
Rolling back tool executions on failure
Rolling back tool executions on failure
Sometimes you need to undo previous agent actions when a later step fails. Restate makes it easy to implement compensation patterns (Sagas) for AI agents.Just track the rollback actions as you go, let the agent raise terminal tool errors, and execute the rollback actions in reverse order.Have a look at the saga guide to learn more.
Long-running background agents
Long-running background agents
Restate supports implementing scheduling and timer logic in your agents.
This allows you to build agents that run periodically, wait for specific times, or implement complex scheduling logic.
Agents can either be long-running or reschedule themselves for later execution.Have a look at the scheduling docs to learn more.
Streaming back intermediate results
Streaming back intermediate results
Have a look at the pub-sub example.
Interrupting agents
Interrupting agents
Have a look at the interruptible coding agent.
Summary
Durable Execution, paired with your existing SDKs, gives your agents a powerful upgrade:- Durable Execution: Automatic recovery from failures without losing progress
- Persistent memory and context: Persistent conversation history and context
- Observability by default across your agents and workflows
- Human-in-the-Loop: Seamless approval workflows with timeouts
- Multi-Agent Coordination: Reliable orchestration of specialized agents
- Suspensions to save costs on function-as-a-service platforms when agents need to wait
- Advanced Patterns: Real-time progress updates, interruptions, and long-running workflows