Skip to main content
Build fault-tolerant processing pipelines where each step transforms the previous step’s output. If any step fails, Restate automatically resumes from that exact point. For example: 
+----------+    +---------------+    +-------------+    +--------------+
|  Input   | => | Extract       | => | Sort        | => | Format as    |
| Message  |    | Metrics       |    | Results     |    | Table        |
+----------+    +---------------+    +-------------+    +--------------+

​
How does Restate help?

The benefits of using Restate here are:
  • Automatic retries of failed tasks: LLM API down, timeouts, infrastructure failures, etc.
  • Recovery of previous progress: After a failure, Restate recovers the progress the execution did before the crash.
  • Works with any LLM SDK (Vercel AI, LangChain, LiteLLM, etc.) and any programming language supported by Restate (TypeScript, Python, Go, etc.).

​
Example

Wrap each step in the chain with ctx.run() to ensure fault tolerance and automatic recovery. Restate uses durable execution to persist the result of each step as it completes, so if any step fails, Restate will retry from that exact point without losing previous progress or re-executing completed steps. 
async function process(ctx: Context, report: { message: string }) {
  // Step 1: Extract metrics
  const extract = await ctx.run(
    "Extract metrics",
    // Use your preferred LLM SDK here
    async () =>
      llmCall(`Extract numerical values and their metrics from the text. 
            Format as 'Metric Name: Value' per line. Input: ${report.message}`),
    { maxRetryAttempts: 3 },
  );

  // Step 2: Process the result from Step 1
  const sortedMetrics = await ctx.run(
    "Sort metrics",
    async () =>
      llmCall(`Sort lines in descending order by value: ${extract.text}`),
    { maxRetryAttempts: 3 },
  );

  // Step 3: Format as table
  const table = await ctx.run(
    "Format as table",
    async () =>
      llmCall(`Format the data as a markdown table: ${sortedMetrics.text}`),
    { maxRetryAttempts: 3 },
  );

  return table.text;
}
View on GitHub: TS / Python The Restate UI shows how each step in the chain is executed and persisted: Chaining LLM calls - UI
This pattern is implementable with any of our SDKs and any AI SDK. If you need help with a specific SDK, please reach out to us via Discord or Slack.
1

Requirements

  • AI SDK of your choice (e.g., OpenAI, LangChain, Pydantic AI, LiteLLM, etc.) to make LLM calls.
  • API key for your model provider.
2

Download the example

git clone https://github.com/restatedev/ai-examples.git &&
cd typescript-patterns &&
npm install
3

Start the Restate Server

restate-server
4

Start the Service

Export the API key of your model provider as an environment variable and then start the agent. For example, for OpenAI:
export OPENAI_API_KEY=your_openai_api_key
npm run dev
5

Register the services

  • UI
  • CLI
Service Registration
6

Send a request

In the UI (http://localhost:9070), click on the process handler of the CallChainingService to open the playground and send a default request:Chaining LLM calls - UI
7

Check the Restate UI

You see in the Invocations Tab of the UI how the LLM is called multiple times, and how each result is persisted in Restate:Chaining LLM calls - UI