Skip to main content

Tour of Restate

This tutorial guides you through the development of an end-to-end Restate application, and covers all the essential features. After this tutorial, you should have a firm understanding of how Restate can help you and feel comfortable to tackle your next application on your own.

This tutorial implements a ticket reservation application for a theatre. It allows users to add tickets for specific seats to their shopping cart. After the user adds a ticket, the seat gets reserved for 15 minutes. If the user doesn't pay for the ticket within that time interval, the reservation is released and the ticket becomes available to other users.

Restate applications are made up of services that expose methods to each other and the outside world. Services communicate with one another using Remote Procedure Calls (RPC). Our ticket example consists of three services having the following responsibilities:

  1. User session: keeps track of the tickets in the shopping cart.
  2. Ticket service: manages the ticket allocation.
  3. Checkout: handles the checkout process of the ticket sale.

Tour of Restate diagram

As we go, you will discover how Restate can help you with some intricacies in this application.

Prerequisites

  • Latest stable version of NodeJS >= v18.17.1 and npm CLI >= 9.6.7 installed.
  • Docker Engine or Podman to launch the Restate runtime (not needed for the app implementation itself).
  • curl

This guide is written for:

  • TypeScript SDK version: @restatedev/restate-sdk:0.7.0
  • Restate runtime Docker image: docker.io/restatedev/restate:0.7

Getting started

Setting up the tutorial

Clone the GitHub repository of the tutorial:

git clone --depth 1 --branch v0.7.0 [email protected]:restatedev/tour-of-restate.git && cd tour-of-restate/typescript

This GitHub repository contains the basic skeleton of the NodeJS/TypeScript services that you develop in this tutorial.

First, get all dependencies and build tools. Then build the app:

npm install && npm run build

Starting the components

Let's start a basic version of the Restate services that we will be using for this tour. This is where our application logic will reside.

npm run app

You can run the solution of a specific part by running, for example for part 2:

npm run part2

Next, let's launch Restate itself. Restate can be deployed as a simple standalone binary, all the way up to a fully managed service. In this tour we'll be running it standalone from a container image.

docker run --name restate_dev --rm -d --network=host docker.io/restatedev/restate:0.7

When you are finished with the tour, you can stop Restate with docker stop restate_dev. This also deletes any persisted state because we start the container with the --rm flag. You can restart Restate at a later time (without losing state) with docker restart restate_dev. You can view Restate logs using docker logs restate_dev or follow them live by adding --follow.

Registering services with Restate

When Restate first comes up, it doesn't know anything about our services. Let's tell it where they are running. You can trigger service discovery by calling the Restate admin API informing it about the service deployment where our service is listening.

curl localhost:9070/deployments -H 'content-type: application/json' -d '{"uri": "http://localhost:9080"}'

Restate will send a discovery request to the service deployment to find the services and methods run behind this endpoint. Restate keeps track of which services run where (including of the method signatures). Restate also keeps track of multiple revisions as your APIs evolve, and routes requests appropriately.

tip

You can pipe the output of curl into the jq utility to reformat the JSON output and make it easier to read.

You should now see the registered services printed out to your terminal:

{
"id": "bG9jYWxob3N0OjgwODAv",
"services": [
{
"name": "TicketService",
"revision": 1
/* ... Additional information on registered methods ...*/
},
{
"name": "UserSession",
"revision": 1
/* ... Additional information on registered methods ...*/
},
{
"name": "Checkout",
"revision": 1
/* ... Additional information on registered methods ...*/
}
]
}

The Restate logs will show the new service registrations taking place.

Show the runtime logs
2023-08-15T13:24:52.879977Z INFO restate_schema_impl::schemas_impl
Registering deployment
restate.deployment.id: bG9jYWxob3N0OjgwODAv
restate.deployment.url: http://localhost:9080/
2023-08-15T13:24:52.880010Z INFO restate_schema_impl::schemas_impl
Registering service
rpc.service: "UserSession"
restate.deployment.url: http://localhost:9080/
2023-08-15T13:24:52.881233Z INFO restate_schema_impl::schemas_impl
Registering service
rpc.service: "TicketService"
restate.deployment.url: http://localhost:9080/
2023-08-15T13:24:52.881422Z INFO restate_schema_impl::schemas_impl
Registering service
rpc.service: "Checkout"
restate.deployment.url: http://localhost:9080/

For MacOS users, the address will be http://host.docker.internal:9080/.

Calling services from outside Restate

Add a ticket to a cart by calling UserSession/addTicket.

curl localhost:8080/UserSession/addTicket \
-H 'content-type: application/json' \
-d '{"key": "123", "request": "456"}'

If this prints out true, then you have a working setup.

You can call the UserSession/checkout function to proceed with the purchase, as follows:

curl localhost:8080/UserSession/checkout \
-H 'content-type: application/json' \
-d '{"key": "123"}'

In src/app you will find the skeletons of the various services to help you start implementing the app. The app.ts file contains the definition of the server that hosts the set of services.

tip

Only public Restate services are callable from outside Restate using the ingress endpoint. You can mark services as private which makes them available only to other Restate services.

Services and concurrency

Restate supports three distinct kinds of services:

  1. Keyed service: All service invocations are sharded on a user-defined key. Function invocations of keyed services are serialized on the user-defined key. Therefore, at most one function can run at a time for a given key within such service.
  2. Unkeyed service: No key defined. No concurrency guarantees or limitations. Invocations are processed as they come in. You would get the same behavior with a keyed service with random keys.
  3. Singleton service (not available for TypeScript Handler API): No key defined. There is at most one concurrent invocation for this entire service. You can see this as a keyed service where all invocations have an identical key. This service type does not scale up, so don't use it for heavy load.

In this example app, the Checkout service is an unkeyed service, while the TicketService and UserSession services are keyed services. The TicketService is keyed by the ticket ID, while the UserSession service is keyed by the user ID. So at most one invocation can run at a time for the same ticket ID or user ID respectively.

Specifying the service type

Specifying unkeyed services

You specify an unkeyed service by creating a router with its set of functions.

const serviceRouter = restate.router({
hello: async (ctx: restate.RpcContext, request: Request) => { ... },
callMe: async (ctx: restate.RpcContext) => { ... },
maybe: async () => { ... }
});

The functions of an unkeyed router have two optional parameters:

  • A ctx of type restate.RpcContext which allows to interact with Restate.
  • A request of type JSON object which represents additional invocation data.

You can choose the parameter names differently.

For example, for the Checkout service, you create a router with the checkout function:

const checkout = async (
ctx: restate.RpcContext,
request: { userId: string; tickets: string[] },
) => {
return true;
};

export const checkoutRouter = restate.router({
checkout,
// ... optional other functions ...
});

The Checkout service only has a single function, but you can add more functions to the router if you want.

Specifying keyed services

You specify a keyed service by creating a keyed router with its set of functions:

const keyedServiceRouter = restate.keyedRouter({
hello: async (ctx: restate.RpcContext, key: string, request: Request) => { ... },
callMe: async (ctx: restate.RpcContext, key: string) => { ... },
maybe: async (ctx: restate.RpcContext) => { ... },
withSomething: async () => { ... }
});

The functions of a keyed router have three optional parameters

  • A ctx of type restate.RpcContext which allows to interact with Restate.
  • A key parameter of type string which represents the key of the invoked service.
  • A request parameter of type JSON object which represents additional invocation data.

You can choose the parameter names differently.

For example, the UserSession service has the following keyed router:

export const userSessionRouter = restate.keyedRouter({ addTicket, expireTicket, checkout });

As mentioned, for keyed services there can be at most one concurrent invocation to any method of the service. This means that for the same value of key, an unfinished call to addTicket() will block subsequent calls to expireTicket() or checkout() until addTicket() has finished. This holds true even if you omit the key parameter from the function signature, as we have done with the maybe() and withSomething() functions above.

Exporting the service API

You export the service API by creating a service API instance which specifies under which path the service is reachable.

For example, for the UserSession service, you export the following service API:

export const userSessionApi: restate.ServiceApi<typeof userSessionRouter> = {
path: "UserSession",
};

You can now call the service at the endpoint http://localhost:8080/<servicePath>/<functionName>. For example, you call the addTicket function of the UserSession service at http://localhost:8080/UserSession/addTicket, as you did earlier.

tip

The concurrency guarantees of keyed services make it a lot easier to reason about interaction with external systems.

Imagine you have a keyed service which is the single writer to a database and every key only interacts with an isolated set of database rows. Then you can scale your application and never have concurrent invocations writing to the same database row. This resolves common data consistency issues such as lost updates or non-repeatable reads.

Have a look at the product service of the shopping cart example. The service is keyed on product ID, just like the database table it writes to. Restate ensures that there are never concurrent writes to the same product.

caution

Take into account the concurrency limitations when designing your applications!

  • Time-consuming operations in a keyed service, for example sleeps, lock that key/service for the entire operation. Other invocations for that key/service are enqueued, until the invocation has completed.
  • Deadlocks: Watch out for cyclical request-response calls in your application. For example, if A calls B, and B calls C, and C calls A again. Each service waits on the response of the other and none of them can progress. The keys remain locked and the system can't process any more requests.

Service calls

note

Implement it yourself or follow along by looking at the code under part1.

One of the key parts of distributed applications is service-to-service communication. Service method calls are implemented as RPC messages. Restate makes service-to-service communication reliable by ensuring that messages do not get lost.

Calling services within Restate

The tour begins with an introduction to request-response calls, in which one service calls another service and waits for the response.

When the UserSession/addTicket function is called, it first needs to reserve the ticket for the user. It does that by calling the TicketService/reserve function. Add the highlighted code snippet to the addTicket function in the UserSession service.

import * as restate from "@restatedev/restate-sdk";
import { ticketServiceApi } from "./ticket_service";

const addTicket = async (
ctx: restate.RpcContext,
userId: string,
ticketId: string,
) => {
const success = await ctx.rpc(ticketServiceApi).reserve(ticketId);
return success;
};

To make the call, you supply the exported ticketServiceApi to the RpcContext via the rpc function to specify which service to invoke. Then you call the reserve function, which returns a Promise that gets resolved with the boolean response.

Try it out by running the services via npm run app and send a request to UserSession/addTicket, as we did previously.

Have a look at the SDK and runtime logs, to see how the ingress request triggers the execution of the addTicket and reserve functions.

Show the logs
[restate] [2023-08-15T20:53:51.026Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHDd1Z6crtc63up0wAD] : Invoking function.
[restate] [2023-08-15T20:53:51.026Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHDd1Z6crtc63up0wAD] : Adding message to journal and sending to Restate ; InvokeEntryMessage
[restate] [2023-08-15T20:53:51.027Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHDd1Z6crtc63up0wAD] : Scheduling suspension in 30000 ms
[restate] [2023-08-15T20:53:51.029Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHEvH1866jAt-dCQtFY] : Invoking function.
[restate] [2023-08-15T20:53:51.029Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHEvH1866jAt-dCQtFY] : Journaled and sent output message ; OutputStreamEntryMessage
[restate] [2023-08-15T20:53:51.029Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHEvH1866jAt-dCQtFY] : Function completed successfully.
[restate] [2023-08-15T20:53:51.031Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHDd1Z6crtc63up0wAD] : Received completion message from Restate, adding to journal. ; CompletionMessage
[restate] [2023-08-15T20:53:51.031Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHDd1Z6crtc63up0wAD] : Journaled and sent output message ; OutputStreamEntryMessage
[restate] [2023-08-15T20:53:51.031Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHDd1Z6crtc63up0wAD] : Function completed successfully.

To better understand what's going on, you can increase the log level by setting the environment variable RESTATE_DEBUG_LOGGING=JOURNAL. You can silence the logging by removing this environment variable in package.json, where the scripts are defined. To also log the messages, use RESTATE_DEBUG_LOGGING=JOURNAL_VERBOSE.

Reliable delivery and suspension

The call you just added seems like a regular RPC call that you might be familiar with from other service frameworks. This similarity is superficial; under the hood the Restate runtime and SDK cooperate to deliver resiliency and efficiency.

When an incoming request is sent to Restate, the runtime first persists it. It establishes a connection to the user session service, and sends the request over to begin a new invocation. All communications between the service and other Restate components goes over this connection. In particular, the user session service does not communicate directly with the ticket service.

When the user session service makes an RPC to the ticket service, this request goes over the open connection to the runtime. Once again, this is durably persisted in the Restate journal, before it is passed on (by Restate) to the ticket service. In case of a failure, Restate takes care of retrying according to the configured policy. We will cover Resiliency in more detail later on. When the ticket service responds, its response is then sent back to the runtime, which persists it and forwards it to the user session service to continue execution.

When services take a while to respond, Restate suspends the calling execution to free up the resources for other invocations. When the calling service is unblocked to resume, Restate invokes it again and sends over a replay log. This replay log contains all the state that the previously completed steps generated prior to suspension. The service replays the log, and the service resumes execution from the point where it left off prior to suspending.

tip

The suspension mechanism is especially beneficial if you deploy your services to a FaaS platform such as AWS Lambda. This way you don't pay for the idle time a caller spends waiting for a response when making synchronous service calls within Restate!

Waiting for slow services

To see this work in practice, let's pretend that ticket reservations take a while by making the TicketService/reserve function sleep for a while.

caution

This is not the best way to sleep in a Restate service! The Restate SDK offers a native way to suspend execution for a period of time. We will cover suspendable sleep a bit later on in the tour.

import { setTimeout } from "timers/promises";

const reserve = async (ctx: restate.RpcContext) => {
await setTimeout(35000);
return true;
};

Call UserSession/addTicket again and have a look at the SDK logs. Now that the ticket service responds after 35 seconds, you see that the addTicket function suspends after 30 seconds. Once the runtime has received the response, it invokes the addTicket function again and the original call finishes.

Show the logs
[restate] [2023-08-15T20:58:25.433Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHF0pFyZoYvJ5G_WZMF] : Invoking function.
[restate] [2023-08-15T20:58:25.434Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHF0pFyZoYvJ5G_WZMF] : Adding message to journal and sending to Restate ; InvokeEntryMessage
[restate] [2023-08-15T20:58:25.434Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHF0pFyZoYvJ5G_WZMF] : Scheduling suspension in 30000 ms
[restate] [2023-08-15T20:58:25.436Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHF0pNxZqnhNkoJPdn7] : Invoking function.
[restate] [2023-08-15T20:58:55.436Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHF0pFyZoYvJ5G_WZMF] : Writing suspension message to journal. ; SuspensionMessage
[restate] [2023-08-15T20:58:55.438Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHF0pFyZoYvJ5G_WZMF] : Suspending function.
[restate] [2023-08-15T20:59:00.440Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHF0pNxZqnhNkoJPdn7] : Journaled and sent output message ; OutputStreamEntryMessage
[restate] [2023-08-15T20:59:00.441Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHF0pNxZqnhNkoJPdn7] : Function completed successfully.
[restate] [2023-08-15T20:59:00.458Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHF0pFyZoYvJ5G_WZMF] : Resuming (replaying) function.
[restate] [2023-08-15T20:59:00.459Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHF0pFyZoYvJ5G_WZMF] : Matched and replayed message from journal ; InvokeEntryMessage
[restate] [2023-08-15T20:59:00.460Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHF0pFyZoYvJ5G_WZMF] : Journaled and sent output message ; OutputStreamEntryMessage
[restate] [2023-08-15T20:59:00.460Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHF0pFyZoYvJ5G_WZMF] : Function completed successfully.
Terminating invocations

By default Restate will keep retrying failed invocations until they succeed. If at any point during testing you create an invocation that you want to permanently terminate, you can kill it using the Admin API and the invocation id from the logs:

curl -X DELETE http://localhost:9070/invocations/<invocation_id>

Each invocation gets an ID assigned by Restate. You can find the ID printed in the service logs.

📝 Try it out

Make the UserSession/checkout function call the Checkout/checkout function.

For the request field, you can use a hard-coded string array for now: ["456"]. You will fix this later on.

Solution

Add the following code to the UserSession/checkout function:

import { checkoutApi } from "./checkout";

const checkout = async (ctx: restate.RpcContext, userId: string) => {
const checkoutRequest = { userId: userId, tickets: ["456"] };
const success = await ctx.rpc(checkoutApi).checkout(checkoutRequest);

return sucess;
};

Call the UserSession/checkout function as you did earlier and have a look at the logs again to see what happened:

[restate] [2023-08-15T21:06:12.672Z] DEBUG: [UserSession/checkout] [K1qex52CPYkBiiHHQJdwvLuBqEHXFQU3] : Invoking function.
[restate] [2023-08-15T21:06:12.673Z] DEBUG: [UserSession/checkout] [K1qex52CPYkBiiHHQJdwvLuBqEHXFQU3] : Adding message to journal and sending to Restate ; InvokeEntryMessage
[restate] [2023-08-15T21:06:12.674Z] DEBUG: [UserSession/checkout] [K1qex52CPYkBiiHHQJdwvLuBqEHXFQU3] : Scheduling suspension in 30000 ms
[restate] [2023-08-15T21:06:12.676Z] DEBUG: [Checkout/checkout] [H32MEMFENfUBiiHHQJxy5oLQ-W4akbii] : Invoking function.
[restate] [2023-08-15T21:06:12.677Z] DEBUG: [Checkout/checkout] [H32MEMFENfUBiiHHQJxy5oLQ-W4akbii] : Journaled and sent output message ; OutputStreamEntryMessage
[restate] [2023-08-15T21:06:12.677Z] DEBUG: [Checkout/checkout] [H32MEMFENfUBiiHHQJxy5oLQ-W4akbii] : Function completed successfully.
[restate] [2023-08-15T21:06:12.679Z] DEBUG: [UserSession/checkout] [K1qex52CPYkBiiHHQJdwvLuBqEHXFQU3] : Received completion message from Restate, adding to journal. ; CompletionMessage
[restate] [2023-08-15T21:06:12.679Z] DEBUG: [UserSession/checkout] [K1qex52CPYkBiiHHQJdwvLuBqEHXFQU3] : Journaled and sent output message ; OutputStreamEntryMessage
[restate] [2023-08-15T21:06:12.679Z] DEBUG: [UserSession/checkout] [K1qex52CPYkBiiHHQJdwvLuBqEHXFQU3] : Function completed successfully.

Notice that the UserSession didn't get suspended because the response came before the suspension timeout hit.

Reliable message sending without queues

Up to now, all our service calls waited for a response. This is referred to as request-response or synchronous RPC. With Restate, you can also make one-way calls where you don't wait for the response.

Update expireTicket to use RpcContext.send to send a one-way message to release the reservation:

const addTicket = async (
ctx: restate.RpcContext,
userId: string,
ticketId: string,
) => {
ctx.send(ticketServiceApi).reserve(ticketId);
return true;
};

Note that RpcContext.send isn't an asynchronous operation which returns a promise. Therefore, there is no need to await it.

Once you have adapted the code, try it out by calling the UserSession/addTicket function, as explained earlier. In the service logs, you see that the addTicket function doesn't wait for a response and finishes earlier than the reserve function.

Show the logs
[restate] [2023-08-15T21:09:59.681Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHJmC53vpTRfQdpRgKE] : Invoking function.
[restate] [2023-08-15T21:09:59.682Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHJmC53vpTRfQdpRgKE] : Adding message to journal and sending to Restate ; BackgroundInvokeEntryMessage
[restate] [2023-08-15T21:09:59.683Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHJmC53vpTRfQdpRgKE] : Journaled and sent output message ; OutputStreamEntryMessage
[restate] [2023-08-15T21:09:59.683Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHJmC53vpTRfQdpRgKE] : Function completed successfully.
[restate] [2023-08-15T21:09:59.686Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHJmDBwp6kJ82nETxIy]: Invoking function.
[restate] [2023-08-15T21:10:34.688Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHJmDBwp6kJ82nETxIy]: Journaled and sent output message ; OutputStreamEntryMessage
[restate] [2023-08-15T21:10:34.689Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHJmDBwp6kJ82nETxIy]: Function completed successfully.
Good news!

Just as with synchronous calls, Restate persists and retries failed one-way invocations. There is no need to set up message queues to ensure delivery!

📝 Try it out

In this example, when you add a seat to your shopping cart, it gets reserved for 15 minutes. When a user didn't proceed with the payment before the timeout, you need to call the UserSession/expireTicket function. Let the expireTicket function call the TicketService/unreserve function.

When you are done with the implementation, send a request to UserSession/expireTicket to check if it works.

Solution
const expireTicket = async (
ctx: restate.RpcContext,
userId: string,
ticketId: string,
) => {
ctx.send(ticketServiceApi).unreserve(ticketId);
};

Call the expireTicket function with:

curl localhost:8080/UserSession/expireTicket \
-H 'content-type: application/json' \
-d '{"key": "123", "request": "456"}'

Have a look at the logs again to see what happened:

[restate] [2023-08-15T21:13:54.707Z] DEBUG: [UserSession/expireTicket] [K1qex52CPYkBiiHLgq92RZrEuQ502v1K] : Invoking function.
[restate] [2023-08-15T21:13:54.708Z] DEBUG: [UserSession/expireTicket] [K1qex52CPYkBiiHLgq92RZrEuQ502v1K] : Adding message to journal and sending to Restate ; BackgroundInvokeEntryMessage
[restate] [2023-08-15T21:13:54.708Z] DEBUG: [UserSession/expireTicket] [K1qex52CPYkBiiHLgq92RZrEuQ502v1K] : Journaled and sent output message ; OutputStreamEntryMessage
[restate] [2023-08-15T21:13:54.709Z] DEBUG: [UserSession/expireTicket] [K1qex52CPYkBiiHLgq92RZrEuQ502v1K] : Function completed successfully.
[restate] [2023-08-15T21:13:54.713Z] DEBUG: [TicketService/unreserve] [IwF2v7ci3eUBiiHLgrFz6rkvK_3tRk6Z] : Invoking function.
[restate] [2023-08-15T21:13:54.713Z] DEBUG: [TicketService/unreserve] [IwF2v7ci3eUBiiHLgrFz6rkvK_3tRk6Z] : Journaled and sent output message ; OutputStreamEntryMessage
[restate] [2023-08-15T21:13:54.713Z] DEBUG: [TicketService/unreserve] [IwF2v7ci3eUBiiHLgrFz6rkvK_3tRk6Z] : Function completed successfully.
info

🚩 Explore the intermediate solution in part1, and run it with npm run part1

Durable timers

note

Implement it yourself or follow along by looking at the code under part2.

Restate suspends a function invocation when it waits on external input. The partial progress of the service invocation is durably stored in the log and can be resumed once the external input has arrived. Restate offers the same mechanism for timers.

Suspendable sleep

Earlier in this tutorial, you saw how Restate caller suspension works when you make an RPC request to another service. We artificially slowed down the target service by using platform-native sleep. Sometimes you intentionally want to suspend execution for a period of time, for example to back off from overloading a dependency or simply you are implementing a long-running business workflow. In those situations you can explicitly suspend execution for arbitrarily long periods of time using the Restate context's sleep function. Adapt the TicketService/reserve function to use ctx.sleep:

const reserve = async (ctx: restate.RpcContext) => {
await ctx.sleep(35000);
return true;
};

Send an addTicket request, as you did earlier. In the service logs, you can see the reserve function processing the sleep, then suspending, and then resuming again after the sleep completed. The addTicket function did a one-way call to the reserve function so didn't suspend but just finished its invocation.

Show the logs
[restate] [2023-08-15T21:20:50.312Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHMg255bahRoMD2cNRH] : Invoking function.
[restate] [2023-08-15T21:20:50.313Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHMg255bahRoMD2cNRH] : Adding message to journal and sending to Restate ; BackgroundInvokeEntryMessage
[restate] [2023-08-15T21:20:50.314Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHMg255bahRoMD2cNRH] : Journaled and sent output message ; OutputStreamEntryMessage
[restate] [2023-08-15T21:20:50.314Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHMg255bahRoMD2cNRH] : Function completed successfully.
[restate] [2023-08-15T21:20:50.317Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHMg3F8P4i8kn5RHEEj] : Invoking function.
[restate] [2023-08-15T21:20:50.317Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHMg3F8P4i8kn5RHEEj] : Adding message to journal and sending to Restate ; SleepEntryMessage
[restate] [2023-08-15T21:20:50.317Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHMg3F8P4i8kn5RHEEj] : Scheduling suspension in 30000 ms
[restate] [2023-08-15T21:21:20.319Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHMg3F8P4i8kn5RHEEj] : Writing suspension message to journal. ; SuspensionMessage
[restate] [2023-08-15T21:21:20.320Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHMg3F8P4i8kn5RHEEj] : Suspending function.
[restate] [2023-08-15T21:21:25.342Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHMg3F8P4i8kn5RHEEj] : Resuming (replaying) function.
[restate] [2023-08-15T21:21:25.342Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHMg3F8P4i8kn5RHEEj] : Matched and replayed message from journal ; SleepEntryMessage
[restate] [2023-08-15T21:21:25.342Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHMg3F8P4i8kn5RHEEj] : Journaled and sent output message ; OutputStreamEntryMessage
[restate] [2023-08-15T21:21:25.343Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHMg3F8P4i8kn5RHEEj] : Function completed successfully.

In UserSession/addTicket, revert the call to reserve to a request-response call, to see how the suspensions work across different services.

Replace the RpcContext.send() with RpcContext.rpc(), to end up with:

const addTicket = async (
ctx: restate.RpcContext,
userId: string,
ticketId: string,
) => {
const reservationSuccess = await ctx.rpc(ticketServiceApi).reserve(ticketId);
return reservationSuccess;
};
Show the logs
[restate] [2023-08-15T21:23:49.536Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHN6ft5nYhPYl4Q3r4Y] : Invoking function.
[restate] [2023-08-15T21:23:49.537Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHN6ft5nYhPYl4Q3r4Y] : Adding message to journal and sending to Restate ; InvokeEntryMessage
[restate] [2023-08-15T21:23:49.538Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHN6ft5nYhPYl4Q3r4Y] : Scheduling suspension in 30000 ms
[restate] [2023-08-15T21:23:49.540Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHN6gh7cpkCpv4OREYE] : Invoking function.
[restate] [2023-08-15T21:23:49.540Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHN6gh7cpkCpv4OREYE] : Adding message to journal and sending to Restate ; SleepEntryMessage
[restate] [2023-08-15T21:23:49.540Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHN6gh7cpkCpv4OREYE] : Scheduling suspension in 30000 ms
[restate] [2023-08-15T21:24:19.541Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHN6ft5nYhPYl4Q3r4Y] : Writing suspension message to journal. ; SuspensionMessage
[restate] [2023-08-15T21:24:19.547Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHN6ft5nYhPYl4Q3r4Y] : Suspending function.
[restate] [2023-08-15T21:24:19.551Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHN6gh7cpkCpv4OREYE] : Writing suspension message to journal. ; SuspensionMessage
[restate] [2023-08-15T21:24:19.552Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHN6gh7cpkCpv4OREYE] : Suspending function.
[restate] [2023-08-15T21:24:24.555Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHN6gh7cpkCpv4OREYE] : Resuming (replaying) function.
[restate] [2023-08-15T21:24:24.556Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHN6gh7cpkCpv4OREYE] : Matched and replayed message from journal ; SleepEntryMessage
[restate] [2023-08-15T21:24:24.557Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHN6gh7cpkCpv4OREYE] : Journaled and sent output message ; OutputStreamEntryMessage
[restate] [2023-08-15T21:24:24.557Z] DEBUG: [TicketService/reserve] [IwF2v7ci3eUBiiHN6gh7cpkCpv4OREYE] : Function completed successfully.
[restate] [2023-08-15T21:24:24.564Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHN6ft5nYhPYl4Q3r4Y] : Resuming (replaying) function.
[restate] [2023-08-15T21:24:24.565Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHN6ft5nYhPYl4Q3r4Y] : Matched and replayed message from journal ; InvokeEntryMessage
[restate] [2023-08-15T21:24:24.565Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHN6ft5nYhPYl4Q3r4Y] : Journaled and sent output message ; OutputStreamEntryMessage
[restate] [2023-08-15T21:24:24.565Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHN6ft5nYhPYl4Q3r4Y] : Function completed successfully.

Now you see in the logs that both services get suspended. The user session service gets suspended because it waits for a response from the reserve function and the ticket service gets suspended because it waits for the sleep to be completed. Restate keeps track of how long the service should sleep and then triggers it to resume the invocation. Finally, you see the responses of both functions coming in.

Reduce the time the TicketService/reserve function sleeps to a second, so you won't see the suspensions anymore.

caution

In keyed or singleton services, sleeping blocks all further processing (for the specific single key / all invocations, respectively).

Delayed calls

Take a look at a slightly different usage of timers. In the application, a ticket gets reserved for 15 minutes. If the user doesn't pay within that time interval, then it becomes available again to other users.

To do this, you can use a delayed call. This is a one-way call that gets delayed by a specified duration. This would make the UserSession/addTicket function look as follows:

const addTicket = async (
ctx: restate.RpcContext,
userId: string,
ticketId: string,
) => {
const reservationSuccess = await ctx
.rpc(ticketServiceApi)
.reserve(ticketId);

if (reservationSuccess) {
ctx
.sendDelayed(userSessionApi, 15 * 60 * 1000)
.expireTicket(userId, ticketId);
}

return reservationSuccess;
};

To test it out, put the delay to a lower value, for example 5 seconds, call the addTicket function, and see in the logs how the call to UserSession/expireTicket is executed 5 seconds later.

Show the logs
... logs from reserve call ...
[restate] [2023-08-15T21:29:44.426Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHT-at6er1g5Tkk14Ni] : Matched and replayed message from journal ; InvokeEntryMessage
[restate] [2023-08-15T21:29:44.427Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHT-at6er1g5Tkk14Ni] : Adding message to journal and sending to Restate ; BackgroundInvokeEntryMessage
[restate] [2023-08-15T21:29:44.427Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHT-at6er1g5Tkk14Ni] : Journaled and sent output message ; OutputStreamEntryMessage
[restate] [2023-08-15T21:29:44.427Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHT-at6er1g5Tkk14Ni] : Function completed successfully.
[restate] [2023-08-15T21:29:49.442Z] DEBUG: [UserSession/expireTicket] [K1qex52CPYkBiiHU-694wqUVn-2P15Au] : Invoking function.
[restate] [2023-08-15T21:29:49.443Z] DEBUG: [UserSession/expireTicket] [K1qex52CPYkBiiHU-694wqUVn-2P15Au] : Adding message to journal and sending to Restate ; BackgroundInvokeEntryMessage
[restate] [2023-08-15T21:29:49.444Z] DEBUG: [UserSession/expireTicket] [K1qex52CPYkBiiHU-694wqUVn-2P15Au] : Journaled and sent output message ; OutputStreamEntryMessage
[restate] [2023-08-15T21:29:49.444Z] DEBUG: [UserSession/expireTicket] [K1qex52CPYkBiiHU-694wqUVn-2P15Au] : Function completed successfully.
[restate] [2023-08-15T21:29:49.456Z] DEBUG: [TicketService/unreserve] [IwF2v7ci3eUBiiHT-bh91JO16nzGk-X0] : Invoking function.
[restate] [2023-08-15T21:29:49.457Z] DEBUG: [TicketService/unreserve] [IwF2v7ci3eUBiiHT-bh91JO16nzGk-X0] : Journaled and sent output message ; OutputStreamEntryMessage
[restate] [2023-08-15T21:29:49.457Z] DEBUG: [TicketService/unreserve] [IwF2v7ci3eUBiiHT-bh91JO16nzGk-X0] : Function completed successfully.

Don't forget to set the delay back to 15 minutes.

caution

You can implement this pattern in different ways. You could also sleep for 15 minutes at the end of the addTicket function and then call the TicketService/unreserve function:

await ctx.sleep(15 * 60 * 1000);
ctx.send(ticketServiceApi).unreserve({
userId: userId,
ticketId: ticketId,
});

Be aware that sleeping in a keyed service blocks any invocations for that key. In this case, the user would not be able to add any other tickets, nor buy the tickets.

If you do a sleep operation, the invocation is ongoing. If you do a delayed call, the invocation isn't ongoing until the delay has passed, so no key is locked.

Persistent application state

Applications often need to keep state. For example, the user session service needs to track the shopping cart items.

Restate offers a key-value store to persistently store application state.

Good news!

Restate's state is guaranteed to be consistent across retries and invocations. This eliminates the need for a session database.

The isolation level of the application state depends on the service type:

  1. Keyed service: Application state is isolated per key. All the invocations for the same key have access to the same application state. Restate's state feature is most useful for this service type.
  2. Unkeyed service: State is isolated per invocation. Using state isn't useful for this service type.

Getting and setting K/V state

Adapt the UserSession/addTicket function to keep track of the cart items. After reserving the product, you add the ticket to the shopping cart. Have a look at the highlighted code:

const addTicket = async (
ctx: restate.RpcContext,
userId: string,
ticketId: string) => {

// ... reserve call returning success ...

if (reservationSuccess) {
const tickets = (await ctx.get<string[]>("tickets")) ?? [];
tickets.push(ticketId);
ctx.set("tickets", tickets);

// ... expireTicket call ...
}

return reservationSuccess;
}

To retrieve the cart, you use ctx.get. This returns null if the value has never been set.

You can store multiple key-value pairs, by using different state keys. Here, you get the value under the key "tickets". Restate ensures that you get the cart belonging to the current user ID (key of the service).

After you added the ticket to the cart array, you set the state to the new value with ctx.set.

Run the services and call the addTicket function, to see the interaction with state.

Show the logs
... logs from reserve call ...
[restate] [2023-08-16T07:32:02.415Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHVZ9t6j52X1BcPKa1N] : Adding message to journal and sending to Restate ; GetStateEntryMessage
[restate] [2023-08-16T07:32:02.415Z] DEBUG: [UserSession/addTicket] [K1qex52CPYkBiiHVZ9t6j52X1BcPKa1N] : Adding message to journal and sending to Restate ; SetStateEntryMessage
... logs from expireTicket call ...

You see that when you request the state, it gets logged in the journal.

info

You can store any object in the state as long as the value can be serialized with Buffer.from(JSON.stringify(yourObject)).

When starting the invocation, Restate attaches the application state to the request. So when you operate on the state in your function with ctx.get and ctx.set, you get access to a local copy of the state for fast access.

Inspecting K/V state

Restate exposes information on invocations and application state via its Introspection SQL API. You can use this to gain insight into the status of invocations and the service state that is stored. You can inspect the application state of the UserSession service by using psql, as follows:

psql -h localhost -p 9071 -c "SELECT * FROM state WHERE service like '%UserSession%';"

Or watch how the state changes with:

watch -n 1 'psql -h localhost -p 9071 -c "SELECT * FROM state WHERE service like '\''%UserSession%'\''";'

Add several tickets to the state to see how the query result gets updated.

You can also stop and relaunch the service or restart the runtime (docker restart restate_dev), to see that this has no influence on the tickets.

Also adapt the UserSession/checkout function, to use the tickets:

const checkout = async (ctx: restate.RpcContext, userId: string) => {
// 1. Retrieve the tickets from state
const tickets = (await ctx.get<string[]>("tickets")) ?? [];

// 2. If there are no tickets, return `false`
if (tickets.length === 0) {
return false;
}

// 3. Call the `checkout` function of the checkout service with the tickets
const checkoutSuccess = await ctx
.rpc(checkoutApi)
.checkout({ userId: userId, tickets: tickets });

// 4. If this was successful, empty the tickets.
// Otherwise, let the user try again.
if (checkoutSuccess) {
ctx.clear("tickets");
}

return checkoutSuccess;
};

After the tickets are checked out, you clear the state with ctx.clear. Send a checkout request as earlier, and check via psql that the state is empty.

tip

Have a look at the documentation on introspection to learn more about inspecting the invocation status and application state.

📝 Try it out

Finishing UserSession/expireTicket

You have almost fully implemented the user session service. After checkout, let's finish UserSession/expireTicket. At the moment you have the following code:

const expireTicket = async (
ctx: restate.RpcContext,
userId: string,
ticketId: string,
) => {
ctx.send(ticketServiceApi).unreserve(ticketId);
};

Before you call unreserve, you first need to check if the ticket is still hold by the user. Retrieve the state and check if the ticket ID is in there. If this is the case, then you call TicketService/unreserve and remove it from the state.

Solution
const expireTicket = async (
ctx: restate.RpcContext,
userId: string,
ticketId: string,
) => {
const tickets = (await ctx.get<string[]>("tickets")) ?? [];

const ticketIndex = tickets.findIndex(ticket => ticket === ticketId);

if (ticketIndex != -1) {
tickets.splice(ticketIndex, 1);
ctx.set("tickets", tickets);

ctx.send(ticketServiceApi).unreserve(ticketId);
}
};

Call the expireTicket function with:

curl localhost:8080/UserSession/expireTicket \
-H 'content-type: application/json' \
-d '{"key": "123", "request": "456"}'

Implementing the ticket service

You can track the status of the tickets in the ticket service by storing it in the state. Implement the TicketService/reserve, TicketService/unreserve, and TicketService/markAsSold functions to do this.

While you are developing them, you can use psql to monitor the state of the TicketService:

psql -h localhost -p 9071 -c "SELECT * FROM state WHERE service like '\''%TicketService%'\''";'

Use watch to see the state change in real-time:

watch -n 1 'psql -h localhost -p 9071 -c "select service, service_key_utf8, key, value_utf8 from state where service like '\''%TicketService%'\''";'
  1. Implement the TicketService/reserve function. The function first retrieves the value for the "status" state key. If the value is set to TicketStatus.Available, then change it to TicketStatus.Reserved and return true (reservation successful). If the status isn't set to TicketStatus.Available, then return false. Remove the sleep that you added before.
Solution
const reserve = async (ctx: restate.RpcContext) => {
const status =
(await ctx.get<TicketStatus>("status")) ?? TicketStatus.Available;

if (status === TicketStatus.Available) {
ctx.set("status", TicketStatus.Reserved);
return true;
} else {
return false;
}
};

Now, you can't reserve the same ticket multiple times anymore. Call addTicket multiple times for the same ID. The first time it returns true, afterwards false.

  1. Implement the TicketService/unreserve function. The function should clear the value of the "status" key if it's not on status TicketStatus.Sold.
Solution
const unreserve = async (ctx: restate.RpcContext) => {
const status =
(await ctx.get<TicketStatus>("status")) ?? TicketStatus.Available;

if (status !== TicketStatus.Sold) {
ctx.clear("status");
}
};

Now, the ticket reservation status is cleared when the delayed expireTicket call triggers. Play around with reducing the delay of the expireTicket call in the addTicket function. Try to reserve the same ticket ID multiple times, and see how you are able to reserve it again after the unreserve function executed.

  1. Implement the TicketService/markAsSold function. The function should set the value of the "status" key to TicketStatus.Sold if it's reserved.
Solution
const markAsSold = async (ctx: restate.RpcContext) => {
const status =
(await ctx.get<TicketStatus>("status")) ?? TicketStatus.Available;

if (status === TicketStatus.Reserved) {
ctx.set("status", TicketStatus.Sold);
}
}
};

In the next section, you implement the Checkout/checkout function that calls markAsSold. This ties the final parts together.

info

🚩 Explore the intermediate solution in part2, and run it with:

npm run part2

Side effects

Implement it yourself or follow along by looking at the code under part3.

Restate's replay mechanism makes applications resilient to failures. It only requires your code to be deterministic.

If you need to execute a non-deterministic code snippet, for example generating a UUID or communicating to an external system, then you should wrap it in a side effect. The side effect executes the supplied function and eagerly stores the return value in Restate. Upon replay, the stored value gets inserted and the function doesn't get re-executed.

tip

You can use a side effect to avoid re-execution during replay for any arbitrary piece of user code. This includes pieces of time-consuming, deterministic user code.

Use a side effect in Checkout/checkout to create and store a UUID (with the uuid library) or idempotency key:

import { v4 as uuid } from "uuid";

const checkout = async (
ctx: restate.RpcContext,
request: { userId: string; tickets: string[] },
) => {
const idempotencyKey = await ctx.sideEffect<string>(async () => uuid());
return true;
};

The highlighted line of code wraps the creation of the UUID in a side effect with a string return type. When the checkout() function gets re-executed upon replay, Restate injects the stored value.

To experiment, you can print the idempotency key to the console and add a sleep to see how the UUID gets replayed after the suspension:

console.log("My idempotency key: " + idempotencyKey);
await ctx.sleep(31000);

Call UserSession/checkout and have a look at the logs to see what happens.

Show the logs
... logs of `UserSession/checkout` ...
[restate] [2023-08-16T08:00:37.965Z] DEBUG: [Checkout/checkout] [f7_cNhBPYp4BiiHV5Zxwno4-FrEGNoOL] : Invoking function.
[restate] [2023-08-16T08:00:37.966Z] DEBUG: [Checkout/checkout] [f7_cNhBPYp4BiiHV5Zxwno4-FrEGNoOL] : Adding message to journal and sending to Restate ; undefined
[restate] [2023-08-16T08:00:37.966Z] DEBUG: [Checkout/checkout] [f7_cNhBPYp4BiiHV5Zxwno4-FrEGNoOL] : Scheduling suspension in 30000 ms
[restate] [2023-08-16T08:00:37.967Z] DEBUG: [Checkout/checkout] [f7_cNhBPYp4BiiHV5Zxwno4-FrEGNoOL] : Received completion message from Restate, adding to journal. ; CompletionMessage
My idempotency key: dd92178a-85ca-4002-bfd5-0f3ed167317d
[restate] [2023-08-16T08:00:37.967Z] DEBUG: [Checkout/checkout] [f7_cNhBPYp4BiiHV5Zxwno4-FrEGNoOL] : Adding message to journal and sending to Restate ; SleepEntryMessage
[restate] [2023-08-16T08:00:37.967Z] DEBUG: [Checkout/checkout] [f7_cNhBPYp4BiiHV5Zxwno4-FrEGNoOL] : Scheduling suspension in 30000 ms
[restate] [2023-08-16T08:01:07.965Z] DEBUG: [UserSession/checkout] [K1qex52CPYkBiiHV5Zl7vY2Gie86MAZl] : Writing suspension message to journal. ; SuspensionMessage
[restate] [2023-08-16T08:01:07.966Z] DEBUG: [UserSession/checkout] [K1qex52CPYkBiiHV5Zl7vY2Gie86MAZl] : Suspending function.
[restate] [2023-08-16T08:01:07.967Z] DEBUG: [Checkout/checkout] [f7_cNhBPYp4BiiHV5Zxwno4-FrEGNoOL] : Writing suspension message to journal. ; SuspensionMessage
[restate] [2023-08-16T08:01:07.967Z] DEBUG: [Checkout/checkout] [f7_cNhBPYp4BiiHV5Zxwno4-FrEGNoOL] : Suspending function.
[restate] [2023-08-16T08:01:08.981Z] DEBUG: [Checkout/checkout] [f7_cNhBPYp4BiiHV5Zxwno4-FrEGNoOL] : Resuming (replaying) function.
[restate] [2023-08-16T08:01:08.983Z] DEBUG: [Checkout/checkout] [f7_cNhBPYp4BiiHV5Zxwno4-FrEGNoOL] : Matched and replayed message from journal ; undefined
My idempotency key: dd92178a-85ca-4002-bfd5-0f3ed167317d
[restate] [2023-08-16T08:01:08.983Z] DEBUG: [Checkout/checkout] [f7_cNhBPYp4BiiHV5Zxwno4-FrEGNoOL] : Matched and replayed message from journal ; SleepEntryMessage
[restate] [2023-08-16T08:01:08.984Z] DEBUG: [Checkout/checkout] [f7_cNhBPYp4BiiHV5Zxwno4-FrEGNoOL] : Journaled and sent output message ; OutputStreamEntryMessage
[restate] [2023-08-16T08:01:08.984Z] DEBUG: [Checkout/checkout] [f7_cNhBPYp4BiiHV5Zxwno4-FrEGNoOL] : Function completed successfully.

... logs of `UserSessionService/Checkout` ...

Remove the sleep again, before you continue.

caution

You can't execute any RestateContext calls from within a side effect. This is invalid code.

📝 Try it out

Executing the payment

The checkout function triggers the payment via some external payment provider (PaymentClient). You can find a payment client stub in auxiliary/PaymentClient. The payment client has two methods:

  • get() to create a new client,
  • call(idempotencyKey, amount) to execute the payment for a certain idempotency key (String) and total amount (double/number). The payment provider makes sure that exactly one payment gets processed for a single idempotency key.

Create a new payment client in CheckoutService/Checkout. Assume every ticket costs 40 dollars. Then execute the payment with the created idempotency key and store the boolean result of the call as the return value of the side effect. Also return the boolean result at the end of the checkout function.

Solution
const checkout = async (
ctx: restate.RpcContext,
request: { userId: string; tickets: string[] },
) => {
const idempotencyKey = await ctx.sideEffect<string>(async () => uuid());

const totalPrice = request.tickets.length * 40;

const paymentClient = PaymentClient.get();
const success = await ctx.sideEffect(() => paymentClient.call(idempotencyKey, totalPrice));

return success;
};

Add some tickets to your cart and then call UserSession/checkout. You should see similar logs:

// ... UserSessionService/Checkout logs ...
[restate] [2023-08-16T08:08:45.987Z] DEBUG: [Checkout/checkout] [f7_cNhBPYp4BiiHV5Zxwno4-FrEGNoOL] : Invoking function.
[restate] [2023-08-16T08:08:45.987Z] DEBUG: [Checkout/checkout] [f7_cNhBPYp4BiiHV5Zxwno4-FrEGNoOL] : Adding message to journal and sending to Restate ; undefined
[restate] [2023-08-16T08:08:45.987Z] DEBUG: [Checkout/checkout] [f7_cNhBPYp4BiiHV5Zxwno4-FrEGNoOL] : Scheduling suspension in 30000 ms
[restate] [2023-08-16T08:08:45.988Z] DEBUG: [Checkout/checkout] [f7_cNhBPYp4BiiHV5Zxwno4-FrEGNoOL] : Received completion message from Restate, adding to journal. ; CompletionMessage
Payment call succeeded for idempotency key 923d1558-bf17-4c94-b027-4cfd778049fe and amount 40
[restate] [2023-08-16T08:08:45.988Z] DEBUG: [Checkout/checkout] [f7_cNhBPYp4BiiHV5Zxwno4-FrEGNoOL] : Journaled and sent output message ; OutputStreamEntryMessage
[restate] [2023-08-16T08:08:45.988Z] DEBUG: [Checkout/checkout] [f7_cNhBPYp4BiiHV5Zxwno4-FrEGNoOL] : Function completed successfully.
[restate] [2023-08-16T08:08:45.989Z] DEBUG: [UserSession/checkout] [K1qex52CPYkBiiHV5Zl7vY2Gie86MAZl] : Received completion message from Restate, adding to journal. ; CompletionMessage
[restate] [2023-08-16T08:08:45.989Z] DEBUG: [UserSession/checkout] [K1qex52CPYkBiiHV5Zl7vY2Gie86MAZl] : Adding message to journal and sending to Restate ; ClearStateEntryMessage
[restate] [2023-08-16T08:08:45.989Z] DEBUG: [UserSession/checkout] [K1qex52CPYkBiiHV5Zl7vY2Gie86MAZl] : Journaled and sent output message ; OutputStreamEntryMessage
[restate] [2023-08-16T08:08:45.989Z] DEBUG: [UserSession/checkout] [K1qex52CPYkBiiHV5Zl7vY2Gie86MAZl] : Function completed successfully.
// ... UserSessionService/Checkout logs ...

You see the UserSession service retrieving the cart and calling the checkout service. The Checkout service then does a side effect, the payment, and another side effect. The checkout success is propagated back to the UserSession service.

Finishing the checkout workflow

Once this works, you implement the rest of the checkout workflow:

  • If the payment call was successful, then:
    • use the EmailClient to notify the users of the payment success. Prevent duplicate emails during retries by using a side effect.
    • let the calling UserSession mark all tickets as sold via TicketService/markAsSold.
  • If the payment call was unsuccessful, then:
    • use the EmailClient to notify the users of the payment failure. Prevent duplicate emails during retries by using a side effect.
Solution

The Checkout/checkout function now looks like the following:

const checkout = async (
ctx: restate.RpcContext,
request: { userId: string; tickets: string[] },
) => {
const idempotencyKey = await ctx.sideEffect(async () => uuid());

const totalPrice = request.tickets.length * 40;

const paymentClient = PaymentClient.get();

const success = await ctx.sideEffect(() => paymentClient.call(idempotencyKey, totalPrice));

const email = EmailClient.get();

if (success) {
await ctx.sideEffect(() => email.notifyUserOfPaymentSuccess(request.userId));
} else {
await ctx.sideEffect(() => email.notifyUserOfPaymentFailure(request.userId));
}

return success;
};

The UserSession/checkout function now looks like the following:

const checkout = async (ctx: restate.RpcContext, userId: string) => {
const tickets = (await ctx.get<string[]>("tickets")) ?? [];

if (tickets.length === 0) {
return false;
}

const checkoutSuccess = await ctx
.rpc(checkoutApi)
.checkout({ userId: userId, tickets: tickets! });

if (checkoutSuccess) {
// mark tickets as sold if checkout was successful
for (const ticketId of tickets) {
ctx.send(ticketServiceApi).markAsSold(ticketId);
}
ctx.clear("tickets");
}

return checkoutSuccess;
};

🥳 Except for a few resiliency tweaks, you have now fully implemented the ticket reservation system! Try it out by reserving some new tickets and buying them by checking out the cart.

info

🚩 Explore the intermediate solution in part3, and run it with:

npm run part3

Resiliency

Implement it yourself or follow along by looking at the code under part4.

As you have discovered throughout this tutorial, Restate makes your applications resilient out-of-the-box by:

  • Making sure messages can't get lost by durably storing all calls in the log. No more queues needed for asynchronous calls.
  • Retrying failed invocations. No more retry logic required for inter-service communication.
  • Restoring partial progress of invocations, via its durable execution mechanism. For example, if a service goes down due to an infrastructure failure, the ongoing invocations resume at the point in the user code where they left off.
  • Ensuring consistent application state with its key-value store.
  • Providing end-to-end exactly-once guarantees for incoming invocations.
Good news!

These features are switched on by default. No need to do anything.

Side effect retries

Also side effects are retried until they succeed. You can observe this behavior by using the paymentClient.failingCall in Checkout/checkout:

const paymentClient = PaymentClient.get();
const doPayment = () => paymentClient.failingCall(idempotencyKey, amount)
const success = await ctx.sideEffect(doPayment);

You now call paymentClient.failingCall which fails two times before succeeding. A failing side effect is retried until it succeeds. Have a look at the logs to see the retries.

Show the logs
[restate] [2023-08-16T08:24:28.507Z] DEBUG: [Checkout/checkout] [bE19gwjrZxIBiiHKisNy5Jjvmc0fMHyk] : Invoking function.
[restate] [2023-08-16T08:24:28.507Z] DEBUG: [Checkout/checkout] [bE19gwjrZxIBiiHKisNy5Jjvmc0fMHyk] : Adding message to journal and sending to Restate ; undefined
[restate] [2023-08-16T08:24:28.507Z] DEBUG: [Checkout/checkout] [bE19gwjrZxIBiiHKisNy5Jjvmc0fMHyk] : Scheduling suspension in 30000 ms
[restate] [2023-08-16T08:24:28.508Z] DEBUG: [Checkout/checkout] [bE19gwjrZxIBiiHKisNy5Jjvmc0fMHyk] : Received completion message from Restate, adding to journal. ; CompletionMessage
Payment call failed for idempotency key adab6b35-0536-4348-8912-35f8cd474cf8 and amount 40. Retrying...
[restate] [2023-08-16T08:24:28.508Z] DEBUG: [Checkout/checkout] [bE19gwjrZxIBiiHKisNy5Jjvmc0fMHyk] : Adding message to journal and sending to Restate ; undefined
[restate] [2023-08-16T08:24:28.508Z] DEBUG: [Checkout/checkout] [bE19gwjrZxIBiiHKisNy5Jjvmc0fMHyk] : Scheduling suspension in 30000 ms
[restate] [2023-08-16T08:24:28.509Z] DEBUG: [Checkout/checkout] [bE19gwjrZxIBiiHKisNy5Jjvmc0fMHyk] : Received completion message from Restate, adding to journal. ; CompletionMessage
[restate] [2023-08-16T08:24:28.509Z] DEBUG: Error while executing side effect 'side-effect': Error - Payment call failed
[restate] [2023-08-16T08:24:28.519Z] DEBUG: Error: Payment call failed
at PaymentClient.failingCall (/Users/till/restate/git/tour-of-restate-typescript/src/aux/payment_client.ts:37:13)
at doPayment (/Users/till/restate/git/tour-of-restate-typescript/src/part4/checkout.ts:29:47)
at AsyncLocalStorage.run (node:async_hooks:346:14)
at executeAndLogSideEffect (/Users/till/restate/git/tour-of-restate-typescript/node_modules/@restatedev/restate-sdk/src/restate_context_impl.ts:234:69)
at executeWithRetries (/Users/till/restate/git/tour-of-restate-typescript/node_modules/@restatedev/restate-sdk/src/restate_context_impl.ts:428:20)
at RestateGrpcContextImpl.sideEffect (/Users/till/restate/git/tour-of-restate-typescript/node_modules/@restatedev/restate-sdk/src/restate_context_impl.ts:300:12)
at RpcContextImpl.sideEffect (/Users/till/restate/git/tour-of-restate-typescript/node_modules/@restatedev/restate-sdk/src/restate_context_impl.ts:552:21)
at checkout (/Users/till/restate/git/tour-of-restate-typescript/src/part4/checkout.ts:30:29)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at dispatchUnkeyedRpcHandler (/Users/till/restate/git/tour-of-restate-typescript/node_modules/@restatedev/restate-sdk/src/server/base_restate_server.ts:366:18)
[restate] [2023-08-16T08:24:28.520Z] DEBUG: Retrying in 10 ms
[restate] [2023-08-16T08:24:28.520Z] DEBUG: [Checkout/checkout] [bE19gwjrZxIBiiHKisNy5Jjvmc0fMHyk] : Adding message to journal and sending to Restate ; SleepEntryMessage
[restate] [2023-08-16T08:24:28.520Z] DEBUG: [Checkout/checkout] [bE19gwjrZxIBiiHKisNy5Jjvmc0fMHyk] : Scheduling suspension in 30000 ms
[restate] [2023-08-16T08:24:28.532Z] DEBUG: [Checkout/checkout] [bE19gwjrZxIBiiHKisNy5Jjvmc0fMHyk] : Received completion message from Restate, adding to journal. ; CompletionMessage
Payment call failed for idempotency key adab6b35-0536-4348-8912-35f8cd474cf8 and amount 40. Retrying...
[restate] [2023-08-16T08:24:28.532Z] DEBUG: [Checkout/checkout] [bE19gwjrZxIBiiHKisNy5Jjvmc0fMHyk] : Adding message to journal and sending to Restate ; undefined
[restate] [2023-08-16T08:24:28.532Z] DEBUG: [Checkout/checkout] [bE19gwjrZxIBiiHKisNy5Jjvmc0fMHyk] : Scheduling suspension in 30000 ms
[restate] [2023-08-16T08:24:28.532Z] DEBUG: [Checkout/checkout] [bE19gwjrZxIBiiHKisNy5Jjvmc0fMHyk] : Received completion message from Restate, adding to journal. ; CompletionMessage
[restate] [2023-08-16T08:24:28.532Z] DEBUG: Error while executing side effect 'side-effect': Error - Payment call failed
[restate] [2023-08-16T08:24:28.534Z] DEBUG: Error: Payment call failed
at PaymentClient.failingCall (/Users/till/restate/git/tour-of-restate-typescript/src/aux/payment_client.ts:37:13)
at doPayment (/Users/till/restate/git/tour-of-restate-typescript/src/part4/checkout.ts:29:47)
at AsyncLocalStorage.run (node:async_hooks:346:14)
at executeAndLogSideEffect (/Users/till/restate/git/tour-of-restate-typescript/node_modules/@restatedev/restate-sdk/src/restate_context_impl.ts:234:69)
at executeWithRetries (/Users/till/restate/git/tour-of-restate-typescript/node_modules/@restatedev/restate-sdk/src/restate_context_impl.ts:428:20)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at checkout (/Users/till/restate/git/tour-of-restate-typescript/src/part4/checkout.ts:30:19)
at dispatchUnkeyedRpcHandler (/Users/till/restate/git/tour-of-restate-typescript/node_modules/@restatedev/restate-sdk/src/server/base_restate_server.ts:366:18)
at HostedGrpcServiceMethod.invoke (/Users/till/restate/git/tour-of-restate-typescript/node_modules/@restatedev/restate-sdk/src/types/grpc.ts:49:23)
[restate] [2023-08-16T08:24:28.534Z] DEBUG: Retrying in 20 ms
[restate] [2023-08-16T08:24:28.534Z] DEBUG: [Checkout/checkout] [bE19gwjrZxIBiiHKisNy5Jjvmc0fMHyk] : Adding message to journal and sending to Restate ; SleepEntryMessage
[restate] [2023-08-16T08:24:28.534Z] DEBUG: [Checkout/checkout] [bE19gwjrZxIBiiHKisNy5Jjvmc0fMHyk] : Scheduling suspension in 30000 ms
[restate] [2023-08-16T08:24:28.556Z] DEBUG: [Checkout/checkout] [bE19gwjrZxIBiiHKisNy5Jjvmc0fMHyk] : Received completion message from Restate, adding to journal. ; CompletionMessage
Payment call succeeded for idempotency key adab6b35-0536-4348-8912-35f8cd474cf8 and amount 40
[restate] [2023-08-16T08:24:28.557Z] DEBUG: [Checkout/checkout] [bE19gwjrZxIBiiHKisNy5Jjvmc0fMHyk] : Adding message to journal and sending to Restate ; undefined
[restate] [2023-08-16T08:24:28.557Z] DEBUG: [Checkout/checkout] [bE19gwjrZxIBiiHKisNy5Jjvmc0fMHyk] : Scheduling suspension in 30000 ms
[restate] [2023-08-16T08:24:28.557Z] DEBUG: [Checkout/checkout] [bE19gwjrZxIBiiHKisNy5Jjvmc0fMHyk] : Received completion message from Restate, adding to journal. ; CompletionMessage
Payment successful. Notifying user about shipment.
Notifying user 123 of payment success
[restate] [2023-08-16T08:24:28.557Z] DEBUG: [Checkout/checkout] [bE19gwjrZxIBiiHKisNy5Jjvmc0fMHyk] : Adding message to journal and sending to Restate ; undefined
[restate] [2023-08-16T08:24:28.557Z] DEBUG: [Checkout/checkout] [bE19gwjrZxIBiiHKisNy5Jjvmc0fMHyk] : Scheduling suspension in 30000 ms
[restate] [2023-08-16T08:24:28.558Z] DEBUG: [Checkout/checkout] [bE19gwjrZxIBiiHKisNy5Jjvmc0fMHyk] : Received completion message from Restate, adding to journal. ; CompletionMessage
[restate] [2023-08-16T08:24:28.558Z] DEBUG: [Checkout/checkout] [bE19gwjrZxIBiiHKisNy5Jjvmc0fMHyk] : Journaled and sent output message ; OutputStreamEntryMessage
[restate] [2023-08-16T08:24:28.558Z] DEBUG: [Checkout/checkout] [bE19gwjrZxIBiiHKisNy5Jjvmc0fMHyk] : Function completed successfully.