Skip to main content

Tour of Restate

This tutorial guides you through the development of an end-to-end Restate application, and covers all the essential features. After this tutorial, you should have a firm understanding of how Restate can help you and feel comfortable to tackle your next application on your own.

This tutorial implements a ticket reservation application for a theatre. It allows users to add tickets for specific seats to their shopping cart. After the user adds a ticket, the seat gets reserved for 15 minutes. If the user doesn't pay for the ticket within that time interval, the reservation is released and the ticket becomes available to other users.

Restate applications are made up of services that expose handlers. Handlers are functions that implement business logic. Restate manages their invocation and execution. Services communicate with one another using Remote Procedure Calls (RPC). Our ticket example consists of three services:

As we go, you will discover how Restate can help you with some intricacies in this application.

Prerequisites

This guide is written for:

  • TypeScript SDK version: 1.4.0
  • Restate Server Docker image: docker.io/restatedev/restate:1.1

Getting Started

1
Set up the services

Download the example and run locally with an IDE:

restate example typescript-tour-of-restate && cd typescript-tour-of-restate

Install the dependencies and build the app:

npm install && npm run build

Run the services

npm run app-dev

This GitHub repository contains the basic skeleton of the TypeScript services that you develop in this tutorial.

2
Launch Restate

Restate is a single self-contained binary. No external dependencies needed. Check out our Local Dev page for instructions on how to install Restate Server, then do:

restate-server

3
Register the services with Restate

Now, we need to tell Restate where our services are running. You can register services by calling the Restate Admin API (default port 9070) and supplying it the service endpoint URI:

restate deployments register http://localhost:9080
Output
❯ SERVICES THAT WILL BE ADDED:

- CheckoutService

Type: Service

HANDLER  INPUT TYPE                                   OUTPUT TYPE

handle   one of "empty or value of content-type */*"  value with content-type "application/json"



- CartObject

Type: VirtualObject ⬅️ 🚶🚶🚶

HANDLER       INPUT TYPE                                   OUTPUT TYPE

addTicket     one of "empty or value of content-type */*"  value with content-type "application/json"

expireTicket  one of "empty or value of content-type */*"  value with content-type "application/json"

checkout      one of "empty or value of content-type */*"  value with content-type "application/json"



- TicketObject

Type: VirtualObject ⬅️ 🚶🚶🚶

HANDLER     INPUT TYPE                                   OUTPUT TYPE

reserve     one of "empty or value of content-type */*"  value with content-type "application/json"

markAsSold  one of "empty or value of content-type */*"  value with content-type "application/json"

unreserve   one of "empty or value of content-type */*"  value with content-type "application/json"





✔ Are you sure you want to apply those changes? · yes

✅ DEPLOYMENT:

SERVICE          REV

TicketObject     1

CheckoutService  1

CartObject       1

If you run Restate with Docker, replace http://localhost:9080 by http://host.docker.internal:9080.

🚀
All set up!

In src/app you will find the skeletons of the various services to help you start implementing the app. For example:

checkout_service.ts
export const checkoutService = restate.service({

  name: "CheckoutService",

  handlers: {

    async handle(ctx: restate.Context, request: { userId: string; tickets: string[] }){

      return true;

    },

  }

});



export const CheckoutService: typeof checkoutService = { name: "CheckoutService"};

Restate handlers have the Restate Context supplied as the first argument. This is the entrypoint to the SDK.

The app.ts file contains the definition of the endpoint that hosts the services.

Invoking Handlers

Implement it yourself or follow along by looking at the code under part1

Handlers can be invoked in several ways: via HTTP requests, programmatically with the SDK, or via Kafka events.

Request-response calls over HTTP

Let's start with invoking our handler over HTTP using curl.

For example, add a ticket seat2B to the cart of Mary by calling the addTicket handler of the CartObject:

curl localhost:8080/CartObject/Mary/addTicket -H 'content-type: application/json' -d '"seat2B"'

If this prints out true, then you have a working setup.

When Mary wants to proceed with the purchase, call the checkout handler of the CartObject:

curl -X POST localhost:8080/CartObject/Mary/checkout

We will use these two curl commands often when developing the code, so keep them handy.

Restate as proxy

Restate acts as a proxy for your services. It forwards the request to the correct service and handler. Therefore, the request is sent to Restate and not directly to the service.

Why does the path contain Mary?

Handlers are either a part of plain services or Virtual Objects. Virtual Objects are a special type of service that allows you to group handlers together, share state between them, and control concurrency. Each Virtual Object has a unique key. We will cover the difference in more detail later. For now, it's important to note that when invoking a handler within a Virtual Object, you need to specify its key. In our example, the CartObject and TicketObject are Virtual Objects, while the CheckoutService is a plain service. To add the ticket to Mary's cart, we need to specify the key Mary in the path to reach her Virtual Object.

We can do the same programmatically within a handler by using the SDK. Let's try this out!

Request-response calls between handlers

You can also call other handlers programmatically by using the clients generated by the Restate SDK. Let's try this out!

When we add a ticket to the cart, the CartObject/addTicket handler first needs to reserve the ticket for the user. It does that by calling the TicketObject/reserve handler:

cart_object.ts
async addTicket(ctx: restate.ObjectContext, ticketId: string) {

  const reservationSuccess = await ctx.objectClient(TicketObject, ticketId).reserve();



  return true;

},
Service logs
// withClass highlight-line

[restate] [CartObject/addTicket][inv_1gdJBtdVEcM95mw1LLMMJY1Y0thJ9ugFGN][2024-03-18T16:30:28.790Z] DEBUG: Invoking function.

[restate] [CartObject/addTicket][inv_1gdJBtdVEcM95mw1LLMMJY1Y0thJ9ugFGN][2024-03-18T16:30:28.792Z] DEBUG: Adding message to journal and sending to Restate ; InvokeEntryMessage

[restate] [CartObject/addTicket][inv_1gdJBtdVEcM95mw1LLMMJY1Y0thJ9ugFGN][2024-03-18T16:30:28.792Z] DEBUG: Scheduling suspension in 30000 ms

// withClass highlight-line

[restate] [TicketObject/reserve][inv_1k78Krj3GqrK6odGaRe866kHZilkVf1H4l][2024-03-18T16:30:28.796Z] DEBUG: Invoking function.

[restate] [TicketObject/reserve][inv_1k78Krj3GqrK6odGaRe866kHZilkVf1H4l][2024-03-18T16:30:28.796Z] DEBUG: Journaled and sent output message ; OutputStreamEntryMessage

// withClass highlight-line

[restate] [TicketObject/reserve][inv_1k78Krj3GqrK6odGaRe866kHZilkVf1H4l][2024-03-18T16:30:28.796Z] DEBUG: Function completed successfully.

[restate] [CartObject/addTicket][inv_1gdJBtdVEcM95mw1LLMMJY1Y0thJ9ugFGN][2024-03-18T16:30:28.799Z] DEBUG: Received completion message from Restate, adding to journal. ; CompletionMessage

[restate] [CartObject/addTicket][inv_1gdJBtdVEcM95mw1LLMMJY1Y0thJ9ugFGN][2024-03-18T16:30:28.800Z] DEBUG: Journaled and sent output message ; OutputStreamEntryMessage

// withClass highlight-line

[restate] [CartObject/addTicket][inv_1gdJBtdVEcM95mw1LLMMJY1Y0thJ9ugFGN][2024-03-18T16:30:28.800Z] DEBUG: Function completed successfully.
  1. Create the client via ctx.serviceClient or ctx.objectClient (for Virtual Objects). Specify the service definition (TicketObject) and optionally the Virtual Object key (ticketId).
  2. Specify the handler you want to call and supply the request. Here reserve().
  3. Await the response of the call.

Send a request to CartObject/addTicket as we did previously, and have a look at the service logs.

Sending messages between handlers

We can also let handlers send messages to other handlers without waiting for a response.

In the example, when a seat gets added to the shopping cart, it gets reserved for 15 minutes. When a user didn't proceed with the payment before the timeout, the CartObject/expireTicket handler is triggered. Let the expireTicket handler call the TicketObject/unreserve handler.

cart_object.ts
async expireTicket(ctx: restate.ObjectContext, ticketId: string) {

  ctx.objectSendClient(TicketObject, ticketId).unreserve();

},

Specify that you want to call the TicketObject by supplying ticketObjectApi to the send function. Then call the unreserve handler on the TicketObject.

Once you have added this to the code, call the CartObject/expireTicket handler:

curl localhost:8080/CartObject/Mary/expireTicket -H 'content-type: application/json' -d '"seat2B"'
Service logs
[restate] [CartObject/expireTicket][inv_1gdJBtdVEcM942bjcDmb1c1khoaJe11Hbz][2024-03-18T16:14:24.671Z] DEBUG: Invoking function.

[restate] [CartObject/expireTicket][inv_1gdJBtdVEcM942bjcDmb1c1khoaJe11Hbz][2024-03-18T16:14:24.672Z] DEBUG: Adding message to journal and sending to Restate ; BackgroundInvokeEntryMessage

[restate] [CartObject/expireTicket][inv_1gdJBtdVEcM942bjcDmb1c1khoaJe11Hbz][2024-03-18T16:14:24.673Z] DEBUG: Journaled and sent output message ; OutputStreamEntryMessage

// withClass highlight-line

[restate] [CartObject/expireTicket][inv_1gdJBtdVEcM942bjcDmb1c1khoaJe11Hbz][2024-03-18T16:14:24.673Z] DEBUG: Function completed successfully.

[restate] [TicketObject/unreserve][inv_1k78Krj3GqrK3GuJXkgaXBbg69R47TCeAN][2024-03-18T16:14:24.677Z] DEBUG: Invoking function.

[restate] [TicketObject/unreserve][inv_1k78Krj3GqrK3GuJXkgaXBbg69R47TCeAN][2024-03-18T16:14:24.677Z] DEBUG: Journaled and sent output message ; OutputStreamEntryMessage

// withClass highlight-line

[restate] [TicketObject/unreserve][inv_1k78Krj3GqrK3GuJXkgaXBbg69R47TCeAN][2024-03-18T16:14:24.677Z] DEBUG: Function completed successfully.

The service logs show how the expireTicket handler gets executed and then the unreserve handler. The call to expireTicket finishes earlier than the unreserve handler because expireTicket didn't wait for the response of the unreserve handler.

Restate as message queue

Restate persists and retries failed one-way invocations. There is no need to set up message queues to ensure delivery!

Sending messages via `curl`

To send messages via curl, add /send to the handler path:

curl localhost:8080/CartObject/Mary/addTicket/send -H 'content-type: application/json' -d '"seat2B"'
Output
{"invocationId":"inv_1aiqX0vFEFNH1Umgre58JiCLgHfTtztYK5","status":"Accepted"}

This returns the invocation ID. This is a unique identifier for the invocation. You can use it to track the progress of the invocation via the CLI, and to correlate logs and metrics.

📝 Try it out

Make the CartObject/checkout handler call the CheckoutService/handle handler.

For the request field, you can use a hard-coded string array for now: ["seat2B"]. You will fix this later on. Note that the CheckoutService is not a Virtual Object, so you don't need to specify a key.

Solution

Add the following code to the CartObject/checkout handler:

cart_object.ts
async checkout(ctx: restate.ObjectContext) {

  const success = await ctx.serviceClient(CheckoutService)

      .handle({userId: ctx.key, tickets: ["seat2B"]});



  return success;

},

Call CartObject/checkout as you did earlier and have a look at the logs again to see what happened:

[restate] [CartObject/checkout][inv_1gdJBtdVEcM919dOBhoVBm3fUMlaIHnANr][2024-03-19T07:57:24.018Z] DEBUG: Invoking function.

[restate] [CartObject/checkout][inv_1gdJBtdVEcM919dOBhoVBm3fUMlaIHnANr][2024-03-19T07:57:24.019Z] DEBUG: Adding message to journal and sending to Restate ; InvokeEntryMessage

[restate] [CartObject/checkout][inv_1gdJBtdVEcM919dOBhoVBm3fUMlaIHnANr][2024-03-19T07:57:24.020Z] DEBUG: Scheduling suspension in 30000 ms

[restate] [CheckoutService/handle][inv_16WnWCiCVV5G2gUUevDM4uIli4v7TN9k2d][2024-03-19T07:57:24.023Z] DEBUG: Invoking function.

[restate] [CheckoutService/handle][inv_16WnWCiCVV5G2gUUevDM4uIli4v7TN9k2d][2024-03-19T07:57:24.024Z] DEBUG: Journaled and sent output message ; OutputStreamEntryMessage

[restate] [CheckoutService/handle][inv_16WnWCiCVV5G2gUUevDM4uIli4v7TN9k2d][2024-03-19T07:57:24.024Z] DEBUG: Function completed successfully.

[restate] [CartObject/checkout][inv_1gdJBtdVEcM919dOBhoVBm3fUMlaIHnANr][2024-03-19T07:57:24.026Z] DEBUG: Received completion message from Restate, adding to journal. ; CompletionMessage

[restate] [CartObject/checkout][inv_1gdJBtdVEcM919dOBhoVBm3fUMlaIHnANr][2024-03-19T07:57:24.027Z] DEBUG: Journaled and sent output message ; OutputStreamEntryMessage

[restate] [CartObject/checkout][inv_1gdJBtdVEcM919dOBhoVBm3fUMlaIHnANr][2024-03-19T07:57:24.027Z] DEBUG: Function completed successfully.

Durable Execution

The calls we just did seem like regular RPC calls as you might know them from other service frameworks. But under the hood a lot more is happening.

Restate makes RPC calls resilient by letting the Restate Server and SDK cooperate. Restate tracks the execution of code in a journal and can replay it in case of a failure. This is called Durable Execution.

Have a look at the animation to understand what happened under-the-hood:

Journals

This animation shows you what happened under the hood when we did the reserve call from the `CartObject` to the `TicketObject`. The animation uses the TypeScript SDK.

CartObject[Mary]
suspended...
async function addTicket(ctx, ticketId){
  const success = await ctx
    .objectClient(ticketObject)
    .reserve(ticketId);

  return success;
}
TicketObject[seat2B]
suspended...
async function reserve(ctx, ticketId){
  ...
  return success;
}

Whenever a failure would happen, Restate would be able to recover the latest state of the handler by sending over the journal. The code would fast-forward to the point where it crashed, and continue executing from there on.

To see the recovery of partial progress in practice, let's make the CartObject/addTicket handler crash right after the call.

cart_object.ts
async addTicket(ctx: restate.ObjectContext, ticketId: string) {

  const reservationSuccess = await ctx.objectClient(TicketObject, ticketId).reserve();



  return true;

},

Add the following code after the reservation call, to let the code throw an error after the call:

throw new Error("Failing");

Call CartObject/addTicket again and have a look at the service logs.

Service logs
// withClass highlight-line

[restate] [CartObject/addTicket][inv_1aiqX0vFEFNH0TF1pLRFBDFosQCCTAN1M5][2024-04-16T13:28:20.245Z] DEBUG:  Adding message to journal and sending to Restate ; InvokeEntryMessage

[restate] [CartObject/addTicket][inv_1aiqX0vFEFNH0TF1pLRFBDFosQCCTAN1M5][2024-04-16T13:28:20.246Z] DEBUG:  Scheduling suspension in 30000 ms

[restate] [TicketObject/reserve][inv_1iGFK6hGrtOf3jcD8PupmCOJz1SDzvfPi1][2024-04-16T13:28:20.296Z] DEBUG:  Invoking function.

[restate] [TicketObject/reserve][inv_1iGFK6hGrtOf3jcD8PupmCOJz1SDzvfPi1][2024-04-16T13:28:20.296Z] DEBUG:  Journaled and sent output message ; OutputEntryMessage

// withClass highlight-line

[restate] [TicketObject/reserve][inv_1iGFK6hGrtOf3jcD8PupmCOJz1SDzvfPi1][2024-04-16T13:28:20.296Z] DEBUG:  Function completed successfully.

[restate] [CartObject/addTicket][inv_1aiqX0vFEFNH0TF1pLRFBDFosQCCTAN1M5][2024-04-16T13:28:20.362Z] DEBUG:  Received completion message from Restate, adding to journal.

// withClass highlight-line ; CompletionMessage

Trace: [restate] [CartObject/addTicket][inv_1aiqX0vFEFNH0TF1pLRFBDFosQCCTAN1M5][2024-04-16T13:28:20.363Z] TRACE:  Function completed with an error: Failing Error: Failing

... rest of trace ...

[restate] [CartObject/addTicket][inv_1aiqX0vFEFNH0TF1pLRFBDFosQCCTAN1M5][2024-04-16T13:28:20.372Z] DEBUG:  Invocation ended with retryable error. ; ErrorMessage

// withClass highlight-line

[restate] [CartObject/addTicket][inv_1aiqX0vFEFNH0TF1pLRFBDFosQCCTAN1M5][2024-04-16T13:28:20.437Z] DEBUG:  Resuming (replaying) function.

// withClass highlight-line

[restate] [CartObject/addTicket][inv_1aiqX0vFEFNH0TF1pLRFBDFosQCCTAN1M5][2024-04-16T13:28:20.437Z] DEBUG:  Matched and replayed message from journal ; InvokeEntryMessage

// withClass highlight-line

Trace: [restate] [CartObject/addTicket][inv_1aiqX0vFEFNH0TF1pLRFBDFosQCCTAN1M5][2024-04-16T13:28:20.437Z] TRACE:  Function completed with an error: Failing Error: Failing

... rest of trace ...

You see the retries taking place. And you see that only the first time the call to the CheckoutService was made. The other times, the call was skipped and the journaled response was replayed.

By default, Restate will keep retrying failed invocations until they succeed. If you want to cancel an invocation in a retry loop, you can use the CLI to do this. Let's have a look at that next.

Debugging with the CLI

Now that we have a failing invocation, let's take the opportunity to show you how you can get more information about what is going on with the CLI. The CLI is a management tool that lets you interact with the Restate Server. You can use it to boostrap a new project, but also to get information about the services and invocations.

Have a look at some useful commands and try them out yourself:

restate services list
Output
NAME             REVISION  FLAVOR    DEPLOYMENT TYPE  DEPLOYMENT ID

🌎  CartObject       1         ⬅️ 🚶🚶🚶  HTTP 2           dp_11pXug0mWsff2NOoRBZbOcV

🌎  CheckoutService  1                   HTTP 2           dp_11pXug0mWsff2NOoRBZbOcV

🌎  TicketObject     1         ⬅️ 🚶🚶🚶  HTTP 2           dp_11pXug0mWsff2NOoRBZbOcV`,

`$ restate services describe CartObject

📜 Service Information:

―――――――――――――――――――――――

Name:             CartObject

Service type:     VirtualObject

Revision:         1

Public:           true

Deployment ID:    dp_11pXug0mWsff2NOoRBZbOcV

Deployment Type:  HTTP 2

Protocol Style:   Streaming

Endpoint:         http://localhost:9080/

Created at:       2024-04-23T12:32:16.691000000Z





🔌 Handlers:

――――――――――――

HANDLER       INPUT TYPE                 OUTPUT TYPE

addTicket     one of "empty or value of  value with content-type "application/json"

content-type */*"

checkout      one of "empty or value of  value with content-type "application/json"

content-type */*"

expireTicket  one of "empty or value of  value with content-type "application/json"

content-type */*"
restate invocations list
Output
❯ [2024-04-23 14:41:59.365 +02:00] inv_1fmRNvSNVxNp5PTqHI4HLJ17HpxzhB3MEV

Target:      CartObject/Mary/addTicket

Status:      backing-off  (18 seconds and 284 ms. Retried 9 time(s). Next

retry in in 8 seconds and 220 ms))

Deployment:  dp_11pXug0mWsff2NOoRBZbOcV [required]

Error:       [2024-04-23 14:42:13.706 +02:00]

[500] Failing

Caused by:   UNKNOWN
restate invocations describe inv_1fmRNvSNVxNp5PTqHI4HLJ17HpxzhB3MEV
Output
📜 Invocation Information:

――――――――――――――――――――――――――

Created at:   2024-04-23 14:41:59.365 +02:00 (a minute ago)

Target:       CartObject/Mary/addTicket

Status:       backing-off  (1 minute, 23 seconds and 937 ms. Retried 14

time(s). Next retry in in 991 ms))

Deployment:   dp_11pXug0mWsff2NOoRBZbOcV [required]

Error:        [2024-04-23 14:43:13.248 +02:00]

[500] Failing

Caused by:    UNKNOWN

Modified at:  2024-04-23 14:41:59.388 +02:00



💡   This invocation is bound to run on deployment 'dp_11pXug0mWsff2NOoRBZbOcV'. To guarantee

safety and correctness, invocations that made progress on a deployment

cannot move to newer deployments automatically.



🚂 Invocation Progress:

―――――――――――――――――――――――

[Ingress]

└──(this)─> CartObject/Mary/addTicket

▸

├──── ☑️  #1 Call TicketObject/seat2B/reserve inv_19maBIcE9uRD1CrHgpGXZ7FcXPsz4bzkbL

└────>> backing-off
restate invocations cancel --kill inv_1fmRNvSNVxNp5PTqHI4HLJ17HpxzhB3MEV
Output
❯ [2024-04-23 14:41:59.365 +02:00] inv_1fmRNvSNVxNp5PTqHI4HLJ17HpxzhB3MEV

Target:      CartObject/Mary/addTicket

Status:      backing-off  (25 minutes, 29 seconds and 200 ms. Retried 141

time(s). Next retry in in 12 seconds and 94 ms))

Deployment:  dp_11pXug0mWsff2NOoRBZbOcV [required]

Error:       [2024-04-23 15:07:27.860 +02:00]

[500] Failing

Caused by:   UNKNOWN



✔ Are you sure you want to kill this invocation · yes



✅ Request was sent successfully
note

Remove the throwing of the exception from your code before you continue.

info

🚩 Explore the intermediate solution in part1, and run it with:

npm run part1

Scheduling Async Tasks

Implement it yourself or follow along by looking at the code under part2

When a handler calls another handler, Restate registers the call and makes sure it happens. You can also ask Restate to execute the call at a later point in the future, by adding a delay parameter to the call. Restate then registers the call and triggers it after the delay has passed.

In the application, a ticket gets reserved for 15 minutes. If the user doesn't pay within that time interval, then it becomes available again to other users.

Let the CartObject/addTicket handler call the CartObject/expireTicket handler with a delay of 15 minutes:

cart_object.ts
async addTicket(ctx: restate.ObjectContext, ticketId: string) {

  const reservationSuccess = await ctx.objectClient(TicketObject, ticketId).reserve();



  if (reservationSuccess) {

    ctx.objectSendClient(CartObject, ctx.key, {delay: 15 * 60 * 1000})

        .expireTicket(ticketId);

  }



  return reservationSuccess;

},

To test it out, put the delay to a lower value, for example 5 seconds, call the addTicket function, and see in the logs how the call to CartObject/expireTicket is executed 5 seconds later.

Service logs
... logs from reserve call ...

[restate] [CartObject/addTicket][inv_1gdJBtdVEcM90xbqbDEnOzNgilf2WmjZTP][2024-03-19T08:49:43.081Z] DEBUG: Received completion message from Restate, adding to journal. ; CompletionMessage

// withClass highlight-line

[restate] [CartObject/addTicket][inv_1gdJBtdVEcM90xbqbDEnOzNgilf2WmjZTP][2024-03-19T08:49:43.081Z] DEBUG: Adding message to journal and sending to Restate ; BackgroundInvokeEntryMessage

[restate] [CartObject/addTicket][inv_1gdJBtdVEcM90xbqbDEnOzNgilf2WmjZTP][2024-03-19T08:49:43.081Z] DEBUG: Journaled and sent output message ; OutputStreamEntryMessage

[restate] [CartObject/addTicket][inv_1gdJBtdVEcM90xbqbDEnOzNgilf2WmjZTP][2024-03-19T08:49:43.081Z] DEBUG: Function completed successfully.

[restate] [CartObject/expireTicket][inv_1gdJBtdVEcM93r8tce9IfwnbiAsk8lCevD][2024-03-19T08:49:48.092Z] DEBUG: Invoking function.

[restate] [CartObject/expireTicket][inv_1gdJBtdVEcM93r8tce9IfwnbiAsk8lCevD][2024-03-19T08:49:48.093Z] DEBUG: Adding message to journal and sending to Restate ; BackgroundInvokeEntryMessage

[restate] [CartObject/expireTicket][inv_1gdJBtdVEcM93r8tce9IfwnbiAsk8lCevD][2024-03-19T08:49:48.093Z] DEBUG: Journaled and sent output message ; OutputStreamEntryMessage

[restate] [CartObject/expireTicket][inv_1gdJBtdVEcM93r8tce9IfwnbiAsk8lCevD][2024-03-19T08:49:48.093Z] DEBUG: Function completed successfully.

// withClass highlight-line

[restate] [TicketObject/unreserve][inv_1k78Krj3GqrK529L4BRmz8ntFtiw2DkahH][2024-03-19T08:49:48.141Z] DEBUG: Invoking function.

[restate] [TicketObject/unreserve][inv_1k78Krj3GqrK529L4BRmz8ntFtiw2DkahH][2024-03-19T08:49:48.141Z] DEBUG: Journaled and sent output message ; OutputStreamEntryMessage

[restate] [TicketObject/unreserve][inv_1k78Krj3GqrK529L4BRmz8ntFtiw2DkahH][2024-03-19T08:49:48.141Z] DEBUG: Function completed successfully.

Don't forget to set the delay back to 15 minutes.

No workflow orchestrator or cron jobs needed!

Durable timers are a powerful feature that can be used to implement workflows, schedule async tasks, or plan background jobs. Restate makes them resilient to failures and ensures that they get executed. No extra infrastructure needed!

Suspendable sleep

Another timer-like feature of the SDK is suspendable sleep. Restate will make sure that the function gets resumed after the specified duration has passed. When running on function-as-a-service platforms, your function can suspend in the meantime, so you don't pay for the wait time.

await ctx.sleep(15 * 60 * 1000);
info

🚩 Explore the intermediate solution in part2, and run it with:

npm run part2

Virtual Objects vs. Services

Implement it yourself or follow along by looking at the code under part3

At the beginning of this tutorial, we mentioned that the TicketObject and CartObject services are Virtual Objects.

Virtual Objects are identified by a key and allow you to store K/V state in Restate. For each Virtual Object (key), only one invocation can run at a time (across all the handlers of that Virtual Object).

Services, on the other hand, do not have access to K/V state, and handlers can run concurrently.

Virtual Objects simplify many use cases

With access to consistent K/V state and strong concurrency guarantees, implementing the TicketObject in a resilient and consistent way becomes straightforward. When a user reserves a ticket, we want to be sure that no concurrent other requests are reserving the same ticket at the same time. To get this behaviour, we key the TicketObject on ticket ID. We now have a single Virtual Object per ticket.

Long-running operations in Virtual Objects

If you do long-running operations in a Virtual Object, no other invocations are processed the meantime. For example, if you would implement the expiration of the ticket in the CartObject service by sleeping for 15 minutes:

await ctx.sleep(15 * 60 * 1000);

ctx.objectSendClient(TicketObject, ticketId).unreserve();

The user wouldn't be able to add any other tickets, nor buy the tickets. If you do a delayed call, the invocation isn't ongoing until the delay has passed, so the Virtual Object is not locked.

Consistent K/V state

Restate offers a key-value store to store application state for Virtual Objects.

No need for session databases!

Restate's state is guaranteed to be consistent across retries and invocations. This eliminates the need for a session database.

Getting and setting K/V state

Adapt the CartObject/addTicket function to keep track of the cart items. After reserving the product, you add the ticket to the shopping cart. Have a look at the highlighted code:

cart_object.ts
async addTicket(ctx: restate.ObjectContext, ticketId: string) {

  const reservationSuccess = await ctx.objectClient(TicketObject, ticketId).reserve();



  if (reservationSuccess) {

    const tickets = (await ctx.get<string[]>("tickets")) ?? [];

    tickets.push(ticketId);

    ctx.set("tickets", tickets);



    ctx.objectSendClient(CartObject, ctx.key, {delay: 15 * 60 * 1000})

        .expireTicket(ticketId);

  }



  return reservationSuccess;

},

To retrieve the cart, you use ctx.get. This returns null if the value has never been set.

After you added the ticket to the cart array, you set the state to the new value with ctx.set.

You can store multiple key-value pairs, by using different state keys. Here, you get the value under the key "tickets". Restate returns the cart belonging to the current Virtual Object (for example, user Mary).

Run the services and call the addTicket function, to see the interaction with state.

Service logs
... logs from reserve call ...

// withClass highlight-line

[restate] [CartObject/addTicket][inv_1gdJBtdVEcM97yGYEG5NWYtbMlnSAGHGY9][2024-03-19T08:55:20.941Z] DEBUG: Adding message to journal and sending to Restate ; GetStateEntryMessage

// withClass highlight-line

[restate] [CartObject/addTicket][inv_1gdJBtdVEcM97yGYEG5NWYtbMlnSAGHGY9][2024-03-19T08:55:20.941Z] DEBUG: Adding message to journal and sending to Restate ; SetStateEntryMessage

... logs from expireTicket call ...
Local state access

When starting the invocation, Restate attaches the application state to the request. So when you operate on the state in your function, you get access to a local copy of the state for fast access.

Also adapt the CartObject/checkout function, to use the tickets:

cart_object.ts
async checkout(ctx: restate.ObjectContext) {

  const tickets = (await ctx.get<string[]>("tickets")) ?? [];



  if (tickets.length === 0) {

    return false;

  }



  const success = await ctx.serviceClient(CheckoutService)

      .handle({userId: ctx.key, tickets});



  if (success) {

    ctx.clear("tickets");

  }



  return success;

},

After the tickets are checked out, you clear the state with ctx.clear.

Inspecting K/V state

Restate exposes information on invocations and application state. You can watch the state of the CartObject service, via:

restate kv get -w -n 1 CartObject Mary
Output
🤖 State:

―――――――――



Service  CartObject

Key      Mary



KEY      VALUE

tickets  [

           "seat2B"

         ]

Add some tickets to the state to see how the query result gets updated.

Then, send a checkout request as earlier, and notice that the state is now empty.

📝 Try it out

Finishing CartObject/expireTicket

You have almost fully implemented the CartObject. Let's finish CartObject/expireTicket.

Before you call unreserve, you first need to check if the ticket is still held by the user. Retrieve the state and check if the ticket ID is in there. If this is the case, then you call TicketObject/unreserve and remove it from the state.

Solution
cart_object.ts
async expireTicket(ctx: restate.ObjectContext, ticketId: string) {

  const tickets = (await ctx.get<string[]>("tickets")) ?? [];



  const ticketIndex = tickets.findIndex((ticket) => ticket === ticketId);



  if (ticketIndex != -1) {

    tickets.splice(ticketIndex, 1);

    ctx.set("tickets", tickets);



    ctx.objectSendClient(TicketObject, ticketId).unreserve();

  }

},

Call the expireTicket handler with:

curl localhost:8080/CartObject/Mary/expireTicket -H 'content-type: application/json' -d '"seat2B"'

Implementing the TicketObject

Track the status of the tickets in the TicketObject by storing it in the state and transitioning from one state to another, like a state machine. The possible states are available (default), reserved, and sold. Implement the handlers in the TicketObject to reserve, unreserve, and mark a ticket as sold.

While you are developing them, monitor the state of the TicketObject via:

restate kv get -w -n 1 TicketObject seat2B
1
TicketObject/reserve

  1. Retrieve the value for the "status" state key.
  2. If the value is set to TicketStatus.Available, then change it to TicketStatus.Reserved and return true (reservation successful).
  3. If the status isn't set to TicketStatus.Available, then return false.
Solution
ticket_object.ts
async reserve(ctx: restate.ObjectContext) {

  const status =

      (await ctx.get<TicketStatus>("status")) ?? TicketStatus.Available;



  if (status === TicketStatus.Available) {

    ctx.set("status", TicketStatus.Reserved);

    return true;

  } else {

    return false;

  }

},

Now, you can't reserve the same ticket multiple times anymore. Call addTicket multiple times for the same ID. The first time it returns true, afterwards false.

2
TicketObject/unreserve

Clear the "status", if it's not equal to TicketStatus.Sold.

Solution
ticket_object.ts
async unreserve(ctx: restate.ObjectContext) {

  const status =

      (await ctx.get<TicketStatus>("status")) ?? TicketStatus.Available;



  if (status !== TicketStatus.Sold) {

    ctx.clear("status");

  }

},

Now, the ticket reservation status is cleared when the delayed expireTicket call triggers. Play around with reducing the delay of the expireTicket call in the addTicket handler. Try to reserve the same ticket ID multiple times, and see how you are able to reserve it again after the unreserve handler executed.

3
TicketObject/markAsSold

Set the "status" to TicketStatus.Sold if it's reserved.

Solution
ticket_object.ts
async markAsSold(ctx: restate.ObjectContext) {

  const status =

      (await ctx.get<TicketStatus>("status")) ?? TicketStatus.Available;



  if (status === TicketStatus.Reserved) {

    ctx.set("status", TicketStatus.Sold);

  }

},

In the next section, you implement the CheckoutService/handle function that calls markAsSold. This ties the final parts together.

info

🚩 Explore the intermediate solution in part3, and run it with:

npm run part3

Journaling actions

Implement it yourself or follow along by looking at the code under part4

Restate's Durable Execution mechanism tracks the progress of the code execution in a journal. Once an action/result has made it to the journal, it will not be re-executed on retries.

You can store the return value of any function in the journal, by using ctx.run. This lets you capture potentially non-deterministic computation and interaction with external systems in a safe way. The SDK also offers helper functions for creating UUIDs and generating random numbers.

Handler logic needs to be deterministic

For the replay to work, code needs to be deterministic, otherwise the replayed entries do not line up with the code execution on retries. So use ctx.run to store the result of non-deterministic operations!

We can use this feature to do exactly-once payments in CheckoutService/handle:

1
Generate an idempotency token

Let's use the SDK helper functions to generate a unique payment identifier and store it in Restate. Once the token is stored, it will be the same on retries. Try it out by printing the idempotency key and then throwing an error:

checkout_service.ts
async handle(

  ctx: restate.Context,

  request: { userId: string; tickets: string[] }

) {

  const idempotencyKey = ctx.rand.uuidv4();

  console.info("My idempotency key: " + idempotencyKey);

  throw new Error("Something happened!");



  return true;

},

Call CartObject/checkout and have a look at the logs to see what happens.

Service logs
... logs of `CartObjectService/CheckoutService` ...

[restate] [CheckoutService/handle][inv_1jhuapyO2Bpg3prqzrAJOFs99mt7jv5x3r][2024-03-19T09:15:30.498Z] DEBUG: Invoking function.

// withClass highlight-line

My idempotency key: e25b747f-ecfb-443b-8939-1935392aab6b

Trace: [restate] [CheckoutService/handle][inv_1jhuapyO2Bpg3prqzrAJOFs99mt7jv5x3r][2024-03-19T09:15:30.499Z] TRACE: Function completed with an error: Something happened! Error: Something happened!

... rest of trace  ...

[restate] [CheckoutService/handle][inv_1jhuapyO2Bpg3prqzrAJOFs99mt7jv5x3r][2024-03-19T09:15:30.512Z] DEBUG: Invocation ended with retryable error. ; ErrorMessage

[restate] [CheckoutService/handle][inv_1jhuapyO2Bpg3prqzrAJOFs99mt7jv5x3r][2024-03-19T09:15:30.568Z] DEBUG: Invoking function.

// withClass highlight-line

My idempotency key: e25b747f-ecfb-443b-8939-1935392aab6b

Trace: [restate] [CheckoutService/handle][inv_1jhuapyO2Bpg3prqzrAJOFs99mt7jv5x3r][2024-03-19T09:15:30.568Z] TRACE: Function completed with an error: Something happened! Error: Something happened!

... rest of trace  ...

[restate] [CheckoutService/handle][inv_1jhuapyO2Bpg3prqzrAJOFs99mt7jv5x3r][2024-03-19T09:15:30.568Z] DEBUG: Invocation ended with retryable error. ; ErrorMessage

... retries continue ...

2
Trigger the payment

Execute the payment via an external payment provider via PaymentClient.get().call(idempotencyKey, amount). The payment provider will deduplicate payments based on the idempotency token. We assume every ticket costs 40 dollars.

checkout_service.ts
async handle(

  ctx: restate.Context,

  request: { userId: string; tickets: string[] }

) {

  const totalPrice = request.tickets.length * 40;



  const idempotencyKey = ctx.rand.uuidv4();

  const success = await ctx.run(() =>

    PaymentClient.get().call(idempotencyKey, totalPrice)

  );



  return success;

},

📝 Try it out

Let's finish the checkout flow by sending the email notifications and marking the tickets as sold.

1
Implement the email notifications

After the CheckoutService/handle handler has handled the payment, you need to notify the users of the payment status:

  • Payment success: notify the users via EmailClient.get().notifyUserOfPaymentSuccess(request.getUserId()).
  • Payment failure: notify the users via the EmailClient.get().notifyUserOfPaymentFailure(request.getUserId()).
Solution
checkout_service.ts
async handle(ctx: restate.Context, request: { userId: string; tickets: string[] }) {

  const totalPrice = request.tickets.length * 40;



  const idempotencyKey = ctx.rand.uuidv4();

  const success = await ctx.run(() => PaymentClient.get().call(idempotencyKey, totalPrice));



  if (success) {

    await ctx.run(() => EmailClient.get().notifyUserOfPaymentSuccess(request.userId));

  } else {

    await ctx.run(() => EmailClient.get().notifyUserOfPaymentFailure(request.userId));

  }



  return success;

},

2
Mark tickets as sold

Let the CartObject/checkout handler mark all tickets as sold by calling TicketObject/markAsSold for each ticket.

Solution
cart_object.ts
async checkout(ctx: restate.ObjectContext) {

  const tickets = (await ctx.get<string[]>("tickets")) ?? [];



  if (tickets.length === 0) {

    return false;

  }



  const success = await ctx.serviceClient(CheckoutService)

      .handle({userId: ctx.key, tickets});



  if (success) {

    for (const ticketId of tickets) {

      ctx.objectSendClient(TicketObject, ticketId).markAsSold();

    }

    ctx.clear("tickets");

  }



  return success;

},

🥳 You have now fully implemented the ticket reservation system! Try it out by reserving some new tickets and buying them by checking out the cart.

info

🚩 Explore the intermediate solution in part4, and run it with:

npm run part4

Idempotency for any request

As you saw, generating idempotency keys inside your handlers and storing them in Restate is easy. But this doesn't guard us yet against retries of the HTTP request to Restate.

For example, if the caller of the addTicket handler didn't receive the success response of its first request, it might retry the request.

The second request will return false because the ticket already got reserved the first time, but the caller won't know about this.

To cover this, you can add an idempotency-key header to the incoming request to let Restate deduplicate them.

In our example, when we call the CartObject/addTicket handler, the first time the response is true and the second time it's false. However, if we use the same idempotency key, the second call will return true as well, because it will return the result of the first call:

curl localhost:8080/CartObject/Mary/addTicket -H 'content-type: application/json' \

    -H 'idempotency-key: ad5472esg4dsg525dssdfa5loi'  \

    -d '"seat2C"'

You can also see from the service logs that the handler wasn't executed the second time.

End-to-end idempotency

Restate gives you idempotency for any service, handler and request for free. No extra setup.

info

You only need this when invoking handlers over HTTP. When a handler calls another handler, Restate automatically takes care of the idempotency.

Tracing

Restate exposes OpenTelemetry traces of your invocations.

1
Run Jaeger

docker run -d --name jaeger -e COLLECTOR_OTLP_ENABLED=true \

    -p 4317:4317 -p 16686:16686 jaegertracing/all-in-one:1.46

2
Relaunch Restate with tracing enabled

restate-server --tracing-endpoint http://localhost:4317

3
Send a few requests

curl localhost:8080/CartObject/Mary/addTicket -H 'content-type: application/json' -d '"seat2A"'

curl localhost:8080/CartObject/Mary/addTicket -H 'content-type: application/json' -d '"seat2B"'

curl localhost:8080/CartObject/Mary/addTicket -H 'content-type: application/json' -d '"seat2C"'

curl -X POST localhost:8080/CartObject/Mary/checkout

4
Go to the Jaeger UI

http://localhost:16686

5
Inspect the traces

Select the CartObject service from the service dropdown.

You should see the addTicket and checkout requests listed. Have a look at the traces of the checkout call:

CheckoutService call traces

You can see the calls that were done to Restate, for example invoke, sleep, one way call, get state, etc., and their timings. If you expand one of the traces, you can see tags describing some metadata of the context call, for example invocation ID and the request.

For more information, have a look at the tracing docs.

🏁 The end

You reached the end of this tutorial!

Let's recap what you did! You have built a ticket reservation system that is resilient, consistent, and scalable. We used Restate to provide us with durable, distributed building blocks to simplify the implementation of the system. Let's list a few of them:

What you implementedWhat you didn't implement, as Restate handles it for you
✅ Request-response invocations❌ Handling retries, timeouts, etc.
✅ Sending messages❌ Deploy and operate message queues for async requests
✅ Idempotent HTTP calls❌ Write deduplication logic
✅ Durable Execution: retries, partial progress recovery, and suspensions❌ Manual retry logic and partial progress recovery
✅ Durable timers: sleeping and scheduling async tasks❌ Workflow orchestrators or cron jobs for scheduling tasks
✅ Virtual Objects: concurrency guarantees and shared state❌ Guards for keeping state consistent across retries, concurrent requests, and scaling out.
✅ K/V state: storing and inspecting❌ Session databases for state. State consistency guards.
✅ Storing computation results in the journal❌ Logic to make operations idempotent (e.g. generate idempotency keys)

You now know the essentials to start developing Restate services! Have a look at the next steps to explore further.

Next steps