Introspection

Restate exposes information on invocations and application state via its CLI and Introspection SQL API. You can use this to gain insight into the status of invocations and the service state that is stored. This can be useful for troubleshooting. For example, a Virtual Object might be blocked and you want to kill the invocation that is blocking it, but you don’t know the invocation ID. Or you want to check what is currently stored in the state of a service. You can inspect what is stored in Restate via the UI, via the CLI (via commands or SQL queries), and via curl.

You can use the UI to debug your applications. Have a look at the UI announcement blog post to get some inspiration on how you can use the UI for debugging and understanding your applications.

SQL over the data in Restate

Restate exposes the following SQL tables:

sys_invocation: to inspect invocations
sys_journal: to inspect the invocations’ journal
sys_journal_events: to inspect the events of an invocation, such as failures
sys_inbox: to inspect the queue of invocations waiting for a Virtual Object or Workflow key
sys_locks: to inspect the locks held on Virtual Object and Workflow keys
sys_vqueues, sys_vqueue_meta, sys_vqueue_entry_status: to inspect the virtual queues in which invocations wait for their turn to run
sys_scheduler: to inspect the scheduler state of virtual queues
sys_rules, sys_user_limits: to inspect concurrency-limit rules and the limits in use
sys_service: to inspect the registered services
sys_deployment: to inspect service deployments
sys_promise: to inspect durable promises of workflows
state: to inspect application state

You can find the schema of each of the tables in the references. The CLI documents them as well: restate sql --help lists the tables and restate sql describe <table> shows the columns of a table. The Restate Introspection SQL API has been implemented based on DataFusion and supports standard SQL syntax.

Inspecting invocations

You can execute SQL queries via the CLI or over HTTP.

Listing ongoing invocations

For each of the queries we will show the CLI command and the equivalent SQL query that you can execute via the CLI or over HTTP.

restate invocations list

restate sql --json "select * from sys_invocation;"

curl localhost:9070/query --json '{ "query" : "select * from sys_invocation" }'

Active invocations are invocations that haven’t completed yet. Completed invocations are retained for the configured retention time, after which they are removed from the sys_invocation table.The CLI command only shows the active invocations, not the completed ones. Use --all to see completed ones as well.

Retrieving the status of an invocation

restate invocations describe my_invocation_id

restate sql --json "select * from sys_invocation where id = 'my_invocation_id';"

curl localhost:9070/query --json "{ \"query\" : \"select * from sys_invocation where id = 'my_invocation_id'\" }"

The status is either:

scheduled: will start at a later time (a delayed call)
pending: queued behind other invocations, for example on the same Virtual Object key or for a concurrency limit
ready: ready to be processed, but not yet running
running: actively processing
backing-off: retrying due to a failure
suspended: waiting on some external input (e.g. request-response call, awakeable, sleep, …)
paused: stopped until resumed, because it ran out of retries or was paused
completed: completed, successfully or with a failure (shown until the retention time expires)

The status column of the sys_invocation table shows an invocation that is retrying as ready. The CLI combines it with the sys_vqueues table to show backing-off and the retry count, see Inspecting invocation retries.

Inspecting the invocation journal

restate invocations journal my_invocation_id

restate sql --json "select * from sys_journal where id = 'my_invocation_id';"

curl localhost:9070/query --json "{ \"query\" : \"select * from sys_journal where id = 'my_invocation_id'\" }"

restate invocations describe shows a preview of the journal, with the first and last entries. Use restate invocations journal to see all of it:

# The whole journal, instead of the first and last entries
restate invocations journal my_invocation_id --all
# A single entry or an inclusive range of entries
restate invocations journal my_invocation_id 20..30
# Include the entry payloads: inputs, outputs and state values
restate invocations journal my_invocation_id --payload

Inspecting invocation retries

To have a look at the invocations that are currently in a retry loop, you can execute:

restate invocations list --status backing-off

restate sql --json "select entry_id from sys_vqueues where stage = 'inbox' and status = 'backing-off';"

curl localhost:9070/query --json "{ \"query\" : \"select entry_id from sys_vqueues where stage = 'inbox' and status = 'backing-off'\" }"

The sys_invocation table shows a retrying invocation as ready, so the SQL query looks at the virtual queues instead. Join with sys_invocation on the entry_id column to get the details of the retrying invocations, or use restate invocations describe to see the retry count and the last failure.

Listing invocations that are blocking a Virtual Object

You can retrieve the invocation ID that is currently blocking a Virtual Object via:

restate invocations list MyServiceName/myKey

restate sql --json "select acquired_by from sys_locks where lock_name = 'MyServiceName/myKey' and scope is null;"

curl localhost:9070/query --json "{ \"query\" : \"select acquired_by from sys_locks where lock_name = 'MyServiceName/myKey' and scope is null\" }"

The lock of a Virtual Object key has the name service/key. Each scope has its own Virtual Object instance with its own lock, with the same lock_name but with the scope column set. Filter on scope is null for the unscoped instance, or on scope = 'my-scope' for a scoped one:

restate sql --json "select acquired_by from sys_locks where lock_name = 'MyServiceName/myKey' and scope = 'my-scope';"

curl localhost:9070/query --json "{ \"query\" : \"select acquired_by from sys_locks where lock_name = 'MyServiceName/myKey' and scope = 'my-scope'\" }"

With the CLI, you can also drill down and list only invocations that are blocking any Virtual Object:

restate invocations list --virtual-objects-only

Add --key my-key to list only invocations that are blocking a specific Virtual Object.You can then use the invocation ID to cancel the invocation.

Checking the last time an invocation was modified

restate invocations describe my_invocation_id

restate sql --json "select modified_at from sys_invocation where id = 'my_invocation_id';"

curl localhost:9070/query --json "{ \"query\" : \"select modified_at from sys_invocation where id = 'my_invocation_id'\" }"

This includes any modification to the invocation “data”, for example when the service last switched its status from running to suspended, or when the last journal entry was added.

Checking how an invocation was triggered

To find out if an invocation was triggered via the ingress or by another service:

restate invocations describe my_invocation_id

restate sql --json "select invoked_by, invoked_by_service_name, invoked_by_id from sys_invocation where id = 'my_invocation_id';"

curl localhost:9070/query --json "{ \"query\" : \"select invoked_by, invoked_by_service_name, invoked_by_id from sys_invocation where id = 'my_invocation_id'\" }"

With the CLI, you see the caller in the invocation information. For an invocation triggered via the ingress:

Invoked by:           [Ingress]

For an invocation triggered by another service, you see the calling handler and its invocation ID:

Invoked by:           Greeter/greet inv_1eqitSmJJyOT43RR3rGklqtYYKNrqQYDZS

For the SQL queries, the invoked_by field contains ingress, service, or subscription for invocations created by a Kafka subscription. If the invocation was triggered by another service, then the fields invoked_by_service_name and invoked_by_id will supply more information about the invoking service. See the sys_invocation reference for all values.

Retrieving the trace ID of an invocation

restate invocations describe my_invocation_id

restate sql --json "select trace_id from sys_invocation where id = 'my_invocation_id';"

curl localhost:9070/query --json "{ \"query\" : \"select trace_id from sys_invocation where id = 'my_invocation_id'\" }"

Afterwards, you can use this trace ID to search for spans in Jaeger.

Listing inactive invocations

To list the oldest invocations that are not making progress:

restate invocations list --order asc --status pending,backing-off,suspended

restate sql --json "select * from sys_invocation where modified_at <= now() - interval '1' hour;"

curl localhost:9070/query --json "{ \"query\" : \"select * from sys_invocation where modified_at <= now() - interval '1' hour\" }"

Listing zombie invocations

Zombie invocations are invocations that are pinned to a specific deployment but that deployment was forcefully removed. You can list them by executing:

restate invocations list --zombie

restate sql --json "select * from sys_invocation where pinned_deployment_id = 'my_deployment_id';"

curl localhost:9070/query --json "{ \"query\" : \"select * from sys_invocation where pinned_deployment_id = 'my_deployment_id'\" }"

For the SQL queries, you need to know the deployment ID of the deployment that was forcefully removed.

Inspecting application state

With the CLI

To retrieve the state of a specific service and service key, do:

restate state get <SERVICE_NAME> <SERVICE_KEY>

Example output:

State:
――――――
 KEY   VALUE
 seen  8

If the values are not JSON-encoded UTF-8 strings, then it is also possible to use the --binary flag, and get the value as base64 encoded string. Use --plain to retrieve the state as a JSON object. This can be useful in combination with tools like jq for example:

restate state get counter bob --plain | jq '.seen'

Each scope has its own Virtual Object instance with its own state. Add --scope <SCOPE> to retrieve the state of a scoped instance.

With SQL queries

You can query the application state via the state table.

restate sql --json "select * from state where service_name = 'MyServiceName' and service_key = 'myKey';"

curl localhost:9070/query --json "{ \"query\" : \"select * from state where service_name = 'MyServiceName' and service_key = 'myKey'\" }"

If your state value is a regular string, then you can access its content in the column value_utf8. The scope column is NULL for the unscoped Virtual Object instance, and contains the scope for scoped instances. To retrieve the state of a specific service name, service key and state key, do:

restate sql --json "select * from state where service_name = 'MyServiceName' and service_key = 'myKey' and key = 'myStateKey';"

curl localhost:9070/query --json "{ \"query\" : \"select * from state where service_name = 'MyServiceName' and service_key = 'myKey' and key = 'myStateKey'\" }"

The state key is the name you used to store the state with the SDK. For example, the code snippet ctx.set("count", 1) stores 1 under the key count.

Joining the sys_invocation and state table

To join the sys_invocation and state table:

restate sql --json "select * from sys_invocation JOIN state on sys_invocation.target_service_name = state.service_name and sys_invocation.target_service_key = state.service_key;"

curl localhost:9070/query --json '{ "query" : "select * from sys_invocation JOIN state on sys_invocation.target_service_name = state.service_name and sys_invocation.target_service_key = state.service_key" }'

Edit application state

You can edit the application state either via the state tab of the UI or via the CLI:

restate state edit <SERVICE_NAME> <SERVICE_KEY>

This command opens the editor set in RESTATE_EDITOR, VISUAL or EDITOR (in that order), falling back to vi. It sends the new state values back to the runtime to be applied. Use --binary if the values are not JSON-encoded UTF-8 strings. In this case, you need to decode the base64-encoded string, and encode it back to base64 after editing. Add --scope <SCOPE> to edit the state of a scoped Virtual Object instance.

If during the editing of the state with the CLI, an invocation changed the state as well, then the edit of the CLI will not take affect. If you want the CLI state edit to be applied even if the state has changed in the meantime, then use the --force flag.

An example on how to edit the K/V state of the service counter for the key bob: restate state edit needs a terminal. Scripts and coding agents change the state with restate state patch instead, which applies a JSON Patch to the state and supports --dry-run and --yes:

restate state patch counter bob --patch '[{"op": "replace", "path": "/seen", "value": 8}]'

To delete the state of a key, or of all keys of a Virtual Object or Workflow, use restate state clear:

restate state clear counter/bob
restate state clear counter

SDKs

Services

Restate Cloud / BYOC / OSS

References

SQL over the data in Restate

Inspecting invocations

Inspecting application state

With the CLI

With SQL queries

Edit application state

​SQL over the data in Restate

​Inspecting invocations

​Inspecting application state

​With the CLI

​With SQL queries

​Edit application state

SQL over the data in Restate

Inspecting invocations

Inspecting application state

With the CLI

With SQL queries

Edit application state