Introspection

Restate exposes information on invocations and application state via its CLI and Introspection SQL API. You can use this to gain insight into the status of invocations and the service state that is stored. This can be useful for troubleshooting. For example, a Virtual Object might be blocked and you want to kill the invocation that is blocking it, but you don’t know the invocation ID. Or you want to check what is currently stored in the state of a service. You can inspect what is stored in Restate via the UI, via the CLI (via commands or SQL queries), and via curl.

You can use the UI to debug your applications. Have a look at the UI announcement blog post to get some inspiration on how you can use the UI for debugging and understanding your applications.

SQL over the data in Restate

Restate exposes the following SQL tables:

sys_invocation: to inspect invocations
sys_inbox: to inspect queue of pending invocations
sys_keyed_service_status: to inspect the status of a Virtual Object
sys_journal: to inspect the invocations’ journal
sys_service: to inspect the registered services
sys_deployment: to inspect service deployments
sys_idempotency: to inspect idempotency keys
state: to inspect application state

You can find the schema of each of the tables in the references. The Restate Introspection SQL API has been implemented based on DataFusion and supports standard SQL syntax.

Inspecting invocations

You can execute SQL queries via the CLI or over HTTP.

Listing ongoing invocations

For each of the queries we will show the CLI command and the equivalent SQL query that you can execute via the CLI or over HTTP.

restate invocations list

restate sql --json "select * from sys_invocation;"

curl localhost:9070/query --json '{ "query" : "select * from sys_invocation" }'

Restate only retains the entries for active invocations, workflows or invocations that were invoked with an idempotency key. Active invocations are invocations that haven’t completed yet and are either invoked or suspended. For workflows and invocations that were invoked with an idempotency key, the entries are retained for their specified retention time.The CLI command only shows the active invocations, not the completed ones. Use --all to see completed ones as well.

Retrieving the status of an invocation

restate invocations describe my_invocation_id

restate sql --json "select * from sys_invocation where id = 'my_invocation_id';"

curl localhost:9070/query --json '{ "query" : "select * from sys_invocation where id = 'my_invocation_id';" }'

The status is either:

pending: enqueued waiting for its turn
ready: ready to be processed, but not yet running
running: actively processing
backing-off: retrying due to a failure
suspended: waiting on some external input (e.g. request-response call, awakeable, sleep, …)
completed: completed (this is shown only for idempotent invocations)

Inspecting the invocation journal

restate invocations describe my_invocation_id

restate sql --json "select * from sys_journal where id = 'my_invocation_id';"

curl localhost:9070/query --json '{ "query" : "select * from sys_journal where id = 'my_invocation_id';" }'

You see the journal printed in the output.

Inspecting invocation retries

To have a look at the invocations that are currently in a retry loop, you can execute:

restate invocations list --status backing-off

restate sql --json "select * from sys_invocation where retry_count > 1;"

curl localhost:9070/query --json '{ "query" : "select * from sys_invocation where retry_count > 1;" }'

Listing invocations that are blocking a Virtual Object

You can retrieve the invocation ID that is currently blocking a Virtual Object via:

restate invocations list --service MyService --key myKey

restate sql --json "select invocation_id from sys_keyed_service_status where service_name = 'test.MyServiceName' and service_key = 'myKey';"

curl localhost:9070/query --json '{ "query" : "select invocation_id from sys_keyed_service_status where service_name = 'test.MyServiceName' and service_key = 'myKey';" }'

With the CLI, you can also drill down and list only invocations that are blocking any Virtual Object:

restate invocations list --virtual-objects-only

Add --key my-key to list only invocations that are blocking a specific Virtual Object.You can then use the invocation ID to cancel the invocation.

Checking the last time an invocation was modified

restate invocations describe my_invocation_id

restate sql --json "select modified_at from sys_invocation where id = 'my_invocation_id';"

curl localhost:9070/query --json '{ "query" : "select modified_at from sys_invocation where id = 'my_invocation_id';" }'

This includes any modification to the invocation “data”, for example when the service last switched its status from invoked to suspended, or when the last journal entry was added.

Checking how an invocation was triggered

To find out if an invocation was triggered via the ingress or by another service:

restate invocations describe my_invocation_id

restate sql --json "select invoked_by, invoked_by_service_name, invoked_by_id from sys_invocation where id = 'my_invocation_id';"

curl localhost:9070/query --json '{ "query" : "select invoked_by, invoked_by_service_name, invoked_by_id from sys_invocation where id = 'my_invocation_id';" }'

With the CLI, you see the caller at the root of the tree in the invocation progress:

🚂 Invocation Progress:
―――――――――――――――――――――――
[Ingress]
└──(this)─> Greeter/greet

For the SQL queries, the invoked_by field contains either ingress or service. If the invocation was triggered by another service, then the fields invoked_by_service_name and invoked_by_id will supply more information about the invoking service.

Retrieving the trace ID of an invocation

restate invocations describe my_invocation_id

restate sql --json "select trace_id from sys_invocation where id = 'my_invocation_id';"

curl localhost:9070/query --json '{ "query" : "select trace_id from sys_invocation where id = 'my_invocation_id';" }'

Afterwards, you can use this trace ID to search for spans in Jaeger.

Listing inactive invocations

To list the oldest invocations that are not making progress:

restate invocations list --oldest-first --status pending,backing-off,suspended

restate sql --json "select * from sys_invocation where to_timestamp(modified_at) <= now() - interval '1' hour;"

curl localhost:9070/query --json '{ "query" : "select * from sys_invocation where to_timestamp(modified_at) <= now() - interval '1' hour;" }'

Listing zombie invocations

Zombie invocations are invocations that are pinned to a specific deployment but that deployment was forcefully removed. You can list them by executing:

restate invocations list --zombie

restate sql --json "select * from sys_invocation where pinned_deployment_id = 'my_deployment_id';"

curl localhost:9070/query --json '{ "query" : "select * from sys_invocation where pinned_deployment_id = 'my_deployment_id';" }'

For the SQL queries, you need to know the deployment ID of the deployment that was forcefully removed.

Inspecting application state

With the CLI

To retrieve the state of a specific service and service key, do:

restate kv get <SERVICE_NAME> <SERVICE_KEY>

Example output:

🤖 State:
―――――――――

service  counter
key      bob

KEY   VAL
seen  8

If the values are not JSON-encoded UTF-8 strings, then it is also possible to use the --binary flag, and get the value as base64 encoded string.

With SQL queries

You can query the application state via the state table.

restate sql --json "select * from state where service_name = 'test.MyServiceName' and service_key = 'myKey';"

curl localhost:9070/query --json '{ "query" : "select * from state where service_name = 'test.MyServiceName' and service_key = 'myKey';" }'

If your state value is a regular string, then you can access its content in the column value_utf8. To retrieve the state of a specific service name, service key and state key, do:

restate sql --json "select * from state where service_name = 'MyServiceName' and service_key = 'myKey' and key = 'myStateKey';"

curl localhost:9070/query --json '{ "query" : "select * from state where service_name = 'MyServiceName' and service_key = 'myKey' and key = 'myStateKey';" }'

The state key is the name you used to store the state with the SDK. For example, the code snippet ctx.set("count", 1) stores 1 under the key count.

Joining the sys_invocation and state table

To join the sys_invocation and state table:

restate sql --json "select * from sys_invocation JOIN state on sys_invocation.target_service_name = state.service_name and sys_invocation.target_service_key = state.service_key;"

curl localhost:9070/query --json '{ "query" : "select * from sys_invocation JOIN state on sys_invocation.target_service_name = state.service_name and sys_invocation.target_service_key = state.service_key;" }'

Edit application state

You can edit the application state either via the state tab of the UI or via the CLI:

restate kv edit <SERVICE_NAME> <SERVICE_KEY>

This command opens your default editor (as configured in the cli env). It sends the new state values back to the runtime to be applied. Use --binary if the values are not JSON-encoded UTF-8 strings. In this case, you need to decode the base64-encoded string, and encode it back to base64 after editing. Use --plain to retrieve the state as a JSON object. This can be useful in combination with tools like jq for example:

restate kv get counter bob --plain | jq '.seen'

If during the editing of the state with the CLI, an invocation changed the state as well, then the edit of the CLI will not take affect. If you want the CLI state edit to be applied even if the state has changed in the meantime, then use the --force flag.

An example on how to edit the K/V state of the service counter for the key bob:

SDKs

Services

Restate Cloud

Restate BYOC

Self-hosted Restate

References

SQL over the data in Restate

Inspecting invocations

Inspecting application state

With the CLI

With SQL queries

Edit application state

​SQL over the data in Restate

​Inspecting invocations

​Inspecting application state

​With the CLI

​With SQL queries

​Edit application state

SQL over the data in Restate

Inspecting invocations

Inspecting application state

With the CLI

With SQL queries

Edit application state