Introspection

Restate exposes information on invocations and application state via its CLI and Introspection SQL API. You can use this to gain insight into the status of invocations and the service state that is stored.

This can be useful for troubleshooting. For example, a handler might be blocked for a specific key, and you want to kill the invocation that is blocking it, but you don't know the invocation ID. Or you want to check what is currently stored in the state of a service.

Setting up introspection

CLI

The CLI is the easiest way to get started with introspection.
Have a look at the installation guide for the CLI.

psql

Restate exposes the following SQL tables, queryable via the psql client:

• sys_invocation table to inspect invocations
• sys_inbox table to inspect queue of pending invocations
• sys_keyed_service_status table to inspect the status of a Virtual Object
• sys_journal table to inspect the invocations' journal
• sys_service table to inspect the registered services
• sys_deployment table to inspect service deployments
• sys_idempotency table to inspect idempotency keys
• state table to inspect application state

You can find the schema of each of the tables in the references.

Install the psql client.

Connect with psql to Restate:

psql -h localhost -p 9071

You can change the default port 9071 by supplying the -p argument.

To retrieve the schema of the any of the tables, connect the psql client to Restate and execute:

describe <TABLE_NAME>;

For example:

describe sys_invocation;

The Restate Introspection SQL API has been implemented based on DataFusion and supports standard SQL syntax.

Listing ongoing invocations

CLI

restate inv ls

info

The CLI only allows you to interact with active invocations. Active invocations are invocations that haven't completed yet and are either invoked or suspended.

psql

select * from sys_invocation;

info

The sys_invocation table only contains entries for active invocations. Active invocations are invocations that haven't completed yet and are either invoked or suspended.

Retrieving the status of an invocation

CLI

restate inv get <INVOCATION_ID>

Check the status field in the invocation information in the output.

psql

select target_service_name, target_handler_name, status from sys_invocation where id = '<INVOCATION_ID>';

The status is either:

• pending: enqueued waiting for its turn
• ready: ready to be processed, but not yet running
• running: actively processing
• backing-off: retrying due to a failure
• suspended: waiting on some external input (e.g. request-response call, awakeable, sleep, ...)
• completed: completed (this is shown only for idempotent invocations)

Inspecting the invocation journal

CLI

restate inv get <INVOCATION_ID>

You see the journal printed in the output.

psql

You can retrieve the journal of an invocation by querying the sys_journal table:

select * from sys_journal where id = '<INVOCATION_ID>';

If you don't know the invocation id of your request, you can find it in the sys_invocation table as described above

Inspecting invocation retries

To have a look at the invocations that are currently in a retry loop, you can execute:

CLI

restate inv ls --status backing-off

psql

select * from sys_invocation where retry_count > 1;

The metadata around these attempts is not committed durably to storage, but is held in memory in the invoker component. This metadata can be found in the sys_invocation table.

Listing invocations that are blocking a specific key

You can retrieve the invocation ID that is currently blocking a service-key combination via:

CLI

restate inv ls --key <KEY> --service <SERVICE>

psql

select invocation_id from sys_keyed_service_status where service_name = 'test.MyServiceName' and service_key = 'myKey';

You can then use the invocation ID to kill the invocation.

Checking the last time an invocation was modified

CLI

restate inv get <INVOCATION_ID>

Check the Modified at field in the invocation information in the output.

psql

select modified_at from sys_invocation where id = '<INVOCATION_ID>';

This includes any modification to the row in the table (e.g. when the service last switched its status from invoked to suspended, or when the last journal entry was added).

Checking how an invocation was triggered

To find out if an invocation was triggered via the ingress or by another service:

CLI

restate inv get <INVOCATION_ID>

You can check this at the root of the tree in the invocation progress.

psql

select invoked_by, invoked_by_service_name, invoked_by_id from sys_invocation where id = '<INVOCATION_ID>';

The invoked_by field contains either ingress or service. If the invocation was triggered by another service, then the fields invoked_by_service_name and invoked_by_id will supply more information about the invoking service.

Retrieving the trace ID of an invocation

CLI

restate inv get <INVOCATION_ID>

psql

select trace_id from sys_invocation where id = '<INVOCATION_ID>';

Afterwards, you can use this trace ID to search for spans in Jaeger.

Listing inactive invocations

CLI

To list the oldest invocations that are not making progress:

restate inv ls --oldest-first --status pending,backing-off,suspended

psql

To retrieve all invocations that have not seen any activity for more than 1 hour:

select * from sys_invocation where to_timestamp(modified_at) <= now() - interval '1' hour;

Listing zombie invocations

Zombie invocations are invocations that are pinned to a specific deployment but that deployment was forcefully removed. You can list them by executing:

CLI

restate inv ls --zombie

psql

This is not possible via only psql.

If you know the ID of the deployment that got forcefully removed, you can check if there are zombie invocations pinned to it via:

select * from sys_invocation where pinned_deployment_id = '<DEPLOYMENT_ID>';

Inspecting application state

To retrieve the state of a specific service and service key, do:

CLI

restate state get <SERVICE_NAME> <SERVICE_KEY>

Instead of state, you can use the alias kv.

For example:

restate kv get counter bob

This will output the state of the service counter with the key bob:

🤖 State:

―――――――――

service counter

key bob

KEY VAL

seen 8

info

If the values are not JSON-encoded UTF-8 strings, then it is also possible to use the --binary flag, and get the value as base64 encoded string.

psql

You can query the application state via the state table.

select * from state where service_name = 'test.MyServiceName' and service_key = 'myKey';

Take into account the data types of the included columns

If your state value is a regular string, then you can access its content in the column value_utf8.

To retrieve the state of a specific service name, service key and state key, do:

select * from state where service_name = 'MyServiceName' and service_key = 'myKey' and key = 'myStateKey';

The state key is the name you used to store the state with the SDK. For example, the code snippet ctx.set("count", 1) stores 1 under the key count.

Joining the tables

To join the sys_invocation and state table:

select * from sys_invocation JOIN state on sys_invocation.target_service_name = state.service_name and sys_invocation.target_service_key = state.service_key;

To edit application state

CLI

To edit the application state, do:

restate state edit <SERVICE_NAME> <SERVICE_KEY>

This command opens your default editor (as configured in the cli env). It sends the new state values back to the runtime to be applied.

State as binary

Use --binary if the values are not JSON-encoded UTF-8 strings. In this case, you need to decode the base64-encoded string, and encode it back to base64 after editing.

State as JSON objects

Use --plain to retrieve the state as a JSON object. This can be useful in combination with tools like jq for example:

restate kv get counter bob --plain | jq '.seen'

Editing state during ongoing invocations

If during the editing of the state with the CLI, an invocation changed the state as well, then the edit of the CLI will not take affect. If you want the CLI state edit to be applied even if the state has changed in the meantime, then use the --force flag.

An example on how to edit the K/V state of the service counter for the key bob:

restate state edit counter bob

After changing the state in our editor, we get:

ℹ️ About to write the following state :

―――――――――――――――――――――――――――――――――――――――

service counter

key bob

force? false

binary? false

KEY VAL

seen 8

✔ Are you sure? · yes

Enqueued successfully for processing