Skip to main content

Introspection

Restate exposes information on invocations and application state via its CLI and Introspection SQL API. You can use this to gain insight into the status of invocations and the service state that is stored.

This can be useful for troubleshooting. For example, a handler might be blocked for a specific key, and you want to kill the invocation that is blocking it, but you don't know the invocation ID. Or you want to check what is currently stored in the state of a service.

Setting up introspection​

The CLI is the easiest way to get started with introspection.
Have a look at the installation guide for the CLI.

Inspecting invocations​

To get an overview of invocation inspection with the CLI, do:

restate inv --help
info

The CLI only allows you to interact with active invocations. Active invocations are invocations that haven't completed yet and are either invoked or suspended.

Below, we list some common queries that you can execute to inspect invocations.

Listing ongoing invocations​

restate inv ls

Retrieving the status of an invocation​

restate inv get <INVOCATION_ID>

Check the status field in the invocation information in the output.

Inspecting the invocation journal​

restate inv get <INVOCATION_ID>

You see the journal printed in the output.

Inspecting invocation retries​

To have a look at the invocations that are currently in a retry loop, you can execute:

restate inv ls --status backing-off

Listing invocations that are blocking a specific key​

You can retrieve the invocation ID (id) that is currently blocking a service-key combination via:

restate inv ls --key <KEY> --service <SERVICE>

You can then use the invocation ID to kill the invocation.

Checking the last time an invocation was modified​

restate inv get <INVOCATION_ID>

Check the Modified at field in the invocation information in the output.

Checking how an invocation was triggered​

To find out if an invocation was triggered via the ingress or by another service:

restate inv get <INVOCATION_ID>

You can check this at the root of the tree in the invocation progress.

Retrieving the trace ID of an invocation​

restate inv get <INVOCATION_ID>

Afterwards, you can use this trace ID to search for spans in Jaeger.

Listing inactive invocations​

To list the oldest invocations that are not making progress:

restate inv ls --oldest-first --status pending,backing-off,suspended

Listing zombie invocations​

Zombie invocations are invocations that are pinned to a specific deployment but that deployment was forcefully removed. You can list them by executing:

restate inv ls --zombie

Inspecting application state​

To retrieve the state of a specific service and service key , do:

restate state get <SERVICE_NAME> <SERVICE_KEY>

Instead of state, you can use the alias kv.

For example:

restate kv get counter bob

This will output the state of the service counter with the key bob:

πŸ€– State:
―――――――――

service counter
key bob

KEY VAL
seen 8
info

If the values are not JSON-encoded UTF-8 strings, then it is also possible to use the --binary flag, and get the value as base64 encoded string.

To edit application state​

To edit the application state, do:

restate state edit <SERVICE_NAME> <SERVICE_KEY>

This command opens your default editor (as configured in the cli env). It sends the new state values back to the runtime to be applied.

--binary flag

Use --binary if the values are not JSON-encoded UTF-8 strings. In this case, you need to decode the base64-encoded string, and encode it back to base64 after editing.

--plain flag

Use --plain to retrieve the state as a JSON object. This can be useful in combination with tools like jq for example:

restate kv get counter bob --plain | jq '.seen'
Editing state during ongoing invocations

If during the editing of the state with the CLI, an invocation changed the state as well, then the edit of the CLI will not take affect. If you want the CLI state edit to be applied even if the state has changed in the meantime, then use the --force flag.

An example on how to edit the K/V state of the service counter for the key bob:

restate state edit counter bob

After changing the state in our editor, we get:

ℹ️  About to write the following state :
―――――――――――――――――――――――――――――――――――――――


service counter
key bob
force? false
binary? false

KEY VAL
seen 8

βœ” Are you sure? Β· yes


Enqueued successfully for processing