Restate supports the following tracing features:
- Runtime execution tracing per invocation
- Exporting traces to OTLP-compatible systems (e.g. Jaeger)
- Correlating parent traces of incoming gRPC/Connect HTTP requests, using the W3C TraceContext specification.
Setting up OTLP exporter
Set up the OTLP exporter by pointing the configuration entry
observability.tracing.endpoint to your trace collector.
The exporter sends OTLP trace data via gRPC.
Exporting OTLP traces to Jaeger
Jaeger accepts OTLP trace data via gRPC on port
Start Jaeger with the environment variable
COLLECTOR_OTLP_ENABLED=true to enable OTLP.
Refer to the Jaeger documentation for more details on how to deploy Jaeger.
Configure the tracing endpoint in Restate as a fully specified URL:
You can configure a span/event filter in a similar fashion to the log filter setting the
observability.tracing.filter configuration entry.
Try it out locally
To experiment with this feature locally, run the Jaeger container via:
docker run -d --name jaeger \
-e COLLECTOR_OTLP_ENABLED=true \
-p 4317:4317 \
-p 16686:16686 \
Then launch Restate with the tracing endpoint defined as an environment variable:
docker run --name restate_dev --rm -d -e RESTATE_OBSERVABILITY__TRACING__ENDPOINT=http://localhost:4317 --network=host docker.io/restatedev/restate:0.7
docker run --name restate_dev --rm -d -e RESTATE_OBSERVABILITY__TRACING__ENDPOINT=http://host.docker.internal:4317 -p 8080:8080 -p 9070-9071:9070-9071 docker.io/restatedev/restate:0.7
Go to the Jaeger UI at http://localhost:16686.
If you now spin up your Restate services and send requests to them, you will see the traces appear in the Jaeger UI.
An example from the shopping cart example:
Setting up Jaeger file exporter
If you can't configure a Jaeger agent, you can export traces by writing them to files, using the Jaeger JSON format. To do so, set up the configuration entry
observability.tracing.json_file_export_path pointing towards the path where trace files should be written.
You can import the trace files using the Jaeger UI:
The traces contain detailed information about the context calls that were done during the invocation (e.g. sleep, one-way calls, interaction with state):
ingress_invoke spans show when the gRPC/Connect HTTP request was received by Restate. The
invoke span beneath it shows when Restate invoked the service deployment to process the request.
The tags of the spans contain the metadata of the context calls (e.g. call arguments, invocation id).
When a service invokes another service, the child invocation is linked automatically to the parent invocation, as you can see in the image. Note that the spans of one-way calls are shown as separate traces. The parent invocation only shows that the one-way call was scheduled, not its entire tracing span. To see this information, search for the trace of the one-way call by filtering on the invocation id tag:
Traces export attributes and tags that correlate the trace with the service and/or invocation. For example, in the Jaeger UI, you can filter on the invocation id (
restate.invocation.id) or any other tag: