Metadata Storage

Restate stores cluster configuration data in a dedicated metadata store. This page explains the available metadata backend options, and how to choose and configure the right one for your deployment.

To understand the terminology used on this page, it might be helpful to read through the architecture reference.

What is Metadata?

Metadata in Restate includes critical cluster coordination information:

Cluster membership: Which nodes are part of the cluster and their roles
Partition assignments: How partitions are distributed across worker nodes
Service deployments: Registered service endpoints and their versions
Log configuration: How the replicated log is configured and which nodes participate

The metadata store is essential for cluster operation. While metadata is updated relatively infrequently, its availability is crucial for cluster health.

Choose your metadata provider during initial deployment based on your infrastructure and requirements. While migration between providers is possible from replicated to external stores, this operation requires a maintenance window (downtime) and careful execution.

Metadata Storage Providers

Restate supports four metadata storage providers, each suited to different deployment scenarios:

Replicated (Default) - Built-in Raft-based consensus metadata server
Etcd - External etcd cluster integration
Amazon S3 - We strongly recommend against using other S3-compatible stores for metadata

When to Use Each Provider

Replicated (Default)

The replicated metadata server uses Raft consensus to provide a highly-available metadata store built directly into Restate nodes. Use cases:

Standard production deployments
On-premises or multi-cloud environments
Deployments where you want minimal external dependencies
Environments where you already run stateful services

Requirements:

Nodes must run the metadata-server role
Persistent storage for metadata server nodes
Odd number of metadata-server nodes for quorum (typically 3 or 5)

Trade-offs:

✅ No external dependencies - everything runs within Restate
✅ Battle-tested Raft consensus for strong consistency
✅ Works in any environment (cloud, on-prem, hybrid)
⚠️ Needs careful attention to node placement for fault tolerance
⚠️ Requires manual provisioning after majority of nodes are up to avoid split brain (other backends safely support auto-provisioning in a multi-node cluster)

Etcd

Use an external etcd cluster as the metadata store. This is ideal if you already operate etcd infrastructure or prefer to centralize metadata management. Use cases:

Organizations with existing etcd infrastructure
Environments with dedicated etcd operations expertise
Multi-tenant deployments with centralized control

Requirements:

External etcd cluster (v3.x) must be available
Network connectivity between Restate nodes and etcd
etcd cluster properly configured for production use

Trade-offs:

✅ Leverage existing etcd infrastructure
✅ Separate metadata concerns from Restate node management
✅ Allows sharing metadata across clusters
⚠️ Adds external dependency that must be separately managed
⚠️ Network latency to etcd can affect cluster operations
⚠️ Requires etcd operations expertise

Object Store (Amazon S3)

Use Amazon S3 as a metadata store. This provides a lightweight alternative to running metadata servers while keeping all data in object storage. Use cases:

AWS deployments with existing S3 infrastructure
Minimizing stateful components
Environments optimizing for S3 as primary storage

Requirements:

Amazon S3 bucket (not S3-compatible alternatives)
IAM permissions for S3 operations
Very low-latency access to S3 (same region recommended)

Trade-offs:

✅ No metadata servers to manage
✅ Integrates with existing S3 buckets
✅ Simple backup and recovery
⚠️ Only Amazon S3 is tested and supported (notably, MinIO is not supported for metadata)
⚠️ Tied to a single region!
⚠️ Network latency to S3 directly affects cluster operations
⚠️ Must ensure very low-latency access (typically same region)

For object store metadata: Only Amazon S3 is currently tested and supported. Using S3-compatible alternatives is not supported and may lead to correctness issues. MinIO in particular is known to be incompatible as a metadata backend.

Configuration Examples

Replicated (Default)

This is the default configuration for multi-node clusters. Each node needs to know about all metadata server addresses.

restate.toml

# Node configuration - each node needs these settings
node-name = "node-1"
cluster-name = "my-cluster"

# Enable the metadata-server role on this node
roles = [
    "metadata-server",
    "worker",
    "log-server",
    "admin",
    "http-ingress"
]

# List all metadata server nodes (all nodes need this)
[metadata-client]
type = "replicated"
addresses = [
    "http://node-1:5122/",
    "http://node-2:5122/",
    "http://node-3:5122/"
]

For fault tolerance, run metadata-server on an odd number of nodes (typically 3 or 5). To tolerate n node failures, you need at least 2n + 1 metadata server nodes.

Etcd

Configure Restate to use an external etcd cluster:

restate.toml

# No metadata-server role needed
roles = [
    "worker",
    "log-server",
    "admin",
    "http-ingress"
]

[metadata-client]
type = "etcd"
# Etcd cluster addresses (host:port format)
addresses = [
    "etcd-1.example.com:2379",
    "etcd-2.example.com:2379",
    "etcd-3.example.com:2379"
]

# Optional: Adjust timeouts if needed
connect-timeout = "5s"
keep-alive-interval = "10s"

Object Store (Amazon S3)

Configure Restate to use Amazon S3 for metadata storage:

restate.toml

# No metadata-server role needed
roles = [
    "worker",
    "log-server",
    "admin",
    "http-ingress"
]

[metadata-client]
type = "object-store"
# S3 bucket and optional prefix
path = "s3://restate-metadata-bucket/cluster-prod"

# Authentication options
aws-profile = "my-profile"
# Or use explicit credentials:
# aws-access-key-id = "..."
# aws-secret-access-key = "..."
# aws-session-token = "..."

aws-region = "us-east-1"

# Optional: For testing with local S3-compatible storage
# aws-endpoint-url = "http://localhost:9000"
# aws-allow-http = true

Object store metadata requires low-latency access to S3. Deploy your Restate cluster in the same AWS region as your S3 bucket to minimize latency impact on cluster operations.

Migrating Between Providers

Restate supports migrating metadata from the replicated provider to external providers (etcd, or object-store) using the restatectl metadata migrate command. This allows you to change metadata providers without losing cluster state.

Important limitations:

Migration only supports moving FROM replicated TO external providers (not the reverse)
Migration requires all cluster nodes to be restarted in migration mode
Plan for a maintenance window as the cluster will not process invocations during migration

Migration Process

Prerequisites

Before starting the migration:

Target metadata store must be accessible and ready:
- Etcd: Cluster running and accessible
- Object Store: S3 bucket created and accessible
Prepare target configuration:
- Have your target [metadata-client] configuration ready in TOML format
- Test connectivity from Restate nodes to the target store
Plan for downtime:
- All nodes will need to be restarted in migration mode
- Invocation processing will be paused during migration

Step-by-Step Migration

Prepare target metadata store

For Etcd:

# Ensure your etcd cluster is running and accessible
etcdctl endpoint health

For Object Store:

# Ensure S3 bucket exists and is accessible
aws s3 ls s3://restate-metadata-bucket/

Restart all nodes in migration mode

Stop all cluster nodes and restart them with the --metadata-migration-mode flag:

restate --metadata-migration-mode --config-file restate.toml

In migration mode, only the Admin and MetadataServer roles are active. The cluster will not process invocations or serve requests until migration is complete and nodes are restarted normally.

Create target configuration file

Create a TOML file (e.g., target-metadata.toml) with your target metadata client configuration:For Etcd:

target-metadata.toml

[metadata-client]
type = "etcd"
addresses = [
    "etcd-1.example.com:2379",
    "etcd-2.example.com:2379",
    "etcd-3.example.com:2379"
]

For Object Store:

target-metadata.toml

[metadata-client]
type = "object-store"
path = "s3://restate-metadata-bucket/cluster-prod"
aws-profile = "my-profile"
aws-region = "us-east-1"

Run migration command

Execute the migration using restatectl:

restatectl metadata migrate --toml target-metadata.toml

By default, the migration will fail if any keys already exist in the target store. To override existing keys:

restatectl metadata migrate --toml target-metadata.toml --force

Use --force with caution. It will overwrite any existing metadata in the target store, which could cause issues if the target is shared with other clusters.

The migration command will:

Read all metadata from the replicated store
Write it to the target store
Verify the migration succeeded
Display next steps

Expected output:

✅ Metadata migration completed

1. Please make sure to update all nodes config with
   [metadata-client]
   type = "etcd"
   addresses = [
       "etcd-1.example.com:2379",
       "etcd-2.example.com:2379",
       "etcd-3.example.com:2379"
   ]

2. Restart all nodes without the --metadata-migration-mode

Update node configurations

Update the [metadata-client] section in all node configuration files with the target configuration:

restate.toml

# Remove or comment out metadata-server role if present
roles = [
    "worker",
    "log-server",
    "admin",
    "http-ingress"
]

# Replace [metadata-client] section with target configuration
[metadata-client]
type = "etcd"
addresses = [
    "etcd-1.example.com:2379",
    "etcd-2.example.com:2379",
    "etcd-3.example.com:2379"
]

Also remove or comment out the [metadata-server] section if present, as it’s no longer needed with external metadata providers.

Restart all nodes normally

Stop all nodes running in migration mode and restart them with their updated configurations:

restate --config-file restate.toml

Verify the cluster is healthy:

restatectl status
restatectl nodes list

Troubleshooting Migration

Migration fails with “target keys exist”:

The target store already contains metadata keys
Either use --force to override, or clean the target store first
Verify you’re not accidentally sharing a metadata store with another cluster

Migration command times out:

Check that nodes are actually running in migration mode
Verify network connectivity to both source and target stores
Check Restate server logs for errors

Cluster won’t start after migration:

Verify all nodes have the updated [metadata-client] configuration
Verify etcd cluster is accessible if using etcd
Check server logs for connection errors

Cluster Configuration: Understanding Restate cluster deployments
Architecture Reference: Deep dive into metadata architecture
Server Configuration: Complete server configuration guide
Configuration Reference: Detailed metadata client options
AWS Deployment Guide: Deploying Restate on AWS
Kubernetes Deployment: Deploying Restate on Kubernetes

​What is Metadata?

​Metadata Storage Providers

​When to Use Each Provider

​Replicated (Default)

​Etcd

​Object Store (Amazon S3)

​Configuration Examples

​Replicated (Default)

​Etcd

​Object Store (Amazon S3)

​Migrating Between Providers

​Migration Process

​Prerequisites

​Step-by-Step Migration

​Troubleshooting Migration

​Related Documentation

What is Metadata?

Metadata Storage Providers

When to Use Each Provider

Replicated (Default)

Etcd

Object Store (Amazon S3)

Configuration Examples

Replicated (Default)

Etcd

Object Store (Amazon S3)

Migrating Between Providers

Migration Process

Prerequisites

Step-by-Step Migration

Troubleshooting Migration

Related Documentation