> ## Documentation Index
> Fetch the complete documentation index at: https://docs.restate.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Migrating Snapshot Storage Backend

> Move a multi-node Restate cluster from one snapshot destination to another.

This guide describes how to migrate a multi-node Restate cluster from one snapshot storage backend to another (for example, between different buckets/prefixes, or from MinIO to GCS).

The migration temporarily increases partition replication to ensure every node hosts every partition before snapshots are disabled. This prevents trim-gap failures during rolling restarts. The migration leverages Restate's `worker.durability-mode` configuration option to prevent any log trimming during the transition, ensuring no data loss even if the old snapshots become unavailable before new ones are created.

<Info>
  **Prerequisites:**

  * Restate server version **1.6 or later** (required for `worker.durability-mode` configuration)
  * Rolling restart capability for your cluster
  * Access to both old and new snapshot storage backends during migration
  * Capacity to temporarily run each partition on every worker node (partition replication = cluster size)
  * [`restatectl`](/server/clusters#controlling-clusters-with-restatectl) CLI configured to communicate with your cluster
</Info>

<Steps>
  <Step title="Record current replication settings">
    Capture the current cluster replication settings so you can restore them later:

    ```shell theme={null}
    restatectl config get
    ```

    Note the current **Partition replication** value (for example `{node: 2}`).
  </Step>

  <Step title="Temporarily increase partition replication to all nodes">
    Set partition replication to your cluster size `N` (the number of worker nodes). This ensures every node has a local copy of every partition before you disable snapshots.

    ```shell theme={null}
    restatectl config set --partition-replication N
    ```

    Do **not** use `--replication` here unless you also want to increase log replication.

    <Info>
      **Why increase partition replication?**

      Without this step, when the cluster controller reconfigures partition replica sets during rolling restarts, some nodes may be unable to serve a given partition depending on their prior local partition store state, the log trim point, and available snapshots. With partition replication matching cluster size, every node will maintain a warm replica of every partition and is able to resume without the need for a new snapshot on restart, provided the log was not trimmed during its downtime.
    </Info>
  </Step>

  <Step title="Wait for replicas to catch up">
    Wait until every partition has a replica on every node and followers have no lag:

    ```shell theme={null}
    restatectl partitions list
    ```

    Verify that:

    * Each partition ID appears `N` times (once per node)
    * All rows show `LSN-LAG` of `0` (or consistently near `0`)

    For example, with 8 partitions and 3 nodes, you should see 24 rows total.
  </Step>

  <Step title="Disable automatic log trimming">
    Roll out a configuration update to disable automatic snapshots and switch to conservative durability mode:

    ```toml restate.toml theme={null}
    [worker.snapshots]
    # Disable automatic snapshots by removing/commenting destination
    # destination = "s3://old-bucket/prefix"

    [worker]
    # Use the strictest mode - requires BOTH replicas AND snapshots for trim
    # When snapshot destination is not set, this halts all log trimming
    durability-mode = "snapshot-and-replica-set"
    ```

    This effectively disables both snapshotting and log trimming. The system will log a warning every 60 seconds: *"Detected cluster environment with no snapshot repository configured. Automatic log trimming is disabled..."* - this is expected during the migration.

    Perform a rolling restart of all cluster nodes with the new configuration. Restart one node at a time, waiting for it to rejoin and partitions to become active before proceeding to the next node.

    <Tip>
      **Live traffic during migration**

      With partition replication matching cluster size, rolling restarts have minimal impact on incoming ingress requests. Make sure to use [idempotency keys](/foundations/invocations#idempotency) with incoming requests to make retries safe and reduce error rates further.
    </Tip>
  </Step>

  <Step title="Verify that log trimming has stopped">
    Check the cluster status to confirm all partitions are active:

    ```shell theme={null}
    restatectl partitions list
    ```

    You should see all partitions with the `ARCHIVED` column empty or unchanged:

    ```
    ID  NODE     MODE    STATUS  EPOCH  APPLIED  DURABLE  ARCHIVED  LSN-LAG  UPDATED
    0   N1:1     Leader  Active  5      1234     1234     -         0        2s ago
    1   N2:1     Leader  Active  5      5678     5678     -         0        1s ago
    ...
    ```

    The `ARCHIVED` column shows `-` (due to no known snapshot). This is expected.

    The applied LSN should increase over time if there is cluster activity but the archived LSN should remain `-`:
  </Step>

  <Step title="Configure new snapshot repository">
    Roll out a configuration update with the new snapshot destination:

    ```toml restate.toml theme={null}
    [worker.snapshots]
    destination = "s3://new-bucket/prefix"   # New repository

    [worker]
    # Use conservative settings
    durability-mode = "snapshot-and-replica-set"
    trim-delay-interval = "24h"
    ```

    Perform a rolling restart of all cluster nodes (one at a time, verifying health between each).
  </Step>

  <Step title="Create snapshots in the new repository">
    Trigger manual snapshots for all partitions to populate the new repository immediately:

    ```shell theme={null}
    restatectl snapshot create
    ```

    You should see output confirming each partition was snapshotted:

    ```
    Snapshot created for partition 0: snap_15GSJBOfxk3x8k1CfPwfxrb (log 0 @ LSN >= 49622035)
    Snapshot created for partition 1: snap_2xHJKLMnop4y9z2DgQwgAbc (log 1 @ LSN >= 49622040)
    ...
    ```
  </Step>

  <Step title="Verify snapshots in the new storage backend">
    Check that snapshots exist in the new storage backend. For S3:

    ```shell theme={null}
    aws s3 ls s3://new-bucket/prefix/ --recursive | head -20
    ```

    Each partition should have a `latest.json` file and a snapshot directory:

    ```
    prefix/0/latest.json
    prefix/0/lsn_00000000000000860864-snap_13yBpep1H1jKGAzHhqkmCyt/...
    prefix/1/latest.json
    prefix/1/lsn_00000000000000860870-snap_2xHJKLMnop4y9z2DgQwgAbc/...
    ...
    ```

    Confirm the archived LSN column now shows the snapshot LSN values:

    ```shell theme={null}
    restatectl partitions list
    ```

    Expected output:

    ```
    ID  NODE     MODE    STATUS  EPOCH  APPLIED  DURABLE  ARCHIVED  LSN-LAG  UPDATED
    0   N1:1     Leader  Active  5      1250     1250     1234      0        2s ago
    1   N2:1     Leader  Active  5      5700     5700     5678      0        1s ago
    ```
  </Step>

  <Step title="Restore partition replication">
    After the new snapshot repository is verified, restore the original partition replication value you recorded earlier:

    ```shell theme={null}
    restatectl config set --partition-replication <previous-value>
    ```
  </Step>

  <Step title="Restore normal operations">
    Roll out a configuration update with production settings:

    ```toml restate.toml theme={null}
    [worker]
    # Return to balanced mode (recommended for production)
    durability-mode = "balanced"
    ```

    Perform a rolling restart of all cluster nodes (one at a time, verifying health between each).
  </Step>

  <Step title="Verify normal operations">
    Check that the cluster status is healthy:

    ```shell theme={null}
    restatectl status --extra
    ```

    All nodes should be healthy and all partitions active with no warnings.

    Confirm log trimming has resumed:

    ```shell theme={null}
    restatectl log list
    ```

    The trim point should gradually increase as durability conditions are met.
  </Step>

  <Step title="Clean up old snapshots">
    After confirming the cluster is migrated to the new snapshot backend:

    1. Remove old snapshots
    2. Revoke access to the old storage backend
  </Step>
</Steps>

## Durability mode reference

| Mode                       | Description                                        | Use case                                       |
| -------------------------- | -------------------------------------------------- | ---------------------------------------------- |
| `balanced`                 | Requires snapshot AND at least one replica flushed | Production default (when snapshots configured) |
| `snapshot-and-replica-set` | Requires snapshot AND all replicas flushed         | Migration phase (strictest)                    |
| `snapshot-only`            | Requires only snapshot, ignores replicas           | Special cases                                  |
| `replica-set-only`         | Requires all replicas flushed, ignores snapshots   | Default without snapshots                      |
| `none`                     | Disables automatic durability tracking             | Testing only                                   |

## Rollback plan

If you encounter issues during migration, the rollback procedure depends on how far you've progressed:

**During steps 1-3** (before log trimming is disabled):

No destructive changes have been made. Simply restore partition replication to the original value:

```shell theme={null}
restatectl config set --partition-replication <original-value>
```

**During steps 4-5** (log trimming disabled, no new snapshots yet):

Restore the original configuration pointing to the old snapshot repository, perform a rolling restart, then restore partition replication:

```shell theme={null}
restatectl config set --partition-replication <original-value>
```

**During steps 6-8** (configuring new repository, creating snapshots):

If no log trimming has occurred since the original repository was disabled, you can safely discard the new repository and revert to the original configuration. Restore partition replication after the rollback.

**After step 9** (partition replication restored, normal operations):

If logs have been trimmed based on snapshot LSNs published to the new repository, you must follow the same migration process to return to the original destination: disable log trimming, update snapshot destination, create and verify snapshots, then re-enable log trimming.

## See also

* [Configuring automatic snapshotting](/server/snapshots#configuring-automatic-snapshotting)
* [Controlling clusters with restatectl](/server/clusters#controlling-clusters-with-restatectl)