Kubernetes

This page describes how to deploy Restate and Restate services on Kubernetes.

Deploying Restate Server on K8S

The recommended Kubernetes deployment strategy is a one-replica StatefulSet. We recommend installing Restate in its own namespace. The easiest way to do this is with our Helm chart:

helm install restate oci://ghcr.io/restatedev/restate-helm --namespace restate --create-namespace

Restate Kubernetes Operator

If you want to run multiple Restate clusters in Kubernetes, or want advanced functionality like online volume expansion and network policies, you can also use the Restate Operator. Details of how to install it and deploy a cluster can be found in the README.

Deploying Restate services on K8S

Service deployments can be deployed like any Kubernetes service; a Deployment of more than one replica is generally appropriate. The deployment should be load balanced at L7 if you want multiple service deployment pods. Native Kubernetes ClusterIP load balancing will lead to the Restate binary sending all requests to a single pod, as HTTP2 connections are aggressively reused. This is fine for local testing, but in production an approach must be found. If your infrastructure already has an approach for L7 load balancing services, you can use the same approach here. Otherwise, some recommended approaches are detailed below:

Infrastructure	Approach
Knative	Use Knative for autoscaling, scale to zero and the integrated L7 load balancing
Istio / LinkerD	Ensure sidecar is injected into Restate pod and all service pods
Cilium	Ensure Cilium is installed with loadBalancer.l7.backend=envoy, and annotate service pods with service.cilium.io/lb-l7=enabled
Minikube	For local development it's likely not worth worrying about; see below
Any	Use an envoy sidecar on the Restate pod; see below

Local Kubernetes development

A simple deployment setup (eg, for local use with Minikube) with a single pod in Kubernetes is as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: service
  template:
    metadata:
      labels:
        app: service
    spec:
      containers:
        - name: service
          image: path.to/yourrepo:yourtag
          env:
            - name: PORT
              value: "9080"
          ports:
            - containerPort: 9080
              name: http2
---
apiVersion: v1
kind: Service
metadata:
  name: service
spec:
  selector:
    app: service
  ports:
    - port: 9080
      name: http2
  type: ClusterIP

L7 load balancing is not needed when there is only one pod, so it's acceptable to use a normal ClusterIP Service.

Knative

Restate supports Knative services. Knative allows scaling to zero when there are no in-flight invocations and automatically configures an L7 load balancer. There are no special requirements to deploy a service deployment container with Knative:

$ kn service create service-name --port h2c:9080 --image path.to/yourrepo:yourtag

Or using the YAML manifest:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: service-name
spec:
  template:
    spec:
      containers:
        - image: path.to/yourrepo:yourtag
          ports:
            - name: h2c
              containerPort: 9080

The service will be accessible at http://<service-name>.<namespace>.svc.

By default Knative exposes the service through the Ingress. This is not required by Restate, and you can disable this behavior adding the argument --cluster-local to the aforementioned creation command.

Simple L7 load balancing with an Envoy sidecar

A simple approach to L7 load balancing is to set up an Envoy sidecar in the Restate pod which acts as a transparent HTTP proxy which will resolve and L7 load balance to Kubernetes Services based on their DNS name. For this to work, Services must be deployed as Headless, ie without a ClusterIP. This is achieved by specifying clusterIP: None in a type: ClusterIP Service.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: restate
  namespace: restate
spec:
  # ...
  template:
    spec:
      containers:
        - name: restate
          # ...
          env:
            - name: HTTP_PROXY
              value: http://127.0.0.1:10001
          # ...
        - name: envoy
          image: envoyproxy/envoy:distroless-v1.27-latest
          volumeMounts:
            - name: envoy-config
              mountPath: /etc/envoy
      # ...
      volumes:
        - name: envoy-config
          configMap:
            name: envoy-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-config
data:
  envoy.yaml: |
    static_resources:
      listeners:
        - name: proxy
          address:
            socket_address:
              address: "127.0.0.1"
              port_value: 10001
          filter_chains:
            - filters:
                - name: envoy.filters.network.http_connection_manager
                  typed_config:
                    '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                    stat_prefix: egress_http
                    stream_idle_timeout: 0s
                    route_config:
                      name: local_route
                      virtual_hosts:
                        - name: http
                          domains: ['*']
                          routes:
                            - match: {prefix: /}
                              route:
                                cluster: dynamic_forward_proxy_cluster
                                timeout: 0s
                    http_filters:
                      - name: envoy.filters.http.dynamic_forward_proxy
                        typed_config:
                          '@type': type.googleapis.com/envoy.extensions.filters.http.dynamic_forward_proxy.v3.FilterConfig
                          sub_cluster_config: {}
                      - name: envoy.filters.http.router
                        typed_config:
                          "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
      clusters:
        - name: dynamic_forward_proxy_cluster
          connect_timeout: 1s
          lb_policy: CLUSTER_PROVIDED
          # assume http2 in upstream
          http2_protocol_options:
            connection_keepalive:
              interval: 40s
              timeout: 20s
          cluster_type:
            name: envoy.clusters.dynamic_forward_proxy
            typed_config:
              '@type': type.googleapis.com/envoy.extensions.clusters.dynamic_forward_proxy.v3.ClusterConfig
              sub_clusters_config:
                lb_policy: ROUND_ROBIN