Skip to main content

Kubernetes

This page describes how to deploy Restate and Restate services on Kubernetes.

Deploying Restate Server on K8S

The recommended Kubernetes deployment strategy is a one-replica StatefulSet. We recommend installing Restate in its own namespace. The easiest way to do this is with our Helm chart:


helm install restate oci://ghcr.io/restatedev/restate-helm --namespace restate --create-namespace

Restate Kubernetes Operator

If you want to run multiple Restate clusters in Kubernetes, or want advanced functionality like online volume expansion and network policies, you can also use the Restate Operator. Details of how to install it and deploy a cluster can be found in the README.

Deploying Restate services on K8S

Service deployments can be deployed like any Kubernetes service; a Deployment of more than one replica is generally appropriate. The deployment should be load balanced at L7 if you want multiple service deployment pods. Native Kubernetes ClusterIP load balancing will lead to the Restate binary sending all requests to a single pod, as HTTP2 connections are aggressively reused. This is fine for local testing, but in production an approach must be found. If your infrastructure already has an approach for L7 load balancing services, you can use the same approach here. Otherwise, some recommended approaches are detailed below:

InfrastructureApproach
KnativeUse Knative for autoscaling, scale to zero and the integrated L7 load balancing
Istio / LinkerDEnsure sidecar is injected into Restate pod and all service pods
CiliumEnsure Cilium is installed with loadBalancer.l7.backend=envoy, and annotate service pods with service.cilium.io/lb-l7=enabled
MinikubeFor local development it's likely not worth worrying about; see below
AnyUse an envoy sidecar on the Restate pod; see below

Local Kubernetes development

A simple deployment setup (eg, for local use with Minikube) with a single pod in Kubernetes is as follows:


apiVersion: apps/v1
kind: Deployment
metadata:
name: service
spec:
replicas: 1
selector:
matchLabels:
app: service
template:
metadata:
labels:
app: service
spec:
containers:
- name: service
image: path.to/yourrepo:yourtag
env:
- name: PORT
value: "9080"
ports:
- containerPort: 9080
name: http2
---
apiVersion: v1
kind: Service
metadata:
name: service
spec:
selector:
app: service
ports:
- port: 9080
name: http2
type: ClusterIP

L7 load balancing is not needed when there is only one pod, so it's acceptable to use a normal ClusterIP Service.

Knative

Restate supports Knative services. Knative allows scaling to zero when there are no in-flight invocations and automatically configures an L7 load balancer. There are no special requirements to deploy a service deployment container with Knative:


$ kn service create service-name --port h2c:9080 --image path.to/yourrepo:yourtag

Or using the YAML manifest:


apiVersion: serving.knative.dev/v1
kind: Service
metadata:
name: service-name
spec:
template:
spec:
containers:
- image: path.to/yourrepo:yourtag
ports:
- name: h2c
containerPort: 9080

The service will be accessible at http://<service-name>.<namespace>.svc.

By default Knative exposes the service through the Ingress. This is not required by Restate, and you can disable this behavior adding the argument --cluster-local to the aforementioned creation command.

Simple L7 load balancing with an Envoy sidecar

A simple approach to L7 load balancing is to set up an Envoy sidecar in the Restate pod which acts as a transparent HTTP proxy which will resolve and L7 load balance to Kubernetes Services based on their DNS name. For this to work, Services must be deployed as Headless, ie without a ClusterIP. This is achieved by specifying clusterIP: None in a type: ClusterIP Service.


apiVersion: apps/v1
kind: StatefulSet
metadata:
name: restate
namespace: restate
spec:
# ...
template:
spec:
containers:
- name: restate
# ...
env:
- name: HTTP_PROXY
value: http://127.0.0.1:10001
# ...
- name: envoy
image: envoyproxy/envoy:distroless-v1.27-latest
volumeMounts:
- name: envoy-config
mountPath: /etc/envoy
# ...
volumes:
- name: envoy-config
configMap:
name: envoy-config
---
apiVersion: v1
kind: ConfigMap
metadata:
name: envoy-config
data:
envoy.yaml: |
static_resources:
listeners:
- name: proxy
address:
socket_address:
address: "127.0.0.1"
port_value: 10001
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
'@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: egress_http
stream_idle_timeout: 0s
route_config:
name: local_route
virtual_hosts:
- name: http
domains: ['*']
routes:
- match: {prefix: /}
route:
cluster: dynamic_forward_proxy_cluster
timeout: 0s
http_filters:
- name: envoy.filters.http.dynamic_forward_proxy
typed_config:
'@type': type.googleapis.com/envoy.extensions.filters.http.dynamic_forward_proxy.v3.FilterConfig
sub_cluster_config: {}
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
clusters:
- name: dynamic_forward_proxy_cluster
connect_timeout: 1s
lb_policy: CLUSTER_PROVIDED
# assume http2 in upstream
http2_protocol_options:
connection_keepalive:
interval: 40s
timeout: 20s
cluster_type:
name: envoy.clusters.dynamic_forward_proxy
typed_config:
'@type': type.googleapis.com/envoy.extensions.clusters.dynamic_forward_proxy.v3.ClusterConfig
sub_clusters_config:
lb_policy: ROUND_ROBIN