High availability

This page runs several Luvio servers that work together. If one server stops, the others keep serving the web console, the sign-in of the desktop clients and the agents of the virtual desktops. The desktop streams never go through the Luvio server: the users already connected keep their desktops in any case.

How it works

Every server is active. Each one serves the web console, the API and the agents, and runs its own template transfer worker. They are called the nodes.
One address in front. The agents, the desktop clients and the browsers reach the nodes through one address: a Kubernetes Service, a virtual IP address (VIP), or your load balancer. This address is the Server URL of every node.
One database. The nodes share one PostgreSQL database, which must be highly available itself. Each node keeps a small data volume of its own.
One leader. One node runs the background work: the tasks that resume after a restart, the checks of the hypervisors and of the pools, the renewal of the certificate. When it stops, another node takes over by itself.

Choose a mode

Installation	High availability
Kubernetes (Helm)	Supported: several replicas, on a CloudNativePG cluster that the chart creates, or on your own highly available PostgreSQL. See Kubernetes.
Docker, application nodes	Supported: two or three Docker hosts, on your own highly available PostgreSQL, behind a VIP (keepalived, included) or your load balancer. See Docker application nodes.
Docker, standalone cluster	A lab and a preview only, not for production: the cluster mode of the standalone image.
.deb or .rpm package, Windows	One server. Its database can be your own highly available PostgreSQL (Use your own PostgreSQL), but the server itself is one machine.

Before you begin

A highly available PostgreSQL, version 12 or later, with one read-write address that always leads to the primary. With Kubernetes, the chart can create it with CloudNativePG. Otherwise: Patroni, pg_auto_failover, a managed database…
- Connect Luvio to PostgreSQL directly. Never through PgBouncer in transaction mode: Luvio holds locks and listens for notifications on its connections.
- Allow about 45 connections per node.
A fixed DNS name or IP address for the whole: the Kubernetes Service or the VIP. Every node gives this address to the agents, and the HTTPS certificate is made for it.
The same Luvio release on every node.
The network flows of Plan your installation, for every node: each node connects to the hypervisors and to the agents (TCP 8081) from its own address.

Kubernetes with CloudNativePG

This procedure runs three Luvio pods on a PostgreSQL cluster of three instances. Read Install Luvio on Kubernetes first: the steps below only add what high availability needs.

Install the CloudNativePG operator, once per Kubernetes cluster:
Terminal window
```
helm repo add cnpg https://cloudnative-pg.github.io/charts
helm upgrade --install cnpg cnpg/cloudnative-pg --namespace cnpg-system --create-namespace
kubectl get crd clusters.postgresql.cnpg.io
```
Without Internet access, push the images of the operator and cloudnative-pg/postgresql:18 to your private registry too: the Luvio offline bundle does not hold them.

Write luvio-values.yaml:

replicaCount: 3
postgresql:
  enabled: false        # not the bundled PostgreSQL
cnpg:
  enabled: true         # a CloudNativePG cluster, 3 instances by default
service:
  type: LoadBalancer
  loadBalancerIP: 192.0.2.50
server:
  publicHostname: luvio.example.com   # resolves to 192.0.2.50

Add the values of your case, as in Write your values. Set server.publicHostname: it must be the same for every pod.

Install the chart (step 3 of the Kubernetes guide).
Watch the pods start:
Terminal window
```
kubectl get pods -n luvio
kubectl get clusters.postgresql.cnpg.io -n luvio
```
luvio-db-1, luvio-db-2 and luvio-db-3 are the PostgreSQL instances; the cluster luvio-db turns to Cluster in healthy state. Until the operator has created the database, the Luvio pods wait in CreateContainerConfigError: this is normal. Then luvio-0, luvio-1 and luvio-2 turn 2/2 Running.
Open http://<server.publicHostname>:3080 and follow First setup, or give the settings in a configuration file to skip it.
Save the encryption key (step 6 of the Kubernetes guide).

Value	Default	Use
`replicaCount`	`1`	Number of Luvio pods
`cnpg.enabled`	`false`	A CloudNativePG cluster created by the chart, named `luvio-db`
`cnpg.instances`	`3`	PostgreSQL instances: one primary, the others replicas
`cnpg.storage.size`, `cnpg.storage.storageClass`	`10Gi`, default class	Volume of each instance
`cnpg.postgresql.parameters.max_connections`	`200`	Raise it with `replicaCount` (about 45 per pod)
`cnpg.backup`	none	The backup of the cluster (object store, volume snapshots), as in the CloudNativePG documentation
`podAntiAffinity`	`soft`	`soft` prefers one Luvio pod per node, `hard` requires it (as many nodes as pods), `none`
`podDisruptionBudget.maxUnavailable`	`1`	A node drain stops one Luvio pod at a time
`service.sessionAffinity`	`None`	Not needed: a user can reach any pod. `ClientIP` keeps each address on one pod
`tls.existingSecret`	none	Your certificate, for every pod (see What the nodes share)

Things to know:

The agents and the desktop clients connect to the LoadBalancer Service, on ports 3080 and 3443. An ingress can serve the web console to the browsers, with server.trustedProxies (Ingress).
helm uninstall deletes the CloudNativePG cluster and its volumes. Take a backup first.
After a key rotation (Administration > High availability), put the new key in the Secret (After a key rotation): a pod added later would start with the previous one.
The bundled PostgreSQL (postgresql.enabled) is one pod without replication. The chart accepts several Luvio pods with it, but warns: the database stays a single point of failure.
Your own PostgreSQL in place of CloudNativePG: set postgresql.enabled: false and externalDatabase.* (External PostgreSQL), with externalDatabase.host the address that always leads to the primary.

Docker application nodes

This procedure runs Luvio on two or three Docker hosts, all active, on a highly available PostgreSQL that you provide. Each host runs one Compose project with Luvio and its transfer worker. In front of them, either keepalived moves a VIP between the hosts, or your own load balancer spreads the connections.

You need, on each host: Docker Engine 24 or later with the Compose plugin 2.24 or later, the time synchronized with NTP, TCP ports 3080 and 3443 free.

Prepare the database. On your PostgreSQL cluster, create an account and a database for Luvio, for example:
```
CREATE ROLE luvio LOGIN PASSWORD '<password>';
CREATE DATABASE luvio OWNER luvio;
```
Luvio builds its tables at its first start. Every host must reach the read-write address of the cluster.
Get the images on every host. With a registry, the image is <REGISTRY>/luvio:<version>, and <REGISTRY>/luvio:<version>-standalone for keepalived. Without Internet access, load both offline bundles of Docker:
Terminal window
```
sudo docker load -i luvio-docker-<version>.tar
sudo docker load -i luvio-docker-<version>-standalone.tar   # for the VIP only
```
On the first host, create the directory of the node and put in it the three files of an application node: the ha-node folder of the Docker bundle (compose.yml, .env.example, with the image of the bundle already set, and keepalived.sh).
Terminal window
```
sudo mkdir -p /opt/luvio-node/config /opt/luvio-node/tls
cd /opt/luvio-node
sudo cp .env.example .env
sudo chmod 600 .env
```

Fill in .env. Generate the two secrets once, with openssl rand -hex 48 for each:

Line	Value
`LUVIO_IMAGE`	The image, for example `luvio:<version>`
`LUVIO_PUBLIC_HOSTNAME`	The DNS name or the address of the VIP or of the load balancer, without `http://` and without a port
`LUVIO_PROTOCOL`	`http`, `https` or `both`
`LUVIO_SESSION_SECRET`	A secret generated once, the same on every node
`LUVIO_CONFIG_ENCRYPTION_KEY`	A secret generated once, the same on every node. It must never change: without it, the database cannot be read
`LUVIO_DB_HOST`, `LUVIO_DB_PORT`	The read-write address of the PostgreSQL cluster
`LUVIO_DB_NAME`, `LUVIO_DB_USER`, `LUVIO_DB_PASSWORD`	The database and its account
`LUVIO_DB_SSLMODE`	`require` (TLS, the default of the file) or empty (no TLS)
`LUVIO_TRUSTED_PROXIES`	The addresses of your load balancer, if it proxies HTTP

Copy .env to your password safe: it holds the keys.

Optional, the same files on every node: a configuration file in config/luvio-config.yaml (and LUVIO_CONFIG_FILE=/etc/luvio/config/luvio-config.yaml in .env, the variables it refers to in config.env), your certificate in tls/tls.crt and tls/tls.key (and the two LUVIO_TLS_*_FILE lines of .env). The container reads them as uid 1000:
Terminal window
```
sudo chown -R 1000:1000 config tls
```

Choose what is in front.

With the VIP (keepalived, on the hosts themselves), fill in the VRRP lines of .env:

Line	Value
`LUVIO_VIP`	A free address of the hosts’ network. `LUVIO_PUBLIC_HOSTNAME` resolves to it
`LUVIO_VRRP_AUTH_PASS`	A password of 8 characters at most, the same on every node
`LUVIO_VRRP_PEERS`	The addresses of the other nodes, separated by commas
`LUVIO_VRRP_PRIORITY`	Optional, 1 to 200: the ready node with the highest one holds the VIP
`LUVIO_VRRP_ROUTER_ID`	Optional, `61` by default: change it if another VRRP group of the network uses it

Let VRRP (IP protocol 112) pass between the hosts. With firewalld:

sudo firewall-cmd --permanent --add-rich-rule='rule protocol value="vrrp" accept'
sudo firewall-cmd --reload

With your load balancer: forward TCP 3080 and 3443 to every node, on the same port numbers, with a health check GET /readyz on port 3080 (it answers 200 when the node can serve).

Start the first node:

cd /opt/luvio-node
sudo docker compose --profile vip up -d    # with the VIP
sudo docker compose up -d                  # with your load balancer

Check it:

sudo docker compose ps
curl -fsS http://localhost:3080/readyz

After a minute, luvio is (healthy) and /readyz answers {"status":"ready"...: Luvio has built its database. If not, read sudo docker compose logs luvio.

Copy the directory to the other hosts, with its .env (keep it chmod 600). On each one, change LUVIO_VRRP_PEERS to the addresses of the other nodes, then start it the same way.
Open http://<LUVIO_PUBLIC_HOSTNAME>:3080 and follow First setup, unless the configuration file completed the installation.

The agents, the desktop clients and the browsers always use LUVIO_PUBLIC_HOSTNAME, never the address of a node. Docker opens ports 3080 and 3443 in the packet filter of each host.

Back up the database with the tools of your PostgreSQL cluster, and keep .env with it. The data volume of a node holds nothing that the database cannot give again, except after a key rotation.

After a key rotation (Administration > High availability), every node keeps the new key in its data volume, and .env has the previous one: a node added later, or whose volume is lost, would start with it and read none of the stored passwords. Read the new key on one node, set LUVIO_CONFIG_ENCRYPTION_KEY to it in .env on every node and in your password safe, then apply it with sudo docker compose --profile vip up -d (or sudo docker compose up -d), one node at a time:

sudo docker compose exec luvio cat /var/lib/luvio/data/cluster-key

What	How
Sign-in sessions	In the database: a user signed in through one node stays signed in on the others. No sticky sessions are needed.
Settings	In the database. A change made on one node applies to every node.
HTTPS certificate	One certificate for every node. By default, Luvio makes a self-signed certificate for the Server URL and shares it through the database. Or give your own: import it in Administration > Server & storage > HTTPS certificate, or use `tls.existingSecret` with Kubernetes (a `kubernetes.io/tls` Secret), `LUVIO_TLS_CERT_FILE` and `LUVIO_TLS_KEY_FILE` with Docker. Luvio reads your files again when they change (a renewal), without a restart.
Background work	One node, the leader, runs it. When the leader stops, another node takes over within seconds; up to a minute when its host is lost suddenly.
VM consoles	Any node can open the console of a machine, whatever node the browser reached first.
ISO uploads	An upload goes on in the node that received the file. If that node is gone for more than 5 minutes, the upload fails with “The server that received this ISO is no longer running: upload the ISO again”.
Template transfers	Each node has its own transfer worker and its own working folder.

Check the failover

Kubernetes. In a terminal, watch the service from outside the cluster:

while true; do curl -s -o /dev/null -w '%{http_code}\n' http://luvio.example.com:3080/readyz; sleep 1; done

Find the leader in the web console: Administration > High availability > Cluster instances lists the nodes, and marks the leader with Leader. With Kubernetes, the host name of a node is the name of its pod. In another terminal, delete that pod:

kubectl delete pod -n luvio <leader pod>

The service keeps answering 200. Kubernetes starts the pod again, and another pod becomes the leader. Do the same with the primary of the database:

kubectl get clusters.postgresql.cnpg.io luvio-db -n luvio -o jsonpath='{.status.currentPrimary}'; echo
kubectl delete pod -n luvio <primary pod>

CloudNativePG promotes a replica, and the Luvio pods connect to it. In Luvio’s tests with three pods on CloudNativePG, the service through the Service stayed available in both cases.

Docker application nodes. On the host that holds the VIP (ip addr shows it), stop Luvio:

sudo docker compose stop luvio

Within a few seconds, another host holds the VIP, and http://<LUVIO_PUBLIC_HOSTNAME>:3080 answers again. Start Luvio again with sudo docker compose start luvio.

Upgrade

Upgrade every node to the new release, then the agents (Templates, agent and pools).

Kubernetes. Run helm upgrade as usual (Upgrade). The pods restart one at a time; the next one restarts once the new one is ready. A pod that cannot serve, for example with an invalid configuration file, never turns ready: the update stops there, and the other pods keep serving the previous version.

Docker application nodes. One node at a time:

Load the new images (both, with the VIP), or skip this step with a registry.
Change LUVIO_IMAGE in .env to the new version.
Apply, with the same command as at the start: sudo docker compose --profile vip up -d, or sudo docker compose up -d.
Wait until /readyz answers {"status":"ready"... on this node, then go on with the next one.

From a chart that ran a Deployment

Charts before high availability ran Luvio as a Deployment, with a volume luvio-data. The new chart runs a StatefulSet: the pod luvio-0 gets a new volume, data-luvio-0. Nothing of the old volume is needed: the database holds the settings, the machines, the templates and the users. With the change, the server is a new node, and it makes a new self-signed certificate (unless you give tls.existingSecret): it sends it to the agents by itself, and the desktop clients ask their users to trust it again.

Let the running tasks finish: ISO uploads, template transfers, pool deployments.
Take a backup.
If the encryption key was ever rotated (Administration > High availability), the current key is in the old volume only. Check:
Terminal window
```
kubectl exec -n luvio deploy/luvio -c server -- cat /var/lib/luvio/data/cluster-key
```
No such file or directory: the key never changed, go on. If it prints a key, put it in the Secret before the upgrade (After a key rotation): the new volume does not have it, and Luvio could not read its passwords.
helm upgrade deletes the old volume with the Deployment. To keep it for a while, annotate it first:
Terminal window
```
kubectl annotate pvc -n luvio luvio-data helm.sh/resource-policy=keep
```
Delete it once the new release runs: kubectl delete pvc -n luvio luvio-data.
Upgrade with your values file and replicaCount: 1.
Once luvio-0 is 2/2 Running, raise replicaCount and run helm upgrade again.

From the bundled PostgreSQL to CloudNativePG

Install the CloudNativePG operator (step 1).

Back up the database:

kubectl exec -n luvio luvio-postgresql-0 -- pg_dump -U luvio -Fc luvio > luvio-db.dump

In your values, set postgresql.enabled: false, cnpg.enabled: true and replicaCount: 0, then run helm upgrade. Keep the same secrets: the encryption key must not change.

Wait until kubectl get clusters.postgresql.cnpg.io -n luvio shows Cluster in healthy state, then restore the backup into the primary:

PRIMARY=$(kubectl get clusters.postgresql.cnpg.io luvio-db -n luvio -o jsonpath='{.status.currentPrimary}')
kubectl exec -i -n luvio "$PRIMARY" -c postgres -- pg_restore -d luvio --no-owner --role=luvio < luvio-db.dump

Set replicaCount: 3 and run helm upgrade again.
Once Luvio works, delete the volume of the old database: kubectl delete pvc -n luvio data-luvio-postgresql-0.

Limits

Storage fill limit: each node counts the space of the copies it is running itself. Two large copies started at the same moment on two nodes can fill a storage beyond the Maximum storage fill.
Template transfers: the working files of Universal Templates stay in the node that made them. A transfer made again on another node prepares them again.
Upgrade from the web console: not available with Docker and Kubernetes. Upgrade with the images and the chart.
Adding a node: the nodes of this page join by sharing the database and the secrets. The Join a high-availability deployment choice of the setup wizard is for the standalone cluster lab: do not use it here.