Install Luvio on Kubernetes

This guide installs the Luvio server on a Kubernetes cluster with its Helm chart. The chart runs a pod with Luvio and its template transfer worker, and by default a PostgreSQL 17 pod for its database. To run several Luvio pods for high availability, read this guide, then Kubernetes with CloudNativePG.

At the end of this guide, the Luvio web console answers on http://<external IP>:3080 and you go on with First setup.

Tip

Read Plan your installation first: it lists the network ports to open between the server, the hypervisors, the virtual machines and the users’ computers.

Before you begin

You need:

• A Kubernetes cluster, version 1.25 or later, with x86-64 or ARM64 nodes. This guide was validated on k3s.
• kubectl and Helm 3 or later, configured for the cluster, on your workstation.
• A load balancer for LoadBalancer services, with a free address for Luvio: a cloud load balancer, MetalLB, kube-vip, or the one of your distribution (k3s ServiceLB, for example). The virtual machines connect to Luvio through it.
• A StorageClass that provides ReadWriteOnce volumes: 20 GB for the data of each Luvio pod, 10 GB for the bundled PostgreSQL. kubectl get storageclass marks the default one (default); if there is none, name yours in your values at step 2, before the installation. Prefer network storage (Ceph RBD, Longhorn, NFS, a cloud disk): a class that keeps the volume on one node (local-path, OpenEBS LocalPV, hostPath) ties Luvio and PostgreSQL to that node, and they cannot start on another one while it is down. PostgreSQL must own its files: it does not start on an NFS export that maps every user to another one (all_squash). Give it another class than Luvio’s data if needed.
• About 1 vCPU and 2 GB of RAM free for the Luvio pod, 0.5 vCPU and 1 GB for PostgreSQL.

Important

Luvio tells the agents in the virtual machines to connect to the port it listens on itself: 3080 (HTTP) or 3443 (HTTPS). The load balancer must publish that same port number, and the virtual machines must reach its address. An ingress serves the web console only: the agents and the desktop clients do not go through it.

1. Get the chart and the images

With access to the registry, the chart is oci://<HELM_REPOSITORY>/luvio and the images come from <REGISTRY>. If the registry requires credentials, create a pull secret in the namespace:

kubectl create namespace luvio
kubectl create secret docker-registry luvio-registry -n luvio \
  --docker-server=<REGISTRY> --docker-username=<user> --docker-password=<password>

Without Internet access, download the offline bundle of your nodes’ processors from <DOWNLOAD_URL>: luvio-helm-<version>-offline-amd64.tar.gz (x86-64) or luvio-helm-<version>-offline-arm64.tar.gz (ARM64). It holds the chart, the images and a script that imports them. Follow its README-offline.md up to the import of the images: it explains how to push them to your private registry (recommended) or to import them on every node. In short:

tar -xzf luvio-helm-<version>-offline-amd64.tar.gz
cd luvio-helm-<version>-offline-amd64
sha256sum -c SHA256SUMS
./import-images.sh registry registry.example.internal/luvio   # private registry
sudo ./import-images.sh node                                  # or: on each node

2. Write your values

Keep your settings in a file, luvio-values.yaml. You pass it at the installation and at every upgrade. The minimum:

service:
  type: LoadBalancer
server:
  # The address the virtual machines reach Luvio at: the external IP of the service, or a DNS
  # name that points to it. Without http:// and without a port.
  publicHostname: 192.0.2.50

server.publicHostname is the Server URL that Luvio gives to the agents in the virtual machines. Inside its pod, Luvio only sees the pod’s own address (for example 10.42.0.12), which the virtual machines cannot reach. If the load balancer chooses the external IP, you know it only after the installation: leave publicHostname out for now and add it at step 4.

Add, depending on your case:

# Images pushed to a private registry (offline bundle, method A)
global:
  imageRegistry: registry.example.internal/luvio

# Registry that requires credentials
imagePullSecrets:
  - name: luvio-registry

# A fixed address on the load balancer (MetalLB, kube-vip...), the one the agents use
service:
  type: LoadBalancer
  loadBalancerIP: 192.0.2.50
server:
  publicHostname: 192.0.2.50

# No default StorageClass, or another one than the default (NAME in kubectl get storageclass)
persistence:
  storageClass: longhorn
postgresql:
  persistence:
    storageClass: longhorn

Set the StorageClass before the installation: the one of the bundled PostgreSQL cannot be changed afterwards (see Troubleshooting).

With MetalLB, the address of the service comes from an IPAddressPool. A pool with autoAssign: false only gives the addresses that a service asks for, with service.loadBalancerIP. If no pool holds the address you chose for Luvio, create one, and announce it on your network (layer 2 here, or BGP):

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: luvio
  namespace: metallb-system
spec:
  addresses:
    - 192.0.2.50/32
  autoAssign: false
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: luvio
  namespace: metallb-system
spec:
  ipAddressPools:
    - luvio

3. Install the chart

From the registry:

helm install luvio oci://<HELM_REPOSITORY>/luvio --version <version> \
  --namespace luvio --create-namespace --values luvio-values.yaml

From the offline bundle:

helm install luvio ./luvio-<version>.tgz \
  --namespace luvio --create-namespace --values luvio-values.yaml

The chart generates the secrets of the installation (the session secret, the encryption key and the database password) and keeps them across upgrades.

4. Check that Luvio runs

1. Watch the pods start:

   kubectl get pods -n luvio

   luvio-0 must be 2/2 Running (Luvio and its transfer worker) and luvio-postgresql-0 1/1 Running. The first start takes a minute or two: Luvio builds its database.

2. Find the external address of the service:

   kubectl get service luvio -n luvio

   The EXTERNAL-IP column gives it. If it stays <pending>, the load balancer gives no address to the service: kubectl describe service luvio -n luvio says why (see Troubleshooting).

3. If server.publicHostname is not in your values yet, add it now with that address (or a DNS name that points to it), then apply it:

   helm upgrade luvio <chart> --namespace luvio --values luvio-values.yaml

   <chart> is the one of step 3. The pod restarts with the address.

5. Open the web console

Open http://<EXTERNAL-IP>:3080 and follow First setup. Luvio has built its database at its first start: the wizard opens on the hypervisor. In Administration > Server & storage, the Server URL shows http://<server.publicHostname>:3080 with From environment.

To skip the wizard, give the hypervisor and the administrator in a configuration file, in your values. For physical machines only, without a hypervisor, set admin.password and config: {hypervisors: []}.

6. Save the encryption key

config-encryption-key encrypts the hypervisor passwords and the other secrets Luvio stores in its database. Without it, a backup of the database cannot be restored, and it must never change. Copy it to your password safe now:

kubectl get secret luvio -n luvio -o jsonpath='{.data.config-encryption-key}' | base64 -d; echo

After a key rotation

Administration > High availability can rotate the key: Luvio encrypts its secrets again with a new key, which each Luvio pod keeps in its data volume. The Secret keeps the previous one: a pod with a new volume (one more pod, a deleted volume) would start with it, and read none of the passwords. Such a pod never turns ready: /readyz answers "reason":"encryption-key-mismatch" and its log says This node cannot decrypt what the database holds. Right after the rotation, put the new key in the Secret:

1. Read it:

   kubectl exec -n luvio luvio-0 -c server -- cat /var/lib/luvio/data/cluster-key

2. Put it in the Secret (the one of secrets.existingSecret if you gave one), then restart the pods. Never put an empty value: the next helm upgrade would generate a new key.

   kubectl patch secret luvio -n luvio --type merge \
     -p '{"stringData":{"config-encryption-key":"<the key of step 1>"}}'
   kubectl rollout restart statefulset/luvio -n luvio

3. Copy the new key to your password safe, in place of the previous one. helm upgrade keeps it.

Settings

Change your values file, then apply it:

helm upgrade luvio <chart> --namespace luvio --values luvio-values.yaml

<chart> is oci://<HELM_REPOSITORY>/luvio --version <version> or ./luvio-<version>.tgz. The main values:

Value	Default	Use
replicaCount	1	Number of Luvio pods: see High availability
server.publicHostname	empty	External IP or DNS name of the service, the Server URL of the agents. Empty: set the Server URL in Administration > Server & storage instead
server.protocol	http	http, https or both
server.httpPort / server.httpsPort	3080 / 3443	Listening ports, also published by the service
service.type	ClusterIP	LoadBalancer for the virtual machines and the desktop clients
ingress.enabled	false	Web console through your ingress controller
server.trustedProxies	[]	Network of the ingress controller’s pods: see Ingress
persistence.size	20Gi	Luvio’s data volume
transfer.workingFolder.persistence.enabled	false	A volume of its own (200 GB by default) for Universal Templates
tls.existingSecret	empty	Your certificate for HTTPS: see HTTPS
config, configEnv	empty	The configuration file and its variables
admin.password, admin.existingSecret	empty	Password of the first admin account, created only when there is none
postgresql.enabled	true	Bundled PostgreSQL; false with cnpg.enabled or externalDatabase.*
cnpg.enabled	false	A CloudNativePG cluster created by the chart: see High availability

The chart’s own README lists every value: helm show readme ./luvio-<version>.tgz.

The service publishes the ports of server.httpPort and server.httpsPort: the protocol and the ports are set in the values, and Administration > Server & storage shows them with From environment.

HTTPS

Set server.protocol: both (or https). The service then publishes 3443 too. By default, Luvio serves HTTPS with a self-signed certificate that it creates and renews itself: browsers show a warning, and the desktop clients ask the user to trust it the first time.

To serve your own certificate, put it in a kubernetes.io/tls Secret (made by cert-manager, or with kubectl create secret tls luvio-tls -n luvio --cert=luvio.crt --key=luvio.key) and name it in your values:

tls:
  existingSecret: luvio-tls

The certificate must be valid for server.publicHostname. Luvio reads it again when the Secret changes (a renewal), without a restart. An ingress with your certificate can also serve the web console.

With both, port 3080 stays open for the agents, the desktop clients, the identity providers and the probes, and the web console moves to HTTPS: a browser that opens http://<address>:3080 is sent to https://<address>:3443, and the sign-in and the console’s API refuse HTTP. A session opened over HTTPS has its own cookie, which browsers never send over HTTP. The ingress keeps serving the web console through port 3080, as a trusted proxy: with both, the chart requires server.trustedProxies (Ingress). HTTPS answers tell the browser to use HTTPS for this host name for one day (HSTS). Single sign-on to the web console reached on port 3443 comes back to that HTTPS address: the identity provider must accept https://<address>:3443/api/auth/oidc/callback (OpenID Connect) or the SAML metadata of https://<address>:3443/api/auth/saml/metadata, and a Redirect URI set in the OpenID Connect settings must be that HTTPS address.

Luvio gives its certificate to the agents, and confirms its new one when it changes. The agents accept no other certificate, except one that the system of the virtual machine trusts: they connect to the service, not to the ingress. A firewall or a proxy that inspects TLS between the virtual machines and Luvio must let their traffic through unchanged.

Ingress

Set server.trustedProxies to the addresses the ingress controller connects from: the network of its pods, usually the pod network of the cluster (for example 10.42.0.0/16 with k3s and RKE2, 10.244.0.0/16 with kubeadm and Flannel):

server:
  trustedProxies:
    - 10.42.0.0/16

Luvio then reads the X-Forwarded-For, X-Forwarded-Proto and X-Forwarded-Host headers that the controller adds: it sees the users’ own addresses (the sign-in limits count per address), and knows when the ingress serves HTTPS. A session opened through an HTTPS ingress then gets a Secure cookie, and the answers carry HSTS. Luvio ignores these headers from any other address. Without the value, every user has the controller’s address, and a session opened through an HTTPS ingress gets a cookie without Secure. With server.protocol: both, the chart refuses an ingress without it: Luvio sends to HTTPS the web console requests that do not come from a trusted proxy.

A client that connects from a trusted address can choose the address and the protocol that Luvio sees. On a LoadBalancer service, set service.externalTrafficPolicy: Local: with the default, Cluster, the connections of the agents and the desktop clients can reach the pod from an address of the pod network.

External PostgreSQL

To use your own PostgreSQL (12 or later):

postgresql:
  enabled: false
externalDatabase:
  host: postgres.example.internal
  port: 5432
  database: luvio
  user: luvio
  existingSecret: luvio-db      # a Secret with the password under the key "password"
  sslMode: require

The account needs the CREATEDB right, unless the database exists already and belongs to it. Luvio builds the database at its first start. Connect it to PostgreSQL directly, never through PgBouncer in transaction mode. The chart can also create a CloudNativePG cluster for you (cnpg.enabled): see High availability.

Upgrade

1. Take a backup.

2. Offline: import the images of the new bundle the same way as at the installation.

3. Upgrade with the same values file:

   helm upgrade luvio <new chart> --namespace luvio --values luvio-values.yaml

Pass your values file rather than --reuse-values, which ignores the defaults of values a new chart adds. With one pod, Luvio stops for the duration of the upgrade (a minute or two). With several, the pods restart one at a time, and the others keep serving (High availability).

Important

From a chart that ran Luvio as a Deployment (before high availability), follow From a chart that ran a Deployment: the upgrade gives Luvio a new data volume and a new self-signed certificate.

Then upgrade the agents: the Luvio server and its agents must be the same release, and an agent older than the server refuses its settings and its commands (see Upgrade Luvio and every agent together in Templates, agent and pools). With several Luvio servers in a cluster, upgrade every server node before the agents.

Universal Templates on Proxmox VE: from a version that did not check the SSH host keys of the nodes, copies stop until you trust the key of each node, in the Universal Templates panel of the hypervisor (see Universal Templates).

Back up and restore

A backup is two things: the database and the keys of the installation.

kubectl exec -n luvio luvio-postgresql-0 -- pg_dump -U luvio -Fc luvio > luvio-db-$(date +%F).dump
kubectl get secret luvio -n luvio \
  -o go-template='{{range $k, $v := .data}}{{$k}}={{$v | base64decode}}{{"\n"}}{{end}}' \
  > luvio-keys-$(date +%F).env

Keep both files outside the cluster: luvio-keys-<date>.env holds the keys. The bundled PostgreSQL has no replication and no automatic backup: schedule this backup, or use your own PostgreSQL for production. With CloudNativePG, back up the database with cnpg.backup (High availability), and keep the keys the same way.

To restore into a new installation:

1. Create the namespace and a Secret with the saved keys:

   kubectl create namespace luvio
   kubectl create secret generic luvio-keys -n luvio --from-env-file=luvio-keys-<date>.env

2. Add it to your values, then install the chart (step 3) without opening the web console:

   secrets:
     existingSecret: luvio-keys

3. Once luvio-postgresql-0 runs, restore the database and restart Luvio:

   kubectl exec -i -n luvio luvio-postgresql-0 -- pg_restore -U luvio -d luvio --clean --if-exists < luvio-db-<date>.dump
   kubectl rollout restart statefulset/luvio -n luvio

A restore brings Luvio back to the time of the backup: desktops created after it are unknown to Luvio; delete them from the hypervisor.

Uninstall

helm uninstall luvio --namespace luvio

This keeps the Secret with the keys, the PostgreSQL volume and the data volumes of the Luvio pods: installing again with the same release name and namespace finds them. A CloudNativePG cluster created by the chart is deleted with its volumes. To delete everything, after a backup:

kubectl delete namespace luvio

Troubleshooting

kubectl get pods -n luvio
kubectl describe pod -n luvio <pod>
kubectl logs -n luvio luvio-0 -c server     # web console, API
kubectl logs -n luvio luvio-0 -c transfer   # template transfers

Symptom	Cause and fix
Pod in ErrImagePull or ImagePullBackOff	The node cannot get the image: check global.imageRegistry and the pull secret, or import the images on that node (offline bundle). kubectl describe pod shows the image name.
EXTERNAL-IP stays <pending>	No load balancer serves LoadBalancer services in the cluster, or it has no address for this one. MetalLB says no available IPs in kubectl describe service luvio -n luvio when its pools are full or only give the addresses services ask for (autoAssign: false): set service.loadBalancerIP to an address of a pool, or create one (see step 2), then run helm upgrade.
luvio-postgresql-0 restarts, its logs say data directory "/var/lib/postgresql/data/pgdata" has wrong ownership	Its volume does not keep the owner of the files: an NFS export with all_squash, for example. Give PostgreSQL another class with postgresql.persistence.storageClass; like in the next row, helm upgrade cannot change it: uninstall, delete the PVC and install again.
Pods Pending, kubectl get pvc -n luvio shows Pending volumes without a StorageClass	No default StorageClass. helm upgrade cannot fix it: it fails with updates to statefulset spec for fields other than ... are forbidden, because the volumes of the bundled PostgreSQL and of Luvio are fixed when they are created. As nothing is stored yet, start over: helm uninstall luvio -n luvio, kubectl delete pvc data-luvio-postgresql-0 data-luvio-0 -n luvio (the uninstallation keeps them), then install again with persistence.storageClass and postgresql.persistence.storageClass in your values.
The machines stay Not connected in the console	The Server URL is the pod’s address, with the warning The virtual machines probably cannot reach this address: set server.publicHostname and run helm upgrade. Luvio sends the new address to the agents that were given the pod’s address; for a machine that stays Not connected, use Resend Agent Config in Manage Machines. Or the virtual machines do not reach the external IP on port 3080.
server.publicHostname takes a DNS name or an IP address when you install or upgrade	The value has http://, a port or a path: give the name or the address alone.
Unsupported state or unable to authenticate data in the logs	The encryption key is not the one the database was encrypted with: restore the saved Secret.
A Luvio pod stays 1/2, not ready, its logs say [DeploymentConfig] … is invalid	The configuration file has an error: nothing of it is applied. The logs list the errors. Fix config, then run helm upgrade. During an upgrade, the other pods keep serving.
Luvio pods in CreateContainerConfigError with cnpg.enabled	They wait for the Secret luvio-db-app, which the CloudNativePG operator creates with the database. Check that the operator is installed: kubectl get crd clusters.postgresql.cnpg.io.
helm install or helm upgrade says Choose exactly one database	Set one of postgresql.enabled, cnpg.enabled or externalDatabase.host, and turn the others off.