Backup and Restore

The procedure described below explains how to do a full backup of your ESS Pro installation, and how to restore it.

It is also possible to do incremental backups for the s3 storage and the PostgreSQL databases. Doing so would avoid the need to stop the services.

Database Backups and caveats

Daily backups

Doing daily backups only can cause issues on restore. E2EE server-side will be desynchronized with the clients, causing a various of identifiers and key conflicts. It will cause a lot of UTDs, prevent users from reading their room history.

It should be used carefully, as a simple way to backup your Synapse & MAS database, but you should be prepared for issues during restore.

Point-in-time backups

Point-in-time backups allow you to restore your data at any point in time on a given retention period. It usually consists of a daily full-backup and incremental backups restored on top of it.

Cloud-managed databases usually handle this as "Automated backups".

Alternatively, wal-g is often used for the purpose of storing Postgres WAL files on an external object storage.

When restoring your database, make sure to restore the most frequent and valid database to avoid the E2EE issues mentioned above.

Matrix Authentication Service Tokens TTL

If using snapshots without also archiving WALs, you should configure Matrix Authentication Service to have an access_token_ttl value longer than twice the frequency of your snapshots:

matrixAuthenticationService:
  additional:
    access-token-ttl.yml:
      config: |
        experimental:
            access_token_ttl: 1200s

When restoring a database, if the client Access token is not valid any more according to its TTL window, it will cause a disconnection and the local encryption keys will be nuked. Users will have to reconnect their encryption storage after logging in. Having Access Tokens TTL longer than the expected snapshot restore time avoids this issue.

Full backups of the ESS deployment

You need to backup a couple of things to be able to restore your deployment:

Follow the documentation to stop ESS Pro services
The database. You need to backup your database and restore it on a new deployment.
1. If you are using the provided Postgres database, build a dump using the command kubectl exec --namespace ess -it sts/ess-postgres -- pg_dumpall -U postgres > dump.sql. Adjust to your own Kubernetes namespace and release name if required.
2. If you are using your own Postgres database, please build your backup according to your database documentation.
Your values files used to deploy the chart
The chart will generate some credentials in a Secret if you do not provide them. To copy them to a local file, you can run the following command: kubectl get secrets -l "app.kubernetes.io/managed-by=matrix-tools-init-secrets" -n ess -o yaml > secrets.yaml. Adjust to your own Kubernetes namespace if required.
The chart will generate some flags/markers in a ConfigMap to ensure that helm upgrade with different values doesn't put the installation in an invalid state. To copy them to a local file, you can run the following command: kubectl get configmap -l "app.kubernetes.io/managed-by=matrix-tools-deployment-markers" -n ess -o yaml > configmaps.yaml. Adjust to your own Kubernetes namespace if required.
The media files: if you are using S3 storage, follow your S3 provider backup recommendations. If you are storing media in persistent volume, Synapse stores media that should be backed up. On a default K3s setup, you can find where synapse media is stored on your node using the command kubectl get pv -n ess -o yaml | grep synapse-media.
Run the helm upgrade --install.... command again to restore your workload's pods.

Restore procedure

Recreate the namespace and the backed-up secret in step 3:

kubectl create ns ess
kubectl -n ess apply -f secrets.yaml
kubectl -n ess apply -f configmaps.yaml

Redeploy the chart using the values backed-up in step 2.
Follow the documentation to stop ESS Pro services

Restore the PostgreSQL dump. If you are using the provided PostgreSQL database, this can be achieved using the following commands:

# Drop newly created databases and roles
kubectl exec -n ess sts/ess-postgres -- psql -U postgres -c 'DROP DATABASE matrixauthenticationservice'
kubectl exec -n ess sts/ess-postgres -- psql -U postgres -c 'DROP DATABASE synapse'
kubectl exec -n ess sts/ess-postgres -- psql -U postgres -c 'DROP ROLE synapse_user'
kubectl exec -n ess sts/ess-postgres -- psql -U postgres -c 'DROP ROLE matrixauthenticationservice_user'
kubectl cp dump.sql ess-postgres-0:/tmp -n ess
kubectl exec -n ess sts/ess-postgres -- bash -c "psql -U postgres -d postgres < /tmp/dump.sql"
kubectl exec -n ess sts/ess-postgres -- psql -U postgres -d synapse -c 'TRUNCATE e2e_one_time_keys_json'

Adjust to your own Kubernetes namespace and release name if required.

If you are not using S3 for Synapse media, you need to restore the synapse media files. Since the Synapse container is distroless and lacks tar, you cannot use kubectl cp directly. Instead, create a debug pod with the matrix-tools debug image that mounts the Synapse PVC, then copy your files to it:

# Create a debug pod with the Synapse PVC mounted
kubectl run synapse-media-debug --image=matrix-tools:0.17.9-debug -n ess --restart=Never --stdin --tty --rm --overrides='{
   "spec": {
     "securityContext": {
       "runAsNonRoot": true,
       "seccompProfile": {
         "type": "RuntimeDefault"
       }
     },
     "imagePullSecrets": [
       {
         "name": "ess-registry-element-io"
       }
     ],
     "containers": [
       {
         "name": "debug",
         "image": "registry.element.io/matrix-tools:0.17.9-debug",
         "command": [
           "sleep",
           "infinity"
         ],
         "securityContext": {
           "allowPrivilegeEscalation": false,
           "capabilities": {
             "drop": [
               "ALL"
             ]
           },
           "runAsNonRoot": true,
           "fsGroup": 10091,
           "runAsUser": 10091,
           "runAsGroup": 10091
         },
         "volumeMounts": [
           {
             "name": "synapse-media",
             "mountPath": "/media"
           }
         ]
       }
     ],
     "volumes": [
       {
         "name": "synapse-media",
         "persistentVolumeClaim": {
           "claimName": "ess-synapse-media"
         }
       }
     ]
   }
 }'

# Wait for the pod to be ready
kubectl wait --for=condition=ready pod/synapse-media-debug -n ess --timeout=300s

# Copy your media files to the debug pod (which writes to the PVC)
kubectl cp <your_media_files_path> synapse-media-debug:/media/media_store -n ess

# Clean up the debug pod
kubectl delete pod synapse-media-debug -n ess

If you are using K3s, you can alternatively find where the new persistent volume has been mounted with kubectl get pv -n ess -o yaml | grep synapse-media and copy your files directly to the destination path.

Follow the documentation to start ESS Pro services