ESS LTS 24.10 Change Logs and Upgrade Notes
Upgrade Notes for the 24.10 LTS
If you plan on upgrading to this LTS we always recommend upgrading to the latest patch version of your current LTS and then updating to the latest version of this LTS.
If you plan on updating, we recommend installing the latest patch version.
Whether upgrading or updating, you should be aware of all significant upgrade notes from each prior patch version. Any highlighted patch notes for this specific LTS have been collated for convenience below, you can find the full changelogs of each release thereafter.
- 24.10.01-gui:
The required Python versions are now 3.10, 3.11, 3.12.
As a result, Ubuntu 24.04 is now supported but Ubuntu 20.04 support is dropped. Please consult the Ubuntu documentation for upgrading between Ubuntu LTS versions.
The installer will attempt to install the required packages in some scenarios.
Airgapped customers should ensure that Python 3.12 packages are available in their package mirrors.
Alternatively, Python 3.10, 3.11, or 3.12 can be preinstalled on the server in all situations.
24.10.02-gui
Security Issues
- Upgrade Element Web to v1.11.85, fixes CVE-2024-50336, CVE-2024-51749 and CVE-2024-51750.
Bug Fixes
- When setting securityContext for pods, also set runAsGroup.
24.10.01-gui
Release Summary
The required Python versions are now 3.10, 3.11, 3.12. As a result, Ubuntu 24.04 is now supported but Ubuntu 20.04 support is dropped. Please consult the Ubuntu documentation for upgrading between Ubuntu LTS versions. The installer will attempt to install the required packages in some scenarios. Airgapped customers should ensure that Python 3.12 packages are available in their package mirrors. Alternatively, Python 3.10, 3.11, or 3.12 can be preinstalled on the server in all situations.
New Features
- XMPP Bridge and IRC Bridge both support Authenticated Media. Their signing key is generated automatically by the installer UI.
- Authenticated Media is now enforced by default. All components but Matrix Content Scanner are compatible with it. If you need to disable it, please add
enable_authenticated_media: false
to Synapse -> Additional YAML. - Add the possibility to allow/deny rooms and log events for AuditBot.
- Support overriding just the server and path in the image digest ConfigMap.
- Support Element Call in Element X.
- Matrix Authentication Service and Synapse only use internal paths to communicate, removing the need for
hostAliases
setup between the two. - All ESS Images are now hosted behind
registry.element.io
. - Synapse workers supporting multiple replicas can now be configured for automatic horizontal scaling.
- Expose
images_digests.yml
file in the Download screen for Airgapped customers who want to sync their registry directly withregistry.element.io
.
Upgrade Notes
- Upgrade to cert-manager 1.15.3.
- Operator - Upgrade Python to 3.12, Ansible to 2.17.
- Upgrade Synapse to v1.116.0.
- Upgrade Element Web to v1.11.82.
- Update XMPP Bridge to 2.0.1.
- Update AdminBot and AuditBot to 6.3.1.
- Update IRC Bridge to 3.0.2.
- Update Hydrogen to 0.5.0.
- Update Admin Console to v16.105.4.
- Upgrade
microk8s
to 1.31.
As per 24.10 releases, the standalone installer only supports upgrading microk8s
installed from 23.10 releases.
As per 23.10.35/24.04.05/24.05.01, the standalone installer now upgrades microk8s
automatically. The microk8s
upgrade procedure does not involve an uninstall/reinstall of microk8s
any more. It now will automatically upgrade microk8s
to the expected version, and as such, the --upgrade-cluster
flag has been removed.
Any customization to CNI Configuration in /var/snap/microk8s/current/args/cni-network/cni.yaml
will have to be reconfigured.
During the upgrade, microk8s
& workloads will restart several times. Managed addons that require upgrading will be temporarily disabled to be upgraded.
This all will induce a small downtime of a couple of minutes.
- The installer now makes sure the upgrade comes from a supported version.
Security Issues
- Upgrade to Ansible 9 for security fixes and Python compatibility.
Bug Fixes
- Allow only one VoIP platform (Jitsi or Element Call) to be enabled.
- Fix migration of authentication settings from 24.07.01 with Matrix Authentication Service installed.
- Fix an issue where, after update, the installer UI would ask to save for changes on the Host screen when the user actually did not click anything.
- Fix monitoring integration tab not rendering.
- Fix AuditBot logs viewer when Matrix Authentication Service is setup.
Non-LTS Monthly Release Changes
This section summarises all the changes between the previous LTS and this one during the monthly non-LTS releases. Duplicate entries where individual components received upgrades have been removed so only the latest version is mentioned.
You can then compare the below changelog against the above LTS releases for an accurate overall changelog if upgrading from a previous LTS.
Info
Some changes added to non-LTS monthly releases are backported into older LTS releases if required. As such, some of the below features may already be present in a previous LTS. You can check the associated LTS books' respective changelog page to compare.
Release Summary
The required Python versions are now 3.9, 3.10, 3.11. These are available on all supported OS distributions. The installer will attempt to install the required packages in some scenarios. Airgapped customers should ensure that Python 3.9 packages are available in their package mirrors. Alternatively, Python 3.9, 3.10, or 3.11 can be preinstalled on the server in all situations.
- This release adds the possibility to enable Matrix Authentication Service during initial setup. Enabling Matrix Authentication Service is experimental; a couple of features do not work yet with it (AuditBot, AdminBot, Element Call, GroupSync, Admin UI). Enabling MAS allows you to use Element X with OIDC or LDAP login.
- This release now makes ESS ElementX ready by default. Any new installation will deploy Matrix Authentication Service. Existing setups will not profit from this change, migration paths are planned later in the future.
New Features
- Support knocking with
generic_worker
federation. - Major Change: The standalone installer now upgrades
microk8s
gracefully and automatically. Themicrok8s
upgrade procedure no longer involves an uninstall/reinstall ofmicrok8s
. It now automatically upgradesmicrok8s
to the expected version, and the--upgrade-cluster
flag has been removed.
Any customization to CNI Configuration in /var/snap/microk8s/current/args/cni-network/cni.yaml
will need to be reconfigured. During the upgrade, microk8s
will restart, and addons will be disabled to force an upgrade. This process may induce a small downtime of a couple of minutes.
- Status watchers are now golang containers, reducing resources used by the operator and updater.
- Allow configuration of Synapse database connection pool sizes.
- Add a ServiceMonitor to scrape metrics of
microk8s
ingress. - Expose Operator & Updater metrics.
- Add support for Outbound webhooks in Hookshot.
- Synapse OIDC support attribute requirements.
- Add a new experimental feature to enable Matrix Authentication Service during ESS bootstrap.
- Simplification of the OIDC provider configuration. After upgrading, please make sure that your OIDC settings were properly migrated to the new view.
- It is now possible to enable the new Matrix Authentication Service when bootstrapping a new ESS setup. It is an experimental feature, incompatible with GroupSync, Element Call, AuditBot, and AdminBot at this time. It is required to try out Element X with OIDC login.
- It is now possible to use LDAP with Matrix Authentication Service.
- Properly enforce patterns check in UI inputs under cards that can be enabled/disabled.
- Display deployment availability in the UI, in addition to the reconciliation status.
- Element Call is now MAS-Compatible.
- Add the possibility to configure a matrix stats endpoint.
- Setup the
onprem-admin
user as a MAS admin. - Allow configuration of empty (no) disallowed IP ranges in Hookshot.
- Validate Synapse Telemetry is consistently set.
- Synapse improve worker configuration.
- Allow blocking of non-scanned media.
- AdminBot/AuditBot + MAS compatibility.
- The UI now properly marks secrets as required when necessary.
- The reconciliation process now ensures that all secrets are present and shows missing secrets if necessary.
- Add Hookshot permissions configuration.
- Add the possibility to manage Federation dynamically from the Admin Console when Secure Border Gateway is enabled.
- Speed up initial Synapse deployment.
- Add the possibility to configure user deprovisioning and room clean up in GroupSync.
- Synapse auto invite: use Synapse native feature, run on background worker if it exists.
- Allow to override a container image without configuring a new digest.
- Support MSC4186 / Simplified Sliding Sync natively in Synapse.
- Support authenticated media APIs (MSC3916) in Synapse.
- Scrape Synapse HAProxy metrics.
- Scrape AdminBot and AuditBot HAProxy metrics.
- Set default volume sizes for Matrix Content Scanner volumes.
- Set default volume sizes for AdminBot, AuditBot & Sydent volumes.
- The administration interface can now manage users on deployments using Matrix OIDC.
- Administrators can now configure the SBG allowlist within the Admin UI.
- The user management page now allows admins to toggle the locked status of users.
- The user management page now displays the primary email address of users.
- The user management page will now default to showing locked and deactivated users when searching by name.
- Enabling MAS is not experimental any more, and is now the default setup mode.
- Allow to override a container image without configuring a new digest.
- Allow configuration of the operator and updater with debug logs.
- Check for supported Python versions when starting a deployment run. Recreate the virtual environment if it is using the wrong Python version.
- The installer now makes sure that the
microk8s
version on the host is supported before starting the upgrade process. - Speed improvements in the operator/updater reconciliation process.
Upgrade Notes
- Upgrade Telegram bridge to 0.15.1-mod-1.
- Upgrade WhatsApp bridge to 0.10.7-mod-1.
- Upgrade Sygnal to 0.14.3 to support the latest Firebase API.
- Update Synapse Admin to v16.92.0.
- Update AdminBot to Pipe 6.1.1.
- Matrix Content Scanner upgrade to 1.0.8.
- On RHEL and derived platforms, it now requires
python 3.11
installed. - Upgrade SecureBorderGateway to v1.2.0.
- Upgrade AuditBot to 6.1.2 to improve overall request handling efficiency, especially at high-loads.
- Upgrade to Synapse 1.114.0.
- Upgrade to Element Call 0.6.3 with improved call layout.
- Upgrade to Matrix Authentication Service 0.11.0 and support password auth.
- Synapse registration and password policy settings are now moved to Authentication configuration, under Local Password Database mode.
- Upgrade Hydrogen to v0.4.1-fix.
- Upgrade to cert-manager 1.12.13.
- Upgrade ElementWeb to v1.11.81.
- Services got renamed,
-headless
suffixes are all removed. If you are using Network Policies, those will need to be upgraded to the new names. - Global upgrade of the monitoring stack. Victoria Metrics is now on version 1.101.
- Now that Synapse brings native Sliding Sync protocol, the Sliding Sync proxy has been discontinued. Its PostgreSQL cluster instance is being cleaned-up.
Security Issues
- Previous update might have enabled unexpectedly outbound webhooks in Hookshot. If you don't need this feature, make sure that it is disabled in Hookshot integration, under Generic Webhooks settings.
- Better image signatures, enterprise is now published to
sigstore
. - Upgrade to Ansible 8 for security fixes.
Bug Fixes
- Fix Remove button not working for some integrations.
- Fix cert-manager upgrade failing to remove old resources.
- Fix operator and updater having permissions issues under OpenShift.
- Fix Jitsi JVB failing to get ready when STUN servers list is empty and Coturn is not deployed.
- Fix missing storage class on some Monitoring PVCs.
- Fix media screen on standalone setup.
- Remove
--upgrade-cluster
parameter asmicrok8s
is now upgraded gracefully. - Fix inconsistent behaviour when switching between S3/Persistent volume option under the media tab.
- Fix watchers to avoid triggering unneeded reconciliation loops.
- GroupSync: Fix issue when LDAP identities contain commas in their names.
- Configuring monitoring stack persistent volumes properly in
microk8s
requires recreating theirstatefulsets
. - Fix haproxy failing on IPv4-only nodes.
- The installer no longer flakes between bootstrap and installer view when the Kubernetes cluster is intermittently unreachable.
- Fix an Ansible error when installing the telemetry script on the local host when user GID != UID.
- Allow well-known delegation to omit configuration of the ingress entirely without triggering unknown variable errors.
- Allow configuration of Matrix Content Scanner without a storage class name.
- Mark Postgres configuration as required for all components that use a Postgres database.
- Mark the source for GroupSync as required.
- Remove workloads and dependent CRs from statuses when they're no longer deployed.
- Fix provisioning of users that are not rate-limited.
- Better identification for the Telegram and WhatsApp bridges in their respective apps.
- Fix an issue where the cert-manager issuer would try to be created but the cert-manager webhook would not be ready.
- Fix haproxy failing on IPv4-only nodes.
- Fix monitoring of Kube
etcd
and Kube scheduler onmicrok8s
. - Don't include cert-manager in the Airgapped tarball. ESS doesn't install or manage cert-manager in Airgapped deploys.
- Avoid leaking Postgres connections when there are issues provisioning Synapse users.
- SIPBridge - Disable Virtual rooms.
- Attempt to detect OpenShift and configure operator & updater installation values appropriately.
- Fix an issue preventing setup when a proxy is configured on the host.
- Fix a critical issue which would prevent users from accessing AdminBot and AuditBot UI.
- Fixes an issue where AuditBot UI would fail to open because tokens were unable to refresh.
- Revert change of 24.04.07 which prevented AdminBot and AuditBot from doing an initial sync.
- Create new devices for AdminBot and AuditBot to work with the new Rust SDK cryptographic libraries.
- Reduce secrets leaks from operator & updater logs. If you need, for debugging purposes, to enable secrets logging, you must edit the operator & updater deployments and set the environment variable
DEBUG_MANIFESTS=1
. - Refactor Synapse config files to own the priority of each setting managed by ESS.
- Sygnal upgrade to 0.15.0 for further Firebase API fixes.
- AdminBot and AuditBot are currently incompatible with MAS.
- Synapse - Override
botocore
CA bundle to allow pushing against non-AWS S3 providers. - Add support for Element Call configuration in Element Well Known file.
- Matrix Authentication Service - Fix UI configuration of certificates for ingresses.
- Minor speed up to initial setup of Synapse.
- Prevent users from manually editing the AuditBot/AdminBot passphrase.
- Fix display of the status of the reconciliation.
- Fix Coturn page causing a memory leak.
- Ensure the
nf_conntrack
module is loaded in the kernel when deploying in standalone mode. - Fix
microk8s
services subnet parsing. - Fix some CVEs in the operator/updater/conversion webhook.
- Fix Matrix Content Scanner not working as expected.
- Configure max upload size in Secure Border Gateway request body size limit.
- Prevent users from editing AuditBot and AdminBot passphrases in the UI.
- Enforce pattern checks against inputs under options.
- Increase Matrix Content Scanner ClamAV startup reliability.
- Reduce false positives from Matrix Content Scanner.
- On RHEL and derived platforms, the installer should not rely on
platform-python
for tasks other than Firewalld and SELinux tasks formicrok8s
setup. - Fix proxy variables configuration check preventing the installer to go through.
- Fix an issue preventing setup when a proxy is configured on the host. On a proxy configuration errors, the installer will now continue the setup process after displaying the verification error message.
- Enable MSC 3967 on Synapse to avoid some device verification issues.
- Setup the
onprem-admin
user as a MAS admin. - Fix pulling operator & updater images from behind a proxy.
- Expired sessions are now automatically logged out of the admin interface.
- OIDC sessions are now refreshed correctly when the token expires.
- An error is now displayed when the standalone admin UI cannot load the audit/admin interface configuration.
- Ensure operator and updater metrics are correctly scraped.
- Ensure Telemetry room permissions are consistent.
- Ensure component settings for storageClassName override the global setting.
- Removing an item from a list field will now only delete one item.
- Setup the
onprem-admin
user as a MAS admin. - Fix Synapse being stuck with registration closed even if explicitly allowed.
- Improve reliability of changing the Postgres password in cluster if the password seed changes.
- Fix potential permissions issues during
microk8s
upgrades. - Construct storage for Matrix Content Scanner if deploying on ESS managed
microk8s
. - Correctly import Airgapped registry settings when upgrading from before 24.04.
- Remove unneeded reconciliations due to bad orphan detection.
- Fix updater metrics scraping.
- Improve reliability of setting up CoreDNS.
- Validate that the node IP is excluded from an HTTP Proxy if one is configured.
- Fix empty dashboards (
nginx
, Kubernetes Workloads, etc) in Grafana. - Fix missing VMAlert component which is required to gather record metrics.
- Fix
microk8s
stop command not stopping running containers. - Improve reliability of some
microk8s
interactions.
Deprecations
- Element Call participants limits feature is deprecated. The option has been removed from the UI.
- Jitsi and Element Call can not be deployed together.