Upgrades

Andromeda migrates each tenant database itself, the first time a broker opens it. There is no separate migration step to run. A broker can upgrade any tenant database created by an earlier stable release, and it refuses to serve a database whose schema doesn't match the one it was built for.

Every tenant database is stamped with its schema version and a digest of the migrations that built it.

Upgrade procedure

1. Read the changelog for the release you are moving to. A major release is the only one that may change observable behaviour.
2. Back up your databases.
3. Run andromeda configuration validate against your configuration with the new binary or image.
4. Stop the old broker and start the new one on the same databases. Each tenant is migrated the first time the broker opens it.
5. Check the log for tenant schema contract: lines. None means every tenant was accepted.

Pin the full version of the image in production, such as X.Y.Z, so an upgrade happens when you change it and not when a tag moves. See Project status for how releases are numbered.

A configuration that leaves out database.maintenance.mode runs maintenance in automatic mode. Set it to "observe" or "disabled" before upgrading a deployment whose database is maintained by other means.

What happens when a broker opens a tenant

The broker compares the tenant's stamp with the one it was built with.

• If the database is new, has no stamp yet, or has an older schema version, the broker migrates it and stamps it.
• If it is stamped with the same version and digest, the broker serves it.
• Anything else is drift the broker can't fix, and it refuses the tenant.

Drift covers a different migrations digest, a newer schema version than the broker's own, and a PostgreSQL server below the minimum version or not using UTF-8. It also covers PostGIS or TimescaleDB below their minimum versions.

A refused tenant answers every request with 500 InternalError. The log line starts with tenant schema contract: and says what didn't match. For example, migrations digest mismatch, expected ... but found ....

The platform database, which holds the tenant registry, is migrated the same way. Its migrator refuses a migration file that changed since it was applied.

Extension updates

After an image upgrade, each tenant database also updates its TimescaleDB and PostGIS extensions when a broker opens it. Tenants that no request opens are brought up to date without one.

At least every five seconds, the broker's wake pass loads up to tenancy.residency.max_wakes_per_pass tenants (default 4), counting the tenants it wakes for due work first. It loads only into free slots of the tenant cache and never unloads another tenant to make room. It starts with the tenants furthest behind. Those are the tenants no broker of this version has opened yet, followed by the oldest TimescaleDB, the oldest PostGIS, and the database with the most committed transactions. Each loaded tenant unloads after tenancy.residency.woken_lifetime, like any woken tenant.

A tenant is loaded for this once per version the server ships, so one whose role may not update an extension is not loaded again until the next image upgrade. Until then maintenance keeps reporting it as extension-update-available. PostgreSQL says how extension updates run and what they need.

Migrations that need more than a restart

Stable releases seal their migrations. A released migration never changes again, and every schema change is a new file that upgrades a database created by the previous release.

A few kinds of change need more from you than a restart. Their migrations say so in their own header, with the time measured on a large tenant.

• A new index on the history tables is built without blocking writes, one chunk at a time.
• If that build is cut short (a cancelled statement, a lost connection), the index is left invalid and the migration isn't recorded. The next start then fails on the existing name, instead of keeping a broken index quietly.
• To recover, drop the index with DROP INDEX <name>, which drops its chunk indexes too. Then restart the broker to build it again.
• A new unique constraint on a populated history table needs a planned pause in writes while the constraint is built.
• A change to the shape of a stored composite type means rows queued by the old release can't be read by the new one.
• For that, either drain the queues, stop every writer, then migrate and start the new binary.
• Or the migration moves the data to a new type beside the old one.

Snapshot branches

Each Snapshot keeps its own copy of the entity, history and subscription tables, and that copy is upgraded too. When a tenant is migrated, or a broker opens a tenant whose Snapshots are behind, every Snapshot below the current version is upgraded in turn.

A Snapshot whose upgrade fails keeps its old version. The error is recorded in snapshot_preparation.branch_upgrade_error. The upgrade moves on to the next Snapshot, and the tenant keeps serving.

Until it is upgraded, the Snapshot is quarantined.

• A request scoped to it gets InternalError naming it, for example Snapshot urn:ngsi-ld:Snapshot:... is quarantined: its stored branch has not been upgraded to this broker's schema.
• Its notification deliveries are held.
• It can't be cloned.
• It can still be deleted, and expiry and quota eviction still remove it.

How Snapshots behave otherwise is in Snapshots. Fix the cause, then let the upgrade run again. It runs whenever a broker loads the tenant, which a restart does. A successful upgrade lifts the quarantine.

Pre-releases and development builds

The migrations of an alpha, beta or rc release, and of a build from the main branch, are not sealed. They can still change before the stable release, and a broker refuses a tenant database that an earlier build of the same migrations created.

Don't upgrade a database you want to keep to one of these builds. To start over, delete the database volume and start PostgreSQL again. With Docker Compose that is docker compose down -v, then docker compose up -d, run against the compose file you started it with. down -v deletes the volumes, so all data in that stack is lost. The broker creates and migrates tenant databases again on first use.

If it goes wrong

• Every request to a tenant gets 500, and the log says tenant schema contract: migrations digest mismatch

  The database was created by a different build. If you only ran stable releases, it was changed by hand. If you ran a pre-release or development build against it, recreate the database or restore a backup.

• The log says tenant schema contract: schema version mismatch with a found version above the expected one

  A newer broker has already migrated this database. Run that newer build against it, or restore the backup you took before the upgrade.

• A Snapshot request gets InternalError saying the Snapshot is quarantined

  Read snapshot_preparation.branch_upgrade_error for that Snapshot, fix the cause, and restart the broker so it loads the tenant again.