Skip to main content

Snapshots

A Snapshot is a schema of its own inside the tenant's database, so it takes disk space and, while one is being prepared or cloned, vacuum progress. This page covers how a Snapshot behaves from creation to deletion and what it costs the tenant database. The Snapshots guide covers using them, and Upgrades covers how Snapshots are upgraded. The members a Snapshot body may carry are under Snapshots and EntityMaps in Strictness, and the q that Purge Snapshots accepts is in the Query language.

Lifecycle​

A created or cloned Snapshot answers 201 in status preparing and is prepared in the background. It ends in one of four terminal statuses:

  • success: every query ran and the results are captured.
  • partial: some queries captured data and at least one failed. The Snapshot is usable with what was captured.
  • empty: every query ran and none matched. The Snapshot scopes to an empty dataset.
  • failure: preparation failed. The Snapshot publishes an empty dataset, so it stays addressable.

A preparation captures into a staging copy, which is published, together with the terminal status, in one transaction. A reader sees either no content or all of it.

A Snapshot with an endpoint gets a status notification when it reaches a terminal status and when its metadata is updated. Notifications are delivered from a durable queue under the same retry policy as subscription notifications ([notifications.delivery]). A retryable failure is retried until notifications.delivery.max_attempts attempts have been made, and a terminal failure is not retried.

Preparation​

Creating or cloning starts preparing at once. That first pass claims the new Snapshot by its identifier and waits out a brief hold on its row, such as the client reading the Snapshot it just created, rather than passing over it as the scheduled passes do, so that concurrent workers each take a different Snapshot. A Snapshot it cannot claim, because a worker already holds it or because it clones a Snapshot still preparing, is logged and left to its holder, to the pass that ends its source, or to the scheduled sweep.

Scoped requests​

A request scoped to a preparing Snapshot waits for its terminal status for up to snapshot.preparation.scoped_operation_timeout (default 20 s). If the Snapshot is still preparing then, the request is answered 503 with Retry-After: 1. A quarantined Snapshot is refused with InternalError, see Upgrades.

Lifetime and expiry​

  • A Snapshot created without snapshotLifetime lives snapshot.default_lifetime (default 1 h).
  • A longer lifetime than snapshot.max_lifetime (default 30 days) is cut down to it, not refused. A lifetime that is not positive is refused.
  • Patching snapshotLifetime restarts the lifetime from the time of the patch.
  • A clone takes its own snapshotLifetime, or else its source's, counted from the clone's creation.
  • An expired Snapshot is never served. A read of it answers 404 and deletes it, and the sweep every expiry.sweep_interval (default 30 s) deletes the rest.

Priority and last use​

snapshotPriority is an integer from 1 to 10, default 5, and can be patched. Retrieving a Snapshot and every request scoped to it record its last use. Priority and last use decide which Snapshot a tenant loses when it goes over its quota.

Quotas​

snapshot.quota.max_per_tenant bounds how many Snapshots a tenant keeps, and snapshot.quota.max_storage_per_tenant bounds how much disk they take. Both default to "unlimited". The storage figure is the on-disk size of every table of every Snapshot schema, published and staging, with indexes and TOAST. Both quotas count preparing Snapshots.

The quotas are checked after every create and clone, and after every preparation ends, since a preparation grows the storage. While a tenant is over either quota, Snapshots are deleted one at a time in this order:

  1. lowest snapshotPriority
  2. earliest expiry
  3. oldest last use, a Snapshot never used counting as the oldest
  4. smallest id

The Snapshot just created is a candidate like any other, and so is one still preparing. A tenant that creates a Snapshot of the lowest priority over a full quota can therefore lose that Snapshot at once. Each eviction logs Snapshot evicted by tenant resource quota with the Snapshot's id.

A Snapshot being in use never changes the order. An eviction is a deletion, so when a clone is copying into or out of the Snapshot first in line, its copy is ended, as Cloning a Snapshot describes. Any other request holding that Snapshot is waited for, for up to 5 seconds, and the order is read again once the request ends. A check that fails is logged as Snapshot quota enforcement after a create or clone failed or Snapshot quota enforcement after preparation failed. The create, clone or preparation that ran the check still succeeds, and the next one checks again.

Cloning a Snapshot​

A clone copies its source as of one moment. It takes the source's queries, query details, priority, endpoint and receiverInfo unless the clone request gives its own. A quarantined source is refused.

The copy is one REPEATABLE READ transaction. Every copy statement reads the snapshot taken by the transaction's first statement, so the copied entities, history and discovery counts agree with each other while writers on the source carry on. That is why the copy can't be split into smaller transactions. The copy takes as long as the source is large, and no statement timeout bounds it. It is prepared in the background.

Both Snapshots stay in place while the copy runs. Deleting or evicting either one ends the copy's database session first, so the deletion doesn't wait for the copy to finish. An expiry is different and waits for the copy. A deleted clone is gone. A clone whose source was deleted ends failure. The copy's session is identified by its process id together with the start of its transaction, so a pooled session that has moved on to other work is never ended.

To PostgreSQL, a REPEATABLE READ transaction is a client transaction, and it keeps the oldest row versions it could still see. The vacuum horizon of the tenant database can't pass them until the clone commits or fails, so no vacuum in that database can remove rows that died after the clone started, live tables included. Other tenant databases are not affected.

A clone of a large Snapshot on a busy tenant therefore shows the same symptoms as any long transaction. Dead space in current_attributes, entities and the work queues grows, and a catch-up vacuum finishes without reclaiming it.

What maintenance does about it​

The maintenance controller reads the holders of the vacuum horizon before a vacuum, and a running clone is among them as a client backend.

  • A clone that has been open for more than 30 seconds defers expensive maintenance work on that tenant database, the same as any other transaction open that long. The deferral is logged with its reason, and the work runs on a later pass once the clone has finished.
  • A vacuum that runs anyway and leaves the table still past its dead-space threshold is logged as ineffective, and the log lists every holder with its age. The clone appears there.
  • When transaction ID wraparound pressure is overdue or critical, the visit logs every holder of the horizon in any database, and the clone appears there too.

Maintenance describes the gates, and Maintenance describes the wraparound report.

Deletion​

A Snapshot is deleted by its id, by Purge Snapshots, by expiry or by eviction. Every route drops its schema and its record in one transaction, so a Snapshot is either whole or gone.

Background work that reads every Snapshot's schema can meet a schema dropped after it listed the Snapshots. It lists the Snapshots again for as long as such a failure is explained by a Snapshot gone since, and a notification whose Snapshot's schema is gone is treated like one whose subscription was deleted.

If it goes wrong​

  • Vacuums on a tenant keep coming back ineffective while a Snapshot is being created from another

    A clone holds the horizon. Wait for it to finish, because the horizon moves on as soon as it commits. Find it in pg_stat_activity as a transaction with an old backend_xmin in the tenant database.

  • A clone runs for a long time

    The copy takes as long as the source Snapshot is large. Ending its backend frees the horizon at once but cancels the clone, and the new Snapshot is not published. Do that only if the dead space it holds back is the greater cost.

  • A Snapshot disappeared without being deleted

    It expired or it was evicted. Search the broker log for Snapshot evicted by tenant resource quota with the Snapshot's id. With no such line, check its expiresAt: a Snapshot past it is gone. To keep Snapshots longer, raise the lifetime within snapshot.max_lifetime. To stop evictions, raise the quotas or give the Snapshots worth keeping a higher snapshotPriority.

  • A scoped request fails with 503 and a preparation timeout

    The Snapshot is still preparing. Retry after the Retry-After delay, or raise snapshot.preparation.scoped_operation_timeout. A tenant with more Snapshots to prepare than snapshot.preparation.max_concurrent queues the rest.

Last updated on