Skip to main content

Store outages

When PostgreSQL restarts, fails over or drops off the network, Andromeda answers the requests that need it with 503 and a Retry-After header. It picks up again by itself once the database is back. Clients get a clear "try again shortly", never a 500, and nothing is lost in the background.

Every write Andromeda offers is safe to repeat, so a client can always retry a 503.

What clients get​

Three database conditions are answered with 503. Each has its own delay in Retry-After.

  • Database unreachable or connection lost: Retry-After: 5, detail the datastore is temporarily unavailable; retry shortly.
  • No connection free in time: Retry-After: 1, detail the datastore connection pool is saturated; retry shortly.
  • Write rolled back under contention on every attempt: Retry-After: 1, detail the operation was rolled back by a concurrent transaction; retry shortly.

The second case covers a request that waited past database.acquire_timeout, and one the pooler's query_wait_timeout disconnected. Keep database.acquire_timeout below request_limits.timeout, so a full pool shows up as a quick 503 rather than a slow 500. A read request only counts the connections reads may use (see Database connections).

HTTP/1.1 503 Service Unavailable
Content-Type: application/json
Retry-After: 5

{
    "type": "https://uri.etsi.org/ngsi-ld/errors/ServiceUnavailable",
    "title": "Service unavailable",
    "status": 503,
    "detail": "the datastore is temporarily unavailable; retry shortly"
}

The type is ServiceUnavailable and the status 503, because the condition will most likely clear after a short wait. InternalError is kept for 500, a failure that retrying won't fix, so a client can base its retry policy on the type as well as the status.

Why five seconds for an outage and not one? A restart, a failover or a pooler reconnect takes seconds. PgBouncer also remembers a failed login to PostgreSQL for server_login_retry, 15 seconds by default. Telling clients to come back after one second would only pile load onto a database that is trying to come back.

A load balancer or client that retries a 503 straight away just gets another 503. The Retry-After hint is what paces clients.

A connection setting the driver can't use at all is a deployment mistake, not an outage. That gets a 500.

Writes that may or may not have happened​

If the connection drops after the COMMIT went out but before its answer came back, nobody knows whether the write landed. The client still gets 503, because there is nothing more honest to say. The client is the one who can decide to send it again.

That is safe. Upsert, update, replace, merge and delete give the same result when repeated. A repeated Create Entity answers 409 AlreadyExists, which tells the client the first attempt got through. The answer comes from the rows in the database.

Andromeda never repeats such a write by itself. It only re-runs a transaction the database rolled back under contention, which is known to have left nothing behind.

Federated requests​

Every distributed operation first reads the tenant's context source registrations from Andromeda's own database. If that read fails, the request gets the same 503 as a local one.

This is different from a context source failing. A read whose forwarding fails at a source still answers with what Andromeda holds, plus a warning. But if Andromeda can't even read its registrations, answering with local data alone would hide every source without a word. So it answers 503 instead.

Background work​

Durable notification drains, expiry sweeps, Snapshot preparation and the temporal history queue all treat a lost database like a busy one. The pass fails for now. The rows it had claimed are released when their lease runs out. The broker carries on with them once the database is back.

Nothing is dead-lettered or quarantined because of an outage.

A notification the drain has already sent is the exception to giving up at once. A released row would be sent again, so the drain keeps asking the database to record the outcome, with a wait that grows from a tenth of a second to two seconds, until the lease would run out. Each failed request logs notification_delivery_acknowledgement_retried. Only if the database stays unreachable for the whole lease is the notification sent a second time.

What to watch​

  • persistence adapter unavailable at warn: logged once at each statement that met an outage, with the driver's own message as cause. A burst of these is the outage, and they stop when the database is back.
  • Saturation and contention at debug: a full pool and a rolled-back write are load rather than faults, so they log at debug.
  • Configuration faults at error: a connection setting the driver can't use. This won't fix itself.

The driver's own text never reaches a client.

If it goes wrong​

  • A wave of 503 with Retry-After: 5

    The database is unreachable. Check that PostgreSQL is up and reachable from the broker, and that PgBouncer can log in to it, if you use one.

  • Steady 503 with Retry-After: 1 while the database is up

    The pool is too small for the load, or requests hold connections for too long. See Database connections.

  • A 500 while the database is up

    Look for error lines. A connection setting the driver can't use is the usual cause.

Which driver errors count as the database being unreachable
  • SQLSTATE 57P01 (the backend was terminated), 57P02 (a crash reset) and 57P03 (the server is starting or shutting down).
  • Every SQLSTATE in class 08 (connection exceptions). PgBouncer also reports server login has been failing this way while it can't log in to PostgreSQL. Its query_wait_timeout disconnect shares SQLSTATE 08P01, but it counts as a full pool.
  • An I/O failure on the connection: refused, reset, aborted, broken pipe, or an unexpected end of stream. Running out of local sockets on the broker's host counts as load, not an outage.
  • A TLS or protocol failure on the connection, a closed pool, a crashed connection worker, and a transaction that couldn't begin.

Where Andromeda interprets the specification​

The specification allows the HTTP binding's own status codes alongside its error table, which binds InternalError to 500. It names no type for a 503, so ServiceUnavailable is Andromeda's own, published under the ETSI errors namespace like the gateway types. RFC 9110 section 15.6.4 defines 503 for a condition that "will likely be alleviated after some delay", and Retry-After for how long. The warning for a failed source is clause 6.3.17.

Last updated on