Troubleshooting

Most problems a running deployment meets come down to the connections between the broker and its database. Each entry starts with what you see, and the text under it says what to change.

Connections and load

• database_connection_capacity_exceeded is logged at warn when the broker starts

  The broker's pools, all full at once, would need more connections than the server lets its role open. Raise the server's max_connections or shrink the pools before the load gets there. Connection capacity shows how the broker counts.

• Requests get 503 with Retry-After: 1 under load

  A pool is full. Raise the pool that is short. If the cause is the number of loaded tenants, put PgBouncer in front.

• Tenants load slowly, again and again

  More tenants are in use than tenancy.residency.max_resident allows, so tenants keep being unloaded and loaded again. The andromeda_tenant_cold_build_duration_seconds histogram shows how often and for how long. Raise the limit if the server has the connections to spare, or move to PgBouncer.

Database outages

• Requests get 503 with Retry-After: 5

  The broker can't reach the database. After a PostgreSQL restart, a failover or a PgBouncer restart, clients get this until the database is back, and the broker's background work then resumes by itself. With PgBouncer, keep server_login_retry short, so a restarted PostgreSQL is picked up within a few retries. Store outages has the details.

Starting the broker

• The broker doesn't start after a configuration change

  Run andromeda configuration validate. It reports the same problem the broker stops on.

On each page

Every deployment page ends with the problems that belong to it.

• PostgreSQL

  Temporal queries that miss history, too_many_connections, migrations failing with 55P03.

• PostgreSQL image

  A server that exits at startup, a password that doesn't change, exec format error.

• PgBouncer

  Tenants that wait for a connection, steady 503 under load, a broker that can't connect through it.

• Security

  Tenants that see each other's data, notifications blocked by an egress policy, @context documents that can't be fetched.

• TLS

  Stored URLs that start with http://, a rotated certificate that isn't served, a broker that won't start with TLS on.

• HTTP versions

  HTTP/2 that never starts, uploads refused with 411, clients that never switch to HTTP/3.

• Browser clients

  A CORS change the broker refuses, writes refused with 403, headers a script can't read.