Skip to main content

Choosing a setup

Every Andromeda deployment has two parts, the broker and one PostgreSQL server with PostGIS and TimescaleDB. Around them you can add a reverse proxy or API gateway in front of the broker, PgBouncer between the broker and the database, an MQTT broker for MQTT notifications, and other NGSI-LD brokers as context sources. Most of the decisions come down to how the broker reaches the database, where TLS ends, what the broker keeps and for how long, and which of the optional parts you add.

Andromeda v1.0 runs one broker process per database. Running several against the same database is not supported yet, as Replicas explains, so this page assumes one.

The Deployment planner turns the decisions on this page into files. Simple asks about a dozen questions and settles the rest as the docs recommend, Advanced asks every question and ends in a browser over every setting, and Check my configuration reads a configuration you already have. A plan gives the steps for your case, each with the page that covers it, the warnings that apply, a configuration file, an environment file and the files to run the broker with Docker Compose or Docker behind nginx, Caddy or Traefik.

You can also write the configuration yourself. Configuration lists every setting and explains how the file, the environment and the flags combine. Every setting except database.connection has a default, so a broker that is told where its database is runs with no other configuration.

How many tenants​

A tenant is a separate set of data with its own database, and clients pick one with the NGSILD-Tenant header. The number of tenants decides how the broker spends its connections to PostgreSQL, so it shapes more of the setup than any other answer.

With only the default tenant, turn tenants off with tenancy.mode = "disabled". The broker then refuses the header with 501, so no request can create a tenant database by naming a new one. With tenants on, the first write to a new tenant name creates it, and nothing checks that the caller may.

With a few tenants, up to a few dozen, leave the defaults. The broker keeps up to tenancy.residency.max_resident tenants loaded (16 by default), and each loaded tenant has a pool of up to tenancy.databases.max_connections connections (4 by default). The default tenant has its own pool of tenancy.default_tenant.max_connections (32). All of these pools together must fit in the server's max_connections.

With hundreds of tenants, direct pools no longer fit. Run PgBouncer in transaction mode and give its address in database.pooler.connection. A loaded tenant that isn't serving a request then holds almost no server connections, so tenancy.residency.max_resident can grow to the number of tenants in use at once, and database.pooler.max_connections_per_tenant (2 by default) can stay small. database.connection keeps naming PostgreSQL itself, because creating tenants, migrations and maintenance need a direct connection.

Whatever the count, a request that names a new tenant creates and migrates its database before the operation runs, even if the operation is then refused. A tenant whose database still holds nothing after tenancy.provisional_tenant_lifetime (10 minutes) is dropped again. Until then, a client that keeps sending made-up names costs you one database per name, which is another reason to turn tenants off or to restrict the header at the proxy.

No API call deletes a tenant. To remove one, suspend it and drop its database yourself. Tenant databases are named from the tenant name and tenancy.databases.name_prefix (andromeda_tenant by default), and a tenant nobody uses is unloaded after tenancy.residency.idle_eviction_after (15 minutes).

Where PostgreSQL is and how many connections it allows​

The broker keeps all its data in one PostgreSQL server with PostGIS and TimescaleDB. PostgreSQL lists the versions and extensions the server needs, the rights the broker's role needs, and what the broker sets on each database itself. You can run the PostgreSQL image published for Andromeda, a server of your own, or a managed database that meets those requirements.

database.connection has no default, and the broker refuses to start without it. It takes a connection URL or a table of its parts, host, port, user, password, database and parameters. Next to the broker in Docker Compose, the host is the service name. On a separate server or a managed database, use its address and a role made for the broker. That role creates a database for each named tenant and installs the extensions in it, so it is normally a superuser. To keep the password out of the configuration file, write the other parts in the file and pass the password in ANDROMEDA__DATABASE__CONNECTION__PASSWORD, from a secret store. It then needs no escaping, as it would inside a URL. ANDROMEDA__DATABASE__CONNECTION takes the whole URL instead.

Every pool counts against the server's max_connections. Check the value with SHOW max_connections, then compare it with what these pools can hold at once:

  • the platform pool, database.platform_max_connections (32)
  • maintenance and tenant loading
  • without PgBouncer, the default tenant's pool and every loaded tenant's pool

If they don't fit, lower the pool sizes, lower tenancy.residency.max_resident, raise the server's limit, or add PgBouncer, and leave room for your own tools. Connection capacity has the sums, and the postgresql component of GET /ngsi-ld/v1/version reports the result as a verdict.

When a pool is full, a request waits up to database.acquire_timeout (5 seconds) for a connection and then gets 503 with Retry-After: 1. That timeout must stay below request_limits.timeout, or the broker refuses to start.

If heavy reads run beside steady writes, raise database.write_reserve_percent (25 by default). Reads never take a whole pool, because the reserve keeps that share of it for writes, so writes don't queue behind a crowd of readers.

Each loaded tenant also caches the attribute and entity type names it uses, in two caches with a fixed memory budget each, and a name missing from a cache costs a database read. Ordinary data models fit the default of 16 MiB. The worst case is the loaded tenants times two caches times the budget, so cap database.name_cache.max_size_per_dictionary when there are many tenants or names you don't control, and raise it when software generates the names. For a batch that runs in rare bursts, set database.name_cache.idle_eviction_after to "unlimited". The names then survive the quiet hours, and only the budget evicts them.

What sits in front of the broker​

The broker serves the API on server.host and server.port, 0.0.0.0:1056 by default, in plaintext. Its probes are on a separate management port, 1086. Clients reach the broker either through a reverse proxy, such as nginx, Traefik, Caddy or a cloud load balancer, or straight on port 1056.

A proxy is optional, but most setups use one. It gives the broker a public address, can terminate TLS, and is where per-caller rules go. Without a proxy, the broker listens where clients connect, and a firewall should limit who can reach port 1056. Behind a proxy, the broker should listen only on a network the proxy reaches.

Either way, anyone who reaches the API can call every operation in every tenant, because the broker has no rules per caller. Rules such as who may write, or who may use which tenant, belong in the proxy. Without a proxy, keep the broker on a network where every client is trusted. Security covers this.

TLS termination​

TLS is what turns a connection into HTTPS, and TLS termination is where one component holds the certificate and decrypts the traffic. The broker can terminate TLS itself, or a proxy can do it and pass the broker plain HTTP over a network you control. Terminating at the proxy is the usual choice when there is a proxy anyway.

Terminate in the broker when there is no proxy, when the proxy passes encrypted traffic through unchanged, or when the broker must check client certificates. Terminate nowhere only if the whole path stays on a network you trust, such as a closed plant network or a test setup, because everything then crosses it as plain text. TLS has the settings.

Client certificates, also called mutual TLS, make each client present a certificate from your CA, and the broker refuses connections without one (tls.client_authentication.mode). A certificate decides who can connect, not what they may do. Use "required" when your devices or services already carry certificates from your own CA, and "optional" while you move clients over.

HTTP/3 runs over UDP and copes better with lossy or mobile networks. Every NGSI-LD client works over HTTP/1.1 or HTTP/2, so turn HTTP/3 on only for clients that want it. It always terminates TLS in the broker, even when a proxy handles TLS for TCP, so the broker needs a certificate of its own and the API port must be open for UDP. See HTTP versions.

The address clients use​

Behind a TLS proxy the broker receives plain HTTP, so on its own it would assume clients use http://. It writes its own address into data it stores, such as @context links, and those would point clients at the wrong scheme. You can tell it the right address in two ways.

A fixed public address, server.public_url, is the simpler and safer way. Use it when everyone reaches the broker under one name. The broker builds every absolute URL on it, and http.forwarded.trust stays at "none". The address is the proxy's, not the broker's own port, and it includes the path prefix if the proxy serves the broker under one, such as https://192.0.2.10/context-broker. A path prefix needs the fixed address, and so do subscriptions forwarded to context sources, because they name it as the place to send their notifications back to. Without server.public_url, subscriptions are not forwarded.

The forwarded headers are the other way, for clients that reach the broker under several names, such as an IP address inside your network and a host name outside it. Set http.forwarded.trust to "proxies" and list the proxy's addresses in http.forwarded.proxies, as the broker sees requests arrive from them. That is the proxy's own address, or the subnet its replicas run in. The broker then believes Forwarded, X-Forwarded-Proto and X-Forwarded-Host only from those peers. The proxy must overwrite those headers on every request rather than append to them, or a client could choose the scheme and host of the URLs the broker stores. Trusting forwarded fields safely explains it.

Browser applications​

Browsers stop a web page from calling an API on another address unless the API allows it (CORS), and they need protection against requests forged by other sites (CSRF). Neither matters for servers, scripts, IoT agents or devices, or for a web application that calls its own backend, which then calls the broker.

When JavaScript in the user's browser calls the broker directly, replace the permissive CORS default with an explicit list of origins, and turn on CSRF protection for the same origins. An origin is the scheme, host and port the application is opened at. A proxy in front must pass Origin and Host on unchanged, or the CSRF check refuses requests it should allow.

Allow credentials only if the browser attaches cookies or stored credentials by itself. An application whose code sets its own headers, such as Authorization, doesn't need it.

Strict-Transport-Security tells browsers to use only HTTPS for the broker, for 180 days by default. Turn it on once HTTPS works and clients use a host name, because browsers ignore it for an IP address. Extend it to subdomains only when all of them serve HTTPS, since it is hard to undo. The broker sends it only on its own TLS listener, so behind a TLS proxy the proxy adds it. See Browser clients.

Callers you don't fully trust​

A subscription or a registration holds an address the broker will call later, and a @context URL in a request makes the broker download it. A caller could aim any of them at an internal service that the broker can reach and they cannot. When every client is yours, nothing more is needed.

When partners, customers or the public can make the broker call out, set these to "refused":

Each one takes allowed_addresses and denied_addresses lists beside it for the exceptions, such as a server in your own network that hosts your contexts. Limit the broker's outgoing traffic with a firewall as well, and let only trusted callers create subscriptions and registrations at the proxy.

The broker downloads a @context the first time it sees it, which is fine when it can reach the internet and callers are trusted. On an offline or locked-down network, or with callers you don't trust, set jsonld.fetch.mode = "disabled" and preload a local copy of every @context your clients use, each as a [[jsonld.preloaded_contexts]] table with its url and path. A request naming any other remote @context is then refused.

What the broker keeps​

A subscription makes the broker send a notification whenever matching data changes. By default notifications are sent from memory and a failed one is only recorded, which suits dashboards and live views where the next change will do. When a notification triggers something that must not be missed, such as an alarm, billing or copying data to another system, set notifications.delivery.mode = "durable". Each notification is then queued in the database with the write that raised it and retried with a growing wait. After notifications.delivery.max_attempts (5) failures it is dead-lettered and kept for 7 days as a record of what never arrived.

A write is confirmed to the client once PostgreSQL has made it durable. database.synchronous_commit ("on" by default) is set on every database the broker uses. Leave it on. With "off", a server crash can lose writes the clients were already told had succeeded.

Temporal history is every past value of each attribute, read through the Temporal API. If only the latest values matter, set temporal.history.core_writes = "not-recorded", so Core API writes change only the current state. If a separate process loads history itself through the Temporal API, set the same, or everything is stored twice.

To keep history, decide for how long. Kept forever, it needs the disk to hold it. To set a limit, set temporal.storage.retention.mode = "enabled" and temporal.storage.retention.delete_after, a whole number of days. History is deleted in whole partitions once all of a partition is older than the limit, so a little more than the limit is kept for a while. Partitions older than temporal.storage.columnstore.convert_after (30 days) are converted to the columnstore before that, when TimescaleDB is the TSL edition.

Context sources​

Andromeda can be linked to other NGSI-LD brokers or data services, called context sources, and answer queries with their data as well. A registration says what a source covers and where it is. Each forwarded request gets federation.forwarding.connect_timeout (3 seconds) to connect and federation.forwarding.request_timeout (5 seconds) to answer, and one operation has at most federation.forwarding.max_concurrent (16) forwarded requests in flight. A registration's own management.timeout takes precedence over the request timeout.

Three choices matter in a deployment:

Registrations that set management.cacheDuration also use a response cache in each tenant, of up to federation.response_cache.max_entries answers (1024).

Geoqueries​

Each broker process can keep an in-memory index of entity locations in each tenant, to narrow a geoquery on location before PostgreSQL runs it. Answers are exact in every mode, because the index only proposes candidates and PostgreSQL checks each one against the full query. entity.spatial_index.consistency decides what the index costs.

  • "watermark", the default, catches the index up with the database before it narrows a query. It is exact in every deployment, at the cost of one indexed read per geoquery.
  • "commit-synchronous" updates the index with every committed write, which saves that read, but it is exact only when exactly one broker process writes the database. Another process's writes reach the index only at the next rebuild, up to entity.spatial_index.rebuild_interval (30 minutes) later.
  • "store-verified" keeps no index at all, and PostgreSQL answers every geoquery with its own GiST indexes. It is exact everywhere and uses the least memory, because a tenant allocates nothing for spatial lookups and a tenant that isn't loaded yet loads without reading its locations.

The index is the only part of a broker process whose memory grows with the number of entities. Plan about 1 GiB per million located entities in each tenant, rebuilds included, and see Spatial index for the sums. Choose "store-verified" when memory matters more than the extra read, and "commit-synchronous" only when you are sure there is one writer.

Batches and Snapshots​

A batch of entity.batch.large_threshold entities (512) or more is cut into chunks of entity.batch.chunk_size entities (512), and a few chunks are prepared and written at once (entity.batch.max_concurrent_chunks and entity.batch.max_concurrent_chunk_writes, both 4).

A client that sends Prefer: respond-async can have a batch's notifications and temporal history handed to background workers, through a queue kept in the database and retried with a growing wait. By default every batch that sends the preference is handled this way. Set entity.batch.async_side_effect_threshold to limit it to batches of that size and up.

A Snapshot is a stored copy of the result of a query. One created without snapshotLifetime lives for snapshot.default_lifetime (1 hour), and a longer request is cut down to snapshot.max_lifetime (30 days). Neither the number of Snapshots nor the storage they take is limited unless you set snapshot.quota.max_per_tenant and snapshot.quota.max_storage_per_tenant. Set both when tenants share a disk or aren't trusted.

Optional extensions​

Andromeda's features beyond the NGSI-LD specification are opt-in. Each has its own mode and is off by default:

  • temporal delete-by-query
  • grouped temporal aggregation
  • linked entity retrieval on the temporal operations
  • datasetId and deleteAll on Purge Entities
  • tenant management
  • a near geoquery bounded from both sides
  • Accept-Language read as the language filter
  • language tags as the target language of a q bracket

Turn one on only when a client needs it. Tenant management in particular lets any caller list tenants and suspend them, so leave it off, or allow /ngsi-ld/v1/tenants only to operators at the proxy. Extensions describes each one.

Cursor pagination is the exception, and it is on by default. A request that names neither offset nor cursor still pages by offset, and a client opts in to a cursor walk with an empty cursor=. Set extensions.cursor_pagination.fresh_requests to "cursor" and such a request walks by cursor instead. With extensions.cursor_pagination.mode = "disabled", collections page by offset only and a cursor is refused.

Set extensions.mode = "conformance" to force every extension off, cursor pagination included, whatever their own settings say.

Expiry​

Sweeps every expiry.sweep_interval (30 seconds) delete what has passed its expiresAt, whether an entity, an attribute, a subscription, a registration, a Snapshot or an Entity Map. An expired resource may stay in storage for up to one interval, but the API never serves it, so a longer interval only delays getting the disk space back.

Limits on a request​

request_limits.timeout (30 seconds), request_limits.max_body (64 MiB) and the other request_limits keys bound what one request may cost the broker, and the defaults suit most deployments. They protect the broker from mistakes and oversized requests, not from a determined caller, so set rate limits and connection limits at the proxy too. Security lists them.

Responses are compressed with the codings in http.compression.algorithms, and request bodies decoded with those in http.decompression.algorithms. Both lists hold gzip, brotli, deflate and zstd by default, and an empty list turns either off. Bodies from http.body_decoding.blocking_threshold (256 KiB) up are parsed on a separate thread pool, so one large request doesn't hold up the others. Every response carries a Server-Timing header with its total time, unless http.server_timing.mode is "disabled".

Watching and maintaining it​

Probes tell an orchestrator such as Docker whether the broker is alive and ready to serve, on /livez and /readyz. Metrics are numbers over time, such as request rates and queue lengths, collected by Prometheus or something that scrapes like it. They are off by default. Set observability.metrics.prometheus.mode = "enabled" to serve them at /metrics.

Probes and metrics are not for clients, so by default they get the separate management port, 1086, which clients shouldn't be able to reach. It runs on its own, so probes keep answering while the API is overloaded or draining. The management listener can terminate TLS too, with the same certificate (management.transport). On a platform that gives each container a single port, set management.mode = "shared-port" to serve them on the API port. The proxy should then forward only /ngsi-ld/, so /metrics, /livez and /readyz stay internal. Monitoring and Metrics cover what to watch.

The broker always logs to standard output, at observability.log.level (info by default), which Docker and systemd collect. On a plain server where nothing collects it, the broker can also write JSON log files and rotate them (observability.log.file.mode), into a directory that exists and that it can write to.

PostgreSQL tables and indexes bloat over time and need vacuuming and rebuilding, and the broker does that itself by default. database.maintenance.mode is "automatic", so the broker samples every tenant database, warns about wraparound and held WAL, rebuilds bloated indexes, runs catch-up vacuums and recompresses history that changed. An action the database role has no right to take is only reported.

  • Set "observe" to sample and log what the broker would do without changing anything. That suits a database you maintain on your own schedule but want the warnings for.
  • Set "disabled" if a database administrator or a managed service does it all.
  • Set database.maintenance.profile to "responsive" when data churns a lot, with frequent updates and deletions, and you want smaller signs of bloat acted on sooner.

Repacking a bloated table is reported but not run by default, because it copies the table online and needs free disk of up to twice its size, which the broker can't measure. Once you know the disk has room, set database.maintenance.repack = "performed". A table larger than database.maintenance.max_repack_size (4 GiB) is never repacked. Maintenance lists the rights each action needs.

Stopping, upgrading and backing up​

When the broker is told to stop, it stops accepting connections and gives the requests in flight up to server.graceful_drain_timeout (30 seconds) to finish. With live delivery, it then spends up to notifications.evaluation.drain_timeout (30 seconds) evaluating the changes already committed, so their notifications still go out. The whole stop takes at most the two together, 60 seconds by default, however many tenants are loaded, and whatever still runs then is abandoned and logged. Give the broker the two together plus 10 seconds before your orchestrator kills it. Docker and Podman wait only 10 seconds by default, so set stop_grace_period: 70s in Compose, or --stop-timeout 70 with docker run. A broker killed early loses the live notifications it hadn't evaluated, and durable ones are sent after the restart.

The broker migrates each database when it opens it, so an upgrade is a restart on the new version. Take a backup first, because a newer broker's migrations can't be undone by an older one. With the published images, a tag of only the major and minor version, such as 1.0, picks up fixes when you pull, and a full version pins the exact release. Upgrades describes the procedure.

Every tenant has its own database on the one server, so back up the whole server, not one database. Use a physical backup tool with WAL archiving, as PostgreSQL describes.

Checking it works​

Check the configuration without starting the broker, with the same file and environment the broker will run with, since the database password usually comes from the environment:

andromeda configuration validate

It prints Configuration valid: and the file it read, or the first problem it finds. Every key can come from the environment as well as the file, such as ANDROMEDA__SERVER__PORT for server.port, and each part of a database connection has a variable of its own, such as ANDROMEDA__DATABASE__CONNECTION__PASSWORD. andromeda configuration generate -o andromeda.toml writes a file with every key and its default, and andromeda configuration show prints what the broker will use, with the database connections redacted. Check my configuration in the Deployment planner reads the same file and environment without leaving your browser, and lists unknown keys, refused values and broken rules all at once.

Then start the broker and ask the management port whether it is ready:

curl -i http://localhost:1086/readyz

A 200 means every health indicator is up or degraded. A 503 means one is down, often the database. Last, check the connection capacity from the API port:

curl http://localhost:1056/ngsi-ld/v1/version

The postgresql component should report "verdict": "sufficient". If it reports "verdict": "exceeded", raise max_connections on the server, or lower tenancy.databases.max_connections and tenancy.residency.max_resident.

What to watch​

  • database_connection_capacity_exceeded at warn

    Logged at startup when the pools, all full at once, would need more connections than the server lets the broker's role open. Size the server or the pools before load gets there.

If it goes wrong​

  • Requests get 503 with Retry-After: 1 under load

    A pool is full. Raise the pool that is short, or put PgBouncer in front if the cause is the number of loaded tenants.

  • Tenants load slowly, again and again

    More tenants are in use than tenancy.residency.max_resident allows, so tenants keep being unloaded and loaded again. The andromeda_tenant_cold_build_duration_seconds histogram shows how often and for how long. Raise the limit if the server has the connections, or move to PgBouncer.

  • The broker doesn't start after a configuration change

    Run andromeda configuration validate. It reports the same problem the broker stops on.

Last updated on