Monitoring

Andromeda serves its liveness and readiness probes and its Prometheus metrics on a separate management port, 1086. The NGSI-LD API stays on 1056. The management endpoints keep answering when the API is swamped or draining, which is exactly when you need them. You can also firewall them off from application traffic.

The probes and the scrape answer from their own listener. Setting it up is a configuration choice, covered under Choosing a layout.

The two ports

The data plane is the NGSI-LD API. It listens on port 1056 by default and serves everything under /ngsi-ld/v1, including its own probes and version.

• /ngsi-ld/v1/health and /ngsi-ld/v1/ready: the full health report, 503 when an indicator is down.
• /ngsi-ld/v1/live: 200 while the process answers, whatever its dependencies say.
• /ngsi-ld/v1/version: the broker version.

The management plane listens on port 1086 by default. It serves the probes at their conventional paths.

• /livez: liveness. It checks nothing and answers 200 while the process is up.
• /readyz: readiness. It answers 200 when every health indicator is Up or Degraded, and 503 when any is Down.
• /metrics: the Prometheus scrape, when enabled.

The management listener runs its own accept loop with a bare request stack. It handles request ids, redaction of sensitive headers, request tracing, a request timeout and trailing-slash handling. A scraper or a probe never has to deal with CORS, a CSRF token or a Content Security Policy.

Because the loop is separate, the probes and the scrape still answer while the API is saturated or shutting down.

A degraded indicator still gets 200 from /readyz. You might expect a slow database to take the broker out of rotation, but only a Down indicator does that. A degraded store keeps the broker in service and shows up in the health report.

/livez checks nothing, so a stuck database never makes the orchestrator restart the process.

Checking readiness from the command line

andromeda health asks a running broker whether it is ready. It exits with status 0 when the broker answers with success and 1 otherwise, so a script or a container healthcheck can act on the exit code alone. The command reads the same configuration the broker does and probes whichever endpoint answers readiness in your layout. A ready broker prints Broker ready: followed by the URL it probed, and a broker that is not ready gets the reason on standard error. The broker image uses this command as its healthcheck, and the broker image page describes which endpoint it picks and how it handles TLS.

The health report

/ngsi-ld/v1/health lists one entry per component under healthCheck.components: postgresql, mqtt when the default tenant delivers over MQTT, and backgroundTasks.

The backgroundTasks component covers the broker's own tasks and the tasks of every tenant runtime that is loaded. A task stops for good when its failures outlast its restarts. Its criticality then decides what happens:

• A tenant-fatal or process-fatal task stops every task of its tenant, or the broker's own tasks as well, and is not listed in the component.
• tenantDegrading leaves the other tasks running. The component is degraded for as long as any such task is stopped. Durable notification delivery, live notification events, context source notification events, temporal intent delivery and entity side-effect delivery are tenantDegrading. While the delivery drain of a tenant is stopped, its queued deliveries stay queued.
• observed is reported only. The task is listed and the component stays up.

/readyz answers 200 for a degraded component, so a stopped task does not take the broker out of rotation. Alert on the backgroundTasks status, or read the details:

"backgroundTasks": {
    "status": "degraded",
    "details": {
        "stoppedTasks": [
            {
                "name": "notification-delivery",
                "tenant": "@default",
                "criticality": "tenantDegrading",
                "failureClass": "invariantViolation",
                "failureSummary": "task execution stopped outside its runtime contract: task 42 panicked"
            }
        ]
    }
}

tenant is the tenant's name, @default for the unnamed default tenant, and absent for a task of the whole broker. failureClass is transient, permanent or invariantViolation, and failureSummary is the failure the task reported.

Evicting a tenant's runtime drops its stopped tasks from the report. The tasks run again when the tenant loads, and the component returns to up. A stopped task of the unnamed default tenant stays until the broker restarts, because that tenant is never evicted.

Choosing a layout

management.mode picks one of three layouts.

• "separate-port" (default): the management endpoints get their own listener on management.host and management.port. This is the one to firewall. Applications reach the data port, and only operators and Prometheus reach the management port.
• "shared-port": everything shares the data port. /livez, /readyz and, if enabled, /metrics sit at the root of the data listener, next to /ngsi-ld/v1. They then go through the API's full web edge, which is the price of one port. Use it for a single container where one exposed port is simpler and the isolation isn't needed.
• "disabled": no management endpoints at all. The probes under /ngsi-ld/v1 stay, but /livez, /readyz and /metrics are gone, and metrics can't be turned on.

Configuration

[management]
mode = "separate-port"
host = "0.0.0.0"
port = 1086
transport = "plaintext"

[observability.metrics.prometheus]
mode = "disabled"

Key	Default	What it does
management.mode	"separate-port"	"separate-port", "shared-port" or "disabled", as above
management.host and management.port	"0.0.0.0" and 1086	Where the management listener binds. Read only in "separate-port" mode
management.transport	"plaintext"	"tls" makes the management listener terminate TLS with the broker's shared [tls] identity, client-certificate policy included. Read only in "separate-port" mode
observability.metrics.prometheus.mode	"disabled"	"enabled" installs the Prometheus recorder and serves /metrics. While it is "disabled", no recorder is installed and no scrape endpoint exists

Every key can also be set from the environment, with the ANDROMEDA__ prefix and a double underscore between levels, such as ANDROMEDA__OBSERVABILITY__METRICS__PROMETHEUS__MODE=enabled.

If it goes wrong

• The broker refuses to start, complaining about the management port

  In "separate-port" mode the management port can't be 0, and it can't be the same as the data port. The two are compared on the port alone, because a wildcard host overlaps every host.

• The broker refuses to start with metrics turned on

  observability.metrics.prometheus.mode = "enabled" needs somewhere to serve them, so it doesn't work with management.mode = "disabled". With "shared-port" it's fine, because the scrape uses the data port.

• /readyz answers 503

  A health indicator is down. /ngsi-ld/v1/health on the data port gives the full report, with each component and its status.

• /ngsi-ld/v1/health reports degraded and /readyz still answers 200

  Look at backgroundTasks under components. A task stopped for good, and its entry names the task, the tenant and the failure. Fix the cause the summary gives. Restarting the broker runs every task again.

These checks run when the configuration is loaded, before any listener starts.