Metrics

With observability.metrics.prometheus.mode set to "enabled", Andromeda serves Prometheus metrics at /metrics on the management port. This page lists every metric it emits, what its labels hold, and which ones to alert on.

Metrics are off by default, and need management.mode set to "separate-port" or "shared-port". Every name starts with andromeda_, and every metric carries a # HELP line. Durations end in _duration_seconds and sizes in _bytes, and everything specific to PostgreSQL starts with andromeda_postgresql_.

Label values come from fixed lists, apart from tenant (the tenant name), the HTTP endpoint (a route template) and the HTTP status (a status code). An outcome label is ok or error, and cancelled where an operation can be cancelled.

A gauge or counter stays in the scrape for as long as the value it reports holds, even when nothing changes it for hours. Only the pool, table and history chunk series carry a tenant label, and they are the only series that ever leave the scrape. They go five minutes after their tenant is unloaded.

Scraping

The scrape answers in the Prometheus text format, with Content-Type: text/plain; version=0.0.4. Point Prometheus at the management port, never the data port.

scrape_configs:
  - job_name: andromeda
    metrics_path: /metrics
    static_configs:
      - targets: ["andromeda:1086"]

With management.transport = "tls", add scheme: https and the CA under tls_config. In "shared-port" mode, the scrape is on the data port instead. Monitoring covers the management port and the layouts.

Metrics are kept in the broker's memory and start from zero when it restarts.

Build

• andromeda_build_info (gauge, version): always 1. version is the version of the running broker, so andromeda_build_info joined on instance shows which version each broker runs.

HTTP requests

These cover every request the API listener handles. In "separate-port" mode, the management port isn't counted. In "shared-port" mode, the scrape itself isn't counted.

• andromeda_http_requests_total (counter, endpoint, method, status): requests answered. method is the request method, or OTHER for a method HTTP doesn't define. endpoint is the route the request matched, such as /ngsi-ld/v1/entities/{entity_id}, never the raw path, and unmatched for a request no route matched. status is the response status code.
• andromeda_http_request_duration_seconds (histogram, endpoint, method, status): how long each request took to answer, from 5 ms to 10 s.
• andromeda_http_requests_pending (gauge, endpoint, method): requests in progress. A route that has had a request reads 0 once nothing is in progress.

Entities, history and subscriptions

• andromeda_entity_operations_total (counter, operation, outcome) and andromeda_entity_operation_duration_seconds (histogram, operation): entity operations and how long they took, from 1 ms to 10 s.
• operation is one of create, merge, replace, delete, retrieve, query, batch-create, batch-upsert, batch-update, batch-merge or batch-delete.
• outcome is ok or error.
• andromeda_temporal_operations_total (counter, operation, outcome) and andromeda_temporal_operation_duration_seconds (histogram, operation): Temporal API operations.
• operation is one of create-or-update, add-attributes, delete, delete-attribute, modify-attribute-instance, delete-attribute-instance, retrieve, query or query-ids.
• andromeda_subscription_operations_total (counter, operation, outcome): subscriptions created, updated and deleted. operation is create, update or delete.
• andromeda_subscription_candidates (histogram, stage): for each change, how many subscriptions were still in play after each step of matching.
• The stages run in order: active, routed (entity selectors), watched (triggers and watched attributes), evaluated, then matched.
• evaluated far below active means the routing index is doing its job.

Notifications

• andromeda_notification_deliveries_total (counter, outcome) and andromeda_notification_delivery_duration_seconds (histogram): HTTP and MQTT delivery attempts, and how long each took, from 1 ms to 10 s. ok means the endpoint accepted the notification. error covers a refusal, a timeout and an endpoint that couldn't be reached.
• andromeda_notification_oversized_total (counter, limit): notifications refused before sending because they were too large. limit says which limit refused them:
• max-http-body is notifications.delivery.max_http_body.
• max-mqtt-packet is notifications.delivery.max_mqtt_packet.
• mqtt-server-announced is a Maximum Packet Size the MQTT broker announced below max_mqtt_packet. Raising max_mqtt_packet doesn't help with this one, because the MQTT broker's own limit has to go up.

Queues and background tasks

• andromeda_queue_outcomes_total (counter, queue, outcome): what happened to each item claimed from a work queue in the tenant databases.
• queue is notification-deliveries, pending-side-effects (batches answered with Prefer: respond-async) or temporal-append-intents (history waiting to be written).
• outcome is completed (done), retried (failed, and scheduled for another attempt), exhausted (failed for the last time, and left the queue), lease-lost (its claim ran out before it was done, so another pass takes it), held-back (released without an attempt, to wait for an earlier item it must follow), or stuck.
• stuck is a history write, or the notifications of a respond-async batch, that failed again after temporal.history.stuck_after_attempts or entity.side_effects.stuck_after_attempts attempts. It stays queued and is retried, since what the broker accepted is never thrown away. Every such failure counts once, alongside its retried.
• andromeda_task_failures_total (counter, class, criticality): background tasks that failed for good, after their restarts ran out. The observed and tenant-degrading ones stay in the backgroundTasks component of the health report until the tenant runtime is evicted or the broker restarts. The metric labels keep the kebab-case spelling shown here, while the health report spells the same values in camelCase (tenantDegrading, invariantViolation).
• class is transient, permanent or invariant-violation.
• criticality says what the failure did: observed (listed in the backgroundTasks component of the health report, which stays up), tenant-degrading (that component is degraded), tenant-fatal (every task of the tenant was stopped) or process-fatal (the broker's own tasks were stopped too).

Database

• andromeda_postgresql_statement_duration_seconds (histogram, statement) and andromeda_postgresql_statement_failures_total (counter, statement): how long the hottest statements took, from 0.5 ms to 10 s, and how often they failed.

statement names the kind of statement, never its SQL.

• Entity writes: entity-insert, entity-update, entity-delete.
• The notification queue: notification-delivery-claim, notification-delivery-acknowledge, notification-delivery-hold.
• The queue of respond-async batches: pending-side-effect-claim, pending-side-effect-release.
• The history queue: temporal-intent-claim, temporal-intent-retry, temporal-intent-release, temporal-intent-release-probe, temporal-intent-application.
• Entity creation: entity-evolution-withdrawal, which withdraws the Temporal Evolution a creation established.
• Identity row locks: entity-row-lock. Before a commit or a history application runs any of its writes, it locks every identity row it is about to write, in identifier order. For a commit, the same statement also reads the version that a write leaving current state unchanged is held to.
• Temporal queries: temporal-candidate-entities (which entities a query selects), temporal-candidate-entity-count, temporal-history, temporal-scope-history, temporal-aggregation, temporal-aggregation-shape-check (whether the instances to aggregate can be aggregated).
• Temporal queries over entities a context source contributed to: temporal-source-filter-evaluation, temporal-source-window-start-instances, temporal-source-window-start-scope, temporal-source-window-relationships.

The pool and table series below describe each tenant's database.

• andromeda_postgresql_pool_connections and andromeda_postgresql_pool_idle_connections (gauges, tenant): open and idle connections in one tenant's pool, sampled every five seconds. Open connections at the pool's size (tenancy.default_tenant.max_connections or tenancy.databases.max_connections) with few idle means requests are waiting for one (see Database connections).
• andromeda_postgresql_table_bytes and andromeda_postgresql_table_index_bytes (gauges, tenant, table): on-disk size of entities, entity_search, current_attributes, subscriptions and subscription_delivery_state, and of their indexes.
• These are sampled once a minute, and only when the tenant has been written to since the last sample. Between samples, they keep the last value while the tenant stays loaded.
• andromeda_postgresql_table_updates_total and andromeda_postgresql_table_hot_updates_total (counters, tenant, table): rows the same tables had updated, and how many of those updates were heap-only (they wrote no index entry). They are read from PostgreSQL's table statistics on the same samples, and a statistics reset never moves them backwards.
• rate(andromeda_postgresql_table_hot_updates_total[1h]) / rate(andromeda_postgresql_table_updates_total[1h]) is the heap-only share of the last hour's updates, and has no value for a table nothing updated.
• andromeda_postgresql_history_largest_rowstore_chunk_bytes (gauge, tenant, table): the largest attribute_history chunk still waiting for columnstore conversion. It is 0 without a columnstore or with nothing to convert. Alert at about 4 GB (see Maintenance).

The tenant label holds the tenant name, and @default for the default tenant. A tenant name starts with a letter or an underscore, so no named tenant shares that value, not even one called default. These series stop being updated when a tenant is unloaded, and they are dropped five minutes later.

PostgreSQL maintenance

Maintenance explains what each of these tells you and when to act. None carries a tenant label.

• andromeda_postgresql_maintenance_decisions_total (counter, decision): what each maintenance pass decided. The values are disabled, global-lock-unavailable, no-tenant, tenant-unavailable, schema-behind, idle, probed, queued, would-act, deferred, acted, action-failed and cancelled.
• andromeda_postgresql_maintenance_actions_total (counter) and andromeda_postgresql_maintenance_action_duration_seconds (histogram), both by action and outcome: actions taken and how long they ran, from 0.1 s to an hour.
• action is reindex, vacuum, analyze, recovery, recompress or repack.
• outcome is ok, error or cancelled.
• andromeda_postgresql_maintenance_reclaimed_bytes (histogram, action) and andromeda_postgresql_maintenance_removed_dead_tuples (histogram, action): what a successful rebuild or repack gave back, from 1 MiB to 64 GiB, and the dead rows a vacuum removed, from 100 to a billion.
• andromeda_postgresql_maintenance_repack_phase_seconds (histogram, phase): how long each phase of a table repack took, from 0.1 s to six hours. phase is setup, copy, index-build, replay, swap, drop or analyze. Repacking tables says what a slow phase means.
• andromeda_postgresql_freeze_age_max, andromeda_postgresql_multixact_age_max and andromeda_postgresql_freeze_pressure_databases (gauges, the last by level): the oldest transaction and multixact ages across the tenant databases, and how many databases are at normal, overdue and critical.
• andromeda_postgresql_autovacuum_workers_busy, andromeda_postgresql_autovacuum_workers_max and andromeda_postgresql_overdue_relations (gauges): autovacuum workers in use and allowed, and how many tables the latest tenant visit found overdue for a vacuum.
• andromeda_postgresql_replication_slot_max_retained_wal_bytes (gauge): the most WAL any replication slot keeps on disk. 0 without a slot.

Tenants

• andromeda_tenant_runtimes_cached (gauge): tenants loaded right now.
• andromeda_tenant_cold_builds_total (counter, outcome) and andromeda_tenant_cold_build_duration_seconds (histogram): tenants loaded from cold, and how long loading took, from 10 ms to a minute. outcome is ok or error.
• andromeda_tenant_build_admission_wait_duration_seconds (histogram): how long a tenant waited for its turn to load, from 1 ms to 10 s.
• andromeda_tenant_requests_shed_total (counter, wait): requests answered 503 because a tenant could not be reached or loaded in time, by where they waited. Each also logs one warn line, the request was shed as saturated, with tenant, wait and waited_seconds. wait is:
• build-lock: another request was loading the same tenant.
• build-permit: other tenants held every loading slot.
• teardown-slot: every slot for closing a loaded tenant was busy, so no room could be made.
• provisioning: a new tenant's database was still being created. Creating it goes on after the request is answered.
• runtime-build: the tenant was still being loaded. Loading it goes on after the request is answered.
• registry-read or registry-write: the platform pool for the tenant's registry row.
• registration-transaction: the platform pool for registering a new tenant.
• migration-pool: the connection a new tenant's migrations run over.
• runtime-pool: a connection the load acquires.
• andromeda_live_changes_abandoned_total (counter): committed entity changes that got no notification because a stopping broker, or a tenant being unloaded, ran out of notifications.evaluation.drain_timeout before evaluating them. Each such stop also logs one warn line, stopping tenant gave up evaluating committed changes, with abandoned. Only live delivery can abandon changes.
• andromeda_tenant_evictions_total (counter, reason): tenants unloaded.
• reason is idle, excess (the cache was full), dormant (its next work is scheduled for later), orphaned (its registry row was removed) or suspended.
• andromeda_tenant_wakes_total (counter, outcome): attempts to load a tenant for its scheduled work.
• outcome is already-cached, loaded, nothing-due, deferred, build-failed or unregistered.
• A wake pass claims only as many due tenants as it has places for. A place is either free or held by a loaded tenant the pass may unload, which means no request is using it and it is past tenancy.residency.woken_lifetime or its due work has already run. When there is no place at all, the pass claims nothing and waits for one to free up, and the tenants it left keep their turn.
• A wake is deferred only when a request took the place in the moment between the pass counting it and the load. The wake is retried as soon as one of the woken tenants may give up its place.
• A wake is build-failed when the tenant could not be loaded, and is retried tenancy.residency.min_wake_interval later.
• andromeda_tenant_overdue_wakes (gauge) and andromeda_tenant_overdue_wake_max_age_seconds (gauge): wakes overdue as the last wake pass ended, and how long ago the oldest of them fell due. A wake is overdue when it was deferred, or when it was due and a pass left it for a later one, either because the pass had already loaded as many tenants as tenancy.residency.max_wakes_per_pass allows or because it had no place for it. Both are zero while the broker loads tenants as fast as their work falls due. While any wake is overdue, the broker also logs a warn line at most once a minute, tenant wakes are overdue: more tenants have work due than the broker can load, with overdue and oldest_overdue_seconds.

Tenants explains loading and unloading.

Name dictionaries

Each loaded tenant caches its attribute names and its entity type names, up to database.name_cache.max_size_per_dictionary each. These series add up every loaded tenant's caches, and dictionary is attribute-names or entity-type-names. Name caches explains the caches.

• andromeda_name_dictionary_lookups_total (counter, dictionary, outcome): names looked up in a cache, on reads and writes. outcome is hit (the cache held the name) or miss (the name was read from the database). A miss rate that stays high means the cache is too small for the names in use.
• andromeda_name_dictionary_evictions_total (counter, dictionary, cause): names dropped from a cache. cause is size (the cache was full) or idle (unused for database.name_cache.idle_eviction_after). An evicted name is read from the database the next time it is needed.
• andromeda_name_dictionary_entries (gauge, dictionary): names the caches hold right now.
• andromeda_name_dictionary_insertions_total (counter, dictionary): names added to a dictionary for the first time. A name another writer added first isn't counted again, and a write that is rolled back still counts the names it added.

Federation

• andromeda_federation_requests_total (counter, operation, outcome) and andromeda_federation_request_duration_seconds (histogram, operation): requests sent to context sources, and how long they took, from 1 ms to 10 s.
• operation is query, retrieve or provision (a write forwarded to a source).
• andromeda_federation_upstream_status_total (counter, status): what the sources answered. status is the class of the HTTP status code: 1xx, 2xx, 3xx, 4xx or 5xx, and other for a code outside them.

What to watch

• The backgroundTasks component of /ngsi-ld/v1/health being degraded, or andromeda_task_failures_total with criticality="tenant-degrading": a task of a tenant, or of the broker, stopped for good and its work is not running. Queued deliveries stay queued.
• andromeda_task_failures_total with criticality="tenant-fatal" or "process-fatal": a tenant's background work has stopped, or the broker's own has.
• andromeda_queue_outcomes_total with outcome="stuck" or "exhausted": a queue item keeps failing, or gave up.
• For notification-deliveries, an endpoint is down or refusing.
• For pending-side-effects, the notifications of accepted batches are late. A stuck job keeps being retried, and exhausted means a job whose stored changes could not be read back.
• A stuck history write keeps being retried, so the history of an accepted write is late, and the entity's later history waits behind it. Deleting that entity's Temporal Evolution removes the stuck write along with the rest of its history.
• rate(andromeda_notification_deliveries_total{outcome="error"}[5m]): endpoints refusing notifications. The subscription's lastFailure and notification.status show which one (see Delivery status).
• andromeda_postgresql_statement_failures_total rising: the database refuses statements. See Store outages.
• andromeda_tenant_cold_builds_total{outcome="error"}: a tenant couldn't be loaded, so the request or scheduled work that needed it failed.
• andromeda_tenant_overdue_wake_max_age_seconds growing: more tenants have work due than tenancy.residency.max_resident lets the broker load, and periodic notifications fall behind without bound. Raise max_resident. Lowering tenancy.residency.woken_lifetime won't help, because while every place is taken a woken tenant already gives up its place as soon as its work has run. Subscriptions and notifications says when this happens.

Configuration

Key	Default	What it does
observability.metrics.prometheus.mode	"disabled"	"enabled" installs the Prometheus recorder and serves /metrics. While it is "disabled", nothing is recorded. It needs management.mode to be "separate-port" or "shared-port"