Skip to main content

Distributed

In a distributed deployment, part of the data stays where it already is, in a gateway at a site, a service next to the devices or a system you can't move. Andromeda registers each of these as a context source and answers for it. A client queries one broker and gets one set of entities, wherever each of them is held.

  1. ConsumersApplications, dashboards
  2. answers, notifications
  3. AndromedaDistribution broker, keeps the registrations
  4. answers to what the broker forwards
  5. Site gatewayEntities of one site
    Legacy adapterOne attribute family
    Another brokerWhole entities
AndromedaA system Andromeda doesn't run

When it fits​

  • The data has a natural home, and copying it into the broker would mean keeping two copies in step.
  • The sources belong to the same deployment, run by the same team or by teams close enough to agree on a data model.
  • Consumers shouldn't need to know where an entity lives.

When the sources are whole brokers run by other teams, federated fits better. The parts are the same, and what changes is who runs them.

What runs​

  • One Andromeda and its PostgreSQL server, as in a single broker. The broker can hold data of its own beside what the sources hold.
  • The sources. Each one needs to serve only the NGSI-LD operations its registration lists, and none of them has to be Andromeda, or even a broker.

The broker is its own registry. Registrations are stored in it, and a consumer can read them at GET /ngsi-ld/v1/csourceRegistrations and go to a source directly if it prefers. A consumer that doesn't care asks the broker and never learns which of the two answered.

What a registration decides​

A registration's mode decides how a source relates to the broker's own data. The Distributed guide goes through the four modes with examples.

  • inclusive, the default

    The broker and the source both hold data for the same entities, and their answers are merged.

  • auxiliary

    The source fills in only what nothing else supplies. It never overrides the broker, and it takes part only in reads.

  • exclusive and redirect

    The source owns the registered data. The broker holds none of it, and passes writes on to the source when the registration lists the write operations.

A registration also lists the operations its source accepts. By default only reads are forwarded, so a source never receives a write it didn't ask for.

What to set​

  • server.public_url

    Set it to the address the sources reach the broker at. Without it the broker forwards no subscriptions to its sources, and a subscription sees only what the broker itself holds.

  • federation.cooldown.mode

    The default, "disabled", holds back only registrations that set management.cooldown. With "exponential", a source that fails is skipped for twice as long after each failure in a row, and requests stop waiting on it.

  • federation.forwarding.request_timeout

    Five seconds by default. Set it to what the slowest source needs and no more, because every request that reaches that source can wait that long.

  • federation.forwarding.private_addresses

    "allowed" by default, which suits sources on a private network. "refused" rules out private and reserved addresses, and federation.forwarding.allowed_addresses then exempts the subnets your sources sit in.

  • federation.query.split_entities

    "complete" by default, meaning every source holds whole entities. Split entities describes the other choice.

Example deployment​

One broker with its database, and one source. sensor-gateway stands for your own service. Any image that serves the NGSI-LD query operations on port 8080 will do, and the image name here is a placeholder.

Start it​

echo "POSTGRES_PASSWORD=$(openssl rand -hex 24)" > .env
docker-compose.yaml
name: andromeda

services:
postgresql:
image: ghcr.io/mskvarc/postgresql:18.6-timescaledb2.31.0-postgis3.6.4-tsl
restart: unless-stopped
environment:
POSTGRES_USER: andromeda
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}
POSTGRES_DB: andromeda
mem_limit: 4g
cpus: 2
volumes:
- postgresql-data:/var/lib/postgresql

andromeda:
image: ghcr.io/mskvarc/andromeda:1.0.0
restart: unless-stopped
depends_on:
postgresql:
condition: service_healthy
environment:
ANDROMEDA__DATABASE__CONNECTION: postgres://andromeda:${POSTGRES_PASSWORD}@postgresql:5432/andromeda
ANDROMEDA__SERVER__PUBLIC_URL: http://andromeda:1056/
ANDROMEDA__FEDERATION__IDENTITY__SOURCE_ALIAS: andromeda
ANDROMEDA__FEDERATION__COOLDOWN__MODE: exponential
ANDROMEDA__FEDERATION__FORWARDING__REQUEST_TIMEOUT: 3s
ports:
- "1056:1056"

# Your own service, standing for the source. Replace the image.
sensor-gateway:
image: registry.example.org/sensor-gateway:1
restart: unless-stopped
expose:
- "8080"

volumes:
postgresql-data:

Start it with docker compose up -d.

Register the source​

Register the gateway as the source of the Vehicle entities it holds.

curl -i http://localhost:1056/ngsi-ld/v1/csourceRegistrations \
-H 'Content-Type: application/json' \
-d '{
"id": "urn:ngsi-ld:ContextSourceRegistration:sensor-gateway",
"type": "ContextSourceRegistration",
"information": [
{
"entities": [
{
"type": "Vehicle"
}
]
}
],
"endpoint": "http://sensor-gateway:8080/"
}'

Query through the broker​

A query for Vehicle entities at the broker now reaches the gateway as well.

curl -s 'http://localhost:1056/ngsi-ld/v1/entities?type=Vehicle' \
-H 'Accept: application/ld+json'

If the gateway is down, the broker still answers with everything else and adds a warning that names the registration.

If it goes wrong​

  • A query returns the broker's own entities and none from the source

    The registration doesn't match the query. Compare its information with the query's types and attributes, and its location and scope with any geoquery or scopeQ. The Distributed guide shows what reaches a source.

  • Every answer carries a warning that names the source, and the source is up

    The broker can't reach the address in endpoint from where it runs. Inside Compose that address is the service name, not localhost. With federation.forwarding.private_addresses set to "refused", a private address also has to be listed in federation.forwarding.allowed_addresses.

  • Subscriptions see changes in the broker and none in the source

    Either server.public_url is not set or the source can't reach it. A subscription is forwarded only when the source can call the broker back.