Distributed operations

When context source registrations cover part of a request, Andromeda forwards that part to the sources, checks what comes back, and merges it with its own data. Andromeda leaves a source that keeps failing alone for a while instead of asking it again on every request.

Which sources are asked

A query, a temporal query, an EntityMap creation and a purge choose their sources the same way. A registration takes part when it matches the request's entities, supports the operation and passes every filter the request carries.

• Entities. The whole type selection counts. type=Vehicle,Boat reaches the sources registered for either type, and type=(Home;Vehicle) only those whose registration lists both types for the same entities. Andromeda matches each element of a POST body's entities on its own, so {"id": "urn:ngsi-ld:A", "type": "Vehicle"} and {"id": "urn:ngsi-ld:B", "type": "Boat"} together don't reach a source registered only for urn:ngsi-ld:A as a Boat.
• csf, read against the properties the registration gives its source.
• scopeQ, read against the registration's scope. A registration whose scope lists only Scopes that scopeQ doesn't match takes no part. A registration without a scope may hold entities in any Scope, so it takes part and receives scopeQ with the request. With splitEntities, scopeQ selects no sources. Andromeda asks every matching source and applies scopeQ to the merged entities. This holds for queries, temporal queries and EntityMap creations.
• A geoquery, read against the registration's location. A registration without a location may hold entities anywhere, so it takes part.

Andromeda never asks a source that would send the request round in a loop. The answer carries a 199 warning naming its registration instead. That is a source whose contextSourceAlias is already listed in the request's Via header. It is also one whose alias names this broker, when the request would land back in the tenant it came from.

Each tenant answers to its own alias. The default tenant uses federation.identity.source_alias. Without that key, it uses the host of server.public_url without its port, and without a server.public_url the machine's host name. The broker logs the alias it uses at startup as source_identity. Set it on two brokers that would otherwise end up with the same alias, such as two on one host.

A named tenant adds ~ and its name to the alias, as in broker-a~fleet, and either spelling names this broker. The tenant a request lands in is the registration's tenant, or the default tenant when it names none. So a registration aliased broker-a with "tenant": "fleet" loops when you register it in fleet, and doesn't when you register it in the default tenant.

A purge never asks an auxiliary source, since an auxiliary source takes only reads. An EntityMap creation does ask it, because an entity only an auxiliary source holds is still part of the answer, and the map lists every source that returned data.

What a source is asked for

Core API reads ask the source for the normalized representation, whatever format or options you used. The specification makes normalized the one representation every source must support, and Andromeda renders the representation you asked for once, after merging. Temporal API reads ask for the normalized instances in the window, whatever representation you asked for. An aggregated read asks for the sources' aggregates or their instances, as federation.query.aggregation decides (see How answers are merged).

A write forwards what Andromeda accepted, not the bytes you sent. What goes on the wire is the normalized representation under the registration's forwarding @context, in the specification version the registration says it conforms to, so an entity you wrote in concise or simplified form reaches the source normalized. Andromeda sends a source only the attributes its registration covers, comparing them by expanded name. Exclusive and redirect registrations take their attributes away from local processing, while Andromeda also processes an inclusive registration's attributes locally.

A deletion reaches a source in the form its version reads.

• A source whose ngsildConformance is 1.6 or later, or that names no version, receives the NGSI-LD Null.
• A 1.4 or older source receives JSON null for a member of a subscription update or of a partial attribute update. It receives no deletion of an attribute itself, which that version can't express in an update.
• A 1.5 source receives no deletion at all, because that version has none.

Andromeda leaves out a deletion a source can't read, so the source keeps that member.

One @context per forwarded request

Andromeda writes every name a forwarded request carries, in its body and in its query parameters, against one @context. The specification has the broker compact a forwarded request "over both payload and query parameters", and a request can link only one context, so there is no way to spell two vocabularies in one forward.

The context is the registration's jsonldContext when it has one. Otherwise it is your request's @context when a single URL names it, and otherwise the core @context, under which your own terms travel as full IRIs. Andromeda decides whether a registration covers a type or an attribute by expanded IRI, so a source whose context gives an IRI a different term still matches.

How an answer is read

Andromeda reads a source's answer one element at a time. It expands each element under the element's own @context if it has one, and under the forwarding context if not. It reads a term that context defines for a core term, such as an alias of @type or of values, as that core term, and does the same for a member shaped by its term's container. Two sources that spell an attribute differently still contribute to the same attribute, for entities and Temporal Evolutions alike.

Andromeda leaves out an element it would refuse from a local write, and keeps the rest of the page. It reports the elements a source's answer loses this way, over all its pages, with one 111 Revalidation Failed warning in the NGSILD-Warning header for that source. It says how many were left out, for example 111 - "3 of the entities context source urn:ngsi-ld:ContextSourceRegistration:x answered cannot be used". The warning names no entity, because what the source sent can be any text. That goes to Andromeda's log instead, one line per element with the reason.

Before merging, Andromeda drops an entity whose expiresAt has passed, and every attribute instance whose own expiresAt has passed.

A retrieval of one entity takes only that entity from a source. If the source answers with an entity of another id, or with an array instead of a single entity, Andromeda merges nothing from it and reports the answer as 111 Revalidation Failed. When the retrieval asks for join=flat, the source's answer is an array of the entity and its Linked Entities. Andromeda joins the Linked Entities to the entity rather than merging them into it.

Andromeda compacts the merged result once, under your @context.

How answers are merged

On the Core API, Andromeda merges the versions of one entity attribute by attribute, by expanded name. It adds an entity that only a source returned to the answer.

Instances with different datasetIds sit side by side. When two holders return an instance with the same datasetId, Andromeda keeps one.

1. Andromeda drops an instance past its expiresAt.
2. Otherwise the one with the latest observedAt wins, and an instance that has an observedAt wins over one that has none.
3. When neither has an observedAt, or both have the same one, the latest modifiedAt wins. A source sends modifiedAt only when your request asks for system attributes.
4. When nothing tells them apart, the broker's own instance stays. Between two sources, the one whose registration id sorts first wins, so the same data always gives the same answer.

An auxiliary source never overrides data the broker holds, and Andromeda uses its data only to fill in. So an auxiliary source can do only the following.

• It adds an attribute only when nobody else supplied it.
• It adds an entity nobody else returned, with every attribute it sent, since nobody else supplied any of them.
• It can't extend an entity's expiry, or make an entity disappear by leaving it out.

On the Temporal API, Andromeda combines the instances every holder returned for one entity, and an instance whose instanceId is already there counts once. An auxiliary source fills only instants no other holder has, so an entity only auxiliary sources hold comes back with their instances. Andromeda applies lastN after merging, so for every attribute you get the lastN most recent instances, wherever they came from.

Aggregated temporal reads

When more than one holder returns an entity, Andromeda computes its aggregates over the instances of every part, in a query and in a retrieval of that one entity alike. Two holders' aggregates added together are not the aggregate of the merged entity. If Andromeda holds 30 speed instances of a Vehicle and one source holds 30 more, the Vehicle gets a totalCount of 60.

federation.query.aggregation decides how Andromeda gathers the instances.

• overlapping-instances (the default) asks the sources for their aggregates first, and asks again for instances only for an entity more than one holder returned. An entity a single holder returned keeps that holder's aggregate.
• instances asks every source for its instances in one round, and Andromeda aggregates every entity a source returned.

Without splitEntities, each holder, Andromeda included, applies q, the geoquery and scopeQ to its own entities before the merge. A local entity that fails them drops out of the result, so its instances don't count. Take q=speed>50 on a Vehicle whose local speed instances are all 50 or less, while two sources each return 30 matching instances. It gets a totalCount of 60, not 63. With splitEntities Andromeda applies those filters after merging, so every part's instances count.

Temporal filters under splitEntities

With splitEntities, a forwarded temporal query keeps its identifier, type and time restrictions, and drops q, the geoquery, scopeQ and attrs. Andromeda applies those filters itself, after merging what every source and its own store returned.

• A filter reads the instances valid inside the window. That includes each attribute's instance in effect when the window opens.
• Andromeda decides the instance in effect across all holders. The latest instance before the window starts wins, and a newer deletion held by one holder ends an older instance held by another.
• To find out what a source held before the window, Andromeda asks it with standard parameters only, namely timerel=before at the start of the window with lastN=1. It asks once for the attributes the filters read and once on deletedAt. The window query's bound applies to each of these, and going over it answers TooManyResults. A source that fails adds a warning.
• Andromeda judges an entity a q hop reaches the same way, at every depth. It asks each of its holders for the attributes the hop reads of it, with the same two retrievals, and a holder that fails adds a warning.

Some sources apply lastN per attribute rather than per attribute and datasetId, and so return only one dataset of each attribute. Where another holder shows a second dataset, Andromeda asks again for that dataset. Andromeda can't ask for a dataset that no holder shows, because a source may apply lastN per attribute and there is no standard request that lists the datasets a source holds. It doesn't see that dataset's instance in effect.

Sorting and the local scope

You can use orderBy as long as no registration takes part in the answer. That means either you sent local=true, or no registration matches the query. When a registration does take part, the query gets 400 BadRequestData.

Temporal queries accept no orderBy other than id, whatever the scope.

Don't confuse this with the rule against queries that narrow nothing, which goes by the request alone. A query with no type, attribute list, q, geoquery or id needs local=true even when there are no registrations at all. That rule protects the broker from an unbounded query, and an empty registry doesn't make "every entity" any cheaper. The sorting rule is about getting a correct sorted answer, and an answer that no registration took part in is correct.

Registrations and time

When you discover context source registrations, or subscribe to them, a temporal query decides which registrations count. Without one, only registrations that declare no time interval match, since those describe sources of latest information. With one, only registrations that declare the interval the timeproperty reads match. That is observationInterval for observedAt, the default, and managementInterval for createdAt, modifiedAt and deletedAt.

The interval matches when it shares at least one instant with the period the query describes, its endpoints included. timerel=after describes everything from timeAt on, timerel=before everything up to timeAt, and between the window from timeAt to endTimeAt. An interval without endAt is open-ended. So a source registered for August 2020 to August 2021 matches timerel=after&timeAt=2020-06-01T00:00:00Z, because it holds observations made after June 2020.

Forwarded queries don't look at these intervals. A Temporal API query goes to every registration that matches its entities and attributes and supports the operation.

Sources that fail: cooldown

After a source fails, Andromeda holds it back for a while. Andromeda treats a held-back registration like a timeout. It adds a 199 warning saying the registration is cooling down, and doesn't contact the source.

Andromeda holds back subscription copies the same way. While a source is cooling down, Andromeda doesn't place, update or delete a subscription's copy there. It logs the skipped step, and your subscription itself is unaffected. Andromeda places a copy it skipped the next time the registration changes after the window ends. A copy it couldn't delete stays at the source until its expiresAt, and Andromeda refuses the notifications it still sends.

The registration's own management.cooldown is the minimum, and by default the only hold, so Andromeda contacts a registration that sets none on every request. Set federation.cooldown.mode to "exponential" or "constant" for Andromeda's policy to hold back every failing source. That can make a window longer, never shorter.

The endpoint failed

No response, a timeout, a refused connection, or a DNS or TLS failure. Andromeda holds back every registration naming that endpoint, so a thousand registrations on one dead endpoint cost one failure, not a thousand.

The source answered with an error

Any status other than 2xx or 404. Andromeda holds back only that registration, because the error was its answer to its own query. A 404 counts as a success.

Any HTTP answer proves the endpoint is reachable and ends its window. Failures that arrive while a window is already open count once. Up to federation.forwarding.max_concurrent requests can be in flight to one endpoint and fail together, and they move the sequence on only once. A success starts the sequence over.

The policy has three modes.

• "exponential" starts the first window at federation.cooldown.retry_interval and doubles it on each further failure, up to federation.cooldown.max_retry_interval. Its windows carry random jitter of up to their own length, so Andromeda doesn't retry sources that failed together in lockstep.
• "constant" keeps every window at federation.cooldown.retry_interval.
• "disabled", the default, leaves only the registration's own management.cooldown.

The cap is a trade-off. With the default max_retry_interval of 24 hours in "exponential" mode, Andromeda contacts a source that stays down about once a day after roughly seventeen failed attempts. It doesn't contact a source that comes back until its current window ends, which after a long outage can take up to federation.cooldown.max_retry_interval. If your sources recover quickly and you need them back straight away, lower federation.cooldown.max_retry_interval or switch to "constant".

Warnings

Andromeda reports problems with sources in the NGSILD-Warning header, and the code says what came back.

Code	Meaning
110	Andromeda reused a cached answer instead of asking the source.
111	The source answered in time, but Andromeda can't use what it sent. That covers an element a local write would refuse, a body that isn't NGSI-LD, an answer to a retrieval that isn't the requested entity, a 206 without a readable Content-Range, and a paged answer that can't be read to its end. A paged answer can't be read to its end when a next link points back at a page already read, an empty page still points onward, or a next link goes to another origin. A temporal source whose 206 Content-Range ends no later than the point the walk resumed it at can't be read further. The answer is then 206 with a Content-Range ending at the history held, as the specification requires when the full representation can't be given at once.
199	No answer arrived. The source timed out, refused the connection or is cooling down, the request would loop back, or Andromeda couldn't build the request at all. In that last case the warning says Andromeda never sent the request, and it records nothing on the registration, because timesSent and timesFailed count only requests it sent.
299	The source answered with an error status, including a 404 for a page its own next link pointed at. A 404 for the request as a whole is an empty answer and adds no warning.

A warning names the registration and what went wrong. It doesn't repeat what the source sent, such as a URL or a refused value, which goes to the log instead. Andromeda passes on a warning a source reports about the sources behind it. It cuts each warning's text to 256 bytes, ending in ..., and writes it as a valid quoted string. It escapes quotes and backslashes, and percent-encodes every character outside visible ASCII.

Even so, a query over thousands of failing registrations would produce a header far bigger than clients and proxies accept. Node.js takes 16 KiB, and many reverse proxies buffer only 4 or 8 KiB for the whole response header. So Andromeda keeps the header short.

• It drops exact duplicates.
• It lists warnings in order up to 2 KiB of values.
• Each code gets one summary for the rest, such as 199 - "<n> further warnings with this code are not listed".

Each registration's own record stays complete. Every request Andromeda sends it updates its status, timesFailed and lastFailure.

What gets refused

Response	When	What to do
400 BadRequestData	orderBy on a query a registration takes part in	Send local=true, or sort on the client
403 TooManyResults	A query gathers more than federation.query.max_entities distinct entities, or the sources send more than federation.query.max_source_rows entities. The detail names the setting the query exceeded	Narrow the query

Andromeda reads every source's answer to its end before cutting your page, so it bounds what it gathers. federation.query.max_entities counts distinct entities across the local store and every source. An entity three sources return counts once, and an entity only an auxiliary source returns counts too, because it is part of the answer. federation.query.max_source_rows counts every entity the sources send, so the same entity from three sources counts three times, and auxiliary sources count too.

Elements a source returns that Andromeda would refuse on a local write are a different case, because they don't refuse your request. Andromeda leaves them out of the answer and reports them as 111 Revalidation Failed. Distributed operations under Strictness has the details.

Configuration

Key	Default	What it does
federation.cooldown.mode	"disabled"	How Andromeda holds back failed sources. See Sources that fail for the three modes
federation.cooldown.retry_interval	1 s	The first window, or every window in "constant" mode
federation.cooldown.max_retry_interval	24 h	The longest window in "exponential" mode
federation.forwarding.max_concurrent	16	How many forwarded requests one operation has in flight at once
federation.forwarding.private_addresses	"allowed"	"refused" stops the broker from contacting a source whose endpoint resolves into loopback, private or reserved address space. The check runs on the addresses the endpoint actually resolves to, not on its host name. The answer reports a refused source as a warning. Whatever this is set to, the broker never follows a redirect from a source, and a 3xx answer counts as that source failing
federation.forwarding.allowed_addresses	empty	IP addresses and subnets, IPv4 or IPv6 and mixed in one list (["10.8.0.0/16", "10.3.0.17", "fd00:1234::/48"]), that a source may resolve into even though private_addresses is "refused". It lets a deployment whose sources sit in private space keep the protection on for the rest of it, and changes nothing while private_addresses is "allowed". An entry is an address or a subnet, never a host name or a port. Restrict ports with a firewall
federation.forwarding.denied_addresses	empty	IP addresses and subnets that no source may resolve into, public addresses included, whatever private_addresses says. A denied address wins over an allowed one, so a narrow entry here carves a hole out of a wide entry in allowed_addresses. Andromeda judges an IPv4-mapped IPv6 address such as ::ffff:10.2.0.5 as the IPv4 address it embeds, and also refuses a NAT64 address (64:ff9b::/96) when the IPv4 address it translates to is denied. An allowed IPv4 entry never exempts a NAT64 address
federation.forwarding.request_timeout	5 s	How long a forwarded request may take, connecting and reading the answer included, unless the registration sets its own management.timeout. It must stay below request_limits.timeout, so Andromeda reports a source that doesn't answer in time as failed before the request itself runs out of time, and federation.forwarding.connect_timeout can be no longer than it
federation.query.aggregation	overlapping-instances	How the instances of an aggregated temporal read are gathered. See Aggregated temporal reads
federation.query.max_entities	10000	The most distinct entities one federated query gathers
federation.query.max_source_rows	30000	The most entities the sources of one federated query send, duplicates included. It can't be lower than federation.query.max_entities
federation.identity.source_alias	derived from server.public_url	The alias this broker answers to, used to recognise a request that loops back. See Which sources are asked

Where Andromeda interprets the specification

The specification defines distributed operations, and normalized is the one representation every source must support. The broker compacts "over both payload and query parameters" with a registration's jsonldContext, which is why a forwarded request carries exactly one context. Andromeda spells a request whose own @context has several entries, or is inline, in the core @context instead.

The specification covers forwarded writes separately, and a broker rejects invalid Context Source data. Attribute coverage works by expanded name, and a broker processes inclusive registrations locally. The specification defines warnings and their codes, and the warn-text is the quoted string of the HTTP caching RFC. The registration status members count the distributed operations a registration triggered, so a request that never left the broker counts as neither sent nor failed.

Merging follows the specification's rules for combining instances, which keep "the one with the most recent observedAt DateTime, if present, and otherwise the one with the most recent modifiedAt DateTime". Andromeda reads "if present" per instance, so an instance without observedAt never has the most recent one. The specification only asks sources, as a "should", to support modifiedAt, so Andromeda can't rely on it to settle a pair. For a pair nothing settles, the specification leaves the choice open, and Andromeda makes it by registration id.

The registration-matching rules, read with the full type selection language, decide which registrations take part. The entity selectors of a query body are a union, so Andromeda matches each on its own. The broker has to respect every constraint a registration specifies, "geofenced areas" among them, and the specification lists the context source filter among the inputs of a purge and of an EntityMap creation.

The matching rules compare no scope. The specification defines a registration's scope as the Scopes "for which the Context Source has Entities", so a registration that declares Scopes, none of which the Scope query matches, can't contribute, and Andromeda passes it over. A registration that declares no scope says nothing about where its entities lie. Andromeda treats it like one without a location, and it takes part.

Without splitEntities the Scope query travels with the request and the source applies it, so the answer holds that source's matching entities and no others. Under splitEntities Andromeda removes the Scope query before forwarding and applies it after aggregation, so it passes over no source for it. A source can hold attributes of an entity whose Scope another source supplies, and passing it over would lose data the merged answer should contain. Querying or subscribing to registrations is different. There Andromeda matches the Scope query against scope, and a registration without a scope matches no Scope query, as an entity without a scope matches none.

An auxiliary registration "never overrides data held directly within a Context Broker", and its data "is only included if it is supplementary". The specification applies that per attribute, since auxiliary data "is added to the payload only when an Attribute is not present in any of the Attribute data received elsewhere", and per instant for temporal reads. An entity nobody else returned has no attribute present elsewhere, so Andromeda adds all of it.

A retrieval answers ResourceNotFound only when no local entity exists "and no matching registrations apply", so it asks the matching auxiliary sources before it answers 404. An EntityMap lists "the registrations that were fired by the previous request and successfully returned data", with no exception for a mode, so an auxiliary source that returned an entity is in the map. An auxiliary source shares no data with an exclusive or redirect source, so it never stops that source's failure being the operation's answer.

The specification raises BadRequestData "If the ordering parameter is present and the execution of the operation is not limited to the local scope", and says "Sort ordering is never applied to distributed operations". Local execution uses "only... information available in the Context Source or Context Broker directly targeted by the request". Andromeda decides "limited to the local scope" by what happens, not by what the client sent. Reading it as "the client sent local=true" would make orderBy, which the ordering language requires, unusable on a broker with no registrations at all, which is exactly the deployment the specification describes. The specification states the temporal orderBy limit separately.

The specification splits registrations by whether a temporal query is present, and picks the interval by timeproperty. For between it asks for an overlap. For before and after it says the timeAt "is contained in or is an endpoint of a time period included in the specified time interval". Read as "the instant lies inside the interval", a query for everything after the epoch would reach no source, so Andromeda reads before and after as an overlap too. Forwarded reads match by attribute coverage, which has no time axis.

Under the cooldown rules a held-back registration answers as a timeout, and a broker may override the recommended cooldown. A broker may decline requests until the cooldown has passed, and 404 counts as a success.

Under splitEntities, the dropped filters "have to be applied after the Entity information from different Context Sources and local information, if there is any, has been aggregated". The specification's validity rules and an example show the instance in effect, and Andromeda bounds the walk. The specification allows per-attribute lastN. Periods describes the period such aggregates report.