Spatial index

Each broker process can keep an in-memory index of entity locations per tenant. The index narrows a geoquery on location before PostgreSQL runs it. Whichever mode you choose, answers stay exact. The modes trade memory and a small read per query against one another, and only one deployment shape makes the cheaper modes safe.

entity.spatial_index.consistency defaults to "watermark", which is exact however the database is written.

What the index is for

The in-memory index gives PostgreSQL a list of candidate entities for a geoquery on location. PostgreSQL then applies the real relation to each candidate and drops the ones that don't match.

So an extra candidate costs one comparison. A missing one would be an entity silently lost from the answer. Every mode is built around never losing one.

A short list replaces PostgreSQL's own GiST scan. PostgreSQL looks each candidate up and reads no spatial index for the default instance. Past a few dozen candidates, looking each one up costs more than the GiST scan of the query's area, so PostgreSQL ignores the list and scans its index instead. The answer is the same either way.

The index can only add candidates, never drop any. You might expect a narrowing index to be a filter. It is only a pre-filter that PostgreSQL re-checks in full.

The index holds only the default instance of location, the one without a datasetId. Instances with a datasetId are always evaluated by PostgreSQL in full. So an entity whose only matching location carries a datasetId is found in every mode.

A geoquery on any other GeoProperty is never narrowed by the index. Each instance is also checked against its own expiresAt at query time. An expired instance never matches, however long the expiry sweep takes to delete it.

Snapshot branches are never indexed. A query against a Snapshot always goes to PostgreSQL.

Choosing a mode

"watermark" (default)

Before the index narrows a query, it catches up with the database. It then reflects every change any broker process committed before the query arrived. That is what makes it safe even when something other than this process writes the database, which one process can't know.

Each catch-up reads the entities whose search row was written since the previous catch-up. The read goes through an index on the writing transaction's ID. Concurrent geoqueries share one catch-up, so a burst of them costs one read.

A catch-up stops at the newest transaction ID its own snapshot can see. A database restored from a dump keeps the transaction IDs of the cluster it was dumped from, and those can be ahead of the new cluster's counter. A catch-up leaves such a row out until a write gives it a new ID. Nothing is lost, because the rebuild that seeds the index has already read the row, so a restore needs no extra step.

The cost is that read per geoquery. There is also one more B-tree entry, kept up to date for every write that changes an entity's types, attribute names, Scope or geometry. A write that changes only values doesn't touch it.

"commit-synchronous"

Every write updates the index before it returns. The index never needs to catch up, and the per-query read goes away.

That covers every path that changes current state. It includes single and batch writes, merges, replaces and attribute writes. It also covers purges, the attribute expiry sweep, deletions, and the entity a temporal write creates.

The mode is exact only when one broker process writes the database. A write from any other process reaches this process's index only at the next rebuild, up to entity.spatial_index.rebuild_interval later. Use it only when you are sure there is exactly one writer.

"store-verified"

No index is built, seeded or kept. PostgreSQL answers every geoquery with its own GiST indexes.

It is exact in every deployment, and it uses the least memory. A tenant allocates nothing for spatial lookups, and loading a cold tenant reads no locations.

Rebuilds

The index is seeded when a tenant is loaded, and fully rebuilt every entity.spatial_index.rebuild_interval. The rebuild also clears out entities that were deleted.

In "watermark" mode, deleted entities stay in the index as extra candidates until the next rebuild. That costs nothing but a comparison. In "commit-synchronous" mode, the rebuild is also what brings in other processes' writes.

If the seed can't read the database, the index stays empty. PostgreSQL then answers every geoquery until a rebuild succeeds, and the tenant still loads.

Memory

The index is the only part of a broker process whose memory grows with the number of entities. Every other cache has a fixed bound in its configuration. For $E$ located entities in a tenant, the peak is about:

$$M_{\text{peak}} \approx \frac{E}{10^{6}} \left( M_{\text{seed}} + M_{\text{writes}} + M_{\text{rebuild}} \right)$$

where, for one million entities located by a Point, with identifiers about 20 characters long:

• $M_{\text{seed}}$ is about 350 MiB, the index once it is seeded or rebuilt.
• $M_{\text{writes}}$ is up to about 200 MiB more after many entities are written between two rebuilds, which brings the index to about 550 MiB. It remembers each written entity until the next rebuild, so that the rebuild doesn't undo the write.
• $M_{\text{rebuild}}$ is about 350 MiB more while a rebuild runs. The rebuild reads every location from the database and builds the new index beside the old one, and each partition swaps over as it finishes.

That is 350 + 200 + 350 = 900 MiB per million entities, so plan about 1 GiB. A LineString or Polygon adds 16 bytes per position, and a longer identifier adds its length. The figure is on top of what the broker needs without the index. Every broker process keeps its own index, and "store-verified" keeps none.

Configuration

Key	Default	What it does
entity.spatial_index.consistency	"watermark"	"watermark", "commit-synchronous" or "store-verified"
entity.spatial_index.rebuild_interval	"30m"	How often the index is rebuilt from the database. Zero is refused

If it goes wrong

• Geoqueries get slower while a long transaction is open somewhere

  In "watermark" mode, each catch-up re-reads everything written since the oldest transaction that was running at the previous catch-up. A long-running transaction anywhere in the PostgreSQL cluster holds that point back. Answers stay exact, and the reads shrink again once the transaction ends. End the transaction.

• The database can't be read during a catch-up

  The query goes to PostgreSQL without narrowing. It doesn't wait on an index that may be behind.

• A geoquery misses an entity that something else just wrote

  You are running "commit-synchronous" while another process writes the same database. Switch to "watermark".

Where Andromeda interprets the specification

NGSI-LD v1.9.1 clause 4.10 defines each georel. Clause 5.7.2 answers a query with the entities whose stored geometry satisfies it.

Clause 5.7.2.4 adds that "if there are multiple instances of the GeoProperty on which the geoquery is based, it is sufficient if any of these instances meets the geospatial restrictions". Clause 5.8.6 says the same of a subscription's geoQ. Clause 4.22 makes an instance invalid at its own expiresAt, and lets its deletion lag.