Skip to main content

Rows, not documents

Andromeda stores an entity as typed rows. Each attribute instance is a row, its value sits in a column of its own kind, and a read puts the entity back together. Storage describes the layout. This page explains why. Until October 2026 Andromeda kept each entity as one JSON document, the design most brokers use, and the rows grew out of what went wrong with it.

Where Andromeda started​

The first schema kept the current state of an entity in one row of entities. The whole entity, in expanded JSON-LD, sat in a payload column of type jsonb, and that document was the source of truth. The fields queries needed most were copied out of it into indexed columns of their own: the types, the Scopes, and the location, observationSpace and operationSpace geometries. A GIN index covered the whole document with the jsonb_ops operator class, and a second one with jsonb_path_ops followed, for whatever a q filter might ask.

It is a natural design. NGSI-LD entities have no fixed shape, a document holds any shape, and a read hands back what was written. The problems below all came up while running it. Each fix moved some data out of the document, and in the end nothing was left in it.

Filters couldn't use an index​

A q filter on a document is a JSON path, and the first compiler turned temperature>50 into jsonb_path_query_first(payload, …)::numeric > 50. That is a function of the whole document, so no index can answer it, and every such filter read every entity.

The two GIN indexes didn't help, and it took a while to see why. A GIN index on payload serves an operator only when its left side is the bare column. The broker's attribute test had the form payload->0 ? $1, an expression over the column, so the jsonb_ops index was never eligible. The jsonb_path_ops index serves only containment (@>), and no query used containment. Neither index ever served a query, yet both were kept up to date, and they were the largest cost on the write path. Every entity write indexed the entire document twice. They were dropped.

What this came down to is that an index reaches a value only when the value has a column of its own.

A second copy fixed filters, not answers​

The fix for filters was a typed projection beside the document. It held one row per attribute instance, with the value in a typed column and an index on each column. q, orderBy and geo-queries read the rows, and reads still rendered the document. Filters became index scans, and every write now stored each value twice.

The two copies didn't hold the same things. Sub-attributes lived only in the document, so a q on a sub-attribute fell back to a JSON path over every candidate document. That path took the first instance at each level, where the specification asks for any value to match. A traversal followed only the column that held a single Relationship object, so it never reached the targets of a Relationship with several. Each of these gave wrong results. The two copies never disagreed on a value, but they covered different parts of the entity, and a query that crossed the gap went wrong.

A write rewrote the whole entity​

PostgreSQL never changes a stored value in place. An update writes a new version of the row, and a changed jsonb value is written out whole. To read it in, PostgreSQL builds the whole document as a tree in memory first. A write to one attribute therefore paid for the entire entity.

A large geometry shows it most clearly. An entity whose location is a polygon of a million positions is about 65 MB of expanded JSON-LD text and 107 MB as jsonb. Measured on PostgreSQL 18, building that tree took about 2.2 GB of server memory for a single write. Replacing one member with jsonb_set costs the same, because it rebuilds the tree too. So changing the entity's status rewrote the polygon.

Today a write compares the state it read with the state it commits and rewrites only the rows of the instances that changed. A sub-attribute change rewrites the row of the one instance it belongs to, and a write that restates stored content writes no attribute row at all. What a write costs depends on what it changes, and the size of the entity doesn't come into it.

A read decoded the whole entity​

The document was the only thing a read could render from, so every read fetched and decoded the whole entity, however little of it the request wanted. attrs naming two attributes of an entity with five hundred still read and decoded all five hundred.

With rows, a projection by attrs, pick or omit reads only the rows of the attributes it names, and a projection that names no attribute the entities hold reads no attribute row at all.

A value is compared as what it is​

A JSON path knows two kinds of scalar. It compares a JSON number as a number and a JSON string as a string. NGSI-LD tells more kinds apart, and in a document they all look like strings.

  • A time is a string in JSON

    Take a testedAt holding the DateTime 2024-01-01T02:00:00+02:00, which is midnight UTC, and the filter q=testedAt>2024-01-01T00:30:00Z. The stored instant is half an hour before the operand, but its text sorts after it. Compared as text, the filter matches.

  • A declared number is a string in JSON

    {"@value": "4", "@type": "xsd:integer"} is the number 4. A JSON path that compares the string "4" with the number 3 is never true, so q=reading>3 doesn't match it.

The typed projection had a version of the same problem. It stored numbers as double-precision floats, so two numbers that differ only in their twentieth digit compared equal. The rows hold numbers as numeric, exactly. Andromeda now decides what a value is once, when it is written, and stores it in the column of that kind: a declared DateTime in timestamptz, a number or a declared number in numeric. A q term compares that column with an operand of the same type, and the column's index answers it. Values has the rules for which values count as times and numbers.

What else the document lost​

  • The written order

    A jsonb object sorts its keys and keeps no record of the order they were written in, so the document couldn't return attributes in the order they arrived. Each row now carries the written position of its attribute and instance.

  • Space

    Expanded JSON-LD spells every attribute name and type as a full IRI, such as https://uri.etsi.org/ngsi-ld/default-context/temperature, and the document repeated it in every entity. The rows hold a four-byte key from a per-tenant dictionary instead. Name dictionaries covers them.

  • A fixed comparison

    jsonb compares strings under the database's default collation, so the order of values held in a document depends on how the PostgreSQL server was set up. Every column whose comparison decides a result now declares its collation, and every query that orders strings names its own. Collation fixed in the schema explains how.

The other tables followed​

After the current state moved to rows, the rest of the schema went the same way. The temporal history, the notification queue and the queue of pending side effects hold attribute instances in the same typed form as the current state, written by one flattening and read back by one assembly. No table holds an entity or an attribute instance as a JSON document. jsonb remains only for values that are JSON by nature, such as a structured value, a JsonProperty or a LanguageProperty map.

What rows give up​

Rows have costs, and some of them are things the document did well.

  • Reading a whole entity does more work

    A document is the entity, ready to send. Andromeda reads the entity's rows in one statement and assembles them in the broker.

  • The schema has to fit every shape

    A document holds whatever it is given. The rows have to give every NGSI-LD value family a place, from LanguageProperty maps to ListRelationships to sub-attributes nested many levels deep, and constraints check that each row holds exactly one value in the column its kind names.

  • The query compiler is larger

    A JSON path evaluates any filter the same way. Andromeda's compiler chooses between several plans for a filter on a sub-attribute, because PostgreSQL can't estimate well how many entities such a filter keeps. A range or a pattern on a sub-attribute still has no index and walks the entities in page order.

These costs are accepted on purpose. Each problem above was found while running the document layout, and each fix moved data out of it. Many systems read from a broker, so it should give the same answer to the same question on every server, have an index behind each comparison it makes, and charge a write only for what the write changes. The rows are how Andromeda gets there.