Pagination

Big answers come in pages. For a collection, such as entities, subscriptions or registrations, follow the rel="next" link until there isn't one. For a temporal read, a 206 tells you there is more, and a next link, or a request you build yourself, fetches it. A page holds at most limit elements, and limit tops out at 1,000.

Walking a collection

GET /ngsi-ld/v1/entities?type=Building&limit=2&count=true

HTTP/1.1 200 OK
NGSILD-Results-Count: 5
Link: </ngsi-ld/v1/entities?type=Building&count=true&limit=2&offset=4>; rel="last"; type="application/json", </ngsi-ld/v1/entities?type=Building&count=true&limit=2&offset=2>; rel="next"; type="application/json"

This is the first page of three, so it has next and last but no prev. The request names neither offset nor cursor, so its links page by offset. Cursor or offset shows the same walk by cursor.

Each operation has its own default limit. With count=true you also get the size of the whole collection in NGSILD-Results-Count. If you only want that total, send limit=0 together with count=true. limit=0 on its own isn't accepted.

A page can carry four links.

• next leads to the following page. The last page has none.
• prev leads to the previous page. The first page has none.
• first goes back to the start. Only pages in the middle have it.
• last leads to the final page. Every page except the last can have it.

On the last page only prev is left, and that is how you tell the walk is over. A collection that fits on one page has no pagination links at all. Every link carries the response's media type as its type.

An offset past the end

An offset equal to or larger than the size of the result set gets 200 with an empty array. The page carries only a prev link, as the last page does, and NGSILD-Results-Count still reports the whole count where the listing sends it. Offsets count from 0, so the position at the size of the result set holds nothing, and every larger offset holds nothing too. A client that steps its offset by the page size reaches an empty page and stops, and never gets the last element twice.

The specification words this case as a "should". A too-large offset is "assumed to be equal to the size of the result set, i.e. only the last element of the result set is to be returned". The rule and its explanation disagree, since the last element sits at the size minus one, and Andromeda follows the rule. Following the explanation would make an offset equal to the size answer an empty page and the next offset answer the last element, so an offset walk would see an element repeat where it expects the walk to end.

Cursor or offset

Andromeda pages in one of two ways.

With an offset, which is the default, the links carry offset and limit, and you can set both yourself. A request that names neither offset nor cursor pages this way.

With a cursor, the links carry an opaque cursor token. Start the walk with an empty cursor=, then pass each token back by following the link. The empty cursor= is the only cursor you write yourself.

GET /ngsi-ld/v1/entities?type=Building&limit=2&count=true&cursor=

HTTP/1.1 200 OK
NGSILD-Results-Count: 5
Link: </ngsi-ld/v1/entities?type=Building&count=true&limit=2&cursor=...>; rel="next"; type="application/json", </ngsi-ld/v1/entities?type=Building&count=true&limit=2&cursor=...>; rel="last"; type="application/json"

A cursor link also carries pageQuery, a digest of the query the walk belongs to. A cursor followed with another q, type, geo-query, temporal window or any other parameter that decides which elements match gets 400 BadRequestData. Changing limit, options, format, lang, pick, omit, count, details, join or joinLevel on a followed link is fine. So is adding entityMap and entityMapLifetime, because an EntityMap records the entities the same query matches. A link followed with entityMap=true gets 201 with an NGSILD-EntityMap header, and the map holds every entity the query matches, not only that page.

The difference shows when the collection changes during the walk. A cursor page starts after the last identifier the previous page returned, not at a position. An entity created mid-walk with a higher identifier turns up on a later page, and deleting one you already passed doesn't make its neighbours shift. An offset walk can skip or repeat elements in the same situation. Start with cursor= when a walk has to see each element once while others write to the collection.

Cursors are on by default, even though the specification treats them as an extension. It allows opaque page pointers outright, so serving them is still conformant. Cursor pagination only counts as an extension because the cursor parameter is Andromeda's own name, which is why you can switch it off. On a default broker, a client that never sends cursor gets offset paging with the extension on or off.

A cursor walk goes in identifier order. A request that names offset or orderBy always gets offset links. Sending cursor together with offset or orderBy gets 400 BadRequestData, and so does a malformed token.

GET /ngsi-ld/v1/entities?type=Building&offset=10&cursor=...

HTTP/1.1 400 Bad Request
Content-Type: application/json

{
    "type": "https://uri.etsi.org/ngsi-ld/errors/BadRequestData",
    "status": 400
}

Only type and status are shown here, and the real body carries the rest of the problem document. Drop the cursor and keep the offset, or the other way round.

Queries that involve context sources page the same way, by offset or by cursor. Andromeda merges its own entities with what every source answered, sorts the lot by identifier, and cuts the page from that. NGSILD-Results-Count is the size of the whole merged set.

Which operations page how

• Cursor and offset

  GET /entities, GET /temporal/entities, GET /subscriptions, GET /csourceSubscriptions, GET /csourceRegistrations, GET /jsonldContexts, and GET /tenants when tenant management is on. All of them go in identifier order, except tenants, which go in name order.

• Offset only, with no Link header

  POST /entityOperations/query and POST /temporal/entityOperations/query. A query sent in a body can't be put into a link, so these answers report only NGSILD-Results-Count, and a cursor there gets 400 InvalidRequest.

• Offset only

  EntityMap creation on /entityMaps and /temporal/entityMaps. The map holds every matching entity whatever limit and offset say, it hands out no page links, and a cursor there gets 400 InvalidRequest.

• No paging

  Purge Entities (DELETE /entities) acts on every match, so limit, offset and cursor are refused there.

Paging temporal instances

When a temporal read leaves instances out, the answer is 206 with a Content-Range header. Reading forwards, the range ends at the most recent instant in the response. Reading backwards, which is a lastN above Andromeda's maximum, it ends at the earliest one. A page never splits the instances of one attribute that share an instant.

Following the next link

A 206 from Retrieve Temporal Evolution of an Entity carries a Link with rel="next". Follow it to get the next page. The link replaces timerel, timeAt, endTimeAt and lastN, and keeps every other parameter. The last page is a 200 with no link.

Reading forwards with a month or a year in aggrPeriodDuration, the link's timeAt is a boundary from which the same calendar grid follows. The next page lays its periods on the grid of the first page even though its window starts later, so the pages put together give exactly the periods of one unpaged answer. To make that work, a page ends only on such a boundary, which can come before the last period that would otherwise fit on it (see Temporal aggregation). The link needs no extra parameter for this.

The link can't carry headers, so send your @context and NGSILD-EntityMap headers again on the request that follows it.

Query Temporal Evolution of Entities has no such link. It pages two things at once, the entities through the collection links above and the instances of each entity. One next link can't continue both, so to get more instances from a query you build the next request yourself.

Building the next request yourself

When reading forwards, don't reuse the end of the range as the next timeAt. after and the start of between include their own instant, so you would get the instances at the edge twice. Start one microsecond later instead. Andromeda holds instants to the microsecond, so nothing can fall between the two.

In the tables, e is the end of the range from Content-Range, and e + 1 µs is that instant plus one microsecond.

Reading forwards, build the next request like this.

Request	Next request
timerel=after&timeAt=t	timerel=after&timeAt=e + 1 µs
no time window	timerel=after&timeAt=e + 1 µs
timerel=between&timeAt=t&endTimeAt=u	timerel=between&timeAt=e + 1 µs&endTimeAt=u
timerel=before&timeAt=u	timerel=between&timeAt=e + 1 µs&endTimeAt=u

Once e + 1 µs is no longer before the end of the window, you have everything.

Reading backwards, build it like this, where n' is the lastN you still need (see below).

Request	Next request
timerel=before&timeAt=u	timerel=before&timeAt=e&lastN=n'
no time window	timerel=before&timeAt=e&lastN=n'
timerel=between&timeAt=t&endTimeAt=u	timerel=between&timeAt=t&endTimeAt=e&lastN=n'
timerel=after&timeAt=t	timerel=between&timeAt=t&endTimeAt=e&lastN=n'

An aggregated page (format=aggregatedValues) ends on a period boundary rather than an instant. Reading forwards, start the next request on the boundary itself, timeAt=e, because the next period begins there. Reading backwards, end at the boundary as above.

lastN over several pages

lastN counts per attribute stream, meaning one attribute with one datasetId. A backward walk spreads one lastN over several pages. The next link asks for your lastN minus the fewest instances any stream got on that page, and the walk ends when that reaches zero.

Every stream gets at least its last lastN instances, and never the same one twice. A stream that got more than the fewest on a page can end up with more than lastN over the whole walk. With lastN=50 over a dense stream of 100 instances and a sparse one of 30, the dense stream gets all 100. That is as close as a walk can get, since one request can't ask for a different lastN per attribute.

Context sources and temporal pages

When a temporal read merges answers from context sources, the merged page is only complete up to the earliest point any source stopped (reading forwards), or the latest (reading backwards). A source may have cut its own page in the middle of an instant, so Andromeda holds back the instances at each source's last instant and serves them on the next page.

If holding them back would leave the page empty, Andromeda serves them anyway and adds a 111 warning naming the source. Whatever that source had at that instant beyond its own page is then out of reach for the walk.

Calendar periods and a hand-built continuation

An aggregated read with a month or a year in aggrPeriodDuration lays its periods on a calendar grid anchored at timeAt. Every boundary takes the anchor's day of month, moved back to the last day when a month is shorter. An anchor on the last day of its month keeps every boundary on the last day of its month (see Temporal aggregation).

A request you build yourself from a boundary lays a fresh grid from that boundary. It continues the original grid only when the same rule, applied to that boundary, gives the remaining boundaries. Take a monthly grid from 30 January in a common year. Starting again from its boundary on 30 March gives the same grid. Starting again from its boundary on 28 February does not. That day is the last of its month, so the fresh grid runs on month ends and puts its next boundary on 31 March, where the original has 30 March.

So a forward page ends only on a boundary that lays the original grid, and a request built from the end of such a page, timeAt=e, continues it with no further parameter. The next link does exactly this.

When a page of whole months or whole years holds no such boundary within its budget, the read is refused with TooManyResults. A duration that mixes months or years with days or time is never paged at all (see Temporal aggregation).

Configuration

• extensions.cursor_pagination.mode (default "enabled"): whether cursors are handed out. Offset paging is always available.
• extensions.cursor_pagination.fresh_requests (default "offset"): how a request that names neither offset nor cursor is paged while cursors are handed out. Set it to "cursor" to walk such a request by cursor.

Together, the two settings decide what a request gets. The first row is the default.

mode	fresh_requests	Request with neither	Request with cursor=
"enabled"	"offset"	offset	cursor
"enabled"	"cursor"	cursor	cursor
"disabled"	either	offset	400 InvalidRequest

Conformance mode, extensions.mode = "conformance", turns cursors off along with every other extension, so it pages every collection by offset and answers a cursor with 400 InvalidRequest. Run the ETSI conformance suite against the same binary you deploy, with that mode on.

What gets refused

• 403 TooManyResults for a limit above 1,000. It is refused, not quietly served as a smaller page. See Andromeda's own bounds.
• 403 TooManyResults for a page that renders to more than request_limits.max_page_body. The page is refused whole, naming its size and the ceiling. Ask for a smaller limit and follow the next page pointer.
• 400 BadRequestData for cursor with offset or orderBy, or a malformed cursor. Leave one of them out, and never edit a cursor token.
• 400 BadRequestData for a cursor followed under another query, or without its pageQuery. A cursor names a position in one result, so to page a different query, start a new walk.

Where Andromeda interprets the specification

The Link pointers and the 206 with Content-Range are defined by the HTTP binding. The page size is the requested limit, the last page keeps only prev, and pointers may be "implemented in a completely opaque way", which is what a cursor is. Offset paging is defined separately. The first and last relations are Andromeda's addition, which web linking allows. cursor is Andromeda's own parameter, so cursor paging is an extension that can be turned off.

A link target must be one "that could be dereferenced". A query carried in a body can't be one, so the body queries have no links. Purge Entities refuses paging parameters.

The specification says "Sort ordering is never applied to distributed operations". Andromeda reads that as covering the orderBy a client asks for, not the identifier order a broker uses to cut pages. The specification asks for "consistent pagination" of a distributed query, which the merged identifier order gives.

The specification doesn't say how to continue a temporal walk, so the forms above are Andromeda's. They rest on the rule that after and the start of between include their instant. The 111 warning is the code the specification gives to a response that arrived in time but can't be used.

Aggregation periods are one element per period of the time range, but the specification doesn't say where the periods start. Andromeda starts a calendar grid from its anchor, and it ends a page only on a boundary from which the same grid follows. A continuation therefore needs nothing beyond timeAt, and Andromeda adds no parameter of its own for it.