9. Route a whole-schema projection to per-type collections

Date: 2026-07-10

Status

Accepted

Extends the writer model of ADR 6 (Make the Writer transaction-aware) and the reference model of ADR 8 (Resolve reference labels from per-reference label sources); updates the catalog grain of search as a Configurable Pipeline instance (#534) from one collection to several (#590).

Partly superseded by ADR 12 (Bound memory by the unit of work, not the input) and ADR 13 (Project inside the batch, per root type), which reverse this ADR by halves:

• Still current – the fan-out’s placement: composed in @lde/search-pipeline over N single-collection engine writers (option (a) below), not a multi-collection writer inside an engine adapter; each collection committing, sweeping and failing in isolation; abort finalizing only the collections that did not go live. That is what this ADR set out to decide, and it stands.

• Still current – the per-document searchType (TypedSearchDocument). N stages still write to one terminal (source → transform → sink), and write(dataset, items) carries no stage identity, so the type must travel with the item. What changes is that a stage mints it – it was constructed for one type – rather than projectGraph discovering it from rdf:type. searchIndexWriter remains the router this ADR made it; it loses projection and buffering, nothing else.

• Superseded – the projection mechanism: the whole-schema single-scan mixed stream, and the buffer-until-flush. Both are structures sized by the input rather than by a bounded unit. Projection happens per type, per batch, over roots the selector supplied (ADR 13).

Implemented by #606.

For the record: the buffering was never this ADR’s subject. It arrived two PRs earlier, in @lde/search-pipeline’s first commit (#565), as a documented deferral of “bounded entity batches within one huge dataset”; this ADR inherited it as background and restated it while deciding the fan-out. Which is why the Consequences below never weigh memory – memory was not on the table.

Context

One validated SearchSchema declares several root types. The Dataset Register indexes four: datasets plus the Organization / Class / TerminologySource label collections its references resolve against (ADR 8). Each type derives its own Typesense collection definition, so the types cannot share a collection – they are four independent blue/green rebuilds, each with its own versioned collection, alias and single-flight lock.

ADR 6 made the engine Writer a transactional, single-collection rebuild (BlueGreenRebuild / InPlaceRebuild), bound to one SearchType. projectGraph already projects the whole schema in one pass – a single scan of the quads builds the subject index every type frames off – yielding one mixed stream of documents. Nothing joined the two: a consumer that wanted several collections had to forge a per-type schema brand, hand-roll a per-type rebuild loop and cast a per-type projection. The per-collection fan-out had no home.

Two placement options: (a) a composition in @lde/search-pipeline over N single-collection engine writers, or (b) a new multi-collection writer inside @lde/search-typesense. (a) keeps the engine adapter a single-collection concern and keeps the fan-out engine-agnostic (an OpenSearch adapter would reuse it unchanged), so the routing is a pipeline-glue concern, not an engine one.

Decision

• The projection tags each document with its type. projectGraph yields { searchType, document } (TypedSearchDocument), not a bare SearchDocument. The whole schema still projects into one mixed, single-scan stream; the tag is what lets the write side route each document to the collection for its type without re-deriving the type from the document. A single-collection consumer just reads document.

• @lde/search-pipeline’s searchIndexWriter fans out (option a). It takes the schema and a writerFor(searchType) factory, builds one engine writer per root type once, and on each run opens one engine run per type. A dataset’s quads are buffered until its flush, projected once (whole schema), then split by type and dispatched to each type’s run. The pipeline still drives one uniform openRun → write* → commit/abort and never branches on the multi-collection shape – consistent with ADR 6’s single lifecycle. A single-collection deployment is just the N = 1 case.

• Each collection commits, sweeps and fails in isolation. A type whose projection is empty this run affects only its own collection, never another’s; the empty-selection guard (#569) is therefore per collection by construction, not a single global gate. commit finalizes every collection independently and, if any fails, throws an AggregateError after attempting them all: a non-critical label-collection failure never blocks the collections that did commit – in particular the datasets index still goes live – while the failure is still surfaced, so a stale collection is never silent.

• abort finalizes only the collections that did not go live. The pipeline aborts a run whose commit throws (ADR 6), so a partial commit reaches abort. Because a committed blue/green rebuild’s abort would drop its now-live collection, abort skips the collections that already committed and drops only the half-built ones (and releases their locks). A failure while opening the per-type runs rolls the already-opened ones back the same way.

Consequences

• The Dataset Register consumer collapses: project the schema, hand the stream to searchIndexWriter with a writerFor that returns a BlueGreenRebuild per type. The forged schema brand, the per-type projection casts and the hand-rolled per-type rebuild loop all go away.

• Breaking for @lde/search: projectGraph yields TypedSearchDocument, not SearchDocument. Breaking for @lde/search-pipeline: searchIndexWriter takes writerFor (per type) instead of a single writer.

• The partial-failure policy is fail-the-run-but-commit-what-can: the safest default (no silent staleness) that still honours the isolation requirement. A skip-and-report or critical/non-critical-per-type policy can layer on later without changing the routing, if a deployment needs the datasets index to go live without the run being marked failed.

• The fan-out is engine-agnostic: it composes any transactional Writer<SearchDocument>, so a second engine adapter reuses it unchanged.