@lde/search-indexer

The @lde/search-pipeline indexer as a bootable process and prebuilt Docker image: mount a schema-declaration module, point it at a dataset registry and Typesense, and one run selects the datasets, extracts and projects each root type per the schema, and rebuilds the engine collections – then exits. Run it as a cron job; the rebuild writers’ cross-pod lock makes an overlapping run fail fast instead of corrupting a collection.

This is the write-path counterpart of @lde/search-api-server (#652): the composition layer that binds the engine-agnostic searchIndexerPipeline to the @lde/search-typesense rebuild writers. Mount the same schema module into both images and point both at the same TYPESENSE_* coordinates, and the write and the read side cannot disagree about the schema. A deployment that needs more – a bespoke root selector, per-stage tuning, another engine – composes searchStages, searchIndexWriter and Pipeline directly instead.

Installation

npm install @lde/search-indexer

Run

docker run --rm \
  --volume ./search-schema.mjs:/config/search-schema.mjs:ro \
  --env REGISTRY_ENDPOINT=https://registry.example.org/sparql \
  --env TYPESENSE_HOST=typesense.internal \
  --env TYPESENSE_API_KEY=admin-key \
  ghcr.io/ldelements/search-indexer

Or without Docker (the same environment variables apply):

npx @lde/search-indexer

Dataset IRIs can also be passed as arguments, which override DATASETS:

npx @lde/search-indexer https://example.org/id/dataset/1

--check validates the configuration and schema module, then exits without indexing – for CI and init containers.

The schema module

The mounted module default-exports the deployment’s search type declarations as plain data – see the API server’s README for the format and authoring guidance; both images load the module with the same @lde/search/module loader. The indexer reads only the default export: the declarations drive one pipeline stage and one engine collection per root type, and each field’s path drives the extraction CONSTRUCT. The read-side extras (schemaOptions, engineOptions) are ignored here, so one file serves both images.

Configuration

Variable	Default	Meaning
SCHEMA_MODULE	/config/search-schema.mjs	Path of the mounted schema-declaration module
REGISTRY_ENDPOINT	required	SPARQL endpoint of the DCAT dataset registry
DATASETS	all registry datasets	Dataset IRIs to index (whitespace- or comma-separated)
DATASET_CRITERIA	all registry datasets	Search criteria as a JSON object, in @lde/dataset-registry-client’s criteria format (mutually exclusive w/ DATASETS)
TYPESENSE_HOST	required	Typesense host
TYPESENSE_PORT	8108	Typesense port
TYPESENSE_PROTOCOL	http	http or https
TYPESENSE_API_KEY	required	An admin key: the indexer creates, writes and swaps collections
REBUILD_MODE	in-place	in-place (update the live collection) or blue-green (swap on commit)
COLLECTION_PREFIX	none	Prefix for every derived collection name (configure the read side to match)
PROVENANCE_FILE	none	JSON file remembering per-dataset processing, to skip unchanged datasets
PIPELINE_VERSION	none	Version keying the skip decisions; required with PROVENANCE_FILE
QLEVER_IMAGE	none	Enables the QLever import path (see below), e.g. adfreiburg/qlever:latest
IMPORT_STRATEGY	sparql	sparql, sparqlWithImportFallback or import; requires QLEVER_IMAGE
DATA_DIR	/data	Directory for downloaded dumps and QLever index caches
QLEVER_NETWORK	none	Docker network the spawned QLever joins; set when the indexer itself runs containerized (see below)

A misconfigured boot reports all problems in one error, not one per crash loop. PROVENANCE_FILE must sit on a durable volume, and cannot be combined with REBUILD_MODE=blue-green: a skipped dataset would be missing from the fresh collection the swap makes live. Its directory must be writable by the image’s runtime user (uid 1000) – the image pre-creates /provenance with that ownership, so a named volume mounted there just works, while a bind mount must be chowned on the host. A run fails at start when the file is not writable, instead of silently never skipping.

The QLever import path

By default the indexer serves only datasets that publish a live SPARQL endpoint (the pipeline’s endpoint-only default). Setting QLEVER_IMAGE enables the import path: data dumps are downloaded to DATA_DIR and imported into a QLever instance the pipeline itself creates and controls – one sibling container per dataset, spawned over the Docker socket (ADR 16 explains why a statically-declared QLever service cannot work). That requires:

• the Docker socket mounted into the indexer container (--volume /var/run/docker.sock:/var/run/docker.sock), plus a group grant to reach it as the image’s non-root user (--group-add $(stat -c %g /var/run/docker.sock));
• DATA_DIR on a volume whose host path is identical for the indexer and the spawned QLever containers – the pipeline passes DATA_DIR as a bind mount to a sibling container, where it resolves against the host.

How the indexer reaches the QLever it spawned depends on where the indexer itself runs:

• On the host (npx @lde/search-indexer with Docker available): leave QLEVER_NETWORK unset. QLever’s port is published on the host and addressed as localhost.
• In a container on a bridge network (the docker compose default): set QLEVER_NETWORK to a network the indexer is attached to. QLever joins that network and is addressed by container name (qlever-<network>) instead – a containerized indexer’s localhost is its own network namespace, so it cannot reach a host-published port. Alternatively, run the indexer with network_mode: host and leave QLEVER_NETWORK unset.

QLEVER_NETWORK must name a network the indexer container is actually attached to – for compose, the prefixed name Docker creates (e.g. myproject_default), not the short service-file name. A wrong or unattached network is not caught at boot: each dataset imports fully, then fails when the endpoint never becomes reachable. And run at most one indexer per network – the QLever container name is derived from the network, so two indexers sharing one would remove each other’s QLever.

This mode does not work on container runtimes without a Docker socket (containerd-based Kubernetes); there, run endpoint-only for now.

Building the image

The image is built from the workspace’s own outputs – the compiled package, the same-commit builds of its @lde/* dependencies and a pruned lockfile – never from npm, so it exists for any commit (ADR 15):

npx nx run @lde/search-indexer:docker:build   # → packages-search-indexer
npx nx run @lde/search-indexer:docker:smoke   # boots it with --check

CI runs docker:smoke for affected PRs; each release rebuilds and pushes ghcr.io/ldelements/search-indexer:<version> (.github/workflows/docker.yml).

Programmatic use

The bin is a thin wrapper over the exported API, usable in tests or a custom boot:

import {
  configFromEnvironment,
  createSearchIndexer,
} from '@lde/search-indexer';

const pipeline = await createSearchIndexer(configFromEnvironment(process.env));
await pipeline.run();

The configuration types are exported too: IndexerConfig (what configFromEnvironment produces) and its parts TypesenseConnection, ProvenanceConfig (PROVENANCE_FILE + PIPELINE_VERSION) and QleverConfig (the QLever import path).