@lde/distribution-probe

Probes a DCAT Distribution to check availability and gather metadata. Returns SparqlProbeResult, DataDumpProbeResult, or NetworkError – the probe never throws.

Installation

npm install @lde/distribution-probe

Usage

import { Distribution } from '@lde/dataset';
import { probe } from '@lde/distribution-probe';

const distribution = new Distribution(
  new URL('https://example.org/data.ttl'),
  'text/turtle',
);
const result = await probe(distribution);

Behaviour

SPARQL endpoints

Sends POST with the configured query (default SELECT * { ?s ?p ?o } LIMIT 1). The query type is detected (ASK / SELECT / CONSTRUCT / DESCRIBE) and drives both the Accept header and how the response is validated:

• ASK / SELECT request application/sparql-results+json, with application/sparql-results+xml as a lower-priority fallback. The response Content-Type must be one of those – anything else fails the probe (isSuccess() === false), which rules out HTML error pages served with 200 OK. The body must parse and contain a results document (a results object for SELECT, a boolean for ASK); empty bodies, invalid JSON/XML, and missing results all fail with a failureReason.
• CONSTRUCT / DESCRIBE request the common RDF serializations (text/turtle, application/n-triples, application/rdf+xml, application/ld+json, application/n-quads, application/trig) and accept any of them. A 2xx RDF response confirms availability, and an empty graph is a valid answer – so an empty body does not fail the probe (unlike a data dump, which must be non-empty). The body is not parse-validated.

Data dumps

Reachability (the default)

Sends HEAD with Accept: <distribution.mimeType>, */*;q=0.5 (or Accept: */* when the distribution declares no media type) and Accept-Encoding: identity. The */* fallback keeps servers that answer any specific Accept with 406 Not Acceptable (notably Dataverse’s file-access endpoint) probeable, while content-negotiating servers still pick the declared type – preserving the probe’s ability to detect real Content-Type mismatches. A successful HEAD settles reachability and gathers metadata (Content-Length, Last-Modified) without reading the body. If HEAD is unsuccessful – e.g. a server that returns 405/501 because it does not implement HEAD – the probe falls back to a body-less GET to confirm the endpoint is up. The body is never downloaded.

This is deliberately cheap: reading a body forces a slow, generate-on-the-fly endpoint (a TriplyDB dump, a SPARQL CONSTRUCT export) to start producing its export, which a HEAD does not.

• Content-Type is checked as a soft warning, not a hard failure. If the server’s Content-Type disagrees with the distribution’s declared mimeType, a message is appended to result.warnings but isSuccess() stays true. Compression wrappers (application/gzip, application/x-gzip, application/octet-stream) are skipped so a gzipped Turtle file doesn’t trigger a warning. For a compressed distribution the check accepts both the declared compressed form (Distribution.compressedMimeType, e.g. application/n-quads+gzip) and the bare mimeType – a server that serves the uncompressed representation is not flagged, while a different base serialization still is.

Content validation (opt-in)

Set validateRdfContent: true to additionally confirm that a dump actually carries RDF. It applies only to distributions whose declared mimeType is an RDF serialization (text/turtle, application/n-triples, application/n-quads, application/trig, text/n3, application/ld+json, application/rdf+xml); non-RDF and undeclared-type distributions stay reachability-only.

When on, the probe GETs the dump – regardless of size – and reads only a bounded prefix (256 KiB), never the whole body:

• It settles on the first triple and stops, so a large dump is validated from its opening chunk. The line/statement-oriented serializations and RDF/XML stream a triple out of the prefix; JSON-LD is not streamable (its parser needs the whole document), so a JSON-LD dump is only validated when it fits the prefix in full – a larger one is reported reachable but unvalidated.
• A gzip body that fetch did not decompress (a .gz dump, or one served with a non-standard Content-Encoding) is inflated in-place; a gzip that will not inflate when the complete compressed body was read fails as Distribution is not valid gzip.
• Empty bodies (Distribution is empty) and bodies that parse to zero triples (Distribution contains no RDF triples) fail the probe. A deliberately truncated prefix is never mistaken for either – it is inconclusive.
• Reachability is settled by the response, so validation never turns a reachable dump into a failure. If no triple surfaces within rdfValidationBudgetMs (default min(timeoutMs, 2000), clamped to timeoutMs), the read is aborted and the distribution is reported reachable but unvalidated (no failureReason). This bounds the extra latency content validation adds on slow, generate-on-the-fly endpoints.

Request headers and credentials

Pass extra HTTP headers (e.g. a User-Agent) via ProbeOptions.headers; they are merged with the probe-generated headers, and caller-supplied values win on conflict.

A distribution URL with embedded credentials (https://user:pass@host/path) is supported: the credentials are stripped from the URL and sent as an Authorization: Basic header instead – unless the caller already set an Authorization header, which takes precedence.

Network errors

A thrown exception from fetch (DNS failure, connection refused, socket reset, TLS error, timeout after the configured timeoutMs – default 5 000 ms) is a connection-level failure. The probe retries these up to retries times (default 2) with a linear backoff – the nth retry waits n × 250 ms – before giving up and returning a NetworkError. This turns a transient transport blip into a reliable single measurement without looking backward across checks. A genuine outage still resolves to a NetworkError on the current check – every attempt fails – but note each attempt gets its own timeoutMs, so an endpoint that fails only by timing out takes up to (retries + 1) × timeoutMs (plus backoff) to be reported down. HTTP error responses (4xx/5xx) and content-validation failures are real ‘down’ states and are never retried.

NetworkError.message includes the underlying error.cause (e.g. ECONNRESET, UND_ERR_SOCKET “other side closed”) when Node wraps one, so observations record what actually failed rather than a bare ‘fetch failed’.

Result fields

SparqlProbeResult and DataDumpProbeResult share:

• statusCode, statusText – the HTTP response status
• contentType – the response Content-Type header, or null
• lastModified – parsed from the Last-Modified header, or null
• failureReason – why the probe failed despite an HTTP response (invalid SPARQL results, failed content validation, …), or null
• warnings – soft findings such as a Content-Type mismatch; never affect success
• responseTimeMs – the successful attempt’s latency
• isSuccess() – true when the status is 200–399 and failureReason is null; a SparqlProbeResult additionally requires the response Content-Type to be one of its acceptedContentTypes

SparqlProbeResult adds acceptedContentTypes: the content types the probe was prepared to accept as a valid answer for the query type. DataDumpProbeResult adds contentSize, parsed from the Content-Length header (or null).

NetworkError carries url, message, and responseTimeMs – the total time spent failing, across every attempt and backoff, so observations do not understate the cost of a down endpoint.

Probing many distributions

probeMany probes an array of distributions concurrently and returns one result per input, in input order. Each distribution is probed once with probe, so every behaviour above applies per distribution; like probe, probeMany never throws – a probe that fails is reported as a NetworkError in its slot.

import { probeMany } from '@lde/distribution-probe';

const results = await probeMany(distributions, {
  concurrency: 20, // max probes in flight across all hosts (default 20)
  perHostConcurrency: 4, // max probes in flight against one host (default 4)
  validateRdfContent: true, // any ProbeOptions are forwarded to each probe
});

Two caps bound the batch:

• concurrency bounds the total fan-out, so a large catalogue does not exhaust sockets or buffer too many response bodies at once.
• perHostConcurrency bounds the burst any one server sees, keeping the batch a polite client: a catalogue that declares many distributions on a single host (e.g. a download endpoint per named graph) will not trip that server’s rate limiter (HTTP 429). Distributions sharing a host (by accessUrl) contend for the same budget; a probe whose host is saturated waits while probes for other hosts proceed, so one busy host never idles the global pool.

All other ProbeOptions (timeoutMs, retries, validateRdfContent, and the rest) are forwarded unchanged to every probe.

Progress reporting

onProgress is called once after each probe settles, with (completed, total) – the number of probes completed so far and the batch size. It fires total times, ending at (total, total), in completion order rather than input order, and is never called for an empty batch. A throwing callback rejects the whole batch, so keep it cheap and side-effect-only:

const results = await probeMany(distributions, {
  onProgress: (completed, total) =>
    process.stderr.write(`\rProbed ${completed}/${total}`),
});