33. Report the values a typed field could not read
Date: 2026-11-10
Status
Accepted
Context
A typed field reads a value or leaves it out. A date over Z.j, an integer over onbekend, a reference over a literal: each is dropped, and the document is indexed without it. Nothing says so. A publisher’s unreadable dates vanish from every date filter, and the only way to find them is to inventory the dumps by hand.
That inventory, run on the four Linked Open Limburg datasets on 10 November 2026, found 1 unreadable dateCreated of 756 in the Limburgs Museum, 11 of 136 in the Discovery Museum and 27 temporalCoverage values of 31 in Tracé decorbladen (ldelements/lde#1022). LDMax declined to reinterpret such dates in its ETL and asked for them to be corrected in the collection system (mightymax/ldmax-feedback#16). A correction at the source needs the list first.
The pipeline already reports items a writer refuses (itemsRejected). That is a different failure: the item is gone, where here only a value is.
Decision
The projection reports every value a typed field leaves out, and the pipeline reports them per dataset.
- Every kind that can refuse a value reports:
integerandnumber(NaN),date,period,reference(a literal where an IRI belongs) andboolean(a value outsidetrue/false/1/0). Akeywordortextfield reads any literal, so it has nothing to report. Reporting two kinds would let a reader assume the others are clean. - Both value routes report. A
derivemostly passes a source value through, so a value it returns that the field cannot read is usually the source’s too; when it is the derive’s own bug, that is worth surfacing as well. A value that is not a string is reported asString(value). @lde/searchreports, and knows nothing of reporters.ProjectionContext.onUnreadable({ field, value, id })is called once per value.fieldis dotted for a nested referent;idis the node the document came from, which is what a publisher can look up.@lde/pipelinetallies. ABatchTransformreports throughBatchContext.valueUnread(target, field, { id, value }), and the pipeline keeps a count and at most 5 distinct examples per (target, field) for the dataset in progress. It emitsProgressReporter.valuesUnreadper field just beforedatasetComplete, and nothing for a dataset that failed, since nothing of it is kept. Tallying in the pipeline, rather than insearchIndexerPipeline, means aPipelinebuilt fromsearchStagesdirectly reports too.ConsoleReporterprints it under the dataset, shaped likeitemsRejected, rather than in a summary at the end of the run, which would hold every dataset’s report until then.
Consequences
- An operator sees, per dataset and field, how many values the index lost and a few of them with the node they came from.
- Memory follows the schema, not the data: one tally per field that rejected anything, bounded in examples (ADR 12).
- Every
BatchTransformreceivesvalueUnread, and may ignore it. - A report a publisher reads without an operator in between is not part of this. It is another reporter on the same hook.