Editorial illustration in editorial-gouache style depicting a batch validation pipeline with cache warmup, parallelized workers, and per-fingerprint aggregation
by: Daniel Friedman

Resource Validation in a Batch Pipeline

Validating one resource on a REST write is a lightweight operation. Validating a million resources in an import pipeline is a different shape entirely. Cache warmth, batching semantics, per-resource result accounting, and downstream OperationOutcome aggregation all become first-order concerns. Designing the pipeline for validation up front is cheaper than retrofitting it after the pipeline is chugging on a large export. The site's R4 resource vetter is single-resource by scope, but the mechanics carry over. For the wider FHIR framing, FHIR vendor evaluation guides have more.

The Shape Of A Batch Pipeline

  • Ingest source — ndjson, Bundle, streaming resource events
  • Structural validation stage
  • Optional profile validation stage
  • Optional terminology validation stage
  • Result accumulation — pass, fail per resource
  • OperationOutcome emission — per resource or aggregate

Each stage is a candidate for parallelization. Each has different cache and warm-up characteristics.

Warm The Terminology Cache First

The largest single latency in a validation pipeline is the first-time value-set expansion. Warm the cache before the pipeline starts:

  • Load all value sets required by the profiles you validate against
  • Load all code systems referenced by those value sets
  • Verify the load succeeded before the pipeline processes its first resource

That warm-up is a fixed cost. Skipping it produces a slow-start pattern where the first thousand resources take three times as long as the millionth.

For the mechanics of terminology cost, terminology validation and the tradeoffs of enabling it is the entry.

Parallelize Structural, Serialize Terminology

Structural validation is stateless — each resource is independent. Parallelize aggressively.

Terminology validation shares a cache. Concurrent access is fine but the cache warmup should be serialized. Once warm, terminology validation scales with worker count.

Profile validation sits in between — StructureDefinition loading is shared, invariant expression evaluation is per-resource.

Accumulate Results With Per-Resource Fingerprints

For a million-resource batch, storing every OperationOutcome individually is expensive. Aggregate by fingerprint at emission time:

  • Group issues by (severity, code, expression-tail)
  • Store the aggregate group with a count and a sample per-resource entry
  • Store the per-resource pass/fail decision

That aggregation reduces storage by orders of magnitude and produces a triage-friendly summary. For the response-shape side, OperationOutcome design for validator responses is the entry.

Fail-Open Or Fail-Closed

  • Fail-open — resources that fail validation still land, marked for review
  • Fail-closed — resources that fail validation do not land

For batch imports, fail-open is often the right choice. The pipeline continues, the downstream systems see the loaded data with a warning flag, and the operator triages later.

Fail-closed produces "the batch is failing, we do not know why" incidents. Fail-open with clear accounting is usually better.

Retry Policy For Transient Failures

Terminology service outages should not fail a batch. Cache misses on a warm cache should not either. Both are transient.

Design the pipeline to retry transient failures a small number of times before falling through to fail-open. Structured failures — malformed JSON, missing required elements — should not retry.

Progress Reporting

A pipeline validating a million resources should emit progress. Percent complete, per-fingerprint counts, throughput per minute. Silent pipelines are indistinguishable from stuck pipelines.

The reporting cost is small. The operational win is large.

The Boundary With Bundle Validation

Batch pipelines usually process resources one-at-a-time, not as Bundles. Cross-resource reference validation is out of scope. If your pipeline needs it, run a second pass after the per-resource validation completes. For the mechanic, single-resource validation vs Bundle validation: knowing the boundary covers the split.

The Short Version

Warm the terminology cache first. Parallelize structural, share terminology, cache profiles. Aggregate results by fingerprint. Fail-open with accounting. Retry transient failures. Report progress. Cross-resource checks are a second pass.

Editorial-gouache diagram of a batch validation pipeline with cache warmup, parallelized structural workers, shared terminology cache, and per-fingerprint aggregation stages annotated, in soft coral gouache tones on cream paper

Sources