stack/docs/superpowers/specs/2026-10-25-bcda-since-reconciliation-design.md

BCDA _since pulls: ingest, reconcile, and prove it — design (2026-10-25)

Part of P52 (#1139). Companion to 2026-10-25-bcda-v3-design.md.

Intent

A _since export is cheap (minutes, megabytes) where a full pull is not (the Large entity's EOBs are 6 min and 2.2 GB). Today the manager can start one (the Exports page has a date picker, the CLI a --since flag) but nothing downstream knows what to do with the result: the loader flattens "the newest export directory per type", so a delta silently replaces the full pull it was meant to extend. This design makes incremental pulls safe end to end: choose the right _since, refuse the wrong one, ingest the delta, reconcile it with everything already on disk under explicit tiebreakers, and keep a ledger that proves the invariants held.

What the API actually promises (sources of truth)

From bcda.cms.gov (How to Filter Claims Data, How to Migrate to v3, FAQs) and the vendored bcda-app (validation.go, requests.go):

• _since filters on meta.lastUpdated, FHIR instant with offset; the server rejects an unparseable value or one in the future (400).
• /Group/all with _since returns resources updated after the instant for existing enrollees and all resources for newly attributed enrollees (RetrieveNewBeneHistData); /Patient returns only updated resources. So a delta legitimately contains rows whose lastUpdated predates _since — backfill for new beneficiaries, not late data.
• Recommended cadence: one unfiltered pull, then each later pull's _since = the previous job's transactionTime (in the completed-job manifest).
• No v3 resource has lastUpdated before 2026-03-26; an earlier _since is a full pull in disguise.
• Each version of a claim has a unique FHIR id; track claims by claim control number (identifier.system = …/CLM-CNTL-NUM). Corrected claims show a newer lastUpdated. An adjustment may carry the original control number as a related claim (clm_orig_cntl_num).
• Final-Action tag: FinalAction / NotFinalAction (9% of the Large entity). Within one snapshot a control number appears once (verified on the real pull: 0 of 9,342 control numbers repeated; 887 all-NotFinalAction).
• Default _typeFilter omits SharedSystem; a delta pulled with a different filter scope than its baseline covers a different population of claims.
• Attribution files update monthly; attribution_status reports last_attribution_update. Enrollees who leave attribution are not signalled by a delta — only a new baseline reveals attrition.
• Completed jobs serve files for 24 h.

Observed on the real Large-entity pull: Patient ids are BENE_SK (-100040134), Coverage ids are composite (part-c--100040134-Z0004--01), EOB FHIR ids are opaque with a CLM-CNTL-NUM identifier on every row; meta.versionId is absent everywhere, so lastUpdated is the only version clock.

Decisions

Vocabulary

• Baseline: a complete job without _since. Authoritative snapshot for the types it carries, as of its transactionTime.
• Delta: a complete job with _since.
• Population: current (Group/all and /Patient both describe the currently attributed enrollees) or runout (Group/runout, last year's). This is the hard boundary: a delta never feeds a baseline of the other population. A /Patient delta over a Group/all baseline is allowed and warned (backfill_scope: no history for newly attributed enrollees).
• Scope: (population, _typeFilter). Watermarks are kept per scope and type. A delta whose _typeFilter differs from its baseline's is still reconciled (its rows are genuine newer versions) but warned (scope_filter) and noted in the ledger as scope drift: claims outside its filter were not refreshed.
• Watermark per (scope, type): the greatest transactionTime among complete (downloaded) jobs. The next _since is the watermark.
• Applied: a job whose rows have been reconciled into the current tables (recorded in the sync ledger).

First order — which row is current (per resource)

Identity keys:

table	key	why
bcda.patient	bene_id (falls back to id)	Patient.id is BENE_SK; MBIs rotate
bcda.coverage	id	composite bene × plan × sequence
bcda.explanation_of_benefit	clm_cntl_num, else clm_id, else id	a claim version gets a new FHIR id; the control number is the claim
bcda.eob_item	follows its header (clm_id + job)	lines are replaced wholesale with their header

Candidate rows for a type = rows of the newest applied baseline carrying it ∪ rows of applied deltas of the same population whose transactionTime ≥ that baseline's. Deltas older than the authoritative baseline are ignored for current state (the baseline already contains their effect, and must be allowed to retire rows they carried); deltas of the other population are ignored with a ledger note.

Winner per key, in order, each applied only when the previous ties:

1. greatest last_updated (the resource's own version clock);
2. source rank — NationalClaimsHistory/DDPS over SharedSystem (the connector's existing rule, so the two layers agree);
3. greatest job transactionTime (the server's view is later);
4. greatest applied sequence (we saw it later);
5. greatest FHIR id string (total order; never reached in practice, present so the result is a pure function of the inputs).

A winner tagged NotFinalAction stays in the table as a tombstone: it records that the claim's latest version is not final, and the connector (_stg_claim_header) already drops it from input_layer. Rows are never deleted from current tables by a delta; only a baseline retires rows.

Second order — consistency across resources and across time

Computed per applied job and stored in the ledger as counts:

• new / replaced / unchanged / stale_ignored — key not seen before / seen with a different id or older last_updated / identical id and last_updated (an overlap re-delivery) / older than what we hold.
• backfill vs late — delta rows with last_updated < _since. Backfill is the API's promise for newly attributed enrollees (bene not in the prior Patient set); late is the same shape for a bene we already had, which the API says should not happen, so it is a warning.
• tombstoned — winners tagged NotFinalAction.
• adjusted — EOB winners carrying clm_orig_cntl_num; the original control number is marked superseded in bcda.eob_supersession so analyses can follow the chain.
• retired — keys present before a baseline that the baseline no longer carries (attrition, opt-out, or scope change). Only a baseline can retire.
• orphan_claims / orphan_coverage — EOB or Coverage rows whose beneficiary is not in current Patient (Patient delta missing, or Patient baseline older than the claims baseline).
• scope_drift — a delta whose _typeFilter differs from its baseline's.

Third order — invariants over the whole store, checked on every apply

1. Keys are unique in every current table.
2. Count conservation per type: |current'| = |current| + new − retired.
3. Watermark per (scope, type) never decreases.
4. Every eob_item.clm_id belongs to a current EOB header.
5. Replay determinism: current state is a pure function of the set of applied jobs — applying in a different order yields identical tables (tested), and re-applying a job is a no-op (idempotent; tested).
6. The ledger is append-only and every current row names the job it came from (_job_key, _transaction_time), so any value can be traced to an ndjson line on disk.

A violated invariant raises SyncInvariantError, the ledger row is written with ok = false, and the current tables are left as they were.

Temporality checks before a _since export starts

check_since(store, since, endpoint, types, type_filter) returns findings with levels ok / warn / block; the UI and CLI refuse to start on block unless --force:

code	level	rule
format	block	not a FHIR instant with offset
future	block	later than now (server would 400)
floor	warn	before 2026-03-26 — this is a full pull in disguise
no_baseline	block	no complete baseline for a requested type in this scope
gap	block	since > watermark: the window (watermark, since) would never be fetched
overlap	ok	since < watermark: re-delivery, reconciliation is idempotent
endpoint_mix	block	baseline and delta describe different populations (Group/runout vs the rest)
backfill_scope	warn	/Patient delta over a Group/all baseline: newly attributed enrollees arrive without history
scope_filter	warn	_typeFilter differs from the baseline's; rows reconcile, nothing outside the filter refreshes
inflight	warn	a job for the same scope/type is still pending or polling
unapplied	warn	downloaded jobs not yet reconciled
attribution_newer	warn	last_attribution_update > baseline transactionTime: expect backfill for new enrollees; attrition stays invisible until the next baseline
stale_baseline	warn	baseline older than 24 h × 30: files are gone, only a new baseline can retire rows

next_since() formats the watermark as YYYY-MM-DDThh:mm:ss.sss+00:00.

Storage and data flow

exports/<key>/*.ndjson, manifest.json      (store: + transaction_time, request on the job row)
        │  flatten (existing v3 flatteners, per job)
        ▼
flat/history/<key>/{patient,coverage,explanation_of_benefit,eob_item}.parquet
        │  + _job_key, _transaction_time, _since, _kind, _applied_seq
        │  reconcile (DuckDB over the history partitions)
        ▼
flat/{patient,coverage,explanation_of_benefit,eob_item}.parquet   ← current state
flat/eob_supersession.parquet, flat/sync_ledger.parquet
        │  load_bcda (unchanged read path) → bcda.* → connector → input_layer.* → replica

load_bcda() gains reconcile=True: when the store has complete jobs it applies the unapplied ones and loads the current tables (plus bcda.sync_ledger, bcda.sync_watermark, bcda.eob_supersession). An explicit ndjson_dir keeps the legacy "flatten these directories" path.

Details settled while building:

• The store now writes exports/<key>/manifest.json (the completed-job body) and puts transaction_time and request on the job row; jobs recorded before that fall back to the manifest, then to the instant the job was started (its first ledger row): the server's snapshot cannot be newer than when it began processing, so that _since overlaps and never gaps, whereas the completion instant would be later than the true cut-off and could skip updates made while the job ran. The ledger records which source was used, and stack bcda resume <key> replaces the fallback with the server's value while the job is still listed. An adopted job's _since is read from its request line.
• Current tables carry _job_key, _transaction_time, _since, _kind, _applied_seq; the connector's _ensure() drops _-prefixed columns before the stage contracts see them.
• _since on a combined (multi-type) job resolves to the earliest watermark across the requested types.
• The sandbox reports transactionTime as midnight UTC of the run day (a /Patient delta run 19:25–19:38Z carried 2026-10-25T00:00:00Z), so a watermark _since can overlap by up to a day. Overlap reconciles to unchanged; it costs download time, never correctness.
• A delta with nothing to deliver (200, empty output) has no file rows; its manifest makes it a job and it advances the watermark.
• Verified on the real Large-entity pull (2026-10-25): applying the three baselines over the legacy flat/ tables reported every row unchanged, zero new/retired, and the loaded counts were identical to the earlier full load (221,021 medical lines, 11,968 pharmacy, 9,582 eligibility). Replay of the tests' delta scenario in either order yields byte-identical tables.

Surfaces

• bcda.sync: parse_instant, format_instant, watermarks, next_since, check_since, apply, status, SyncReport, SyncInvariantError.
• stack bcda sync status|check|pull|apply; stack bcda export --since auto.
• Manager: Exports page gets a _since mode (off / watermark / date) with the checks shown before the run and block respected; new Sync page (watermarks, pending, apply, ledger).
• Notebook section 8 · Incremental.

Non-goals

• Deleting rows on a delta. The API cannot express deletion; only a baseline retires rows.
• Reconciling across scopes (e.g. Group/all baseline with Group/runout deltas) or across API versions.
• _until (not offered by BCDA).

Assumptions (made here, not confirmed by the user)

• Claims identity by control number is correct for all three claim systems (NCH, DDPS, SharedSystem); for the rare row without one, the FHIR id is the key and such a claim can only be replaced by an identical id.
• Overlap by default is zero: _since = watermark exactly, as BCDA advises. SINCE_OVERLAP_SECONDS (conf) widens it; reconciliation makes overlap free.