a05cc51916
Some checks failed
Infra CI / notebooks (push) Successful in 1m6s
CI / lint (push) Successful in 46s
CI / notebooks-smoke (push) Successful in 1m44s
CI / test (push) Failing after 2m36s
Infra CI / zotero (push) Successful in 21s
Infra CI / llm (push) Successful in 55s
Infra CI / bcda (push) Successful in 1m38s
Infra CI / docs (push) Successful in 1m46s
Infra CI / api (push) Successful in 1m8s
Infra CI / mc (push) Failing after 36s
CI / lint (pull_request) Successful in 43s
CI / test (pull_request) Failing after 2m30s
Infra CI / zotero (pull_request) Successful in 15s
Infra CI / llm (pull_request) Successful in 16s
CI / notebooks-smoke (pull_request) Successful in 1m52s
Infra CI / docs (pull_request) Successful in 15s
Infra CI / api (pull_request) Successful in 15s
Infra CI / notebooks (pull_request) Successful in 49s
Infra CI / bcda (pull_request) Successful in 18s
Infra CI / mc (pull_request) Failing after 35s
Design: docs/superpowers/specs/2026-10-25-bcda-since-reconciliation-design.md. Until now a _since export was loaded like a full one: the loader took the newest export directory per type, so a delta silently replaced the baseline it was meant to extend. bcda.sync makes incremental pulls safe end to end. First order — which row is current. A baseline (no _since) is the authoritative snapshot for its types; deltas of the same population at or after its cut-off extend it. Keys: Patient BENE_SK, Coverage id, EOB claim control number (each claim *version* has a new FHIR id — migration guide). Winner per key: newest meta.lastUpdated, then NCH/DDPS over SharedSystem (the connector's rule), then later transactionTime, later applied sequence, larger id. NotFinalAction winners stay as tombstones; only a baseline retires rows (the API cannot express deletion). eob_item follows its header. Second order — what a job did, per table, in the ledger: new / replaced / unchanged / stale_ignored, backfill (lastUpdated < _since for an enrollee we did not have — the API's promise for newly attributed enrollees on Group/all) vs late (same shape for one we had — should not happen), tombstoned, adjusted (clm_orig_cntl_num → bcda.eob_supersession), retired, orphans (claims/coverage without a current Patient), scope drift. Third order — invariants on every apply before anything is published: unique keys, count conservation (|cur'| = |cur| + new − retired), watermark monotone, every item has a header; replay determinism and idempotency are tested (apply in either order → byte-identical tables; re-apply → no-op). Current tables are re-derived from the applied partitions each time, so the state is a function of the set of applied jobs, not their arrival order. Temporality checks before a _since export starts (block/warn/ok): format, future (server 400s), floor (no v3 lastUpdated before 2026-03-26), no_baseline, gap (since > watermark: a window nobody would ever fetch), overlap (free), endpoint_mix (runout vs current population), backfill_scope (/Patient delta over a Group/all baseline), scope_filter (_typeFilter drift), inflight, unapplied, attribution_newer, stale_baseline. next_since() is the previous job's transactionTime, as BCDA recommends; a combined job takes the earliest watermark of its types. Store: job rows carry transaction_time and request, exports/<key>/ manifest.json keeps the completed-job body, resume() fills both in for jobs completed before this; adopted jobs read _since from their request line; a delta with nothing to deliver (200, empty output) still counts and advances the watermark. flatten_export(out_dir=) writes per-job partitions under flat/history/<key>/; current tables keep _job_key/_transaction_time/ _since/_kind/_applied_seq and the connector's _ensure() drops them before the stage contracts. load_bcda(reconcile=True) applies pending jobs and also loads bcda.sync_ledger, bcda.sync_watermark, bcda.eob_supersession; an explicit ndjson_dir or a store without job rows keeps the legacy path. Surfaces: `stack bcda sync status|check|apply|pull`, `stack bcda export --since auto [--force]`, `stack bcda resume`; the manager's Exports page gets a _since mode (off / watermark / date) with the checks shown and blocks respected, and a Sync page (watermarks, pending, reconcile, ledger); notebook section 8. Verified on the real Large-entity store: applying the three baselines over the legacy flat/ tables reported every row unchanged (0 new, 0 retired) and loaded counts identical to the full load (221,021 medical lines, 11,968 pharmacy, 9,582 eligibility); a real /Patient-endpoint EOB delta with a _typeFilter reconciled as 248 unchanged with the drift on record; the sandbox's transactionTime is midnight UTC of the run day, so a same-day zero-row delta is correctly ignored as predating its baseline. refs #1139