stack/docs/superpowers/specs/2026-10-25-bcda-v3-design.md

P52 — BCDA v3: client, crosswalk-driven flatteners, vendored upstream, proving notebook

Tracker: milestone P52, #1139. Builds on the v2 bcda module (client/store/flatten, #6, #8) and the bcda.express.cclf input-layer pipeline.

Why now

BCDA v3 went GA on 2026-07-01 and CMS discontinues v1/v2 on 2027-07-30. The module targets v2 end to end: Client defaults to /api/v2, and bcda.express.flatten matches BlueButton extensions by the retired https://bluebutton.cms.gov/resources/variables/<name> URL suffix. On a v3 payload that match never fires, so almost every coded column comes out NULL without an error. The live spike against the public Large Model Entity sandbox (2026-10-25) confirmed the transport still works on v3 and that the payload is where the migration lives.

Sources of truth (in priority order)

1. BCDA v3 Data Dictionary (BCDA_v3_Data_Dictionary.xlsx, published 2026-07-01). Each v3 field has a fhirPath, the IDR sourceColumn, and two crosswalk columns: ccwMapping (the CCW/v2 variable names the current table models are named after) and cclfMapping (CCLF file.element). The CCLF-BCDA v3 Mapping sheet is the CCLF-side view of the same crosswalk; CCLF fields absent from BCDA v3 lists what cannot be reproduced (HICNs, FIPS county, address lines, BETOS, several CCLFA/B amounts).
2. Migration guide (/api-documentation/how-to-migrate-v3.html) and the July 9 2026 office-hours transition deck: endpoint path change only; /auth and /data unchanged; Claim/ClaimResponse retired into EOB; _typeFilter with the System-Type tag (SharedSystem, NationalClaimsHistory, DDPS; default omits SharedSystem); extension/CodeSystem URLs move to fhir/StructureDefinition/<CODE> and fhir/CodeSystem/<CODE>; Patient/Coverage/EOB ids are new (match on MBI and claim control number); Coverage drops reference year/month for Coverage.period and gains a Dual Medicare/Medicaid row; BETOS removed; v3 deduplicates claim versions within a source.
3. Official v3 sample files (v3_Patient/Coverage/ExplanationOfBenefit.ndjson from bcda.cms.gov) — the fixtures the flatteners are tested against.
4. CMSgov/bcda-app — the server. Vendored sparsely: the v3 handler and its tests, request validation (_typeFilter, _since, Prefer/Accept rules), the v3 OperationOutcome writer, the v3 Postman collection, gzip download fixtures, the sandbox model config, and LICENSE.

Decisions

1. One set of tables, two payload dialects. bcda.patient, bcda.coverage, bcda.explanation_of_benefit, bcda.eob_item keep their CCW-named columns (the ccwMapping column is exactly the crosswalk from v3 fields onto them). v3-native data with no v2 home gets new nullable columns rather than being crammed into monthly slots. The flattener detects the dialect per resource (meta.profile carries C4BB-…|2.2.0, or any bluebutton.cms.gov/fhir/ URL) and dispatches; v2 seeds and tests keep passing.
2. v3 extraction is table-driven. bcda.express.v3 has a handful of primitives that mirror the dictionary's fhirPath idioms (identifier by system, type coding by system, supportingInfo by category code → code/timing/value, header adjudication/total by Adjudication code, item adjudication by Adjudication code, item extension by StructureDefinition code, careTeam by role, contained provider by provider.reference) and declarative maps from v3 codes to columns. Every map entry cites the dictionary sourceColumn, so a reader can check it against the spreadsheet.
3. Identity follows the migration guide. bene_id is the v3 Patient.id (BENE_SK; the EOB patient.reference uses it). bene_mbi_id is the current MBI (the us-mbi identifier without a period.end, else the latest). clm_id is the C4BB unique claim id; clm_cntl_num is the claim control number and the key to pair a SharedSystem EOB with its NationalClaimsHistory successor. system_type and final_action come from meta.tag; eob_type from the EOB-TYPE coding, clm_type_cd from CLM-TYPE-CD.
4. Coverage is one row per plan per period, as v3 delivers it: class_plan ∈ {Part A, Part B, Part C, Part D, Dual Medicare/Medicaid}, period_start/period_end plus the plan's extensions in v3-named columns. The legacy part-A/B/D begin/end columns are filled from the matching plan's period so bcda.express.cclf keeps working; dual_01 is left NULL and the pipeline reads coalesce(dual_01, dual_stus_cd) after a self-join to the Dual row — the only pipeline change in scope.
5. Race. v3 moves race to the US Core extension (OMB codes). us_core_race holds the OMB code; bene_race_cd is derived through the published CCW crosswalk (2106-3→1, 2054-5→2, 2028-9/2076-8→4, 1002-5→6, Hispanic ethnicity→5, unknown→0) so the eligibility race labels do not go blank.
6. Client: default v3; type_filter= on start_export/export; jobs() sends Prefer: respond-async + Accept: application/fhir+json (required on v2 and v3) and returns the Task resources from the Bundle; every non-2xx surfaces the OperationOutcome diagnostics (BcdaHttpError, still an httpx.HTTPStatusError); a poll that hits 404 Job has been cancelled raises JobCancelledError (the public sandbox credentials are shared, so this happens). /_version and /_health are exposed for the notebook's preflight.
7. Crosswalk is data, not prose. The dictionary and both CCLF sheets are flattened to dev/seeds/bcda_v3_data_dictionary.csv, dev/seeds/bcda_v3_cclf_crosswalk.csv, dev/seeds/bcda_v3_cclf_absent.csv and land in reference_data via stack load seed, so the notebook (and later the chat) can show, per table column, where it came from and which CCLF element it replaces.
8. Vendoring is a manifest, not a submodule. dev/vendor/MANIFEST.toml pins the bcda-app commit and lists each vendored path with its git blob sha; bcda.cms.gov downloads are pinned by sha256. dev/scripts/vendor_bcda_app.py --check compares against upstream (GitHub tree API + content hashes) and exits non-zero on drift; --sync refreshes files and the manifest. A weekly workflow runs --check and files an issue on drift (no auto-commit: a payload change needs a human to look at the flattener). tests/bcda/test_vendor.py verifies the tree matches the manifest offline, and the flattener tests run on the vendored official samples.
9. The notebook proves, it does not analyse. notebooks/bcda_v3_sandbox.py walks preflight → auth → attribution → export (reusing a completed job younger than 24 h when one exists, since a fresh Large-entity export takes 20–40 min) → Store download → flatten → DuckDB load (duckdb_batch + publish_replica, never a long-lived writer) → checks. Each check is a sentence with a number and a pass/fail mark: every Patient has an MBI, Coverage rows per plan, EOB counts by System-Type × EOB-TYPE, claim-control-number uniqueness within a source, item amounts populated, dictionary field population per table. Pulls are gated behind BCDA_PULL=1 or the button so the nightly notebooks-integration run only re-validates data already on disk.

Data flow

sandbox /api/v3 ──Client──▶ Store (data/bcda/exports/<key>/*.ndjson, jobs/files parquet)
                              └─ flatten_export ──dialect?──▶ v2 flatteners (unchanged)
                                                 └──────────▶ bcda.express.v3 ──▶ flat/*.parquet
                                                                                   └─ load_bcda ──▶ aco.duckdb bcda.* ──▶ publish_replica
BCDA_v3_Data_Dictionary.xlsx ──gen_bcda_v3_crosswalk.py──▶ dev/seeds/bcda_v3_*.csv ──stack load seed──▶ reference_data.*
CMSgov/bcda-app + bcda.cms.gov ──vendor_bcda_app.py──▶ dev/vendor/{bcda-app,bcda-site}/ (+ MANIFEST.toml) ──▶ tests, weekly drift check

Out of scope

Reworking bcda.express.cclf semantics for v3 (claim routing by nch_near_line_rec_ident_cd still works because v3 carries CLM_NRLN_RIC_CD; a v3-native routing on eob_type is a follow-up). Partially adjudicated exports end to end (the client supports _typeFilter; the sandbox Large entity is adjudicated-only). Databricks/Iceberg landing of the new columns.

Results (2026-10-25, Large Model Entity sandbox)

Pulled as three single-type jobs: the EOB job (89596, 6 min server-side, 192 files / 2.2 GB) was adopted in 44 s; Patient (11 MB) and Coverage (44 MB) exports took 24 and 32 min, most of it queued as Pending. Flatten + load of the whole pull: ~2 min.

table	rows
bcda.patient	9,582
bcda.coverage	19,396 (Part A 5,643 · Part B 5,649 · Part C 3,236 · Dual 4,868)
bcda.explanation_of_benefit	44,361 (PDE 14,395 · CARRIER 6,298 · DME 6,242 · SNF 5,813 · HOSPICE 2,942 · INPATIENT 2,910 · OUTPATIENT 2,883 · HHA 2,878)
bcda.eob_item	248,528

All 20 notebook checks pass: every Patient has a unique current MBI and bene_id == Patient.id; every Coverage row has period.start, every Part A row ms_cd, every Dual row dual_stus_cd, no orphan beneficiaries; every EOB has a claim id, a claim control number, System-Type and Final-Action tags, EOB-TYPE and CLM-TYPE-CD; control numbers are unique within a source (v3 deduplication holds); lines carry HCPCS or NDC on 100% and paid/submitted amounts on 94% (Part D lines carry drug-cost amounts instead); no orphan lines. The entity is adjudicated-only (no SharedSystem claims), so the control-number pairing check is exercised but finds zero pairs — partially adjudicated data needs an entity with PAC access and _typeFilter.

Connector pipeline on v3 semantics (bcda.express.cclf, 2026-10-25)

The input_layer connector was still written for v2 and had never run end to end (its contracts named CCLF columns the EOB table does not have, and the runner's exact-match check would have refused every stage). Rewritten:

1. One header stage carries the v3 semantics. _stg_claim_header drops NotFinalAction claims, derives claim_family from EOB-TYPE → CLM-TYPE-CD → pde_id → the v2 near-line RIC (which v3 emits on only some claims: HHA shows U/W/V/NULL), collapses a claim seen in two sources on its claim control number (NationalClaimsHistory/DDPS over SharedSystem, then newest last_updated; rows without a control number are never collapsed), and resolves the MBI. The four family stages are filters on it.
2. One mapping for medical_claim. _medical_claim(kind) produces the full aco.table.input_layer.InputLayerMedicalClaim shape for institutional, carrier and DME lines: admission/discharge from the C4BB admission period, NUBC type of bill (clm_bill_type_cd, rebuilt as 0 + the three-character v3 frequency code when the coding is missing), MS-DRG, revenue centre, HCPCS and five modifiers, attending→operating→rendering / rendering / rendering→ordering NPIs, billing Organization NPI and tax number, paid / allowed / charge / coinsurance / deductible from the consolidated v3 Adjudication amounts, paid date from clm_pd_dt → NCH weekly process → CMS process date, 12 header diagnoses with POA on the first, first ICD procedure.
3. Part D uses the PDE id, the prescribing care-team NPI, the contained pharmacy Organization NPI, NDC quantity, total drug cost as charge and paid-by-patient as copayment; plan-paid falls back line → header (the sandbox emits neither).
4. Eligibility reads the Part A row (Part B fallback) for status, entitlement and the enrollment period and the Dual row for dual status; gender accepts CCW codes and the FHIR words the v2 flattener passed through; enrollment_status comes from Coverage.status.
5. Contracts inherit. The header stages subclass ExplanationOfBenefit; the outputs subclass the canonical InputLayer* models, so the same downstream pipelines run on BCDA output as on CCLF output. load_bcda(run_pipeline=True) / stack load bcda --run-pipeline and stack run bcda land input_layer.medical_claim, pharmacy_claim, eligibility.

Run on the Large-entity pull (94 s including re-flattening): 221,021 medical lines on 28,282 claims (16,468 institutional, 11,814 professional — equal to the FinalAction non-PDE EOB counts by type; 4,111 NotFinalAction excluded), 11,968 pharmacy rows (= FinalAction PDEs), 9,582 eligibility rows (7,290 with a Part A/B row, 4,868 with dual status). Every institutional line has a 4-character type of bill, admission date, paid, charge, deductible and coinsurance; every professional line has paid, rendering NPI and (99 %) place of service.

BCDA manager (bcda.ui, stack bcda, 2026-10-25)

A Streamlit app and a CLI group over the same service layer, mirroring the surface of the CMS server (CMSgov/bcda-app, vendored): /_version, /_health, metadata, /auth/token, attribution_status, $export on Group/all · Group/runout · Patient with _type / _since / _typeFilter, job history, status, cancel, /data download — plus the local side: the Store's job and file history, flatten + load into bcda.*, the connector into input_layer.*, the replica for browsing, the v3 data dictionary and CCLF crosswalk, and the vendored-upstream manifest.

Decisions:

1. bcda.ui.service has no Streamlit in it. Every operation takes explicit objects (a Connection, a Client, a Store), so the CLI reuses it verbatim and tests hand in mock transports and temp directories. Views are thin.
2. Connections are presets, environment or custom. The four public sandbox entities from bcda.cms.gov are built in (the legacy REACH pair pinned to v2); BCDA_CLIENT_ID / BCDA_CLIENT_SECRET (+ BCDA_BASE_URL, BCDA_API_VERSION) win when set; production is a selectable environment.
3. Long work runs on background threads (bcda.ui.runner): a process-level registry that survives Streamlit reruns, captures the bcda.* log lines emitted on each run's own thread (job started, progress, files stored) and flips state last so readers never see a finished run with an incomplete log. A st.fragment(run_every="5s") panel shows runs on every page that starts them.
4. Exports default to one job per type with cancel retries, because that is the only shape that survives the shared sandbox's sweeps; the _typeFilter builder composes the System-Type _tag, outcome and service-date subqueries from the transition deck; completed jobs under 24 h are offered for adopt, in-progress ones for cancel.
5. Data pages read the read-only replica only, so the manager never holds the primary's single-writer lock; the load page republishes the replica after load_bcda(run_pipeline=…).
6. Launch: stack bcda ui (port, headless flags) runs streamlit run src/bcda/ui/app.py; Streamlit lives in the bcda extra. Containerising it (image + traefik route + dashboard tile) is deliberately left out of this slice.

Tests: service with httpx.MockTransport, runner for thread-routed logs and state ordering, every page through AppTest.from_string with the service stubbed (buttons, forms, error paths), the app shell through AppTest.from_file, and the CLI through CliRunner — all at 100 % line coverage.

Containerised (2026-10-25)

infra/images/bcda.Dockerfile (same base and install pattern as the api image; uv sync --extra cli --extra bcda, Streamlit settings in infra/images/bcda/streamlit.toml, health at /_stcore/health), compose service bcda on the gateway network with ./data read-write (exports, aco.duckdb, replica publish) and ./dev read-only (vendored upstream, crosswalk seeds, vendor script), Traefik route bcda.<domain> behind git-sso, a dashboard tile, the bcda subdomain in stack.toml (CoreDNS hosts and the SSO cert list follow) and an [images.bcda] entry so the deploy workflow builds, scans and pushes it on changes to src/**, pyproject.toml or the image files. Public sandbox credentials are built in; BCDA_CLIENT_ID / BCDA_CLIENT_SECRET / BCDA_BASE_URL in .env override.