stack/docs/superpowers/specs/2026-10-25-bcda-v3-design.md

# P52 — BCDA v3: client, crosswalk-driven flatteners, vendored upstream, proving notebook

Tracker: milestone P52, #1139. Builds on the v2 `bcda` module (client/store/flatten, #6, #8) and the `bcda.express.cclf` input-layer pipeline.

## Why now

BCDA v3 went GA on 2026-07-01 and CMS discontinues v1/v2 on 2027-07-30. The module targets v2 end to end: `Client` defaults to `/api/v2`, and `bcda.express.flatten` matches BlueButton extensions by the retired `https://bluebutton.cms.gov/resources/variables/<name>` URL suffix. On a v3 payload that match never fires, so almost every coded column comes out NULL without an error. The live spike against the public Large Model Entity sandbox (2026-10-25) confirmed the transport still works on v3 and that the payload is where the migration lives.

## Sources of truth (in priority order)

1. **BCDA v3 Data Dictionary** (`BCDA_v3_Data_Dictionary.xlsx`, published 2026-07-01). Each v3 field has a `fhirPath`, the IDR `sourceColumn`, and two crosswalk columns: `ccwMapping` (the CCW/v2 variable names the current table models are named after) and `cclfMapping` (CCLF file.element). The `CCLF-BCDA v3 Mapping` sheet is the CCLF-side view of the same crosswalk; `CCLF fields absent from BCDA v3` lists what cannot be reproduced (HICNs, FIPS county, address lines, BETOS, several CCLFA/B amounts).
2. **Migration guide** (`/api-documentation/how-to-migrate-v3.html`) and the **July 9 2026 office-hours transition deck**: endpoint path change only; `/auth` and `/data` unchanged; Claim/ClaimResponse retired into EOB; `_typeFilter` with the `System-Type` tag (`SharedSystem`, `NationalClaimsHistory`, `DDPS`; default omits SharedSystem); extension/CodeSystem URLs move to `fhir/StructureDefinition/<CODE>` and `fhir/CodeSystem/<CODE>`; Patient/Coverage/EOB ids are new (match on MBI and claim control number); Coverage drops reference year/month for `Coverage.period` and gains a Dual Medicare/Medicaid row; BETOS removed; v3 deduplicates claim versions within a source.
3. **Official v3 sample files** (`v3_Patient/Coverage/ExplanationOfBenefit.ndjson` from bcda.cms.gov) — the fixtures the flatteners are tested against.
4. **CMSgov/bcda-app** — the server. Vendored sparsely: the v3 handler and its tests, request validation (`_typeFilter`, `_since`, Prefer/Accept rules), the v3 OperationOutcome writer, the v3 Postman collection, gzip download fixtures, the sandbox model config, and LICENSE.

## Decisions

1. **One set of tables, two payload dialects.** `bcda.patient`, `bcda.coverage`, `bcda.explanation_of_benefit`, `bcda.eob_item` keep their CCW-named columns (the `ccwMapping` column is exactly the crosswalk from v3 fields onto them). v3-native data with no v2 home gets new nullable columns rather than being crammed into monthly slots. The flattener detects the dialect per resource (`meta.profile` carries `C4BB-…|2.2.0`, or any `bluebutton.cms.gov/fhir/` URL) and dispatches; v2 seeds and tests keep passing.
2. **v3 extraction is table-driven.** `bcda.express.v3` has a handful of primitives that mirror the dictionary's fhirPath idioms (identifier by system, `type` coding by system, supportingInfo by category code → code/timing/value, header `adjudication`/`total` by Adjudication code, item adjudication by Adjudication code, item extension by StructureDefinition code, careTeam by role, contained provider by `provider.reference`) and declarative maps from v3 codes to columns. Every map entry cites the dictionary `sourceColumn`, so a reader can check it against the spreadsheet.
3. **Identity follows the migration guide.** `bene_id` is the v3 `Patient.id` (BENE_SK; the EOB `patient.reference` uses it). `bene_mbi_id` is the current MBI (the `us-mbi` identifier without a `period.end`, else the latest). `clm_id` is the C4BB unique claim id; `clm_cntl_num` is the claim control number and the key to pair a SharedSystem EOB with its NationalClaimsHistory successor. `system_type` and `final_action` come from `meta.tag`; `eob_type` from the `EOB-TYPE` coding, `clm_type_cd` from `CLM-TYPE-CD`.
4. **Coverage is one row per plan per period**, as v3 delivers it: `class_plan` ∈ {Part A, Part B, Part C, Part D, Dual Medicare/Medicaid}, `period_start`/`period_end` plus the plan's extensions in v3-named columns. The legacy part-A/B/D begin/end columns are filled from the matching plan's period so `bcda.express.cclf` keeps working; `dual_01` is left NULL and the pipeline reads `coalesce(dual_01, dual_stus_cd)` after a self-join to the Dual row — the only pipeline change in scope.
5. **Race.** v3 moves race to the US Core extension (OMB codes). `us_core_race` holds the OMB code; `bene_race_cd` is derived through the published CCW crosswalk (2106-3→1, 2054-5→2, 2028-9/2076-8→4, 1002-5→6, Hispanic ethnicity→5, unknown→0) so the eligibility race labels do not go blank.
6. **Client**: default `v3`; `type_filter=` on `start_export`/`export`; `jobs()` sends `Prefer: respond-async` + `Accept: application/fhir+json` (required on v2 and v3) and returns the Task resources from the Bundle; every non-2xx surfaces the OperationOutcome diagnostics (`BcdaHttpError`, still an `httpx.HTTPStatusError`); a poll that hits `404 Job has been cancelled` raises `JobCancelledError` (the public sandbox credentials are shared, so this happens). `/_version` and `/_health` are exposed for the notebook's preflight.
7. **Crosswalk is data, not prose.** The dictionary and both CCLF sheets are flattened to `dev/seeds/bcda_v3_data_dictionary.csv`, `dev/seeds/bcda_v3_cclf_crosswalk.csv`, `dev/seeds/bcda_v3_cclf_absent.csv` and land in `reference_data` via `stack load seed`, so the notebook (and later the chat) can show, per table column, where it came from and which CCLF element it replaces.
8. **Vendoring is a manifest, not a submodule.** `dev/vendor/MANIFEST.toml` pins the bcda-app commit and lists each vendored path with its git blob sha; bcda.cms.gov downloads are pinned by sha256. `dev/scripts/vendor_bcda_app.py --check` compares against upstream (GitHub tree API + content hashes) and exits non-zero on drift; `--sync` refreshes files and the manifest. A weekly workflow runs `--check` and files an issue on drift (no auto-commit: a payload change needs a human to look at the flattener). `tests/bcda/test_vendor.py` verifies the tree matches the manifest offline, and the flattener tests run on the vendored official samples.
9. **The notebook proves, it does not analyse.** `notebooks/bcda_v3_sandbox.py` walks preflight → auth → attribution → export (reusing a completed job younger than 24 h when one exists, since a fresh Large-entity export takes 20–40 min) → `Store` download → flatten → DuckDB load (`duckdb_batch` + `publish_replica`, never a long-lived writer) → checks. Each check is a sentence with a number and a pass/fail mark: every Patient has an MBI, Coverage rows per plan, EOB counts by System-Type × EOB-TYPE, claim-control-number uniqueness within a source, item amounts populated, dictionary field population per table. Pulls are gated behind `BCDA_PULL=1` or the button so the nightly notebooks-integration run only re-validates data already on disk.

## Data flow

```
sandbox /api/v3 ──Client──▶ Store (data/bcda/exports/<key>/*.ndjson, jobs/files parquet)
                              └─ flatten_export ──dialect?──▶ v2 flatteners (unchanged)
                                                 └──────────▶ bcda.express.v3 ──▶ flat/*.parquet
                                                                                   └─ load_bcda ──▶ aco.duckdb bcda.* ──▶ publish_replica
BCDA_v3_Data_Dictionary.xlsx ──gen_bcda_v3_crosswalk.py──▶ dev/seeds/bcda_v3_*.csv ──stack load seed──▶ reference_data.*
CMSgov/bcda-app + bcda.cms.gov ──vendor_bcda_app.py──▶ dev/vendor/{bcda-app,bcda-site}/ (+ MANIFEST.toml) ──▶ tests, weekly drift check
```

## Out of scope

Reworking `bcda.express.cclf` semantics for v3 (claim routing by `nch_near_line_rec_ident_cd` still works because v3 carries `CLM_NRLN_RIC_CD`; a v3-native routing on `eob_type` is a follow-up). Partially adjudicated exports end to end (the client supports `_typeFilter`; the sandbox Large entity is adjudicated-only). Databricks/Iceberg landing of the new columns.

## Results (2026-10-25, Large Model Entity sandbox)

Pulled as three single-type jobs: the EOB job (89596, 6 min server-side, 192 files / 2.2 GB) was adopted in 44 s; Patient (11 MB) and Coverage (44 MB) exports took 24 and 32 min, most of it queued as *Pending*. Flatten + load of the whole pull: ~2 min.

| table | rows |
|---|---|
| `bcda.patient` | 9,582 |
| `bcda.coverage` | 19,396 (Part A 5,643 · Part B 5,649 · Part C 3,236 · Dual 4,868) |
| `bcda.explanation_of_benefit` | 44,361 (PDE 14,395 · CARRIER 6,298 · DME 6,242 · SNF 5,813 · HOSPICE 2,942 · INPATIENT 2,910 · OUTPATIENT 2,883 · HHA 2,878) |
| `bcda.eob_item` | 248,528 |

All 20 notebook checks pass: every Patient has a unique current MBI and `bene_id == Patient.id`; every Coverage row has `period.start`, every Part A row `ms_cd`, every Dual row `dual_stus_cd`, no orphan beneficiaries; every EOB has a claim id, a claim control number, System-Type and Final-Action tags, EOB-TYPE and CLM-TYPE-CD; control numbers are unique within a source (v3 deduplication holds); lines carry HCPCS or NDC on 100% and paid/submitted amounts on 94% (Part D lines carry drug-cost amounts instead); no orphan lines. The entity is adjudicated-only (no SharedSystem claims), so the control-number pairing check is exercised but finds zero pairs — partially adjudicated data needs an entity with PAC access and `_typeFilter`.

## Connector pipeline on v3 semantics (`bcda.express.cclf`, 2026-10-25)

The input_layer connector was still written for v2 and had never run end to end (its contracts named CCLF columns the EOB table does not have, and the runner's exact-match check would have refused every stage). Rewritten:

1. **One header stage carries the v3 semantics.** `_stg_claim_header` drops `NotFinalAction` claims, derives `claim_family` from `EOB-TYPE` → `CLM-TYPE-CD` → `pde_id` → the v2 near-line RIC (which v3 emits on only some claims: HHA shows U/W/V/NULL), collapses a claim seen in two sources on its claim control number (NationalClaimsHistory/DDPS over SharedSystem, then newest `last_updated`; rows without a control number are never collapsed), and resolves the MBI. The four family stages are filters on it.
2. **One mapping for medical_claim.** `_medical_claim(kind)` produces the full `aco.table.input_layer.InputLayerMedicalClaim` shape for institutional, carrier and DME lines: admission/discharge from the C4BB admission period, NUBC type of bill (`clm_bill_type_cd`, rebuilt as `0` + the three-character v3 frequency code when the coding is missing), MS-DRG, revenue centre, HCPCS and five modifiers, attending→operating→rendering / rendering / rendering→ordering NPIs, billing Organization NPI and tax number, paid / allowed / charge / coinsurance / deductible from the consolidated v3 Adjudication amounts, paid date from `clm_pd_dt` → NCH weekly process → CMS process date, 12 header diagnoses with POA on the first, first ICD procedure.
3. **Part D** uses the PDE id, the prescribing care-team NPI, the contained pharmacy Organization NPI, NDC quantity, total drug cost as charge and paid-by-patient as copayment; plan-paid falls back line → header (the sandbox emits neither).
4. **Eligibility** reads the Part A row (Part B fallback) for status, entitlement and the enrollment period and the Dual row for dual status; gender accepts CCW codes and the FHIR words the v2 flattener passed through; `enrollment_status` comes from `Coverage.status`.
5. **Contracts inherit.** The header stages subclass `ExplanationOfBenefit`; the outputs subclass the canonical `InputLayer*` models, so the same downstream pipelines run on BCDA output as on CCLF output. `load_bcda(run_pipeline=True)` / `stack load bcda --run-pipeline` and `stack run bcda` land `input_layer.medical_claim`, `pharmacy_claim`, `eligibility`.

Run on the Large-entity pull (94 s including re-flattening): 221,021 medical lines on 28,282 claims (16,468 institutional, 11,814 professional — equal to the FinalAction non-PDE EOB counts by type; 4,111 NotFinalAction excluded), 11,968 pharmacy rows (= FinalAction PDEs), 9,582 eligibility rows (7,290 with a Part A/B row, 4,868 with dual status). Every institutional line has a 4-character type of bill, admission date, paid, charge, deductible and coinsurance; every professional line has paid, rendering NPI and (99 %) place of service.

## BCDA manager (`bcda.ui`, `stack bcda`, 2026-10-25)

A Streamlit app and a CLI group over the same service layer, mirroring the surface of the CMS server (`CMSgov/bcda-app`, vendored): `/_version`, `/_health`, `metadata`, `/auth/token`, `attribution_status`, `$export` on `Group/all` · `Group/runout` · `Patient` with `_type` / `_since` / `_typeFilter`, job history, status, cancel, `/data` download — plus the local side: the Store's job and file history, flatten + load into `bcda.*`, the connector into `input_layer.*`, the replica for browsing, the v3 data dictionary and CCLF crosswalk, and the vendored-upstream manifest.

Decisions:
1. **`bcda.ui.service` has no Streamlit in it.** Every operation takes explicit objects (a `Connection`, a `Client`, a `Store`), so the CLI reuses it verbatim and tests hand in mock transports and temp directories. Views are thin.
2. **Connections are presets, environment or custom.** The four public sandbox entities from bcda.cms.gov are built in (the legacy REACH pair pinned to v2); `BCDA_CLIENT_ID` / `BCDA_CLIENT_SECRET` (+ `BCDA_BASE_URL`, `BCDA_API_VERSION`) win when set; production is a selectable environment.
3. **Long work runs on background threads** (`bcda.ui.runner`): a process-level registry that survives Streamlit reruns, captures the `bcda.*` log lines emitted on each run's own thread (job started, progress, files stored) and flips `state` last so readers never see a finished run with an incomplete log. A `st.fragment(run_every="5s")` panel shows runs on every page that starts them.
4. **Exports default to one job per type** with cancel retries, because that is the only shape that survives the shared sandbox's sweeps; the `_typeFilter` builder composes the System-Type `_tag`, `outcome` and `service-date` subqueries from the transition deck; completed jobs under 24 h are offered for *adopt*, in-progress ones for *cancel*.
5. **Data pages read the read-only replica only**, so the manager never holds the primary's single-writer lock; the load page republishes the replica after `load_bcda(run_pipeline=…)`.
6. **Launch:** `stack bcda ui` (port, headless flags) runs `streamlit run src/bcda/ui/app.py`; Streamlit lives in the `bcda` extra. Containerising it (image + traefik route + dashboard tile) is deliberately left out of this slice.

Tests: service with `httpx.MockTransport`, runner for thread-routed logs and state ordering, every page through `AppTest.from_string` with the service stubbed (buttons, forms, error paths), the app shell through `AppTest.from_file`, and the CLI through `CliRunner` — all at 100 % line coverage.

### Containerised (2026-10-25)

`infra/images/bcda.Dockerfile` (same base and install pattern as the api image; `uv sync --extra cli --extra bcda`, Streamlit settings in `infra/images/bcda/streamlit.toml`, health at `/_stcore/health`), compose service `bcda` on the gateway network with `./data` read-write (exports, aco.duckdb, replica publish) and `./dev` read-only (vendored upstream, crosswalk seeds, vendor script), Traefik route `bcda.<domain>` behind git-sso, a dashboard tile, the `bcda` subdomain in stack.toml (CoreDNS hosts and the SSO cert list follow) and an `[images.bcda]` entry so the deploy workflow builds, scans and pushes it on changes to `src/**`, `pyproject.toml` or the image files. Public sandbox credentials are built in; `BCDA_CLIENT_ID` / `BCDA_CLIENT_SECRET` / `BCDA_BASE_URL` in `.env` override.