bcda: BCDA v3 — client, crosswalk-driven flatteners, vendored upstream, proving notebook (P52) #1140

Open

kert wants to merge 18 commits from feat/bcda-v3 into main

pull from: feat/bcda-v3

merge into: homelab:main

homelab:main

homelab:feat/zot-fetch-z1

homelab:feat/zotero-fetch-connector

homelab:feat/zotero-ama-coding-collection

Conversation 0 Commits 18 Files Changed 252 +35005 -2211

kert commented 2026-10-25 17:09:15 +00:00

Owner

Copy Link

Closes #1139 (milestone P52). Design: docs/superpowers/specs/2026-10-25-bcda-v3-design.md.

What changes

Client (bcda.client) — default API version v3; _typeFilter on start_export/export with type_filter_tag() for the System-Type tag; jobs() sends the bulk-data headers BCDA requires (Prefer: respond-async, Accept: application/fhir+json — the old call 400'd on v2 and v3) and unwraps the Task Bundle; completed_jobs() lists reusable exports; every non-2xx raises BcdaHttpError carrying the OperationOutcome diagnostics (still an httpx.HTTPStatusError); a poll that hits 404 Job has been cancelled raises JobCancelledError; /_version and /_health.

Store — adopt(client, job_url) records and downloads a job this store did not start (completed jobs stay downloadable 24 h; the public sandbox entity is shared, so someone else's EOB job is as good as ours); export(..., cancel_retries=N) restarts after server-side cancellation and records each cancelled transition; type_filter passthrough.

Flatteners — bcda.express.v3, table-driven from the BCDA v3 Data Dictionary (fhirPath → column, sourceColumn for v3-only columns, ccwMapping as the crosswalk onto the CCW-named columns the tables already had). v3 moved every BlueButton code off resources/variables/<name> onto fhir/StructureDefinition/<CODE> / fhir/CodeSystem/<CODE> and consolidated amounts into adjudication/total/item.adjudication, so the v2 suffix match returned NULL for almost everything. Dispatch is per resource (v3.is_v3); v2 payloads, seeds and tests are unchanged. flatten_export now streams multi-part exports (<Type>.N.ndjson) to Parquet with model-derived Arrow schemas — a Large-entity EOB export is 192 files / 2.2 GB.

Tables — v3-native nullable columns on bcda.patient (contact language, source keys), bcda.coverage (one row per plan + period_start/end, buy-in, entitlement, dual, Part C/D contract/PBP/LIS, Rx class ids), bcda.explanation_of_benefit (System-Type/Final-Action tags, EOB-TYPE, claim control + original control number, IDR dates, header totals and PPS/IME/DSH adjudication amounts, Part D codes) and bcda.eob_item (NDC quantity, revenue-centre indicators, ANSI reason/group codes, Part D and DME line amounts). Race is crosswalked from US Core OMB codes into bene_race_cd; sex from us-core-sex into CCW coding.

Pipeline — cclf.stg_beneficiary_demographics prefers the Part A coverage row and joins the v3 Dual row; eligibility reads coalesce(dual_01, dual_stus_cd). load_bcda() gathers the newest export directory per resource type (a v3 pull is three single-type jobs).

Crosswalk as data — dev/scripts/gen_bcda_v3_crosswalk.py flattens the dictionary into dev/seeds/bcda_v3_data_dictionary.csv, bcda_v3_cclf_crosswalk.csv (293 CCLF elements → v3 field + fhirPath) and bcda_v3_cclf_absent.csv (52 CCLF fields v3 does not carry); tests assert the committed seeds match the vendored workbook.

Vendored upstream — dev/vendor/MANIFEST.toml pins CMSgov/bcda-app at 7decdcc (v3 handler + tests, request validation middleware, OperationOutcome writer, v3 Postman collection, gzip fixtures, sandbox config, LICENSE) and bcda.cms.gov's v3 data dictionary + official sample ndjson. dev/scripts/vendor_bcda_app.py --verify|--check|--sync (stdlib only); new weekly bcda-vendor workflow runs --check and files an issue on drift. Flattener tests run on the official samples.

Notebook — notebooks/bcda_v3_sandbox.py: preflight → auth/attribution → pull (adopt a completed job <24 h, else one export per type with cancel retries) → flatten + load + replica publish → proof checks → dictionary field population + CCLF crosswalk. Network/load steps only with BCDA_PULL=1 / BCDA_LOAD=1 or the buttons, so the nightly notebooks run never starts an export.

Sandbox findings (2026-10-25, Large Model Entity)

• Combined Patient+Coverage+EOB exports never survived: sweeps cancelled every in-progress job on the shared entity at 15:42, 15:55 and 16:26 UTC.
• EOB-only export: 6 min, 192 files, 2.2 GB. Patient-only / Coverage-only jobs sat Pending far longer than the EOB job.
• Flattening the 2.2 GB EOB export takes ~1.6 min and yields ~45k claims / ~250k lines.

Tests

bcda package at 100% line coverage; suite-wide gate (99%) green.

Connector pipeline rewritten for v3 (52ebad2)

bcda.express.cclf / bcda.pipe.cclf had never run end to end (its contracts named CCLF columns the EOB table does not have) and routed on the v2 near-line RIC code, which v3 emits on only some claims. Now:

• _stg_claim_header carries the v3 semantics: drops NotFinalAction claims, derives claim_family from EOB-TYPE → CLM-TYPE-CD → pde_id → RIC, collapses a claim seen in two sources on its claim control number (NCH/DDPS over SharedSystem, newest last_updated), resolves the MBI. The four family stages are filters on it.
• One medical_claim mapping for institutional / carrier / DME lines on the canonical InputLayerMedicalClaim contract: C4BB admission period, NUBC type of bill (new clm_bill_type_cd; rebuilt as 0 + the 3-char v3 frequency code when missing), MS-DRG, revenue centre, HCPCS + modifiers, role-specific NPIs and tax numbers, paid/allowed/charge/coinsurance/deductible from the consolidated v3 Adjudication amounts, paid date from clm_pd_dt → NCH weekly → CMS process.
• Part D on PDE id, prescribing NPI, contained pharmacy NPI, NDC quantity, total drug cost as charge, paid-by-patient as copayment. Eligibility from the Part A/B row (status, entitlement, period) and the Dual row.
• Contracts inherit ExplanationOfBenefit and aco.table.input_layer.InputLayer*; load_bcda(run_pipeline=True), stack load bcda --run-pipeline, stack run bcda.

Large-entity pull (94 s incl. re-flatten): 221,021 medical lines on 28,282 claims (= FinalAction non-PDE EOBs by type; 4,111 NotFinalAction excluded), 11,968 pharmacy rows (= FinalAction PDEs), 9,582 eligibility rows. The notebook's new section 6 reconciles input_layer against bcda.*: 16/16 checks pass. Suite: 15,703 passed, 99.01% coverage.

BCDA manager — Streamlit app + stack bcda CLI

src/bcda/ui/ mirrors the CMS server's surface (CMSgov/bcda-app, vendored): preflight (/_version, /_health, metadata), auth + attribution, $export on Group/all · Group/runout · Patient with _type / _since / _typeFilter (System-Type, outcome, service-date builder), job history with adopt (completed < 24 h) and cancel, plus the local side: store jobs/files/export dirs, flatten + load + connector + replica publish, data browsing on the read-only replica (summaries + SQL), the v3 data dictionary and CCLF crosswalk, and the vendored-upstream manifest with an on-demand drift check.

• bcda.ui.service is Streamlit-free and shared by stack bcda preflight|auth|jobs|export|adopt|cancel|store|load|vendor.
• bcda.ui.runner runs long operations on daemon threads (survives reruns, captures each run's bcda.* log lines, state flips last); a st.fragment panel shows runs live.
• Connections: the four public sandbox entities, BCDA_CLIENT_ID/SECRET from the environment, or custom (sandbox/production, v3/v2).
• stack bcda ui launches streamlit run src/bcda/ui/app.py; Streamlit added to the bcda extra. Containerising it is left for a follow-up.
• Tests: service (MockTransport), runner, every page via AppTest (buttons, forms, error paths), app shell, CLI — new modules at 100% coverage. Suite: 15,754 passed, 99.02%.

Containerised

infra/images/bcda.Dockerfile (api pattern; --extra cli --extra bcda; Streamlit config in infra/images/bcda/streamlit.toml; health /_stcore/health), compose service bcda (gateway network, ./data rw, ./dev ro, BCDA_* from .env), Traefik route bcda.<domain> behind git-sso, dashboard tile, bcda subdomain, [images.bcda] → deploy.yml build/scan/push job. Built locally (2.15 GB), healthy, root 200; running on the host.

/metrics

bcda.ui.metrics (perf.meter / #579 convention) publishes the OTel Prometheus registry on bcda:8502/metrics from a new python -m bcda.ui entrypoint (Streamlit only runs the script when a browser session opens, so the server has to start with the process). Instruments: stack_bcda_runs_total{kind,state}, stack_bcda_runs_inflight{kind}, stack_bcda_run_seconds{kind}, stack_bcda_page_views_total{page}, gauges stack_bcda_store_jobs{status}, stack_bcda_store_bytes. Prometheus target added; verified up with live values. polars added to the bcda extra (the views import it; it was dev-only, which crash-looped the first metrics image). Suite: 15,760 passed, 99.02%.

Published at bcda.fhirworx.io

A new subdomain needs two Cloudflare writes beyond stack.toml/Traefik/CoreDNS: the tunnel's remote ingress list (it overrides the wildcard in infra/cloudflared/config.yml; an unlisted host answers 530 at the edge) and a proxied CNAME to the tunnel. bootstrap_sso.py gains bcda in SUBDOMAINS and a --dns-only mode that does just those two writes and names the missing zone scope on failure (ae9e6f5, 926acd8). Both writes applied; bcda.fhirworx.io resolves publicly and the edge returns the SSO redirect.

House style (6c218b5)

src/bcda/ui/theme.toml carries the dashboard tokens (assets/css/dashboard.css) as a Streamlit theme file; python -m bcda.ui passes it as --theme.base, so the container and stack bcda ui match. bcda.ui.brand adds the Google Fonts import, the dashboard masthead at the top of the sidebar, eyebrows above page titles, and the repo favicon. Family names are quoted in the theme file because Streamlit 1.66 emits the body font unquoted and an unquoted Source Serif 4 is invalid CSS (verified by rendering the rolled container headlessly). tests/bcda/test_ui_brand.py pins the theme to dashboard.css.

Incremental _since pulls (a05cc51)

Design: docs/superpowers/specs/2026-10-25-bcda-since-reconciliation-design.md. bcda.sync turns a _since export from "the newest directory wins" into a reconciled store:

• First order — per key (Patient BENE_SK, Coverage id, EOB claim control number, since each claim version gets a new FHIR id): newest lastUpdated wins, then NCH/DDPS over SharedSystem, then later transactionTime, later applied sequence, larger id. NotFinalAction winners stay as tombstones; only a baseline retires rows; eob_item follows its header.
• Second order — per job and table in the ledger: new / replaced / unchanged / stale_ignored, backfill vs late (lastUpdated before _since, split by whether the enrollee was already known — Group/all backfills new enrollees' full history by contract), tombstoned, adjusted (clm_orig_cntl_num → bcda.eob_supersession), retired, orphans, scope drift.
• Third order — invariants before anything is published: unique keys, count conservation, monotone watermarks, item↔header integrity; order-independence and idempotency are tested (apply in either order → byte-identical tables).
• Temporality checks before a _since export (block / warn / ok): format, future, floor (2026-03-26), no_baseline, gap, overlap, endpoint_mix (runout vs current population), backfill_scope, scope_filter, inflight, unapplied, attribution_newer, stale_baseline. next_since = the previous job's transactionTime, as BCDA recommends.
• Store: transaction_time/request on job rows, exports/<key>/manifest.json, resume backfills both; empty deltas still advance the watermark.
• Surfaces: stack bcda sync status|check|apply|pull, stack bcda export --since auto, stack bcda resume; Exports page _since modes with the checks shown; new Sync page; notebook §8.

Verified on the real store: the three baselines reconciled over the legacy tables as all-unchanged with identical loaded counts; a real /Patient EOB delta with a _typeFilter reconciled as 248 unchanged with the drift on record.

Databricks App playbook (8cd4633) and a fallback fix (af9c088)

cloud/databricks/README.md + cloud/databricks/bcda-app/ scaffold: how to run the manager as a Databricks App for a user without catalog rights — admin pre-provisions catalog/schema/volume/warehouse/secrets, the app's service principal gets schema-level grants (attaching a uc_securable grants USE CATALOG/USE SCHEMA itself), store on a UC volume via fsspec, Delta tables through the warehouse, Data page on the user's own token, compute/egress/source-size optimisation, runbook, phases, open questions, sources.

The legacy transactionTime fallback now uses the job's start instant (overlap-safe) rather than its completion (gap-prone) — caught on the real store, where the sandbox stamps transactionTime as midnight UTC.

Upstream review (2e44650)

docs/superpowers/specs/2026-10-25-bcda-upstream-review.md: bcda-app 7decdcc, its last 400 PRs and the docs source read against what we built. The _since contract we implemented is the server's (_lastUpdated=gt<since>, le<transactionTime>; transactionTime = BFD load completion, PR #1483). Eight gaps filed as #1293–#1300; the two that change claim identity are error files / partial jobs advancing the watermark (#1293) and control-number chains, the DME VMS→NCH digit and Part D natural keys (#1294).

Upstream-review gaps closed (2cc1300, closes #1293–#1300)

Partial exports: manifest error files downloaded as OperationOutcome, every URL attempted, one DownloadError; a partial job reconciles but never becomes the watermark (partial_job check, Store page issues, stack bcda errors). Claim identity per CMS's dedupe guidance: clm_cntl_num_match (VMS→NCH digit), clm_natural_key (Part D), superseded_by (the related chain, walked; new invariant), prvdr_srvc_id_qlfyr_cd; bcda.sync and the connector key claims the same way. Attribution: attribution_expired (block), attribution_floor (warn), attribution_update recorded per job, server 404/400/429 texts explained. completed_jobs() honours the 24 h retention. build_type_filter follows bcda/fhir/param.go (outcome lists, service-date bounds and prefixes). runout_window check and population labels. Vendor manifest pins the behaviour-defining upstream files, the docs source repo and the PAC/unmapped dictionaries. Design record quotes the server's (since, transactionTime] window and the silent exclusions.

Real store re-reconciled with --force: every row unchanged, loaded counts identical.

Provider identity: CCN ↔ NPI ↔ TIN (63e9c00)

Institutional claims carry no TIN in BCDA v3 or CCLF, and CMS publishes no provider TIN on data.cms.gov. The connector now gives professional/DME lines their carrier TIN as billing_tin; stack load cms vendors the institutional universe (PECOS enrollments ×6, Provider of Services, cost reports) into cms.* and derives cms.ccn_crosswalk (65,161 CCN↔NPI rows), cms.provider_tin_observed and cms.ccn_tin (participant-list hook cms.aco_participant_tin), with a coverage line and a Data page query.

Closes #1139 (milestone P52). Design: `docs/superpowers/specs/2026-10-25-bcda-v3-design.md`. ## What changes **Client** (`bcda.client`) — default API version `v3`; `_typeFilter` on `start_export`/`export` with `type_filter_tag()` for the System-Type tag; `jobs()` sends the bulk-data headers BCDA requires (`Prefer: respond-async`, `Accept: application/fhir+json` — the old call 400'd on v2 and v3) and unwraps the Task Bundle; `completed_jobs()` lists reusable exports; every non-2xx raises `BcdaHttpError` carrying the OperationOutcome diagnostics (still an `httpx.HTTPStatusError`); a poll that hits `404 Job has been cancelled` raises `JobCancelledError`; `/_version` and `/_health`. **Store** — `adopt(client, job_url)` records and downloads a job this store did not start (completed jobs stay downloadable 24 h; the public sandbox entity is shared, so someone else's EOB job is as good as ours); `export(..., cancel_retries=N)` restarts after server-side cancellation and records each `cancelled` transition; `type_filter` passthrough. **Flatteners** — `bcda.express.v3`, table-driven from the BCDA v3 Data Dictionary (`fhirPath` → column, `sourceColumn` for v3-only columns, `ccwMapping` as the crosswalk onto the CCW-named columns the tables already had). v3 moved every BlueButton code off `resources/variables/<name>` onto `fhir/StructureDefinition/<CODE>` / `fhir/CodeSystem/<CODE>` and consolidated amounts into `adjudication`/`total`/`item.adjudication`, so the v2 suffix match returned NULL for almost everything. Dispatch is per resource (`v3.is_v3`); v2 payloads, seeds and tests are unchanged. `flatten_export` now streams multi-part exports (`<Type>.N.ndjson`) to Parquet with model-derived Arrow schemas — a Large-entity EOB export is 192 files / 2.2 GB. **Tables** — v3-native nullable columns on `bcda.patient` (contact language, source keys), `bcda.coverage` (one row per plan + `period_start/end`, buy-in, entitlement, dual, Part C/D contract/PBP/LIS, Rx class ids), `bcda.explanation_of_benefit` (System-Type/Final-Action tags, EOB-TYPE, claim control + original control number, IDR dates, header totals and PPS/IME/DSH adjudication amounts, Part D codes) and `bcda.eob_item` (NDC quantity, revenue-centre indicators, ANSI reason/group codes, Part D and DME line amounts). Race is crosswalked from US Core OMB codes into `bene_race_cd`; sex from `us-core-sex` into CCW coding. **Pipeline** — `cclf.stg_beneficiary_demographics` prefers the Part A coverage row and joins the v3 Dual row; `eligibility` reads `coalesce(dual_01, dual_stus_cd)`. `load_bcda()` gathers the newest export directory per resource type (a v3 pull is three single-type jobs). **Crosswalk as data** — `dev/scripts/gen_bcda_v3_crosswalk.py` flattens the dictionary into `dev/seeds/bcda_v3_data_dictionary.csv`, `bcda_v3_cclf_crosswalk.csv` (293 CCLF elements → v3 field + fhirPath) and `bcda_v3_cclf_absent.csv` (52 CCLF fields v3 does not carry); tests assert the committed seeds match the vendored workbook. **Vendored upstream** — `dev/vendor/MANIFEST.toml` pins CMSgov/bcda-app at `7decdcc` (v3 handler + tests, request validation middleware, OperationOutcome writer, v3 Postman collection, gzip fixtures, sandbox config, LICENSE) and bcda.cms.gov's v3 data dictionary + official sample ndjson. `dev/scripts/vendor_bcda_app.py --verify|--check|--sync` (stdlib only); new weekly `bcda-vendor` workflow runs `--check` and files an issue on drift. Flattener tests run on the official samples. **Notebook** — `notebooks/bcda_v3_sandbox.py`: preflight → auth/attribution → pull (adopt a completed job <24 h, else one export per type with cancel retries) → flatten + load + replica publish → proof checks → dictionary field population + CCLF crosswalk. Network/load steps only with `BCDA_PULL=1` / `BCDA_LOAD=1` or the buttons, so the nightly notebooks run never starts an export. ## Sandbox findings (2026-10-25, Large Model Entity) - Combined Patient+Coverage+EOB exports never survived: sweeps cancelled every in-progress job on the shared entity at 15:42, 15:55 and 16:26 UTC. - EOB-only export: 6 min, 192 files, 2.2 GB. Patient-only / Coverage-only jobs sat *Pending* far longer than the EOB job. - Flattening the 2.2 GB EOB export takes ~1.6 min and yields ~45k claims / ~250k lines. ## Tests `bcda` package at 100% line coverage; suite-wide gate (99%) green. ## Connector pipeline rewritten for v3 (52ebad2) `bcda.express.cclf` / `bcda.pipe.cclf` had never run end to end (its contracts named CCLF columns the EOB table does not have) and routed on the v2 near-line RIC code, which v3 emits on only some claims. Now: - **`_stg_claim_header`** carries the v3 semantics: drops `NotFinalAction` claims, derives `claim_family` from `EOB-TYPE` → `CLM-TYPE-CD` → `pde_id` → RIC, collapses a claim seen in two sources on its claim control number (NCH/DDPS over SharedSystem, newest `last_updated`), resolves the MBI. The four family stages are filters on it. - **One `medical_claim` mapping** for institutional / carrier / DME lines on the canonical `InputLayerMedicalClaim` contract: C4BB admission period, NUBC type of bill (new `clm_bill_type_cd`; rebuilt as `0` + the 3-char v3 frequency code when missing), MS-DRG, revenue centre, HCPCS + modifiers, role-specific NPIs and tax numbers, paid/allowed/charge/coinsurance/deductible from the consolidated v3 Adjudication amounts, paid date from `clm_pd_dt` → NCH weekly → CMS process. - **Part D** on PDE id, prescribing NPI, contained pharmacy NPI, NDC quantity, total drug cost as charge, paid-by-patient as copayment. **Eligibility** from the Part A/B row (status, entitlement, period) and the Dual row. - Contracts inherit `ExplanationOfBenefit` and `aco.table.input_layer.InputLayer*`; `load_bcda(run_pipeline=True)`, `stack load bcda --run-pipeline`, `stack run bcda`. Large-entity pull (94 s incl. re-flatten): 221,021 medical lines on 28,282 claims (= FinalAction non-PDE EOBs by type; 4,111 NotFinalAction excluded), 11,968 pharmacy rows (= FinalAction PDEs), 9,582 eligibility rows. The notebook's new section 6 reconciles input_layer against `bcda.*`: 16/16 checks pass. Suite: 15,703 passed, 99.01% coverage. ## BCDA manager — Streamlit app + `stack bcda` CLI `src/bcda/ui/` mirrors the CMS server's surface (CMSgov/bcda-app, vendored): preflight (`/_version`, `/_health`, metadata), auth + attribution, `$export` on Group/all · Group/runout · Patient with `_type` / `_since` / `_typeFilter` (System-Type, outcome, service-date builder), job history with adopt (completed < 24 h) and cancel, plus the local side: store jobs/files/export dirs, flatten + load + connector + replica publish, data browsing on the read-only replica (summaries + SQL), the v3 data dictionary and CCLF crosswalk, and the vendored-upstream manifest with an on-demand drift check. - `bcda.ui.service` is Streamlit-free and shared by `stack bcda preflight|auth|jobs|export|adopt|cancel|store|load|vendor`. - `bcda.ui.runner` runs long operations on daemon threads (survives reruns, captures each run's `bcda.*` log lines, state flips last); a `st.fragment` panel shows runs live. - Connections: the four public sandbox entities, `BCDA_CLIENT_ID`/`SECRET` from the environment, or custom (sandbox/production, v3/v2). - `stack bcda ui` launches `streamlit run src/bcda/ui/app.py`; Streamlit added to the `bcda` extra. Containerising it is left for a follow-up. - Tests: service (MockTransport), runner, every page via `AppTest` (buttons, forms, error paths), app shell, CLI — new modules at 100% coverage. Suite: 15,754 passed, 99.02%. ## Containerised `infra/images/bcda.Dockerfile` (api pattern; `--extra cli --extra bcda`; Streamlit config in `infra/images/bcda/streamlit.toml`; health `/_stcore/health`), compose service `bcda` (gateway network, `./data` rw, `./dev` ro, `BCDA_*` from `.env`), Traefik route `bcda.<domain>` behind git-sso, dashboard tile, `bcda` subdomain, `[images.bcda]` → deploy.yml build/scan/push job. Built locally (2.15 GB), healthy, root 200; running on the host. ## `/metrics` `bcda.ui.metrics` (perf.meter / #579 convention) publishes the OTel Prometheus registry on `bcda:8502/metrics` from a new `python -m bcda.ui` entrypoint (Streamlit only runs the script when a browser session opens, so the server has to start with the process). Instruments: `stack_bcda_runs_total{kind,state}`, `stack_bcda_runs_inflight{kind}`, `stack_bcda_run_seconds{kind}`, `stack_bcda_page_views_total{page}`, gauges `stack_bcda_store_jobs{status}`, `stack_bcda_store_bytes`. Prometheus target added; verified up with live values. `polars` added to the `bcda` extra (the views import it; it was dev-only, which crash-looped the first metrics image). Suite: 15,760 passed, 99.02%. ## Published at bcda.fhirworx.io A new subdomain needs two Cloudflare writes beyond stack.toml/Traefik/CoreDNS: the tunnel's remote ingress list (it overrides the wildcard in `infra/cloudflared/config.yml`; an unlisted host answers 530 at the edge) and a proxied CNAME to the tunnel. `bootstrap_sso.py` gains `bcda` in `SUBDOMAINS` and a `--dns-only` mode that does just those two writes and names the missing zone scope on failure (`ae9e6f5`, `926acd8`). Both writes applied; `bcda.fhirworx.io` resolves publicly and the edge returns the SSO redirect. ## House style (`6c218b5`) `src/bcda/ui/theme.toml` carries the dashboard tokens (`assets/css/dashboard.css`) as a Streamlit theme file; `python -m bcda.ui` passes it as `--theme.base`, so the container and `stack bcda ui` match. `bcda.ui.brand` adds the Google Fonts import, the dashboard masthead at the top of the sidebar, eyebrows above page titles, and the repo favicon. Family names are quoted in the theme file because Streamlit 1.66 emits the body font unquoted and an unquoted `Source Serif 4` is invalid CSS (verified by rendering the rolled container headlessly). `tests/bcda/test_ui_brand.py` pins the theme to `dashboard.css`. ## Incremental `_since` pulls (`a05cc51`) Design: `docs/superpowers/specs/2026-10-25-bcda-since-reconciliation-design.md`. `bcda.sync` turns a `_since` export from "the newest directory wins" into a reconciled store: - **First order** — per key (Patient `BENE_SK`, Coverage id, EOB claim control number, since each claim *version* gets a new FHIR id): newest `lastUpdated` wins, then NCH/DDPS over SharedSystem, then later `transactionTime`, later applied sequence, larger id. `NotFinalAction` winners stay as tombstones; only a baseline retires rows; `eob_item` follows its header. - **Second order** — per job and table in the ledger: new / replaced / unchanged / stale_ignored, backfill vs late (lastUpdated before `_since`, split by whether the enrollee was already known — Group/all backfills new enrollees' full history by contract), tombstoned, adjusted (`clm_orig_cntl_num` → `bcda.eob_supersession`), retired, orphans, scope drift. - **Third order** — invariants before anything is published: unique keys, count conservation, monotone watermarks, item↔header integrity; order-independence and idempotency are tested (apply in either order → byte-identical tables). - **Temporality checks** before a `_since` export (block / warn / ok): format, future, floor (2026-03-26), no_baseline, gap, overlap, endpoint_mix (runout vs current population), backfill_scope, scope_filter, inflight, unapplied, attribution_newer, stale_baseline. `next_since` = the previous job's `transactionTime`, as BCDA recommends. - Store: `transaction_time`/`request` on job rows, `exports/<key>/manifest.json`, `resume` backfills both; empty deltas still advance the watermark. - Surfaces: `stack bcda sync status|check|apply|pull`, `stack bcda export --since auto`, `stack bcda resume`; Exports page `_since` modes with the checks shown; new **Sync** page; notebook §8. Verified on the real store: the three baselines reconciled over the legacy tables as all-unchanged with identical loaded counts; a real `/Patient` EOB delta with a `_typeFilter` reconciled as 248 unchanged with the drift on record. ## Databricks App playbook (`8cd4633`) and a fallback fix (`af9c088`) `cloud/databricks/README.md` + `cloud/databricks/bcda-app/` scaffold: how to run the manager as a Databricks App for a user without catalog rights — admin pre-provisions catalog/schema/volume/warehouse/secrets, the app's service principal gets schema-level grants (attaching a `uc_securable` grants `USE CATALOG`/`USE SCHEMA` itself), store on a UC volume via fsspec, Delta tables through the warehouse, Data page on the user's own token, compute/egress/source-size optimisation, runbook, phases, open questions, sources. The legacy transactionTime fallback now uses the job's start instant (overlap-safe) rather than its completion (gap-prone) — caught on the real store, where the sandbox stamps `transactionTime` as midnight UTC. ## Upstream review (`2e44650`) `docs/superpowers/specs/2026-10-25-bcda-upstream-review.md`: bcda-app `7decdcc`, its last 400 PRs and the docs source read against what we built. The `_since` contract we implemented is the server's (`_lastUpdated=gt<since>`, `le<transactionTime>`; `transactionTime` = BFD load completion, PR #1483). Eight gaps filed as #1293–#1300; the two that change claim identity are error files / partial jobs advancing the watermark (#1293) and control-number chains, the DME VMS→NCH digit and Part D natural keys (#1294). ## Upstream-review gaps closed (`2cc1300`, closes #1293–#1300) Partial exports: manifest `error` files downloaded as OperationOutcome, every URL attempted, one `DownloadError`; a partial job reconciles but never becomes the watermark (`partial_job` check, Store page issues, `stack bcda errors`). Claim identity per CMS's dedupe guidance: `clm_cntl_num_match` (VMS→NCH digit), `clm_natural_key` (Part D), `superseded_by` (the `related` chain, walked; new invariant), `prvdr_srvc_id_qlfyr_cd`; `bcda.sync` and the connector key claims the same way. Attribution: `attribution_expired` (block), `attribution_floor` (warn), `attribution_update` recorded per job, server 404/400/429 texts explained. `completed_jobs()` honours the 24 h retention. `build_type_filter` follows `bcda/fhir/param.go` (outcome lists, service-date bounds and prefixes). `runout_window` check and population labels. Vendor manifest pins the behaviour-defining upstream files, the docs source repo and the PAC/unmapped dictionaries. Design record quotes the server's `(since, transactionTime]` window and the silent exclusions. Real store re-reconciled with `--force`: every row `unchanged`, loaded counts identical. ## Provider identity: CCN ↔ NPI ↔ TIN (`63e9c00`) Institutional claims carry no TIN in BCDA v3 or CCLF, and CMS publishes no provider TIN on data.cms.gov. The connector now gives professional/DME lines their carrier TIN as `billing_tin`; `stack load cms` vendors the institutional universe (PECOS enrollments ×6, Provider of Services, cost reports) into `cms.*` and derives `cms.ccn_crosswalk` (65,161 CCN↔NPI rows), `cms.provider_tin_observed` and `cms.ccn_tin` (participant-list hook `cms.aco_participant_tin`), with a coverage line and a Data page query.

kert added this to the P52: BCDA v3 migration — client, crosswalk-driven flatteners, vendored upstream fixtures, proving notebook milestone 2026-10-25 17:09:15 +00:00

kert added the datanotebookspipeline labels 2026-10-25 17:09:15 +00:00

kert added 4 commits 2026-10-25 17:09:16 +00:00

fix(tests): deploy-range git helper ignores the hook's GIT_DIR/GIT_INDEX_FILE ... 48753e9b80

The pre-commit hook runs the dev tests with GIT_DIR and GIT_INDEX_FILE set
to the real repo; the temp-repo helper inherited them, so every commit that
touched dev/ failed at 'git checkout -b feature' (exit 128) while the same
test passed from a plain shell.

chore(bcda): vendor CMSgov/bcda-app + bcda.cms.gov v3 artefacts sparsely, flatten the v3 data dictionary into crosswalk seeds, weekly drift workflow ... 7e53d9360a

dev/vendor/MANIFEST.toml pins bcda-app@7decdcc (v3 handler + tests, request
validation middleware, OperationOutcome writer, v3 Postman collection, gzip
fixtures, sandbox config, LICENSE) and the bcda.cms.gov v3 data dictionary,
v2 dictionary and official v3 sample ndjson. vendor_bcda_app.py --verify |
--check | --sync (stdlib only); the bcda-vendor workflow runs --check weekly
and files an issue on drift — nothing auto-commits, a payload change upstream
is something a human reads before the flattener is re-pointed.

gen_bcda_v3_crosswalk.py writes dev/seeds/bcda_v3_data_dictionary.csv
(fhirPath, IDR sourceColumn, ccwMapping, cclfMapping per v3 field),
bcda_v3_cclf_crosswalk.csv (293 CCLF elements -> v3 field) and
bcda_v3_cclf_absent.csv (52 CCLF fields v3 does not carry); the CCLF sheet
has label/name swapped on data rows and is normalised. Tests assert the
committed seeds match the vendored workbook.

refs #1139

feat(bcda): BCDA v3 — client, store adoption and cancel retries, dictionary-driven v3 flatteners, streaming multi-part load ... d08401dc53

Client defaults to /api/v3; _typeFilter on start_export/export with
type_filter_tag() for the System-Type tag; jobs() sends the bulk-data headers
BCDA requires (Prefer: respond-async, Accept: application/fhir+json — the old
call 400'd on v2 and v3) and unwraps the Task Bundle; completed_jobs() lists
reusable exports; every non-2xx raises BcdaHttpError carrying the
OperationOutcome diagnostics (still an httpx.HTTPStatusError); a poll that
hits 404 'Job has been cancelled' raises JobCancelledError; /_version and
/_health.

Store.adopt(client, job_url) records and downloads a job this store did not
start (completed jobs stay downloadable 24 h and the public sandbox entity is
shared); export(cancel_retries=N) restarts after server-side cancellation,
recording each 'cancelled' transition.

bcda.express.v3 flattens C4BB 2.2.0 resources into the existing bcda.* tables,
table-driven from the v3 Data Dictionary: v3 moved every BlueButton code off
resources/variables/<name> onto fhir/StructureDefinition|CodeSystem/<CODE> and
consolidated amounts into adjudication/total/item.adjudication, so the v2
suffix match returned NULL for almost everything. Dispatch is per resource
(is_v3); v2 payloads are unchanged. Identity follows the migration guide:
bene_id = Patient.id (BENE_SK), current MBI = us-mbi identifier without
period.end, clm_id = C4BB 'uc', clm_cntl_num pairs SharedSystem with NCH.
Race crosswalked from US Core OMB codes; Coverage is one row per plan+period
(Part A/B/C/D + Dual) with v3-native columns; EOB gains System-Type /
Final-Action / EOB-TYPE, IDR dates, header totals and PPS/IME/DSH amounts;
items gain NDC quantity, revenue-centre indicators, ANSI codes, Part D/DME
amounts.

flatten_export streams <Type>.N.ndjson parts to Parquet with model-derived
Arrow schemas (a Large-entity EOB export is 192 files / 2.2 GB); load_bcda()
gathers the newest export dir per resource type (a v3 pull is three
single-type jobs). cclf.stg_beneficiary_demographics prefers the Part A row
and joins the Dual row; eligibility reads coalesce(dual_01, dual_stus_cd).

bcda package at 100% line coverage; tests run on CMS's official v3 samples.

refs #1139

docs(bcda): v3 design record and the sandbox proving notebook ... a770c2adb9

notebooks/bcda_v3_sandbox.py walks preflight -> auth/attribution -> pull (adopt
a completed job <24 h, else one export per type with cancel retries) ->
flatten + load + replica publish -> 20 proof checks -> dictionary field
population + CCLF crosswalk. Network/load steps only with BCDA_PULL=1 /
BCDA_LOAD=1 or the buttons, so the nightly notebooks run never starts an
export. Proof run against the Large Model Entity: 9,582 patients, 19,396
coverage rows, 44,361 EOBs, 248,528 lines; all checks pass.

refs #1139

kert added 1 commit 2026-10-25 17:43:10 +00:00

feat(bcda): rewrite the input_layer connector for v3 semantics ... 52ebad29cd

The connector (bcda.express.cclf / bcda.pipe.cclf) still routed on the v2
near-line RIC code, read v2-only amount columns, and had never run end to
end: its contracts named CCLF columns the EOB table does not have, so the
runner's exact-match check would have refused every stage.

One header stage now carries the v3 semantics. _stg_claim_header drops
NotFinalAction claims (9% of the sandbox), derives claim_family from
EOB-TYPE → CLM-TYPE-CD → pde_id → RIC (v3 emits the RIC on only some
claims), collapses a claim seen in two sources on its claim control number
(NationalClaimsHistory/DDPS over SharedSystem, then newest last_updated;
rows without a control number are never collapsed) and resolves the MBI.
The four family stages are filters on it.

One mapping produces input_layer.medical_claim for institutional, carrier
and DME lines: admission/discharge from the C4BB admission period, NUBC
type of bill (new clm_bill_type_cd on the EOB table; rebuilt as "0" + the
three-character v3 frequency code when the coding is missing — the v2
fac+clsfctn+freq concatenation yields "88787J" on v3), MS-DRG, revenue
centre, HCPCS + modifiers, attending→operating→rendering / rendering /
rendering→ordering NPIs, billing Organization NPI and tax number,
paid/allowed/charge/coinsurance/deductible from the consolidated v3
Adjudication amounts, paid date from clm_pd_dt → NCH weekly → CMS process.
Part D uses the PDE id, prescribing care-team NPI, contained pharmacy NPI,
NDC quantity, total drug cost as charge and paid-by-patient as copayment.
Eligibility reads the Part A row (Part B fallback) for status, entitlement
and enrollment, the Dual row for dual status, and accepts both CCW sex
codes and the FHIR words the v2 flattener passed through.

Contracts inherit: header stages subclass ExplanationOfBenefit, outputs
subclass aco.table.input_layer.InputLayer*, so downstream pipelines run on
BCDA output exactly as on CCLF output. load_bcda(run_pipeline=True),
stack load bcda --run-pipeline and stack run bcda land input_layer.*.

Large-entity sandbox pull: 221,021 medical lines on 28,282 claims (equal to
the FinalAction non-PDE EOBs by type), 11,968 pharmacy rows, 9,582
eligibility rows; all 16 reconciliation checks in the notebook pass. Tests
run the whole connector on CMS's official v3 samples through DuckDB.

refs #1139

kert added 1 commit 2026-10-25 18:11:48 +00:00

feat(bcda): the BCDA manager — a Streamlit app and stack bcda CLI over the client, store, loader, connector and crosswalk ... eb1b574c04

bcda.ui mirrors the surface of the CMS server (CMSgov/bcda-app, vendored
under dev/vendor): /_version, /_health, metadata, /auth/token,
attribution_status, $export on Group/all · Group/runout · Patient with
_type / _since / _typeFilter, job history, status, cancel and /data
download — plus the local side: the Store's job and file history,
flatten + load into bcda.*, the connector into input_layer.*, the
replica for browsing, the v3 data dictionary and CCLF crosswalk, and the
vendored-upstream manifest with an on-demand drift check.

bcda.ui.service has no Streamlit in it: every operation takes a
Connection / Client / Store, so `stack bcda preflight|auth|jobs|export|
adopt|cancel|store|load|vendor` reuse it verbatim and tests hand in
httpx.MockTransport clients and temp stores. Connections are the four
public sandbox entities, BCDA_CLIENT_ID/SECRET from the environment, or
custom (sandbox or production, v3 or v2). bcda.ui.runner runs exports,
adoption, loads and drift checks on daemon threads in a process-level
registry that survives Streamlit reruns, captures the bcda.* log lines
of each run's own thread, and flips state last so readers never see a
finished run with an incomplete log; a st.fragment panel shows runs
live. Exports default to one job per type with cancel retries (the only
shape that survives the shared sandbox's sweeps); the _typeFilter
builder composes System-Type, outcome and service-date; completed jobs
under 24 h are offered for adopt, in-progress ones for cancel. Data
pages read the read-only replica only.

`stack bcda ui` launches streamlit on src/bcda/ui/app.py; streamlit is
added to the bcda extra. CLI reference pages regenerated (the generator
renumbers sidebar positions, hence the touched pages).

Tests: service, runner, every page through AppTest (buttons, forms,
error paths), the app shell, and the CLI — new modules at 100% line
coverage.

refs #1139

kert added 1 commit 2026-10-25 18:18:37 +00:00

feat(infra): containerise the BCDA manager — bcda image, compose service, Traefik route, dashboard tile ... 191121b542

infra/images/bcda.Dockerfile follows api.Dockerfile (uv base, patched
packages, curl for the healthcheck) with `uv sync --extra cli --extra
bcda`; Streamlit's server settings live in infra/images/bcda/
streamlit.toml (headless, 0.0.0.0:8501, no usage stats, no file
watcher); health is /_stcore/health. The compose service `bcda` sits on
the gateway network, mounts ./data read-write (exports, aco.duckdb,
replica publish) and ./dev read-only (vendored upstream, crosswalk
seeds, vendor script), and takes BCDA_CLIENT_ID / BCDA_CLIENT_SECRET /
BCDA_BASE_URL from .env with the public sandbox credentials as the
default. Routed at bcda.<domain> behind git-sso with secure-headers,
tiled on the dashboard, and added to [platform].subdomains (CoreDNS
hosts and the SSO cert list follow) and [images] (deploy.yml builds,
scans and pushes it on src/**, pyproject.toml or image-file changes;
harden/rebuild-all/infra-ci regenerated accordingly).

Built locally (2.15 GB), /_stcore/health -> ok, root -> 200; brought up
on the host with compose.

refs #1139

kert added 1 commit 2026-10-25 18:43:48 +00:00

feat(bcda): /metrics for the BCDA manager — OTel instruments on a second port, scraped by Prometheus ... e31282c02d

bcda.ui.metrics follows the #579 convention: perf.meter hands back an
OTel meter + PrometheusMetricReader when STACK_TELEMETRY is on and a
no-op MockMeter otherwise. Streamlit cannot mount a route and only
executes the app script when a browser session opens, so the registry is
published from a second port (BCDA_METRICS_PORT, 8502) by a new
`python -m bcda.ui` entrypoint that starts the metrics server first and
then boots Streamlit's CLI in the same process; the image CMD and
`stack bcda ui` use it. Prometheus scrapes bcda:8502/metrics directly
(infra/prometheus/targets/services.yml) like api and llm.

Instruments: stack_bcda_runs_total{kind,state}, stack_bcda_runs_inflight
{kind}, stack_bcda_run_seconds{kind} (recorded by the runner, which now
labels runs export / adopt / load / vendor_check / other),
stack_bcda_page_views_total{page}, and two observable gauges read at
scrape time, stack_bcda_store_jobs{status} and stack_bcda_store_bytes.
Attribute values are closed sets. The image installs the perf extra;
the compose service joins the observability network with
OTEL_SERVICE_NAME=bcda. polars joins the bcda extra: the views import it,
and it had only been a dev dependency, so the first metrics image
crash-looped on `No module named polars` the moment the entrypoint
imported the app at process start.

Verified in the stack: bcda:8502/metrics serves stack_bcda_* series,
the prometheus target is up and stack_bcda_store_jobs answers a query.

refs #1139

kert added 1 commit 2026-10-25 18:49:52 +00:00

fix(infra): publish bcda.fhirworx.io through the tunnel — add it to the hostname list and give bootstrap_sso a --dns-only mode ... ae9e6f5e67

The tunnel runs on a remote ingress list pushed by bootstrap_sso (it
overrides the wildcard in infra/cloudflared/config.yml), and the public
CNAMEs are created from the same SUBDOMAINS list, so a service that only
exists in stack.toml and Traefik answers 530 at the Cloudflare edge and
NXDOMAIN on public DNS — which is what bcda.fhirworx.io did. Adding a
service now means appending it here and running
`CF_API_TOKEN=… uv run python dev/scripts/bootstrap_sso.py --dns-only`,
which does the two Cloudflare writes without the Gitea admin flow.

refs #1139

kert added 1 commit 2026-10-25 18:57:53 +00:00

fix(infra): bootstrap_sso --dns-only says which zone scope the token is missing ... 926acd8f86

Running it with the corwins.media-scoped token updated the tunnel (account
scope) and then failed the DNS step with a bare 'Authentication error';
the record needs Zone:DNS:Edit on fhirworx.io specifically.

refs #1139

kert added 1 commit 2026-10-25 19:21:47 +00:00

feat(bcda): fhirworx house style for the manager — theme file, brand masthead and eyebrows, favicon ... 6c218b5898

src/bcda/ui/theme.toml is a Streamlit theme file carrying the dashboard
tokens from assets/css/dashboard.css (cream paper, near-black ink, navy
primary and sidebar, warm beige surfaces, 2px radius, Playfair Display /
Source Serif 4 / JetBrains Mono). `python -m bcda.ui` passes it as
--theme.base unless the caller set one, so the container and
`stack bcda ui` look the same without a .streamlit/config.toml per cwd;
the image's streamlit.toml keeps only server settings.

bcda.ui.brand adds what a theme file cannot: the Google Fonts import,
the dashboard masthead (rule, display title, monospace eyebrow) at the
top of the sidebar, and brand.title() eyebrows above every page title.
app.py renders its own page links after the masthead (navigation
position="hidden") so the masthead comes first as on the dashboard. The
favicon is the repo's assets/icons/fav32.png, baked into the image.

Family names in the theme file stay double-quoted: Streamlit 1.66 emits
the body font unquoted, and an unquoted family with a digit ("Source
Serif 4") is an invalid font-family, so the browser silently fell back
to its default serif while the headings (quoted on their path) were
fine. Found by rendering the rolled container headlessly on the compose
network and reading computed styles.

tests/bcda/test_ui_brand.py pins the theme's hexes and font stacks to
dashboard.css and renders inject/masthead/title through AppTest;
test_ui_main covers the --theme.base handling.

refs #1139

kert added 1 commit 2026-10-25 20:05:27 +00:00

feat(bcda): incremental _since pulls — watermarks, temporality checks, reconciliation with tiebreakers, ledger and invariants ... a05cc51916

Design: docs/superpowers/specs/2026-10-25-bcda-since-reconciliation-design.md.

Until now a _since export was loaded like a full one: the loader took the
newest export directory per type, so a delta silently replaced the baseline
it was meant to extend. bcda.sync makes incremental pulls safe end to end.

First order — which row is current. A baseline (no _since) is the
authoritative snapshot for its types; deltas of the same population at or
after its cut-off extend it. Keys: Patient BENE_SK, Coverage id, EOB claim
control number (each claim *version* has a new FHIR id — migration guide).
Winner per key: newest meta.lastUpdated, then NCH/DDPS over SharedSystem
(the connector's rule), then later transactionTime, later applied sequence,
larger id. NotFinalAction winners stay as tombstones; only a baseline
retires rows (the API cannot express deletion). eob_item follows its header.

Second order — what a job did, per table, in the ledger: new / replaced /
unchanged / stale_ignored, backfill (lastUpdated < _since for an enrollee we
did not have — the API's promise for newly attributed enrollees on
Group/all) vs late (same shape for one we had — should not happen),
tombstoned, adjusted (clm_orig_cntl_num → bcda.eob_supersession), retired,
orphans (claims/coverage without a current Patient), scope drift.

Third order — invariants on every apply before anything is published:
unique keys, count conservation (|cur'| = |cur| + new − retired), watermark
monotone, every item has a header; replay determinism and idempotency are
tested (apply in either order → byte-identical tables; re-apply → no-op).
Current tables are re-derived from the applied partitions each time, so the
state is a function of the set of applied jobs, not their arrival order.

Temporality checks before a _since export starts (block/warn/ok): format,
future (server 400s), floor (no v3 lastUpdated before 2026-03-26),
no_baseline, gap (since > watermark: a window nobody would ever fetch),
overlap (free), endpoint_mix (runout vs current population),
backfill_scope (/Patient delta over a Group/all baseline), scope_filter
(_typeFilter drift), inflight, unapplied, attribution_newer,
stale_baseline. next_since() is the previous job's transactionTime, as
BCDA recommends; a combined job takes the earliest watermark of its types.

Store: job rows carry transaction_time and request, exports/<key>/
manifest.json keeps the completed-job body, resume() fills both in for
jobs completed before this; adopted jobs read _since from their request
line; a delta with nothing to deliver (200, empty output) still counts and
advances the watermark. flatten_export(out_dir=) writes per-job partitions
under flat/history/<key>/; current tables keep _job_key/_transaction_time/
_since/_kind/_applied_seq and the connector's _ensure() drops them before
the stage contracts. load_bcda(reconcile=True) applies pending jobs and
also loads bcda.sync_ledger, bcda.sync_watermark, bcda.eob_supersession;
an explicit ndjson_dir or a store without job rows keeps the legacy path.

Surfaces: `stack bcda sync status|check|apply|pull`, `stack bcda export
--since auto [--force]`, `stack bcda resume`; the manager's Exports page
gets a _since mode (off / watermark / date) with the checks shown and
blocks respected, and a Sync page (watermarks, pending, reconcile, ledger);
notebook section 8.

Verified on the real Large-entity store: applying the three baselines over
the legacy flat/ tables reported every row unchanged (0 new, 0 retired)
and loaded counts identical to the full load (221,021 medical lines,
11,968 pharmacy, 9,582 eligibility); a real /Patient-endpoint EOB delta
with a _typeFilter reconciled as 248 unchanged with the drift on record;
the sandbox's transactionTime is midnight UTC of the run day, so a
same-day zero-row delta is correctly ignored as predating its baseline.

refs #1139

kert added 2 commits 2026-10-25 20:21:40 +00:00

docs(bcda): playbook for running the manager as a Databricks App without catalog rights ... 8cd4633dac

cloud/databricks/README.md: what the platform guarantees (ephemeral
container, Python 3.11 default → uv pin 3.12, app.yaml command/env with
valueFrom, injected service-principal credentials, user authorization via
X-Forwarded-Access-Token and opt-in scopes, Medium/Large compute, 15 s
SIGTERM, 10 MB per source file, restricted egress policies), the
permission split for a user who cannot create or manage catalogs (admin
provisions catalog/schema/volume/warehouse/secrets once; attaching a
uc_securable grants USE CATALOG/USE SCHEMA automatically; only
CREATE TABLE/MODIFY/SELECT on the schema need a schema manager), the
compose → Databricks mapping (store on a UC volume via fsspec, Delta
tables through the SQL warehouse, no DuckDB file, no second port), the
eight small code changes the port needs, optimisation notes (app as
control plane, size per phase, cache statics, one connection per
session, never pull a table into the app, sync only what the app
imports), a runbook, phases 1–4 and the open questions to verify in the
target workspace. Sources listed.

cloud/databricks/bcda-app/: a bundle of its own (the main bundle creates
schemas and a volume, which needs catalog rights) with the app resource,
its warehouse/secret/volume resources and permissions, app.yaml,
.python-version and a requirements stub.

refs #1139

fix(bcda): legacy cut-off fallback is the job's start, not its completion ... af9c088b84

A job recorded before manifests existed has no transactionTime. The
first version fell back to the instant the `complete` row was written
and called it "safe, slightly overlapping" — I had that backwards: a
_since later than the server's true cut-off is a gap, not an overlap.
The real store showed it: the sandbox stamps transactionTime as midnight
UTC of the run day, so a baseline that finished at 16:53 was being
credited with a 16:53 cut-off and every same-day delta (00:00) was read
as predating it. The fallback is now the job's first ledger row (the
request instant), which can only overlap; Store.job_started_at() reads
it. `stack bcda resume <key>` still replaces the fallback with the
server's value — done for the four pre-manifest jobs in the real store,
whose watermarks now read 2026-10-25T00:00:00Z.

refs #1139

kert added 1 commit 2026-10-25 20:36:16 +00:00

docs(bcda): upstream review — bcda-app 7decdcc, 400 PRs and the docs source against what we built ... 2e4465084e

Eight gaps, two of which change what 'current' means for a claim:
error files dropped (a partial job still advances the watermark), and
claim identity across control-number changes, DME VMS→NCH digit and
Part D natural keys. Also: attribution expiration/January in the checks,
adopting expired or cancellable jobs, the narrower _typeFilter surface,
runout calendar, vendor manifest coverage, and our own docs quoting the
server's (since, transactionTime] window. Each gap has a tracker issue.

refs #1139

kert added 1 commit 2026-10-25 20:36:41 +00:00

docs(bcda): link the upstream review's gaps to their tracker issues (#1293–#1300) ... 3dc44d0650

refs #1139

kert added 1 commit 2026-10-25 21:11:20 +00:00

feat(bcda): close the upstream-review gaps — partial exports, claim identity across systems, attribution checks, filter grammar, vendored behaviour ... 2cc13008a9

Closes #1293 #1294 #1295 #1296 #1297 #1298 #1299 #1300 (upstream review
docs/superpowers/specs/2026-10-25-bcda-upstream-review.md).

Partial exports (#1293). The manifest's `error` list is downloaded as
OperationOutcome[.N].ndjson with resource_type=OperationOutcome; every
URL is attempted and failures are recorded per file, then raised once as
DownloadError. A job with error files is `partial`: it still reconciles
(its rows are real) but never becomes the watermark, check_since warns
`partial_job`, the Store page lists the issues and `stack bcda errors`
prints them.

Claim identity (#1294), per bcda.cms.gov "Matching multiple EOBs to the
same claim": three new header columns — clm_cntl_num_match (VMS leading 1
stripped on SharedSystem DME), clm_natural_key (Part D events without a
control number: bene · line from-date · provider qualifier · service
provider · RX number · fill number) and superseded_by — plus
prvdr_srvc_id_qlfyr_cd. bcda.sync keys claims on match → control number
→ natural key → id, marks every claim another current claim names in
`related` as superseded (chain walked), counts `superseded`, and a new
invariant forbids a live claim being named predecessor by another live
claim. The connector dedupes on the same identity and drops superseded
rows. Partitions and current tables flattened before these columns
existed get them as NULLs.

Attribution (#1295): `attribution_expired` (block) from
expiration_dates, `attribution_floor` (warn) from the earliest
last_attribution_update recorded on job rows (new column), and the
server's 404/400/429 texts explained by explain_server_error.

Adoption (#1296): completed_jobs() drops jobs older than the 24 h file
retention; downloads no longer stop at the first failed file.

_typeFilter (#1297): build_type_filter follows bcda/fhir/param.go —
outcome lists, service-date earliest and latest bounds with eq/gt/ge/lt/le
on YYYY[-MM[-DD]], no comma lists, no conflicting prefixes; CLI and
Exports page expose them.

Runout (#1298): `runout_window` warn after July; population column on
the Sync page ledger.

Vendoring (#1299): the manifest now pins bcda-app's param.go,
bluebutton.go, service.go, constants, models and prod.yml, the docs
source (CMSgov/bcda-static-site: filter/access/migrate/PAC/FAQ pages and
announcements) and the PAC and unmapped-fields dictionaries.

Docs (#1300): the reconciliation design record states the server's
window `(since, transactionTime]`, the lookback bound, the silent
exclusions, re-loads, partial exports and the identity rules.

refs #1139

kert added 1 commit 2026-10-25 21:34:18 +00:00

feat(cms): vendor the institutional provider universe (CCN ↔ NPI ↔ TIN) from data.cms.gov; give carrier lines their billed TIN ... 63e9c0029a

"No TIN available" is a fact about the data, not a bug in ours.
Institutional claims never carry a TIN: the real Large-entity v3 pull has
0 billing TINs on every claim type and 0 CCNs on every institutional
claim (the dictionary's NamingSystem/CCN identifier is not emitted and
half the institutional claims have no billing provider at all), and
CCLF1 has an OSCAR/CCN and NPIs but no TIN either. The only TIN on
claims is the carrier/DME CLM_RNDRG_PRVDR_TAX_NUM, the TIN the line was
billed under, so the connector now gives professional and DME lines that
value as billing_tin too (institutional stays NULL; the facility is its
NPI and CCN). CMS publishes no provider TIN or EIN on data.cms.gov —
checked every enrollment, ownership, change-of-ownership, Provider of
Services and cost-report family in the catalog.

What can be vendored is CCN ↔ NPI ↔ organization. cms.datacms resolves
a dataset family to its newest version in the data.cms.gov catalog,
downloads the CSV (or pages the API) and names columns like the
generated cms.table models. `stack load cms` (aco.load.cms) loads the
institutional universe — hospital, SNF, HHA, hospice, FQHC and RHC
enrollments, the Provider of Services file and the three cost reports —
as cms.* with provenance columns and ingest_log rows, then derives
cms.institutional_provider, cms.ccn_crosswalk (PECOS ∪ POS-only CCNs,
65,161 rows today), cms.provider_tin_observed (NPI → TIN as seen on our
own claims) and cms.ccn_tin (CCN → NPI → TIN, with cms.aco_participant_tin
as the authoritative hook when a participant list is loaded). The
command prints facility-NPI → CCN → TIN coverage; the Data page gets the
same query. On the sandbox nothing resolves — its 8 facility NPIs are
synthetic — which is the expected answer there.

refs #1139

Some checks are pending

Hide all checks

CI / lint (push) Successful in 58s

Details

CI / notebooks-smoke (push) Successful in 1m44s

Details

CI / test (push) Failing after 3m3s

Details

Infra CI / notebooks (push) Successful in 1m13s

Details

Infra CI / docs (push) Successful in 1m46s

Details

Infra CI / llm (push) Successful in 49s

Details

Infra CI / mc (push) Failing after 36s

Details

Infra CI / zotero (push) Successful in 27s

Details

Infra CI / api (push) Successful in 1m9s

Details

Infra CI / bcda (push) Successful in 1m37s

Details

CI / lint (pull_request) Successful in 41s

Details

CI / notebooks-smoke (pull_request) Successful in 1m49s

Details

CI / test (pull_request) Failing after 2m50s

Details

Infra CI / docs (pull_request) Successful in 17s

Details

Infra CI / notebooks (pull_request) Successful in 59s

Details

Infra CI / api (pull_request) Successful in 14s

Details

Infra CI / bcda (pull_request) Successful in 14s

Details

Infra CI / llm (pull_request) Successful in 20s

Details

Infra CI / zotero (pull_request) Successful in 14s

Details

Infra CI / mc (pull_request) Failing after 34s

Details

ci/woodpecker/push/ci

Required

ci/woodpecker/pr/ci

Required

ci/woodpecker/push/infra-ci

Required

ci/woodpecker/pr/infra-ci

Required

Some required checks are missing.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin feat/bcda-v3:feat/bcda-v3

git checkout feat/bcda-v3

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

api

REST API (FastAPI)

auth

Authentication and authorization

bib

Bibliography, Zotero, citations

ci

CI/CD pipelines and Woodpecker plugins

cli

CLI commands and UX

cloud

Cloud provider abstraction (AWS, GCP, Azure, self-hosted)

config

Configuration and wiring

data

Data loading, CCLF, BCDA, seeds

databricks

Databricks jobs and deployment

docs

Documentation and generated references

image:api

image:docs

image:mc

image:notebooks

image:zotero

infra

lakehouse

Nessie, Polaris, Trino, Iceberg

llm

llm module — local RAG over comments + Zotero corpus

notebooks

marimo notebook errors (auto-filed by nb-watcher/nb-integration)

packaging

Python packaging, optional deps, build system

pfs

pipeline

Pipeline execution and validation

pipeline:ci

pipeline:deploy

pipeline:harden

pipeline:infra-ci

pipeline:rebuild-all

pipeline:release

qpp

quality

Testing, coverage, error handling

step:build

step:deploy

step:hadolint

step:lint

step:publish

step:scan

step:test

step:validate

styles

Charts, figures, visual theming

tree

Project directory structure and file organization

No Label data notebooks pipeline

Milestone

No items

No Milestone P52: BCDA v3 migration — client, crosswalk-driven flatteners, vendored upstream fixtures, proving notebook

Projects

Clear projects

No project

Assignees

Clear assignees

kert renovate-bot

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#1140