homelab/stack
1
0
Fork 0
Code Issues 122 Pull Requests 1 Actions Packages Projects 1 Releases Wiki Activity

bcda: migrate client + flatteners to BCDA v3 and prove it against the Large Model Entity sandbox #1139

New Issue
Open
opened 2026-10-25 15:55:05 +00:00 by kert · 0 comments
kert commented 2026-10-25 15:55:05 +00:00
Owner
Copy Link

Why

BCDA v3 went GA 2026-07-01; v1/v2 are discontinued 2027-07-30. The current bcda module targets v2: the client defaults to /api/v2, and the flattener matches BlueButton extensions by the retired resources/variables/<name> URL suffix, so on v3 payloads it silently emits NULL for almost every coded field.

Findings from the live sandbox spike (2026-10-25, Large Model Entity creds)

  • v3 auth/metadata/attribution_status work unchanged (fhirVersion 4.0.1, software r302).
  • GET /api/v{2,3}/jobs requires Prefer: respond-async + Accept: application/fhir+json and returns a FHIR Bundle of Task resources — the existing Client.jobs() was broken on both versions.
  • One job (89592) was reported 404 Job has been cancelled seconds after creation; the public sandbox credentials are shared, so any caller can cancel any job. The client must surface the OperationOutcome diagnostics instead of a bare HTTPStatusError.
  • Export of Patient+Coverage+EOB for the 10k-enrollee entity takes >20 min server-side.

Scope

  1. Client: default v3; _typeFilter support; fix jobs() headers and Bundle shape; OperationOutcome diagnostics on 4xx; /_version + /_health.
  2. Flatteners: v3 extractors driven by the v3 Data Dictionary (fhirPath → sourceColumn, with ccwMapping/cclfMapping crosswalk) into the existing bcda.* tables, with v3-native columns added where v2 had none (Coverage.period per plan, dual Coverage, System-Type/Final-Action tags, claim control number, consolidated adjudication amounts). v2 payloads keep working (auto-detected).
  3. Crosswalk seeds: data dictionary + CCLF↔BCDA v3 mapping + CCLF-fields-absent sheets as dev/seeds/*.csv → reference_data.
  4. Vendored upstream: sparse pin of CMSgov/bcda-app (v3 handler + tests, request validation, OperationOutcome writer, v3 Postman collection, gzip fixtures, sandbox config) plus the official v3 sample ndjson from bcda.cms.gov, with a manifest, a vendor_bcda_app.py --check/--sync script and a weekly workflow that files an issue on drift.
  5. Notebook notebooks/bcda_v3_sandbox.py: metadata → auth → export (or reuse a completed job) → Store download → flatten → DuckDB load → proof queries (MBI coverage, Coverage per plan, EOB by System-Type/EOB-TYPE, claim-control-number dedupe, dictionary field population) and the crosswalk tables.

Out of scope (follow-ups)

  • Rewriting bcda.express.cclf (input_layer pipeline) for v3 semantics beyond keeping it running on the new columns.
  • Partially-adjudicated (SharedSystem) exports — needs _typeFilter, which the client will support, but the sandbox Large entity is adjudicated-only.
## Why BCDA v3 went GA 2026-07-01; v1/v2 are discontinued 2027-07-30. The current `bcda` module targets v2: the client defaults to `/api/v2`, and the flattener matches BlueButton extensions by the retired `resources/variables/<name>` URL suffix, so on v3 payloads it silently emits NULL for almost every coded field. ## Findings from the live sandbox spike (2026-10-25, Large Model Entity creds) - v3 auth/metadata/attribution_status work unchanged (`fhirVersion 4.0.1`, software r302). - `GET /api/v{2,3}/jobs` requires `Prefer: respond-async` + `Accept: application/fhir+json` and returns a FHIR Bundle of Task resources — the existing `Client.jobs()` was broken on both versions. - One job (89592) was reported `404 Job has been cancelled` seconds after creation; the public sandbox credentials are shared, so any caller can cancel any job. The client must surface the OperationOutcome diagnostics instead of a bare HTTPStatusError. - Export of Patient+Coverage+EOB for the 10k-enrollee entity takes >20 min server-side. ## Scope 1. **Client**: default `v3`; `_typeFilter` support; fix `jobs()` headers and Bundle shape; OperationOutcome diagnostics on 4xx; `/_version` + `/_health`. 2. **Flatteners**: v3 extractors driven by the v3 Data Dictionary (fhirPath → `sourceColumn`, with `ccwMapping`/`cclfMapping` crosswalk) into the existing `bcda.*` tables, with v3-native columns added where v2 had none (Coverage.period per plan, dual Coverage, System-Type/Final-Action tags, claim control number, consolidated adjudication amounts). v2 payloads keep working (auto-detected). 3. **Crosswalk seeds**: data dictionary + CCLF↔BCDA v3 mapping + CCLF-fields-absent sheets as `dev/seeds/*.csv` → `reference_data`. 4. **Vendored upstream**: sparse pin of CMSgov/bcda-app (v3 handler + tests, request validation, OperationOutcome writer, v3 Postman collection, gzip fixtures, sandbox config) plus the official v3 sample ndjson from bcda.cms.gov, with a manifest, a `vendor_bcda_app.py --check/--sync` script and a weekly workflow that files an issue on drift. 5. **Notebook** `notebooks/bcda_v3_sandbox.py`: metadata → auth → export (or reuse a completed job) → Store download → flatten → DuckDB load → proof queries (MBI coverage, Coverage per plan, EOB by System-Type/EOB-TYPE, claim-control-number dedupe, dictionary field population) and the crosswalk tables. ## Out of scope (follow-ups) - Rewriting `bcda.express.cclf` (input_layer pipeline) for v3 semantics beyond keeping it running on the new columns. - Partially-adjudicated (SharedSystem) exports — needs `_typeFilter`, which the client will support, but the sandbox Large entity is adjudicated-only.
kert added this to the P52: BCDA v3 migration — client, crosswalk-driven flatteners, vendored upstream fixtures, proving notebook milestone 2026-10-25 15:55:05 +00:00
kert added the datanotebookspipeline labels 2026-10-25 15:55:05 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
api
REST API (FastAPI)
 auth
Authentication and authorization
 bib
Bibliography, Zotero, citations
 ci
CI/CD pipelines and Woodpecker plugins
 cli
CLI commands and UX
 cloud
Cloud provider abstraction (AWS, GCP, Azure, self-hosted)
 config
Configuration and wiring
 data
Data loading, CCLF, BCDA, seeds
 databricks
Databricks jobs and deployment
 docs
Documentation and generated references
 image:api
image:docs
image:mc
image:notebooks
image:zotero
infra
lakehouse
Nessie, Polaris, Trino, Iceberg
 llm
llm module — local RAG over comments + Zotero corpus
 notebooks
marimo notebook errors (auto-filed by nb-watcher/nb-integration)
 packaging
Python packaging, optional deps, build system
 pfs
pipeline
Pipeline execution and validation
 pipeline:ci
pipeline:deploy
pipeline:harden
pipeline:infra-ci
pipeline:rebuild-all
pipeline:release
qpp
quality
Testing, coverage, error handling
 step:build
step:deploy
step:hadolint
step:lint
step:publish
step:scan
step:test
step:validate
styles
Charts, figures, visual theming
 tree
Project directory structure and file organization
No Label data notebooks pipeline
Milestone
No items
No Milestone P52: BCDA v3 migration — client, crosswalk-driven flatteners, vendored upstream fixtures, proving notebook
Projects
Clear projects
No project
Assignees
Clear assignees
kert renovate-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: homelab/stack#1139
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?