diff --git a/docs/docs/cli/load-bcda.md b/docs/docs/cli/load-bcda.md index 60a19e9..58e9b9a 100644 --- a/docs/docs/cli/load-bcda.md +++ b/docs/docs/cli/load-bcda.md @@ -19,6 +19,8 @@ Usage: stack load bcda [OPTIONS] │ --ndjson-dir PATH Directory containing FHIR ndjson files. │ │ --database TEXT DuckDB database path (default: config). │ │ --skip-flatten Skip flattening, load existing Parquet. │ +│ --run-pipeline Also run the bcda connector pipeline into │ +│ input_layer.* │ │ --help Show this message and exit. │ ╰──────────────────────────────────────────────────────────────────────────────╯ ``` diff --git a/docs/superpowers/specs/2026-10-25-bcda-v3-design.md b/docs/superpowers/specs/2026-10-25-bcda-v3-design.md index 6716a31..dde53e0 100644 --- a/docs/superpowers/specs/2026-10-25-bcda-v3-design.md +++ b/docs/superpowers/specs/2026-10-25-bcda-v3-design.md @@ -52,3 +52,15 @@ Pulled as three single-type jobs: the EOB job (89596, 6 min server-side, 192 fil | `bcda.eob_item` | 248,528 | All 20 notebook checks pass: every Patient has a unique current MBI and `bene_id == Patient.id`; every Coverage row has `period.start`, every Part A row `ms_cd`, every Dual row `dual_stus_cd`, no orphan beneficiaries; every EOB has a claim id, a claim control number, System-Type and Final-Action tags, EOB-TYPE and CLM-TYPE-CD; control numbers are unique within a source (v3 deduplication holds); lines carry HCPCS or NDC on 100% and paid/submitted amounts on 94% (Part D lines carry drug-cost amounts instead); no orphan lines. The entity is adjudicated-only (no SharedSystem claims), so the control-number pairing check is exercised but finds zero pairs — partially adjudicated data needs an entity with PAC access and `_typeFilter`. + +## Connector pipeline on v3 semantics (`bcda.express.cclf`, 2026-10-25) + +The input_layer connector was still written for v2 and had never run end to end (its contracts named CCLF columns the EOB table does not have, and the runner's exact-match check would have refused every stage). Rewritten: + +1. **One header stage carries the v3 semantics.** `_stg_claim_header` drops `NotFinalAction` claims, derives `claim_family` from `EOB-TYPE` → `CLM-TYPE-CD` → `pde_id` → the v2 near-line RIC (which v3 emits on only some claims: HHA shows U/W/V/NULL), collapses a claim seen in two sources on its claim control number (NationalClaimsHistory/DDPS over SharedSystem, then newest `last_updated`; rows without a control number are never collapsed), and resolves the MBI. The four family stages are filters on it. +2. **One mapping for medical_claim.** `_medical_claim(kind)` produces the full `aco.table.input_layer.InputLayerMedicalClaim` shape for institutional, carrier and DME lines: admission/discharge from the C4BB admission period, NUBC type of bill (`clm_bill_type_cd`, rebuilt as `0` + the three-character v3 frequency code when the coding is missing), MS-DRG, revenue centre, HCPCS and five modifiers, attending→operating→rendering / rendering / rendering→ordering NPIs, billing Organization NPI and tax number, paid / allowed / charge / coinsurance / deductible from the consolidated v3 Adjudication amounts, paid date from `clm_pd_dt` → NCH weekly process → CMS process date, 12 header diagnoses with POA on the first, first ICD procedure. +3. **Part D** uses the PDE id, the prescribing care-team NPI, the contained pharmacy Organization NPI, NDC quantity, total drug cost as charge and paid-by-patient as copayment; plan-paid falls back line → header (the sandbox emits neither). +4. **Eligibility** reads the Part A row (Part B fallback) for status, entitlement and the enrollment period and the Dual row for dual status; gender accepts CCW codes and the FHIR words the v2 flattener passed through; `enrollment_status` comes from `Coverage.status`. +5. **Contracts inherit.** The header stages subclass `ExplanationOfBenefit`; the outputs subclass the canonical `InputLayer*` models, so the same downstream pipelines run on BCDA output as on CCLF output. `load_bcda(run_pipeline=True)` / `stack load bcda --run-pipeline` and `stack run bcda` land `input_layer.medical_claim`, `pharmacy_claim`, `eligibility`. + +Run on the Large-entity pull (94 s including re-flattening): 221,021 medical lines on 28,282 claims (16,468 institutional, 11,814 professional — equal to the FinalAction non-PDE EOB counts by type; 4,111 NotFinalAction excluded), 11,968 pharmacy rows (= FinalAction PDEs), 9,582 eligibility rows (7,290 with a Part A/B row, 4,868 with dual status). Every institutional line has a 4-character type of bill, admission date, paid, charge, deductible and coinsurance; every professional line has paid, rendering NPI and (99 %) place of service. diff --git a/notebooks/bcda_v3_sandbox.py b/notebooks/bcda_v3_sandbox.py index 7b83c8b..42a784f 100644 --- a/notebooks/bcda_v3_sandbox.py +++ b/notebooks/bcda_v3_sandbox.py @@ -30,11 +30,15 @@ def _(mo): younger than 24 h is **adopted** instead of re-run. 4. **Flatten + load** — `bcda.express.v3` (table-driven from the v3 Data Dictionary's `fhirPath` / `sourceColumn` / `ccwMapping` columns) streams - the ndjson into Parquet and `aco.duckdb` `bcda.*`, then the read-only - replica is republished. + the ndjson into Parquet and `aco.duckdb` `bcda.*`, runs the connector + pipeline (`bcda.pipe.cclf`: v3 routing on EOB-TYPE, Final-Action filter, + SharedSystem→NCH dedupe) into `input_layer.*`, then republishes the + read-only replica. 5. **Proof** — identity, coverage shape, claim provenance, line amounts, referential integrity, and how many dictionary-mapped columns actually carry data, each marked pass / fail. + 6. **input_layer** — the connector's medical_claim / pharmacy_claim / + eligibility reconciled against the `bcda.*` tables they came from. Network steps run only when **`BCDA_PULL=1`** is set or the buttons below are pressed (`BCDA_LOAD=1` flattens and loads what is already in the store without @@ -338,8 +342,8 @@ def _(LOAD, PULL, STORE_PATH, connect, load_btn, mo, pulled, store, time): if _do_load: _dirs = _find_latest_export(STORE_PATH) _t0 = time.time() - load_stats = ( - load_bcda() + load_stats = load_bcda( + run_pipeline=True ) # newest export per resource type → bcda.* in aco.duckdb _dt = time.time() - _t0 _replica = connect.publish_replica("aco") @@ -641,9 +645,187 @@ def _(alt, mo, plans, prov, q): return +@app.cell(hide_code=True) +def _(mo, q): + # ── 6. input_layer — the connector pipeline reconciled against bcda.* ── + _il = q( + "select table_name from information_schema.tables where table_schema = 'input_layer'" + )["table_name"].to_list() + mo.stop( + not {"medical_claim", "pharmacy_claim", "eligibility"} <= set(_il), + mo.md( + "## 6 · input_layer\n\n⚠️ `input_layer.*` not in the replica yet — run the " + "load above (it runs `bcda.pipe.cclf`) or `stack load bcda --skip-flatten --run-pipeline`." + ), + ) + il_checks: list[tuple[bool, str, str]] = [] + + def il(ok: bool, name: str, detail: str) -> None: + il_checks.append((bool(ok), name, detail)) + + _src = q(""" + select + count(*) filter (where final_action = 'FinalAction' and eob_type <> 'PDE') final_medical, + count(*) filter (where final_action = 'FinalAction' and eob_type = 'PDE') final_pde, + count(*) filter (where final_action = 'NotFinalAction') not_final, + count(*) filter (where final_action = 'FinalAction' and eob_type in ('INPATIENT','OUTPATIENT','SNF','HHA','HOSPICE')) final_inst, + count(*) filter (where final_action = 'FinalAction' and eob_type in ('CARRIER','DME')) final_prof + from bcda.explanation_of_benefit + """).to_dicts()[0] + _mc = q(""" + select claim_type, count(*) lines, count(distinct claim_id) claims, + count(person_id) with_person, count(paid_amount) with_paid, count(charge_amount) with_charge, + count(deductible_amount) with_ded, count(coinsurance_amount) with_coins, + count(hcpcs_code) with_hcpcs, count(rendering_npi) with_rendering, count(billing_npi) with_billing, + count(bill_type_code) with_tob, count(drg_code) with_drg, count(admission_date) with_adm, + count(place_of_service_code) with_pos, count(diagnosis_code_1) with_dx, count(paid_date) with_paid_dt + from input_layer.medical_claim group by 1 order by 1 + """) + _by = {r["claim_type"]: r for r in _mc.to_dicts()} + _inst, _prof = _by.get("institutional", {}), _by.get("professional", {}) + _claims = sum(r["claims"] for r in _mc.to_dicts()) + il( + _claims == _src["final_medical"], + "Every final-action medical EOB is in medical_claim, once", + f"{_claims:,} claims = {_src['final_medical']:,} FinalAction non-PDE EOBs; {_src['not_final']:,} NotFinalAction excluded", + ) + il( + _inst.get("claims") == _src["final_inst"] + and _prof.get("claims") == _src["final_prof"], + "Routed by EOB-TYPE: institutional vs professional", + f"institutional {_inst.get('claims', 0):,} / {_src['final_inst']:,} · professional {_prof.get('claims', 0):,} / {_src['final_prof']:,}", + ) + _no_person = q( + "select count(*) n from input_layer.medical_claim where person_id is null" + ).to_dicts()[0]["n"] + il( + _no_person == 0, + "Every line resolves to an MBI (person_id)", + f"{_no_person:,} lines without MBI", + ) + il( + _inst.get("with_adm") == _inst.get("lines") + and _inst.get("with_tob", 0) >= 0.99 * _inst.get("lines", 1), + "Institutional lines carry admission date and NUBC type of bill", + f"TOB {_inst.get('with_tob', 0):,}, admission {_inst.get('with_adm', 0):,} of {_inst.get('lines', 0):,}", + ) + _tob_len = q( + "select count(*) n from input_layer.medical_claim where claim_type='institutional' and length(bill_type_code) <> 4" + ).to_dicts()[0]["n"] + il( + _tob_len == 0, + "Type of bill is the 4-character NUBC code (not fac+clsfctn+freq)", + f"{_tob_len:,} malformed", + ) + il( + _inst.get("with_paid") == _inst.get("lines") + and _inst.get("with_charge") == _inst.get("lines") + and _inst.get("with_ded") == _inst.get("lines") + and _inst.get("with_coins") == _inst.get("lines"), + "Institutional paid / charge / deductible / coinsurance from v3 Adjudication", + f"paid {_inst.get('with_paid', 0):,} · charge {_inst.get('with_charge', 0):,} · ded {_inst.get('with_ded', 0):,} · coins {_inst.get('with_coins', 0):,}", + ) + il( + _prof.get("with_paid") == _prof.get("lines") + and _prof.get("with_pos", 0) >= 0.95 * _prof.get("lines", 1) + and _prof.get("with_rendering") == _prof.get("lines"), + "Professional lines: paid, place of service, rendering NPI", + f"paid {_prof.get('with_paid', 0):,} · POS {_prof.get('with_pos', 0):,} · rendering {_prof.get('with_rendering', 0):,} of {_prof.get('lines', 0):,}", + ) + il( + _inst.get("with_drg", 0) > 0 and _inst.get("with_dx", 0) == _inst.get("lines"), + "Institutional DRG and header diagnoses present", + f"DRG {_inst.get('with_drg', 0):,} · dx1 {_inst.get('with_dx', 0):,}", + ) + il( + (_inst.get("with_paid_dt", 0) + _prof.get("with_paid_dt", 0)) + == (_inst.get("lines", 0) + _prof.get("lines", 0)), + "Paid date resolved (clm_pd_dt → NCH weekly → CMS process)", + f"{_inst.get('with_paid_dt', 0) + _prof.get('with_paid_dt', 0):,} lines", + ) + + _rx = q(""" + select count(*) n, count(distinct claim_id) claims, count(prescribing_provider_npi) prs, count(dispensing_provider_npi) dsp, + count(ndc_code) ndc, count(quantity) qty, count(days_supply) ds, count(refills) rf, count(charge_amount) chg, + count(copayment_amount) cop, count(paid_amount) paid, count(paid_date) pd + from input_layer.pharmacy_claim + """).to_dicts()[0] + il( + _rx["claims"] == _src["final_pde"], + "Every final-action PDE is in pharmacy_claim, once", + f"{_rx['claims']:,} = {_src['final_pde']:,}", + ) + il( + _rx["prs"] == _rx["n"] + and _rx["dsp"] == _rx["n"] + and _rx["ndc"] == _rx["n"] + and _rx["qty"] == _rx["n"] + and _rx["ds"] == _rx["n"], + "Part D: prescriber, pharmacy, NDC, quantity, days supply populated", + f"{_rx['n']:,} rows", + ) + il( + _rx["chg"] == _rx["n"] and _rx["cop"] == _rx["n"], + "Part D: total drug cost and paid-by-patient populated", + f"charge {_rx['chg']:,} · copay {_rx['cop']:,} · plan-paid {_rx['paid']:,} (sandbox emits none)", + ) + + _el = q(""" + select count(*) n, count(distinct person_id) people, count(gender) g, count(enrollment_start_date) es, + count(medicare_status_code) ms, count(original_reason_entitlement_code) orec, count(dual_status_code) dual, + count(enrollment_status) st, count(*) filter (where gender = 'unknown') g_unk, count(*) filter (where race = 'unknown') r_unk + from input_layer.eligibility + """).to_dicts()[0] + _pat = q("select count(*) n from bcda.patient").to_dicts()[0]["n"] + _dual_rows = q( + "select count(distinct subscriber_id) n from bcda.coverage where class_plan = 'Dual Medicare/Medicaid'" + ).to_dicts()[0]["n"] + _ab_rows = q( + "select count(distinct subscriber_id) n from bcda.coverage where class_plan in ('Part A', 'Part B')" + ).to_dicts()[0]["n"] + il( + _el["n"] == _pat and _el["people"] == _pat, + "One eligibility row per Patient", + f"{_el['n']:,} = {_pat:,}", + ) + il( + _el["g_unk"] == 0, + "Gender resolved for every row", + f"{_el['g_unk']:,} unknown; race unknown {_el['r_unk']:,} (OMB UNK)", + ) + il( + _el["es"] == _el["ms"] == _el["st"] == _ab_rows, + "Enrollment period, Medicare status and status from the Part A/B Coverage row", + f"{_el['es']:,} rows = {_ab_rows:,} beneficiaries with a Part A/B row; {_el['n'] - _el['es']:,} patients have none in this pull", + ) + il( + _el["dual"] == _dual_rows, + "Dual status from the Dual Medicare/Medicaid Coverage row", + f"{_el['dual']:,} = {_dual_rows:,} beneficiaries with a Dual row", + ) + + _passed = sum(1 for ok, _, _ in il_checks if ok) + mo.vstack( + [ + mo.md(f"## 6 · input_layer — {_passed} / {len(il_checks)} checks pass"), + mo.md( + "\n".join( + f"- {'✅' if ok else '❌'} **{name}** — {detail}" + for ok, name, detail in il_checks + ) + ), + mo.ui.table( + _mc, + label="medical_claim by claim_type: lines, claims and populated fields", + ), + ] + ) + return (il_checks,) + + @app.cell(hide_code=True) def _(ROOT, con, mo, pl, q): - # ── 6. Dictionary field population + CCLF crosswalk ── + # ── 7. Dictionary field population + CCLF crosswalk ── _cols = q( "select table_name, column_name from information_schema.columns where table_schema = 'bcda' order by 1, 2" ) @@ -721,7 +903,7 @@ def _(ROOT, con, mo, pl, q): mo.vstack( [ mo.md(f""" - ## 6 · Dictionary field population and the CCLF crosswalk + ## 7 · Dictionary field population and the CCLF crosswalk Columns are named after the v3 Data Dictionary's `ccwMapping` (legacy CCW names) or its IDR `sourceColumn` (v3-only). A populated column means the v3 payload actually carried that element diff --git a/src/aco/load/bcda.py b/src/aco/load/bcda.py index f76a877..7fee4d8 100644 --- a/src/aco/load/bcda.py +++ b/src/aco/load/bcda.py @@ -22,6 +22,7 @@ def load_bcda( database: str | None = None, *, skip_flatten: bool = False, + run_pipeline: bool = False, ) -> dict[str, int]: """Flatten BCDA ndjson and load into DuckDB. @@ -37,6 +38,12 @@ def load_bcda( DuckDB database path. Defaults to ``conf.path("db.aco")``. skip_flatten : bool If True, skip the flatten step and load existing Parquet files. + run_pipeline : bool + After loading, run the ``bcda`` connector pipeline + (``bcda.pipe.cclf``) and save ``input_layer.medical_claim``, + ``input_layer.pharmacy_claim`` and ``input_layer.eligibility`` + — the same tables the CCLF loader writes. Off by default + because it replaces whatever source last wrote them. Returns ------- @@ -89,9 +96,44 @@ def load_bcda( stats[table_ref] = count con.close() + if run_pipeline and stats: + stats.update(_run_bcda_pipeline(db_path)) return stats +BCDA_PIPELINE_OUTPUTS = { + "bcda.medical_claim": "input_layer.medical_claim", + "bcda.pharmacy_claim": "input_layer.pharmacy_claim", + "bcda.eligibility": "input_layer.eligibility", +} + + +def _run_bcda_pipeline(database: str) -> dict[str, int]: + """Run ``bcda.pipe.cclf`` against *database* and save the input_layer tables. + + Every expression's contract is validated by the runner (column set + must match exactly), so a successful run is also the proof that the + v3-flattened tables carry what the connector reads. + """ + import narwhals as nw + + from aco.lake.context import DuckDBContext + from bcda.pipe.cclf import pipeline + + ctx = DuckDBContext(database=database, read_only=False) + cache = pipeline.run(ctx.load) + out: dict[str, int] = {} + for source, target in BCDA_PIPELINE_OUTPUTS.items(): + if source in cache: + df = cache[source] + # narwhalify'd expressions hand back native frames; save wants narwhals. + if not isinstance(df, (nw.DataFrame, nw.LazyFrame)): + df = nw.from_native(df) + ctx.save(target, df, mode="replace") + out[target] = len(cache[source]) + return out + + def _find_latest_export(store_path: Path) -> list[Path]: """Export directories holding the newest file of each resource type. diff --git a/src/aco/pipe/__init__.py b/src/aco/pipe/__init__.py index ff2724f..e5bbbbb 100644 --- a/src/aco/pipe/__init__.py +++ b/src/aco/pipe/__init__.py @@ -33,3 +33,12 @@ registry: dict[str, Pipeline] = { "quality_measures": quality_measures.pipeline, "readmissions": readmissions.pipeline, } + +# The BCDA connector lives in the optional ``bcda`` extra; register it +# when installed so ``stack run bcda`` works like ``stack run cclf``. +try: + from bcda.pipe import cclf as _bcda_cclf +except ImportError: # pragma: no cover — stack[bcda] not installed + _bcda_cclf = None +if _bcda_cclf is not None: + registry["bcda"] = _bcda_cclf.pipeline diff --git a/src/bcda/express/cclf.py b/src/bcda/express/cclf.py index 82b85a8..5d8776a 100644 --- a/src/bcda/express/cclf.py +++ b/src/bcda/express/cclf.py @@ -1,55 +1,254 @@ """BCDA FHIR R4 connector — EOB + Patient + Coverage → input_layer tables. -Transforms flattened BCDA FHIR R4 resources into three input_layer -output tables that match what the CCLF pipeline produces: +Transforms the flattened BCDA tables (``bcda.patient``, ``bcda.coverage``, +``bcda.explanation_of_benefit``, ``bcda.eob_item``) into the three +input_layer tables the CCLF pipeline also produces, on the canonical +contracts in ``aco.table.input_layer``: - - input_layer.medical_claim (Part A + Part B carrier + Part B DME) + - input_layer.medical_claim (institutional + professional + DME) - input_layer.pharmacy_claim (Part D) - input_layer.eligibility (Patient + Coverage) -Unlike the CCLF pipeline (19 functions), this pipeline is simpler -(12 functions) because BCDA delivers: - - Current MBIs directly (no CCLF9 crosswalk needed) - - Final-action claims only (no ADR debit/credit reconciliation) - - Inline diagnosis codes (no CCLF4 long→wide pivot) +v3 semantics (BCDA v3 Data Dictionary, migration guide, July-2026 +transition deck) — each is also honoured when the row came from a v2 +export, because the flattener leaves v3-only columns NULL there: -Claim routing uses ``nch_near_line_rec_ident_cd``: - V → Part A institutional → medical_claim (institutional) - O → Part B carrier → medical_claim (professional) - M → Part B DME → medical_claim (professional) - P → Part D pharmacy → pharmacy_claim +**Routing.** v3 carries the claim family on every EOB: the ``EOB-TYPE`` +coding (``eob_type``) and the CCW ``CLM-TYPE-CD`` (``clm_type_cd``). +The v2 near-line record identification code is emitted on only some v3 +claims (HHA shows U/W/V/NULL on the same type), so it is the *last* +resort, not the router:: + + eob_type INPATIENT OUTPATIENT SNF HHA HOSPICE → institutional + CARRIER → professional · DME → dme · PDE → pharmacy + clm_type 10 20 30 40 50 60–64 → institutional · 71 72 → professional + 81 82 → dme · 1–4 → pharmacy + pde_id present → pharmacy + ric V W U → institutional · O → professional · M → dme · P → pharmacy + +**Final action.** v3 tags every EOB ``Final-Action|FinalAction`` or +``NotFinalAction`` (9% of the Large-entity sandbox is NotFinalAction). +Only final-action claims reach input_layer, as with CCLF. v2 rows have +no tag and pass. + +**One claim, two sources.** A partially adjudicated SharedSystem EOB +and its NationalClaimsHistory successor share a claim control number. +The NCH/DDPS row wins; within a source the newest ``last_updated`` wins. +Rows without a control number (v2) are never collapsed. + +**Amounts.** v3 consolidates line amounts under the ``Adjudication`` +CodeSystem; the flattener lands them on the CCW-named columns +(``clm_line_cvrd_pd_amt``, ``clm_line_alowd_chrg_amt``, +``clm_line_sbmtd_chrg_amt`` / ``rev_cntr_tot_chrg_amt``, coinsurance, +deductible), so paid / allowed / charge / coinsurance / deductible are +populated for every family. Part D plan-paid is not emitted by the +sandbox at line level; the pipeline coalesces line → header. + +**Providers.** The billing provider is the contained Organization +(``fac_prvdr_npi_num`` / ``carr_clm_blg_npi_num``, tax number in +``clm_blg_prvdr_tax_num``); institutional rendering is attending → +operating → rendering; DME rendering is rendering → ordering. + +**Dates.** Admission / discharge come from the C4BB ``admissionperiod`` +(``clm_actv_care_from_dt`` / ``clm_dschrg_dt``) with the billable period +as fallback; paid date is ``clm_pd_dt`` → NCH weekly process date → CMS +process date. + +**Type of bill.** v3's ``CLM-BILL-FREQ-CD`` coding carries three +characters (``48C``), so the v2 ``fac + clsfctn + freq`` concatenation +is wrong there; the NUBC code (``clm_bill_type_cd``) is used when +present. + +**Eligibility.** Coverage is one row per plan and period in v3; the +Part A row (Part B fallback) carries status, entitlement and the +enrollment period, and the Dual Medicare/Medicaid row carries dual +status (v2: ``dual_01`` on the Part A row). Pipeline stages:: - BCDA FHIR R4 resources - │ - ▼ stg: filter by claim type, resolve MBI - │ - ▼ int: join header + items, map to input_layer schema - │ - ▼ final: medical_claim, pharmacy_claim, eligibility + bcda.patient ─▶ _stg_beneficiary_xref ─┐ + bcda.explanation_of_benefit ───────────┴▶ _stg_claim_header + (final action · family · dedupe · MBI) + ├─ _stg_part_a_header ──┐ + ├─ _stg_part_b_physician_header ─┤ ⋈ bcda.eob_item + ├─ _stg_part_b_dme_header ───────┤ + │ _int_* ──▶ medical_claim ◀─┘ + └─ _stg_part_d_header ──▶ pharmacy_claim + bcda.patient ⋈ bcda.coverage ─▶ _stg_beneficiary_demographics ─▶ eligibility """ from __future__ import annotations +from typing import Any + import narwhals as nw from narwhals.typing import FrameT +# ── Vocabulary ─────────────────────────────────────────────────── + +INSTITUTIONAL_EOB_TYPES = ("INPATIENT", "OUTPATIENT", "SNF", "HHA", "HOSPICE") +INSTITUTIONAL_CLM_TYPES = ("10", "20", "30", "40", "50", "60", "61", "62", "63", "64") +PROFESSIONAL_CLM_TYPES = ("71", "72") +DME_CLM_TYPES = ("81", "82") +PHARMACY_CLM_TYPES = ("1", "2", "3", "4") + +FAMILIES = ("institutional", "professional", "dme", "pharmacy") + +# Header columns the stages read. Any that the source frame lacks (a +# v2 export, a minimal test fixture) are added as typed NULLs so the +# same expressions run on both dialects. +_HEADER_COLS: dict[str, Any] = { + "clm_id": nw.String, + "clm_cntl_num": nw.String, + "bene_id": nw.String, + "pde_id": nw.String, + "eob_type": nw.String, + "clm_type_cd": nw.String, + "nch_near_line_rec_ident_cd": nw.String, + "system_type": nw.String, + "final_action": nw.String, + "last_updated": nw.String, + "clm_from_dt": nw.Date, + "clm_thru_dt": nw.Date, + "clm_actv_care_from_dt": nw.Date, + "clm_dschrg_dt": nw.Date, + "clm_pd_dt": nw.Date, + "clm_nch_wkly_proc_dt": nw.Date, + "clm_cms_proc_dt": nw.Date, + "clm_admsn_src_cd": nw.String, + "clm_admsn_type_cd": nw.String, + "bene_ptnt_stus_cd": nw.String, + "clm_bill_type_cd": nw.String, + "clm_bill_fac_type_cd": nw.String, + "clm_bill_clsfctn_cd": nw.String, + "clm_bill_freq_cd": nw.String, + "dgns_drg_cd": nw.String, + "dgns_prcdr_icd_ind": nw.String, + "clm_poa_ind": nw.String, + "clm_prcdr_cd": nw.String, + "clm_prcdr_prfrm_dt": nw.Date, + "clm_pmt_amt": nw.Float64, + "clm_mdcr_instnl_tot_chrg_amt": nw.Float64, + "clm_bene_pmt_amt": nw.Float64, + "tot_rx_cst_amt": nw.Float64, + "atndg_prvdr_npi_num": nw.String, + "oprtg_prvdr_npi_num": nw.String, + "rndrg_prvdr_npi_num": nw.String, + "ordrg_prvdr_npi_num": nw.String, + "payto_prvdr_npi_num": nw.String, + "fac_prvdr_npi_num": nw.String, + "carr_clm_blg_npi_num": nw.String, + "clm_rndrg_prvdr_tax_num": nw.String, + "clm_blg_prvdr_tax_num": nw.String, + "clm_prsbng_prvdr_gnrc_id_num": nw.String, + "clm_srvc_prvdr_gnrc_id_num": nw.String, + "clm_line_days_suply_qty": nw.Float64, + "clm_line_rx_fill_num": nw.Float64, + **{f"clm_dgns_{i}_cd": nw.String for i in range(1, 13)}, +} + +_ITEM_COLS: dict[str, Any] = { + "clm_id": nw.String, + "clm_line_num": nw.Float64, + "clm_line_from_dt": nw.Date, + "clm_line_thru_dt": nw.Date, + "clm_line_prod_rev_ctr_cd": nw.String, + "clm_line_hcpcs_cd": nw.String, + "clm_line_ndc_cd": nw.String, + "clm_line_ndc_qty": nw.Float64, + "clm_pos_cd": nw.String, + "clm_line_srvc_unit_qty": nw.Float64, + "clm_line_cvrd_pd_amt": nw.Float64, + "clm_line_alowd_chrg_amt": nw.Float64, + "clm_line_sbmtd_chrg_amt": nw.Float64, + "rev_cntr_tot_chrg_amt": nw.Float64, + "clm_line_coinsrnc_amt": nw.Float64, + "rev_cntr_coinsrnc_wge_adjstd_c": nw.Float64, + "clm_line_bene_ptb_ddctbl_amt": nw.Float64, + "rev_cntr_cash_ddctbl_amt": nw.Float64, + "clm_line_bene_pmt_amt": nw.Float64, + "tot_rx_cst_amt": nw.Float64, + **{f"hcpcs_{i}_mdfr_cd": nw.String for i in range(1, 6)}, +} + # ── Helpers ────────────────────────────────────────────────────── +def _ensure(df: FrameT, cols: dict[str, Any]) -> FrameT: + """Add any missing column as a typed NULL literal.""" + have = set(df.collect_schema().names()) + missing = [ + nw.lit(None).cast(dt).alias(c) for c, dt in cols.items() if c not in have + ] + return df.with_columns(*missing) if missing else df + + +def _case(*pairs: tuple[nw.Expr, nw.Expr], default: nw.Expr) -> nw.Expr: + """Nested ``when/then/otherwise`` — narwhals has no chained ``when``.""" + expr = default + for cond, value in reversed(pairs): + expr = nw.when(cond).then(value).otherwise(expr) + return expr + + +def _claim_family() -> nw.Expr: + """Route an EOB header to a claim family (see module docstring).""" + eob = nw.col("eob_type") + ct = nw.col("clm_type_cd").cast(nw.String) + ric = nw.col("nch_near_line_rec_ident_cd") + lit = nw.lit + return _case( + (eob.is_in(INSTITUTIONAL_EOB_TYPES), lit("institutional")), + (eob == "CARRIER", lit("professional")), + (eob == "DME", lit("dme")), + (eob == "PDE", lit("pharmacy")), + (ct.is_in(INSTITUTIONAL_CLM_TYPES), lit("institutional")), + (ct.is_in(PROFESSIONAL_CLM_TYPES), lit("professional")), + (ct.is_in(DME_CLM_TYPES), lit("dme")), + (ct.is_in(PHARMACY_CLM_TYPES), lit("pharmacy")), + (~nw.col("pde_id").is_null(), lit("pharmacy")), + (ric.is_in(["V", "W", "U"]), lit("institutional")), + (ric == "O", lit("professional")), + (ric == "M", lit("dme")), + (ric == "P", lit("pharmacy")), + default=lit(None).cast(nw.String), + ).alias("claim_family") + + def _icd_indicator() -> nw.Expr: - """Map dgns_prcdr_icd_ind to standard code type.""" - return ( - nw.when(nw.col("dgns_prcdr_icd_ind") == "0") - .then(nw.lit("icd-10-cm")) - .otherwise( - nw.when(nw.col("dgns_prcdr_icd_ind") == "9") - .then(nw.lit("icd-9-cm")) - .otherwise(nw.lit(None)) - ) - .alias("diagnosis_code_type") - ) + """Map dgns_prcdr_icd_ind to the diagnosis code type.""" + ind = nw.col("dgns_prcdr_icd_ind") + return _case( + (ind == "0", nw.lit("icd-10-cm")), + (ind == "9", nw.lit("icd-9-cm")), + default=nw.lit(None).cast(nw.String), + ).alias("diagnosis_code_type") + + +def _pcs_indicator() -> nw.Expr: + ind = nw.col("dgns_prcdr_icd_ind") + return _case( + (ind == "0", nw.lit("icd-10-pcs")), + (ind == "9", nw.lit("icd-9-pcs")), + default=nw.lit(None).cast(nw.String), + ).alias("procedure_code_type") + + +def _paid_date() -> nw.Expr: + return nw.coalesce( + nw.col("clm_pd_dt"), nw.col("clm_nch_wkly_proc_dt"), nw.col("clm_cms_proc_dt") + ).alias("paid_date") + + +def _dedupe_lines(items: FrameT) -> FrameT: + """One row per (claim, line). ``item.sequence`` is unique within an + EOB, so a duplicate can only be the same export part loaded twice + (or the same claim version landing in two parts); keep one.""" + return items.unique(subset=["clm_id", "clm_line_num"], keep="any") + + +def _int_qty(col: str) -> nw.Expr: + return nw.col(col).cast(nw.Float64).round(0).cast(nw.Int64) # ── Stage 1: Beneficiary XREF ─────────────────────────────────── @@ -61,9 +260,10 @@ def stg_beneficiary_xref( ) -> FrameT: """Build MBI crosswalk from BCDA Patient identifiers. - BCDA Patient delivers the current MBI directly in bene_mbi_id. - This deduplicates to one row per beneficiary with a stable - bene_id and the current MBI. + BCDA delivers the current MBI directly in ``bene_mbi_id`` (v2) or as + the ``us-mbi`` identifier without a period end (v3, picked by the + flattener). One row per beneficiary: ``bene_id`` (v2 CCW BENE_ID, + v3 BENE_SK) → current MBI. Operations: select, unique Dependencies: bcda.patient @@ -74,26 +274,281 @@ def stg_beneficiary_xref( ).unique(subset=["bene_id"], keep="any") -# ── Stage 2: Part A Institutional Claims ───────────────────────── +# ── Stage 2: Claim header — final action, family, dedupe, MBI ──── @nw.narwhalify -def stg_part_a_header( +def stg_claim_header( bcda__explanation_of_benefit: FrameT, bcda___stg_beneficiary_xref: FrameT, ) -> FrameT: - """Filter EOB to Part A claims and resolve MBI. + """Every EOB header that belongs in input_layer, tagged with its family. - Filter: nch_near_line_rec_ident_cd == 'V' - Join: bene_id → current_bene_mbi_id from xref + 1. Drop ``NotFinalAction`` claims (v3 tag; v2 rows carry no tag). + 2. Derive ``claim_family`` (v3 EOB-TYPE → CLM-TYPE-CD → pde_id → + v2 near-line RIC). + 3. Collapse a claim seen in two sources: rows sharing a claim + control number keep the NationalClaimsHistory / DDPS row over + the SharedSystem one, then the newest ``last_updated``. Rows + without a control number are left alone. + 4. Resolve the current MBI through the xref. - Operations: filter, join + Operations: filter, with_columns, sort, unique, concat, join Dependencies: bcda.explanation_of_benefit, bcda._stg_beneficiary_xref """ + eob = _ensure(bcda__explanation_of_benefit, _HEADER_COLS) + final = eob.filter( + nw.col("final_action").is_null() | (nw.col("final_action") == "FinalAction") + ).with_columns(_claim_family()) + + rank = ( + nw.when(nw.col("system_type") == "SharedSystem") + .then(nw.lit(1)) + .otherwise(nw.lit(0)) + .alias("_src_rank") + ) + keyed = ( + final.filter(~nw.col("clm_cntl_num").is_null()) + .with_columns(rank) + .sort( + ["clm_cntl_num", "_src_rank", "last_updated"], + descending=[False, False, True], + ) + .unique(subset=["clm_cntl_num"], keep="first", maintain_order=True) + .drop("_src_rank") + ) + unkeyed = final.filter(nw.col("clm_cntl_num").is_null()) + headers = nw.concat([keyed, unkeyed], how="vertical") + xref = bcda___stg_beneficiary_xref - return bcda__explanation_of_benefit.filter( - nw.col("nch_near_line_rec_ident_cd") == "V" - ).join(xref, on="bene_id", how="left") + return headers.join(xref, on="bene_id", how="left") + + +def _family(header: FrameT, family: str) -> FrameT: + return header.filter(nw.col("claim_family") == family) + + +@nw.narwhalify +def stg_part_a_header(bcda___stg_claim_header: FrameT) -> FrameT: + """Institutional claims (INPATIENT, OUTPATIENT, SNF, HHA, HOSPICE). + + Operations: filter + Dependencies: bcda._stg_claim_header + """ + return _family(bcda___stg_claim_header, "institutional") + + +@nw.narwhalify +def stg_part_b_physician_header(bcda___stg_claim_header: FrameT) -> FrameT: + """Part B carrier (professional) claims. + + Operations: filter + Dependencies: bcda._stg_claim_header + """ + return _family(bcda___stg_claim_header, "professional") + + +@nw.narwhalify +def stg_part_b_dme_header(bcda___stg_claim_header: FrameT) -> FrameT: + """Part B DME claims. + + Operations: filter + Dependencies: bcda._stg_claim_header + """ + return _family(bcda___stg_claim_header, "dme") + + +@nw.narwhalify +def stg_part_d_header(bcda___stg_claim_header: FrameT) -> FrameT: + """Part D pharmacy claims (PDE). + + Operations: filter + Dependencies: bcda._stg_claim_header + """ + return _family(bcda___stg_claim_header, "pharmacy") + + +# ── Stage 3: medical_claim ────────────────────────────────────── + + +def _medical_claim(header: FrameT, items: FrameT, kind: str) -> FrameT: + """Header ⋈ items → input_layer.medical_claim for one family. + + ``kind`` is ``institutional``, ``professional`` or ``dme``; the + selection differs only where the dictionary differs (admission + fields, bill type, DRG and revenue centre are institutional; place + of service is professional/DME; the provider roles vary). + """ + hdr = _ensure(header, _HEADER_COLS) + lines = _dedupe_lines(_ensure(items, _ITEM_COLS)) + df = hdr.join(lines, on="clm_id", how="inner", suffix="_item") + inst = kind == "institutional" + + null_str = nw.lit(None).cast(nw.String) + null_date = nw.lit(None).cast(nw.Date) + null_float = nw.lit(None).cast(nw.Float64) + + if inst: + rendering = nw.coalesce( + nw.col("atndg_prvdr_npi_num"), + nw.col("oprtg_prvdr_npi_num"), + nw.col("rndrg_prvdr_npi_num"), + ) + billing = nw.coalesce( + nw.col("fac_prvdr_npi_num"), nw.col("carr_clm_blg_npi_num") + ) + facility = billing + rendering_tin = null_str + elif kind == "professional": + rendering = nw.col("rndrg_prvdr_npi_num") + billing = nw.coalesce( + nw.col("carr_clm_blg_npi_num"), nw.col("fac_prvdr_npi_num") + ) + facility = null_str + rendering_tin = nw.col("clm_rndrg_prvdr_tax_num") + else: # dme + rendering = nw.coalesce( + nw.col("rndrg_prvdr_npi_num"), nw.col("ordrg_prvdr_npi_num") + ) + billing = nw.coalesce( + nw.col("carr_clm_blg_npi_num"), + nw.col("payto_prvdr_npi_num"), + nw.col("fac_prvdr_npi_num"), + ) + facility = null_str + rendering_tin = nw.col("clm_rndrg_prvdr_tax_num") + + if inst: + charge = nw.coalesce( + nw.col("rev_cntr_tot_chrg_amt"), + nw.col("clm_line_sbmtd_chrg_amt"), + nw.col("clm_mdcr_instnl_tot_chrg_amt"), + ) + coinsurance = nw.coalesce( + nw.col("rev_cntr_coinsrnc_wge_adjstd_c"), nw.col("clm_line_coinsrnc_amt") + ) + deductible = nw.coalesce( + nw.col("rev_cntr_cash_ddctbl_amt"), nw.col("clm_line_bene_ptb_ddctbl_amt") + ) + else: + charge = nw.coalesce( + nw.col("clm_line_sbmtd_chrg_amt"), nw.col("rev_cntr_tot_chrg_amt") + ) + coinsurance = nw.col("clm_line_coinsrnc_amt") + deductible = nw.col("clm_line_bene_ptb_ddctbl_amt") + + # NUBC type of bill when the flattener captured it; otherwise rebuild + # it. v3's CLM-BILL-FREQ-CD coding already carries facility + + # classification + frequency (``48C``), so the NUBC code is a leading + # zero plus that; v2 carried one character per part. + freq = nw.col("clm_bill_freq_cd").cast(nw.String) + bill_type = nw.coalesce( + nw.col("clm_bill_type_cd"), + _case( + (freq.str.len_chars() >= 3, nw.lit("0") + freq), + ( + ~freq.is_null(), + nw.col("clm_bill_fac_type_cd").cast(nw.String).fill_null("") + + nw.col("clm_bill_clsfctn_cd").cast(nw.String).fill_null("") + + freq, + ), + default=null_str, + ), + ) + + dx_cols = [ + (nw.col(f"clm_dgns_{i}_cd") if i <= 12 else null_str).alias( + f"diagnosis_code_{i}" + ) + for i in range(1, 26) + ] + poa_cols = [ + (nw.col("clm_poa_ind") if (inst and i == 1) else null_str).alias( + f"diagnosis_poa_{i}" + ) + for i in range(1, 26) + ] + px_code_cols = [ + (nw.col("clm_prcdr_cd") if (inst and i == 1) else null_str).alias( + f"procedure_code_{i}" + ) + for i in range(1, 26) + ] + px_date_cols = [ + (nw.col("clm_prcdr_prfrm_dt") if (inst and i == 1) else null_date).alias( + f"procedure_date_{i}" + ) + for i in range(1, 26) + ] + + admission = nw.coalesce(nw.col("clm_actv_care_from_dt"), nw.col("clm_from_dt")) + discharge = nw.coalesce(nw.col("clm_dschrg_dt"), nw.col("clm_thru_dt")) + drg_type = ( + nw.when(~nw.col("dgns_drg_cd").is_null()) + .then(nw.lit("ms-drg")) + .otherwise(null_str) + ) + + return df.select( + nw.col("clm_id").alias("claim_id"), + nw.col("clm_line_num").cast(nw.Int64).alias("claim_line_number"), + nw.lit("institutional" if inst else "professional").alias("claim_type"), + nw.col("current_bene_mbi_id").alias("person_id"), + nw.col("current_bene_mbi_id").alias("member_id"), + nw.lit("medicare").alias("payer"), + nw.lit("medicare").alias("plan"), + nw.col("clm_from_dt").alias("claim_start_date"), + nw.col("clm_thru_dt").alias("claim_end_date"), + nw.col("clm_line_from_dt").alias("claim_line_start_date"), + nw.col("clm_line_thru_dt").alias("claim_line_end_date"), + (admission if inst else null_date).alias("admission_date"), + (discharge if inst else null_date).alias("discharge_date"), + (nw.col("clm_admsn_src_cd") if inst else null_str).alias("admit_source_code"), + (nw.col("clm_admsn_type_cd") if inst else null_str).alias("admit_type_code"), + (nw.col("bene_ptnt_stus_cd") if inst else null_str).alias( + "discharge_disposition_code" + ), + (null_str if inst else nw.col("clm_pos_cd")).alias("place_of_service_code"), + (bill_type if inst else null_str).alias("bill_type_code"), + (drg_type if inst else null_str).alias("drg_code_type"), + ( + nw.col("dgns_drg_cd").cast(nw.String).str.slice(-3) if inst else null_str + ).alias("drg_code"), + (nw.col("clm_line_prod_rev_ctr_cd") if inst else null_str).alias( + "revenue_center_code" + ), + _int_qty("clm_line_srvc_unit_qty").alias("service_unit_quantity"), + nw.col("clm_line_hcpcs_cd").alias("hcpcs_code"), + nw.col("hcpcs_1_mdfr_cd").alias("hcpcs_modifier_1"), + nw.col("hcpcs_2_mdfr_cd").alias("hcpcs_modifier_2"), + nw.col("hcpcs_3_mdfr_cd").alias("hcpcs_modifier_3"), + nw.col("hcpcs_4_mdfr_cd").alias("hcpcs_modifier_4"), + nw.col("hcpcs_5_mdfr_cd").alias("hcpcs_modifier_5"), + rendering.alias("rendering_npi"), + rendering_tin.alias("rendering_tin"), + billing.alias("billing_npi"), + nw.col("clm_blg_prvdr_tax_num").alias("billing_tin"), + facility.alias("facility_npi"), + _paid_date(), + nw.col("clm_line_cvrd_pd_amt").cast(nw.Float64).alias("paid_amount"), + nw.col("clm_line_alowd_chrg_amt").cast(nw.Float64).alias("allowed_amount"), + charge.cast(nw.Float64).alias("charge_amount"), + coinsurance.cast(nw.Float64).alias("coinsurance_amount"), + null_float.alias("copayment_amount"), + deductible.cast(nw.Float64).alias("deductible_amount"), + null_float.alias("total_cost_amount"), + _icd_indicator(), + *dx_cols, + *poa_cols, + (_pcs_indicator() if inst else null_str.alias("procedure_code_type")), + *px_code_cols, + *px_date_cols, + nw.lit(1).alias("in_network_flag"), + nw.lit("medicare bcda").alias("data_source"), + null_str.alias("file_name"), + null_date.alias("file_date"), + nw.lit(None).cast(nw.Datetime).alias("ingest_datetime"), + ) @nw.narwhalify @@ -101,139 +556,12 @@ def int_institutional_medical_claim( bcda___stg_part_a_header: FrameT, bcda__eob_item: FrameT, ) -> FrameT: - """Join Part A header + line items → medical_claim (institutional). + """Institutional header ⋈ revenue-centre lines → medical_claim. Operations: join, select Dependencies: bcda._stg_part_a_header, bcda.eob_item """ - hdr = bcda___stg_part_a_header - items = bcda__eob_item - - df = hdr.join(items, on="clm_id", how="inner", suffix="_item") - - # Diagnosis codes 1-12 from EOB header, 13-25 null - dx_cols = [] - for i in range(1, 26): - if i <= 12: - dx_cols.append(nw.col(f"clm_dgns_{i}_cd").alias(f"diagnosis_code_{i}")) - else: - dx_cols.append(nw.lit(None).cast(nw.String).alias(f"diagnosis_code_{i}")) - - # POA: slot 1 from header, rest null - poa_cols = [] - for i in range(1, 26): - if i == 1: - poa_cols.append(nw.col("clm_poa_ind").alias("diagnosis_poa_1")) - else: - poa_cols.append(nw.lit(None).cast(nw.String).alias(f"diagnosis_poa_{i}")) - - # Procedure: slot 1 from header, rest null - px_code_cols = [ - nw.col("clm_prcdr_cd").alias("procedure_code_1"), - *[ - nw.lit(None).cast(nw.String).alias(f"procedure_code_{i}") - for i in range(2, 26) - ], - ] - px_date_cols = [ - nw.col("clm_prcdr_prfrm_dt").alias("procedure_date_1"), - *[ - nw.lit(None).cast(nw.Date).alias(f"procedure_date_{i}") - for i in range(2, 26) - ], - ] - - null_str = nw.lit(None).cast(nw.String) - null_date = nw.lit(None).cast(nw.Date) - null_float = nw.lit(None).cast(nw.Float64) - - return df.select( - nw.col("clm_id").alias("claim_id"), - nw.col("clm_line_num").cast(nw.Int64).alias("claim_line_number"), - nw.lit("institutional").alias("claim_type"), - nw.col("current_bene_mbi_id").alias("person_id"), - nw.col("current_bene_mbi_id").alias("member_id"), - nw.lit("medicare").alias("payer"), - nw.lit("medicare").alias("plan"), - nw.col("clm_from_dt").alias("claim_start_date"), - nw.col("clm_thru_dt").alias("claim_end_date"), - nw.col("clm_line_from_dt").alias("claim_line_start_date"), - nw.col("clm_line_thru_dt").alias("claim_line_end_date"), - nw.col("clm_from_dt").alias("admission_date"), - nw.col("clm_thru_dt").alias("discharge_date"), - nw.col("clm_admsn_src_cd").alias("admit_source_code"), - nw.col("clm_admsn_type_cd").alias("admit_type_code"), - nw.col("bene_ptnt_stus_cd").alias("discharge_disposition_code"), - null_str.alias("place_of_service_code"), - # bill_type_code = fac + clsfctn + freq - ( - nw.col("clm_bill_fac_type_cd").cast(nw.String).fill_null("") - + nw.col("clm_bill_clsfctn_cd").cast(nw.String).fill_null("") - + nw.col("clm_bill_freq_cd").cast(nw.String).fill_null("") - ).alias("bill_type_code"), - nw.lit("ms-drg").alias("drg_code_type"), - nw.col("dgns_drg_cd").cast(nw.String).str.slice(-3).alias("drg_code"), - nw.col("clm_line_prod_rev_ctr_cd").alias("revenue_center_code"), - nw.col("clm_line_srvc_unit_qty").cast(nw.Int64).alias("service_unit_quantity"), - nw.col("clm_line_hcpcs_cd").alias("hcpcs_code"), - nw.col("hcpcs_1_mdfr_cd").alias("hcpcs_modifier_1"), - nw.col("hcpcs_2_mdfr_cd").alias("hcpcs_modifier_2"), - nw.col("hcpcs_3_mdfr_cd").alias("hcpcs_modifier_3"), - nw.col("hcpcs_4_mdfr_cd").alias("hcpcs_modifier_4"), - nw.col("hcpcs_5_mdfr_cd").alias("hcpcs_modifier_5"), - nw.col("oprtg_prvdr_npi_num").alias("rendering_npi"), - null_str.alias("rendering_tin"), - nw.col("fac_prvdr_npi_num").alias("billing_npi"), - null_str.alias("billing_tin"), - nw.col("fac_prvdr_npi_num").alias("facility_npi"), - null_date.alias("paid_date"), - nw.col("clm_line_cvrd_pd_amt").cast(nw.Float64).alias("paid_amount"), - null_float.alias("allowed_amount"), - nw.col("clm_mdcr_instnl_tot_chrg_amt").cast(nw.Float64).alias("charge_amount"), - null_float.alias("coinsurance_amount"), - null_float.alias("copayment_amount"), - null_float.alias("deductible_amount"), - null_float.alias("total_cost_amount"), - _icd_indicator(), - *dx_cols, - *poa_cols, - nw.when(nw.col("dgns_prcdr_icd_ind") == "0") - .then(nw.lit("icd-10-pcs")) - .otherwise( - nw.when(nw.col("dgns_prcdr_icd_ind") == "9") - .then(nw.lit("icd-9-pcs")) - .otherwise(nw.lit(None)) - ) - .alias("procedure_code_type"), - *px_code_cols, - *px_date_cols, - nw.lit(1).alias("in_network_flag"), - nw.lit("medicare bcda").alias("data_source"), - null_str.alias("file_name"), - null_date.alias("file_date"), - nw.lit(None).cast(nw.Datetime).alias("ingest_datetime"), - ) - - -# ── Stage 3: Part B Physician Claims ──────────────────────────── - - -@nw.narwhalify -def stg_part_b_physician_header( - bcda__explanation_of_benefit: FrameT, - bcda___stg_beneficiary_xref: FrameT, -) -> FrameT: - """Filter EOB to Part B carrier claims and resolve MBI. - - Filter: nch_near_line_rec_ident_cd == 'O' - - Operations: filter, join - Dependencies: bcda.explanation_of_benefit, bcda._stg_beneficiary_xref - """ - xref = bcda___stg_beneficiary_xref - return bcda__explanation_of_benefit.filter( - nw.col("nch_near_line_rec_ident_cd") == "O" - ).join(xref, on="bene_id", how="left") + return _medical_claim(bcda___stg_part_a_header, bcda__eob_item, "institutional") @nw.narwhalify @@ -241,201 +569,30 @@ def int_physician_medical_claim( bcda___stg_part_b_physician_header: FrameT, bcda__eob_item: FrameT, ) -> FrameT: - """Join Part B physician header + line items → medical_claim. - - claim_type = 'professional'. CCLF5 equivalent. + """Carrier header ⋈ lines → medical_claim (claim_type professional). Operations: join, select Dependencies: bcda._stg_part_b_physician_header, bcda.eob_item """ - hdr = bcda___stg_part_b_physician_header - items = bcda__eob_item - - df = hdr.join(items, on="clm_id", how="inner", suffix="_item") - - # Diagnosis codes 1-12 from EOB header, 13-25 null - dx_cols = [] - for i in range(1, 26): - if i <= 12: - dx_cols.append(nw.col(f"clm_dgns_{i}_cd").alias(f"diagnosis_code_{i}")) - else: - dx_cols.append(nw.lit(None).cast(nw.String).alias(f"diagnosis_code_{i}")) - - poa_cols = [ - nw.lit(None).cast(nw.String).alias(f"diagnosis_poa_{i}") for i in range(1, 26) - ] - px_code_cols = [ - nw.lit(None).cast(nw.String).alias(f"procedure_code_{i}") for i in range(1, 26) - ] - px_date_cols = [ - nw.lit(None).cast(nw.Date).alias(f"procedure_date_{i}") for i in range(1, 26) - ] - - null_str = nw.lit(None).cast(nw.String) - null_date = nw.lit(None).cast(nw.Date) - null_float = nw.lit(None).cast(nw.Float64) - - return df.select( - nw.col("clm_id").alias("claim_id"), - nw.col("clm_line_num").cast(nw.Int64).alias("claim_line_number"), - nw.lit("professional").alias("claim_type"), - nw.col("current_bene_mbi_id").alias("person_id"), - nw.col("current_bene_mbi_id").alias("member_id"), - nw.lit("medicare").alias("payer"), - nw.lit("medicare").alias("plan"), - nw.col("clm_from_dt").alias("claim_start_date"), - nw.col("clm_thru_dt").alias("claim_end_date"), - nw.col("clm_line_from_dt").alias("claim_line_start_date"), - nw.col("clm_line_thru_dt").alias("claim_line_end_date"), - null_date.alias("admission_date"), - null_date.alias("discharge_date"), - null_str.alias("admit_source_code"), - null_str.alias("admit_type_code"), - null_str.alias("discharge_disposition_code"), - nw.col("clm_pos_cd").alias("place_of_service_code"), - null_str.alias("bill_type_code"), - null_str.alias("drg_code_type"), - null_str.alias("drg_code"), - null_str.alias("revenue_center_code"), - nw.col("clm_line_srvc_unit_qty").cast(nw.Int64).alias("service_unit_quantity"), - nw.col("clm_line_hcpcs_cd").alias("hcpcs_code"), - nw.col("hcpcs_1_mdfr_cd").alias("hcpcs_modifier_1"), - nw.col("hcpcs_2_mdfr_cd").alias("hcpcs_modifier_2"), - nw.col("hcpcs_3_mdfr_cd").alias("hcpcs_modifier_3"), - nw.col("hcpcs_4_mdfr_cd").alias("hcpcs_modifier_4"), - nw.col("hcpcs_5_mdfr_cd").alias("hcpcs_modifier_5"), - nw.col("rndrg_prvdr_npi_num").alias("rendering_npi"), - nw.col("clm_rndrg_prvdr_tax_num").alias("rendering_tin"), - nw.col("carr_clm_blg_npi_num").alias("billing_npi"), - null_str.alias("billing_tin"), - null_str.alias("facility_npi"), - null_date.alias("paid_date"), - nw.col("clm_line_cvrd_pd_amt").cast(nw.Float64).alias("paid_amount"), - nw.col("clm_line_alowd_chrg_amt").cast(nw.Float64).alias("allowed_amount"), - null_float.alias("charge_amount"), - null_float.alias("coinsurance_amount"), - null_float.alias("copayment_amount"), - null_float.alias("deductible_amount"), - null_float.alias("total_cost_amount"), - _icd_indicator(), - *dx_cols, - *poa_cols, - null_str.alias("procedure_code_type"), - *px_code_cols, - *px_date_cols, - nw.lit(1).alias("in_network_flag"), - nw.lit("medicare bcda").alias("data_source"), - null_str.alias("file_name"), - null_date.alias("file_date"), - nw.lit(None).cast(nw.Datetime).alias("ingest_datetime"), + return _medical_claim( + bcda___stg_part_b_physician_header, bcda__eob_item, "professional" ) -# ── Stage 4: Part B DME Claims ────────────────────────────────── - - -@nw.narwhalify -def stg_part_b_dme_header( - bcda__explanation_of_benefit: FrameT, - bcda___stg_beneficiary_xref: FrameT, -) -> FrameT: - """Filter EOB to Part B DME claims and resolve MBI. - - Filter: nch_near_line_rec_ident_cd == 'M' - - Operations: filter, join - Dependencies: bcda.explanation_of_benefit, bcda._stg_beneficiary_xref - """ - xref = bcda___stg_beneficiary_xref - return bcda__explanation_of_benefit.filter( - nw.col("nch_near_line_rec_ident_cd") == "M" - ).join(xref, on="bene_id", how="left") - - @nw.narwhalify def int_dme_medical_claim( bcda___stg_part_b_dme_header: FrameT, bcda__eob_item: FrameT, ) -> FrameT: - """Join Part B DME header + line items → medical_claim. + """DME header ⋈ lines → medical_claim (claim_type professional). - claim_type = 'professional'. CCLF6 equivalent. - DME claims have no inline diagnosis or procedure codes. + v3 DME claims carry diagnoses, modifiers and units like carrier + claims; v2 DME rows had none and keep their NULLs. Operations: join, select Dependencies: bcda._stg_part_b_dme_header, bcda.eob_item """ - hdr = bcda___stg_part_b_dme_header - items = bcda__eob_item - - df = hdr.join(items, on="clm_id", how="inner", suffix="_item") - - null_str = nw.lit(None).cast(nw.String) - null_date = nw.lit(None).cast(nw.Date) - null_float = nw.lit(None).cast(nw.Float64) - - dx_cols = [null_str.alias(f"diagnosis_code_{i}") for i in range(1, 26)] - poa_cols = [null_str.alias(f"diagnosis_poa_{i}") for i in range(1, 26)] - px_code_cols = [null_str.alias(f"procedure_code_{i}") for i in range(1, 26)] - px_date_cols = [null_date.alias(f"procedure_date_{i}") for i in range(1, 26)] - - return df.select( - nw.col("clm_id").alias("claim_id"), - nw.col("clm_line_num").cast(nw.Int64).alias("claim_line_number"), - nw.lit("professional").alias("claim_type"), - nw.col("current_bene_mbi_id").alias("person_id"), - nw.col("current_bene_mbi_id").alias("member_id"), - nw.lit("medicare").alias("payer"), - nw.lit("medicare").alias("plan"), - nw.col("clm_from_dt").alias("claim_start_date"), - nw.col("clm_thru_dt").alias("claim_end_date"), - nw.col("clm_line_from_dt").alias("claim_line_start_date"), - nw.col("clm_line_thru_dt").alias("claim_line_end_date"), - null_date.alias("admission_date"), - null_date.alias("discharge_date"), - null_str.alias("admit_source_code"), - null_str.alias("admit_type_code"), - null_str.alias("discharge_disposition_code"), - nw.col("clm_pos_cd").alias("place_of_service_code"), - null_str.alias("bill_type_code"), - null_str.alias("drg_code_type"), - null_str.alias("drg_code"), - null_str.alias("revenue_center_code"), - nw.lit(None).cast(nw.Int64).alias("service_unit_quantity"), - nw.col("clm_line_hcpcs_cd").alias("hcpcs_code"), - null_str.alias("hcpcs_modifier_1"), - null_str.alias("hcpcs_modifier_2"), - null_str.alias("hcpcs_modifier_3"), - null_str.alias("hcpcs_modifier_4"), - null_str.alias("hcpcs_modifier_5"), - nw.col("ordrg_prvdr_npi_num").alias("rendering_npi"), - null_str.alias("rendering_tin"), - nw.col("payto_prvdr_npi_num").alias("billing_npi"), - null_str.alias("billing_tin"), - null_str.alias("facility_npi"), - null_date.alias("paid_date"), - nw.col("clm_line_cvrd_pd_amt").cast(nw.Float64).alias("paid_amount"), - nw.col("clm_line_alowd_chrg_amt").cast(nw.Float64).alias("allowed_amount"), - null_float.alias("charge_amount"), - null_float.alias("coinsurance_amount"), - null_float.alias("copayment_amount"), - null_float.alias("deductible_amount"), - null_float.alias("total_cost_amount"), - null_str.alias("diagnosis_code_type"), - *dx_cols, - *poa_cols, - null_str.alias("procedure_code_type"), - *px_code_cols, - *px_date_cols, - nw.lit(1).alias("in_network_flag"), - nw.lit("medicare bcda").alias("data_source"), - null_str.alias("file_name"), - null_date.alias("file_date"), - nw.lit(None).cast(nw.Datetime).alias("ingest_datetime"), - ) - - -# ── Stage 5: Union medical claims ─────────────────────────────── + return _medical_claim(bcda___stg_part_b_dme_header, bcda__eob_item, "dme") @nw.narwhalify @@ -446,42 +603,24 @@ def medical_claim( ) -> FrameT: """Union institutional + professional + DME into medical_claim. + The three frames come from the same select, so the schemas already + agree column for column. + Operations: concat Dependencies: bcda._int_institutional_medical_claim, - bcda._int_physician_medical_claim, - bcda._int_dme_medical_claim + bcda._int_physician_medical_claim, bcda._int_dme_medical_claim """ - frames = [ - bcda___int_institutional_medical_claim, - bcda___int_physician_medical_claim, - bcda___int_dme_medical_claim, - ] - # Cast all columns to String to align schemas — null-only - # columns may be inferred as Int32 by the backend engine. - cols = frames[0].collect_schema().names() - aligned = [f.with_columns(nw.col(c).cast(nw.String) for c in cols) for f in frames] - return nw.concat(aligned) + return nw.concat( + [ + bcda___int_institutional_medical_claim, + bcda___int_physician_medical_claim, + bcda___int_dme_medical_claim, + ], + how="vertical", + ) -# ── Stage 6: Part D Pharmacy Claims ───────────────────────────── - - -@nw.narwhalify -def stg_part_d_header( - bcda__explanation_of_benefit: FrameT, - bcda___stg_beneficiary_xref: FrameT, -) -> FrameT: - """Filter EOB to Part D pharmacy claims and resolve MBI. - - Filter: nch_near_line_rec_ident_cd == 'P' - - Operations: filter, join - Dependencies: bcda.explanation_of_benefit, bcda._stg_beneficiary_xref - """ - xref = bcda___stg_beneficiary_xref - return bcda__explanation_of_benefit.filter( - nw.col("nch_near_line_rec_ident_cd") == "P" - ).join(xref, on="bene_id", how="left") +# ── Stage 4: pharmacy_claim ───────────────────────────────────── @nw.narwhalify @@ -489,26 +628,28 @@ def pharmacy_claim( bcda___stg_part_d_header: FrameT, bcda__eob_item: FrameT, ) -> FrameT: - """Join Part D header + line items → pharmacy_claim. + """Part D header ⋈ line → input_layer.pharmacy_claim. - CCLF7 equivalent. BCDA FHIR careTeam already resolves - provider identifiers to NPI type, so no qualifier check - is needed (unlike CCLF7 which requires - prvdr_srvc_id_qlfyr_cd == '01'). + v3: the claim id is the PDE id (same value as the claim id), the + prescriber is the ``prescribing`` care-team NPI, the dispensing + pharmacy is the contained billing Organization's NPI, quantity is + the NDC quantity, and amounts come from the line with the header + totals as fallback (the sandbox omits line-level plan-paid). + Charge is total drug cost; copayment is paid-by-patient. Operations: join, select Dependencies: bcda._stg_part_d_header, bcda.eob_item """ - hdr = bcda___stg_part_d_header - items = bcda__eob_item - - df = hdr.join(items, on="clm_id", how="inner", suffix="_item") - + hdr = _ensure(bcda___stg_part_d_header, _HEADER_COLS) + lines = _dedupe_lines(_ensure(bcda__eob_item, _ITEM_COLS)) + df = hdr.join(lines, on="clm_id", how="inner", suffix="_item") null_float = nw.lit(None).cast(nw.Float64) return df.select( - nw.col("pde_id").alias("claim_id"), - nw.lit(1).alias("claim_line_number"), + nw.coalesce(nw.col("pde_id"), nw.col("clm_id")).alias("claim_id"), + nw.coalesce(nw.col("clm_line_num"), nw.lit(1.0)) + .cast(nw.Int64) + .alias("claim_line_number"), nw.col("current_bene_mbi_id").alias("person_id"), nw.col("current_bene_mbi_id").alias("member_id"), nw.lit("medicare").alias("payer"), @@ -516,20 +657,38 @@ def pharmacy_claim( nw.col("clm_prsbng_prvdr_gnrc_id_num") .cast(nw.String) .alias("prescribing_provider_npi"), - nw.col("clm_srvc_prvdr_gnrc_id_num") + nw.coalesce( + nw.col("fac_prvdr_npi_num"), + nw.col("carr_clm_blg_npi_num"), + nw.col("clm_srvc_prvdr_gnrc_id_num"), + ) .cast(nw.String) .alias("dispensing_provider_npi"), - nw.col("clm_line_from_dt").alias("dispensing_date"), + nw.coalesce(nw.col("clm_line_from_dt"), nw.col("clm_from_dt")).alias( + "dispensing_date" + ), nw.col("clm_line_ndc_cd").alias("ndc_code"), - nw.col("clm_line_srvc_unit_qty").cast(nw.Int64).alias("quantity"), - nw.col("clm_line_days_suply_qty").cast(nw.Int64).alias("days_supply"), - nw.col("clm_line_rx_fill_num").cast(nw.Int64).alias("refills"), - nw.col("clm_line_from_dt").alias("paid_date"), - nw.col("clm_line_bene_pmt_amt").cast(nw.Float64).alias("paid_amount"), + nw.coalesce(nw.col("clm_line_ndc_qty"), nw.col("clm_line_srvc_unit_qty")) + .cast(nw.Float64) + .round(0) + .cast(nw.Int64) + .alias("quantity"), + _int_qty("clm_line_days_suply_qty").alias("days_supply"), + _int_qty("clm_line_rx_fill_num").alias("refills"), + nw.coalesce( + nw.col("clm_pd_dt"), nw.col("clm_line_from_dt"), nw.col("clm_from_dt") + ).alias("paid_date"), + nw.coalesce(nw.col("clm_line_cvrd_pd_amt"), nw.col("clm_pmt_amt")) + .cast(nw.Float64) + .alias("paid_amount"), null_float.alias("allowed_amount"), - null_float.alias("charge_amount"), + nw.coalesce(nw.col("tot_rx_cst_amt_item"), nw.col("tot_rx_cst_amt")) + .cast(nw.Float64) + .alias("charge_amount"), null_float.alias("coinsurance_amount"), - nw.col("clm_line_bene_pmt_amt").cast(nw.Float64).alias("copayment_amount"), + nw.coalesce(nw.col("clm_line_bene_pmt_amt"), nw.col("clm_bene_pmt_amt")) + .cast(nw.Float64) + .alias("copayment_amount"), null_float.alias("deductible_amount"), nw.lit(1).alias("in_network_flag"), nw.lit("medicare bcda").alias("data_source"), @@ -539,7 +698,36 @@ def pharmacy_claim( ) -# ── Stage 7: Eligibility ──────────────────────────────────────── +# ── Stage 5: Eligibility ──────────────────────────────────────── + +_PATIENT_COLS: dict[str, Any] = { + "bene_id": nw.String, + "bene_mbi_id": nw.String, + "bene_sex_cd": nw.String, + "bene_race_cd": nw.String, + "bene_dob": nw.Date, + "bene_death_dt": nw.Date, + "bene_1st_name": nw.String, + "bene_midl_name": nw.String, + "bene_last_name": nw.String, + "bene_line_1_adr": nw.String, + "geo_zip_plc_name": nw.String, + "geo_usps_state_cd": nw.String, + "geo_zip5_cd": nw.String, +} +_COVERAGE_COLS: dict[str, Any] = { + "subscriber_id": nw.String, + "class_plan": nw.String, + "status": nw.String, + "ms_cd": nw.String, + "orec": nw.String, + "dual_01": nw.String, + "dual_stus_cd": nw.String, + "bene_part_a_enrlmt_bgn_dt": nw.Date, + "mdcr_covg_a_end_dt": nw.Date, + "bene_part_b_enrlmt_bgn_dt": nw.Date, + "mdcr_covg_b_end_dt": nw.Date, +} @nw.narwhalify @@ -547,110 +735,95 @@ def stg_beneficiary_demographics( bcda__patient: FrameT, bcda__coverage: FrameT, ) -> FrameT: - """Join Patient + Coverage for demographics and enrollment. + """Patient ⋈ Coverage: demographics plus the enrollment row. - Patient provides: name, dob, sex, race, death_dt, address - Coverage provides: ms_cd, orec, enrollment periods + Coverage has several rows per beneficiary (Part A, B, D in v2; + Part A, B, C, D and a Dual Medicare/Medicaid row per period in v3). + The Part A row carries Medicare status, entitlement reason and the + enrollment period, with Part B as fallback; the v3 Dual row is + joined separately for ``dual_stus_cd``. The output is an explicit + column list (the ``_stg_beneficiary_demographics`` contract). - Join on bene_mbi_id = subscriber_id. Coverage has multiple rows - per beneficiary (Part A, B, D in v2; Part A, B, C, D and a Dual - Medicare/Medicaid row per period in v3). The Part A row is the - demographics row — it carries ms_cd / orec / entitlement — with - Part B as the fallback; the v3 Dual row is joined separately for - ``dual_stus_cd`` (v2 keeps ``dual_01`` on the Part A row). - - Operations: filter, sort, unique, join + Operations: filter, sort, unique, join, select Dependencies: bcda.patient, bcda.coverage """ - cols = bcda__coverage.columns - parts = bcda__coverage - if "class_plan" in cols: - parts = parts.filter(nw.col("class_plan").is_in(["Part A", "Part B"])).sort( - "class_plan" # "Part A" < "Part B" + patient = _ensure(bcda__patient, _PATIENT_COLS) + coverage = _ensure(bcda__coverage, _COVERAGE_COLS) + + parts = ( + coverage.filter( + nw.col("class_plan").is_null() + | nw.col("class_plan").is_in(["Part A", "Part B"]) + ) + .sort("class_plan") # "Part A" < "Part B" + .unique(subset=["subscriber_id"], keep="first", maintain_order=True) + .select( + "subscriber_id", + nw.col("class_plan").alias("coverage_plan"), + nw.col("status").alias("coverage_status"), + "ms_cd", + "orec", + "dual_01", + nw.coalesce( + nw.col("bene_part_a_enrlmt_bgn_dt"), nw.col("bene_part_b_enrlmt_bgn_dt") + ).alias("enrollment_start_date"), + nw.coalesce( + nw.col("mdcr_covg_a_end_dt"), nw.col("mdcr_covg_b_end_dt") + ).alias("enrollment_end_date"), ) - parts = parts.unique(subset=["subscriber_id"], keep="first", maintain_order=True) - joined = bcda__patient.join( - parts, - left_on="bene_mbi_id", - right_on="subscriber_id", - how="left", - suffix="_cov", ) - if "dual_stus_cd" in cols and "class_plan" in cols: - dual = ( - bcda__coverage.filter(nw.col("class_plan") == "Dual Medicare/Medicaid") - .select("subscriber_id", nw.col("dual_stus_cd").alias("dual_stus_cd_dual")) - .unique(subset=["subscriber_id"], keep="any") - ) - joined = joined.join( - dual, - left_on="bene_mbi_id", - right_on="subscriber_id", - how="left", - suffix="_dual", - ) - return joined.unique(subset=["bene_mbi_id"], keep="any") + dual = ( + coverage.filter(nw.col("class_plan") == "Dual Medicare/Medicaid") + .select("subscriber_id", nw.col("dual_stus_cd").alias("dual_stus_cd_dual")) + .unique(subset=["subscriber_id"], keep="any") + ) + return ( + patient.select(*_PATIENT_COLS) + .join(parts, left_on="bene_mbi_id", right_on="subscriber_id", how="left") + .join(dual, left_on="bene_mbi_id", right_on="subscriber_id", how="left") + .unique(subset=["bene_mbi_id"], keep="any") + ) @nw.narwhalify def eligibility( bcda___stg_beneficiary_demographics: FrameT, ) -> FrameT: - """Map Patient + Coverage to eligibility schema. + """Demographics → input_layer.eligibility. - Gender crosswalk: 1→male, 2→female, else unknown - Race crosswalk: 1→white, 2→black, 3→other, 4→asian, - 5→hispanic, 6→native, else unknown + Gender: CCW ``1``/``2`` (v3, and v2 after the flattener's crosswalk) + or the FHIR ``male``/``female`` the v2 flattener used to pass + through. Race: CCW codes 1–6. Dual status: v3 Dual row, else the + v2 January slot. Enrollment status from the Part A/B Coverage + status (``active`` / ``cancelled``). Operations: select Dependencies: bcda._stg_beneficiary_demographics """ df = bcda___stg_beneficiary_demographics null_str = nw.lit(None).cast(nw.String) - # v2 keeps dual status on the Part A row (dual_01); v3 carries it on - # the Dual Medicare/Medicaid row that stg_beneficiary_demographics - # joins in as dual_stus_cd_dual. - dual = nw.col("dual_01") - if "dual_stus_cd_dual" in df.columns: - dual = nw.coalesce(nw.col("dual_01"), nw.col("dual_stus_cd_dual")) - - gender = ( - nw.when(nw.col("bene_sex_cd") == "1") - .then(nw.lit("male")) - .otherwise( - nw.when(nw.col("bene_sex_cd") == "2") - .then(nw.lit("female")) - .otherwise(nw.lit("unknown")) - ) - .alias("gender") - ) - - race = ( - nw.when(nw.col("bene_race_cd") == "1") - .then(nw.lit("white")) - .otherwise( - nw.when(nw.col("bene_race_cd") == "2") - .then(nw.lit("black")) - .otherwise( - nw.when(nw.col("bene_race_cd") == "3") - .then(nw.lit("other")) - .otherwise( - nw.when(nw.col("bene_race_cd") == "4") - .then(nw.lit("asian")) - .otherwise( - nw.when(nw.col("bene_race_cd") == "5") - .then(nw.lit("hispanic")) - .otherwise( - nw.when(nw.col("bene_race_cd") == "6") - .then(nw.lit("native")) - .otherwise(nw.lit("unknown")) - ) - ) - ) - ) - ) - .alias("race") - ) + sex = nw.col("bene_sex_cd").cast(nw.String) + gender = _case( + (sex.is_in(["1", "male"]), nw.lit("male")), + (sex.is_in(["2", "female"]), nw.lit("female")), + default=nw.lit("unknown"), + ).alias("gender") + race_cd = nw.col("bene_race_cd").cast(nw.String) + race = _case( + (race_cd == "1", nw.lit("white")), + (race_cd == "2", nw.lit("black")), + (race_cd == "3", nw.lit("other")), + (race_cd == "4", nw.lit("asian")), + (race_cd == "5", nw.lit("hispanic")), + (race_cd == "6", nw.lit("native")), + default=nw.lit("unknown"), + ).alias("race") + status = nw.col("coverage_status").cast(nw.String) + enrollment_status = _case( + (status == "active", nw.lit("active")), + (status == "cancelled", nw.lit("inactive")), + default=null_str, + ).alias("enrollment_status") return df.select( nw.col("bene_mbi_id").alias("person_id"), @@ -664,13 +837,15 @@ def eligibility( .then(nw.lit(1)) .otherwise(nw.lit(0)) .alias("death_flag"), - nw.col("bene_part_a_enrlmt_bgn_dt").alias("enrollment_start_date"), - nw.col("mdcr_covg_a_end_dt").alias("enrollment_end_date"), + nw.col("enrollment_start_date"), + nw.col("enrollment_end_date"), nw.lit("medicare").alias("payer"), nw.lit("medicare").alias("payer_type"), nw.lit("medicare").alias("plan"), nw.col("orec").alias("original_reason_entitlement_code"), - dual.alias("dual_status_code"), + nw.coalesce(nw.col("dual_stus_cd_dual"), nw.col("dual_01")).alias( + "dual_status_code" + ), nw.col("ms_cd").alias("medicare_status_code"), null_str.alias("group_id"), null_str.alias("group_name"), @@ -686,7 +861,10 @@ def eligibility( nw.col("geo_zip5_cd").alias("zip_code"), null_str.alias("phone"), null_str.alias("email"), - null_str.alias("ethnicity"), + nw.when(race_cd == "5") + .then(nw.lit("hispanic")) + .otherwise(null_str) + .alias("ethnicity"), nw.lit("medicare bcda").alias("data_source"), nw.lit(None).cast(nw.String).alias("file_name"), nw.lit(None).cast(nw.Date).alias("file_date"), @@ -694,5 +872,5 @@ def eligibility( nw.lit(None).cast(nw.Int64).alias("hospice_flag"), nw.lit(None).cast(nw.Int64).alias("institutional_snp_flag"), nw.lit(None).cast(nw.Int64).alias("long_term_institutional_flag"), - null_str.alias("enrollment_status"), + enrollment_status, ) diff --git a/src/bcda/express/v3.py b/src/bcda/express/v3.py index d4945f1..614f225 100644 --- a/src/bcda/express/v3.py +++ b/src/bcda/express/v3.py @@ -69,6 +69,7 @@ NDC = "http://hl7.org/fhir/sid/ndc" HIPPS = "HIPPSCodes" # https://www.cms.gov/.../ProspMedicareFeeSvcPmtGen/HIPPSCodes HCPCS_SYSTEMS = ("http://www.ama-assn.org/go/cpt", "HCPCSReleaseCodeSets") POS = "Place_of_Service_Code_Set" +NUBC_TOB = "https://www.nubc.org/CodeSystem/TypeOfBill" # CCW BENE_RACE_CD from the US Core OMB race/ethnicity codes # (CCW codebook: 0 Unknown, 1 White, 2 Black, 3 Other, 4 Asian/Pacific @@ -1070,6 +1071,7 @@ def flatten_eob(resource: dict) -> tuple[ExplanationOfBenefit, list[EobItem]]: if v is not None: for col in cols: kwargs[col] = v + kwargs["clm_bill_type_cd"] = _coding(tob.get("code"), NUBC_TOB) if kwargs.get("bene_ptnt_stus_cd") is None: kwargs["bene_ptnt_stus_cd"] = _si_code_by_system( infos, CS + "CLM-MDCR-NCH-PTNT-STUS-IND-CD" diff --git a/src/bcda/pipe/cclf.py b/src/bcda/pipe/cclf.py index e4e4481..4b68131 100644 --- a/src/bcda/pipe/cclf.py +++ b/src/bcda/pipe/cclf.py @@ -1,30 +1,32 @@ """BCDA FHIR R4 connector pipeline — EOB + Patient + Coverage → input_layer. -Transforms flattened BCDA FHIR R4 resources into three consolidated -output tables: - - input_layer.medical_claim (Part A + Part B carrier + Part B DME) +Transforms the flattened BCDA tables (v2 or v3 exports) into the three +input_layer tables on the canonical ``aco.table.input_layer`` contracts: + - input_layer.medical_claim (institutional + professional + DME) - input_layer.pharmacy_claim (Part D) - input_layer.eligibility (Patient + Coverage) -Pipeline expressions (12 total):: +Pipeline expressions (13 total):: - stg_beneficiary_xref Patient → MBI identity - stg_part_a_header EOB(V) + xref + stg_beneficiary_xref Patient → current MBI + stg_claim_header EOB → final action, claim family, + cross-source dedupe, MBI + stg_part_a_header family = institutional int_institutional_medical_claim header + items → medical_claim - stg_part_b_physician_header EOB(O) + xref + stg_part_b_physician_header family = professional int_physician_medical_claim header + items → medical_claim - stg_part_b_dme_header EOB(M) + xref + stg_part_b_dme_header family = dme int_dme_medical_claim header + items → medical_claim - medical_claim union inst+phys+dme - stg_part_d_header EOB(P) + xref + medical_claim union inst + phys + dme + stg_part_d_header family = pharmacy pharmacy_claim Part D → pharmacy_claim - stg_beneficiary_demographics Patient + Coverage + stg_beneficiary_demographics Patient + Coverage (Part A row, Dual row) eligibility demographics → eligibility -Source data — BCDA bulk FHIR export (v2 or v3, flattened to bcda.*): - - ExplanationOfBenefit (header + items) - - Patient - - Coverage +v3 semantics (routing on EOB-TYPE / CLM-TYPE-CD, Final-Action filter, +SharedSystem→NationalClaimsHistory dedupe by claim control number, +consolidated line amounts, admission period, NUBC type of bill, Coverage +period per plan) are documented in ``bcda.express.cclf``. """ from __future__ import annotations @@ -55,6 +57,7 @@ def _build_pipeline(): BcdaPharmacyClaim, BcdaStgBeneficiaryDemographics, BcdaStgBeneficiaryXref, + BcdaStgClaimHeader, BcdaStgPartAHeader, BcdaStgPartBDmeHeader, BcdaStgPartBPhysicianHeader, @@ -76,26 +79,42 @@ def _build_pipeline(): after=["bcda.patient"], refs=_REFS, description=( - "Deduplicates BCDA Patient to one row per " - "beneficiary with the current MBI. Unlike the " - "CCLF pipeline, no CCLF9-style crosswalk is " - "needed — BCDA delivers current MBIs directly." + "One row per beneficiary with the current MBI. v2 " + "delivers it in bene_mbi_id; v3's flattener picks the " + "us-mbi identifier without a period end. bene_id is " + "the CCW BENE_ID (v2) or the IDR BENE_SK (v3)." ), ), - # -- Part A institutional claims -------------------------- + # -- Claim header: final action, family, dedupe ----------- Expr( - name="bcda._stg_part_a_header", - fn=ex.stg_part_a_header, - output=BcdaStgPartAHeader, + name="bcda._stg_claim_header", + fn=ex.stg_claim_header, + output=BcdaStgClaimHeader, after=[ "bcda.explanation_of_benefit", "bcda._stg_beneficiary_xref", ], refs=_REFS, description=( - "Filters EOB to Part A institutional claims " - "(nch_near_line_rec_ident_cd='V') and resolves " - "beneficiary MBI via the xref." + "Drops NotFinalAction claims (v3 Final-Action tag), " + "routes each EOB to institutional / professional / dme " + "/ pharmacy from the v3 EOB-TYPE and CLM-TYPE-CD " + "codings (v2 near-line RIC as last resort), collapses " + "a claim seen in two sources by claim control number " + "(NationalClaimsHistory over SharedSystem, newest " + "last), and resolves the current MBI." + ), + ), + # -- Institutional claims --------------------------------- + Expr( + name="bcda._stg_part_a_header", + fn=ex.stg_part_a_header, + output=BcdaStgPartAHeader, + after=["bcda._stg_claim_header"], + refs=_REFS, + description=( + "Institutional claims: EOB-TYPE INPATIENT, OUTPATIENT, " + "SNF, HHA, HOSPICE (CLM-TYPE-CD 10–64)." ), ), Expr( @@ -105,42 +124,35 @@ def _build_pipeline(): after=["bcda._stg_part_a_header", "bcda.eob_item"], refs=_REFS, description=( - "Joins Part A header + line items into " - "institutional medical_claim rows with bill type, " - "DRG, revenue center, 12 inline diagnosis codes, " - "and 1 procedure code from the EOB header." + "Institutional header + revenue-centre lines → " + "medical_claim: admission period, NUBC type of bill, " + "MS-DRG, revenue centre, 12 header diagnoses with POA " + "on the first, first ICD procedure, line paid / charge " + "/ coinsurance / deductible from the v3 Adjudication " + "amounts, attending→operating→rendering NPI, billing " + "Organization NPI and tax number." ), ), - # -- Part B physician claims ------------------------------ + # -- Part B carrier claims -------------------------------- Expr( name="bcda._stg_part_b_physician_header", fn=ex.stg_part_b_physician_header, output=BcdaStgPartBPhysicianHeader, - after=[ - "bcda.explanation_of_benefit", - "bcda._stg_beneficiary_xref", - ], + after=["bcda._stg_claim_header"], refs=_REFS, - description=( - "Filters EOB to Part B carrier/physician claims " - "(nch_near_line_rec_ident_cd='O') and resolves " - "beneficiary MBI." - ), + description="Part B carrier claims: EOB-TYPE CARRIER (CLM-TYPE-CD 71, 72).", ), Expr( name="bcda._int_physician_medical_claim", fn=ex.int_physician_medical_claim, output=BcdaIntPhysicianMedicalClaim, - after=[ - "bcda._stg_part_b_physician_header", - "bcda.eob_item", - ], + after=["bcda._stg_part_b_physician_header", "bcda.eob_item"], refs=_REFS, description=( - "Joins Part B physician header + line items into " - "professional medical_claim rows with place of " - "service, rendering NPI/TIN, and 12 inline " - "diagnosis codes." + "Carrier header + lines → professional medical_claim: " + "place of service, HCPCS + modifiers, rendering NPI and " + "tax number, billing NPI, line paid / allowed / " + "submitted / coinsurance / deductible." ), ), # -- Part B DME claims ------------------------------------ @@ -148,16 +160,9 @@ def _build_pipeline(): name="bcda._stg_part_b_dme_header", fn=ex.stg_part_b_dme_header, output=BcdaStgPartBDmeHeader, - after=[ - "bcda.explanation_of_benefit", - "bcda._stg_beneficiary_xref", - ], + after=["bcda._stg_claim_header"], refs=_REFS, - description=( - "Filters EOB to Part B DME claims " - "(nch_near_line_rec_ident_cd='M') and resolves " - "beneficiary MBI." - ), + description="Part B DME claims: EOB-TYPE DME (CLM-TYPE-CD 81, 82).", ), Expr( name="bcda._int_dme_medical_claim", @@ -166,12 +171,12 @@ def _build_pipeline(): after=["bcda._stg_part_b_dme_header", "bcda.eob_item"], refs=_REFS, description=( - "Joins Part B DME header + line items into " - "professional medical_claim rows. DME claims " - "carry no diagnosis or procedure codes." + "DME header + lines → professional medical_claim; " + "rendering NPI falls back to the ordering provider, " + "billing NPI to the pay-to provider." ), ), - # -- Final medical_claim (union) -------------------------- + # -- Union -------------------------------------------------- Expr( name="bcda.medical_claim", fn=ex.medical_claim, @@ -183,25 +188,18 @@ def _build_pipeline(): ], refs=_REFS, description=( - "Unions institutional + professional + DME into " - "the final medical_claim table." + "Union of institutional, carrier and DME lines on the " + "input_layer.medical_claim contract." ), ), - # -- Part D pharmacy claims ------------------------------- + # -- Part D ---------------------------------------------- Expr( name="bcda._stg_part_d_header", fn=ex.stg_part_d_header, output=BcdaStgPartDHeader, - after=[ - "bcda.explanation_of_benefit", - "bcda._stg_beneficiary_xref", - ], + after=["bcda._stg_claim_header"], refs=_REFS, - description=( - "Filters EOB to Part D pharmacy claims " - "(nch_near_line_rec_ident_cd='P') and resolves " - "beneficiary MBI." - ), + description="Part D claims: EOB-TYPE PDE (CLM-TYPE-CD 1–4).", ), Expr( name="bcda.pharmacy_claim", @@ -210,12 +208,14 @@ def _build_pipeline(): after=["bcda._stg_part_d_header", "bcda.eob_item"], refs=_REFS, description=( - "Joins Part D header + line items into " - "pharmacy_claim with NDC, quantity, days supply, " - "prescribing and dispensing provider NPIs." + "Part D header + line → pharmacy_claim: PDE id, " + "prescribing care-team NPI, dispensing pharmacy NPI " + "(contained billing Organization), NDC and NDC " + "quantity, days supply, fill number, total drug cost " + "as charge, paid-by-patient as copayment." ), ), - # -- Eligibility ------------------------------------------ + # -- Eligibility ------------------------------------------- Expr( name="bcda._stg_beneficiary_demographics", fn=ex.stg_beneficiary_demographics, @@ -223,9 +223,10 @@ def _build_pipeline(): after=["bcda.patient", "bcda.coverage"], refs=_REFS, description=( - "Joins Patient + Coverage for demographics and " - "enrollment data. Deduplicates to one row per " - "beneficiary MBI." + "Patient joined to its Part A Coverage row (Part B " + "fallback) for Medicare status, entitlement reason, " + "enrollment period and status, plus the v3 Dual " + "Medicare/Medicaid row for dual status." ), ), Expr( @@ -235,8 +236,8 @@ def _build_pipeline(): after=["bcda._stg_beneficiary_demographics"], refs=_REFS, description=( - "Maps Patient + Coverage to the eligibility " - "schema with gender and race crosswalks." + "Demographics → input_layer.eligibility with gender, " + "race and enrollment-status crosswalks." ), ), ] diff --git a/src/bcda/table/cclf_pipe.py b/src/bcda/table/cclf_pipe.py index 0069a3f..cfec817 100644 --- a/src/bcda/table/cclf_pipe.py +++ b/src/bcda/table/cclf_pipe.py @@ -1,383 +1,175 @@ -"""SQLTable contracts for the BCDA FHIR R4 connector pipeline intermediates. +"""SQLTable contracts for the BCDA FHIR R4 connector pipeline. -Each class corresponds to one ``Expr`` in ``bcda.pipe.cclf`` and documents -the output schema produced by the matching ``bcda.express.cclf`` function. +Each class is the output contract of one ``Expr`` in ``bcda.pipe.cclf`` +and is checked column-for-column by ``aco.pipe.runner`` against what +the matching ``bcda.express.cclf`` function returns. + +The contracts inherit rather than restate: + +- the claim-header stages pass every ``bcda.explanation_of_benefit`` + column through plus ``claim_family`` and the current MBI, so they + subclass :class:`bcda.table.eob.ExplanationOfBenefit`; +- the three outputs are the canonical Tuva-shaped input_layer tables, + so they subclass ``aco.table.input_layer.InputLayer*`` — the same + schema the CCLF pipeline lands, which is what lets the downstream + pipelines run unchanged on either source. """ from __future__ import annotations -from datetime import date, datetime +from datetime import date +from aco.table.input_layer import ( + InputLayerEligibility, + InputLayerMedicalClaim, + InputLayerPharmacyClaim, +) +from bcda.table.eob import ExplanationOfBenefit from conf.table_base import SQLTable # ── Stage 1: Beneficiary XREF (Patient) ────────────────────────────────────── class BcdaStgBeneficiaryXref(SQLTable): - """bcda._stg_beneficiary_xref — deduplicated MBI identity from BCDA Patient.""" + """bcda._stg_beneficiary_xref — one row per beneficiary with the current MBI.""" __schema__ = "bcda" __tablename__ = "_stg_beneficiary_xref" bene_id: str | None = None - """Internal BCDA beneficiary identifier.""" + """Beneficiary key: CCW BENE_ID (v2) or IDR BENE_SK (v3 Patient.id).""" current_bene_mbi_id: str | None = None - """Current Medicare Beneficiary Identifier (delivered directly by BCDA).""" + """Current Medicare Beneficiary Identifier.""" -# ── Stage 2: Part A institutional claim header ──────────────────────────────── +# ── Stage 2: Claim header — final action, family, dedupe, MBI ──────────────── -class BcdaStgPartAHeader(SQLTable): - """bcda._stg_part_a_header — EOB Part A filtered and MBI resolved.""" +class BcdaStgClaimHeader(ExplanationOfBenefit): + """bcda._stg_claim_header — final-action EOB headers tagged with a family. + + Every ``bcda.explanation_of_benefit`` column plus the two below. + """ + + __schema__ = "bcda" + __tablename__ = "_stg_claim_header" + + claim_family: str | None = None + """institutional · professional · dme · pharmacy (see + ``bcda.express.cclf._claim_family``).""" + + current_bene_mbi_id: str | None = None + """Current MBI resolved through ``_stg_beneficiary_xref``.""" + + +class BcdaStgPartAHeader(BcdaStgClaimHeader): + """bcda._stg_part_a_header — institutional claims.""" __schema__ = "bcda" __tablename__ = "_stg_part_a_header" - clm_id: str | None = None - current_bene_mbi_id: str | None = None - bene_id: str | None = None - nch_near_line_rec_ident_cd: str | None = None - clm_from_dt: date | None = None - clm_thru_dt: date | None = None - clm_admsn_dt: date | None = None - nch_bene_dschrg_dt: date | None = None - clm_admsn_src_cd: str | None = None - clm_admsn_type_cd: str | None = None - ptnt_dschrg_stus_cd: str | None = None - clm_bill_fac_type_cd: str | None = None - clm_bill_clsfctn_cd: str | None = None - clm_bill_freq_cd: str | None = None - clm_drg_cd: str | None = None - dgns_prcdr_icd_ind: str | None = None - clm_pmt_amt: float | None = None - clm_tot_chrg_amt: float | None = None - org_npi_num: str | None = None - at_physn_npi: str | None = None - op_physn_npi: str | None = None - -# ── Stage 3: Institutional medical_claim (Part A header + items) ────────────── - - -class BcdaIntInstitutionalMedicalClaim(SQLTable): - """bcda._int_institutional_medical_claim — Part A joined to medical_claim schema.""" - - __schema__ = "bcda" - __tablename__ = "_int_institutional_medical_claim" - - claim_id: str | None = None - claim_line_number: int | None = None - claim_type: str | None = None - person_id: str | None = None - member_id: str | None = None - payer: str | None = None - plan: str | None = None - claim_start_date: date | None = None - claim_end_date: date | None = None - admission_date: date | None = None - discharge_date: date | None = None - admit_source_code: str | None = None - admit_type_code: str | None = None - discharge_disposition_code: str | None = None - bill_type_code: str | None = None - drg_code_type: str | None = None - drg_code: str | None = None - revenue_center_code: str | None = None - hcpcs_code: str | None = None - rendering_npi: str | None = None - billing_npi: str | None = None - facility_npi: str | None = None - paid_amount: float | None = None - allowed_amount: float | None = None - charge_amount: float | None = None - diagnosis_code_type: str | None = None - diagnosis_code_1: str | None = None - diagnosis_code_2: str | None = None - diagnosis_code_3: str | None = None - procedure_code_type: str | None = None - procedure_code_1: str | None = None - in_network_flag: int | None = None - data_source: str | None = None - file_name: str | None = None - file_date: date | None = None - ingest_datetime: datetime | None = None - - -# ── Stage 4: Part B physician claim header ──────────────────────────────────── - - -class BcdaStgPartBPhysicianHeader(SQLTable): - """bcda._stg_part_b_physician_header — EOB Part B carrier filtered and MBI resolved.""" +class BcdaStgPartBPhysicianHeader(BcdaStgClaimHeader): + """bcda._stg_part_b_physician_header — Part B carrier claims.""" __schema__ = "bcda" __tablename__ = "_stg_part_b_physician_header" - clm_id: str | None = None - current_bene_mbi_id: str | None = None - bene_id: str | None = None - nch_near_line_rec_ident_cd: str | None = None - clm_from_dt: date | None = None - clm_thru_dt: date | None = None - clm_pmt_amt: float | None = None - clm_tot_chrg_amt: float | None = None - rndrng_physn_npi: str | None = None - org_npi_num: str | None = None - dgns_prcdr_icd_ind: str | None = None - -# ── Stage 5: Professional medical_claim (Part B physician) ─────────────────── - - -class BcdaIntPhysicianMedicalClaim(SQLTable): - """bcda._int_physician_medical_claim — Part B physician joined to medical_claim schema.""" - - __schema__ = "bcda" - __tablename__ = "_int_physician_medical_claim" - - claim_id: str | None = None - claim_line_number: int | None = None - claim_type: str | None = None - person_id: str | None = None - member_id: str | None = None - payer: str | None = None - plan: str | None = None - claim_start_date: date | None = None - claim_end_date: date | None = None - claim_line_start_date: date | None = None - claim_line_end_date: date | None = None - place_of_service_code: str | None = None - hcpcs_code: str | None = None - hcpcs_modifier_1: str | None = None - rendering_npi: str | None = None - billing_npi: str | None = None - paid_amount: float | None = None - allowed_amount: float | None = None - charge_amount: float | None = None - diagnosis_code_type: str | None = None - diagnosis_code_1: str | None = None - diagnosis_code_2: str | None = None - diagnosis_code_3: str | None = None - in_network_flag: int | None = None - data_source: str | None = None - file_name: str | None = None - file_date: date | None = None - ingest_datetime: datetime | None = None - - -# ── Stage 6: Part B DME claim header ───────────────────────────────────────── - - -class BcdaStgPartBDmeHeader(SQLTable): - """bcda._stg_part_b_dme_header — EOB Part B DME filtered and MBI resolved.""" +class BcdaStgPartBDmeHeader(BcdaStgClaimHeader): + """bcda._stg_part_b_dme_header — Part B DME claims.""" __schema__ = "bcda" __tablename__ = "_stg_part_b_dme_header" - clm_id: str | None = None - current_bene_mbi_id: str | None = None - bene_id: str | None = None - nch_near_line_rec_ident_cd: str | None = None - clm_from_dt: date | None = None - clm_thru_dt: date | None = None - clm_pmt_amt: float | None = None - clm_tot_chrg_amt: float | None = None - rndrng_physn_npi: str | None = None - org_npi_num: str | None = None - -# ── Stage 7: DME medical_claim (Part B DME) ─────────────────────────────────── - - -class BcdaIntDmeMedicalClaim(SQLTable): - """bcda._int_dme_medical_claim — Part B DME joined to medical_claim schema.""" - - __schema__ = "bcda" - __tablename__ = "_int_dme_medical_claim" - - claim_id: str | None = None - claim_line_number: int | None = None - claim_type: str | None = None - person_id: str | None = None - member_id: str | None = None - payer: str | None = None - plan: str | None = None - claim_start_date: date | None = None - claim_end_date: date | None = None - claim_line_start_date: date | None = None - claim_line_end_date: date | None = None - hcpcs_code: str | None = None - rendering_npi: str | None = None - billing_npi: str | None = None - paid_amount: float | None = None - allowed_amount: float | None = None - charge_amount: float | None = None - in_network_flag: int | None = None - data_source: str | None = None - file_name: str | None = None - file_date: date | None = None - ingest_datetime: datetime | None = None - - -# ── Stage 8: Final medical_claim (union inst + phys + dme) ─────────────────── - - -class BcdaMedicalClaim(SQLTable): - """bcda.medical_claim — unioned Part A + Part B carrier + Part B DME claims.""" - - __schema__ = "bcda" - __tablename__ = "medical_claim" - - claim_id: str | None = None - claim_line_number: int | None = None - claim_type: str | None = None - person_id: str | None = None - member_id: str | None = None - payer: str | None = None - plan: str | None = None - claim_start_date: date | None = None - claim_end_date: date | None = None - admission_date: date | None = None - discharge_date: date | None = None - admit_source_code: str | None = None - admit_type_code: str | None = None - discharge_disposition_code: str | None = None - place_of_service_code: str | None = None - bill_type_code: str | None = None - drg_code_type: str | None = None - drg_code: str | None = None - revenue_center_code: str | None = None - hcpcs_code: str | None = None - rendering_npi: str | None = None - billing_npi: str | None = None - facility_npi: str | None = None - paid_amount: float | None = None - allowed_amount: float | None = None - charge_amount: float | None = None - diagnosis_code_type: str | None = None - diagnosis_code_1: str | None = None - diagnosis_code_2: str | None = None - diagnosis_code_3: str | None = None - procedure_code_type: str | None = None - procedure_code_1: str | None = None - in_network_flag: int | None = None - data_source: str | None = None - file_name: str | None = None - file_date: date | None = None - ingest_datetime: datetime | None = None - - -# ── Stage 9: Part D pharmacy claim header ──────────────────────────────────── - - -class BcdaStgPartDHeader(SQLTable): - """bcda._stg_part_d_header — EOB Part D filtered and MBI resolved.""" +class BcdaStgPartDHeader(BcdaStgClaimHeader): + """bcda._stg_part_d_header — Part D pharmacy claims.""" __schema__ = "bcda" __tablename__ = "_stg_part_d_header" - clm_id: str | None = None - current_bene_mbi_id: str | None = None - bene_id: str | None = None - nch_near_line_rec_ident_cd: str | None = None - clm_from_dt: date | None = None - clm_thru_dt: date | None = None - clm_pmt_amt: float | None = None - clm_tot_chrg_amt: float | None = None - prscrbr_id: str | None = None - rx_srvc_rfrnc_num: str | None = None + +# ── Stage 3: medical_claim (input_layer shape) ─────────────────────────────── -# ── Stage 10: Final pharmacy_claim (Part D) ─────────────────────────────────── +class BcdaIntInstitutionalMedicalClaim(InputLayerMedicalClaim): + """bcda._int_institutional_medical_claim — institutional lines.""" + + __schema__ = "bcda" + __tablename__ = "_int_institutional_medical_claim" -class BcdaPharmacyClaim(SQLTable): - """bcda.pharmacy_claim — Part D joined to pharmacy_claim schema.""" +class BcdaIntPhysicianMedicalClaim(InputLayerMedicalClaim): + """bcda._int_physician_medical_claim — carrier lines.""" + + __schema__ = "bcda" + __tablename__ = "_int_physician_medical_claim" + + +class BcdaIntDmeMedicalClaim(InputLayerMedicalClaim): + """bcda._int_dme_medical_claim — DME lines.""" + + __schema__ = "bcda" + __tablename__ = "_int_dme_medical_claim" + + +class BcdaMedicalClaim(InputLayerMedicalClaim): + """bcda.medical_claim — union of the three families; saved to input_layer.medical_claim.""" + + __schema__ = "bcda" + __tablename__ = "medical_claim" + + +# ── Stage 4: pharmacy_claim ────────────────────────────────────────────────── + + +class BcdaPharmacyClaim(InputLayerPharmacyClaim): + """bcda.pharmacy_claim — Part D; saved to input_layer.pharmacy_claim.""" __schema__ = "bcda" __tablename__ = "pharmacy_claim" - claim_id: str | None = None - claim_line_number: int | None = None - person_id: str | None = None - member_id: str | None = None - payer: str | None = None - plan: str | None = None - prescribing_provider_npi: str | None = None - dispensing_provider_npi: str | None = None - dispensing_date: date | None = None - ndc_code: str | None = None - quantity: float | None = None - days_supply: int | None = None - refills: int | None = None - paid_date: date | None = None - paid_amount: float | None = None - allowed_amount: float | None = None - coinsurance_amount: float | None = None - copayment_amount: float | None = None - deductible_amount: float | None = None - in_network_flag: int | None = None - data_source: str | None = None - file_name: str | None = None - file_date: date | None = None - ingest_datetime: datetime | None = None - -# ── Stage 11: Beneficiary demographics staging (Patient + Coverage) ─────────── +# ── Stage 5: eligibility ───────────────────────────────────────────────────── class BcdaStgBeneficiaryDemographics(SQLTable): - """bcda._stg_beneficiary_demographics — Patient + Coverage joined and deduplicated.""" + """bcda._stg_beneficiary_demographics — Patient plus its enrollment Coverage row.""" __schema__ = "bcda" __tablename__ = "_stg_beneficiary_demographics" - current_bene_mbi_id: str | None = None bene_id: str | None = None - bene_birth_dt: date | None = None + bene_mbi_id: str | None = None bene_sex_cd: str | None = None bene_race_cd: str | None = None - bene_fips_state_cd: str | None = None - bene_county_cd: str | None = None - bene_zip_cd: str | None = None + bene_dob: date | None = None + bene_death_dt: date | None = None bene_1st_name: str | None = None + bene_midl_name: str | None = None bene_last_name: str | None = None - crnt_bene_mdcr_stus_cd: str | None = None - covr_yr_mo: str | None = None + bene_line_1_adr: str | None = None + geo_zip_plc_name: str | None = None + geo_usps_state_cd: str | None = None + geo_zip5_cd: str | None = None + coverage_plan: str | None = None + """Which Coverage row supplied the enrollment fields (Part A, else Part B).""" + coverage_status: str | None = None + """Coverage.status of that row (active / cancelled).""" + ms_cd: str | None = None + orec: str | None = None + dual_01: str | None = None + """v2 January dual status (NULL on v3 rows).""" + enrollment_start_date: date | None = None + enrollment_end_date: date | None = None + dual_stus_cd_dual: str | None = None + """v3 dual status from the Dual Medicare/Medicaid Coverage row.""" -# ── Stage 12: Final eligibility (Patient + Coverage) ───────────────────────── - - -class BcdaEligibility(SQLTable): - """bcda.eligibility — Patient + Coverage mapped to eligibility schema.""" +class BcdaEligibility(InputLayerEligibility): + """bcda.eligibility — saved to input_layer.eligibility.""" __schema__ = "bcda" __tablename__ = "eligibility" - - person_id: str | None = None - member_id: str | None = None - gender: str | None = None - race: str | None = None - birth_date: date | None = None - death_date: date | None = None - death_flag: int | None = None - enrollment_start_date: date | None = None - enrollment_end_date: date | None = None - payer: str | None = None - plan: str | None = None - original_reason_entitlement_code: str | None = None - dual_status_code: str | None = None - medicare_status_code: str | None = None - first_name: str | None = None - last_name: str | None = None - address: str | None = None - city: str | None = None - state: str | None = None - zip_code: str | None = None - phone: str | None = None - data_source: str | None = None - file_name: str | None = None - file_date: date | None = None - ingest_datetime: datetime | None = None diff --git a/src/bcda/table/eob.py b/src/bcda/table/eob.py index 9508f5f..3f7a61a 100644 --- a/src/bcda/table/eob.py +++ b/src/bcda/table/eob.py @@ -414,6 +414,12 @@ class ExplanationOfBenefit(SQLTable): clm_sbmt_frmt_cd: str | None = None """Submission format (supportingInfo CLM_SBMT_FRMT_CD).""" + clm_bill_type_cd: str | None = None + """NUBC type of bill, 4 characters (supportingInfo typeofbill coding + https://www.nubc.org/CodeSystem/TypeOfBill) — the v3 CLM-BILL-FREQ-CD + coding carries the last three characters, so this is the only place + the whole code appears.""" + clm_blg_prvdr_name: str | None = None """Billing provider name (contained Organization.name).""" diff --git a/src/cli/load.py b/src/cli/load.py index 35ed28c..eba461c 100644 --- a/src/cli/load.py +++ b/src/cli/load.py @@ -52,6 +52,11 @@ def bcda( "--skip-flatten", help="Skip flattening, load existing Parquet.", ), + run_pipeline: bool = typer.Option( + False, + "--run-pipeline", + help="Also run the bcda connector pipeline into input_layer.*", + ), ) -> None: """Flatten BCDA FHIR exports and load into DuckDB. @@ -68,6 +73,7 @@ def bcda( ndjson_dir=ndjson_dir, database=database, skip_flatten=skip_flatten, + run_pipeline=run_pipeline, ) except FileNotFoundError as e: typer.echo(f"Error: {e}", err=True) diff --git a/tests/aco/test_load_bcda.py b/tests/aco/test_load_bcda.py index 7021a33..5f99fed 100644 --- a/tests/aco/test_load_bcda.py +++ b/tests/aco/test_load_bcda.py @@ -95,12 +95,17 @@ class TestLoadV3EndToEnd: return real_path(key) with patch("conf.path", side_effect=fake_path): - stats = load_bcda(database=str(db)) + stats = load_bcda(database=str(db), run_pipeline=True) assert stats == { "bcda.patient": 94, "bcda.explanation_of_benefit": 538, "bcda.eob_item": 2464, "bcda.coverage": 175, + # the connector pipeline ran and landed input_layer.*; the + # duplicated EOB part collapses on claim control number + "input_layer.medical_claim": 1021, + "input_layer.pharmacy_claim": 102, + "input_layer.eligibility": 94, } con = duckdb.connect(str(db), read_only=True) plans = dict( @@ -121,4 +126,17 @@ class TestLoadV3EndToEnd: ).fetchone()[0] == 0 ) + assert ( + con.execute( + "select count(distinct claim_id) from input_layer.medical_claim" + ).fetchone()[0] + == 131 + ) + assert ( + con.execute( + "select count(*) from input_layer.medical_claim " + "where claim_type = 'institutional' and length(bill_type_code) <> 4" + ).fetchone()[0] + == 0 + ) con.close() diff --git a/tests/bcda/test_cclf_v3.py b/tests/bcda/test_cclf_v3.py new file mode 100644 index 0000000..0c736f0 --- /dev/null +++ b/tests/bcda/test_cclf_v3.py @@ -0,0 +1,923 @@ +"""bcda.express.cclf — the connector pipeline on v2- and v3-shaped tables. + +Routing, final-action filtering, cross-source dedupe, the shared +medical_claim mapping, Part D, demographics and eligibility. Fixtures +are minimal frames; the official-sample run lives in test_pipe_cclf. +""" + +from __future__ import annotations + +from datetime import date + +import narwhals as nw +import polars as pl +import pytest + +from bcda.express import cclf as ex +from bcda.express.cclf import ( + eligibility, + int_dme_medical_claim, + int_institutional_medical_claim, + int_physician_medical_claim, + medical_claim, + pharmacy_claim, + stg_beneficiary_demographics, + stg_beneficiary_xref, + stg_claim_header, + stg_part_a_header, + stg_part_b_dme_header, + stg_part_b_physician_header, + stg_part_d_header, +) +from bcda.table.cclf_pipe import ( + BcdaEligibility, + BcdaMedicalClaim, + BcdaPharmacyClaim, + BcdaStgBeneficiaryDemographics, + BcdaStgClaimHeader, +) + +# ── fixtures ───────────────────────────────────────────────── + + +@pytest.fixture +def xref() -> pl.DataFrame: + return stg_beneficiary_xref( + bcda__patient=pl.DataFrame( + {"bene_id": ["B1", "B2"], "bene_mbi_id": ["MBI-1", "MBI-2"]} + ) + ) + + +def _eob(rows: list[dict]) -> pl.DataFrame: + """Header frame from sparse dicts; missing columns are filled by the stage.""" + cols: dict[str, list] = {} + for r in rows: + for k in r: + cols.setdefault(k, []) + for r in rows: + for k in cols: + cols[k].append(r.get(k)) + return pl.DataFrame(cols) + + +@pytest.fixture +def v3_headers() -> pl.DataFrame: + """Seven v3 claims: one per family, a NotFinalAction, and a SharedSystem/NCH pair.""" + return _eob( + [ + { + "clm_id": "I1", + "clm_cntl_num": "CCN-I1", + "bene_id": "B1", + "eob_type": "INPATIENT", + "clm_type_cd": "60", + "system_type": "NationalClaimsHistory", + "final_action": "FinalAction", + "last_updated": "2026-01-02", + "clm_from_dt": date(2025, 1, 1), + "clm_thru_dt": date(2025, 1, 9), + "clm_actv_care_from_dt": date(2024, 12, 31), + "clm_dschrg_dt": date(2025, 1, 9), + "clm_admsn_src_cd": "7", + "clm_admsn_type_cd": "1", + "bene_ptnt_stus_cd": "01", + "clm_bill_type_cd": "0111", + "clm_bill_fac_type_cd": "1", + "clm_bill_clsfctn_cd": "11", + "clm_bill_freq_cd": "111", + "dgns_drg_cd": "470", + "dgns_prcdr_icd_ind": "0", + "clm_dgns_1_cd": "I10", + "clm_poa_ind": "Y", + "clm_prcdr_cd": "0SRC0J9", + "clm_prcdr_prfrm_dt": date(2025, 1, 2), + "atndg_prvdr_npi_num": "ATT", + "oprtg_prvdr_npi_num": "OPR", + "fac_prvdr_npi_num": "FAC", + "clm_blg_prvdr_tax_num": "TIN-FAC", + "clm_nch_wkly_proc_dt": date(2025, 1, 20), + "clm_mdcr_instnl_tot_chrg_amt": 9999.0, + }, + { + "clm_id": "C1", + "clm_cntl_num": "CCN-C1", + "bene_id": "B1", + "eob_type": "CARRIER", + "clm_type_cd": "71", + "system_type": "NationalClaimsHistory", + "final_action": "FinalAction", + "last_updated": "2026-01-02", + "clm_from_dt": date(2025, 2, 1), + "clm_thru_dt": date(2025, 2, 1), + "dgns_prcdr_icd_ind": "0", + "clm_dgns_1_cd": "E11.9", + "rndrg_prvdr_npi_num": "RND", + "clm_rndrg_prvdr_tax_num": "TIN-RND", + "carr_clm_blg_npi_num": "BLG", + "clm_cms_proc_dt": date(2025, 2, 10), + }, + { + "clm_id": "D1", + "clm_cntl_num": "CCN-D1", + "bene_id": "B2", + "eob_type": "DME", + "clm_type_cd": "82", + "system_type": "NationalClaimsHistory", + "final_action": "FinalAction", + "last_updated": "2026-01-02", + "clm_from_dt": date(2025, 3, 1), + "clm_thru_dt": date(2025, 3, 1), + "ordrg_prvdr_npi_num": "ORD", + "payto_prvdr_npi_num": "PAY", + }, + { + "clm_id": "P1", + "pde_id": "P1", + "clm_cntl_num": "CCN-P1", + "bene_id": "B2", + "eob_type": "PDE", + "clm_type_cd": "1", + "system_type": "DDPS", + "final_action": "FinalAction", + "last_updated": "2026-01-02", + "clm_from_dt": date(2025, 4, 1), + "clm_thru_dt": date(2025, 4, 1), + "clm_prsbng_prvdr_gnrc_id_num": "PRS", + "fac_prvdr_npi_num": "PHARM", + "clm_srvc_prvdr_gnrc_id_num": "LIC", + "clm_line_days_suply_qty": 30.0, + "clm_line_rx_fill_num": 2.0, + "clm_pd_dt": date(2025, 4, 3), + "clm_bene_pmt_amt": 7.5, + "tot_rx_cst_amt": 120.0, + }, + { + "clm_id": "X1", + "clm_cntl_num": "CCN-X1", + "bene_id": "B1", + "eob_type": "SNF", + "clm_type_cd": "20", + "system_type": "NationalClaimsHistory", + "final_action": "NotFinalAction", + "last_updated": "2026-01-02", + "clm_from_dt": date(2025, 5, 1), + "clm_thru_dt": date(2025, 5, 2), + }, + { + "clm_id": "S1", + "clm_cntl_num": "CCN-PAIR", + "bene_id": "B1", + "eob_type": "OUTPATIENT", + "clm_type_cd": "40", + "system_type": "SharedSystem", + "final_action": "FinalAction", + "last_updated": "2026-01-05", + "clm_from_dt": date(2025, 6, 1), + "clm_thru_dt": date(2025, 6, 1), + }, + { + "clm_id": "N1", + "clm_cntl_num": "CCN-PAIR", + "bene_id": "B1", + "eob_type": "OUTPATIENT", + "clm_type_cd": "40", + "system_type": "NationalClaimsHistory", + "final_action": "FinalAction", + "last_updated": "2026-01-01", + "clm_from_dt": date(2025, 6, 1), + "clm_thru_dt": date(2025, 6, 1), + }, + ] + ) + + +@pytest.fixture +def v3_items() -> pl.DataFrame: + return _eob( + [ + { + "clm_id": "I1", + "clm_line_num": 1.0, + "clm_line_from_dt": date(2025, 1, 1), + "clm_line_thru_dt": date(2025, 1, 9), + "clm_line_prod_rev_ctr_cd": "0120", + "clm_line_hcpcs_cd": "99223", + "clm_line_srvc_unit_qty": 1.4, + "clm_line_cvrd_pd_amt": 100.0, + "rev_cntr_tot_chrg_amt": 250.0, + "rev_cntr_coinsrnc_wge_adjstd_c": 10.0, + "rev_cntr_cash_ddctbl_amt": 5.0, + }, + { + "clm_id": "I1", + "clm_line_num": 2.0, + "clm_line_from_dt": date(2025, 1, 2), + "clm_line_thru_dt": date(2025, 1, 2), + "clm_line_prod_rev_ctr_cd": "0450", + "clm_line_hcpcs_cd": None, + "clm_line_srvc_unit_qty": 2.0, + "clm_line_cvrd_pd_amt": 50.0, + "rev_cntr_tot_chrg_amt": 75.0, + }, + { + "clm_id": "C1", + "clm_line_num": 1.0, + "clm_line_from_dt": date(2025, 2, 1), + "clm_line_thru_dt": date(2025, 2, 1), + "clm_line_hcpcs_cd": "99213", + "hcpcs_1_mdfr_cd": "25", + "clm_pos_cd": "11", + "clm_line_srvc_unit_qty": 1.0, + "clm_line_cvrd_pd_amt": 60.0, + "clm_line_alowd_chrg_amt": 75.0, + "clm_line_sbmtd_chrg_amt": 150.0, + "clm_line_coinsrnc_amt": 15.0, + "clm_line_bene_ptb_ddctbl_amt": 0.0, + }, + { + "clm_id": "D1", + "clm_line_num": 1.0, + "clm_line_from_dt": date(2025, 3, 1), + "clm_line_thru_dt": date(2025, 3, 31), + "clm_line_hcpcs_cd": "E1390", + "clm_pos_cd": "12", + "clm_line_srvc_unit_qty": 1.0, + "clm_line_cvrd_pd_amt": 80.0, + "clm_line_alowd_chrg_amt": 100.0, + "clm_line_sbmtd_chrg_amt": 120.0, + }, + { + "clm_id": "P1", + "clm_line_num": 1.0, + "clm_line_from_dt": date(2025, 4, 1), + "clm_line_thru_dt": date(2025, 4, 1), + "clm_line_ndc_cd": "00002143380", + "clm_line_ndc_qty": 30.0, + "clm_line_srvc_unit_qty": 1.0, + "tot_rx_cst_amt": 118.0, + }, + { + "clm_id": "N1", + "clm_line_num": 1.0, + "clm_line_from_dt": date(2025, 6, 1), + "clm_line_thru_dt": date(2025, 6, 1), + "clm_line_hcpcs_cd": "G0463", + "clm_line_cvrd_pd_amt": 30.0, + }, + { + "clm_id": "X1", + "clm_line_num": 1.0, + "clm_line_from_dt": date(2025, 5, 1), + "clm_line_thru_dt": date(2025, 5, 1), + "clm_line_cvrd_pd_amt": 1.0, + }, + ] + ) + + +@pytest.fixture +def v2_headers() -> pl.DataFrame: + """v2-shaped headers: routed by RIC code, no tags, no control numbers.""" + return _eob( + [ + { + "clm_id": "V1", + "bene_id": "B1", + "nch_near_line_rec_ident_cd": "V", + "clm_from_dt": date(2023, 1, 1), + "clm_thru_dt": date(2023, 1, 5), + "clm_bill_fac_type_cd": "1", + "clm_bill_clsfctn_cd": "1", + "clm_bill_freq_cd": "1", + "dgns_drg_cd": "MS-DRG470", + "oprtg_prvdr_npi_num": "OP", + "fac_prvdr_npi_num": "FACV", + "clm_mdcr_instnl_tot_chrg_amt": 5000.0, + }, + { + "clm_id": "O1", + "bene_id": "B1", + "nch_near_line_rec_ident_cd": "O", + "clm_from_dt": date(2023, 2, 1), + "clm_thru_dt": date(2023, 2, 1), + "rndrg_prvdr_npi_num": "RN", + "carr_clm_blg_npi_num": "BL", + }, + { + "clm_id": "M1", + "bene_id": "B2", + "nch_near_line_rec_ident_cd": "M", + "clm_from_dt": date(2023, 3, 1), + "clm_thru_dt": date(2023, 3, 1), + "ordrg_prvdr_npi_num": "ORD", + "payto_prvdr_npi_num": "PAY", + }, + { + "clm_id": "P2", + "pde_id": "PDE-2", + "bene_id": "B2", + "nch_near_line_rec_ident_cd": "P", + "clm_from_dt": date(2023, 4, 1), + "clm_thru_dt": date(2023, 4, 1), + "clm_prsbng_prvdr_gnrc_id_num": "PRS2", + "clm_srvc_prvdr_gnrc_id_num": "DSP2", + "clm_line_days_suply_qty": 30.0, + "clm_line_rx_fill_num": 0.0, + }, + { + "clm_id": "W1", + "bene_id": "B1", + "nch_near_line_rec_ident_cd": "W", + "clm_from_dt": date(2023, 5, 1), + "clm_thru_dt": date(2023, 5, 1), + }, + { + "clm_id": "Z1", + "bene_id": "B1", + "nch_near_line_rec_ident_cd": "Z", + "clm_from_dt": date(2023, 6, 1), + "clm_thru_dt": date(2023, 6, 1), + }, + { + "clm_id": "T1", + "bene_id": "B1", + "clm_type_cd": "72", + "clm_from_dt": date(2023, 7, 1), + "clm_thru_dt": date(2023, 7, 1), + }, + ] + ) + + +@pytest.fixture +def v2_items() -> pl.DataFrame: + return _eob( + [ + { + "clm_id": "V1", + "clm_line_num": 1.0, + "clm_line_from_dt": date(2023, 1, 1), + "clm_line_thru_dt": date(2023, 1, 5), + "clm_line_prod_rev_ctr_cd": "0301", + "clm_line_hcpcs_cd": "99213", + "clm_line_srvc_unit_qty": 1.0, + "clm_line_cvrd_pd_amt": 75.5, + }, + { + "clm_id": "O1", + "clm_line_num": 1.0, + "clm_line_from_dt": date(2023, 2, 1), + "clm_line_thru_dt": date(2023, 2, 1), + "clm_line_hcpcs_cd": "99214", + "clm_pos_cd": "11", + "clm_line_srvc_unit_qty": 2.0, + "clm_line_cvrd_pd_amt": 120.0, + "clm_line_alowd_chrg_amt": 150.0, + }, + { + "clm_id": "M1", + "clm_line_num": 1.0, + "clm_line_from_dt": date(2023, 3, 1), + "clm_line_thru_dt": date(2023, 3, 1), + "clm_line_hcpcs_cd": "E1390", + "clm_pos_cd": "12", + "clm_line_cvrd_pd_amt": 50.0, + "clm_line_alowd_chrg_amt": 60.0, + }, + { + "clm_id": "P2", + "clm_line_num": 1.0, + "clm_line_from_dt": date(2023, 4, 1), + "clm_line_thru_dt": date(2023, 4, 1), + "clm_line_ndc_cd": "12345678901", + "clm_line_srvc_unit_qty": 30.0, + "clm_line_cvrd_pd_amt": 10.0, + "clm_line_bene_pmt_amt": 5.0, + }, + ] + ) + + +# ── helpers ────────────────────────────────────────────────── + + +class TestHelpers: + def test_case_nests(self) -> None: + df = pl.DataFrame({"x": [1, 2, 3]}) + expr = ex._case( + (nw.col("x") == 1, nw.lit("a")), + (nw.col("x") == 2, nw.lit("b")), + default=nw.lit("z"), + ) + out = nw.from_native(df).select(expr.alias("y")).to_native() + assert out["y"].to_list() == ["a", "b", "z"] + + def test_ensure_adds_typed_nulls(self) -> None: + df = nw.from_native(pl.DataFrame({"a": [1]})) + out = ex._ensure(df, {"a": nw.Int64, "b": nw.String, "c": nw.Date}).to_native() + assert out.columns == ["a", "b", "c"] + assert out.schema["b"] == pl.String and out.schema["c"] == pl.Date + assert ex._ensure(df, {"a": nw.Int64}).to_native().columns == ["a"] + + +# ── stg_claim_header ───────────────────────────────────────── + + +class TestStgClaimHeader: + def test_v3_routing_final_action_and_dedupe(self, v3_headers, xref) -> None: + out = stg_claim_header( + bcda__explanation_of_benefit=v3_headers, bcda___stg_beneficiary_xref=xref + ) + fam = dict( + zip(out["clm_id"].to_list(), out["claim_family"].to_list(), strict=True) + ) + assert fam == { + "I1": "institutional", + "C1": "professional", + "D1": "dme", + "P1": "pharmacy", + "N1": "institutional", # NCH row wins the CCN-PAIR + } + assert "X1" not in fam # NotFinalAction + assert "S1" not in fam # SharedSystem twin dropped + # sparse fixture: every output column is in the contract; the runner's + # exact-match check is exercised on real tables in test_pipe_cclf + assert set(out.columns) <= set(BcdaStgClaimHeader.column_names()) + assert {"claim_family", "current_bene_mbi_id"} <= set(out.columns) + mbi = dict( + zip( + out["clm_id"].to_list(), + out["current_bene_mbi_id"].to_list(), + strict=True, + ) + ) + assert mbi["I1"] == "MBI-1" and mbi["D1"] == "MBI-2" + + def test_newest_wins_within_a_source(self, xref) -> None: + eob = _eob( + [ + { + "clm_id": "old", + "clm_cntl_num": "C", + "bene_id": "B1", + "eob_type": "CARRIER", + "system_type": "NationalClaimsHistory", + "final_action": "FinalAction", + "last_updated": "2026-01-01", + }, + { + "clm_id": "new", + "clm_cntl_num": "C", + "bene_id": "B1", + "eob_type": "CARRIER", + "system_type": "NationalClaimsHistory", + "final_action": "FinalAction", + "last_updated": "2026-02-01", + }, + ] + ) + out = stg_claim_header( + bcda__explanation_of_benefit=eob, bcda___stg_beneficiary_xref=xref + ) + assert out["clm_id"].to_list() == ["new"] + + def test_v2_routing_fallbacks(self, v2_headers, xref) -> None: + out = stg_claim_header( + bcda__explanation_of_benefit=v2_headers, bcda___stg_beneficiary_xref=xref + ) + fam = dict( + zip(out["clm_id"].to_list(), out["claim_family"].to_list(), strict=True) + ) + assert fam == { + "V1": "institutional", + "O1": "professional", + "M1": "dme", + "P2": "pharmacy", + "W1": "institutional", # Part B institutional RIC + "Z1": None, # unknown RIC → unrouted, kept for inspection + "T1": "professional", # CLM-TYPE-CD without an EOB-TYPE + } + assert len(out) == 7 # nothing without a control number is collapsed + + def test_pde_id_routes_pharmacy_without_codes(self, xref) -> None: + eob = _eob([{"clm_id": "Q", "pde_id": "Q", "bene_id": "B1"}]) + out = stg_claim_header( + bcda__explanation_of_benefit=eob, bcda___stg_beneficiary_xref=xref + ) + assert out["claim_family"].to_list() == ["pharmacy"] + + def test_family_filters(self, v3_headers, xref) -> None: + hdr = stg_claim_header( + bcda__explanation_of_benefit=v3_headers, bcda___stg_beneficiary_xref=xref + ) + assert sorted( + stg_part_a_header(bcda___stg_claim_header=hdr)["clm_id"].to_list() + ) == ["I1", "N1"] + assert stg_part_b_physician_header(bcda___stg_claim_header=hdr)[ + "clm_id" + ].to_list() == ["C1"] + assert stg_part_b_dme_header(bcda___stg_claim_header=hdr)[ + "clm_id" + ].to_list() == ["D1"] + assert stg_part_d_header(bcda___stg_claim_header=hdr)["clm_id"].to_list() == [ + "P1" + ] + + +# ── medical_claim ──────────────────────────────────────────── + + +@pytest.fixture +def v3_medical(v3_headers, v3_items, xref) -> pl.DataFrame: + hdr = stg_claim_header( + bcda__explanation_of_benefit=v3_headers, bcda___stg_beneficiary_xref=xref + ) + return medical_claim( + bcda___int_institutional_medical_claim=int_institutional_medical_claim( + bcda___stg_part_a_header=stg_part_a_header(bcda___stg_claim_header=hdr), + bcda__eob_item=v3_items, + ), + bcda___int_physician_medical_claim=int_physician_medical_claim( + bcda___stg_part_b_physician_header=stg_part_b_physician_header( + bcda___stg_claim_header=hdr + ), + bcda__eob_item=v3_items, + ), + bcda___int_dme_medical_claim=int_dme_medical_claim( + bcda___stg_part_b_dme_header=stg_part_b_dme_header( + bcda___stg_claim_header=hdr + ), + bcda__eob_item=v3_items, + ), + ) + + +class TestMedicalClaimV3: + def test_contract_and_rows(self, v3_medical) -> None: + assert set(v3_medical.columns) == set(BcdaMedicalClaim.column_names()) + assert sorted(v3_medical["claim_id"].to_list()) == [ + "C1", + "D1", + "I1", + "I1", + "N1", + ] + assert v3_medical.filter(pl.col("claim_id") == "X1").is_empty() + + def test_institutional_line(self, v3_medical) -> None: + r = v3_medical.filter( + (pl.col("claim_id") == "I1") & (pl.col("claim_line_number") == 1) + ).to_dicts()[0] + assert r["claim_type"] == "institutional" and r["person_id"] == "MBI-1" + assert (r["admission_date"], r["discharge_date"]) == ( + date(2024, 12, 31), + date(2025, 1, 9), + ) + assert ( + r["admit_source_code"], + r["admit_type_code"], + r["discharge_disposition_code"], + ) == ("7", "1", "01") + assert r["bill_type_code"] == "0111" # NUBC, not "111111" + assert (r["drg_code_type"], r["drg_code"]) == ("ms-drg", "470") + assert r["revenue_center_code"] == "0120" and r["hcpcs_code"] == "99223" + assert r["service_unit_quantity"] == 1 + assert ( + r["rendering_npi"], + r["billing_npi"], + r["facility_npi"], + r["billing_tin"], + ) == ("ATT", "FAC", "FAC", "TIN-FAC") + assert r["rendering_tin"] is None and r["place_of_service_code"] is None + assert ( + r["paid_amount"], + r["charge_amount"], + r["coinsurance_amount"], + r["deductible_amount"], + ) == (100.0, 250.0, 10.0, 5.0) + assert r["allowed_amount"] is None + assert r["paid_date"] == date(2025, 1, 20) # NCH weekly process date + assert ( + r["diagnosis_code_type"], + r["diagnosis_code_1"], + r["diagnosis_poa_1"], + ) == ("icd-10-cm", "I10", "Y") + assert ( + r["procedure_code_type"], + r["procedure_code_1"], + r["procedure_date_1"], + ) == ("icd-10-pcs", "0SRC0J9", date(2025, 1, 2)) + assert r["diagnosis_code_25"] is None and r["procedure_code_2"] is None + assert r["in_network_flag"] == 1 and r["data_source"] == "medicare bcda" + line2 = v3_medical.filter( + (pl.col("claim_id") == "I1") & (pl.col("claim_line_number") == 2) + ).to_dicts()[0] + assert line2["hcpcs_code"] is None and line2["charge_amount"] == 75.0 + + def test_professional_line(self, v3_medical) -> None: + r = v3_medical.filter(pl.col("claim_id") == "C1").to_dicts()[0] + assert r["claim_type"] == "professional" + assert r["place_of_service_code"] == "11" and r["hcpcs_modifier_1"] == "25" + assert (r["rendering_npi"], r["rendering_tin"], r["billing_npi"]) == ( + "RND", + "TIN-RND", + "BLG", + ) + assert ( + r["facility_npi"] is None + and r["bill_type_code"] is None + and r["admission_date"] is None + ) + assert ( + r["paid_amount"], + r["allowed_amount"], + r["charge_amount"], + r["coinsurance_amount"], + r["deductible_amount"], + ) == (60.0, 75.0, 150.0, 15.0, 0.0) + assert r["paid_date"] == date(2025, 2, 10) # CMS process date fallback + assert ( + r["diagnosis_code_1"] == "E11.9" + and r["procedure_code_type"] is None + and r["diagnosis_poa_1"] is None + ) + + def test_dme_line(self, v3_medical) -> None: + r = v3_medical.filter(pl.col("claim_id") == "D1").to_dicts()[0] + assert r["claim_type"] == "professional" and r["person_id"] == "MBI-2" + assert (r["rendering_npi"], r["billing_npi"]) == ( + "ORD", + "PAY", + ) # ordering / pay-to fallbacks + assert r["place_of_service_code"] == "12" and r["paid_date"] is None + + def test_bill_type_fallback_from_v3_freq_code(self, xref) -> None: + """No NUBC coding captured → leading zero + the 3-char v3 freq code.""" + eob = _eob( + [ + { + "clm_id": "H", + "clm_cntl_num": "C", + "bene_id": "B1", + "eob_type": "HHA", + "final_action": "FinalAction", + "clm_bill_fac_type_cd": "3", + "clm_bill_clsfctn_cd": "32", + "clm_bill_freq_cd": "329", + }, + ] + ) + items = _eob([{"clm_id": "H", "clm_line_num": 1.0}]) + hdr = stg_claim_header( + bcda__explanation_of_benefit=eob, bcda___stg_beneficiary_xref=xref + ) + out = int_institutional_medical_claim( + bcda___stg_part_a_header=stg_part_a_header(bcda___stg_claim_header=hdr), + bcda__eob_item=items, + ) + assert out["bill_type_code"].to_list() == ["0329"] + + def test_pair_survivor_is_nch(self, v3_medical) -> None: + assert v3_medical.filter(pl.col("claim_id") == "N1")[ + "hcpcs_code" + ].to_list() == ["G0463"] + + +class TestMedicalClaimV2: + def test_v2_frames_map(self, v2_headers, v2_items, xref) -> None: + hdr = stg_claim_header( + bcda__explanation_of_benefit=v2_headers, bcda___stg_beneficiary_xref=xref + ) + inst = int_institutional_medical_claim( + bcda___stg_part_a_header=stg_part_a_header(bcda___stg_claim_header=hdr), + bcda__eob_item=v2_items, + ) + prof = int_physician_medical_claim( + bcda___stg_part_b_physician_header=stg_part_b_physician_header( + bcda___stg_claim_header=hdr + ), + bcda__eob_item=v2_items, + ) + dme = int_dme_medical_claim( + bcda___stg_part_b_dme_header=stg_part_b_dme_header( + bcda___stg_claim_header=hdr + ), + bcda__eob_item=v2_items, + ) + out = medical_claim( + bcda___int_institutional_medical_claim=inst, + bcda___int_physician_medical_claim=prof, + bcda___int_dme_medical_claim=dme, + ) + assert set(out.columns) == set(BcdaMedicalClaim.column_names()) + v1 = out.filter(pl.col("claim_id") == "V1").to_dicts()[0] + assert v1["bill_type_code"] == "111" # v2 fac + clsfctn + freq + assert ( + v1["drg_code"] == "470" and v1["charge_amount"] == 5000.0 + ) # header total fallback + assert (v1["rendering_npi"], v1["billing_npi"]) == ("OP", "FACV") + assert (v1["admission_date"], v1["discharge_date"]) == ( + date(2023, 1, 1), + date(2023, 1, 5), + ) + assert v1["paid_date"] is None and v1["coinsurance_amount"] is None + o1 = out.filter(pl.col("claim_id") == "O1").to_dicts()[0] + assert (o1["rendering_npi"], o1["billing_npi"], o1["allowed_amount"]) == ( + "RN", + "BL", + 150.0, + ) + m1 = out.filter(pl.col("claim_id") == "M1").to_dicts()[0] + assert (m1["rendering_npi"], m1["billing_npi"], m1["paid_amount"]) == ( + "ORD", + "PAY", + 50.0, + ) + # W1 and T1 route but have no lines; Z1 is unrouted + assert sorted(set(out["claim_id"].to_list())) == ["M1", "O1", "V1"] + + +# ── pharmacy_claim ─────────────────────────────────────────── + + +class TestPharmacyClaim: + def test_v3(self, v3_headers, v3_items, xref) -> None: + hdr = stg_claim_header( + bcda__explanation_of_benefit=v3_headers, bcda___stg_beneficiary_xref=xref + ) + out = pharmacy_claim( + bcda___stg_part_d_header=stg_part_d_header(bcda___stg_claim_header=hdr), + bcda__eob_item=v3_items, + ) + assert set(out.columns) == set(BcdaPharmacyClaim.column_names()) + r = out.to_dicts()[0] + assert (r["claim_id"], r["claim_line_number"], r["person_id"]) == ( + "P1", + 1, + "MBI-2", + ) + assert (r["prescribing_provider_npi"], r["dispensing_provider_npi"]) == ( + "PRS", + "PHARM", + ) + assert (r["dispensing_date"], r["paid_date"]) == ( + date(2025, 4, 1), + date(2025, 4, 3), + ) + assert (r["ndc_code"], r["quantity"], r["days_supply"], r["refills"]) == ( + "00002143380", + 30, + 30, + 2, + ) + assert ( + r["paid_amount"] is None + ) # sandbox emits no plan-paid; header clm_pmt_amt is NULL too + assert (r["charge_amount"], r["copayment_amount"]) == ( + 118.0, + 7.5, + ) # line drug cost, header paid-by-patient + assert r["allowed_amount"] is None and r["deductible_amount"] is None + + def test_v2(self, v2_headers, v2_items, xref) -> None: + hdr = stg_claim_header( + bcda__explanation_of_benefit=v2_headers, bcda___stg_beneficiary_xref=xref + ) + out = pharmacy_claim( + bcda___stg_part_d_header=stg_part_d_header(bcda___stg_claim_header=hdr), + bcda__eob_item=v2_items, + ) + r = out.to_dicts()[0] + assert r["claim_id"] == "PDE-2" and r["dispensing_provider_npi"] == "DSP2" + assert (r["quantity"], r["days_supply"], r["refills"]) == (30, 30, 0) + assert (r["paid_amount"], r["copayment_amount"], r["charge_amount"]) == ( + 10.0, + 5.0, + None, + ) + assert r["paid_date"] == date(2023, 4, 1) + + +# ── eligibility ────────────────────────────────────────────── + + +def _patients() -> pl.DataFrame: + return pl.DataFrame( + { + "bene_id": ["B1", "B2", "B3"], + "bene_mbi_id": ["M1", "M2", "M3"], + "bene_sex_cd": ["1", "female", "9"], + "bene_race_cd": ["1", "5", None], + "bene_dob": [date(1950, 1, 1), date(1960, 2, 2), None], + "bene_death_dt": [None, date(2025, 1, 1), None], + "bene_1st_name": ["a", "b", "c"], + "bene_midl_name": ["x", None, None], + "bene_last_name": ["A", "B", "C"], + "bene_line_1_adr": ["1 st", None, None], + "geo_zip_plc_name": ["Town", None, None], + "geo_usps_state_cd": ["TX", "VA", None], + "geo_zip5_cd": ["77550", None, None], + } + ) + + +class TestEligibility: + def test_v3_coverage_rows(self) -> None: + coverage = pl.DataFrame( + { + "subscriber_id": ["M1", "M1", "M1", "M2", "M2", "M3"], + "class_plan": [ + "Part C", + "Part B", + "Part A", + "Part B", + "Dual Medicare/Medicaid", + "Part A", + ], + "status": [ + "active", + "active", + "cancelled", + "active", + "active", + "active", + ], + "ms_cd": [None, "10", "20", "10", None, "30"], + "orec": [None, "0", "1", "0", None, "2"], + "dual_stus_cd": [None, None, None, None, "08", None], + "bene_part_a_enrlmt_bgn_dt": [ + None, + None, + date(2020, 1, 1), + None, + None, + date(2021, 1, 1), + ], + "mdcr_covg_a_end_dt": [ + None, + None, + date(2024, 12, 31), + None, + None, + None, + ], + "bene_part_b_enrlmt_bgn_dt": [ + None, + date(2020, 6, 1), + None, + date(2022, 1, 1), + None, + None, + ], + "mdcr_covg_b_end_dt": [None, None, None, None, None, None], + } + ) + demo = stg_beneficiary_demographics( + bcda__patient=_patients(), bcda__coverage=coverage + ).sort("bene_mbi_id") + assert set(demo.columns) == set(BcdaStgBeneficiaryDemographics.column_names()) + assert demo["coverage_plan"].to_list() == ["Part A", "Part B", "Part A"] + assert demo["ms_cd"].to_list() == ["20", "10", "30"] + assert demo["dual_stus_cd_dual"].to_list() == [None, "08", None] + assert demo["enrollment_start_date"].to_list() == [ + date(2020, 1, 1), + date(2022, 1, 1), + date(2021, 1, 1), + ] + elig = eligibility(bcda___stg_beneficiary_demographics=demo).sort("person_id") + assert set(elig.columns) == set(BcdaEligibility.column_names()) + assert elig["gender"].to_list() == ["male", "female", "unknown"] + assert elig["race"].to_list() == ["white", "hispanic", "unknown"] + assert elig["ethnicity"].to_list() == [None, "hispanic", None] + assert elig["enrollment_status"].to_list() == ["inactive", "active", "active"] + assert elig["dual_status_code"].to_list() == [None, "08", None] + assert elig["medicare_status_code"].to_list() == ["20", "10", "30"] + assert elig["original_reason_entitlement_code"].to_list() == ["1", "0", "2"] + assert elig["enrollment_end_date"].to_list() == [date(2024, 12, 31), None, None] + assert elig["death_flag"].to_list() == [0, 1, 0] + assert ( + elig["payer_type"].to_list() == ["medicare"] * 3 + and elig["hospice_flag"].null_count() == 3 + ) + + def test_v2_coverage_without_plan_columns(self) -> None: + coverage = pl.DataFrame( + { + "subscriber_id": ["M1", "M3"], + "ms_cd": ["10", "30"], + "orec": ["0", "2"], + "dual_01": ["04", None], + "bene_part_a_enrlmt_bgn_dt": [date(2019, 1, 1), None], + "mdcr_covg_a_end_dt": [None, None], + } + ) + demo = stg_beneficiary_demographics( + bcda__patient=_patients(), bcda__coverage=coverage + ).sort("bene_mbi_id") + assert demo["coverage_plan"].to_list() == [None, None, None] + elig = eligibility(bcda___stg_beneficiary_demographics=demo).sort("person_id") + assert elig["dual_status_code"].to_list() == ["04", None, None] + assert elig["medicare_status_code"].to_list() == ["10", None, "30"] + assert elig["enrollment_status"].null_count() == 3 + assert elig["enrollment_start_date"].to_list() == [date(2019, 1, 1), None, None] diff --git a/tests/bcda/test_express.py b/tests/bcda/test_express.py index 813bf01..5d4f79e 100644 --- a/tests/bcda/test_express.py +++ b/tests/bcda/test_express.py @@ -20,20 +20,7 @@ from datetime import date import polars as pl import pytest -from bcda.express.cclf import ( - eligibility, - int_dme_medical_claim, - int_institutional_medical_claim, - int_physician_medical_claim, - medical_claim, - pharmacy_claim, - stg_beneficiary_demographics, - stg_beneficiary_xref, - stg_part_a_header, - stg_part_b_dme_header, - stg_part_b_physician_header, - stg_part_d_header, -) +from bcda.express.cclf import stg_beneficiary_xref from bcda.express.flatten import ( _coding_code, _get_ext, @@ -1030,1052 +1017,6 @@ class TestStgBeneficiaryXref: assert set(result["current_bene_mbi_id"].to_list()) == {"MBI-A", "MBI-B"} -# ── stg_part_a_header ─────────────────────────────────────── - - -class TestStgPartAHeader: - """EOB(V) filter + MBI join.""" - - def test_filters_to_part_a( - self, - eob_df: pl.DataFrame, - xref_df: pl.DataFrame, - ) -> None: - result = stg_part_a_header( - bcda__explanation_of_benefit=eob_df, - bcda___stg_beneficiary_xref=xref_df, - ) - assert result.shape[0] == 1 - nch = result["nch_near_line_rec_ident_cd"].to_list() - assert nch == ["V"] - - def test_mbi_resolved( - self, - eob_df: pl.DataFrame, - xref_df: pl.DataFrame, - ) -> None: - result = stg_part_a_header( - bcda__explanation_of_benefit=eob_df, - bcda___stg_beneficiary_xref=xref_df, - ) - assert "current_bene_mbi_id" in result.columns - assert result["current_bene_mbi_id"].to_list() == ["MBI-A"] - - -# ── stg_part_b_physician_header ───────────────────────────── - - -class TestStgPartBPhysicianHeader: - """EOB(O) filter + MBI join.""" - - def test_filters_to_part_b_carrier( - self, - eob_df: pl.DataFrame, - xref_df: pl.DataFrame, - ) -> None: - result = stg_part_b_physician_header( - bcda__explanation_of_benefit=eob_df, - bcda___stg_beneficiary_xref=xref_df, - ) - assert result.shape[0] == 1 - nch = result["nch_near_line_rec_ident_cd"].to_list() - assert nch == ["O"] - - def test_mbi_resolved( - self, - eob_df: pl.DataFrame, - xref_df: pl.DataFrame, - ) -> None: - result = stg_part_b_physician_header( - bcda__explanation_of_benefit=eob_df, - bcda___stg_beneficiary_xref=xref_df, - ) - assert result["current_bene_mbi_id"].to_list() == ["MBI-A"] - - -# ── stg_part_b_dme_header ────────────────────────────────── - - -class TestStgPartBDmeHeader: - """EOB(M) filter + MBI join.""" - - def test_filters_to_dme( - self, - eob_df: pl.DataFrame, - xref_df: pl.DataFrame, - ) -> None: - result = stg_part_b_dme_header( - bcda__explanation_of_benefit=eob_df, - bcda___stg_beneficiary_xref=xref_df, - ) - assert result.shape[0] == 1 - nch = result["nch_near_line_rec_ident_cd"].to_list() - assert nch == ["M"] - - def test_mbi_resolved( - self, - eob_df: pl.DataFrame, - xref_df: pl.DataFrame, - ) -> None: - result = stg_part_b_dme_header( - bcda__explanation_of_benefit=eob_df, - bcda___stg_beneficiary_xref=xref_df, - ) - assert result["current_bene_mbi_id"].to_list() == ["MBI-B"] - - -# ── stg_part_d_header ────────────────────────────────────── - - -class TestStgPartDHeader: - """EOB(P) filter + MBI join.""" - - def test_filters_to_part_d( - self, - eob_df: pl.DataFrame, - xref_df: pl.DataFrame, - ) -> None: - result = stg_part_d_header( - bcda__explanation_of_benefit=eob_df, - bcda___stg_beneficiary_xref=xref_df, - ) - assert result.shape[0] == 1 - nch = result["nch_near_line_rec_ident_cd"].to_list() - assert nch == ["P"] - - def test_mbi_resolved( - self, - eob_df: pl.DataFrame, - xref_df: pl.DataFrame, - ) -> None: - result = stg_part_d_header( - bcda__explanation_of_benefit=eob_df, - bcda___stg_beneficiary_xref=xref_df, - ) - assert result["current_bene_mbi_id"].to_list() == ["MBI-B"] - - -# ── int_institutional_medical_claim ───────────────────────── - - -class TestIntInstitutionalMedicalClaim: - """Part A header + items -> institutional medical_claim.""" - - @pytest.fixture - def staged_part_a( - self, - eob_df: pl.DataFrame, - xref_df: pl.DataFrame, - ) -> pl.DataFrame: - return stg_part_a_header( - bcda__explanation_of_benefit=eob_df, - bcda___stg_beneficiary_xref=xref_df, - ) - - def test_returns_dataframe( - self, - staged_part_a: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = int_institutional_medical_claim( - bcda___stg_part_a_header=staged_part_a, - bcda__eob_item=eob_item_df, - ) - assert isinstance(result, pl.DataFrame) - - def test_claim_type_institutional( - self, - staged_part_a: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = int_institutional_medical_claim( - bcda___stg_part_a_header=staged_part_a, - bcda__eob_item=eob_item_df, - ) - assert result["claim_type"].to_list() == ["institutional"] - - def test_key_output_columns( - self, - staged_part_a: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = int_institutional_medical_claim( - bcda___stg_part_a_header=staged_part_a, - bcda__eob_item=eob_item_df, - ) - expected = { - "claim_id", - "claim_line_number", - "claim_type", - "person_id", - "member_id", - "payer", - "plan", - "claim_start_date", - "claim_end_date", - "admission_date", - "discharge_date", - "bill_type_code", - "drg_code", - "revenue_center_code", - "hcpcs_code", - "rendering_npi", - "billing_npi", - "facility_npi", - "paid_amount", - "charge_amount", - "diagnosis_code_type", - "diagnosis_code_1", - "data_source", - "in_network_flag", - } - assert expected.issubset(set(result.columns)) - - def test_diagnosis_25_columns( - self, - staged_part_a: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = int_institutional_medical_claim( - bcda___stg_part_a_header=staged_part_a, - bcda__eob_item=eob_item_df, - ) - for i in range(1, 26): - assert f"diagnosis_code_{i}" in result.columns - assert f"diagnosis_poa_{i}" in result.columns - - def test_procedure_25_columns( - self, - staged_part_a: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = int_institutional_medical_claim( - bcda___stg_part_a_header=staged_part_a, - bcda__eob_item=eob_item_df, - ) - for i in range(1, 26): - assert f"procedure_code_{i}" in result.columns - assert f"procedure_date_{i}" in result.columns - - def test_person_member_match( - self, - staged_part_a: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = int_institutional_medical_claim( - bcda___stg_part_a_header=staged_part_a, - bcda__eob_item=eob_item_df, - ) - assert result["person_id"].to_list() == result["member_id"].to_list() - - def test_payer_plan_values( - self, - staged_part_a: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = int_institutional_medical_claim( - bcda___stg_part_a_header=staged_part_a, - bcda__eob_item=eob_item_df, - ) - assert result["payer"].to_list() == ["medicare"] - assert result["plan"].to_list() == ["medicare"] - - def test_data_source( - self, - staged_part_a: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = int_institutional_medical_claim( - bcda___stg_part_a_header=staged_part_a, - bcda__eob_item=eob_item_df, - ) - assert result["data_source"].to_list() == ["medicare bcda"] - - def test_icd_indicator_mapping( - self, - staged_part_a: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = int_institutional_medical_claim( - bcda___stg_part_a_header=staged_part_a, - bcda__eob_item=eob_item_df, - ) - assert result["diagnosis_code_type"].to_list() == ["icd-10-cm"] - - def test_bill_type_code_concat( - self, - staged_part_a: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = int_institutional_medical_claim( - bcda___stg_part_a_header=staged_part_a, - bcda__eob_item=eob_item_df, - ) - assert result["bill_type_code"].to_list() == ["111"] - - -# ── int_physician_medical_claim ────────────────────────────── - - -class TestIntPhysicianMedicalClaim: - """Part B physician header + items -> professional - medical_claim.""" - - @pytest.fixture - def staged_part_b( - self, - eob_df: pl.DataFrame, - xref_df: pl.DataFrame, - ) -> pl.DataFrame: - return stg_part_b_physician_header( - bcda__explanation_of_benefit=eob_df, - bcda___stg_beneficiary_xref=xref_df, - ) - - def test_claim_type_professional( - self, - staged_part_b: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = int_physician_medical_claim( - bcda___stg_part_b_physician_header=staged_part_b, - bcda__eob_item=eob_item_df, - ) - assert result["claim_type"].to_list() == ["professional"] - - def test_place_of_service( - self, - staged_part_b: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = int_physician_medical_claim( - bcda___stg_part_b_physician_header=staged_part_b, - bcda__eob_item=eob_item_df, - ) - assert result["place_of_service_code"].to_list() == ["11"] - - def test_null_admission_fields( - self, - staged_part_b: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = int_physician_medical_claim( - bcda___stg_part_b_physician_header=staged_part_b, - bcda__eob_item=eob_item_df, - ) - assert result["admission_date"].to_list() == [None] - assert result["discharge_date"].to_list() == [None] - assert result["admit_source_code"].to_list() == [None] - - def test_rendering_npi_tin( - self, - staged_part_b: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = int_physician_medical_claim( - bcda___stg_part_b_physician_header=staged_part_b, - bcda__eob_item=eob_item_df, - ) - assert result["rendering_npi"].to_list() == ["NPI-RN2"] - assert result["rendering_tin"].to_list() == ["TAX-2"] - - def test_data_source( - self, - staged_part_b: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = int_physician_medical_claim( - bcda___stg_part_b_physician_header=staged_part_b, - bcda__eob_item=eob_item_df, - ) - assert result["data_source"].to_list() == ["medicare bcda"] - - -# ── int_dme_medical_claim ─────────────────────────────────── - - -class TestIntDmeMedicalClaim: - """Part B DME header + items -> professional medical_claim.""" - - @pytest.fixture - def staged_dme( - self, - eob_df: pl.DataFrame, - xref_df: pl.DataFrame, - ) -> pl.DataFrame: - return stg_part_b_dme_header( - bcda__explanation_of_benefit=eob_df, - bcda___stg_beneficiary_xref=xref_df, - ) - - def test_claim_type_professional( - self, - staged_dme: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = int_dme_medical_claim( - bcda___stg_part_b_dme_header=staged_dme, - bcda__eob_item=eob_item_df, - ) - assert result["claim_type"].to_list() == ["professional"] - - def test_null_diagnosis_codes( - self, - staged_dme: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = int_dme_medical_claim( - bcda___stg_part_b_dme_header=staged_dme, - bcda__eob_item=eob_item_df, - ) - for i in range(1, 26): - assert result[f"diagnosis_code_{i}"].to_list() == [None] - - def test_rendering_ordering_npi( - self, - staged_dme: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = int_dme_medical_claim( - bcda___stg_part_b_dme_header=staged_dme, - bcda__eob_item=eob_item_df, - ) - assert result["rendering_npi"].to_list() == ["NPI-ORD3"] - assert result["billing_npi"].to_list() == ["NPI-PAY3"] - - def test_null_drg_fields( - self, - staged_dme: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = int_dme_medical_claim( - bcda___stg_part_b_dme_header=staged_dme, - bcda__eob_item=eob_item_df, - ) - assert result["drg_code"].to_list() == [None] - assert result["drg_code_type"].to_list() == [None] - - def test_data_source( - self, - staged_dme: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = int_dme_medical_claim( - bcda___stg_part_b_dme_header=staged_dme, - bcda__eob_item=eob_item_df, - ) - assert result["data_source"].to_list() == ["medicare bcda"] - - -# ── medical_claim (union) ────────────────────────────────── - - -class TestMedicalClaim: - """Union of institutional + physician + DME.""" - - @pytest.fixture - def three_int_dfs( - self, - eob_df: pl.DataFrame, - xref_df: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> tuple[pl.DataFrame, pl.DataFrame, pl.DataFrame]: - part_a = stg_part_a_header( - bcda__explanation_of_benefit=eob_df, - bcda___stg_beneficiary_xref=xref_df, - ) - part_b = stg_part_b_physician_header( - bcda__explanation_of_benefit=eob_df, - bcda___stg_beneficiary_xref=xref_df, - ) - dme = stg_part_b_dme_header( - bcda__explanation_of_benefit=eob_df, - bcda___stg_beneficiary_xref=xref_df, - ) - inst = int_institutional_medical_claim( - bcda___stg_part_a_header=part_a, - bcda__eob_item=eob_item_df, - ) - phys = int_physician_medical_claim( - bcda___stg_part_b_physician_header=part_b, - bcda__eob_item=eob_item_df, - ) - dme_mc = int_dme_medical_claim( - bcda___stg_part_b_dme_header=dme, - bcda__eob_item=eob_item_df, - ) - return inst, phys, dme_mc - - def test_row_count_is_sum( - self, - three_int_dfs: tuple[ - pl.DataFrame, - pl.DataFrame, - pl.DataFrame, - ], - ) -> None: - inst, phys, dme_mc = three_int_dfs - result = medical_claim( - bcda___int_institutional_medical_claim=inst, - bcda___int_physician_medical_claim=phys, - bcda___int_dme_medical_claim=dme_mc, - ) - expected = inst.shape[0] + phys.shape[0] + dme_mc.shape[0] - assert result.shape[0] == expected - - def test_contains_all_claim_types( - self, - three_int_dfs: tuple[ - pl.DataFrame, - pl.DataFrame, - pl.DataFrame, - ], - ) -> None: - inst, phys, dme_mc = three_int_dfs - result = medical_claim( - bcda___int_institutional_medical_claim=inst, - bcda___int_physician_medical_claim=phys, - bcda___int_dme_medical_claim=dme_mc, - ) - types = set(result["claim_type"].to_list()) - assert "institutional" in types - assert "professional" in types - - def test_columns_match_inputs( - self, - three_int_dfs: tuple[ - pl.DataFrame, - pl.DataFrame, - pl.DataFrame, - ], - ) -> None: - inst, phys, dme_mc = three_int_dfs - result = medical_claim( - bcda___int_institutional_medical_claim=inst, - bcda___int_physician_medical_claim=phys, - bcda___int_dme_medical_claim=dme_mc, - ) - assert set(result.columns) == set(inst.columns) - - -# ── pharmacy_claim ────────────────────────────────────────── - - -class TestPharmacyClaim: - """Part D header + items -> pharmacy_claim.""" - - @pytest.fixture - def staged_part_d( - self, - eob_df: pl.DataFrame, - xref_df: pl.DataFrame, - ) -> pl.DataFrame: - return stg_part_d_header( - bcda__explanation_of_benefit=eob_df, - bcda___stg_beneficiary_xref=xref_df, - ) - - def test_returns_dataframe( - self, - staged_part_d: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = pharmacy_claim( - bcda___stg_part_d_header=staged_part_d, - bcda__eob_item=eob_item_df, - ) - assert isinstance(result, pl.DataFrame) - - def test_key_columns_present( - self, - staged_part_d: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = pharmacy_claim( - bcda___stg_part_d_header=staged_part_d, - bcda__eob_item=eob_item_df, - ) - expected = { - "claim_id", - "claim_line_number", - "person_id", - "member_id", - "payer", - "plan", - "prescribing_provider_npi", - "dispensing_provider_npi", - "dispensing_date", - "ndc_code", - "quantity", - "days_supply", - "refills", - "paid_amount", - "copayment_amount", - "data_source", - "in_network_flag", - } - assert expected.issubset(set(result.columns)) - - def test_claim_id_from_pde( - self, - staged_part_d: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = pharmacy_claim( - bcda___stg_part_d_header=staged_part_d, - bcda__eob_item=eob_item_df, - ) - assert result["claim_id"].to_list() == ["PDE-1"] - - def test_ndc_code( - self, - staged_part_d: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = pharmacy_claim( - bcda___stg_part_d_header=staged_part_d, - bcda__eob_item=eob_item_df, - ) - assert result["ndc_code"].to_list() == ["12345678901"] - - def test_provider_npis( - self, - staged_part_d: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = pharmacy_claim( - bcda___stg_part_d_header=staged_part_d, - bcda__eob_item=eob_item_df, - ) - assert result["prescribing_provider_npi"].to_list() == ["NPI-PRE4"] - assert result["dispensing_provider_npi"].to_list() == ["NPI-DSP4"] - - def test_paid_copay_amounts( - self, - staged_part_d: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = pharmacy_claim( - bcda___stg_part_d_header=staged_part_d, - bcda__eob_item=eob_item_df, - ) - assert result["paid_amount"].to_list() == [5.0] - assert result["copayment_amount"].to_list() == [5.0] - - def test_null_amount_columns( - self, - staged_part_d: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = pharmacy_claim( - bcda___stg_part_d_header=staged_part_d, - bcda__eob_item=eob_item_df, - ) - for col in ( - "allowed_amount", - "charge_amount", - "coinsurance_amount", - "deductible_amount", - ): - assert result[col].to_list() == [None] - - def test_data_source( - self, - staged_part_d: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = pharmacy_claim( - bcda___stg_part_d_header=staged_part_d, - bcda__eob_item=eob_item_df, - ) - assert result["data_source"].to_list() == ["medicare bcda"] - - def test_claim_line_number_always_1( - self, - staged_part_d: pl.DataFrame, - eob_item_df: pl.DataFrame, - ) -> None: - result = pharmacy_claim( - bcda___stg_part_d_header=staged_part_d, - bcda__eob_item=eob_item_df, - ) - assert result["claim_line_number"].to_list() == [1] - - -# ── stg_beneficiary_demographics ──────────────────────────── - - -class TestStgBeneficiaryDemographics: - """Patient + Coverage join for demographics.""" - - @pytest.fixture - def coverage_df(self) -> pl.DataFrame: - return pl.DataFrame( - { - "subscriber_id": ["MBI-A", "MBI-B"], - "ms_cd": ["10", "20"], - "orec": ["0", "1"], - "dual_01": ["00", "01"], - "bene_part_a_enrlmt_bgn_dt": [ - date(2020, 1, 1), - date(2021, 7, 1), - ], - "mdcr_covg_a_end_dt": [ - date(2025, 12, 31), - date(2025, 6, 30), - ], - } - ) - - @pytest.fixture - def full_patient_df(self) -> pl.DataFrame: - return pl.DataFrame( - { - "bene_id": ["B001", "B002"], - "bene_mbi_id": ["MBI-A", "MBI-B"], - "bene_1st_name": ["JOHN", "JANE"], - "bene_midl_name": ["A", "B"], - "bene_last_name": [ - "DOE", - "SMITH", - ], - "bene_dob": [ - date(1950, 1, 15), - date(1960, 3, 20), - ], - "bene_sex_cd": ["1", "2"], - "bene_race_cd": ["1", "2"], - "bene_death_dt": [None, None], - "bene_line_1_adr": [ - "123 ELM", - "456 OAK", - ], - "geo_zip_plc_name": [ - "SPRINGFIELD", - "SHELBYVILLE", - ], - "geo_usps_state_cd": ["IL", "IL"], - "geo_zip5_cd": ["62701", "62565"], - } - ) - - def test_dedup_on_mbi( - self, - full_patient_df: pl.DataFrame, - coverage_df: pl.DataFrame, - ) -> None: - result = stg_beneficiary_demographics( - bcda__patient=full_patient_df, - bcda__coverage=coverage_df, - ) - assert result.shape[0] == 2 - assert result["bene_mbi_id"].n_unique() == result.shape[0] - - def test_coverage_columns_joined( - self, - full_patient_df: pl.DataFrame, - coverage_df: pl.DataFrame, - ) -> None: - result = stg_beneficiary_demographics( - bcda__patient=full_patient_df, - bcda__coverage=coverage_df, - ) - assert "ms_cd" in result.columns - assert "orec" in result.columns - assert "dual_01" in result.columns - - def test_coverage_dedup(self) -> None: - """Multiple coverage rows per subscriber.""" - patient = pl.DataFrame( - { - "bene_id": ["B1"], - "bene_mbi_id": ["MBI-1"], - "bene_1st_name": ["A"], - "bene_midl_name": [None], - "bene_last_name": ["B"], - "bene_dob": [date(1950, 1, 1)], - "bene_sex_cd": ["1"], - "bene_race_cd": ["1"], - "bene_death_dt": [None], - "bene_line_1_adr": ["ADDR"], - "geo_zip_plc_name": ["CITY"], - "geo_usps_state_cd": ["IL"], - "geo_zip5_cd": ["60601"], - } - ) - cov = pl.DataFrame( - { - "subscriber_id": [ - "MBI-1", - "MBI-1", - ], - "ms_cd": ["10", "10"], - "orec": ["0", "0"], - "dual_01": ["00", "00"], - "bene_part_a_enrlmt_bgn_dt": [ - date(2020, 1, 1), - date(2021, 1, 1), - ], - "mdcr_covg_a_end_dt": [ - date(2025, 12, 31), - date(2025, 12, 31), - ], - } - ) - result = stg_beneficiary_demographics( - bcda__patient=patient, - bcda__coverage=cov, - ) - assert result.shape[0] == 1 - - -# ── eligibility (final) ───────────────────────────────────── - - -class TestEligibility: - """Demographics -> eligibility with crosswalks.""" - - @pytest.fixture - def staged_demographics(self) -> pl.DataFrame: - return pl.DataFrame( - { - "bene_mbi_id": [ - "MBI-A", - "MBI-B", - "MBI-C", - ], - "bene_sex_cd": ["1", "2", "0"], - "bene_race_cd": ["1", "2", "5"], - "bene_dob": [ - date(1950, 1, 15), - date(1960, 3, 20), - date(1970, 7, 4), - ], - "bene_death_dt": [ - date(2023, 11, 1), - None, - None, - ], - "bene_part_a_enrlmt_bgn_dt": [ - date(2020, 1, 1), - date(2021, 7, 1), - date(2022, 1, 1), - ], - "mdcr_covg_a_end_dt": [ - date(2025, 12, 31), - date(2025, 6, 30), - date(2025, 12, 31), - ], - "orec": ["0", "1", "2"], - "dual_01": ["00", "01", "02"], - "ms_cd": ["10", "20", "30"], - "bene_1st_name": [ - "JOHN", - "JANE", - "BOB", - ], - "bene_midl_name": ["A", "B", "C"], - "bene_last_name": [ - "DOE", - "SMITH", - "JONES", - ], - "bene_line_1_adr": [ - "123 ELM", - "456 OAK", - "789 PINE", - ], - "geo_zip_plc_name": [ - "SPRINGFIELD", - "SHELBYVILLE", - "CAPITAL CITY", - ], - "geo_usps_state_cd": [ - "IL", - "IL", - "IL", - ], - "geo_zip5_cd": [ - "62701", - "62565", - "62700", - ], - } - ) - - def test_gender_crosswalk(self, staged_demographics: pl.DataFrame) -> None: - result = eligibility( - bcda___stg_beneficiary_demographics=staged_demographics, - ) - genders = result.sort("person_id")["gender"].to_list() - assert genders == [ - "male", - "female", - "unknown", - ] - - def test_race_crosswalk(self, staged_demographics: pl.DataFrame) -> None: - result = eligibility( - bcda___stg_beneficiary_demographics=staged_demographics, - ) - races = result.sort("person_id")["race"].to_list() - assert races == [ - "white", - "black", - "hispanic", - ] - - def test_death_flag(self, staged_demographics: pl.DataFrame) -> None: - result = eligibility( - bcda___stg_beneficiary_demographics=staged_demographics, - ) - flags = result.sort("person_id")["death_flag"].to_list() - assert flags == [1, 0, 0] - - def test_person_member_id_match(self, staged_demographics: pl.DataFrame) -> None: - result = eligibility( - bcda___stg_beneficiary_demographics=staged_demographics, - ) - assert result["person_id"].to_list() == result["member_id"].to_list() - - def test_payer_fields(self, staged_demographics: pl.DataFrame) -> None: - result = eligibility( - bcda___stg_beneficiary_demographics=staged_demographics, - ) - for col in ("payer", "payer_type", "plan"): - vals = result[col].unique().to_list() - assert vals == ["medicare"] - - def test_data_source(self, staged_demographics: pl.DataFrame) -> None: - result = eligibility( - bcda___stg_beneficiary_demographics=staged_demographics, - ) - vals = result["data_source"].unique().to_list() - assert vals == ["medicare bcda"] - - def test_name_fields(self, staged_demographics: pl.DataFrame) -> None: - result = eligibility( - bcda___stg_beneficiary_demographics=staged_demographics, - ) - row = result.filter(pl.col("person_id") == "MBI-A") - assert row["first_name"].to_list() == ["JOHN"] - assert row["last_name"].to_list() == ["DOE"] - assert row["middle_name"].to_list() == ["A"] - - def test_address_fields(self, staged_demographics: pl.DataFrame) -> None: - result = eligibility( - bcda___stg_beneficiary_demographics=staged_demographics, - ) - row = result.filter(pl.col("person_id") == "MBI-B") - assert row["city"].to_list() == ["SHELBYVILLE"] - assert row["state"].to_list() == ["IL"] - assert row["zip_code"].to_list() == ["62565"] - - def test_enrollment_dates(self, staged_demographics: pl.DataFrame) -> None: - result = eligibility( - bcda___stg_beneficiary_demographics=staged_demographics, - ) - row = result.filter(pl.col("person_id") == "MBI-A") - assert row["enrollment_start_date"].to_list() == [date(2020, 1, 1)] - assert row["enrollment_end_date"].to_list() == [date(2025, 12, 31)] - - def test_row_count_preserved(self, staged_demographics: pl.DataFrame) -> None: - result = eligibility( - bcda___stg_beneficiary_demographics=staged_demographics, - ) - assert result.shape[0] == staged_demographics.shape[0] - - def test_race_all_codes(self) -> None: - """Exercise all 7 race code mappings.""" - df = pl.DataFrame( - { - "bene_mbi_id": [f"M{i}" for i in range(7)], - "bene_sex_cd": ["1"] * 7, - "bene_race_cd": [ - "0", - "1", - "2", - "3", - "4", - "5", - "6", - ], - "bene_dob": [date(1950, 1, 1)] * 7, - "bene_death_dt": [None] * 7, - "bene_part_a_enrlmt_bgn_dt": [date(2023, 1, 1)] * 7, - "mdcr_covg_a_end_dt": [date(2023, 12, 31)] * 7, - "orec": ["0"] * 7, - "dual_01": ["00"] * 7, - "ms_cd": ["10"] * 7, - "bene_1st_name": ["A"] * 7, - "bene_midl_name": ["B"] * 7, - "bene_last_name": ["C"] * 7, - "bene_line_1_adr": ["ADDR"] * 7, - "geo_zip_plc_name": ["CITY"] * 7, - "geo_usps_state_cd": ["IL"] * 7, - "geo_zip5_cd": ["60601"] * 7, - } - ) - result = eligibility( - bcda___stg_beneficiary_demographics=df, - ) - races = result.sort("person_id")["race"].to_list() - assert races == [ - "unknown", - "white", - "black", - "other", - "asian", - "hispanic", - "native", - ] - - def test_key_output_columns(self, staged_demographics: pl.DataFrame) -> None: - result = eligibility( - bcda___stg_beneficiary_demographics=staged_demographics, - ) - expected = { - "person_id", - "member_id", - "gender", - "race", - "birth_date", - "death_date", - "death_flag", - "enrollment_start_date", - "enrollment_end_date", - "payer", - "payer_type", - "plan", - "original_reason_entitlement_code", - "dual_status_code", - "medicare_status_code", - "first_name", - "middle_name", - "last_name", - "address", - "city", - "state", - "zip_code", - "data_source", - } - assert expected.issubset(set(result.columns)) - - def test_entitlement_codes(self, staged_demographics: pl.DataFrame) -> None: - result = eligibility( - bcda___stg_beneficiary_demographics=staged_demographics, - ) - row = result.filter(pl.col("person_id") == "MBI-A") - assert row["original_reason_entitlement_code"].to_list() == ["0"] - assert row["dual_status_code"].to_list() == ["00"] - assert row["medicare_status_code"].to_list() == ["10"] - - -# ════════════════════════════════════════════════════════════════ -# Part 3: bcda.express.flatten — coverage gap tests -# ════════════════════════════════════════════════════════════════ - - class TestGetExtQuantityMatch: """_get_ext_quantity: cover the return-value path (line 155).""" @@ -2730,78 +1671,3 @@ class TestFlattenExport: # ── v3 coverage shape through the CCLF demographics stage ─────── - - -class TestDemographicsV3Coverage: - """v3 Coverage: one row per plan; dual status on its own row.""" - - def _frames(self): - import polars as pl - - patient = pl.DataFrame( - { - "bene_mbi_id": ["M1", "M2", "M3"], - "bene_sex_cd": ["1", "2", "1"], - "bene_race_cd": ["1", "2", None], - "bene_dob": [None, None, None], - "bene_death_dt": [None, None, None], - "bene_1st_name": ["a", "b", "c"], - "bene_midl_name": [None, None, None], - "bene_last_name": ["x", "y", "z"], - "bene_line_1_adr": [None, None, None], - "geo_zip_plc_name": [None, None, None], - "geo_usps_state_cd": [None, None, None], - "geo_zip5_cd": [None, None, None], - } - ) - coverage = pl.DataFrame( - { - "subscriber_id": ["M1", "M1", "M1", "M2", "M2", "M3"], - "class_plan": [ - "Part C", - "Part B", - "Part A", - "Part B", - "Dual Medicare/Medicaid", - "Part A", - ], - "ms_cd": [None, "10", "20", "10", None, "30"], - "orec": [None, "0", "1", "0", None, "2"], - "dual_01": [None, None, None, None, None, "04"], - "dual_stus_cd": [None, None, None, None, "08", None], - "bene_part_a_enrlmt_bgn_dt": [None, None, None, None, None, None], - "mdcr_covg_a_end_dt": [None, None, None, None, None, None], - } - ) - return patient, coverage - - def test_part_a_preferred_and_dual_joined(self) -> None: - from bcda.express.cclf import eligibility, stg_beneficiary_demographics - - patient, coverage = self._frames() - demo = stg_beneficiary_demographics(patient, coverage).sort("bene_mbi_id") - assert demo["ms_cd"].to_list() == [ - "20", - "10", - "30", - ] # M1 → Part A row, not Part C/B - assert demo["dual_stus_cd_dual"].to_list() == [None, "08", None] - elig = eligibility(demo).sort("person_id") - assert elig["dual_status_code"].to_list() == [None, "08", "04"] - assert elig["medicare_status_code"].to_list() == ["20", "10", "30"] - assert elig["original_reason_entitlement_code"].to_list() == ["1", "0", "2"] - - def test_v2_frames_without_plan_columns_still_work(self) -> None: - import polars as pl - - from bcda.express.cclf import eligibility, stg_beneficiary_demographics - - patient, coverage = self._frames() - v2cov = coverage.drop("class_plan", "dual_stus_cd").unique( - subset=["subscriber_id"], keep="first" - ) - demo = stg_beneficiary_demographics(patient, v2cov) - assert "dual_stus_cd_dual" not in demo.columns - elig = eligibility(demo).sort("person_id") - assert elig["dual_status_code"].to_list() == [None, None, "04"] - assert isinstance(elig, pl.DataFrame) diff --git a/tests/bcda/test_pipe_cclf.py b/tests/bcda/test_pipe_cclf.py index c01c6c0..a206a6a 100644 --- a/tests/bcda/test_pipe_cclf.py +++ b/tests/bcda/test_pipe_cclf.py @@ -9,21 +9,22 @@ import pytest class TestBuildPipeline: def test_build_returns_pipeline(self) -> None: - """_build_pipeline() returns a Pipeline with 12 exprs.""" + """_build_pipeline() returns a Pipeline with 13 exprs.""" from bcda.pipe.cclf import _build_pipeline pipeline = _build_pipeline() assert hasattr(pipeline, "exprs") - assert len(pipeline.exprs) == 12 + assert len(pipeline.exprs) == 13 assert hasattr(pipeline, "run") def test_expr_names(self) -> None: - """All 12 expected expression names are present.""" + """All 13 expected expression names are present.""" from bcda.pipe.cclf import _build_pipeline pipeline = _build_pipeline() names = [e.name for e in pipeline.exprs] assert "bcda._stg_beneficiary_xref" in names + assert "bcda._stg_claim_header" in names assert "bcda._stg_part_a_header" in names assert "bcda._int_institutional_medical_claim" in names assert "bcda._stg_part_b_physician_header" in names @@ -52,7 +53,7 @@ class TestLazyGetattr: pipeline = cclf_mod.pipeline assert hasattr(pipeline, "exprs") - assert len(pipeline.exprs) == 12 + assert len(pipeline.exprs) == 13 def test_run_attr(self) -> None: """Accessing .run triggers lazy init and returns callable.""" @@ -76,3 +77,78 @@ class TestLazyGetattr: match="has no attribute 'nonexistent'", ): _ = cclf_mod.nonexistent + + +class TestRunOnOfficialSamples: + """The whole connector against CMS's v3 sample files through DuckDB. + + Every expression's output is checked against its contract by the + runner, so this is the proof that the v3-flattened tables carry + what the connector reads and that the outputs are input_layer-shaped. + """ + + def test_end_to_end(self, tmp_path) -> None: + import shutil + from pathlib import Path + + import duckdb + + from aco.lake.context import DuckDBContext + from aco.load.bcda import load_bcda + from bcda.pipe.cclf import pipeline + + samples = Path(__file__).resolve().parents[2] / "dev" / "vendor" / "bcda-site" + src = tmp_path / "ndjson" + src.mkdir() + for rtype in ("Patient", "Coverage", "ExplanationOfBenefit"): + shutil.copy(samples / f"v3_{rtype}.ndjson", src / f"{rtype}.ndjson") + db = tmp_path / "aco.duckdb" + from unittest.mock import patch + + with patch("conf.path", side_effect=lambda k: tmp_path / "store"): + load_bcda(ndjson_dir=src, database=str(db)) + + ctx = DuckDBContext(database=str(db), read_only=True) + cache = pipeline.run(ctx.load) + assert set(cache) >= { + "bcda._stg_claim_header", + "bcda.medical_claim", + "bcda.pharmacy_claim", + "bcda.eligibility", + } + header = cache["bcda._stg_claim_header"] + # 36 NotFinalAction sample claims are dropped; the rest are routed + assert len(header) == 233 + assert set(header["claim_family"].to_list()) == { + "institutional", + "professional", + "dme", + "pharmacy", + } + medical = cache["bcda.medical_claim"] + assert medical["claim_id"].n_unique() == 131 + assert set(medical["claim_type"].to_list()) == {"institutional", "professional"} + inst = medical.filter(medical["claim_type"] == "institutional") + assert inst["bill_type_code"].null_count() == 0 + assert ( + inst["bill_type_code"].str.len_chars().max() == 4 + ) # NUBC, not fac+clsfctn+freq + assert inst["admission_date"].null_count() == 0 + assert inst["paid_amount"].null_count() == 0 + assert inst["deductible_amount"].null_count() == 0 + prof = medical.filter(medical["claim_type"] == "professional") + assert prof["place_of_service_code"].null_count() < len(prof) * 0.05 + assert prof["rendering_npi"].null_count() == 0 + assert len(cache["bcda.pharmacy_claim"]) == 102 + rx = cache["bcda.pharmacy_claim"] + assert rx["ndc_code"].null_count() == 0 and rx["days_supply"].null_count() == 0 + assert rx["charge_amount"].null_count() == 0 # total drug cost + elig = cache["bcda.eligibility"] + assert len(elig) == 94 + assert set(elig["gender"].to_list()) == {"male", "female"} + assert elig["medicare_status_code"].null_count() < len(elig) + assert set(elig["enrollment_status"].drop_nulls().to_list()) <= { + "active", + "inactive", + } + duckdb.connect(str(db), read_only=True).close() diff --git a/tests/bcda/test_v3.py b/tests/bcda/test_v3.py index 27cd093..1dce0b9 100644 --- a/tests/bcda/test_v3.py +++ b/tests/bcda/test_v3.py @@ -1061,6 +1061,7 @@ class TestFlattenEobV3: h.clm_srvc_clsfctn_type_cd, h.clm_bill_freq_cd, ) == ("1", "1", "1", "1") + assert h.clm_bill_type_cd == "0111" def test_c4bb_code_falls_back_to_any_system(self) -> None: r = _eob( diff --git a/tests/cli/test_cli_run.py b/tests/cli/test_cli_run.py index fc75664..7768e58 100644 --- a/tests/cli/test_cli_run.py +++ b/tests/cli/test_cli_run.py @@ -143,6 +143,18 @@ class TestLoadCommands: result = runner.invoke(app, ["load", "bcda", "--ndjson-dir", "/tmp/fake"]) assert result.exit_code == 0 + def test_bcda_run_pipeline_flag(self) -> None: + with patch( + "aco.load.bcda.load_bcda", + return_value={"bcda.patient": 50, "input_layer.eligibility": 50}, + ) as load: + result = runner.invoke( + app, ["load", "bcda", "--ndjson-dir", "/tmp/fake", "--run-pipeline"] + ) + assert result.exit_code == 0 + assert load.call_args.kwargs["run_pipeline"] is True + assert "input_layer.eligibility: 50 rows" in result.output + def test_bcda_not_found(self) -> None: with patch("aco.load.bcda.load_bcda", side_effect=FileNotFoundError("missing")): result = runner.invoke(app, ["load", "bcda", "--ndjson-dir", "/tmp/fake"]) diff --git a/tests/test_dab_generator.py b/tests/test_dab_generator.py index d9a25ec..4c591e1 100644 --- a/tests/test_dab_generator.py +++ b/tests/test_dab_generator.py @@ -169,7 +169,8 @@ class TestPythonResources: def test_schemas_count(self): r = self._resources() - assert len(r.schemas) == 14, f"Expected 14 schemas, got {len(r.schemas)}" + # 14 aco pipelines + the bcda connector registered when stack[bcda] is installed + assert len(r.schemas) == 15, f"Expected 15 schemas, got {len(r.schemas)}" def test_schemas_include_core(self): r = self._resources()