homelab/stack
1
0
Fork 0
Code Issues 262 Pull Requests 2 Actions Packages Projects 1 Releases Wiki Activity
Files
d08401dc53a11e0572d3a17c0df722fee3d69ad8
stack/src/bcda/express/cclf.py
kert d08401dc53 feat(bcda): BCDA v3 — client, store adoption and cancel retries, dictionary-driven v3 flatteners, streaming multi-part load 
Client defaults to /api/v3; _typeFilter on start_export/export with
type_filter_tag() for the System-Type tag; jobs() sends the bulk-data headers
BCDA requires (Prefer: respond-async, Accept: application/fhir+json — the old
call 400'd on v2 and v3) and unwraps the Task Bundle; completed_jobs() lists
reusable exports; every non-2xx raises BcdaHttpError carrying the
OperationOutcome diagnostics (still an httpx.HTTPStatusError); a poll that
hits 404 'Job has been cancelled' raises JobCancelledError; /_version and
/_health.

Store.adopt(client, job_url) records and downloads a job this store did not
start (completed jobs stay downloadable 24 h and the public sandbox entity is
shared); export(cancel_retries=N) restarts after server-side cancellation,
recording each 'cancelled' transition.

bcda.express.v3 flattens C4BB 2.2.0 resources into the existing bcda.* tables,
table-driven from the v3 Data Dictionary: v3 moved every BlueButton code off
resources/variables/<name> onto fhir/StructureDefinition|CodeSystem/<CODE> and
consolidated amounts into adjudication/total/item.adjudication, so the v2
suffix match returned NULL for almost everything. Dispatch is per resource
(is_v3); v2 payloads are unchanged. Identity follows the migration guide:
bene_id = Patient.id (BENE_SK), current MBI = us-mbi identifier without
period.end, clm_id = C4BB 'uc', clm_cntl_num pairs SharedSystem with NCH.
Race crosswalked from US Core OMB codes; Coverage is one row per plan+period
(Part A/B/C/D + Dual) with v3-native columns; EOB gains System-Type /
Final-Action / EOB-TYPE, IDR dates, header totals and PPS/IME/DSH amounts;
items gain NDC quantity, revenue-centre indicators, ANSI codes, Part D/DME
amounts.

flatten_export streams <Type>.N.ndjson parts to Parquet with model-derived
Arrow schemas (a Large-entity EOB export is 192 files / 2.2 GB); load_bcda()
gathers the newest export dir per resource type (a v3 pull is three
single-type jobs). cclf.stg_beneficiary_demographics prefers the Part A row
and joins the Dual row; eligibility reads coalesce(dual_01, dual_stus_cd).

bcda package at 100% line coverage; tests run on CMS's official v3 samples.

refs #1139
2026-10-25 13:08:13 -04:00

699 lines
26 KiB
Python
Raw Blame History

"""BCDA FHIR R4 connector — EOB + Patient + Coverage → input_layer tables.
Transforms flattened BCDA FHIR R4 resources into three input_layer
output tables that match what the CCLF pipeline produces:
- input_layer.medical_claim (Part A + Part B carrier + Part B DME)
- input_layer.pharmacy_claim (Part D)
- input_layer.eligibility (Patient + Coverage)
Unlike the CCLF pipeline (19 functions), this pipeline is simpler
(12 functions) because BCDA delivers:
- Current MBIs directly (no CCLF9 crosswalk needed)
- Final-action claims only (no ADR debit/credit reconciliation)
- Inline diagnosis codes (no CCLF4 long→wide pivot)
Claim routing uses ``nch_near_line_rec_ident_cd``:
V → Part A institutional → medical_claim (institutional)
O → Part B carrier → medical_claim (professional)
M → Part B DME → medical_claim (professional)
P → Part D pharmacy → pharmacy_claim
Pipeline stages::
BCDA FHIR R4 resources
│
▼ stg: filter by claim type, resolve MBI
│
▼ int: join header + items, map to input_layer schema
│
▼ final: medical_claim, pharmacy_claim, eligibility
"""
from __future__ import annotations
import narwhals as nw
from narwhals.typing import FrameT
# ── Helpers ──────────────────────────────────────────────────────
def _icd_indicator() -> nw.Expr:
"""Map dgns_prcdr_icd_ind to standard code type."""
return (
nw.when(nw.col("dgns_prcdr_icd_ind") == "0")
.then(nw.lit("icd-10-cm"))
.otherwise(
nw.when(nw.col("dgns_prcdr_icd_ind") == "9")
.then(nw.lit("icd-9-cm"))
.otherwise(nw.lit(None))
)
.alias("diagnosis_code_type")
)
# ── Stage 1: Beneficiary XREF ───────────────────────────────────
@nw.narwhalify
def stg_beneficiary_xref(
bcda__patient: FrameT,
) -> FrameT:
"""Build MBI crosswalk from BCDA Patient identifiers.
BCDA Patient delivers the current MBI directly in bene_mbi_id.
This deduplicates to one row per beneficiary with a stable
bene_id and the current MBI.
Operations: select, unique
Dependencies: bcda.patient
"""
return bcda__patient.select(
nw.col("bene_id"),
nw.col("bene_mbi_id").alias("current_bene_mbi_id"),
).unique(subset=["bene_id"], keep="any")
# ── Stage 2: Part A Institutional Claims ─────────────────────────
@nw.narwhalify
def stg_part_a_header(
bcda__explanation_of_benefit: FrameT,
bcda___stg_beneficiary_xref: FrameT,
) -> FrameT:
"""Filter EOB to Part A claims and resolve MBI.
Filter: nch_near_line_rec_ident_cd == 'V'
Join: bene_id → current_bene_mbi_id from xref
Operations: filter, join
Dependencies: bcda.explanation_of_benefit, bcda._stg_beneficiary_xref
"""
xref = bcda___stg_beneficiary_xref
return bcda__explanation_of_benefit.filter(
nw.col("nch_near_line_rec_ident_cd") == "V"
).join(xref, on="bene_id", how="left")
@nw.narwhalify
def int_institutional_medical_claim(
bcda___stg_part_a_header: FrameT,
bcda__eob_item: FrameT,
) -> FrameT:
"""Join Part A header + line items → medical_claim (institutional).
Operations: join, select
Dependencies: bcda._stg_part_a_header, bcda.eob_item
"""
hdr = bcda___stg_part_a_header
items = bcda__eob_item
df = hdr.join(items, on="clm_id", how="inner", suffix="_item")
# Diagnosis codes 1-12 from EOB header, 13-25 null
dx_cols = []
for i in range(1, 26):
if i <= 12:
dx_cols.append(nw.col(f"clm_dgns_{i}_cd").alias(f"diagnosis_code_{i}"))
else:
dx_cols.append(nw.lit(None).cast(nw.String).alias(f"diagnosis_code_{i}"))
# POA: slot 1 from header, rest null
poa_cols = []
for i in range(1, 26):
if i == 1:
poa_cols.append(nw.col("clm_poa_ind").alias("diagnosis_poa_1"))
else:
poa_cols.append(nw.lit(None).cast(nw.String).alias(f"diagnosis_poa_{i}"))
# Procedure: slot 1 from header, rest null
px_code_cols = [
nw.col("clm_prcdr_cd").alias("procedure_code_1"),
*[
nw.lit(None).cast(nw.String).alias(f"procedure_code_{i}")
for i in range(2, 26)
],
]
px_date_cols = [
nw.col("clm_prcdr_prfrm_dt").alias("procedure_date_1"),
*[
nw.lit(None).cast(nw.Date).alias(f"procedure_date_{i}")
for i in range(2, 26)
],
]
null_str = nw.lit(None).cast(nw.String)
null_date = nw.lit(None).cast(nw.Date)
null_float = nw.lit(None).cast(nw.Float64)
return df.select(
nw.col("clm_id").alias("claim_id"),
nw.col("clm_line_num").cast(nw.Int64).alias("claim_line_number"),
nw.lit("institutional").alias("claim_type"),
nw.col("current_bene_mbi_id").alias("person_id"),
nw.col("current_bene_mbi_id").alias("member_id"),
nw.lit("medicare").alias("payer"),
nw.lit("medicare").alias("plan"),
nw.col("clm_from_dt").alias("claim_start_date"),
nw.col("clm_thru_dt").alias("claim_end_date"),
nw.col("clm_line_from_dt").alias("claim_line_start_date"),
nw.col("clm_line_thru_dt").alias("claim_line_end_date"),
nw.col("clm_from_dt").alias("admission_date"),
nw.col("clm_thru_dt").alias("discharge_date"),
nw.col("clm_admsn_src_cd").alias("admit_source_code"),
nw.col("clm_admsn_type_cd").alias("admit_type_code"),
nw.col("bene_ptnt_stus_cd").alias("discharge_disposition_code"),
null_str.alias("place_of_service_code"),
# bill_type_code = fac + clsfctn + freq
(
nw.col("clm_bill_fac_type_cd").cast(nw.String).fill_null("")
+ nw.col("clm_bill_clsfctn_cd").cast(nw.String).fill_null("")
+ nw.col("clm_bill_freq_cd").cast(nw.String).fill_null("")
).alias("bill_type_code"),
nw.lit("ms-drg").alias("drg_code_type"),
nw.col("dgns_drg_cd").cast(nw.String).str.slice(-3).alias("drg_code"),
nw.col("clm_line_prod_rev_ctr_cd").alias("revenue_center_code"),
nw.col("clm_line_srvc_unit_qty").cast(nw.Int64).alias("service_unit_quantity"),
nw.col("clm_line_hcpcs_cd").alias("hcpcs_code"),
nw.col("hcpcs_1_mdfr_cd").alias("hcpcs_modifier_1"),
nw.col("hcpcs_2_mdfr_cd").alias("hcpcs_modifier_2"),
nw.col("hcpcs_3_mdfr_cd").alias("hcpcs_modifier_3"),
nw.col("hcpcs_4_mdfr_cd").alias("hcpcs_modifier_4"),
nw.col("hcpcs_5_mdfr_cd").alias("hcpcs_modifier_5"),
nw.col("oprtg_prvdr_npi_num").alias("rendering_npi"),
null_str.alias("rendering_tin"),
nw.col("fac_prvdr_npi_num").alias("billing_npi"),
null_str.alias("billing_tin"),
nw.col("fac_prvdr_npi_num").alias("facility_npi"),
null_date.alias("paid_date"),
nw.col("clm_line_cvrd_pd_amt").cast(nw.Float64).alias("paid_amount"),
null_float.alias("allowed_amount"),
nw.col("clm_mdcr_instnl_tot_chrg_amt").cast(nw.Float64).alias("charge_amount"),
null_float.alias("coinsurance_amount"),
null_float.alias("copayment_amount"),
null_float.alias("deductible_amount"),
null_float.alias("total_cost_amount"),
_icd_indicator(),
*dx_cols,
*poa_cols,
nw.when(nw.col("dgns_prcdr_icd_ind") == "0")
.then(nw.lit("icd-10-pcs"))
.otherwise(
nw.when(nw.col("dgns_prcdr_icd_ind") == "9")
.then(nw.lit("icd-9-pcs"))
.otherwise(nw.lit(None))
)
.alias("procedure_code_type"),
*px_code_cols,
*px_date_cols,
nw.lit(1).alias("in_network_flag"),
nw.lit("medicare bcda").alias("data_source"),
null_str.alias("file_name"),
null_date.alias("file_date"),
nw.lit(None).cast(nw.Datetime).alias("ingest_datetime"),
)
# ── Stage 3: Part B Physician Claims ────────────────────────────
@nw.narwhalify
def stg_part_b_physician_header(
bcda__explanation_of_benefit: FrameT,
bcda___stg_beneficiary_xref: FrameT,
) -> FrameT:
"""Filter EOB to Part B carrier claims and resolve MBI.
Filter: nch_near_line_rec_ident_cd == 'O'
Operations: filter, join
Dependencies: bcda.explanation_of_benefit, bcda._stg_beneficiary_xref
"""
xref = bcda___stg_beneficiary_xref
return bcda__explanation_of_benefit.filter(
nw.col("nch_near_line_rec_ident_cd") == "O"
).join(xref, on="bene_id", how="left")
@nw.narwhalify
def int_physician_medical_claim(
bcda___stg_part_b_physician_header: FrameT,
bcda__eob_item: FrameT,
) -> FrameT:
"""Join Part B physician header + line items → medical_claim.
claim_type = 'professional'. CCLF5 equivalent.
Operations: join, select
Dependencies: bcda._stg_part_b_physician_header, bcda.eob_item
"""
hdr = bcda___stg_part_b_physician_header
items = bcda__eob_item
df = hdr.join(items, on="clm_id", how="inner", suffix="_item")
# Diagnosis codes 1-12 from EOB header, 13-25 null
dx_cols = []
for i in range(1, 26):
if i <= 12:
dx_cols.append(nw.col(f"clm_dgns_{i}_cd").alias(f"diagnosis_code_{i}"))
else:
dx_cols.append(nw.lit(None).cast(nw.String).alias(f"diagnosis_code_{i}"))
poa_cols = [
nw.lit(None).cast(nw.String).alias(f"diagnosis_poa_{i}") for i in range(1, 26)
]
px_code_cols = [
nw.lit(None).cast(nw.String).alias(f"procedure_code_{i}") for i in range(1, 26)
]
px_date_cols = [
nw.lit(None).cast(nw.Date).alias(f"procedure_date_{i}") for i in range(1, 26)
]
null_str = nw.lit(None).cast(nw.String)
null_date = nw.lit(None).cast(nw.Date)
null_float = nw.lit(None).cast(nw.Float64)
return df.select(
nw.col("clm_id").alias("claim_id"),
nw.col("clm_line_num").cast(nw.Int64).alias("claim_line_number"),
nw.lit("professional").alias("claim_type"),
nw.col("current_bene_mbi_id").alias("person_id"),
nw.col("current_bene_mbi_id").alias("member_id"),
nw.lit("medicare").alias("payer"),
nw.lit("medicare").alias("plan"),
nw.col("clm_from_dt").alias("claim_start_date"),
nw.col("clm_thru_dt").alias("claim_end_date"),
nw.col("clm_line_from_dt").alias("claim_line_start_date"),
nw.col("clm_line_thru_dt").alias("claim_line_end_date"),
null_date.alias("admission_date"),
null_date.alias("discharge_date"),
null_str.alias("admit_source_code"),
null_str.alias("admit_type_code"),
null_str.alias("discharge_disposition_code"),
nw.col("clm_pos_cd").alias("place_of_service_code"),
null_str.alias("bill_type_code"),
null_str.alias("drg_code_type"),
null_str.alias("drg_code"),
null_str.alias("revenue_center_code"),
nw.col("clm_line_srvc_unit_qty").cast(nw.Int64).alias("service_unit_quantity"),
nw.col("clm_line_hcpcs_cd").alias("hcpcs_code"),
nw.col("hcpcs_1_mdfr_cd").alias("hcpcs_modifier_1"),
nw.col("hcpcs_2_mdfr_cd").alias("hcpcs_modifier_2"),
nw.col("hcpcs_3_mdfr_cd").alias("hcpcs_modifier_3"),
nw.col("hcpcs_4_mdfr_cd").alias("hcpcs_modifier_4"),
nw.col("hcpcs_5_mdfr_cd").alias("hcpcs_modifier_5"),
nw.col("rndrg_prvdr_npi_num").alias("rendering_npi"),
nw.col("clm_rndrg_prvdr_tax_num").alias("rendering_tin"),
nw.col("carr_clm_blg_npi_num").alias("billing_npi"),
null_str.alias("billing_tin"),
null_str.alias("facility_npi"),
null_date.alias("paid_date"),
nw.col("clm_line_cvrd_pd_amt").cast(nw.Float64).alias("paid_amount"),
nw.col("clm_line_alowd_chrg_amt").cast(nw.Float64).alias("allowed_amount"),
null_float.alias("charge_amount"),
null_float.alias("coinsurance_amount"),
null_float.alias("copayment_amount"),
null_float.alias("deductible_amount"),
null_float.alias("total_cost_amount"),
_icd_indicator(),
*dx_cols,
*poa_cols,
null_str.alias("procedure_code_type"),
*px_code_cols,
*px_date_cols,
nw.lit(1).alias("in_network_flag"),
nw.lit("medicare bcda").alias("data_source"),
null_str.alias("file_name"),
null_date.alias("file_date"),
nw.lit(None).cast(nw.Datetime).alias("ingest_datetime"),
)
# ── Stage 4: Part B DME Claims ──────────────────────────────────
@nw.narwhalify
def stg_part_b_dme_header(
bcda__explanation_of_benefit: FrameT,
bcda___stg_beneficiary_xref: FrameT,
) -> FrameT:
"""Filter EOB to Part B DME claims and resolve MBI.
Filter: nch_near_line_rec_ident_cd == 'M'
Operations: filter, join
Dependencies: bcda.explanation_of_benefit, bcda._stg_beneficiary_xref
"""
xref = bcda___stg_beneficiary_xref
return bcda__explanation_of_benefit.filter(
nw.col("nch_near_line_rec_ident_cd") == "M"
).join(xref, on="bene_id", how="left")
@nw.narwhalify
def int_dme_medical_claim(
bcda___stg_part_b_dme_header: FrameT,
bcda__eob_item: FrameT,
) -> FrameT:
"""Join Part B DME header + line items → medical_claim.
claim_type = 'professional'. CCLF6 equivalent.
DME claims have no inline diagnosis or procedure codes.
Operations: join, select
Dependencies: bcda._stg_part_b_dme_header, bcda.eob_item
"""
hdr = bcda___stg_part_b_dme_header
items = bcda__eob_item
df = hdr.join(items, on="clm_id", how="inner", suffix="_item")
null_str = nw.lit(None).cast(nw.String)
null_date = nw.lit(None).cast(nw.Date)
null_float = nw.lit(None).cast(nw.Float64)
dx_cols = [null_str.alias(f"diagnosis_code_{i}") for i in range(1, 26)]
poa_cols = [null_str.alias(f"diagnosis_poa_{i}") for i in range(1, 26)]
px_code_cols = [null_str.alias(f"procedure_code_{i}") for i in range(1, 26)]
px_date_cols = [null_date.alias(f"procedure_date_{i}") for i in range(1, 26)]
return df.select(
nw.col("clm_id").alias("claim_id"),
nw.col("clm_line_num").cast(nw.Int64).alias("claim_line_number"),
nw.lit("professional").alias("claim_type"),
nw.col("current_bene_mbi_id").alias("person_id"),
nw.col("current_bene_mbi_id").alias("member_id"),
nw.lit("medicare").alias("payer"),
nw.lit("medicare").alias("plan"),
nw.col("clm_from_dt").alias("claim_start_date"),
nw.col("clm_thru_dt").alias("claim_end_date"),
nw.col("clm_line_from_dt").alias("claim_line_start_date"),
nw.col("clm_line_thru_dt").alias("claim_line_end_date"),
null_date.alias("admission_date"),
null_date.alias("discharge_date"),
null_str.alias("admit_source_code"),
null_str.alias("admit_type_code"),
null_str.alias("discharge_disposition_code"),
nw.col("clm_pos_cd").alias("place_of_service_code"),
null_str.alias("bill_type_code"),
null_str.alias("drg_code_type"),
null_str.alias("drg_code"),
null_str.alias("revenue_center_code"),
nw.lit(None).cast(nw.Int64).alias("service_unit_quantity"),
nw.col("clm_line_hcpcs_cd").alias("hcpcs_code"),
null_str.alias("hcpcs_modifier_1"),
null_str.alias("hcpcs_modifier_2"),
null_str.alias("hcpcs_modifier_3"),
null_str.alias("hcpcs_modifier_4"),
null_str.alias("hcpcs_modifier_5"),
nw.col("ordrg_prvdr_npi_num").alias("rendering_npi"),
null_str.alias("rendering_tin"),
nw.col("payto_prvdr_npi_num").alias("billing_npi"),
null_str.alias("billing_tin"),
null_str.alias("facility_npi"),
null_date.alias("paid_date"),
nw.col("clm_line_cvrd_pd_amt").cast(nw.Float64).alias("paid_amount"),
nw.col("clm_line_alowd_chrg_amt").cast(nw.Float64).alias("allowed_amount"),
null_float.alias("charge_amount"),
null_float.alias("coinsurance_amount"),
null_float.alias("copayment_amount"),
null_float.alias("deductible_amount"),
null_float.alias("total_cost_amount"),
null_str.alias("diagnosis_code_type"),
*dx_cols,
*poa_cols,
null_str.alias("procedure_code_type"),
*px_code_cols,
*px_date_cols,
nw.lit(1).alias("in_network_flag"),
nw.lit("medicare bcda").alias("data_source"),
null_str.alias("file_name"),
null_date.alias("file_date"),
nw.lit(None).cast(nw.Datetime).alias("ingest_datetime"),
)
# ── Stage 5: Union medical claims ───────────────────────────────
@nw.narwhalify
def medical_claim(
bcda___int_institutional_medical_claim: FrameT,
bcda___int_physician_medical_claim: FrameT,
bcda___int_dme_medical_claim: FrameT,
) -> FrameT:
"""Union institutional + professional + DME into medical_claim.
Operations: concat
Dependencies: bcda._int_institutional_medical_claim,
bcda._int_physician_medical_claim,
bcda._int_dme_medical_claim
"""
frames = [
bcda___int_institutional_medical_claim,
bcda___int_physician_medical_claim,
bcda___int_dme_medical_claim,
]
# Cast all columns to String to align schemas — null-only
# columns may be inferred as Int32 by the backend engine.
cols = frames[0].collect_schema().names()
aligned = [f.with_columns(nw.col(c).cast(nw.String) for c in cols) for f in frames]
return nw.concat(aligned)
# ── Stage 6: Part D Pharmacy Claims ─────────────────────────────
@nw.narwhalify
def stg_part_d_header(
bcda__explanation_of_benefit: FrameT,
bcda___stg_beneficiary_xref: FrameT,
) -> FrameT:
"""Filter EOB to Part D pharmacy claims and resolve MBI.
Filter: nch_near_line_rec_ident_cd == 'P'
Operations: filter, join
Dependencies: bcda.explanation_of_benefit, bcda._stg_beneficiary_xref
"""
xref = bcda___stg_beneficiary_xref
return bcda__explanation_of_benefit.filter(
nw.col("nch_near_line_rec_ident_cd") == "P"
).join(xref, on="bene_id", how="left")
@nw.narwhalify
def pharmacy_claim(
bcda___stg_part_d_header: FrameT,
bcda__eob_item: FrameT,
) -> FrameT:
"""Join Part D header + line items → pharmacy_claim.
CCLF7 equivalent. BCDA FHIR careTeam already resolves
provider identifiers to NPI type, so no qualifier check
is needed (unlike CCLF7 which requires
prvdr_srvc_id_qlfyr_cd == '01').
Operations: join, select
Dependencies: bcda._stg_part_d_header, bcda.eob_item
"""
hdr = bcda___stg_part_d_header
items = bcda__eob_item
df = hdr.join(items, on="clm_id", how="inner", suffix="_item")
null_float = nw.lit(None).cast(nw.Float64)
return df.select(
nw.col("pde_id").alias("claim_id"),
nw.lit(1).alias("claim_line_number"),
nw.col("current_bene_mbi_id").alias("person_id"),
nw.col("current_bene_mbi_id").alias("member_id"),
nw.lit("medicare").alias("payer"),
nw.lit("medicare").alias("plan"),
nw.col("clm_prsbng_prvdr_gnrc_id_num")
.cast(nw.String)
.alias("prescribing_provider_npi"),
nw.col("clm_srvc_prvdr_gnrc_id_num")
.cast(nw.String)
.alias("dispensing_provider_npi"),
nw.col("clm_line_from_dt").alias("dispensing_date"),
nw.col("clm_line_ndc_cd").alias("ndc_code"),
nw.col("clm_line_srvc_unit_qty").cast(nw.Int64).alias("quantity"),
nw.col("clm_line_days_suply_qty").cast(nw.Int64).alias("days_supply"),
nw.col("clm_line_rx_fill_num").cast(nw.Int64).alias("refills"),
nw.col("clm_line_from_dt").alias("paid_date"),
nw.col("clm_line_bene_pmt_amt").cast(nw.Float64).alias("paid_amount"),
null_float.alias("allowed_amount"),
null_float.alias("charge_amount"),
null_float.alias("coinsurance_amount"),
nw.col("clm_line_bene_pmt_amt").cast(nw.Float64).alias("copayment_amount"),
null_float.alias("deductible_amount"),
nw.lit(1).alias("in_network_flag"),
nw.lit("medicare bcda").alias("data_source"),
nw.lit(None).cast(nw.String).alias("file_name"),
nw.lit(None).cast(nw.Date).alias("file_date"),
nw.lit(None).cast(nw.Datetime).alias("ingest_datetime"),
)
# ── Stage 7: Eligibility ────────────────────────────────────────
@nw.narwhalify
def stg_beneficiary_demographics(
bcda__patient: FrameT,
bcda__coverage: FrameT,
) -> FrameT:
"""Join Patient + Coverage for demographics and enrollment.
Patient provides: name, dob, sex, race, death_dt, address
Coverage provides: ms_cd, orec, enrollment periods
Join on bene_mbi_id = subscriber_id. Coverage has multiple rows
per beneficiary (Part A, B, D in v2; Part A, B, C, D and a Dual
Medicare/Medicaid row per period in v3). The Part A row is the
demographics row — it carries ms_cd / orec / entitlement — with
Part B as the fallback; the v3 Dual row is joined separately for
``dual_stus_cd`` (v2 keeps ``dual_01`` on the Part A row).
Operations: filter, sort, unique, join
Dependencies: bcda.patient, bcda.coverage
"""
cols = bcda__coverage.columns
parts = bcda__coverage
if "class_plan" in cols:
parts = parts.filter(nw.col("class_plan").is_in(["Part A", "Part B"])).sort(
"class_plan" # "Part A" < "Part B"
)
parts = parts.unique(subset=["subscriber_id"], keep="first", maintain_order=True)
joined = bcda__patient.join(
parts,
left_on="bene_mbi_id",
right_on="subscriber_id",
how="left",
suffix="_cov",
)
if "dual_stus_cd" in cols and "class_plan" in cols:
dual = (
bcda__coverage.filter(nw.col("class_plan") == "Dual Medicare/Medicaid")
.select("subscriber_id", nw.col("dual_stus_cd").alias("dual_stus_cd_dual"))
.unique(subset=["subscriber_id"], keep="any")
)
joined = joined.join(
dual,
left_on="bene_mbi_id",
right_on="subscriber_id",
how="left",
suffix="_dual",
)
return joined.unique(subset=["bene_mbi_id"], keep="any")
@nw.narwhalify
def eligibility(
bcda___stg_beneficiary_demographics: FrameT,
) -> FrameT:
"""Map Patient + Coverage to eligibility schema.
Gender crosswalk: 1→male, 2→female, else unknown
Race crosswalk: 1→white, 2→black, 3→other, 4→asian,
5→hispanic, 6→native, else unknown
Operations: select
Dependencies: bcda._stg_beneficiary_demographics
"""
df = bcda___stg_beneficiary_demographics
null_str = nw.lit(None).cast(nw.String)
# v2 keeps dual status on the Part A row (dual_01); v3 carries it on
# the Dual Medicare/Medicaid row that stg_beneficiary_demographics
# joins in as dual_stus_cd_dual.
dual = nw.col("dual_01")
if "dual_stus_cd_dual" in df.columns:
dual = nw.coalesce(nw.col("dual_01"), nw.col("dual_stus_cd_dual"))
gender = (
nw.when(nw.col("bene_sex_cd") == "1")
.then(nw.lit("male"))
.otherwise(
nw.when(nw.col("bene_sex_cd") == "2")
.then(nw.lit("female"))
.otherwise(nw.lit("unknown"))
)
.alias("gender")
)
race = (
nw.when(nw.col("bene_race_cd") == "1")
.then(nw.lit("white"))
.otherwise(
nw.when(nw.col("bene_race_cd") == "2")
.then(nw.lit("black"))
.otherwise(
nw.when(nw.col("bene_race_cd") == "3")
.then(nw.lit("other"))
.otherwise(
nw.when(nw.col("bene_race_cd") == "4")
.then(nw.lit("asian"))
.otherwise(
nw.when(nw.col("bene_race_cd") == "5")
.then(nw.lit("hispanic"))
.otherwise(
nw.when(nw.col("bene_race_cd") == "6")
.then(nw.lit("native"))
.otherwise(nw.lit("unknown"))
)
)
)
)
)
.alias("race")
)
return df.select(
nw.col("bene_mbi_id").alias("person_id"),
nw.col("bene_mbi_id").alias("member_id"),
null_str.alias("subscriber_id"),
gender,
race,
nw.col("bene_dob").alias("birth_date"),
nw.col("bene_death_dt").alias("death_date"),
nw.when(~nw.col("bene_death_dt").is_null())
.then(nw.lit(1))
.otherwise(nw.lit(0))
.alias("death_flag"),
nw.col("bene_part_a_enrlmt_bgn_dt").alias("enrollment_start_date"),
nw.col("mdcr_covg_a_end_dt").alias("enrollment_end_date"),
nw.lit("medicare").alias("payer"),
nw.lit("medicare").alias("payer_type"),
nw.lit("medicare").alias("plan"),
nw.col("orec").alias("original_reason_entitlement_code"),
dual.alias("dual_status_code"),
nw.col("ms_cd").alias("medicare_status_code"),
null_str.alias("group_id"),
null_str.alias("group_name"),
null_str.alias("name_suffix"),
nw.col("bene_1st_name").alias("first_name"),
nw.col("bene_midl_name").alias("middle_name"),
nw.col("bene_last_name").alias("last_name"),
null_str.alias("social_security_number"),
null_str.alias("subscriber_relation"),
nw.col("bene_line_1_adr").alias("address"),
nw.col("geo_zip_plc_name").alias("city"),
nw.col("geo_usps_state_cd").alias("state"),
nw.col("geo_zip5_cd").alias("zip_code"),
null_str.alias("phone"),
null_str.alias("email"),
null_str.alias("ethnicity"),
nw.lit("medicare bcda").alias("data_source"),
nw.lit(None).cast(nw.String).alias("file_name"),
nw.lit(None).cast(nw.Date).alias("file_date"),
nw.lit(None).cast(nw.Datetime).alias("ingest_datetime"),
nw.lit(None).cast(nw.Int64).alias("hospice_flag"),
nw.lit(None).cast(nw.Int64).alias("institutional_snp_flag"),
nw.lit(None).cast(nw.Int64).alias("long_term_institutional_flag"),
null_str.alias("enrollment_status"),
)
Reference in New Issue View Git Blame Copy Permalink