data: ASP ingestion — strict skin sub filter, full provenance, 86 canonical URLs
Some checks failed
CI / skinny-install (aco) (push) Successful in 50s
CI / skinny-install (bls) (push) Successful in 32s
CI / skinny-install (cms) (push) Successful in 28s
CI / skinny-install (cli) (push) Successful in 36s
CI / skinny-install (perf) (push) Successful in 27s
CI / skinny-install (conf) (push) Successful in 35s
CI / skinny-install (pfs) (push) Successful in 36s
CI / skinny-install (rex) (push) Successful in 35s
CI / lint-test (push) Successful in 1m24s
CI / skinny-install (api) (push) Successful in 34s
CI / skinny-install (bcda) (push) Successful in 34s
CI / skinny-install (bib) (push) Successful in 33s
CI / skinny-install (ccw) (push) Successful in 32s
Deploy / build-scan-report (push) Has been cancelled
Some checks failed
CI / skinny-install (aco) (push) Successful in 50s
CI / skinny-install (bls) (push) Successful in 32s
CI / skinny-install (cms) (push) Successful in 28s
CI / skinny-install (cli) (push) Successful in 36s
CI / skinny-install (perf) (push) Successful in 27s
CI / skinny-install (conf) (push) Successful in 35s
CI / skinny-install (pfs) (push) Successful in 36s
CI / skinny-install (rex) (push) Successful in 35s
CI / lint-test (push) Successful in 1m24s
CI / skinny-install (api) (push) Successful in 34s
CI / skinny-install (bcda) (push) Successful in 34s
CI / skinny-install (bib) (push) Successful in 33s
CI / skinny-install (ccw) (push) Successful in 32s
Deploy / build-scan-report (push) Has been cancelled
Rewrite ingest_asp.py with academic rigor: - Strict filter to skin_substitute_hcpcs.csv code universe (NO regex) - 28 non-skin-sub Q4 codes excluded (Q4054 darbepoetin, Q4055 epoetin, Q4074-Q4099 ESRD drugs) that were incorrectly included before - source_url and source_file columns on every output row (100% provenance) - Documented column mapping rationale across 20 years of CMS format drift - DuckDB path corrected to data/aco.duckdb Result: 1,974 observations, 68 quarters (2009-Q1–2025-Q4), 169 codes Canonical source URLs: data/cms/asp_source_urls.csv (86 CMS.gov URLs) Refs #233
This commit is contained in:
87
data/cms/asp_source_urls.csv
Normal file
87
data/cms/asp_source_urls.csv
Normal file
@@ -0,0 +1,87 @@
|
||||
quarter,url
|
||||
2026-Q2,https://www.cms.gov/files/zip/april-2026-medicare-part-b-payment-limit-files-03-24-2026-final-file.zip
|
||||
2026-Q1,https://www.cms.gov/files/zip/january-2026-medicare-part-b-payment-limit-files.zip
|
||||
2025-Q4,https://www.cms.gov/files/zip/october-2025-asp-pricing-final-file.zip
|
||||
2025-Q3,https://www.cms.gov/files/zip/july-2025-asp-pricing-file.zip
|
||||
2025-Q2,https://www.cms.gov/files/zip/april-2025-asp-pricing-file.zip
|
||||
2025-Q1,https://www.cms.gov/files/zip/january-2025-asp-pricing-file-03/11/25-final-file.zip
|
||||
2024-Q4,https://www.cms.gov/files/zip/october-2024-asp-pricing-file.zip
|
||||
2024-Q3,https://www.cms.gov/files/zip/july-2024-asp-pricing-file.zip
|
||||
2024-Q2,https://www.cms.gov/files/zip/april-2024-asp-pricing-file.zip
|
||||
2024-Q1,https://www.cms.gov/files/zip/january-2024-asp-pricing-file.zip
|
||||
2023-Q4,https://www.cms.gov/files/zip/october-2023-asp-pricing-file.zip
|
||||
2023-Q3,https://www.cms.gov/files/zip/july-2023-asp-pricing-file.zip
|
||||
2023-Q2,https://www.cms.gov/files/zip/april-2023-asp-pricing-file.zip
|
||||
2023-Q1,https://www.cms.gov/files/zip/january-2023-asp-pricing-file.zip
|
||||
2022-Q4,https://www.cms.gov/files/zip/october-2022-asp-pricing-file.zip
|
||||
2022-Q3,https://www.cms.gov/files/zip/july-2022-asp-pricing-file.zip
|
||||
2022-Q2,https://www.cms.gov/files/zip/april-2022-asp-pricing-file.zip
|
||||
2022-Q1,https://www.cms.gov/files/zip/january-2022-asp-pricing-file.zip
|
||||
2021-Q4,https://www.cms.gov/files/zip/october-2021-asp-pricing-file.zip
|
||||
2021-Q3,https://www.cms.gov/files/zip/july-2021-asp-pricing-file.zip
|
||||
2021-Q2,https://www.cms.gov/files/zip/april-2021-asp-pricing-file.zip
|
||||
2021-Q1,https://www.cms.gov/files/zip/january-2021-asp-pricing-file.zip
|
||||
2020-Q4,https://www.cms.gov/files/zip/october-2020-asp-pricing-file.zip
|
||||
2020-Q3,https://www.cms.gov/files/zip/july-2020-asp-pricing-file-updated-06012020.zip
|
||||
2020-Q2,https://www.cms.gov/files/zip/april-2020-asp-pricing-file.zip
|
||||
2020-Q1,https://www.cms.gov/files/zip/january-2020-asp-pricing-file.zip
|
||||
2019-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2019-Oct-ASP-Pricing-File.zip
|
||||
2019-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2019-July-ASP-Pricing-File.zip
|
||||
2019-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April-2019-ASP-Pricing-File.zip
|
||||
2019-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2019-January-ASP-Pricing-File.zip
|
||||
2018-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2018-Oct-ASP-Pricing-File.zip
|
||||
2018-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2018-July-ASP-Pricing-File.zip
|
||||
2018-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2018-April-ASP-Pricing-File.zip
|
||||
2018-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2018-January-ASP-Pricing-File.zip
|
||||
2017-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2017-October-ASP-Pricing-File.zip
|
||||
2017-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2017-July-ASP-Pricing-File.zip
|
||||
2017-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2017-April-ASP-Pricing.zip
|
||||
2017-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2017-January-ASP-Pricing-Files.zip
|
||||
2016-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2016-October-ASP-Pricing-File.zip
|
||||
2016-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2016-July-ASP-Pricing-File.zip
|
||||
2016-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2016-April-ASP-Pricing-File.zip
|
||||
2016-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2016-January-ASP-Pricing-File.zip
|
||||
2015-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2015-October-ASP-Pricing-File.zip
|
||||
2015-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2015-July-ASP-Pricing-File.zip
|
||||
2015-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2015-April-ASP-Pricing-File.zip
|
||||
2015-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2015-January-ASP-Pricing-File.zip
|
||||
2014-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2014-October-ASP-Pricing-File.zip
|
||||
2014-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jul-2014-ASP-Pricing-File.zip
|
||||
2014-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Apr-14-ASP-Pricing-File.zip
|
||||
2014-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jan-14-ASP-Pricing-File.zip
|
||||
2013-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2013-October-ASP-Pricing-File.zip
|
||||
2013-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2013-July-ASP-Pricing-File.zip
|
||||
2013-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Apr-13-ASP-Pricing-file.zip
|
||||
2013-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jan-2013-ASP-Pricing-File.zip
|
||||
2012-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/October-2012-ASP-Pricing-File.zip
|
||||
2012-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/July-2012-ASP-Pricing-File.zip
|
||||
2012-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April-2012-ASP-Pricing-file-revised030513.zip
|
||||
2012-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jan_2012_ASP_Pricing_File.zip
|
||||
2011-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Oct_2011_ASP_Pricing_File.zip
|
||||
2011-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/July_2011_ASP_Pricing_File.zip
|
||||
2011-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April_2011_ASP_Pricing_File.zip
|
||||
2011-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/January2011_ASP_PricingFile.zip
|
||||
2010-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/October_2010_ASP_Pricing_File.zip
|
||||
2010-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/July_2010_ASP_Pricing_File.zip
|
||||
2010-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April_2010_ASP_Pricing_File.zip
|
||||
2010-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/January_2010_ASP_Pricing_File.zip
|
||||
2009-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/October_2009_ASP_Pricing_File.zip
|
||||
2009-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/July_2009_ASP_Pricing_File.zip
|
||||
2009-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April_2009_ASP_Pricing_File_by_HCPCS.zip
|
||||
2009-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/JAN_2009_ASP_Pricing_File_by_HCPCS.zip
|
||||
2008-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/October2008ASPPricingFilebyHCPCS.zip
|
||||
2008-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/July2008ASPPricingFilebyHCPCS.zip
|
||||
2008-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April08ASPbyHCPCS.zip
|
||||
2008-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jan08ASPbyHCPCS.zip
|
||||
2007-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/October07ASPbyHCPCS.zip
|
||||
2007-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/July07ASPbyHCPCS_121707.zip
|
||||
2007-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April07ASPbyHCPCS_121707.zip
|
||||
2007-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jan07ASPbyHCPCS_121707.zip
|
||||
2006-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/oct06asp_hcpcs.zip
|
||||
2006-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/july06asp_hcpcs.zip
|
||||
2006-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/apr06pricing.zip
|
||||
2006-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jan06pricing.zip
|
||||
2005-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/oct05aspbyhcpcs033106v2.zip
|
||||
2005-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/jul05aspbyhcpcs_032906.zip
|
||||
2005-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/apr05aspbyhcpcsv8_033106.zip
|
||||
2005-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/jan05aspbyhcpcsv6_033106.zip
|
||||
|
353
dev/scripts/ingest_asp.py
Normal file
353
dev/scripts/ingest_asp.py
Normal file
@@ -0,0 +1,353 @@
|
||||
"""Ingest CMS Average Sales Price (ASP) Drug Pricing Files.
|
||||
|
||||
Reads all quarterly ASP ZIP files from data/cms/asp/, normalises column
|
||||
names across 20+ years of CMS format drift, filters **strictly** to
|
||||
skin substitute HCPCS codes defined in data/cms/skin_substitute_hcpcs.csv,
|
||||
and produces:
|
||||
|
||||
* data/cms/asp/skin_subs_asp_quarterly.csv (flat file with provenance)
|
||||
* skin_subs.asp_quarterly (DuckDB table)
|
||||
|
||||
Every output row carries ``source_url`` and ``source_file`` columns for
|
||||
full provenance — each observation traces back to a specific CMS
|
||||
quarterly ZIP and the file within it that was parsed.
|
||||
|
||||
Source URLs are read from data/cms/asp_source_urls.csv (canonical URLs
|
||||
provided from CMS.gov, one per quarter 2005-Q1 through 2026-Q2).
|
||||
|
||||
Column mapping rationale
|
||||
------------------------
|
||||
CMS has changed column names across 20+ years of ASP files:
|
||||
|
||||
2005–2006: "HCPCS Code", "Short Description", "HCPCS Code Dosage",
|
||||
"Estimated ASP", "Payment Limit"
|
||||
2007–2011: Same but sometimes "Dosage" instead of "HCPCS Code Dosage"
|
||||
2012–2019: "HCPCS Code", "Short Description", "Dosage",
|
||||
"Payment Limit"
|
||||
2020–2025: "HCPCS Code", "Short Description", "HCPCS Code Dosage",
|
||||
"Payment Limit"
|
||||
2026+: "HCPCS Code", "Short Description", "Dosage or Unit",
|
||||
"Payment Limit" (now "Payment Limit" for skin subs is
|
||||
flat $127.14/cm² under reclassification)
|
||||
|
||||
All variants are mapped to: hcpcs_code, short_description,
|
||||
dosage_or_unit, payment_limit.
|
||||
|
||||
The ``asp_per_unit`` column is derived: payment_limit / 1.06, since
|
||||
CMS payment limit = ASP + 6% for drugs/biologicals. For 2026+ skin
|
||||
subs under reclassification, this derivation no longer applies (flat
|
||||
rate), but we retain it for consistency and flag it in notes.
|
||||
|
||||
Filtering
|
||||
---------
|
||||
We filter STRICTLY to codes present in skin_substitute_hcpcs.csv
|
||||
(the code universe built from CMS rulemaking in issue #232). This
|
||||
is critical because the Q4xxx prefix includes non-skin-sub codes
|
||||
(Q4054 darbepoetin alfa, Q4055 epoetin alfa, Q4074-Q4082 various
|
||||
ESRD drugs) that must be excluded.
|
||||
|
||||
Usage:
|
||||
uv run python dev/scripts/ingest_asp.py
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import io
|
||||
import re
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
import duckdb
|
||||
import pandas as pd
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Paths
|
||||
# ---------------------------------------------------------------------------
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
ASP_DIR = ROOT / "data" / "cms" / "asp"
|
||||
HCPCS_REF = ROOT / "data" / "cms" / "skin_substitute_hcpcs.csv"
|
||||
URL_REF = ROOT / "data" / "cms" / "asp_source_urls.csv"
|
||||
DUCKDB_PATH = ROOT / "data" / "aco.duckdb"
|
||||
OUTPUT_CSV = ASP_DIR / "skin_subs_asp_quarterly.csv"
|
||||
|
||||
|
||||
def _load_source_urls() -> dict[str, str]:
|
||||
"""Load canonical source URLs: {quarter: url}."""
|
||||
urls = {}
|
||||
if URL_REF.exists():
|
||||
with URL_REF.open() as f:
|
||||
for row in csv.DictReader(f):
|
||||
urls[row["quarter"]] = row["url"]
|
||||
return urls
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Column normalisation
|
||||
# ---------------------------------------------------------------------------
|
||||
# Mapping from observed CMS column names (lowercased) to our standard names.
|
||||
# Each entry documents which CMS file years use that variant.
|
||||
COL_MAP = {
|
||||
# HCPCS code — consistent across all years
|
||||
"hcpcs code": "hcpcs_code",
|
||||
"hcpcs_code": "hcpcs_code",
|
||||
# Short description — consistent
|
||||
"short description": "short_description",
|
||||
"short_description": "short_description",
|
||||
# Dosage/unit — varies by year
|
||||
"hcpcs code dosage": "dosage_or_unit", # 2005-2006, 2020-2025
|
||||
"hcpcs_code_dosage": "dosage_or_unit",
|
||||
"dosage": "dosage_or_unit", # 2007-2019
|
||||
"dosage or unit": "dosage_or_unit", # 2026+
|
||||
# Payment limit — consistent (ASP + 6%)
|
||||
"payment limit": "payment_limit",
|
||||
"payment_limit": "payment_limit",
|
||||
}
|
||||
|
||||
KEEP_COLS = ["hcpcs_code", "short_description", "dosage_or_unit", "payment_limit"]
|
||||
|
||||
|
||||
def normalise_columns(df: pd.DataFrame) -> pd.DataFrame:
|
||||
"""Lower-case columns, apply COL_MAP, keep only KEEP_COLS."""
|
||||
df.columns = [str(c).strip().lower() for c in df.columns]
|
||||
df = df.rename(columns=COL_MAP)
|
||||
present = [c for c in KEEP_COLS if c in df.columns]
|
||||
return df[present]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Readers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _find_header_row(lines: list[str]) -> int:
|
||||
"""Return the 0-based row index of the column-header row.
|
||||
|
||||
The header row starts with 'HCPCS Code,' (possibly quoted).
|
||||
Must NOT match note lines that incidentally mention 'HCPCS code'.
|
||||
"""
|
||||
for i, line in enumerate(lines):
|
||||
stripped = line.strip().lower()
|
||||
if stripped.startswith("hcpcs code,") or stripped.startswith('"hcpcs code",'):
|
||||
return i
|
||||
return -1
|
||||
|
||||
|
||||
def read_csv_from_zip(zf: zipfile.ZipFile, name: str) -> pd.DataFrame:
|
||||
"""Read a section-508 CSV inside a ZIP, skipping preamble rows."""
|
||||
raw = zf.read(name).decode("latin-1")
|
||||
lines = raw.splitlines()
|
||||
hdr = _find_header_row(lines)
|
||||
if hdr < 0:
|
||||
return pd.DataFrame()
|
||||
buf = "\n".join(lines[hdr:])
|
||||
df = pd.read_csv(io.StringIO(buf), dtype=str, on_bad_lines="skip")
|
||||
return normalise_columns(df)
|
||||
|
||||
|
||||
def read_xls_from_zip(zf: zipfile.ZipFile, name: str) -> pd.DataFrame:
|
||||
"""Read .xls inside a ZIP, skipping preamble rows."""
|
||||
import xlrd
|
||||
|
||||
data = zf.read(name)
|
||||
wb = xlrd.open_workbook(file_contents=data)
|
||||
sh = wb.sheet_by_index(0)
|
||||
hdr_row = -1
|
||||
for r in range(min(30, sh.nrows)):
|
||||
first_cell = str(sh.cell_value(r, 0)).strip().lower()
|
||||
if first_cell == "hcpcs code":
|
||||
hdr_row = r
|
||||
break
|
||||
if hdr_row < 0:
|
||||
return pd.DataFrame()
|
||||
headers = [str(sh.cell_value(hdr_row, c)).strip() for c in range(sh.ncols)]
|
||||
rows = []
|
||||
for r in range(hdr_row + 1, sh.nrows):
|
||||
row = [str(sh.cell_value(r, c)).strip() for c in range(sh.ncols)]
|
||||
if any(row):
|
||||
rows.append(row)
|
||||
df = pd.DataFrame(rows, columns=headers)
|
||||
return normalise_columns(df)
|
||||
|
||||
|
||||
def pick_pricing_file(names: list[str]) -> str | None:
|
||||
"""From a list of files in a ZIP, pick the ASP pricing file.
|
||||
|
||||
Prefer CSV (section 508) over XLS. Skip NOC, crosswalk, NDC,
|
||||
and "not payable" files.
|
||||
"""
|
||||
skip_patterns = ["noc", "crosswalk", "ndc", "not payable", "not_payable"]
|
||||
candidates: list[str] = []
|
||||
for n in names:
|
||||
low = n.lower()
|
||||
if any(s in low for s in skip_patterns):
|
||||
continue
|
||||
if low.endswith((".csv", ".xls", ".xlsx")):
|
||||
candidates.append(n)
|
||||
|
||||
csvs = [c for c in candidates if c.lower().endswith(".csv")]
|
||||
if csvs:
|
||||
for c in csvs:
|
||||
low = c.lower()
|
||||
if "payment limit" in low or "pricing" in low:
|
||||
return c
|
||||
return csvs[0]
|
||||
|
||||
xlss = [c for c in candidates if c.lower().endswith((".xls", ".xlsx"))]
|
||||
if xlss:
|
||||
for c in xlss:
|
||||
low = c.lower()
|
||||
if "payment limit" in low or "pricing" in low or "asp" in low:
|
||||
return c
|
||||
return xlss[0]
|
||||
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Numeric cleaning
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def clean_numeric(series: pd.Series) -> pd.Series:
|
||||
"""Strip $ signs and convert to float, coercing errors to NaN."""
|
||||
return (
|
||||
series.astype(str)
|
||||
.str.replace("$", "", regex=False)
|
||||
.str.replace(",", "", regex=False)
|
||||
.str.strip()
|
||||
.pipe(pd.to_numeric, errors="coerce")
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def main() -> None:
|
||||
# Load skin-substitute HCPCS codes (strict filter — no regex fallback)
|
||||
hcpcs_ref = pd.read_csv(HCPCS_REF, dtype=str)
|
||||
skin_codes = set(hcpcs_ref["hcpcs_code"].str.strip().str.upper())
|
||||
print(f"Loaded {len(skin_codes)} skin-substitute HCPCS codes from reference")
|
||||
|
||||
# Load canonical source URLs
|
||||
source_urls = _load_source_urls()
|
||||
print(f"Loaded {len(source_urls)} source URLs")
|
||||
|
||||
# Ingest all quarters
|
||||
zips = sorted(ASP_DIR.glob("asp_*.zip"))
|
||||
print(f"Found {len(zips)} quarterly ZIP files\n")
|
||||
|
||||
frames: list[pd.DataFrame] = []
|
||||
for zpath in zips:
|
||||
m = re.match(r"asp_(\d{4}-Q\d)", zpath.name)
|
||||
if not m:
|
||||
print(f" SKIP {zpath.name} (cannot parse quarter)")
|
||||
continue
|
||||
quarter = m.group(1)
|
||||
|
||||
try:
|
||||
with zipfile.ZipFile(zpath) as zf:
|
||||
target = pick_pricing_file(zf.namelist())
|
||||
if target is None:
|
||||
print(f" SKIP {quarter}: no pricing file found in ZIP")
|
||||
continue
|
||||
|
||||
if target.lower().endswith(".csv"):
|
||||
df = read_csv_from_zip(zf, target)
|
||||
else:
|
||||
df = read_xls_from_zip(zf, target)
|
||||
|
||||
if df.empty:
|
||||
print(f" SKIP {quarter}: empty after parsing {target}")
|
||||
continue
|
||||
|
||||
df["quarter"] = quarter
|
||||
df["source_file"] = target
|
||||
df["source_url"] = source_urls.get(quarter, "")
|
||||
frames.append(df)
|
||||
print(f" OK {quarter}: {len(df):>5} rows from {target}")
|
||||
except Exception as e:
|
||||
print(f" ERR {quarter}: {e}")
|
||||
|
||||
if not frames:
|
||||
raise RuntimeError("No data parsed from any file")
|
||||
|
||||
all_df = pd.concat(frames, ignore_index=True)
|
||||
print(f"\nTotal rows across all quarters: {len(all_df)}")
|
||||
|
||||
# Normalise HCPCS codes
|
||||
all_df["hcpcs_code"] = all_df["hcpcs_code"].astype(str).str.strip().str.upper()
|
||||
|
||||
# Filter STRICTLY to skin substitute code universe
|
||||
# NO regex fallback — only codes in skin_substitute_hcpcs.csv
|
||||
skin_df = all_df[all_df["hcpcs_code"].isin(skin_codes)].copy()
|
||||
print(f"Skin substitute rows (strict filter): {len(skin_df)}")
|
||||
|
||||
# Show what we excluded
|
||||
q4_but_not_skin = all_df[
|
||||
all_df["hcpcs_code"].str.match(r"^Q4\d{2,3}$")
|
||||
& ~all_df["hcpcs_code"].isin(skin_codes)
|
||||
]["hcpcs_code"].unique()
|
||||
if len(q4_but_not_skin) > 0:
|
||||
print(f" Excluded Q4 codes NOT in skin sub universe: {sorted(q4_but_not_skin)}")
|
||||
|
||||
if not skin_df.empty:
|
||||
skin_df["payment_limit"] = clean_numeric(skin_df["payment_limit"])
|
||||
# ASP per unit = payment_limit / 1.06
|
||||
# (CMS payment limit = ASP + 6% for drugs/biologicals)
|
||||
skin_df["asp_per_unit"] = (skin_df["payment_limit"] / 1.06).round(3)
|
||||
|
||||
# Reorder columns
|
||||
out_cols = [
|
||||
"quarter",
|
||||
"hcpcs_code",
|
||||
"short_description",
|
||||
"asp_per_unit",
|
||||
"payment_limit",
|
||||
"dosage_or_unit",
|
||||
"source_url",
|
||||
"source_file",
|
||||
]
|
||||
for c in out_cols:
|
||||
if c not in skin_df.columns:
|
||||
skin_df[c] = None
|
||||
skin_df = skin_df[out_cols].sort_values(["quarter", "hcpcs_code"])
|
||||
|
||||
# Write CSV
|
||||
skin_df.to_csv(OUTPUT_CSV, index=False)
|
||||
print(f"\nWrote {len(skin_df)} rows to {OUTPUT_CSV}")
|
||||
|
||||
if len(skin_df) > 0:
|
||||
quarters = sorted(skin_df["quarter"].unique())
|
||||
codes = sorted(skin_df["hcpcs_code"].unique())
|
||||
print(f" Quarters: {len(quarters)} ({quarters[0]} to {quarters[-1]})")
|
||||
print(f" Unique HCPCS codes: {len(codes)}")
|
||||
|
||||
# Load into DuckDB
|
||||
print(f"\nLoading into DuckDB at {DUCKDB_PATH} ...")
|
||||
con = duckdb.connect(str(DUCKDB_PATH))
|
||||
con.execute("CREATE SCHEMA IF NOT EXISTS skin_subs")
|
||||
con.execute("DROP TABLE IF EXISTS skin_subs.asp_quarterly")
|
||||
if not skin_df.empty:
|
||||
con.execute(
|
||||
"""
|
||||
CREATE TABLE skin_subs.asp_quarterly AS
|
||||
SELECT * FROM read_csv_auto(?, header=true)
|
||||
""",
|
||||
[str(OUTPUT_CSV)],
|
||||
)
|
||||
count = con.execute(
|
||||
"SELECT count(*) FROM skin_subs.asp_quarterly"
|
||||
).fetchone()[0]
|
||||
print(f" Loaded {count} rows into skin_subs.asp_quarterly")
|
||||
else:
|
||||
print(" WARNING: No skin substitute data found in ASP files")
|
||||
print(" This may be expected — skin subs may not appear in main ASP pricing files")
|
||||
print(" They may be in separate NOC or tissue coding files")
|
||||
con.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user