data: ASP ingestion — strict skin sub filter, full provenance, 86 canonical URLs
Some checks failed
CI / skinny-install (aco) (push) Successful in 50s
CI / skinny-install (bls) (push) Successful in 32s
CI / skinny-install (cms) (push) Successful in 28s
CI / skinny-install (cli) (push) Successful in 36s
CI / skinny-install (perf) (push) Successful in 27s
CI / skinny-install (conf) (push) Successful in 35s
CI / skinny-install (pfs) (push) Successful in 36s
CI / skinny-install (rex) (push) Successful in 35s
CI / lint-test (push) Successful in 1m24s
CI / skinny-install (api) (push) Successful in 34s
CI / skinny-install (bcda) (push) Successful in 34s
CI / skinny-install (bib) (push) Successful in 33s
CI / skinny-install (ccw) (push) Successful in 32s
Deploy / build-scan-report (push) Has been cancelled
Some checks failed
CI / skinny-install (aco) (push) Successful in 50s
CI / skinny-install (bls) (push) Successful in 32s
CI / skinny-install (cms) (push) Successful in 28s
CI / skinny-install (cli) (push) Successful in 36s
CI / skinny-install (perf) (push) Successful in 27s
CI / skinny-install (conf) (push) Successful in 35s
CI / skinny-install (pfs) (push) Successful in 36s
CI / skinny-install (rex) (push) Successful in 35s
CI / lint-test (push) Successful in 1m24s
CI / skinny-install (api) (push) Successful in 34s
CI / skinny-install (bcda) (push) Successful in 34s
CI / skinny-install (bib) (push) Successful in 33s
CI / skinny-install (ccw) (push) Successful in 32s
Deploy / build-scan-report (push) Has been cancelled
Rewrite ingest_asp.py with academic rigor: - Strict filter to skin_substitute_hcpcs.csv code universe (NO regex) - 28 non-skin-sub Q4 codes excluded (Q4054 darbepoetin, Q4055 epoetin, Q4074-Q4099 ESRD drugs) that were incorrectly included before - source_url and source_file columns on every output row (100% provenance) - Documented column mapping rationale across 20 years of CMS format drift - DuckDB path corrected to data/aco.duckdb Result: 1,974 observations, 68 quarters (2009-Q1–2025-Q4), 169 codes Canonical source URLs: data/cms/asp_source_urls.csv (86 CMS.gov URLs) Refs #233
This commit is contained in:
87
data/cms/asp_source_urls.csv
Normal file
87
data/cms/asp_source_urls.csv
Normal file
@@ -0,0 +1,87 @@
|
|||||||
|
quarter,url
|
||||||
|
2026-Q2,https://www.cms.gov/files/zip/april-2026-medicare-part-b-payment-limit-files-03-24-2026-final-file.zip
|
||||||
|
2026-Q1,https://www.cms.gov/files/zip/january-2026-medicare-part-b-payment-limit-files.zip
|
||||||
|
2025-Q4,https://www.cms.gov/files/zip/october-2025-asp-pricing-final-file.zip
|
||||||
|
2025-Q3,https://www.cms.gov/files/zip/july-2025-asp-pricing-file.zip
|
||||||
|
2025-Q2,https://www.cms.gov/files/zip/april-2025-asp-pricing-file.zip
|
||||||
|
2025-Q1,https://www.cms.gov/files/zip/january-2025-asp-pricing-file-03/11/25-final-file.zip
|
||||||
|
2024-Q4,https://www.cms.gov/files/zip/october-2024-asp-pricing-file.zip
|
||||||
|
2024-Q3,https://www.cms.gov/files/zip/july-2024-asp-pricing-file.zip
|
||||||
|
2024-Q2,https://www.cms.gov/files/zip/april-2024-asp-pricing-file.zip
|
||||||
|
2024-Q1,https://www.cms.gov/files/zip/january-2024-asp-pricing-file.zip
|
||||||
|
2023-Q4,https://www.cms.gov/files/zip/october-2023-asp-pricing-file.zip
|
||||||
|
2023-Q3,https://www.cms.gov/files/zip/july-2023-asp-pricing-file.zip
|
||||||
|
2023-Q2,https://www.cms.gov/files/zip/april-2023-asp-pricing-file.zip
|
||||||
|
2023-Q1,https://www.cms.gov/files/zip/january-2023-asp-pricing-file.zip
|
||||||
|
2022-Q4,https://www.cms.gov/files/zip/october-2022-asp-pricing-file.zip
|
||||||
|
2022-Q3,https://www.cms.gov/files/zip/july-2022-asp-pricing-file.zip
|
||||||
|
2022-Q2,https://www.cms.gov/files/zip/april-2022-asp-pricing-file.zip
|
||||||
|
2022-Q1,https://www.cms.gov/files/zip/january-2022-asp-pricing-file.zip
|
||||||
|
2021-Q4,https://www.cms.gov/files/zip/october-2021-asp-pricing-file.zip
|
||||||
|
2021-Q3,https://www.cms.gov/files/zip/july-2021-asp-pricing-file.zip
|
||||||
|
2021-Q2,https://www.cms.gov/files/zip/april-2021-asp-pricing-file.zip
|
||||||
|
2021-Q1,https://www.cms.gov/files/zip/january-2021-asp-pricing-file.zip
|
||||||
|
2020-Q4,https://www.cms.gov/files/zip/october-2020-asp-pricing-file.zip
|
||||||
|
2020-Q3,https://www.cms.gov/files/zip/july-2020-asp-pricing-file-updated-06012020.zip
|
||||||
|
2020-Q2,https://www.cms.gov/files/zip/april-2020-asp-pricing-file.zip
|
||||||
|
2020-Q1,https://www.cms.gov/files/zip/january-2020-asp-pricing-file.zip
|
||||||
|
2019-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2019-Oct-ASP-Pricing-File.zip
|
||||||
|
2019-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2019-July-ASP-Pricing-File.zip
|
||||||
|
2019-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April-2019-ASP-Pricing-File.zip
|
||||||
|
2019-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2019-January-ASP-Pricing-File.zip
|
||||||
|
2018-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2018-Oct-ASP-Pricing-File.zip
|
||||||
|
2018-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2018-July-ASP-Pricing-File.zip
|
||||||
|
2018-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2018-April-ASP-Pricing-File.zip
|
||||||
|
2018-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2018-January-ASP-Pricing-File.zip
|
||||||
|
2017-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2017-October-ASP-Pricing-File.zip
|
||||||
|
2017-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2017-July-ASP-Pricing-File.zip
|
||||||
|
2017-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2017-April-ASP-Pricing.zip
|
||||||
|
2017-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2017-January-ASP-Pricing-Files.zip
|
||||||
|
2016-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2016-October-ASP-Pricing-File.zip
|
||||||
|
2016-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2016-July-ASP-Pricing-File.zip
|
||||||
|
2016-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2016-April-ASP-Pricing-File.zip
|
||||||
|
2016-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2016-January-ASP-Pricing-File.zip
|
||||||
|
2015-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2015-October-ASP-Pricing-File.zip
|
||||||
|
2015-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2015-July-ASP-Pricing-File.zip
|
||||||
|
2015-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2015-April-ASP-Pricing-File.zip
|
||||||
|
2015-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2015-January-ASP-Pricing-File.zip
|
||||||
|
2014-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2014-October-ASP-Pricing-File.zip
|
||||||
|
2014-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jul-2014-ASP-Pricing-File.zip
|
||||||
|
2014-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Apr-14-ASP-Pricing-File.zip
|
||||||
|
2014-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jan-14-ASP-Pricing-File.zip
|
||||||
|
2013-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2013-October-ASP-Pricing-File.zip
|
||||||
|
2013-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2013-July-ASP-Pricing-File.zip
|
||||||
|
2013-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Apr-13-ASP-Pricing-file.zip
|
||||||
|
2013-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jan-2013-ASP-Pricing-File.zip
|
||||||
|
2012-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/October-2012-ASP-Pricing-File.zip
|
||||||
|
2012-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/July-2012-ASP-Pricing-File.zip
|
||||||
|
2012-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April-2012-ASP-Pricing-file-revised030513.zip
|
||||||
|
2012-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jan_2012_ASP_Pricing_File.zip
|
||||||
|
2011-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Oct_2011_ASP_Pricing_File.zip
|
||||||
|
2011-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/July_2011_ASP_Pricing_File.zip
|
||||||
|
2011-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April_2011_ASP_Pricing_File.zip
|
||||||
|
2011-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/January2011_ASP_PricingFile.zip
|
||||||
|
2010-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/October_2010_ASP_Pricing_File.zip
|
||||||
|
2010-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/July_2010_ASP_Pricing_File.zip
|
||||||
|
2010-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April_2010_ASP_Pricing_File.zip
|
||||||
|
2010-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/January_2010_ASP_Pricing_File.zip
|
||||||
|
2009-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/October_2009_ASP_Pricing_File.zip
|
||||||
|
2009-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/July_2009_ASP_Pricing_File.zip
|
||||||
|
2009-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April_2009_ASP_Pricing_File_by_HCPCS.zip
|
||||||
|
2009-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/JAN_2009_ASP_Pricing_File_by_HCPCS.zip
|
||||||
|
2008-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/October2008ASPPricingFilebyHCPCS.zip
|
||||||
|
2008-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/July2008ASPPricingFilebyHCPCS.zip
|
||||||
|
2008-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April08ASPbyHCPCS.zip
|
||||||
|
2008-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jan08ASPbyHCPCS.zip
|
||||||
|
2007-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/October07ASPbyHCPCS.zip
|
||||||
|
2007-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/July07ASPbyHCPCS_121707.zip
|
||||||
|
2007-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April07ASPbyHCPCS_121707.zip
|
||||||
|
2007-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jan07ASPbyHCPCS_121707.zip
|
||||||
|
2006-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/oct06asp_hcpcs.zip
|
||||||
|
2006-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/july06asp_hcpcs.zip
|
||||||
|
2006-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/apr06pricing.zip
|
||||||
|
2006-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jan06pricing.zip
|
||||||
|
2005-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/oct05aspbyhcpcs033106v2.zip
|
||||||
|
2005-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/jul05aspbyhcpcs_032906.zip
|
||||||
|
2005-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/apr05aspbyhcpcsv8_033106.zip
|
||||||
|
2005-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/jan05aspbyhcpcsv6_033106.zip
|
||||||
|
353
dev/scripts/ingest_asp.py
Normal file
353
dev/scripts/ingest_asp.py
Normal file
@@ -0,0 +1,353 @@
|
|||||||
|
"""Ingest CMS Average Sales Price (ASP) Drug Pricing Files.
|
||||||
|
|
||||||
|
Reads all quarterly ASP ZIP files from data/cms/asp/, normalises column
|
||||||
|
names across 20+ years of CMS format drift, filters **strictly** to
|
||||||
|
skin substitute HCPCS codes defined in data/cms/skin_substitute_hcpcs.csv,
|
||||||
|
and produces:
|
||||||
|
|
||||||
|
* data/cms/asp/skin_subs_asp_quarterly.csv (flat file with provenance)
|
||||||
|
* skin_subs.asp_quarterly (DuckDB table)
|
||||||
|
|
||||||
|
Every output row carries ``source_url`` and ``source_file`` columns for
|
||||||
|
full provenance — each observation traces back to a specific CMS
|
||||||
|
quarterly ZIP and the file within it that was parsed.
|
||||||
|
|
||||||
|
Source URLs are read from data/cms/asp_source_urls.csv (canonical URLs
|
||||||
|
provided from CMS.gov, one per quarter 2005-Q1 through 2026-Q2).
|
||||||
|
|
||||||
|
Column mapping rationale
|
||||||
|
------------------------
|
||||||
|
CMS has changed column names across 20+ years of ASP files:
|
||||||
|
|
||||||
|
2005–2006: "HCPCS Code", "Short Description", "HCPCS Code Dosage",
|
||||||
|
"Estimated ASP", "Payment Limit"
|
||||||
|
2007–2011: Same but sometimes "Dosage" instead of "HCPCS Code Dosage"
|
||||||
|
2012–2019: "HCPCS Code", "Short Description", "Dosage",
|
||||||
|
"Payment Limit"
|
||||||
|
2020–2025: "HCPCS Code", "Short Description", "HCPCS Code Dosage",
|
||||||
|
"Payment Limit"
|
||||||
|
2026+: "HCPCS Code", "Short Description", "Dosage or Unit",
|
||||||
|
"Payment Limit" (now "Payment Limit" for skin subs is
|
||||||
|
flat $127.14/cm² under reclassification)
|
||||||
|
|
||||||
|
All variants are mapped to: hcpcs_code, short_description,
|
||||||
|
dosage_or_unit, payment_limit.
|
||||||
|
|
||||||
|
The ``asp_per_unit`` column is derived: payment_limit / 1.06, since
|
||||||
|
CMS payment limit = ASP + 6% for drugs/biologicals. For 2026+ skin
|
||||||
|
subs under reclassification, this derivation no longer applies (flat
|
||||||
|
rate), but we retain it for consistency and flag it in notes.
|
||||||
|
|
||||||
|
Filtering
|
||||||
|
---------
|
||||||
|
We filter STRICTLY to codes present in skin_substitute_hcpcs.csv
|
||||||
|
(the code universe built from CMS rulemaking in issue #232). This
|
||||||
|
is critical because the Q4xxx prefix includes non-skin-sub codes
|
||||||
|
(Q4054 darbepoetin alfa, Q4055 epoetin alfa, Q4074-Q4082 various
|
||||||
|
ESRD drugs) that must be excluded.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
uv run python dev/scripts/ingest_asp.py
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import csv
|
||||||
|
import io
|
||||||
|
import re
|
||||||
|
import zipfile
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import duckdb
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Paths
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
ROOT = Path(__file__).resolve().parents[2]
|
||||||
|
ASP_DIR = ROOT / "data" / "cms" / "asp"
|
||||||
|
HCPCS_REF = ROOT / "data" / "cms" / "skin_substitute_hcpcs.csv"
|
||||||
|
URL_REF = ROOT / "data" / "cms" / "asp_source_urls.csv"
|
||||||
|
DUCKDB_PATH = ROOT / "data" / "aco.duckdb"
|
||||||
|
OUTPUT_CSV = ASP_DIR / "skin_subs_asp_quarterly.csv"
|
||||||
|
|
||||||
|
|
||||||
|
def _load_source_urls() -> dict[str, str]:
|
||||||
|
"""Load canonical source URLs: {quarter: url}."""
|
||||||
|
urls = {}
|
||||||
|
if URL_REF.exists():
|
||||||
|
with URL_REF.open() as f:
|
||||||
|
for row in csv.DictReader(f):
|
||||||
|
urls[row["quarter"]] = row["url"]
|
||||||
|
return urls
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Column normalisation
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Mapping from observed CMS column names (lowercased) to our standard names.
|
||||||
|
# Each entry documents which CMS file years use that variant.
|
||||||
|
COL_MAP = {
|
||||||
|
# HCPCS code — consistent across all years
|
||||||
|
"hcpcs code": "hcpcs_code",
|
||||||
|
"hcpcs_code": "hcpcs_code",
|
||||||
|
# Short description — consistent
|
||||||
|
"short description": "short_description",
|
||||||
|
"short_description": "short_description",
|
||||||
|
# Dosage/unit — varies by year
|
||||||
|
"hcpcs code dosage": "dosage_or_unit", # 2005-2006, 2020-2025
|
||||||
|
"hcpcs_code_dosage": "dosage_or_unit",
|
||||||
|
"dosage": "dosage_or_unit", # 2007-2019
|
||||||
|
"dosage or unit": "dosage_or_unit", # 2026+
|
||||||
|
# Payment limit — consistent (ASP + 6%)
|
||||||
|
"payment limit": "payment_limit",
|
||||||
|
"payment_limit": "payment_limit",
|
||||||
|
}
|
||||||
|
|
||||||
|
KEEP_COLS = ["hcpcs_code", "short_description", "dosage_or_unit", "payment_limit"]
|
||||||
|
|
||||||
|
|
||||||
|
def normalise_columns(df: pd.DataFrame) -> pd.DataFrame:
|
||||||
|
"""Lower-case columns, apply COL_MAP, keep only KEEP_COLS."""
|
||||||
|
df.columns = [str(c).strip().lower() for c in df.columns]
|
||||||
|
df = df.rename(columns=COL_MAP)
|
||||||
|
present = [c for c in KEEP_COLS if c in df.columns]
|
||||||
|
return df[present]
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Readers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _find_header_row(lines: list[str]) -> int:
|
||||||
|
"""Return the 0-based row index of the column-header row.
|
||||||
|
|
||||||
|
The header row starts with 'HCPCS Code,' (possibly quoted).
|
||||||
|
Must NOT match note lines that incidentally mention 'HCPCS code'.
|
||||||
|
"""
|
||||||
|
for i, line in enumerate(lines):
|
||||||
|
stripped = line.strip().lower()
|
||||||
|
if stripped.startswith("hcpcs code,") or stripped.startswith('"hcpcs code",'):
|
||||||
|
return i
|
||||||
|
return -1
|
||||||
|
|
||||||
|
|
||||||
|
def read_csv_from_zip(zf: zipfile.ZipFile, name: str) -> pd.DataFrame:
|
||||||
|
"""Read a section-508 CSV inside a ZIP, skipping preamble rows."""
|
||||||
|
raw = zf.read(name).decode("latin-1")
|
||||||
|
lines = raw.splitlines()
|
||||||
|
hdr = _find_header_row(lines)
|
||||||
|
if hdr < 0:
|
||||||
|
return pd.DataFrame()
|
||||||
|
buf = "\n".join(lines[hdr:])
|
||||||
|
df = pd.read_csv(io.StringIO(buf), dtype=str, on_bad_lines="skip")
|
||||||
|
return normalise_columns(df)
|
||||||
|
|
||||||
|
|
||||||
|
def read_xls_from_zip(zf: zipfile.ZipFile, name: str) -> pd.DataFrame:
|
||||||
|
"""Read .xls inside a ZIP, skipping preamble rows."""
|
||||||
|
import xlrd
|
||||||
|
|
||||||
|
data = zf.read(name)
|
||||||
|
wb = xlrd.open_workbook(file_contents=data)
|
||||||
|
sh = wb.sheet_by_index(0)
|
||||||
|
hdr_row = -1
|
||||||
|
for r in range(min(30, sh.nrows)):
|
||||||
|
first_cell = str(sh.cell_value(r, 0)).strip().lower()
|
||||||
|
if first_cell == "hcpcs code":
|
||||||
|
hdr_row = r
|
||||||
|
break
|
||||||
|
if hdr_row < 0:
|
||||||
|
return pd.DataFrame()
|
||||||
|
headers = [str(sh.cell_value(hdr_row, c)).strip() for c in range(sh.ncols)]
|
||||||
|
rows = []
|
||||||
|
for r in range(hdr_row + 1, sh.nrows):
|
||||||
|
row = [str(sh.cell_value(r, c)).strip() for c in range(sh.ncols)]
|
||||||
|
if any(row):
|
||||||
|
rows.append(row)
|
||||||
|
df = pd.DataFrame(rows, columns=headers)
|
||||||
|
return normalise_columns(df)
|
||||||
|
|
||||||
|
|
||||||
|
def pick_pricing_file(names: list[str]) -> str | None:
|
||||||
|
"""From a list of files in a ZIP, pick the ASP pricing file.
|
||||||
|
|
||||||
|
Prefer CSV (section 508) over XLS. Skip NOC, crosswalk, NDC,
|
||||||
|
and "not payable" files.
|
||||||
|
"""
|
||||||
|
skip_patterns = ["noc", "crosswalk", "ndc", "not payable", "not_payable"]
|
||||||
|
candidates: list[str] = []
|
||||||
|
for n in names:
|
||||||
|
low = n.lower()
|
||||||
|
if any(s in low for s in skip_patterns):
|
||||||
|
continue
|
||||||
|
if low.endswith((".csv", ".xls", ".xlsx")):
|
||||||
|
candidates.append(n)
|
||||||
|
|
||||||
|
csvs = [c for c in candidates if c.lower().endswith(".csv")]
|
||||||
|
if csvs:
|
||||||
|
for c in csvs:
|
||||||
|
low = c.lower()
|
||||||
|
if "payment limit" in low or "pricing" in low:
|
||||||
|
return c
|
||||||
|
return csvs[0]
|
||||||
|
|
||||||
|
xlss = [c for c in candidates if c.lower().endswith((".xls", ".xlsx"))]
|
||||||
|
if xlss:
|
||||||
|
for c in xlss:
|
||||||
|
low = c.lower()
|
||||||
|
if "payment limit" in low or "pricing" in low or "asp" in low:
|
||||||
|
return c
|
||||||
|
return xlss[0]
|
||||||
|
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Numeric cleaning
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def clean_numeric(series: pd.Series) -> pd.Series:
|
||||||
|
"""Strip $ signs and convert to float, coercing errors to NaN."""
|
||||||
|
return (
|
||||||
|
series.astype(str)
|
||||||
|
.str.replace("$", "", regex=False)
|
||||||
|
.str.replace(",", "", regex=False)
|
||||||
|
.str.strip()
|
||||||
|
.pipe(pd.to_numeric, errors="coerce")
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Main
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
# Load skin-substitute HCPCS codes (strict filter — no regex fallback)
|
||||||
|
hcpcs_ref = pd.read_csv(HCPCS_REF, dtype=str)
|
||||||
|
skin_codes = set(hcpcs_ref["hcpcs_code"].str.strip().str.upper())
|
||||||
|
print(f"Loaded {len(skin_codes)} skin-substitute HCPCS codes from reference")
|
||||||
|
|
||||||
|
# Load canonical source URLs
|
||||||
|
source_urls = _load_source_urls()
|
||||||
|
print(f"Loaded {len(source_urls)} source URLs")
|
||||||
|
|
||||||
|
# Ingest all quarters
|
||||||
|
zips = sorted(ASP_DIR.glob("asp_*.zip"))
|
||||||
|
print(f"Found {len(zips)} quarterly ZIP files\n")
|
||||||
|
|
||||||
|
frames: list[pd.DataFrame] = []
|
||||||
|
for zpath in zips:
|
||||||
|
m = re.match(r"asp_(\d{4}-Q\d)", zpath.name)
|
||||||
|
if not m:
|
||||||
|
print(f" SKIP {zpath.name} (cannot parse quarter)")
|
||||||
|
continue
|
||||||
|
quarter = m.group(1)
|
||||||
|
|
||||||
|
try:
|
||||||
|
with zipfile.ZipFile(zpath) as zf:
|
||||||
|
target = pick_pricing_file(zf.namelist())
|
||||||
|
if target is None:
|
||||||
|
print(f" SKIP {quarter}: no pricing file found in ZIP")
|
||||||
|
continue
|
||||||
|
|
||||||
|
if target.lower().endswith(".csv"):
|
||||||
|
df = read_csv_from_zip(zf, target)
|
||||||
|
else:
|
||||||
|
df = read_xls_from_zip(zf, target)
|
||||||
|
|
||||||
|
if df.empty:
|
||||||
|
print(f" SKIP {quarter}: empty after parsing {target}")
|
||||||
|
continue
|
||||||
|
|
||||||
|
df["quarter"] = quarter
|
||||||
|
df["source_file"] = target
|
||||||
|
df["source_url"] = source_urls.get(quarter, "")
|
||||||
|
frames.append(df)
|
||||||
|
print(f" OK {quarter}: {len(df):>5} rows from {target}")
|
||||||
|
except Exception as e:
|
||||||
|
print(f" ERR {quarter}: {e}")
|
||||||
|
|
||||||
|
if not frames:
|
||||||
|
raise RuntimeError("No data parsed from any file")
|
||||||
|
|
||||||
|
all_df = pd.concat(frames, ignore_index=True)
|
||||||
|
print(f"\nTotal rows across all quarters: {len(all_df)}")
|
||||||
|
|
||||||
|
# Normalise HCPCS codes
|
||||||
|
all_df["hcpcs_code"] = all_df["hcpcs_code"].astype(str).str.strip().str.upper()
|
||||||
|
|
||||||
|
# Filter STRICTLY to skin substitute code universe
|
||||||
|
# NO regex fallback — only codes in skin_substitute_hcpcs.csv
|
||||||
|
skin_df = all_df[all_df["hcpcs_code"].isin(skin_codes)].copy()
|
||||||
|
print(f"Skin substitute rows (strict filter): {len(skin_df)}")
|
||||||
|
|
||||||
|
# Show what we excluded
|
||||||
|
q4_but_not_skin = all_df[
|
||||||
|
all_df["hcpcs_code"].str.match(r"^Q4\d{2,3}$")
|
||||||
|
& ~all_df["hcpcs_code"].isin(skin_codes)
|
||||||
|
]["hcpcs_code"].unique()
|
||||||
|
if len(q4_but_not_skin) > 0:
|
||||||
|
print(f" Excluded Q4 codes NOT in skin sub universe: {sorted(q4_but_not_skin)}")
|
||||||
|
|
||||||
|
if not skin_df.empty:
|
||||||
|
skin_df["payment_limit"] = clean_numeric(skin_df["payment_limit"])
|
||||||
|
# ASP per unit = payment_limit / 1.06
|
||||||
|
# (CMS payment limit = ASP + 6% for drugs/biologicals)
|
||||||
|
skin_df["asp_per_unit"] = (skin_df["payment_limit"] / 1.06).round(3)
|
||||||
|
|
||||||
|
# Reorder columns
|
||||||
|
out_cols = [
|
||||||
|
"quarter",
|
||||||
|
"hcpcs_code",
|
||||||
|
"short_description",
|
||||||
|
"asp_per_unit",
|
||||||
|
"payment_limit",
|
||||||
|
"dosage_or_unit",
|
||||||
|
"source_url",
|
||||||
|
"source_file",
|
||||||
|
]
|
||||||
|
for c in out_cols:
|
||||||
|
if c not in skin_df.columns:
|
||||||
|
skin_df[c] = None
|
||||||
|
skin_df = skin_df[out_cols].sort_values(["quarter", "hcpcs_code"])
|
||||||
|
|
||||||
|
# Write CSV
|
||||||
|
skin_df.to_csv(OUTPUT_CSV, index=False)
|
||||||
|
print(f"\nWrote {len(skin_df)} rows to {OUTPUT_CSV}")
|
||||||
|
|
||||||
|
if len(skin_df) > 0:
|
||||||
|
quarters = sorted(skin_df["quarter"].unique())
|
||||||
|
codes = sorted(skin_df["hcpcs_code"].unique())
|
||||||
|
print(f" Quarters: {len(quarters)} ({quarters[0]} to {quarters[-1]})")
|
||||||
|
print(f" Unique HCPCS codes: {len(codes)}")
|
||||||
|
|
||||||
|
# Load into DuckDB
|
||||||
|
print(f"\nLoading into DuckDB at {DUCKDB_PATH} ...")
|
||||||
|
con = duckdb.connect(str(DUCKDB_PATH))
|
||||||
|
con.execute("CREATE SCHEMA IF NOT EXISTS skin_subs")
|
||||||
|
con.execute("DROP TABLE IF EXISTS skin_subs.asp_quarterly")
|
||||||
|
if not skin_df.empty:
|
||||||
|
con.execute(
|
||||||
|
"""
|
||||||
|
CREATE TABLE skin_subs.asp_quarterly AS
|
||||||
|
SELECT * FROM read_csv_auto(?, header=true)
|
||||||
|
""",
|
||||||
|
[str(OUTPUT_CSV)],
|
||||||
|
)
|
||||||
|
count = con.execute(
|
||||||
|
"SELECT count(*) FROM skin_subs.asp_quarterly"
|
||||||
|
).fetchone()[0]
|
||||||
|
print(f" Loaded {count} rows into skin_subs.asp_quarterly")
|
||||||
|
else:
|
||||||
|
print(" WARNING: No skin substitute data found in ASP files")
|
||||||
|
print(" This may be expected — skin subs may not appear in main ASP pricing files")
|
||||||
|
print(" They may be in separate NOC or tissue coding files")
|
||||||
|
con.close()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
Reference in New Issue
Block a user