data: ASP ingestion — strict skin sub filter, full provenance, 86 canonical URLs
Some checks failed
CI / skinny-install (aco) (push) Successful in 50s
CI / skinny-install (bls) (push) Successful in 32s
CI / skinny-install (cms) (push) Successful in 28s
CI / skinny-install (cli) (push) Successful in 36s
CI / skinny-install (perf) (push) Successful in 27s
CI / skinny-install (conf) (push) Successful in 35s
CI / skinny-install (pfs) (push) Successful in 36s
CI / skinny-install (rex) (push) Successful in 35s
CI / lint-test (push) Successful in 1m24s
CI / skinny-install (api) (push) Successful in 34s
CI / skinny-install (bcda) (push) Successful in 34s
CI / skinny-install (bib) (push) Successful in 33s
CI / skinny-install (ccw) (push) Successful in 32s
Deploy / build-scan-report (push) Has been cancelled

Rewrite ingest_asp.py with academic rigor:
- Strict filter to skin_substitute_hcpcs.csv code universe (NO regex)
- 28 non-skin-sub Q4 codes excluded (Q4054 darbepoetin, Q4055 epoetin,
  Q4074-Q4099 ESRD drugs) that were incorrectly included before
- source_url and source_file columns on every output row (100% provenance)
- Documented column mapping rationale across 20 years of CMS format drift
- DuckDB path corrected to data/aco.duckdb

Result: 1,974 observations, 68 quarters (2009-Q1–2025-Q4), 169 codes
Canonical source URLs: data/cms/asp_source_urls.csv (86 CMS.gov URLs)

Refs #233
This commit is contained in:
kert
2026-03-25 09:44:18 -04:00
parent 937430c4d0
commit e16984258f
2 changed files with 440 additions and 0 deletions

View File

@@ -0,0 +1,87 @@
quarter,url
2026-Q2,https://www.cms.gov/files/zip/april-2026-medicare-part-b-payment-limit-files-03-24-2026-final-file.zip
2026-Q1,https://www.cms.gov/files/zip/january-2026-medicare-part-b-payment-limit-files.zip
2025-Q4,https://www.cms.gov/files/zip/october-2025-asp-pricing-final-file.zip
2025-Q3,https://www.cms.gov/files/zip/july-2025-asp-pricing-file.zip
2025-Q2,https://www.cms.gov/files/zip/april-2025-asp-pricing-file.zip
2025-Q1,https://www.cms.gov/files/zip/january-2025-asp-pricing-file-03/11/25-final-file.zip
2024-Q4,https://www.cms.gov/files/zip/october-2024-asp-pricing-file.zip
2024-Q3,https://www.cms.gov/files/zip/july-2024-asp-pricing-file.zip
2024-Q2,https://www.cms.gov/files/zip/april-2024-asp-pricing-file.zip
2024-Q1,https://www.cms.gov/files/zip/january-2024-asp-pricing-file.zip
2023-Q4,https://www.cms.gov/files/zip/october-2023-asp-pricing-file.zip
2023-Q3,https://www.cms.gov/files/zip/july-2023-asp-pricing-file.zip
2023-Q2,https://www.cms.gov/files/zip/april-2023-asp-pricing-file.zip
2023-Q1,https://www.cms.gov/files/zip/january-2023-asp-pricing-file.zip
2022-Q4,https://www.cms.gov/files/zip/october-2022-asp-pricing-file.zip
2022-Q3,https://www.cms.gov/files/zip/july-2022-asp-pricing-file.zip
2022-Q2,https://www.cms.gov/files/zip/april-2022-asp-pricing-file.zip
2022-Q1,https://www.cms.gov/files/zip/january-2022-asp-pricing-file.zip
2021-Q4,https://www.cms.gov/files/zip/october-2021-asp-pricing-file.zip
2021-Q3,https://www.cms.gov/files/zip/july-2021-asp-pricing-file.zip
2021-Q2,https://www.cms.gov/files/zip/april-2021-asp-pricing-file.zip
2021-Q1,https://www.cms.gov/files/zip/january-2021-asp-pricing-file.zip
2020-Q4,https://www.cms.gov/files/zip/october-2020-asp-pricing-file.zip
2020-Q3,https://www.cms.gov/files/zip/july-2020-asp-pricing-file-updated-06012020.zip
2020-Q2,https://www.cms.gov/files/zip/april-2020-asp-pricing-file.zip
2020-Q1,https://www.cms.gov/files/zip/january-2020-asp-pricing-file.zip
2019-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2019-Oct-ASP-Pricing-File.zip
2019-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2019-July-ASP-Pricing-File.zip
2019-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April-2019-ASP-Pricing-File.zip
2019-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2019-January-ASP-Pricing-File.zip
2018-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2018-Oct-ASP-Pricing-File.zip
2018-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2018-July-ASP-Pricing-File.zip
2018-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2018-April-ASP-Pricing-File.zip
2018-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2018-January-ASP-Pricing-File.zip
2017-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2017-October-ASP-Pricing-File.zip
2017-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2017-July-ASP-Pricing-File.zip
2017-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2017-April-ASP-Pricing.zip
2017-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2017-January-ASP-Pricing-Files.zip
2016-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2016-October-ASP-Pricing-File.zip
2016-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2016-July-ASP-Pricing-File.zip
2016-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2016-April-ASP-Pricing-File.zip
2016-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2016-January-ASP-Pricing-File.zip
2015-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2015-October-ASP-Pricing-File.zip
2015-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2015-July-ASP-Pricing-File.zip
2015-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2015-April-ASP-Pricing-File.zip
2015-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2015-January-ASP-Pricing-File.zip
2014-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2014-October-ASP-Pricing-File.zip
2014-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jul-2014-ASP-Pricing-File.zip
2014-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Apr-14-ASP-Pricing-File.zip
2014-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jan-14-ASP-Pricing-File.zip
2013-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2013-October-ASP-Pricing-File.zip
2013-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2013-July-ASP-Pricing-File.zip
2013-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Apr-13-ASP-Pricing-file.zip
2013-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jan-2013-ASP-Pricing-File.zip
2012-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/October-2012-ASP-Pricing-File.zip
2012-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/July-2012-ASP-Pricing-File.zip
2012-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April-2012-ASP-Pricing-file-revised030513.zip
2012-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jan_2012_ASP_Pricing_File.zip
2011-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Oct_2011_ASP_Pricing_File.zip
2011-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/July_2011_ASP_Pricing_File.zip
2011-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April_2011_ASP_Pricing_File.zip
2011-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/January2011_ASP_PricingFile.zip
2010-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/October_2010_ASP_Pricing_File.zip
2010-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/July_2010_ASP_Pricing_File.zip
2010-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April_2010_ASP_Pricing_File.zip
2010-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/January_2010_ASP_Pricing_File.zip
2009-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/October_2009_ASP_Pricing_File.zip
2009-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/July_2009_ASP_Pricing_File.zip
2009-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April_2009_ASP_Pricing_File_by_HCPCS.zip
2009-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/JAN_2009_ASP_Pricing_File_by_HCPCS.zip
2008-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/October2008ASPPricingFilebyHCPCS.zip
2008-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/July2008ASPPricingFilebyHCPCS.zip
2008-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April08ASPbyHCPCS.zip
2008-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jan08ASPbyHCPCS.zip
2007-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/October07ASPbyHCPCS.zip
2007-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/July07ASPbyHCPCS_121707.zip
2007-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April07ASPbyHCPCS_121707.zip
2007-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jan07ASPbyHCPCS_121707.zip
2006-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/oct06asp_hcpcs.zip
2006-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/july06asp_hcpcs.zip
2006-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/apr06pricing.zip
2006-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jan06pricing.zip
2005-Q4,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/oct05aspbyhcpcs033106v2.zip
2005-Q3,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/jul05aspbyhcpcs_032906.zip
2005-Q2,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/apr05aspbyhcpcsv8_033106.zip
2005-Q1,https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/jan05aspbyhcpcsv6_033106.zip
1 quarter url
2 2026-Q2 https://www.cms.gov/files/zip/april-2026-medicare-part-b-payment-limit-files-03-24-2026-final-file.zip
3 2026-Q1 https://www.cms.gov/files/zip/january-2026-medicare-part-b-payment-limit-files.zip
4 2025-Q4 https://www.cms.gov/files/zip/october-2025-asp-pricing-final-file.zip
5 2025-Q3 https://www.cms.gov/files/zip/july-2025-asp-pricing-file.zip
6 2025-Q2 https://www.cms.gov/files/zip/april-2025-asp-pricing-file.zip
7 2025-Q1 https://www.cms.gov/files/zip/january-2025-asp-pricing-file-03/11/25-final-file.zip
8 2024-Q4 https://www.cms.gov/files/zip/october-2024-asp-pricing-file.zip
9 2024-Q3 https://www.cms.gov/files/zip/july-2024-asp-pricing-file.zip
10 2024-Q2 https://www.cms.gov/files/zip/april-2024-asp-pricing-file.zip
11 2024-Q1 https://www.cms.gov/files/zip/january-2024-asp-pricing-file.zip
12 2023-Q4 https://www.cms.gov/files/zip/october-2023-asp-pricing-file.zip
13 2023-Q3 https://www.cms.gov/files/zip/july-2023-asp-pricing-file.zip
14 2023-Q2 https://www.cms.gov/files/zip/april-2023-asp-pricing-file.zip
15 2023-Q1 https://www.cms.gov/files/zip/january-2023-asp-pricing-file.zip
16 2022-Q4 https://www.cms.gov/files/zip/october-2022-asp-pricing-file.zip
17 2022-Q3 https://www.cms.gov/files/zip/july-2022-asp-pricing-file.zip
18 2022-Q2 https://www.cms.gov/files/zip/april-2022-asp-pricing-file.zip
19 2022-Q1 https://www.cms.gov/files/zip/january-2022-asp-pricing-file.zip
20 2021-Q4 https://www.cms.gov/files/zip/october-2021-asp-pricing-file.zip
21 2021-Q3 https://www.cms.gov/files/zip/july-2021-asp-pricing-file.zip
22 2021-Q2 https://www.cms.gov/files/zip/april-2021-asp-pricing-file.zip
23 2021-Q1 https://www.cms.gov/files/zip/january-2021-asp-pricing-file.zip
24 2020-Q4 https://www.cms.gov/files/zip/october-2020-asp-pricing-file.zip
25 2020-Q3 https://www.cms.gov/files/zip/july-2020-asp-pricing-file-updated-06012020.zip
26 2020-Q2 https://www.cms.gov/files/zip/april-2020-asp-pricing-file.zip
27 2020-Q1 https://www.cms.gov/files/zip/january-2020-asp-pricing-file.zip
28 2019-Q4 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2019-Oct-ASP-Pricing-File.zip
29 2019-Q3 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2019-July-ASP-Pricing-File.zip
30 2019-Q2 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April-2019-ASP-Pricing-File.zip
31 2019-Q1 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2019-January-ASP-Pricing-File.zip
32 2018-Q4 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2018-Oct-ASP-Pricing-File.zip
33 2018-Q3 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2018-July-ASP-Pricing-File.zip
34 2018-Q2 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2018-April-ASP-Pricing-File.zip
35 2018-Q1 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2018-January-ASP-Pricing-File.zip
36 2017-Q4 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2017-October-ASP-Pricing-File.zip
37 2017-Q3 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2017-July-ASP-Pricing-File.zip
38 2017-Q2 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2017-April-ASP-Pricing.zip
39 2017-Q1 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2017-January-ASP-Pricing-Files.zip
40 2016-Q4 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2016-October-ASP-Pricing-File.zip
41 2016-Q3 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/Downloads/2016-July-ASP-Pricing-File.zip
42 2016-Q2 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2016-April-ASP-Pricing-File.zip
43 2016-Q1 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2016-January-ASP-Pricing-File.zip
44 2015-Q4 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2015-October-ASP-Pricing-File.zip
45 2015-Q3 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2015-July-ASP-Pricing-File.zip
46 2015-Q2 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2015-April-ASP-Pricing-File.zip
47 2015-Q1 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2015-January-ASP-Pricing-File.zip
48 2014-Q4 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2014-October-ASP-Pricing-File.zip
49 2014-Q3 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jul-2014-ASP-Pricing-File.zip
50 2014-Q2 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Apr-14-ASP-Pricing-File.zip
51 2014-Q1 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jan-14-ASP-Pricing-File.zip
52 2013-Q4 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2013-October-ASP-Pricing-File.zip
53 2013-Q3 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/2013-July-ASP-Pricing-File.zip
54 2013-Q2 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Apr-13-ASP-Pricing-file.zip
55 2013-Q1 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jan-2013-ASP-Pricing-File.zip
56 2012-Q4 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/October-2012-ASP-Pricing-File.zip
57 2012-Q3 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/July-2012-ASP-Pricing-File.zip
58 2012-Q2 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April-2012-ASP-Pricing-file-revised030513.zip
59 2012-Q1 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jan_2012_ASP_Pricing_File.zip
60 2011-Q4 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Oct_2011_ASP_Pricing_File.zip
61 2011-Q3 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/July_2011_ASP_Pricing_File.zip
62 2011-Q2 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April_2011_ASP_Pricing_File.zip
63 2011-Q1 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/January2011_ASP_PricingFile.zip
64 2010-Q4 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/October_2010_ASP_Pricing_File.zip
65 2010-Q3 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/July_2010_ASP_Pricing_File.zip
66 2010-Q2 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April_2010_ASP_Pricing_File.zip
67 2010-Q1 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/January_2010_ASP_Pricing_File.zip
68 2009-Q4 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/October_2009_ASP_Pricing_File.zip
69 2009-Q3 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/July_2009_ASP_Pricing_File.zip
70 2009-Q2 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April_2009_ASP_Pricing_File_by_HCPCS.zip
71 2009-Q1 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/JAN_2009_ASP_Pricing_File_by_HCPCS.zip
72 2008-Q4 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/October2008ASPPricingFilebyHCPCS.zip
73 2008-Q3 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/July2008ASPPricingFilebyHCPCS.zip
74 2008-Q2 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April08ASPbyHCPCS.zip
75 2008-Q1 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jan08ASPbyHCPCS.zip
76 2007-Q4 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/October07ASPbyHCPCS.zip
77 2007-Q3 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/July07ASPbyHCPCS_121707.zip
78 2007-Q2 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/April07ASPbyHCPCS_121707.zip
79 2007-Q1 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jan07ASPbyHCPCS_121707.zip
80 2006-Q4 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/oct06asp_hcpcs.zip
81 2006-Q3 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/july06asp_hcpcs.zip
82 2006-Q2 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/apr06pricing.zip
83 2006-Q1 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/Jan06pricing.zip
84 2005-Q4 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/oct05aspbyhcpcs033106v2.zip
85 2005-Q3 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/jul05aspbyhcpcs_032906.zip
86 2005-Q2 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/apr05aspbyhcpcsv8_033106.zip
87 2005-Q1 https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/downloads/jan05aspbyhcpcsv6_033106.zip

353
dev/scripts/ingest_asp.py Normal file
View File

@@ -0,0 +1,353 @@
"""Ingest CMS Average Sales Price (ASP) Drug Pricing Files.
Reads all quarterly ASP ZIP files from data/cms/asp/, normalises column
names across 20+ years of CMS format drift, filters **strictly** to
skin substitute HCPCS codes defined in data/cms/skin_substitute_hcpcs.csv,
and produces:
* data/cms/asp/skin_subs_asp_quarterly.csv (flat file with provenance)
* skin_subs.asp_quarterly (DuckDB table)
Every output row carries ``source_url`` and ``source_file`` columns for
full provenance — each observation traces back to a specific CMS
quarterly ZIP and the file within it that was parsed.
Source URLs are read from data/cms/asp_source_urls.csv (canonical URLs
provided from CMS.gov, one per quarter 2005-Q1 through 2026-Q2).
Column mapping rationale
------------------------
CMS has changed column names across 20+ years of ASP files:
20052006: "HCPCS Code", "Short Description", "HCPCS Code Dosage",
"Estimated ASP", "Payment Limit"
20072011: Same but sometimes "Dosage" instead of "HCPCS Code Dosage"
20122019: "HCPCS Code", "Short Description", "Dosage",
"Payment Limit"
20202025: "HCPCS Code", "Short Description", "HCPCS Code Dosage",
"Payment Limit"
2026+: "HCPCS Code", "Short Description", "Dosage or Unit",
"Payment Limit" (now "Payment Limit" for skin subs is
flat $127.14/cm² under reclassification)
All variants are mapped to: hcpcs_code, short_description,
dosage_or_unit, payment_limit.
The ``asp_per_unit`` column is derived: payment_limit / 1.06, since
CMS payment limit = ASP + 6% for drugs/biologicals. For 2026+ skin
subs under reclassification, this derivation no longer applies (flat
rate), but we retain it for consistency and flag it in notes.
Filtering
---------
We filter STRICTLY to codes present in skin_substitute_hcpcs.csv
(the code universe built from CMS rulemaking in issue #232). This
is critical because the Q4xxx prefix includes non-skin-sub codes
(Q4054 darbepoetin alfa, Q4055 epoetin alfa, Q4074-Q4082 various
ESRD drugs) that must be excluded.
Usage:
uv run python dev/scripts/ingest_asp.py
"""
from __future__ import annotations
import csv
import io
import re
import zipfile
from pathlib import Path
import duckdb
import pandas as pd
# ---------------------------------------------------------------------------
# Paths
# ---------------------------------------------------------------------------
ROOT = Path(__file__).resolve().parents[2]
ASP_DIR = ROOT / "data" / "cms" / "asp"
HCPCS_REF = ROOT / "data" / "cms" / "skin_substitute_hcpcs.csv"
URL_REF = ROOT / "data" / "cms" / "asp_source_urls.csv"
DUCKDB_PATH = ROOT / "data" / "aco.duckdb"
OUTPUT_CSV = ASP_DIR / "skin_subs_asp_quarterly.csv"
def _load_source_urls() -> dict[str, str]:
"""Load canonical source URLs: {quarter: url}."""
urls = {}
if URL_REF.exists():
with URL_REF.open() as f:
for row in csv.DictReader(f):
urls[row["quarter"]] = row["url"]
return urls
# ---------------------------------------------------------------------------
# Column normalisation
# ---------------------------------------------------------------------------
# Mapping from observed CMS column names (lowercased) to our standard names.
# Each entry documents which CMS file years use that variant.
COL_MAP = {
# HCPCS code — consistent across all years
"hcpcs code": "hcpcs_code",
"hcpcs_code": "hcpcs_code",
# Short description — consistent
"short description": "short_description",
"short_description": "short_description",
# Dosage/unit — varies by year
"hcpcs code dosage": "dosage_or_unit", # 2005-2006, 2020-2025
"hcpcs_code_dosage": "dosage_or_unit",
"dosage": "dosage_or_unit", # 2007-2019
"dosage or unit": "dosage_or_unit", # 2026+
# Payment limit — consistent (ASP + 6%)
"payment limit": "payment_limit",
"payment_limit": "payment_limit",
}
KEEP_COLS = ["hcpcs_code", "short_description", "dosage_or_unit", "payment_limit"]
def normalise_columns(df: pd.DataFrame) -> pd.DataFrame:
"""Lower-case columns, apply COL_MAP, keep only KEEP_COLS."""
df.columns = [str(c).strip().lower() for c in df.columns]
df = df.rename(columns=COL_MAP)
present = [c for c in KEEP_COLS if c in df.columns]
return df[present]
# ---------------------------------------------------------------------------
# Readers
# ---------------------------------------------------------------------------
def _find_header_row(lines: list[str]) -> int:
"""Return the 0-based row index of the column-header row.
The header row starts with 'HCPCS Code,' (possibly quoted).
Must NOT match note lines that incidentally mention 'HCPCS code'.
"""
for i, line in enumerate(lines):
stripped = line.strip().lower()
if stripped.startswith("hcpcs code,") or stripped.startswith('"hcpcs code",'):
return i
return -1
def read_csv_from_zip(zf: zipfile.ZipFile, name: str) -> pd.DataFrame:
"""Read a section-508 CSV inside a ZIP, skipping preamble rows."""
raw = zf.read(name).decode("latin-1")
lines = raw.splitlines()
hdr = _find_header_row(lines)
if hdr < 0:
return pd.DataFrame()
buf = "\n".join(lines[hdr:])
df = pd.read_csv(io.StringIO(buf), dtype=str, on_bad_lines="skip")
return normalise_columns(df)
def read_xls_from_zip(zf: zipfile.ZipFile, name: str) -> pd.DataFrame:
"""Read .xls inside a ZIP, skipping preamble rows."""
import xlrd
data = zf.read(name)
wb = xlrd.open_workbook(file_contents=data)
sh = wb.sheet_by_index(0)
hdr_row = -1
for r in range(min(30, sh.nrows)):
first_cell = str(sh.cell_value(r, 0)).strip().lower()
if first_cell == "hcpcs code":
hdr_row = r
break
if hdr_row < 0:
return pd.DataFrame()
headers = [str(sh.cell_value(hdr_row, c)).strip() for c in range(sh.ncols)]
rows = []
for r in range(hdr_row + 1, sh.nrows):
row = [str(sh.cell_value(r, c)).strip() for c in range(sh.ncols)]
if any(row):
rows.append(row)
df = pd.DataFrame(rows, columns=headers)
return normalise_columns(df)
def pick_pricing_file(names: list[str]) -> str | None:
"""From a list of files in a ZIP, pick the ASP pricing file.
Prefer CSV (section 508) over XLS. Skip NOC, crosswalk, NDC,
and "not payable" files.
"""
skip_patterns = ["noc", "crosswalk", "ndc", "not payable", "not_payable"]
candidates: list[str] = []
for n in names:
low = n.lower()
if any(s in low for s in skip_patterns):
continue
if low.endswith((".csv", ".xls", ".xlsx")):
candidates.append(n)
csvs = [c for c in candidates if c.lower().endswith(".csv")]
if csvs:
for c in csvs:
low = c.lower()
if "payment limit" in low or "pricing" in low:
return c
return csvs[0]
xlss = [c for c in candidates if c.lower().endswith((".xls", ".xlsx"))]
if xlss:
for c in xlss:
low = c.lower()
if "payment limit" in low or "pricing" in low or "asp" in low:
return c
return xlss[0]
return None
# ---------------------------------------------------------------------------
# Numeric cleaning
# ---------------------------------------------------------------------------
def clean_numeric(series: pd.Series) -> pd.Series:
"""Strip $ signs and convert to float, coercing errors to NaN."""
return (
series.astype(str)
.str.replace("$", "", regex=False)
.str.replace(",", "", regex=False)
.str.strip()
.pipe(pd.to_numeric, errors="coerce")
)
# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------
def main() -> None:
# Load skin-substitute HCPCS codes (strict filter — no regex fallback)
hcpcs_ref = pd.read_csv(HCPCS_REF, dtype=str)
skin_codes = set(hcpcs_ref["hcpcs_code"].str.strip().str.upper())
print(f"Loaded {len(skin_codes)} skin-substitute HCPCS codes from reference")
# Load canonical source URLs
source_urls = _load_source_urls()
print(f"Loaded {len(source_urls)} source URLs")
# Ingest all quarters
zips = sorted(ASP_DIR.glob("asp_*.zip"))
print(f"Found {len(zips)} quarterly ZIP files\n")
frames: list[pd.DataFrame] = []
for zpath in zips:
m = re.match(r"asp_(\d{4}-Q\d)", zpath.name)
if not m:
print(f" SKIP {zpath.name} (cannot parse quarter)")
continue
quarter = m.group(1)
try:
with zipfile.ZipFile(zpath) as zf:
target = pick_pricing_file(zf.namelist())
if target is None:
print(f" SKIP {quarter}: no pricing file found in ZIP")
continue
if target.lower().endswith(".csv"):
df = read_csv_from_zip(zf, target)
else:
df = read_xls_from_zip(zf, target)
if df.empty:
print(f" SKIP {quarter}: empty after parsing {target}")
continue
df["quarter"] = quarter
df["source_file"] = target
df["source_url"] = source_urls.get(quarter, "")
frames.append(df)
print(f" OK {quarter}: {len(df):>5} rows from {target}")
except Exception as e:
print(f" ERR {quarter}: {e}")
if not frames:
raise RuntimeError("No data parsed from any file")
all_df = pd.concat(frames, ignore_index=True)
print(f"\nTotal rows across all quarters: {len(all_df)}")
# Normalise HCPCS codes
all_df["hcpcs_code"] = all_df["hcpcs_code"].astype(str).str.strip().str.upper()
# Filter STRICTLY to skin substitute code universe
# NO regex fallback — only codes in skin_substitute_hcpcs.csv
skin_df = all_df[all_df["hcpcs_code"].isin(skin_codes)].copy()
print(f"Skin substitute rows (strict filter): {len(skin_df)}")
# Show what we excluded
q4_but_not_skin = all_df[
all_df["hcpcs_code"].str.match(r"^Q4\d{2,3}$")
& ~all_df["hcpcs_code"].isin(skin_codes)
]["hcpcs_code"].unique()
if len(q4_but_not_skin) > 0:
print(f" Excluded Q4 codes NOT in skin sub universe: {sorted(q4_but_not_skin)}")
if not skin_df.empty:
skin_df["payment_limit"] = clean_numeric(skin_df["payment_limit"])
# ASP per unit = payment_limit / 1.06
# (CMS payment limit = ASP + 6% for drugs/biologicals)
skin_df["asp_per_unit"] = (skin_df["payment_limit"] / 1.06).round(3)
# Reorder columns
out_cols = [
"quarter",
"hcpcs_code",
"short_description",
"asp_per_unit",
"payment_limit",
"dosage_or_unit",
"source_url",
"source_file",
]
for c in out_cols:
if c not in skin_df.columns:
skin_df[c] = None
skin_df = skin_df[out_cols].sort_values(["quarter", "hcpcs_code"])
# Write CSV
skin_df.to_csv(OUTPUT_CSV, index=False)
print(f"\nWrote {len(skin_df)} rows to {OUTPUT_CSV}")
if len(skin_df) > 0:
quarters = sorted(skin_df["quarter"].unique())
codes = sorted(skin_df["hcpcs_code"].unique())
print(f" Quarters: {len(quarters)} ({quarters[0]} to {quarters[-1]})")
print(f" Unique HCPCS codes: {len(codes)}")
# Load into DuckDB
print(f"\nLoading into DuckDB at {DUCKDB_PATH} ...")
con = duckdb.connect(str(DUCKDB_PATH))
con.execute("CREATE SCHEMA IF NOT EXISTS skin_subs")
con.execute("DROP TABLE IF EXISTS skin_subs.asp_quarterly")
if not skin_df.empty:
con.execute(
"""
CREATE TABLE skin_subs.asp_quarterly AS
SELECT * FROM read_csv_auto(?, header=true)
""",
[str(OUTPUT_CSV)],
)
count = con.execute(
"SELECT count(*) FROM skin_subs.asp_quarterly"
).fetchone()[0]
print(f" Loaded {count} rows into skin_subs.asp_quarterly")
else:
print(" WARNING: No skin substitute data found in ASP files")
print(" This may be expected — skin subs may not appear in main ASP pricing files")
print(" They may be in separate NOC or tissue coding files")
con.close()
if __name__ == "__main__":
main()