feat: CMS rulemaking comment analysis module (regulations.gov) #251

New Issue

kert · 2026-03-25T19:29:43Z

kert commented

2026-03-25 19:29:43 +00:00

Objective

Build a module for downloading, extracting, and analyzing public comments on CMS rulemaking dockets, starting with the skin substitute reclassification rules.

Reference project: jacobmr/hti5 — ONC HTI-5 comment analysis (298 comments, position classification, thematic analysis, coordination detection, provision mapping).

Target Dockets

CMS-1834-P (CY 2026 OPPS proposed rule) — skin sub reclassification from drugs/biologicals to incident-to supplies
CMS-1809-FC (CY 2025 OPPS final rule) — high/low cost category changes
CMS-1784-F (CY 2024 PFS final rule) — billing requirements

Pipeline (modeled on hti5)

Download — regulations.gov API bulk download of comments + PDF/DOCX attachments
Extract — pypdf/python-docx text extraction from attachments (70%+ of substance is in PDFs)
Classify — Position on 5-point scale (strongly oppose reclassification → strongly support)
Themes — Tag per comment: ASP methodology, flat rate impact, patient access, innovation, fraud/waste, clinical evidence, manufacturer impact, provider impact, geographic disparities
Stakeholder segmentation — manufacturer, distributor, provider, wound care society, patient advocacy, payer, government agency
Coordination detection — Jaccard similarity on n-grams, keyword fingerprinting, temporal clustering for form letter campaigns
Provision mapping — Map each comment to specific regulatory provisions (ASP+6%, flat rate, HCPCS codes, pass-through, documentation)
Sentiment aggregation — Aggregate position by stakeholder type, by provision, by theme

Architecture

New module: src/rex/comments/ (or src/cms/comments/)
CLI: stack comments fetch <docket-id>, stack comments analyze, stack comments report
Store: skin_subs.rulemaking_comments in DuckDB, items in bib.sqlite
Marimo notebook for interactive exploration
Separate PR/branch — this is the highest priority new feature

Key Insight from hti5

PDF attachment extraction is critical (70%+ of substance)
Deterministic weighted regex for classification is effective and reproducible
Coordination detection reveals industry-organized campaigns
Provision-level mapping is more valuable than binary for/against
regulations.gov API requires User-Agent header and rate limiting

## Objective Build a module for downloading, extracting, and analyzing public comments on CMS rulemaking dockets, starting with the skin substitute reclassification rules. **Reference project:** [jacobmr/hti5](https://github.com/jacobmr/hti5) — ONC HTI-5 comment analysis (298 comments, position classification, thematic analysis, coordination detection, provision mapping). ## Target Dockets - **CMS-1834-P** (CY 2026 OPPS proposed rule) — skin sub reclassification from drugs/biologicals to incident-to supplies - **CMS-1809-FC** (CY 2025 OPPS final rule) — high/low cost category changes - **CMS-1784-F** (CY 2024 PFS final rule) — billing requirements ## Pipeline (modeled on hti5) 1. **Download** — regulations.gov API bulk download of comments + PDF/DOCX attachments 2. **Extract** — pypdf/python-docx text extraction from attachments (70%+ of substance is in PDFs) 3. **Classify** — Position on 5-point scale (strongly oppose reclassification → strongly support) 4. **Themes** — Tag per comment: ASP methodology, flat rate impact, patient access, innovation, fraud/waste, clinical evidence, manufacturer impact, provider impact, geographic disparities 5. **Stakeholder segmentation** — manufacturer, distributor, provider, wound care society, patient advocacy, payer, government agency 6. **Coordination detection** — Jaccard similarity on n-grams, keyword fingerprinting, temporal clustering for form letter campaigns 7. **Provision mapping** — Map each comment to specific regulatory provisions (ASP+6%, flat rate, HCPCS codes, pass-through, documentation) 8. **Sentiment aggregation** — Aggregate position by stakeholder type, by provision, by theme ## Architecture - New module: `src/rex/comments/` (or `src/cms/comments/`) - CLI: `stack comments fetch <docket-id>`, `stack comments analyze`, `stack comments report` - Store: `skin_subs.rulemaking_comments` in DuckDB, items in bib.sqlite - Marimo notebook for interactive exploration - **Separate PR/branch** — this is the highest priority new feature ## Key Insight from hti5 - PDF attachment extraction is critical (70%+ of substance) - Deterministic weighted regex for classification is effective and reproducible - Coordination detection reveals industry-organized campaigns - Provision-level mapping is more valuable than binary for/against - regulations.gov API requires User-Agent header and rate limiting

kert added this to the P30: CMS Rulemaking Comment Analysis — regulations.gov sentiment + coordination detection milestone 2026-03-25 19:31:53 +00:00

kert referenced a pull request that will close this issue

2026-03-25 19:50:49 +00:00

feat: CMS rulemaking comment analysis module (rex.comments) #261

kert referenced this issue from a commit

2026-04-23 15:18:58 +00:00

perf(bib): concurrent CDN attachment downloads + 3× API throughput

kert commented

2026-04-23 15:28:19 +00:00

Backfill perf update — commit 22cbc74

API sleep tightened: 3.7s → 1.3s (under reg.gov 50/min burst cap, ~2700/hr theoretical)
Attachment downloads now run concurrently on the CDN (downloads.regulations.gov, 4 workers) — they don't count against the API rate limit
bib-backfill-comments.service restarted on the new code at 09:33 EDT

Progress: ~34,948 / 56,568 enriched (~62%). Sustained ~14 items/min post-restart; API pacing is no longer the bottleneck — attachment download time is. Errors steady at 7 in this run (1120 lifetime, mostly older dockets).

Attachments land in .state/comments/{docket_id}/{comment_id}/, ready for the #253 extraction pipeline.

**Backfill perf update** — commit 22cbc74 - API sleep tightened: 3.7s → 1.3s (under reg.gov 50/min burst cap, ~2700/hr theoretical) - Attachment downloads now run concurrently on the CDN (`downloads.regulations.gov`, 4 workers) — they don't count against the API rate limit - `bib-backfill-comments.service` restarted on the new code at 09:33 EDT **Progress**: ~34,948 / 56,568 enriched (~62%). Sustained ~14 items/min post-restart; API pacing is no longer the bottleneck — attachment download time is. Errors steady at 7 in this run (1120 lifetime, mostly older dockets). Attachments land in `.state/comments/{docket_id}/{comment_id}/`, ready for the #253 extraction pipeline.

kert commented

2026-07-10 20:49:05 +00:00

Closing the umbrella — the pipeline's foundation is shipped and the remaining analysis stages have their own open issues carrying all the detail:

Download ✓ — stack bib backfill-comments (regulations.gov API, comments + attachments); ~164k source:regulations-gov items in bib, routed to Zotero Rules/Comments/<docket> collections (https://git.fhirworx.io/homelab/stack/commit/c53571c), including the target dockets.
Extract ✓ — #253 (closed): src/rex/comments/, combined.md per comment in Zotero.
Classify / Themes → #254 (open)
Stakeholders / Coordination → #255 (open)
Aggregate / Dashboard / Provision mapping → #256 (open)

The skin_subs.rulemaking_comments DuckDB table already exists with the full column schema for those stages (position, position_score, themes, stakeholder_type, coordination_group, is_form_letter, provisions) but only a 15-row pilot — the at-scale runs are what #254–#256 track.

Closing the umbrella — the pipeline's foundation is shipped and the remaining analysis stages have their own open issues carrying all the detail: - **Download** ✓ — `stack bib backfill-comments` (regulations.gov API, comments + attachments); ~164k `source:regulations-gov` items in bib, routed to Zotero `Rules/Comments/<docket>` collections (https://git.fhirworx.io/homelab/stack/commit/c53571c), including the target dockets. - **Extract** ✓ — #253 (closed): `src/rex/comments/`, combined.md per comment in Zotero. - **Classify / Themes** → #254 (open) - **Stakeholders / Coordination** → #255 (open) - **Aggregate / Dashboard / Provision mapping** → #256 (open) The `skin_subs.rulemaking_comments` DuckDB table already exists with the full column schema for those stages (position, position_score, themes, stakeholder_type, coordination_group, is_form_letter, provisions) but only a 15-row pilot — the at-scale runs are what #254–#256 track.

kert closed this issue

2026-07-10 20:49:05 +00:00

kert referenced this issue

2026-07-10 20:49:06 +00:00

comments: position classification + thematic tagging #254

kert referenced this issue

2026-07-10 20:49:07 +00:00

comments: extract author/org metadata from attachment text #416

Sign in to join this conversation.