comments: regulations.gov API client — bulk download comments + attachments #252

New Issue

kert · 2026-03-25T19:30:23Z

kert commented

2026-03-25 19:30:23 +00:00

Objective

Download all public comments and their PDF/DOCX attachments for a given CMS docket.

Tasks

regulations.gov API v4 client with API key, rate limiting, pagination
Download inline comment text to JSON
Download PDF/DOCX attachments via attachment URLs
Deduplication (hti5 found 7 dupes in 305 comments)
Store raw data: data/cms/comments/{docket_id}/raw/
Metadata: commenter name, org, posted date, received date, attachment count
Target: CMS-1834-P (CY 2026 OPPS proposed rule, skin sub reclassification)

API Reference

Base: https://api.regulations.gov/v4/
Endpoints: /documents?filter[docketId]=CMS-1834-P, /comments/{id}, attachments via fileUrl
Rate limit: 1000 req/hr with API key
Requires X-Api-Key header (obtain from regulations.gov)

## Objective Download all public comments and their PDF/DOCX attachments for a given CMS docket. ## Tasks - [ ] regulations.gov API v4 client with API key, rate limiting, pagination - [ ] Download inline comment text to JSON - [ ] Download PDF/DOCX attachments via attachment URLs - [ ] Deduplication (hti5 found 7 dupes in 305 comments) - [ ] Store raw data: `data/cms/comments/{docket_id}/raw/` - [ ] Metadata: commenter name, org, posted date, received date, attachment count - [ ] Target: CMS-1834-P (CY 2026 OPPS proposed rule, skin sub reclassification) ## API Reference - Base: `https://api.regulations.gov/v4/` - Endpoints: `/documents?filter[docketId]=CMS-1834-P`, `/comments/{id}`, attachments via `fileUrl` - Rate limit: 1000 req/hr with API key - Requires `X-Api-Key` header (obtain from regulations.gov)

kert added this to the P30: CMS Rulemaking Comment Analysis — regulations.gov sentiment + coordination detection milestone 2026-03-25 19:31:53 +00:00

kert referenced this issue

2026-03-25 19:50:49 +00:00

feat: CMS rulemaking comment analysis module (rex.comments) #261

kert commented

2026-04-15 23:00:59 +00:00

Delivered as bib/regulations_gov.py + bib/federalregister.py + CLI commands stack bib {discover-pfs-rules, fetch-docket-comments, fetch-pfs-comments, backfill-comments}. Currently a multi-day backfill is running against all 9 PFS dockets 2017–2025 (~164K comments + attachments). Spec deviations:

Storage layout: attachments land in bib attachments table + data/storage/<key>/<filename> instead of data/cms/comments/{docket}/raw/. Functionally equivalent, also lets the same files land in Zotero on stack bib sync-zotero.
Target docket: covered all 9 PFS proposed-rule dockets rather than just CMS-1834-P. Tooling is identical for OPPS — single CLI invocation away (stack bib fetch-docket-comments CMS-1834-P --attachments).
Dedup: at upsert layer (URL match) rather than a separate pass.

Sub-issues #253 (PDF/DOCX text extraction) and #254-#256 (NLP analytics) remain open.

**Delivered** as `bib/regulations_gov.py` + `bib/federalregister.py` + CLI commands `stack bib {discover-pfs-rules, fetch-docket-comments, fetch-pfs-comments, backfill-comments}`. Currently a multi-day backfill is running against all 9 PFS dockets 2017–2025 (~164K comments + attachments). Spec deviations: * Storage layout: attachments land in bib `attachments` table + `data/storage/<key>/<filename>` instead of `data/cms/comments/{docket}/raw/`. Functionally equivalent, also lets the same files land in Zotero on `stack bib sync-zotero`. * Target docket: covered all 9 PFS proposed-rule dockets rather than just CMS-1834-P. Tooling is identical for OPPS — single CLI invocation away (`stack bib fetch-docket-comments CMS-1834-P --attachments`). * Dedup: at upsert layer (URL match) rather than a separate pass. Sub-issues #253 (PDF/DOCX text extraction) and #254-#256 (NLP analytics) remain open.

kert closed this issue

2026-04-15 23:00:59 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#252