comments: finish the detail scrape — 160.5k unenriched stubs cap every text-based exhibit #662

New Issue

kert · 2026-08-19T18:12:11Z

kert commented

2026-08-19 18:12:11 +00:00

Section 4 of the palliative RFI notebook disclosed that only 28.4k of 166.9k docketed PFS comments have locally retrievable text. Diagnosis (2026-08-19):

160,523 comment items are unenriched (enriched:ok on only 6,418; enriched:gone 0): the initial farm captured list-level metadata only (extra_json = {"doc_type": "Public Comment"} — no inline text, no attachment manifest). It is NOT a case of downloaded-but-unextracted PDFs.
The repo already has the full resumable machinery: stack bib backfill-comments hits /v4/comments/{id}?include=attachments at ~970 req/h (reg.gov hourly cap), downloads attachments to .state/comments/, tags enriched:ok|gone. 160.5k pending ≈ 165 hours ≈ 7 days of continuous crawl on the one configured API key.
Separately, 2,393 already-extracted comments carry ocr_needed attachments (image-only PDFs/JPGs); stack comments extract-ocr is still the phase-2 stub (#253 follow-up). That gap persists after the backfill and grows with it.

Plan:

Run the backfill docket-by-docket in palliative-value order (2018 → 2023 → 2024 → 2025 → 2019 → 2021 → 2022 → 2020) as a persistent background service, so high-density dockets land first.
After each docket completes: stack comments extract --docket <D>, then re-run dev/scripts/tag_palliative_comments.py (idempotent) so notebook Section 4 tracks coverage as it improves.
OCR remains out of scope here — tracked as its own phase-2 item (GPU fleet, P45 hardware is live).

Related: #615 (post-close re-farm of CMS-2026-2377), #660 (the exhibit this feeds).

Section 4 of the palliative RFI notebook disclosed that only 28.4k of 166.9k docketed PFS comments have locally retrievable text. Diagnosis (2026-08-19): - **160,523 comment items are unenriched** (`enriched:ok` on only 6,418; `enriched:gone` 0): the initial farm captured list-level metadata only (`extra_json = {"doc_type": "Public Comment"}` — no inline text, no attachment manifest). It is NOT a case of downloaded-but-unextracted PDFs. - The repo already has the full resumable machinery: `stack bib backfill-comments` hits `/v4/comments/{id}?include=attachments` at ~970 req/h (reg.gov hourly cap), downloads attachments to `.state/comments/`, tags `enriched:ok|gone`. **160.5k pending ≈ 165 hours ≈ 7 days** of continuous crawl on the one configured API key. - Separately, **2,393 already-extracted comments carry `ocr_needed` attachments** (image-only PDFs/JPGs); `stack comments extract-ocr` is still the phase-2 stub (#253 follow-up). That gap persists after the backfill and grows with it. Plan: 1. Run the backfill docket-by-docket in palliative-value order (2018 → 2023 → 2024 → 2025 → 2019 → 2021 → 2022 → 2020) as a persistent background service, so high-density dockets land first. 2. After each docket completes: `stack comments extract --docket <D>`, then re-run `dev/scripts/tag_palliative_comments.py` (idempotent) so notebook Section 4 tracks coverage as it improves. 3. OCR remains out of scope here — tracked as its own phase-2 item (GPU fleet, P45 hardware is live). Related: #615 (post-close re-farm of CMS-2026-2377), #660 (the exhibit this feeds).

kert added the bib label 2026-08-19 18:12:11 +00:00

kert referenced this issue

2026-08-19 18:12:31 +00:00

comments: implement extract-ocr — 2,393 comments have image-only attachments invisible to all text exhibits #663

kert commented

2026-08-19 18:31:31 +00:00

Backfill launched 2026-08-19 ~14:45 ET (pid on rack, nohup loop): docket order CMS-2018-0076 → 2023 → 2024 → 2025 → 2019 → 2021 → 2022 → 2020, each followed by stack comments extract --docket and an idempotent re-run of the palliative sweep so #660's notebook section improves as dockets land. First docket: 11,143 pending, enriching at ~970/hr (≈11.5 h). Total ≈ 165 h ≈ 7 days. Progress: tail .state/comments/backfill.log. Gotcha fixed at launch: the CLI needs REGULATIONS_GOV_API_KEY exported — a bare nohup shell doesn't source .env; the loop now does set -a; . .env.

Backfill launched 2026-08-19 ~14:45 ET (pid on rack, nohup loop): docket order CMS-2018-0076 → 2023 → 2024 → 2025 → 2019 → 2021 → 2022 → 2020, each followed by `stack comments extract --docket` and an idempotent re-run of the palliative sweep so #660's notebook section improves as dockets land. First docket: 11,143 pending, enriching at ~970/hr (≈11.5 h). Total ≈ 165 h ≈ 7 days. Progress: `tail .state/comments/backfill.log`. Gotcha fixed at launch: the CLI needs REGULATIONS_GOV_API_KEY exported — a bare nohup shell doesn't source .env; the loop now does `set -a; . .env`.

kert commented

2026-08-19 19:01:27 +00:00

Superseded the API crawl with a bulk route (e5c00bd): stack bib backfill-comments --docket D --mirror pulls from the Mirrulations public S3 mirror (AWS Open Data, s3://mirrulations) — comment detail JSON is the v4 API response verbatim (raw-data/CMS/{D}/text-{D}/comments/{cid}.json), attachments sit beside it in the binary tree. No API key, no rate cap, and the mirror is a superset of our farm (15,368 vs 13,245 for CMS-2018-0076; 3,433 vs 2,728 for the live docket) so mirror-only comments get created with the standard tag set.

The API loop was killed at 425/11,143 (~40 min) and the mirror loop launched over all ten dockets (including CMS-2017-0092 and CMS-2026-2377 — mirror is fresher everywhere): 1,800 comments enriched in the first ~90 seconds. The ~7-day crawl is now a same-day job. Each docket is still followed by stack comments extract + the palliative sweep, so #660's exhibit updates as dockets land.

Note on the "many droplets" idea that prompted this: reg.gov's cap is per API key, not per IP, so a VM fleet wouldn't have helped — the mirror sidesteps the API entirely. The reg.gov API remains for freshness gap-fill on the live docket (#615).

Superseded the API crawl with a bulk route (e5c00bd): **`stack bib backfill-comments --docket D --mirror`** pulls from the Mirrulations public S3 mirror (AWS Open Data, `s3://mirrulations`) — comment detail JSON is the v4 API response verbatim (`raw-data/CMS/{D}/text-{D}/comments/{cid}.json`), attachments sit beside it in the binary tree. No API key, no rate cap, and the mirror is a **superset** of our farm (15,368 vs 13,245 for CMS-2018-0076; 3,433 vs 2,728 for the live docket) so mirror-only comments get created with the standard tag set. The API loop was killed at 425/11,143 (~40 min) and the mirror loop launched over all ten dockets (including CMS-2017-0092 and CMS-2026-2377 — mirror is fresher everywhere): **1,800 comments enriched in the first ~90 seconds**. The ~7-day crawl is now a same-day job. Each docket is still followed by `stack comments extract` + the palliative sweep, so #660's exhibit updates as dockets land. Note on the "many droplets" idea that prompted this: reg.gov's cap is per API key, not per IP, so a VM fleet wouldn't have helped — the mirror sidesteps the API entirely. The reg.gov API remains for freshness gap-fill on the live docket (#615).

kert commented

2026-08-20 13:12:47 +00:00

Complete (2026-08-19 16:27 ET, ~90 min wall clock, 0 errors on all ten dockets):

Corpus: 166,941 → 196,739 comments — the mirror surfaced ~30k comments the original farm never listed (19,745 in CMS-2020-0088 alone, +806 on the live docket).
Enrichment: 196,701 of 196,739 (99.98%) now carry inline text, org tags, and attachment manifests — up from 6,418 (3.8%) yesterday morning.
28,228 comments have extracted attachment text locally; per-docket stack comments extract ran inside the loop.
Palliative exhibit (#660): topic:palliative grew 278 → 417 (+50%); biggest gains CMS-2018-0076 78→129, CMS-2025-0304 28→69, CMS-2023-0121 44→57. Notebook Section 4 reads all of this live.
Residual: 38 unenriched stragglers (mirror gaps — API gap-fill candidates), and the OCR backlog grew 2,393 → 2,741 image-only attachments (#663).

Complete (2026-08-19 16:27 ET, ~90 min wall clock, 0 errors on all ten dockets): - Corpus: **166,941 → 196,739 comments** — the mirror surfaced ~30k comments the original farm never listed (19,745 in CMS-2020-0088 alone, +806 on the live docket). - Enrichment: **196,701 of 196,739 (99.98%)** now carry inline text, org tags, and attachment manifests — up from 6,418 (3.8%) yesterday morning. - 28,228 comments have extracted attachment text locally; per-docket `stack comments extract` ran inside the loop. - Palliative exhibit (#660): **`topic:palliative` grew 278 → 417** (+50%); biggest gains CMS-2018-0076 78→129, CMS-2025-0304 28→69, CMS-2023-0121 44→57. Notebook Section 4 reads all of this live. - Residual: 38 unenriched stragglers (mirror gaps — API gap-fill candidates), and the OCR backlog grew 2,393 → 2,741 image-only attachments (#663).

kert closed this issue

2026-08-20 13:12:47 +00:00

kert referenced this issue from a commit

2026-08-26 19:56:39 +00:00

feat(bib): backfill-comments --mirror — bulk enrichment from the Mirrulations S3 mirror (refs #662)

kert referenced this issue from a commit

2026-09-09 14:32:23 +00:00

chore(comments): re-farm CMS-2026-2377 mirror-first, API steps as gap-fill (refs #615 #662)

Sign in to join this conversation.