Some checks failed
CI / lint (push) Successful in 54s
CI / notebooks-smoke (push) Successful in 2m21s
Deploy / changes (push) Successful in 19s
CI / test (push) Failing after 4m46s
Infra CI / notebooks (push) Successful in 1m7s
Infra CI / zotero (push) Successful in 29s
Infra CI / api (push) Successful in 22s
Infra CI / bcda (push) Successful in 21s
Infra CI / llm (push) Successful in 17s
Infra CI / docs (push) Successful in 1m32s
Deploy / notebooks (push) Has been skipped
Deploy / zotero (push) Has been skipped
Infra CI / mc (push) Successful in 19s
Deploy / docs (push) Successful in 50s
Deploy / api (push) Successful in 1m40s
Deploy / bcda (push) Successful in 1m59s
Deploy / mc (push) Has been skipped
Deploy / llm (push) Successful in 1m11s
Deploy / report (push) Successful in 16s
3.7 KiB
3.7 KiB
title, sidebar_position
| title | sidebar_position |
|---|---|
| stack bib backfill-comments | 41 |
stack bib backfill-comments
Usage: stack bib backfill-comments [OPTIONS]
Enrich every reg-gov comment stub with body + attachments.
Walks items tagged ``source:regulations-gov`` whose ``abstract`` is
still empty and hits ``/v4/comments/{id}?include=attachments`` for
each one. Resumable: stop any time, start again, it picks up where
it left off by skipping already-enriched items. At 1.3s/req the API
calls run at ~2700/hr; attachment downloads are concurrent on the
CDN and don't count against the rate limit.
``--mirror`` swaps the API for the Mirrulations public S3 mirror
(#662): no API key, no cap, and mirror-only comments get created.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --docket <str> Limit to one docket (e.g. CMS-2026-2377). Without │
│ it the walk covers every un-enriched stub in the │
│ store — a multi-day crawl at the reg.gov rate │
│ cap. │
│ --limit -n <int> Cap items processed this run. 0 = no cap │
│ (multi-day crawl). │
│ [default: 0] │
│ --sleep <float> Seconds between API calls. 3.7s ≈ 970/hr — just │
│ under the 1000/hr reg.gov hourly cap (the binding │
│ limit; the 50/min burst cap is looser). CDN │
│ downloads are concurrent and don't count. │
│ [default: 3.7] │
│ --log <path> Append-only progress log (survives restarts). │
│ [default: /tmp/bib-backfill-comments.log] │
│ --mirror Pull from the Mirrulations S3 mirror instead of │
│ the reg.gov API — bulk speed, no rate cap, also │
│ ingests comments the original farm never listed. │
│ Requires --docket. │
│ --force Run even if the docket is sealed. Bypasses the │
│ docket seal only: items already tagged │
│ enriched:ok (or enriched:gone) are still skipped, │
│ so this resumes an interrupted enrichment — it │
│ never re-enriches an item. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯