Files
stack/docs/docs/cli/bib-backfill-comments.md
kert a790b74179
Some checks failed
CI / lint (push) Successful in 54s
CI / notebooks-smoke (push) Successful in 2m21s
Deploy / changes (push) Successful in 19s
CI / test (push) Failing after 4m46s
Infra CI / notebooks (push) Successful in 1m7s
Infra CI / zotero (push) Successful in 29s
Infra CI / api (push) Successful in 22s
Infra CI / bcda (push) Successful in 21s
Infra CI / llm (push) Successful in 17s
Infra CI / docs (push) Successful in 1m32s
Deploy / notebooks (push) Has been skipped
Deploy / zotero (push) Has been skipped
Infra CI / mc (push) Successful in 19s
Deploy / docs (push) Successful in 50s
Deploy / api (push) Successful in 1m40s
Deploy / bcda (push) Successful in 1m59s
Deploy / mc (push) Has been skipped
Deploy / llm (push) Successful in 1m11s
Deploy / report (push) Successful in 16s
test(pfs,llm): home-visit dossier golden (fixture + live) and chat golden entry; CLI docs
2026-11-13 01:44:12 -05:00

3.7 KiB

title, sidebar_position
title sidebar_position
stack bib backfill-comments 41

stack bib backfill-comments

Usage: stack bib backfill-comments [OPTIONS]                                   
                                                                                
 Enrich every reg-gov comment stub with body + attachments.                     
                                                                                
 Walks items tagged ``source:regulations-gov`` whose ``abstract`` is            
 still empty and hits ``/v4/comments/{id}?include=attachments`` for             
 each one. Resumable: stop any time, start again, it picks up where             
 it left off by skipping already-enriched items. At 1.3s/req the API            
 calls run at ~2700/hr; attachment downloads are concurrent on the              
 CDN and don't count against the rate limit.                                    
                                                                                
 ``--mirror`` swaps the API for the Mirrulations public S3 mirror               
 (#662): no API key, no cap, and mirror-only comments get created.              
                                                                                
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --docket          <str>    Limit to one docket (e.g. CMS-2026-2377). Without │
│                            it the walk covers every un-enriched stub in the  │
│                            store — a multi-day crawl at the reg.gov rate     │
│                            cap.                                              │
│ --limit   -n      <int>    Cap items processed this run. 0 = no cap          │
│                            (multi-day crawl).                                │
│                            [default: 0]                                      │
│ --sleep           <float>  Seconds between API calls. 3.7s ≈ 970/hr — just   │
│                            under the 1000/hr reg.gov hourly cap (the binding │
│                            limit; the 50/min burst cap is looser). CDN       │
│                            downloads are concurrent and don't count.         │
│                            [default: 3.7]                                    │
│ --log             <path>   Append-only progress log (survives restarts).     │
│                            [default: /tmp/bib-backfill-comments.log]         │
│ --mirror                   Pull from the Mirrulations S3 mirror instead of   │
│                            the reg.gov API — bulk speed, no rate cap, also   │
│                            ingests comments the original farm never listed.  │
│                            Requires --docket.                                │
│ --force                    Run even if the docket is sealed. Bypasses the    │
│                            docket seal only: items already tagged            │
│                            enriched:ok (or enriched:gone) are still skipped, │
│                            so this resumes an interrupted enrichment — it    │
│                            never re-enriches an item.                        │
│ --help                     Show this message and exit.                       │
╰──────────────────────────────────────────────────────────────────────────────╯