eb1b574c04
Some checks failed
CI / lint (push) Successful in 48s
CI / notebooks-smoke (push) Successful in 1m46s
Infra CI / zotero (push) Successful in 15s
Infra CI / llm (push) Successful in 1m4s
Infra CI / notebooks (pull_request) Successful in 55s
Infra CI / docs (pull_request) Successful in 17s
Infra CI / api (pull_request) Successful in 14s
Infra CI / llm (pull_request) Successful in 20s
CI / test (push) Failing after 2m26s
Infra CI / notebooks (push) Successful in 52s
Infra CI / api (push) Successful in 1m14s
Infra CI / docs (push) Successful in 1m45s
Infra CI / mc (push) Failing after 42s
CI / lint (pull_request) Successful in 27s
CI / notebooks-smoke (pull_request) Successful in 1m41s
CI / test (pull_request) Failing after 2m20s
Infra CI / zotero (pull_request) Successful in 21s
Infra CI / mc (pull_request) Failing after 32s
Package Supply Chain / pkg-supply-chain (push) Successful in 2m3s
bcda.ui mirrors the surface of the CMS server (CMSgov/bcda-app, vendored under dev/vendor): /_version, /_health, metadata, /auth/token, attribution_status, $export on Group/all · Group/runout · Patient with _type / _since / _typeFilter, job history, status, cancel and /data download — plus the local side: the Store's job and file history, flatten + load into bcda.*, the connector into input_layer.*, the replica for browsing, the v3 data dictionary and CCLF crosswalk, and the vendored-upstream manifest with an on-demand drift check. bcda.ui.service has no Streamlit in it: every operation takes a Connection / Client / Store, so `stack bcda preflight|auth|jobs|export| adopt|cancel|store|load|vendor` reuse it verbatim and tests hand in httpx.MockTransport clients and temp stores. Connections are the four public sandbox entities, BCDA_CLIENT_ID/SECRET from the environment, or custom (sandbox or production, v3 or v2). bcda.ui.runner runs exports, adoption, loads and drift checks on daemon threads in a process-level registry that survives Streamlit reruns, captures the bcda.* log lines of each run's own thread, and flips state last so readers never see a finished run with an incomplete log; a st.fragment panel shows runs live. Exports default to one job per type with cancel retries (the only shape that survives the shared sandbox's sweeps); the _typeFilter builder composes System-Type, outcome and service-date; completed jobs under 24 h are offered for adopt, in-progress ones for cancel. Data pages read the read-only replica only. `stack bcda ui` launches streamlit on src/bcda/ui/app.py; streamlit is added to the bcda extra. CLI reference pages regenerated (the generator renumbers sidebar positions, hence the touched pages). Tests: service, runner, every page through AppTest (buttons, forms, error paths), the app shell, and the CLI — new modules at 100% line coverage. refs #1139
3.7 KiB
3.7 KiB
title, sidebar_position
| title | sidebar_position |
|---|---|
| stack bib backfill-comments | 30 |
stack bib backfill-comments
Usage: stack bib backfill-comments [OPTIONS]
Enrich every reg-gov comment stub with body + attachments.
Walks items tagged ``source:regulations-gov`` whose ``abstract`` is
still empty and hits ``/v4/comments/{id}?include=attachments`` for
each one. Resumable: stop any time, start again, it picks up where
it left off by skipping already-enriched items. At 1.3s/req the API
calls run at ~2700/hr; attachment downloads are concurrent on the
CDN and don't count against the rate limit.
``--mirror`` swaps the API for the Mirrulations public S3 mirror
(#662): no API key, no cap, and mirror-only comments get created.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --docket TEXT Limit to one docket (e.g. CMS-2026-2377). Without │
│ it the walk covers every un-enriched stub in the │
│ store — a multi-day crawl at the reg.gov rate │
│ cap. │
│ --limit -n INTEGER Cap items processed this run. 0 = no cap │
│ (multi-day crawl). │
│ [default: 0] │
│ --sleep FLOAT Seconds between API calls. 3.7s ≈ 970/hr — just │
│ under the 1000/hr reg.gov hourly cap (the binding │
│ limit; the 50/min burst cap is looser). CDN │
│ downloads are concurrent and don't count. │
│ [default: 3.7] │
│ --log PATH Append-only progress log (survives restarts). │
│ [default: /tmp/bib-backfill-comments.log] │
│ --mirror Pull from the Mirrulations S3 mirror instead of │
│ the reg.gov API — bulk speed, no rate cap, also │
│ ingests comments the original farm never listed. │
│ Requires --docket. │
│ --force Run even if the docket is sealed. Bypasses the │
│ docket seal only: items already tagged │
│ enriched:ok (or enriched:gone) are still skipped, │
│ so this resumes an interrupted enrichment — it │
│ never re-enriches an item. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯