homelab/stack
1
0
Fork 0
Code Issues 59 Pull Requests 1 Actions Packages Projects 1 Releases Wiki Activity

chore(comments): re-farm script is the plain chain now that every stage is incremental (refs #615)

Browse Source
This commit is contained in:
kert
2026-09-08 14:37:13 -04:00
parent bac63363c6
commit d7fd23f83a
1 changed files with 6 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
dev/scripts/refarm_cms_2026_2377.sh
View File

@@ -22,24 +22,15 @@ step() { # step <label> <cmd...>: run, count failures, keep going
"$@" || { echo "WARN: $label rc=$?"; FAILURES=$((FAILURES+1)); } "$@" || { echo "WARN: $label rc=$?"; FAILURES=$((FAILURES+1)); }
} }
# Fast path first: the Mirrulations S3 mirror (#662) is a superset of the # Every step is incremental: sealed dockets are skipped outright, open
# live docket and has no rate cap (~1,500 comments/min vs the API's ~970/hr). # dockets pull from the stored watermark, extract/index compare cheap
# It creates comments the farm never listed, enriches, attaches and tags # fingerprints before doing any work. Mirror first (bulk, no cap), then
# them identically (reg-docket:/rule:/enriched:ok), so the index is usable # the API for anything the mirror lags on.
# minutes in rather than the next day.
step "mirror backfill" uv run stack bib backfill-comments --docket CMS-2026-2377 --mirror step "mirror backfill" uv run stack bib backfill-comments --docket CMS-2026-2377 --mirror
step "extract" uv run stack comments extract --docket CMS-2026-2377
# extract-ocr is a phase-2 stub (always exits 2, no --docket) — see #253.
# ocr_needed attachments stay flagged in combined.md frontmatter for now.
step "index" uv run stack llm index --collection comments --docket CMS-2026-2377
# Gap-fill from the reg.gov API for anything the mirror lags on. All
# resumable: the list walk upserts, the backfill skips enriched:ok, and
# extract/index only touch new or changed items.
step "api fetch" uv run stack bib fetch-pfs-comments --docket CMS-2026-2377 step "api fetch" uv run stack bib fetch-pfs-comments --docket CMS-2026-2377
step "api backfill" uv run stack bib backfill-comments --docket CMS-2026-2377 step "api backfill" uv run stack bib backfill-comments --docket CMS-2026-2377
step "extract (gap)" uv run stack comments extract --docket CMS-2026-2377 step "extract" uv run stack comments extract --docket CMS-2026-2377
step "index (gap)" uv run stack llm index --collection comments --docket CMS-2026-2377 step "index" uv run stack llm index --collection comments --docket CMS-2026-2377
COUNTS=$(uv run python - <<'EOF' COUNTS=$(uv run python - <<'EOF'
from conf import connect from conf import connect