homelab/stack
1
0
Fork 0
Code Issues 20 Pull Requests Actions Packages Projects 1 Releases Wiki Activity

chore(comments): re-farm CMS-2026-2377 mirror-first, API steps as gap-fill (refs #615 #662)

Browse Source 
The API backfill enriches ~970 comments/hr, so 15k new comments meant a
next-day index. The Mirrulations mirror is a superset of the live docket
(18,651 vs 18,647 comments on 2026-09-08) and runs ~600/min, creating,
tagging and enriching items identically. Run mirror → extract → index
first, then the API list walk + backfill + incremental extract/index to
catch anything the mirror lags on.
This commit is contained in:
kert
2026-09-08 08:40:59 -04:00
parent 86b44d9ae8
commit 832bfd511f
1 changed files with 22 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
dev/scripts/refarm_cms_2026_2377.sh
View File

@@ -16,13 +16,30 @@ echo "=== re-farm run $(date -Is) ==="
FAILURES=0
uv run stack bib fetch-pfs-comments --docket CMS-2026-2377 || { echo "WARN: fetch-pfs-comments rc=$?"; FAILURES=$((FAILURES+1)); }
uv run stack bib backfill-comments --docket CMS-2026-2377 || { echo "WARN: backfill rc=$?"; FAILURES=$((FAILURES+1)); }
uv run stack comments extract --docket CMS-2026-2377 || { echo "WARN: extract rc=$?"; FAILURES=$((FAILURES+1)); }
step() { # step <label> <cmd...>: run, count failures, keep going
local label=$1; shift
echo "--- $label: $(date -Is)"
"$@" || { echo "WARN: $label rc=$?"; FAILURES=$((FAILURES+1)); }
}
# Fast path first: the Mirrulations S3 mirror (#662) is a superset of the
# live docket and has no rate cap (~1,500 comments/min vs the API's ~970/hr).
# It creates comments the farm never listed, enriches, attaches and tags
# them identically (reg-docket:/rule:/enriched:ok), so the index is usable
# minutes in rather than the next day.
step "mirror backfill" uv run stack bib backfill-comments --docket CMS-2026-2377 --mirror
step "extract" uv run stack comments extract --docket CMS-2026-2377
# extract-ocr is a phase-2 stub (always exits 2, no --docket) — see #253.
# ocr_needed attachments stay flagged in combined.md frontmatter for now.
uv run stack llm index --collection comments --docket CMS-2026-2377 \
|| { echo "WARN: index rc=$?"; FAILURES=$((FAILURES+1)); }
step "index" uv run stack llm index --collection comments --docket CMS-2026-2377
# Gap-fill from the reg.gov API for anything the mirror lags on. All
# resumable: the list walk upserts, the backfill skips enriched:ok, and
# extract/index only touch new or changed items.
step "api fetch" uv run stack bib fetch-pfs-comments --docket CMS-2026-2377
step "api backfill" uv run stack bib backfill-comments --docket CMS-2026-2377
step "extract (gap)" uv run stack comments extract --docket CMS-2026-2377
step "index (gap)" uv run stack llm index --collection comments --docket CMS-2026-2377
COUNTS=$(uv run python - <<'EOF'
from conf import connect