stack/docs/superpowers/plans/2026-08-19-mirrulations-backfill-p43.md

Mirrulations Mirror Backfill Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Replace the 7-day regulations.gov API crawl with a bulk pull from the Mirrulations public S3 mirror (AWS Open Data) — hours instead of days, zero API-key pressure (refs #662).

Architecture: s3://mirrulations (public, HTTPS-listable, no auth) mirrors every docket: raw-data/CMS/{D}/text-{D}/comments/{cid}.json is byte-compatible with the v4 /comments/{id} detail response ({"data": {...}}, no included block), and raw-data/CMS/{D}/binary-{D}/comments_attachments/{cid}_attachment_N.{ext} holds the attachment binaries. The mirror is a superset of our farm (15,368 vs 13,245 for CMS-2018-0076; 3,433 vs 2,728 for the live docket). A new backfill_from_mirror in bib/regulations_gov.py reuses _parse_comment + upsert_comment (creation of mirror-only comments) and the enrichment steps of backfill_details (abstract, org tag, attach_file, enriched:ok), with S3 I/O fanned out over a thread pool and ALL sqlite work on the main thread (same pattern as prisma/fetch.py). VMs are unnecessary: the constraint was the per-key API cap, which the mirror sidesteps entirely; the reg.gov API remains only for freshness gap-fill (#615).

Tech Stack: httpx (S3 REST list + GET), concurrent.futures.ThreadPoolExecutor, existing bib.store.Store / Source / Tag machinery; pytest + httpx.MockTransport.

Spec: Issue #662 plus this plan.

Global Constraints

• No git stash; explicit-path staging; git status --short pre-commit; ruff format+check first; commits in background (hook runs mapped tests).
• No Claude co-author trailer.
• All Store/sqlite calls on the main thread; worker threads do network + file I/O only.
• Kill the running API backfill loop BEFORE launching the mirror run (two enrichers racing the same docket would double-attach).
• Local attachment naming must match the existing layout: .state/comments/{D}/{cid}/attachment_N.{ext} (strip the {cid}_ prefix from mirror basenames).
• Mirror-only comments get the same tag set as farmed ones: source:regulations-gov, doctype:comment, reg-docket:{D}, rule:{cms_id} (derived from an existing item of the docket), year:{posted[:4]}, plus org: when present.

---

Task 1: Mirror client + backfill_from_mirror

Files:

• Modify: src/bib/regulations_gov.py
• Modify: src/cli/bib.py (add --mirror to backfill-comments)
• Test: tests/bib/test_regulations_gov_mirror.py (new)

Interfaces:

• Consumes: _parse_comment(row) -> Comment, upsert_comment(store, comment, cms_id=, extra_tags=), Store.add_tag/attach_file/update, _slug, _docket_from_comment_id.

• Produces:

  • class Mirror — list_keys(prefix) -> list[str] (paged, continuation-token), comment_ids(docket) -> list[str], attachment_keys(docket) -> dict[str, list[str]], get_json(key) -> dict, download(key, dest) -> Path | None. Base https://mirrulations.s3.amazonaws.com; constructor takes optional httpx.Client for testing.
  • backfill_from_mirror(store, mirror, *, docket, limit=None, log_path=None, commit_every=200, scratch_root=Path(".state/comments"), workers=12) -> dict[str, int] with stats keys {"created", "enriched", "attached", "errors", "seen"}.
  • CLI: stack bib backfill-comments --docket D --mirror routes to the mirror path (docket required with --mirror).

• Step 1: Write the failing tests (tests/bib/test_regulations_gov_mirror.py) — a MockTransport that serves: a paged list response (two pages via NextContinuationToken), comment JSONs for one existing-stub cid and one mirror-only cid, one attachment binary (%PDF...). Assert: paging concatenates keys; backfill_from_mirror on a tmp Store (seed one stub item with source:regulations-gov + reg-docket: tags and the comment URL) → stub gets abstract + enriched:ok, mirror-only cid gets created with the full tag set, attachment lands at scratch/{D}/{cid}/attachment_1.pdf and is registered via attach_file, stats match.

• Step 2: Run to verify failure — uv run pytest tests/bib/test_regulations_gov_mirror.py -q → ImportError.

• Step 3: Implement Mirror + backfill_from_mirror per the interfaces; thread pool fetches (cid) -> (json_dict, [downloaded attachment paths]) into scratch; main thread applies DB effects per completed future; rule: cms_id looked up once per docket from any existing item's rule: tag (empty string when the docket has none). Wire the CLI flag.

• Step 4: Tests green — the new file plus uv run pytest tests/bib -q (background; large).

• Step 5: Live smoke — uv run stack bib backfill-comments --docket CMS-2017-0092 --mirror --limit 25 (smallest docket; verify enriched counts + attachment files).

• Step 6: Ruff, status check, commit — feat(bib): backfill-comments --mirror — bulk enrichment from the Mirrulations S3 mirror (refs #662).

Task 2: Swap the running crawl over to the mirror

• Step 1: Kill the API loop (pkill -f backfill-comments — anchored; also the wrapping bash pid) after noting its progress.
• Step 2: Launch the mirror loop (same docket order + extract + sweep chain, .env sourced), all ten dockets including CMS-2017-0092 and CMS-2026-2377 (mirror is fresher than our farm even there).
• Step 3: Verify first-docket throughput; estimate completion (expect hours).
• Step 4: Comment on #662 (mirror route live, API loop retired to gap-fill duty); leave open until the run + extraction + sweep complete.

Task 3: Close out

• Update memory (palliative_rfi_project.md + the answer to "many VMs?": per-key limit, mirror route).