55 lines
5.8 KiB
Markdown
55 lines
5.8 KiB
Markdown
# Mirrulations Mirror Backfill Implementation Plan
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
|
|
**Goal:** Replace the 7-day regulations.gov API crawl with a bulk pull from the Mirrulations public S3 mirror (AWS Open Data) — hours instead of days, zero API-key pressure (refs #662).
|
|
|
|
**Architecture:** `s3://mirrulations` (public, HTTPS-listable, no auth) mirrors every docket: `raw-data/CMS/{D}/text-{D}/comments/{cid}.json` is byte-compatible with the v4 `/comments/{id}` detail response (`{"data": {...}}`, no `included` block), and `raw-data/CMS/{D}/binary-{D}/comments_attachments/{cid}_attachment_N.{ext}` holds the attachment binaries. The mirror is a superset of our farm (15,368 vs 13,245 for CMS-2018-0076; 3,433 vs 2,728 for the live docket). A new `backfill_from_mirror` in `bib/regulations_gov.py` reuses `_parse_comment` + `upsert_comment` (creation of mirror-only comments) and the enrichment steps of `backfill_details` (abstract, org tag, attach_file, `enriched:ok`), with S3 I/O fanned out over a thread pool and ALL sqlite work on the main thread (same pattern as `prisma/fetch.py`). VMs are unnecessary: the constraint was the per-key API cap, which the mirror sidesteps entirely; the reg.gov API remains only for freshness gap-fill (#615).
|
|
|
|
**Tech Stack:** httpx (S3 REST list + GET), `concurrent.futures.ThreadPoolExecutor`, existing `bib.store.Store` / `Source` / `Tag` machinery; pytest + `httpx.MockTransport`.
|
|
|
|
**Spec:** Issue #662 plus this plan.
|
|
|
|
## Global Constraints
|
|
|
|
- No git stash; explicit-path staging; `git status --short` pre-commit; ruff format+check first; commits in background (hook runs mapped tests).
|
|
- No Claude co-author trailer.
|
|
- All `Store`/sqlite calls on the main thread; worker threads do network + file I/O only.
|
|
- Kill the running API backfill loop BEFORE launching the mirror run (two enrichers racing the same docket would double-attach).
|
|
- Local attachment naming must match the existing layout: `.state/comments/{D}/{cid}/attachment_N.{ext}` (strip the `{cid}_` prefix from mirror basenames).
|
|
- Mirror-only comments get the same tag set as farmed ones: `source:regulations-gov`, `doctype:comment`, `reg-docket:{D}`, `rule:{cms_id}` (derived from an existing item of the docket), `year:{posted[:4]}`, plus `org:` when present.
|
|
|
|
---
|
|
|
|
### Task 1: `Mirror` client + `backfill_from_mirror`
|
|
|
|
**Files:**
|
|
- Modify: `src/bib/regulations_gov.py`
|
|
- Modify: `src/cli/bib.py` (add `--mirror` to `backfill-comments`)
|
|
- Test: `tests/bib/test_regulations_gov_mirror.py` (new)
|
|
|
|
**Interfaces:**
|
|
- Consumes: `_parse_comment(row) -> Comment`, `upsert_comment(store, comment, cms_id=, extra_tags=)`, `Store.add_tag/attach_file/update`, `_slug`, `_docket_from_comment_id`.
|
|
- Produces:
|
|
- `class Mirror` — `list_keys(prefix) -> list[str]` (paged, continuation-token), `comment_ids(docket) -> list[str]`, `attachment_keys(docket) -> dict[str, list[str]]`, `get_json(key) -> dict`, `download(key, dest) -> Path | None`. Base `https://mirrulations.s3.amazonaws.com`; constructor takes optional `httpx.Client` for testing.
|
|
- `backfill_from_mirror(store, mirror, *, docket, limit=None, log_path=None, commit_every=200, scratch_root=Path(".state/comments"), workers=12) -> dict[str, int]` with stats keys `{"created", "enriched", "attached", "errors", "seen"}`.
|
|
- CLI: `stack bib backfill-comments --docket D --mirror` routes to the mirror path (docket required with --mirror).
|
|
|
|
- [ ] **Step 1: Write the failing tests** (`tests/bib/test_regulations_gov_mirror.py`) — a MockTransport that serves: a paged list response (two pages via `NextContinuationToken`), comment JSONs for one existing-stub cid and one mirror-only cid, one attachment binary (`%PDF...`). Assert: paging concatenates keys; `backfill_from_mirror` on a tmp `Store` (seed one stub item with `source:regulations-gov` + `reg-docket:` tags and the comment URL) → stub gets abstract + `enriched:ok`, mirror-only cid gets created with the full tag set, attachment lands at `scratch/{D}/{cid}/attachment_1.pdf` and is registered via `attach_file`, stats match.
|
|
- [ ] **Step 2: Run to verify failure** — `uv run pytest tests/bib/test_regulations_gov_mirror.py -q` → ImportError.
|
|
- [ ] **Step 3: Implement** `Mirror` + `backfill_from_mirror` per the interfaces; thread pool fetches `(cid) -> (json_dict, [downloaded attachment paths])` into scratch; main thread applies DB effects per completed future; `rule:` cms_id looked up once per docket from any existing item's `rule:` tag (empty string when the docket has none). Wire the CLI flag.
|
|
- [ ] **Step 4: Tests green** — the new file plus `uv run pytest tests/bib -q` (background; large).
|
|
- [ ] **Step 5: Live smoke** — `uv run stack bib backfill-comments --docket CMS-2017-0092 --mirror --limit 25` (smallest docket; verify enriched counts + attachment files).
|
|
- [ ] **Step 6: Ruff, status check, commit** — `feat(bib): backfill-comments --mirror — bulk enrichment from the Mirrulations S3 mirror (refs #662)`.
|
|
|
|
### Task 2: Swap the running crawl over to the mirror
|
|
|
|
- [ ] **Step 1:** Kill the API loop (`pkill -f backfill-comments` — anchored; also the wrapping bash pid) after noting its progress.
|
|
- [ ] **Step 2:** Launch the mirror loop (same docket order + extract + sweep chain, `.env` sourced), all ten dockets including CMS-2017-0092 and CMS-2026-2377 (mirror is fresher than our farm even there).
|
|
- [ ] **Step 3:** Verify first-docket throughput; estimate completion (expect hours).
|
|
- [ ] **Step 4:** Comment on #662 (mirror route live, API loop retired to gap-fill duty); leave open until the run + extraction + sweep complete.
|
|
|
|
### Task 3: Close out
|
|
|
|
- [ ] Update memory (`palliative_rfi_project.md` + the answer to "many VMs?": per-key limit, mirror route).
|