stack/docs/superpowers/specs/2026-09-08-comment-pipeline-seal-and-skip-design.md

comment pipeline — docket seals and cheap skips: never redo what is known complete

Tracker: new milestone (P47) with one issue per component; refs #615 (re-farm), #662 (mirror backfill), #253 (OCR stub, out of scope).

Goal

A default run of any stage of the regulations.gov comment pipeline — bib fetch-pfs-comments, bib backfill-comments, comments extract, llm index — must do no work for a docket that is known complete, and must decide per item whether there is work before touching the network, the filesystem, or a PDF. --force is the only way to redo known-complete work, and it applies to one run.

Success criteria, measured on the live data (2026-09-08: 12 dockets, 201,874 comments, 28,493 PDFs / 13.3 GB, 224,351 bib items):

• stack llm index --collection all with nothing changed finishes in under a minute, opens zero PDFs, and does not copy zotero.sqlite.
• stack bib fetch-pfs-comments with every docket sealed makes at most 4 API calls (the Federal Register rule search) and writes zero rows.
• stack comments extract with nothing changed reads zero combined.md files.
• Re-running any stage twice produces byte-identical bib.sqlite content (no updated_at / access_date churn, no tag rewrites, no duplicate attachment rows).

Current state (measured 2026-09-08)

What a default run redoes today, per stage. File refs are to the tree at 832bfd5.

Stage	Skips	Redoes	Cost
fetch (cli/bib.py:189-298, bib/regulations_gov.py:198-260)	nothing	rediscovers every rule, one FR GET per rule, one rate-limited reg.gov call per CMS id (even with --docket, which filters after the resolve call), re-walks every docket from page 1 (the lastModifiedDate cursor is intra-call only), re-upserts every comment	~170 API calls per large docket; Store.upsert (bib/store.py:207-245) stamps access_date, bumps updated_at, deletes and rewrites every tag row (store.py:429-453); attach_file (store.py:502-533) mints a duplicate row and copies the file again — 33,379 duplicated (item_id, filename) groups covering 66,917 of 78,017 rows in the live store
backfill API (regulations_gov.py:378-544)	enriched:ok / enriched:gone	nothing significant	one anti-join scan — already correct
backfill mirror (regulations_gov.py:657-776)	enriched:ok	lists the whole S3 prefix for comments and attachments before filtering	40–80 LIST requests per docket
extract (rex/comments/walker.py:79-118, combine.py:42-132)	dirs with a combined.md (stat only)	reads + YAML-parses every skipped combined.md for sibling derivation and fires the attach callback; loads every comment abstract from bib at startup (cli/comments.py:52-63); never detects a stale extraction	28,228 files / 757 MB parsed per run
index comments (llm/index.py:90-143, llm/source.py:104-132, llm/pages.py:61-83)	sha256(text) match in index_state	chunks and opens every PDF page (enrich_pdf_pages) before the hash check; one _year_of query per comment; reads every combined.md; store.get per comment without one	full PyMuPDF decode of 13.3 GB per no-op run
index corpus (source.py:265-357, cli/llm.py:24-27)	same	hydrates all 224,351 items (3 queries each) to discard 201,874 comments; re-extracts every attachment; copies the 1.95 GB zotero.sqlite unconditionally	~670k queries + 1.95 GB copy per run

The one signal that would let a stage skip a whole docket — the reg.gov commentEndDate — is fetched every run (cli/bib.py:170, :272) and used only as a filter predicate. No docket-level state exists anywhere.

Decisions

1. Two-level skip. A docket-level seal removes whole dockets from every walk; a per-item fingerprint (cheap facts only) is checked before any load. Content hashes stay as the third line.
2. Seal = auto after close + quiet period. A docket seals when its comment close date is ≥ seal_quiet_days (default 30) in the past and the most recent completed pull after that date found zero new comments. Manual seal / unseal exist. Sealing happens at the end of a fetch walk because fetch is the only stage that learns about new comments.
3. Docket state lives in a dockets table in bib.sqlite, not bib_meta JSON, not tags, not .state/ (cache-only by convention). Index-side docket completion lives in the llm Postgres DB next to index_state.
4. Fix every default-path redo for open dockets too (fetch watermark, no-op upsert, idempotent attachments, extract skip cost, index ordering, bulk queries, lazy Zotero snapshot).
5. Attachment duplicates are cleaned up now — guard plus a one-time migration — because the guard alone leaves 66k dead rows that every later stage pays for.
6. --force semantics unchanged and uniform: bypasses seal, fingerprint, hash, and docket-completion for that run; never unseals.

Architecture

                    dockets (bib.sqlite)                     index_state(+fingerprint), index_docket_state (llm pg)
                    id, close date, watermark, sealed_at            (item_key, collection) → fingerprint, content_hash
                          │                                                       │
   fetch ──► sealed? ──skip──┐   backfill ──► sealed? ──skip──┐   extract ──► sealed? ──skip──┐   index ──► sealed & complete? ──skip──┐
     │ no                   │      │ no                       │      │ no                    │      │ no                              │
     ▼                      │      ▼                          │      ▼                       │      ▼                                 │
   walk from watermark      │   pending set from tags         │   fingerprint per dir        │   fingerprint per DocRef                │
   upsert → unchanged? no-op│   (mirror: list S3, open only)  │   differs? extract           │   differs? load → hash differs? embed   │
   walk clean → advance     │                                 │                              │   clean over sealed docket → completion │
   watermark, maybe seal ───┘                                 └──────────────────────────────┴──────────────────────────────────────────┘

Components

bib/schema.sql + bib/store.py — dockets table and store API

CREATE TABLE IF NOT EXISTS dockets (
  id               TEXT PRIMARY KEY,   -- reg.gov docket id, e.g. CMS-2026-2377
  rule_cms_id      TEXT,               -- CMS-1848-P
  fr_document_id   TEXT,               -- reg.gov document id carrying the comments
  fr_object_id     TEXT,               -- attributes.objectId (what /comments filters on)
  comment_end_date TEXT,               -- YYYY-MM-DD from document attributes
  pull_watermark   TEXT,               -- max lastModifiedDate seen on a completed walk
  last_pull_at     TEXT,
  last_pull_new    INTEGER,            -- comments created by that walk
  sealed_at        TEXT,
  seal_reason      TEXT,               -- 'auto' | 'manual'
  counts_json      TEXT                -- {"comments":n,"enriched":n,"extracted":n} at seal
);
CREATE UNIQUE INDEX IF NOT EXISTS attachments_item_filename ON attachments(item_id, filename);

The unique index is created by Store.connect only when no duplicates remain (see migration); fresh databases get it from the schema.

Store methods (all thin SQL, unit-tested against an in-memory store):

• docket_get(id) -> Docket | None, docket_upsert(Docket), dockets() -> list[Docket], sealed_docket_ids() -> set[str].
• docket_seal(id, reason, counts), docket_unseal(id).
• docket_of_item(key) -> str | None reads the reg-docket: tag.
• upsert(item) -> key keeps its signature; new upsert_status(item) -> tuple[key, Literal["created","updated","unchanged"]] compares the normalized row (all columns except access_date, updated_at, created_at) and the tag/collection sets, and returns unchanged without writing anything. stamp_access runs only on create/update.
• attach_file(key, path, title) returns the existing attachment key when (item_id, filename) already exists; no copy, no insert.

Docket is a frozen dataclass in new bib/dockets.py, together with the pure should_seal(docket, today, quiet_days) -> bool.

bib/dockets.py (new, pure)

• should_seal: comment_end_date present, today >= end + quiet_days, last_pull_at >= end + quiet_days, last_pull_new == 0, not already sealed.
• fingerprint_files(paths) -> str: sha256 over sorted (name, size, mtime_ns) — shared by extract and index.

bib/regulations_gov.py — watermark walk, sealed skips

• iter_comments(object_id, since: str | None = None): initial cursor = since. Boundary rows re-yield (>= filter); the no-op upsert absorbs them.
• walk_docket(store, api, docket: Docket, *, attachments, limit) -> WalkResult(new, updated, unchanged, watermark, clean) extracted from the CLI loop so both fetch commands share it. Advances pull_watermark / last_pull_at / last_pull_new only when clean (no HTTPStatusError break, no exception). Calls should_seal and docket_seal(..., reason="auto") when true.
• backfill_comments(..., docket=...) and backfill_from_mirror(...): return early with a log line when the docket is sealed and force is false. Mirror listing is unchanged for open dockets.

cli/bib.py — fetch commands

fetch-pfs-comments per proposed rule: split CMS ids → for each, store.docket_get. Sealed → echo " <docket>: sealed <date>, skipping" and continue before the federal_register() rule fetch and the resolve_docket call. Unknown → resolve_docket + find_documents_in_docket once, insert the Docket row with comment_end_date, fr_document_id, fr_object_id. Known and open → walk_docket from the watermark using the stored object id (no resolve, no document listing). --docket filters before any API call (the current help text becomes true). fetch-docket-comments takes the same path for one docket. Both accept --force (walk from page 1, ignore seal, never unseal).

dev/scripts/dedupe_attachments.py (one-time migration)

Groups attachments by (item_id, filename); keeps the row with the smallest rowid; deletes the others' rows and their files under the attachment store when no other row references the same path. Prints a report (groups, rows removed, bytes freed, and the list of removed keys that carry a Zotero mapping, for a separate Zotero-side pass — Zotero itself is not touched). --dry-run default; --apply executes inside one transaction. After a clean apply, Store.connect creates the unique index on next open.

rex/comments/walker.py + combine.py + cli/comments.py — extract

• walk_and_extract(root, ..., sealed: set[str], force, reattach): docket dirs in sealed are skipped at the loop head (walker.py:95) unless force.
• Per dir, "up to date" = combined.md exists and its frontmatter sources fingerprint equals fingerprint_files over the dir's current attachment/body files. extract_comment writes sources into the frontmatter.
• Skipped dirs no longer go through derive_siblings_from_combined or on_extracted; --reattach restores the old behaviour for repair runs.
• _build_bib_helpers loads abstracts in batches keyed by the todo list instead of the whole table.
• New commands: stack comments dockets (table: id, rule, close date, comments/enriched/extracted, watermark, sealed), stack comments seal <id> [--reason], stack comments unseal <id>, stack comments dockets --discover (one-time populate of the 12 known dockets from reg-docket: tags, resolving close dates with one API call each).

llm/source.py — lazy DocRef

@dataclass(frozen=True)
class DocRef:
    key: str
    collection: str
    docket: str | None
    fingerprint: str
    load: Callable[[], Doc]   # builds the Doc (reads combined.md / attachments / fr_anchors)

• iter_comment_refs(store, root, docket=None, sealed_complete: set[str]): one SQL query per docket for (key, comment_id, year, updated_at, has_combined); skips dockets in sealed_complete; fingerprint = fingerprint_files(combined.md + attachment files) + updated_at. load does today's iter_comment_docs body for one item.
• iter_corpus_refs: keys selected in SQL (tag = corpus tag AND NOT source:regulations-gov), no hydration; fingerprint = updated_at + attachment file stats. load hydrates and extracts attachments.
• iter_rule_refs: fingerprint = fr_anchor_docs.sha256.
• ZoteroPdfIndex becomes lazy: constructed with the source path; snapshot() runs on first lookup() miss of a local PDF, and only copies when the source mtime is newer than the existing .state/llm/zotero.sqlite.

llm/migrate.py + llm/index.py — fingerprint-first loop

• Migration: ALTER TABLE index_state ADD COLUMN IF NOT EXISTS fingerprint TEXT; CREATE TABLE IF NOT EXISTS index_docket_state (collection TEXT, docket TEXT, sealed_at TIMESTAMPTZ, indexed_at TIMESTAMPTZ, PRIMARY KEY (collection, docket)).
• index_refs(refs, collection, *, force, sealed: dict[str, sealed_at]), per ref:
  1. not force and state[key].fingerprint == ref.fingerprint → skip (no load).
  2. doc = ref.load(); h = content_hash(doc.text); not force and state[key].content_hash == h → update fingerprint only, skip.
  3. chunks = enrich_pdf_pages(doc, chunk_doc(doc)) → embed → write chunks + state row (fingerprint, hash, chunk_count).
• After a run over a docket whose seal is set and which ended with zero errors, upsert index_docket_state(collection, docket, sealed_at). On the next run iter_*_refs receives sealed_complete = {docket for rows where sealed_at == dockets.sealed_at}; an unseal/reseal changes sealed_at and invalidates the row.
• stats output adds fingerprint_skipped, hash_skipped, docket_skipped.

cli/llm.py

index builds the ref iterators, passes store.sealed_docket_ids() joined with index_docket_state, honours --docket for comments, --key for rules, and --force. The Zotero snapshot line moves into the lazy index. --collection all runs the three in sequence as today.

Config

stack.toml: [comments] seal_quiet_days = 30. No new env.

dev/scripts/refarm_cms_2026_2377.sh

Collapses to the plain chain (mirror backfill → extract → index → API fetch → API backfill → extract → index) with no special ordering justification: every step is now cheap when there is nothing to do. The 2026-09-15 crontab line stays.

Data flow for one incremental run (open docket)

1. fetch-pfs-comments: FR search (≤4 calls) → 11 dockets sealed, skipped with no calls → CMS-2026-2377 open: walk from watermark 2026-09-08T12:00:00 → 40 new, 3 updated, 12 unchanged boundary rows → watermark advanced, last_pull_new=40, should_seal false (close date 2026-09-14 not past).
2. backfill --mirror --docket CMS-2026-2377: not sealed → list S3 → 40 pending → enrich, attach (idempotent).
3. comments extract --docket CMS-2026-2377: 40 dirs lack combined.md, 2 have a changed fingerprint → 42 extracted; the rest are skipped without a read.
4. llm index --collection comments: 11 dockets in index_docket_state → not iterated; CMS-2026-2377: 18,691 refs, 42 fingerprints differ → 42 loaded, hashed, PDF-paged, embedded; 18,649 skipped without a load.

Error handling

• A walk that breaks on an HTTP error leaves the watermark untouched; the next run resumes from the old watermark (re-yields are absorbed by the no-op upsert).
• Sealing never happens on a run that had errors (clean=False).
• A fingerprint mismatch caused by a touch (mtime only) costs one load and hash, then settles.
• index_docket_state is written only after a zero-error run; any embed failure leaves the docket iterable.
• The dedupe migration is dry-run by default and transactional on --apply; it refuses to run if any duplicate group's files differ in size (reports them instead).
• A sealed docket named by --docket without --force prints a one-line notice and exits 0.

Testing

Pure units first (TDD): should_seal (boundary days, missing close date, unclean pull), fingerprint_files (order-independent, mtime-sensitive), upsert_status (created/updated/unchanged, tag-set change → updated, access stamp only on change), attach_file idempotency, iter_comments(since=) passes the filter and re-yields the boundary row, walk_docket watermark advance only when clean, index_refs ordering (a ref whose fingerprint matches never has load called; hash match updates fingerprint without embedding; PDFs opened only on the embed path — inverts tests/llm/test_index.py::test_enriches_chunks_before_add).

CLI: extend tests/cli/test_bib_exercise.py (TestFetchPfsComments: sealed docket makes no reg.gov call; --docket filters before resolve), tests/bib/test_regulations_gov_mirror.py::test_idempotent_rerun (sealed → no S3 listing), tests/rex/comments/test_walker.py (skipped dirs not read, --reattach), tests/cli/test_comments.py (dockets, seal, unseal), tests/cli/test_llm_exercise.py (corpus run without Zotero copy when no fallback needed), tests/llm/test_migrate.py (new column/table idempotent).

Migration: fixture DB with duplicate groups → dry-run report → apply → unique index creatable → second apply is a no-op.

Acceptance on live data (recorded in the tracker): the four success criteria above, timed.

Rollout

1. Deploy code; Store.connect creates dockets; llm migration adds the column and table on container start.
2. stack comments dockets --discover populates 12 rows (12 API calls, once).
3. dev/scripts/dedupe_attachments.py --dry-run, review, --apply; reopen the store → unique index created.
4. stack comments seal <id> --reason manual for the 11 historical dockets (CMS-2017-0092 … CMS-2025-0304); CMS-2026-2377 stays open and auto-seals after 2026-10-14 once a pull finds nothing new.
5. One last full walk: stack llm index --collection all stamps fingerprints for all 196k items (stat-only, no PDF decode, no embedding) and writes index_docket_state for the 11 sealed dockets. Then a second run must meet the sub-minute criterion.
6. stack comments extract once to stamp sources fingerprints into existing combined.md frontmatter (read-only pass for unchanged dirs; rewrites frontmatter only).
7. Re-farm script simplified; crontab unchanged.

Out of scope

• OCR for image-only attachments (#253); extract-ocr stays a stub.
• Zotero-side removal of already-synced duplicate attachments (listed by the migration report; separate pass like #665).
• Reducing mirror S3 listing cost for open dockets.
• Chunking, embedding model, or retrieval changes.