Files

kert 23370e604c docs(plan): Task 3b data-preserving upsert + abstract recovery step after the 2026-09-08 clobber (refs #615 )

2026-09-08 12:18:53 -04:00

137 KiB

Raw Blame History

Comment Pipeline Seal-and-Skip Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: A default run of fetch, backfill, extract, or index does zero work for sealed dockets and decides per item whether there is work before touching the network, the filesystem, or a PDF.

Architecture: A dockets table in bib.sqlite carries each reg.gov docket's close date, pull watermark, and seal. Every stage consults it first. Per item, a cheap fingerprint (file stats + updated_at) is compared before any load; the sha256 content hash stays as the second line. --force bypasses everything for one run.

Tech Stack: Python 3.13, uv run, pytest (uv run --no-sync pytest), SQLite via bib.store.Store, Postgres/pgvector via SQLAlchemy in llm/, typer CLIs under src/cli/, PyMuPDF (fitz) in tests that need PDFs.

Spec: docs/superpowers/specs/2026-09-08-comment-pipeline-seal-and-skip-design.md

Global Constraints

Run tests with uv run --no-sync pytest <path> -q -p no:cacheprovider. The pre-commit hook runs ruff + the cli test suite; keep it green.
Commit messages: conventional prefix (feat(bib):, fix(llm):, chore(comments):), (refs #615) where relevant. No Co-Authored-By trailer (house rule).
Before every commit run git status --short and stage only your own files (other sessions share this worktree).
--force semantics are uniform: bypass seal, fingerprint, hash, and docket-completion for that run; never unseal.
Nothing under .state/ is a source of truth. Docket state lives in bib.sqlite; index state lives in the llm Postgres DB.
Config knob: [comments] seal_quiet_days = 30 in stack.toml; read via conf.cfg with a default when the section is absent ("comments" in cfg).
One deviation from the spec, decided here: extract staleness uses mtime comparison (combined.md mtime vs newest source file mtime), not a sources fingerprint in frontmatter. A frontmatter check requires reading the file, which violates the "zero combined.md reads" criterion. Spec rollout step 6 (frontmatter stamping) is therefore dropped.

File Structure

File	Responsibility
`src/bib/dockets.py` (new)	`Docket` dataclass, `should_seal`, `fingerprint_files`, `quiet_days()` — pure, no I/O except stat
`src/bib/schema.sql`	`dockets` table, `idx_items_url`, attachments unique index (guarded)
`src/bib/store.py`	docket CRUD, `upsert_status` (no-op upsert), idempotent `attach_file`
`src/bib/regulations_gov.py`	`iter_comments(since=, on_error=)`, `walk_docket`, sealed-skip in both backfills, `discover_docket`
`src/cli/bib.py`	fetch commands rewired to `walk_docket`; sealed skip before any API call
`src/cli/comments.py`	`dockets`, `seal`, `unseal` commands; extract with sealed skip, `--reattach`, lazy body lookup
`src/rex/comments/walker.py`	mtime-based currency, no reads for skipped dirs, `skip_dockets`, `reattach`
`src/rex/comments/combine.py`	`source_paths`, `is_current`
`src/llm/migrate.py`	`fingerprint` column, `index_docket_state` table
`src/llm/source.py`	`DocRef`, `iter_comment_refs`, `iter_rule_refs`, `iter_corpus_refs`, lazy `ZoteroPdfIndex`
`src/llm/index.py`	`index_refs` fingerprint-first loop; `index_docs` kept as a thin wrapper
`src/cli/llm.py`	wires refs + sealed/complete sets
`dev/scripts/dedupe_attachments.py` (new)	one-time duplicate cleanup
`dev/scripts/refarm_cms_2026_2377.sh`	plain chain

Task 1: `bib/dockets.py` — Docket, should_seal, fingerprint_files, quiet_days

Files:

Create: src/bib/dockets.py
Test: tests/bib/test_dockets.py

Interfaces:

Produces: Docket (frozen dataclass), should_seal(docket, today, quiet_days) -> bool, fingerprint_files(paths) -> str, quiet_days() -> int.
Step 1: Write the failing tests

# tests/bib/test_dockets.py
"""bib.dockets — pure docket state helpers."""

from __future__ import annotations

import os
from datetime import date
from pathlib import Path

from bib.dockets import Docket, fingerprint_files, quiet_days, should_seal


def _d(**kw) -> Docket:
    base = dict(
        id="CMS-2026-2377",
        rule_cms_id="CMS-1848-P",
        fr_document_id="CMS-2026-2377-0001",
        fr_object_id="0900006482921ba1",
        comment_end_date="2026-09-14",
        pull_watermark="",
        last_pull_at="",
        last_pull_new=None,
        sealed_at="",
        seal_reason="",
        counts_json="{}",
    )
    base.update(kw)
    return Docket(**base)


class TestShouldSeal:
    def test_seals_after_quiet_period_with_no_new(self):
        d = _d(last_pull_at="2026-10-15T03:00:00Z", last_pull_new=0)
        assert should_seal(d, date(2026, 10, 15), 30) is True

    def test_not_before_quiet_period(self):
        d = _d(last_pull_at="2026-10-13T03:00:00Z", last_pull_new=0)
        assert should_seal(d, date(2026, 10, 13), 30) is False

    def test_not_when_last_pull_found_new(self):
        d = _d(last_pull_at="2026-10-20T03:00:00Z", last_pull_new=3)
        assert should_seal(d, date(2026, 10, 20), 30) is False

    def test_not_when_last_pull_predates_quiet_period(self):
        # Pull happened before end+quiet even though today is well past it.
        d = _d(last_pull_at="2026-09-20T03:00:00Z", last_pull_new=0)
        assert should_seal(d, date(2026, 12, 1), 30) is False

    def test_not_without_close_date(self):
        d = _d(comment_end_date="", last_pull_at="2027-01-01T00:00:00Z", last_pull_new=0)
        assert should_seal(d, date(2027, 1, 1), 30) is False

    def test_not_when_already_sealed(self):
        d = _d(sealed_at="2026-10-15T00:00:00Z", last_pull_at="2026-10-15T03:00:00Z", last_pull_new=0)
        assert should_seal(d, date(2026, 10, 15), 30) is False

    def test_not_when_never_pulled(self):
        d = _d(last_pull_at="", last_pull_new=None)
        assert should_seal(d, date(2027, 1, 1), 30) is False


class TestFingerprintFiles:
    def test_order_independent(self, tmp_path: Path):
        a = tmp_path / "a.pdf"
        b = tmp_path / "b.pdf"
        a.write_bytes(b"aa")
        b.write_bytes(b"bbb")
        assert fingerprint_files([a, b]) == fingerprint_files([b, a])

    def test_changes_when_size_changes(self, tmp_path: Path):
        a = tmp_path / "a.pdf"
        a.write_bytes(b"aa")
        f1 = fingerprint_files([a])
        a.write_bytes(b"aaaa")
        assert fingerprint_files([a]) != f1

    def test_changes_when_mtime_changes(self, tmp_path: Path):
        a = tmp_path / "a.pdf"
        a.write_bytes(b"aa")
        f1 = fingerprint_files([a])
        os.utime(a, ns=(1_000_000_000_000_000_000, 1_000_000_000_000_000_000))
        assert fingerprint_files([a]) != f1

    def test_missing_file_is_recorded_not_fatal(self, tmp_path: Path):
        assert fingerprint_files([tmp_path / "nope.pdf"]) == fingerprint_files([tmp_path / "nope.pdf"])
        assert fingerprint_files([tmp_path / "nope.pdf"]) != fingerprint_files([])

    def test_empty_is_stable(self):
        assert fingerprint_files([]) == fingerprint_files([])
        assert len(fingerprint_files([])) == 64


class TestQuietDays:
    def test_default_when_section_missing(self, monkeypatch):
        from conf import _Cfg
        import bib.dockets as mod

        monkeypatch.setattr(mod, "_cfg", lambda: _Cfg({}))
        assert quiet_days() == 30

    def test_reads_section(self, monkeypatch):
        from conf import _Cfg
        import bib.dockets as mod

        monkeypatch.setattr(mod, "_cfg", lambda: _Cfg({"comments": {"seal_quiet_days": 7}}))
        assert quiet_days() == 7

Step 2: Run to verify failure

Run: uv run --no-sync pytest tests/bib/test_dockets.py -q -p no:cacheprovider Expected: FAIL with ModuleNotFoundError: No module named 'bib.dockets'

Step 3: Implement

# src/bib/dockets.py
"""Docket-level state for the regulations.gov comment pipeline.

A *docket* row (``dockets`` table in bib.sqlite) remembers what every
stage would otherwise re-derive from the network: the reg.gov document
that carries the comments, the comment close date, the pull watermark,
and — once the docket is known complete — a *seal*. Sealed dockets are
skipped by fetch, backfill, extract and index unless ``--force``.

Pure helpers only; the store owns persistence (``Store.docket_*``).
"""

from __future__ import annotations

import hashlib
from dataclasses import dataclass
from datetime import date, timedelta
from pathlib import Path
from typing import Iterable

DEFAULT_QUIET_DAYS = 30


@dataclass(frozen=True)
class Docket:
    id: str
    rule_cms_id: str = ""
    fr_document_id: str = ""
    fr_object_id: str = ""
    comment_end_date: str = ""  # YYYY-MM-DD
    pull_watermark: str = ""  # max lastModifiedDate seen on a clean walk
    last_pull_at: str = ""  # ISO-8601 UTC
    last_pull_new: int | None = None
    sealed_at: str = ""
    seal_reason: str = ""
    counts_json: str = "{}"

    @property
    def sealed(self) -> bool:
        return bool(self.sealed_at)


def should_seal(docket: Docket, today: date, quiet_days: int) -> bool:
    """True when the comment period closed ≥ *quiet_days* ago and the
    most recent completed pull, itself after that quiet boundary, found
    nothing new."""
    if docket.sealed or not docket.comment_end_date or not docket.last_pull_at:
        return False
    if docket.last_pull_new is None or docket.last_pull_new > 0:
        return False
    try:
        end = date.fromisoformat(docket.comment_end_date[:10])
        pulled = date.fromisoformat(docket.last_pull_at[:10])
    except ValueError:
        return False
    boundary = end + timedelta(days=quiet_days)
    return today >= boundary and pulled >= boundary


def fingerprint_files(paths: Iterable[Path]) -> str:
    """sha256 over sorted ``(name, size, mtime_ns)`` — cheap change
    detection without reading contents. Missing files contribute their
    name with size -1 so a deletion changes the fingerprint too."""
    rows: list[str] = []
    for p in paths:
        p = Path(p)
        try:
            st = p.stat()
            rows.append(f"{p.name}\x00{st.st_size}\x00{st.st_mtime_ns}")
        except FileNotFoundError:
            rows.append(f"{p.name}\x00-1\x000")
    rows.sort()
    return hashlib.sha256("\n".join(rows).encode()).hexdigest()


def _cfg():
    from conf import cfg

    return cfg


def quiet_days() -> int:
    """``[comments] seal_quiet_days`` from stack.toml, default 30."""
    c = _cfg()
    if "comments" in c and "seal_quiet_days" in c.comments:
        return int(c.comments.seal_quiet_days)
    return DEFAULT_QUIET_DAYS

Then add to stack.toml after the [db] block:

[comments]
seal_quiet_days = 30   # days after a docket's comment close date before an empty pull seals it

Step 4: Run to verify pass

Run: uv run --no-sync pytest tests/bib/test_dockets.py -q -p no:cacheprovider Expected: 14 passed

Step 5: Commit

git add src/bib/dockets.py tests/bib/test_dockets.py stack.toml
git commit -m "feat(bib): Docket model, should_seal, fingerprint_files, seal_quiet_days knob (refs #615)"

Task 2: `dockets` table + Store docket CRUD + url index

Files:

Modify: src/bib/schema.sql (append after fr_links)
Modify: src/bib/store.py (new section after Attachments & Notes)
Test: tests/bib/test_store_dockets.py

Interfaces:

Consumes: bib.dockets.Docket.
Produces: Store.docket_get(id) -> Docket | None, Store.docket_for_rule(cms_id) -> Docket | None, Store.docket_upsert(d: Docket) -> None, Store.dockets() -> list[Docket], Store.sealed_dockets() -> dict[str, str] (id → sealed_at), Store.docket_seal(id, *, reason, counts: dict) -> None, Store.docket_unseal(id) -> None.
Step 1: Write the failing tests

# tests/bib/test_store_dockets.py
"""Store.docket_* — persistence for bib.dockets.Docket."""

from __future__ import annotations

from bib.dockets import Docket
from bib.store import Store


def _store() -> Store:
    return Store(":memory:", storage_dir="/tmp/nope")


def test_table_exists_after_init():
    s = _store()
    names = {r[0] for r in s._con().execute("SELECT name FROM sqlite_master WHERE type='table'")}
    assert "dockets" in names
    idx = {r[0] for r in s._con().execute("SELECT name FROM sqlite_master WHERE type='index'")}
    assert "idx_items_url" in idx


def test_upsert_then_get_roundtrip():
    s = _store()
    d = Docket(id="CMS-2026-2377", rule_cms_id="CMS-1848-P", fr_object_id="obj", comment_end_date="2026-09-14")
    s.docket_upsert(d)
    got = s.docket_get("CMS-2026-2377")
    assert got == d
    assert s.docket_get("CMS-0000-0000") is None


def test_upsert_replaces_fields():
    s = _store()
    s.docket_upsert(Docket(id="D1", pull_watermark="a"))
    s.docket_upsert(Docket(id="D1", pull_watermark="b", last_pull_new=4))
    got = s.docket_get("D1")
    assert got.pull_watermark == "b"
    assert got.last_pull_new == 4


def test_docket_for_rule():
    s = _store()
    s.docket_upsert(Docket(id="D1", rule_cms_id="CMS-1848-P"))
    assert s.docket_for_rule("CMS-1848-P").id == "D1"
    assert s.docket_for_rule("CMS-9999-P") is None


def test_seal_unseal_and_sealed_dockets():
    s = _store()
    s.docket_upsert(Docket(id="D1"))
    s.docket_upsert(Docket(id="D2"))
    s.docket_seal("D1", reason="manual", counts={"comments": 3})
    d1 = s.docket_get("D1")
    assert d1.sealed and d1.seal_reason == "manual"
    assert '"comments": 3' in d1.counts_json
    assert set(s.sealed_dockets()) == {"D1"}
    assert s.sealed_dockets()["D1"] == d1.sealed_at
    s.docket_unseal("D1")
    assert not s.docket_get("D1").sealed
    assert s.sealed_dockets() == {}


def test_dockets_lists_sorted_by_id():
    s = _store()
    s.docket_upsert(Docket(id="CMS-2019-0111"))
    s.docket_upsert(Docket(id="CMS-2017-0092"))
    assert [d.id for d in s.dockets()] == ["CMS-2017-0092", "CMS-2019-0111"]

Step 2: Run to verify failure

Run: uv run --no-sync pytest tests/bib/test_store_dockets.py -q -p no:cacheprovider Expected: FAIL — AttributeError: 'Store' object has no attribute 'docket_upsert' (and the first test fails on the missing table).

Step 3: Implement

Append to src/bib/schema.sql:


-- ── Dockets (P47) ────────────────────────────────────────────────
-- One row per reg.gov docket: what every stage would otherwise re-fetch
-- (document/object ids, close date), the pull watermark, and the seal
-- that marks the docket complete. See bib/dockets.py.
CREATE TABLE IF NOT EXISTS dockets (
    id               TEXT PRIMARY KEY,
    rule_cms_id      TEXT NOT NULL DEFAULT '',
    fr_document_id   TEXT NOT NULL DEFAULT '',
    fr_object_id     TEXT NOT NULL DEFAULT '',
    comment_end_date TEXT NOT NULL DEFAULT '',
    pull_watermark   TEXT NOT NULL DEFAULT '',
    last_pull_at     TEXT NOT NULL DEFAULT '',
    last_pull_new    INTEGER,
    sealed_at        TEXT NOT NULL DEFAULT '',
    seal_reason      TEXT NOT NULL DEFAULT '',
    counts_json      TEXT NOT NULL DEFAULT '{}'
);

-- upsert() dedupes by URL on every comment; without this every upsert
-- was a full scan of items.
CREATE INDEX IF NOT EXISTS idx_items_url ON items(url);

Add to src/bib/store.py — import at top: from bib.dockets import Docket and import json (json is not yet imported there; add it). New section before # ── Attachments & Notes:

    # ── Dockets ──────────────────────────────────────────────────

    _DOCKET_COLS = (
        "id", "rule_cms_id", "fr_document_id", "fr_object_id",
        "comment_end_date", "pull_watermark", "last_pull_at",
        "last_pull_new", "sealed_at", "seal_reason", "counts_json",
    )

    def _docket_from_row(self, row: sqlite3.Row | None) -> Docket | None:
        if row is None:
            return None
        d = {c: row[c] for c in self._DOCKET_COLS}
        return Docket(**d)

    def docket_get(self, docket_id: str) -> Docket | None:
        row = self._con().execute(
            "SELECT * FROM dockets WHERE id = ?", (docket_id,)
        ).fetchone()
        return self._docket_from_row(row)

    def docket_for_rule(self, cms_rule_id: str) -> Docket | None:
        row = self._con().execute(
            "SELECT * FROM dockets WHERE rule_cms_id = ? ORDER BY id LIMIT 1",
            (cms_rule_id,),
        ).fetchone()
        return self._docket_from_row(row)

    def docket_upsert(self, docket: Docket) -> None:
        cols = ", ".join(self._DOCKET_COLS)
        marks = ", ".join(f":{c}" for c in self._DOCKET_COLS)
        sets = ", ".join(f"{c} = excluded.{c}" for c in self._DOCKET_COLS if c != "id")
        con = self._con()
        con.execute(
            f"INSERT INTO dockets ({cols}) VALUES ({marks}) "  # noqa: S608
            f"ON CONFLICT(id) DO UPDATE SET {sets}",
            {c: getattr(docket, c) for c in self._DOCKET_COLS},
        )
        con.commit()

    def dockets(self) -> list[Docket]:
        rows = self._con().execute("SELECT * FROM dockets ORDER BY id").fetchall()
        return [self._docket_from_row(r) for r in rows]

    def sealed_dockets(self) -> dict[str, str]:
        """docket id → sealed_at for every sealed docket."""
        rows = self._con().execute(
            "SELECT id, sealed_at FROM dockets WHERE sealed_at <> ''"
        ).fetchall()
        return {r["id"]: r["sealed_at"] for r in rows}

    def docket_seal(self, docket_id: str, *, reason: str, counts: dict[str, int]) -> None:
        from datetime import datetime, timezone

        now = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
        con = self._con()
        con.execute(
            "UPDATE dockets SET sealed_at = ?, seal_reason = ?, counts_json = ? WHERE id = ?",
            (now, reason, json.dumps(counts, sort_keys=True), docket_id),
        )
        con.commit()

    def docket_unseal(self, docket_id: str) -> None:
        con = self._con()
        con.execute(
            "UPDATE dockets SET sealed_at = '', seal_reason = '' WHERE id = ?",
            (docket_id,),
        )
        con.commit()

Step 4: Run to verify pass

Run: uv run --no-sync pytest tests/bib/test_store_dockets.py tests/bib/test_store.py -q -p no:cacheprovider Expected: all pass.

Step 5: Commit

git add src/bib/schema.sql src/bib/store.py tests/bib/test_store_dockets.py
git commit -m "feat(bib): dockets table + Store.docket_* CRUD; index items.url (refs #615)"

Task 3: No-op upsert — `Store.upsert_status`

Files:

Modify: src/bib/store.py:207-245 (upsert)
Test: tests/bib/test_store_upsert_noop.py

Interfaces:

Produces: Store.upsert_status(item, *, tags=None, collection="") -> tuple[str, str] where the second element is "created" | "updated" | "unchanged". Store.upsert keeps its signature and returns the key only.
Step 1: Write the failing tests

# tests/bib/test_store_upsert_noop.py
"""Store.upsert must not rewrite rows/tags when nothing changed."""

from __future__ import annotations

from bib.item import Source
from bib.store import Store


def _store() -> Store:
    return Store(":memory:", storage_dir="/tmp/nope")


def _item(**kw) -> Source:
    it = Source(title="T", url="https://www.regulations.gov/comment/CMS-2026-2377-1")
    it.abstract = kw.get("abstract", "body")
    for t in kw.get("tags", ["a:1", "b:2"]):
        it.add_tag(t)
    return it


def _snapshot(s: Store, key: str) -> tuple:
    con = s._con()
    row = con.execute("SELECT access_date, updated_at FROM items WHERE key=?", (key,)).fetchone()
    tags = con.execute(
        "SELECT it.rowid FROM item_tags it JOIN items i ON i.id=it.item_id WHERE i.key=? ORDER BY 1",
        (key,),
    ).fetchall()
    return (row["access_date"], row["updated_at"], [t[0] for t in tags])


def test_first_upsert_is_created():
    s = _store()
    key, status = s.upsert_status(_item())
    assert status == "created" and key


def test_identical_upsert_is_unchanged_and_writes_nothing():
    s = _store()
    key, _ = s.upsert_status(_item())
    before = _snapshot(s, key)
    # different Python object, same content
    key2, status = s.upsert_status(_item())
    assert key2 == key and status == "unchanged"
    assert _snapshot(s, key) == before  # no access/updated stamp, no tag row churn


def test_changed_abstract_is_updated():
    s = _store()
    key, _ = s.upsert_status(_item(abstract="v1"))
    _, status = s.upsert_status(_item(abstract="v2"))
    assert status == "updated"
    assert s.get(key).abstract == "v2"


def test_new_tag_is_updated_and_merged():
    s = _store()
    key, _ = s.upsert_status(_item(tags=["a:1"]))
    _, status = s.upsert_status(_item(tags=["c:3"]))
    assert status == "updated"
    assert set(s.get(key).tags) >= {"a:1", "c:3"}


def test_subset_of_existing_tags_is_unchanged():
    """Upsert never removes tags (#624); a subset therefore changes nothing."""
    s = _store()
    key, _ = s.upsert_status(_item(tags=["a:1", "b:2"]))
    _, status = s.upsert_status(_item(tags=["a:1"]))
    assert status == "unchanged"


def test_upsert_keeps_returning_key():
    s = _store()
    key = s.upsert(_item())
    assert s.upsert(_item()) == key

Step 2: Run to verify failure

Run: uv run --no-sync pytest tests/bib/test_store_upsert_noop.py -q -p no:cacheprovider Expected: FAIL — AttributeError: 'Store' object has no attribute 'upsert_status'

Step 3: Implement

Replace the body of Store.upsert in src/bib/store.py with:

    _COMPARE_COLS = (
        "item_type", "title", "url", "date_published",
        "abstract", "institution", "extra", "extra_json",
    )

    def upsert(
        self,
        item: Item,
        *,
        tags: list[Any] | None = None,
        collection: str = "",
    ) -> str:
        """Create or update an item, deduplicating by URL."""
        return self.upsert_status(item, tags=tags, collection=collection)[0]

    def upsert_status(
        self,
        item: Item,
        *,
        tags: list[Any] | None = None,
        collection: str = "",
    ) -> tuple[str, str]:
        """Like :meth:`upsert` but also report what happened:
        ``"created"``, ``"updated"`` or ``"unchanged"``.

        ``unchanged`` means every compared column, the merged tag set and
        the merged collection set already match the stored row — nothing
        is written, no ``access_date``/``updated_at`` stamp, no tag row
        churn. A re-farm over a complete docket is therefore a read-only
        pass over ``items``.
        """
        if tags:
            for tag in tags:
                label = tag.label if hasattr(tag, "label") else str(tag)
                item.add_tag(label)
        if collection and collection not in item.collections:
            item.collections.append(collection)

        if not item.url:
            return self.create(item), "created"

        con = self._con()
        existing = con.execute(
            "SELECT key FROM items WHERE url = ?", (item.url,)
        ).fetchone()
        if not existing:
            return self.create(item), "created"

        ekey = existing["key"]
        # Merge with what's already stored — a re-ingest must never
        # clobber tags/collections curated on the row since the last
        # ingest (#624). Deliberate removal goes through remove_tag.
        current = self.get(ekey)
        merged_tags = list(dict.fromkeys([*current.tags, *item.tags]))
        merged_cols = list(dict.fromkeys([*current.collections, *item.collections]))

        new_row = item.to_row()
        cur_row = current.to_row()
        same_cols = all(new_row.get(c, "") == cur_row.get(c, "") for c in self._COMPARE_COLS)
        if same_cols and set(merged_tags) == set(current.tags) and set(merged_cols) == set(current.collections):
            return ekey, "unchanged"

        item.stamp_access()
        row = item.to_row()
        row.pop("key", None)
        row["tags"] = merged_tags
        row["collections"] = merged_cols
        self.update(ekey, **row)
        return ekey, "updated"

Note: Item.to_row() for subclasses packs subclass fields into extra_json; comparing extra_json strings works because both sides serialize through the same to_row. current came from Item.from_row, so current.to_row()["extra_json"] is the canonical serialization of the stored data.

Step 4: Run to verify pass

Run: uv run --no-sync pytest tests/bib/test_store_upsert_noop.py tests/bib/test_store.py tests/bib -q -p no:cacheprovider Expected: all pass. If an existing test asserts that access_date changes on an identical re-upsert, update that test: the new contract is "unchanged → no stamp".

Step 5: Commit

git add src/bib/store.py tests/bib/test_store_upsert_noop.py
git commit -m "feat(bib): Store.upsert_status — identical re-upsert writes nothing (refs #615)"

Task 3b: Data-preserving upsert — never blank a stored value

Files:

Modify: src/bib/store.py (upsert_status)
Test: tests/bib/test_store_upsert_noop.py (extend)

Why (incident 2026-09-08): the fetch list walk builds a Source whose abstract is empty (the list endpoint has no body) and re-upserts every comment it sees. With the old upsert that overwrote 17,607 enriched bodies in CMS-2026-2377 and 1,115 in CMS-2017-0092. Task 3's comparison alone would still call that "updated" and write the empty abstract. Upsert must merge, not replace: an empty incoming value carries no information.

Interfaces:

Consumes: Store.upsert_status from Task 3.
Produces: same signature; for every column in _COMPARE_COLS (except extra_json) an empty incoming string keeps the stored value. extra_json is compared/written as today (it is always a full serialization).
Step 1: Write the failing tests (append to tests/bib/test_store_upsert_noop.py)

def test_empty_incoming_abstract_keeps_stored_body():
    """A list-walk row (no body) must not blank an enriched comment."""
    s = _store()
    key, _ = s.upsert_status(_item(abstract="enriched body"))
    _, status = s.upsert_status(_item(abstract=""))
    assert status == "unchanged"
    assert s.get(key).abstract == "enriched body"


def test_empty_incoming_title_keeps_stored_title():
    s = _store()
    it = _item(); it.title = "Org: CMS-2026-2377-1"
    key, _ = s.upsert_status(it)
    it2 = _item(); it2.title = ""
    _, status = s.upsert_status(it2)
    assert status == "unchanged"
    assert s.get(key).title == "Org: CMS-2026-2377-1"


def test_non_empty_incoming_still_updates():
    s = _store()
    key, _ = s.upsert_status(_item(abstract="v1"))
    _, status = s.upsert_status(_item(abstract="v2"))
    assert status == "updated" and s.get(key).abstract == "v2"

Step 2: Run to verify failure

Run: uv run --no-sync pytest tests/bib/test_store_upsert_noop.py -q -p no:cacheprovider Expected: the first two new tests FAIL (status == "updated", abstract/title blanked).

Step 3: Implement

In upsert_status, after cur_row = current.to_row() and before the same_cols check:

        # Merge, don't replace: an empty incoming value carries no
        # information (a list-walk row has no body), so the stored value
        # wins. extra_json is always a full serialization and is exempt.
        for c in self._COMPARE_COLS:
            if c != "extra_json" and not new_row.get(c) and cur_row.get(c):
                new_row[c] = cur_row[c]
                setattr(item, c, cur_row[c])

setattr(item, c, …) keeps the later item.to_row() (used to build the written row) consistent with the merged values. Item is a pydantic model; plain attribute assignment is allowed on these fields.

Step 4: Run to verify pass

Run: uv run --no-sync pytest tests/bib -q -p no:cacheprovider Expected: pass.

Step 5: Commit

git add src/bib/store.py tests/bib/test_store_upsert_noop.py
git commit -m "fix(bib): upsert never blanks a stored value — empty incoming columns keep the stored ones (refs #615)"

Task 4: Idempotent `attach_file`

Files:

Modify: src/bib/store.py:502-533 (attach_file)
Test: tests/bib/test_store_attach_idempotent.py

Interfaces:

Produces: Store.attach_file(item_key, path, *, title="") -> str returns the existing attachment key when the item already has an attachment whose filename equals title or path.name.
Step 1: Write the failing test

# tests/bib/test_store_attach_idempotent.py
from __future__ import annotations

from pathlib import Path

from bib.item import Source
from bib.store import Store


def test_second_attach_of_same_filename_returns_existing_key(tmp_path: Path):
    s = Store(str(tmp_path / "bib.sqlite"), storage_dir=tmp_path / "storage")
    key = s.create(Source(title="T", url="https://x/1"))
    f = tmp_path / "attachment_1.pdf"
    f.write_bytes(b"%PDF")
    k1 = s.attach_file(key, f, title="attachment_1.pdf")
    k2 = s.attach_file(key, f, title="attachment_1.pdf")
    assert k1 == k2
    n = s._con().execute("SELECT count(*) FROM attachments").fetchone()[0]
    assert n == 1
    # exactly one storage copy
    assert len(list((tmp_path / "storage").iterdir())) == 1


def test_different_filename_creates_second_row(tmp_path: Path):
    s = Store(str(tmp_path / "bib.sqlite"), storage_dir=tmp_path / "storage")
    key = s.create(Source(title="T", url="https://x/1"))
    f1 = tmp_path / "a.pdf"; f1.write_bytes(b"a")
    f2 = tmp_path / "b.pdf"; f2.write_bytes(b"b")
    assert s.attach_file(key, f1) != s.attach_file(key, f2)

Step 2: Run to verify failure

Run: uv run --no-sync pytest tests/bib/test_store_attach_idempotent.py -q -p no:cacheprovider Expected: first test FAILS (k1 != k2, count 2).

Step 3: Implement

In attach_file, after the item lookup and before generating a key:

        filename = title or path.name
        dup = con.execute(
            "SELECT key FROM attachments WHERE item_id = ? AND filename = ?",
            (row["id"], filename),
        ).fetchone()
        if dup:
            return dup["key"]

and use filename in the INSERT instead of title or path.name.

Step 4: Run to verify pass

Run: uv run --no-sync pytest tests/bib/test_store_attach_idempotent.py tests/bib -q -p no:cacheprovider Expected: pass.

Step 5: Commit

git add src/bib/store.py tests/bib/test_store_attach_idempotent.py
git commit -m "fix(bib): attach_file is idempotent on (item, filename) (refs #615)"

Task 5: Dedupe migration script + guarded unique index

Files:

Create: dev/scripts/dedupe_attachments.py
Modify: src/bib/store.py (_init_schema)
Test: tests/scripts/test_dedupe_attachments.py

Interfaces:

Produces: dedupe_attachments.plan(con) -> list[Group], dedupe_attachments.apply(con, groups) -> Report; CLI uv run python dev/scripts/dedupe_attachments.py [--db PATH] [--apply]. Store._init_schema creates idx_attachments_item_filename (UNIQUE on (item_id, filename)) only when no duplicate groups remain.
Step 1: Write the failing tests

# tests/scripts/test_dedupe_attachments.py
from __future__ import annotations

import importlib.util
import sqlite3
from pathlib import Path

from bib.item import Source
from bib.store import Store

_SCRIPT = Path(__file__).resolve().parents[2] / "dev" / "scripts" / "dedupe_attachments.py"
spec = importlib.util.spec_from_file_location("dedupe_attachments", _SCRIPT)
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)


def _seed(tmp_path: Path) -> tuple[Store, str]:
    s = Store(str(tmp_path / "bib.sqlite"), storage_dir=tmp_path / "storage")
    key = s.create(Source(title="T", url="https://x/1"))
    f = tmp_path / "attachment_1.pdf"
    f.write_bytes(b"%PDF-dup")
    # Simulate the old non-idempotent attach: three rows, three copies.
    con = s._con()
    item_id = con.execute("SELECT id FROM items WHERE key=?", (key,)).fetchone()[0]
    for k in ("AAAAAAAA", "BBBBBBBB", "CCCCCCCC"):
        d = tmp_path / "storage" / k
        d.mkdir(parents=True)
        (d / "attachment_1.pdf").write_bytes(b"%PDF-dup")
        con.execute(
            "INSERT INTO attachments (item_id, key, filename, content_type, storage_path) VALUES (?,?,?,?,?)",
            (item_id, k, "attachment_1.pdf", "application/pdf", str(d / "attachment_1.pdf")),
        )
    con.commit()
    return s, key


def test_plan_finds_group_and_keeps_oldest(tmp_path: Path):
    s, _ = _seed(tmp_path)
    groups = mod.plan(s._con())
    assert len(groups) == 1
    g = groups[0]
    assert g.keep == "AAAAAAAA"
    assert sorted(g.remove) == ["BBBBBBBB", "CCCCCCCC"]


def test_apply_removes_rows_and_files(tmp_path: Path):
    s, _ = _seed(tmp_path)
    rep = mod.apply(s._con(), mod.plan(s._con()))
    assert rep.rows_removed == 2
    assert rep.files_removed == 2
    assert s._con().execute("SELECT count(*) FROM attachments").fetchone()[0] == 1
    assert (tmp_path / "storage" / "AAAAAAAA" / "attachment_1.pdf").is_file()
    assert not (tmp_path / "storage" / "BBBBBBBB").exists()
    assert mod.plan(s._con()) == []


def test_refuses_group_with_differing_sizes(tmp_path: Path):
    s, _ = _seed(tmp_path)
    (tmp_path / "storage" / "CCCCCCCC" / "attachment_1.pdf").write_bytes(b"different-longer")
    groups = mod.plan(s._con())
    assert groups[0].conflict is True
    rep = mod.apply(s._con(), groups)
    assert rep.rows_removed == 0 and rep.skipped_conflicts == 1


def test_unique_index_created_only_when_clean(tmp_path: Path):
    s, _ = _seed(tmp_path)
    s.close()
    s2 = Store(str(tmp_path / "bib.sqlite"), storage_dir=tmp_path / "storage")
    idx = {r[0] for r in s2._con().execute("SELECT name FROM sqlite_master WHERE type='index'")}
    assert "idx_attachments_item_filename" not in idx
    mod.apply(s2._con(), mod.plan(s2._con()))
    s2.close()
    s3 = Store(str(tmp_path / "bib.sqlite"), storage_dir=tmp_path / "storage")
    idx = {r[0] for r in s3._con().execute("SELECT name FROM sqlite_master WHERE type='index'")}
    assert "idx_attachments_item_filename" in idx

Step 2: Run to verify failure

Run: uv run --no-sync pytest tests/scripts/test_dedupe_attachments.py -q -p no:cacheprovider Expected: FAIL — script file missing.

Step 3: Implement the script

#!/usr/bin/env python3
"""One-time cleanup of duplicate bib attachments (refs #615).

``Store.attach_file`` used to mint a new row + storage copy on every
call, so re-farms produced thousands of ``(item_id, filename)``
duplicates (33,379 groups / 66,917 of 78,017 rows on 2026-09-08). This
keeps the oldest row of each group, deletes the others' rows and their
storage copies, and prints a report. Dry-run by default.

Zotero is not touched: ``bib.sync`` already dedupes child attachments by
filename, so a duplicate that was synced once is a single Zotero child
attachment and stays valid.

Usage::

    uv run python dev/scripts/dedupe_attachments.py            # report only
    uv run python dev/scripts/dedupe_attachments.py --apply    # delete
"""

from __future__ import annotations

import argparse
import os
import sqlite3
import sys
from dataclasses import dataclass, field
from pathlib import Path


@dataclass
class Group:
    item_id: int
    filename: str
    keep: str
    remove: list[str] = field(default_factory=list)
    remove_paths: list[str] = field(default_factory=list)
    conflict: bool = False  # sizes differ → do not touch


@dataclass
class Report:
    groups: int = 0
    rows_removed: int = 0
    files_removed: int = 0
    bytes_freed: int = 0
    skipped_conflicts: int = 0
    removed_keys: list[str] = field(default_factory=list)


def plan(con: sqlite3.Connection) -> list[Group]:
    con.row_factory = sqlite3.Row
    rows = con.execute(
        """
        SELECT a.item_id, a.filename, a.key, a.storage_path, a.rowid AS rid
        FROM attachments a
        WHERE (a.item_id, a.filename) IN (
            SELECT item_id, filename FROM attachments
            GROUP BY item_id, filename HAVING count(*) > 1
        )
        ORDER BY a.item_id, a.filename, a.rowid
        """
    ).fetchall()
    groups: dict[tuple[int, str], Group] = {}
    for r in rows:
        gkey = (r["item_id"], r["filename"])
        g = groups.get(gkey)
        if g is None:
            groups[gkey] = Group(item_id=r["item_id"], filename=r["filename"], keep=r["key"])
            continue
        g.remove.append(r["key"])
        g.remove_paths.append(r["storage_path"])
    # conflict check: every copy must have the same size as the kept one
    for g in groups.values():
        keep_path = con.execute(
            "SELECT storage_path FROM attachments WHERE key = ?", (g.keep,)
        ).fetchone()["storage_path"]
        keep_size = _size(keep_path)
        for p in g.remove_paths:
            if _size(p) not in (keep_size, -1):
                g.conflict = True
                break
    return list(groups.values())


def _size(path: str) -> int:
    try:
        return os.stat(path).st_size
    except OSError:
        return -1


def apply(con: sqlite3.Connection, groups: list[Group]) -> Report:
    rep = Report(groups=len(groups))
    con.execute("BEGIN")
    try:
        for g in groups:
            if g.conflict:
                rep.skipped_conflicts += 1
                continue
            for key, path in zip(g.remove, g.remove_paths):
                still_referenced = con.execute(
                    "SELECT count(*) FROM attachments WHERE storage_path = ? AND key <> ?",
                    (path, key),
                ).fetchone()[0]
                con.execute("DELETE FROM attachments WHERE key = ?", (key,))
                rep.rows_removed += 1
                rep.removed_keys.append(key)
                if not still_referenced:
                    p = Path(path)
                    if p.is_file():
                        rep.bytes_freed += p.stat().st_size
                        p.unlink()
                        rep.files_removed += 1
                        try:
                            p.parent.rmdir()  # the per-key dir, if now empty
                        except OSError:
                            pass
        con.execute("COMMIT")
    except Exception:
        con.execute("ROLLBACK")
        raise
    return rep


def main(argv: list[str] | None = None) -> int:
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("--db", default="", help="bib.sqlite path (default: stack.toml db.bib)")
    ap.add_argument("--apply", action="store_true", help="delete duplicates (default: report only)")
    args = ap.parse_args(argv)
    if args.db:
        db = args.db
    else:
        sys.path.insert(0, str(Path(__file__).resolve().parents[2] / "src"))
        from conf import path

        db = str(path("db.bib"))
    con = sqlite3.connect(db, isolation_level=None)
    groups = plan(con)
    conflicts = sum(1 for g in groups if g.conflict)
    rows = sum(len(g.remove) for g in groups)
    print(f"{db}: {len(groups)} duplicate groups, {rows} rows to remove, {conflicts} conflicts (size mismatch, skipped)")
    if not args.apply:
        print("dry run — pass --apply to delete")
        return 0
    rep = apply(con, groups)
    print(
        f"removed rows={rep.rows_removed} files={rep.files_removed} "
        f"freed={rep.bytes_freed / 1e6:.1f} MB skipped_conflicts={rep.skipped_conflicts}"
    )
    print("reopen the store once (any `stack bib` command) to create the unique index")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Then in src/bib/store.py _init_schema, after executescript(ddl):

        # The (item_id, filename) unique index can only exist once the
        # dedupe migration has run (dev/scripts/dedupe_attachments.py);
        # until then we leave it off rather than fail to open the store.
        con = self._con()
        dup = con.execute(
            "SELECT 1 FROM attachments GROUP BY item_id, filename HAVING count(*) > 1 LIMIT 1"
        ).fetchone()
        if dup is None:
            con.execute(
                "CREATE UNIQUE INDEX IF NOT EXISTS idx_attachments_item_filename "
                "ON attachments(item_id, filename)"
            )
            con.commit()

Step 4: Run to verify pass

Run: uv run --no-sync pytest tests/scripts/test_dedupe_attachments.py tests/bib -q -p no:cacheprovider Expected: pass.

Step 5: Commit

git add dev/scripts/dedupe_attachments.py src/bib/store.py tests/scripts/test_dedupe_attachments.py
git commit -m "feat(bib): dedupe_attachments migration + guarded unique index on (item, filename) (refs #615)"

Task 6: `iter_comments(since=, on_error=)`

Files:

Modify: src/bib/regulations_gov.py:198-258 (Client.iter_comments)
Test: tests/bib/test_regulations_gov_since.py

Interfaces:

Produces: Client.iter_comments(object_id, *, since: str = "", on_error: Callable[[Exception], None] | None = None) -> Iterator[Comment]. since seeds the filter[lastModifiedDate][ge] cursor; on_error is called (then the walk stops, as today) when a page returns an HTTP error.
Step 1: Write the failing tests

# tests/bib/test_regulations_gov_since.py
from __future__ import annotations

import httpx

from bib.regulations_gov import Client


def _row(cid: str, lm: str) -> dict:
    return {"id": cid, "attributes": {"lastModifiedDate": lm, "postedDate": lm, "docketId": "D", "commentOnId": "x"}}


def _client(handler) -> Client:
    http = httpx.Client(transport=httpx.MockTransport(handler), headers={"X-Api-Key": "k"})
    return Client(api_key="k", sleep=0, client=http)


def test_since_seeds_the_date_filter():
    seen: list[dict] = []

    def handler(req: httpx.Request) -> httpx.Response:
        seen.append(dict(req.url.params))
        return httpx.Response(200, json={"data": [_row("D-1", "2026-09-08T12:00:00Z")], "meta": {"totalPages": 1}})

    api = _client(handler)
    out = list(api.iter_comments("obj", since="2026-09-01T00:00:00Z"))
    assert [c.id for c in out] == ["D-1"]
    assert seen[0]["filter[lastModifiedDate][ge]"] == "2026-09-01 00:00:00"


def test_no_since_means_no_date_filter():
    seen: list[dict] = []

    def handler(req: httpx.Request) -> httpx.Response:
        seen.append(dict(req.url.params))
        return httpx.Response(200, json={"data": [], "meta": {"totalPages": 1}})

    list(_client(handler).iter_comments("obj"))
    assert "filter[lastModifiedDate][ge]" not in seen[0]


def test_on_error_called_when_page_fails():
    errors: list[Exception] = []

    def handler(req: httpx.Request) -> httpx.Response:
        return httpx.Response(500, json={})

    out = list(_client(handler).iter_comments("obj", on_error=errors.append))
    assert out == []
    assert len(errors) == 1 and isinstance(errors[0], httpx.HTTPStatusError)

Step 2: Run to verify failure

Run: uv run --no-sync pytest tests/bib/test_regulations_gov_since.py -q -p no:cacheprovider Expected: FAIL — TypeError: iter_comments() got an unexpected keyword argument 'since'

Step 3: Implement

Change the signature and the two touched lines in iter_comments:

    def iter_comments(
        self,
        object_id: str,
        *,
        since: str = "",
        on_error: "Callable[[Exception], None] | None" = None,
    ) -> Iterator[Comment]:
        """... (keep the existing docstring; add:)

        *since* seeds the ``lastModifiedDate`` cursor so an incremental
        walk starts where the last clean one ended (the boundary row is
        re-yielded; the store's no-op upsert absorbs it). *on_error* is
        invoked with the ``HTTPStatusError`` before the walk stops, so a
        caller can tell "walked to the end" from "gave up".
        """
        cursor: str | None = _reg_date(since) if since else None
        page = 1
        while True:
            ...
            try:
                data = self._get("/comments", **params)
            except httpx.HTTPStatusError as e:
                log.warning(...)  # unchanged
                if on_error is not None:
                    on_error(e)
                break

Add from typing import TYPE_CHECKING, Callable, Iterator at the top (Callable joins the existing import).

Step 4: Run to verify pass

Run: uv run --no-sync pytest tests/bib/test_regulations_gov_since.py tests/bib/test_regulations_gov_exercise.py -q -p no:cacheprovider Expected: pass.

Step 5: Commit

git add src/bib/regulations_gov.py tests/bib/test_regulations_gov_since.py
git commit -m "feat(bib): iter_comments since= watermark + on_error hook (refs #615)"

Task 7: `discover_docket` + `walk_docket` (watermark, counts, auto-seal)

Files:

Modify: src/bib/regulations_gov.py (new section after upsert_comment)
Test: tests/bib/test_walk_docket.py

Interfaces:

Consumes: Store.docket_get/docket_upsert/docket_seal, Client.iter_comments(since=, on_error=), Client.find_documents_in_docket, Client.attachments_for, Client.download_attachment, Store.upsert_status, bib.dockets.should_seal.
Produces:
- discover_docket(api, docket_id, *, rule_cms_id="") -> Docket — one find_documents_in_docket call; picks the document with an objectId and the latest commentEndDate.
- WalkResult dataclass: created, updated, unchanged: int, watermark: str, clean: bool, sealed: bool.
- walk_docket(store, api, docket, *, attachments=False, limit=0, force=False, quiet_days=30, today=None, scratch_root=Path(".state/comments"), echo=None) -> WalkResult.
Step 1: Write the failing tests

# tests/bib/test_walk_docket.py
from __future__ import annotations

from datetime import date
from unittest.mock import MagicMock

import httpx

from bib.dockets import Docket
from bib.regulations_gov import Comment, WalkResult, discover_docket, walk_docket
from bib.store import Store

D = "CMS-2026-2377"


def _c(n: int, lm: str) -> Comment:
    return Comment(
        id=f"{D}-{n}", title="", posted_date=lm[:10], received_date=lm[:10],
        docket_id=D, comment_on_id="x", raw={"attributes": {"lastModifiedDate": lm}},
    )


def _store() -> Store:
    return Store(":memory:", storage_dir="/tmp/nope")


def _docket(**kw) -> Docket:
    base = dict(id=D, rule_cms_id="CMS-1848-P", fr_object_id="obj", comment_end_date="2026-09-14")
    base.update(kw)
    return Docket(**base)


def test_discover_docket_picks_commentable_doc():
    api = MagicMock()
    api.find_documents_in_docket.return_value = [
        {"id": "X-1", "attributes": {"objectId": "o1"}},  # no comment window
        {"id": "X-2", "attributes": {"objectId": "o2", "commentEndDate": "2026-09-14T03:59:59Z"}},
    ]
    d = discover_docket(api, D, rule_cms_id="CMS-1848-P")
    assert d == Docket(id=D, rule_cms_id="CMS-1848-P", fr_document_id="X-2", fr_object_id="o2", comment_end_date="2026-09-14")
    api.find_documents_in_docket.assert_called_once_with(D)


def test_walk_creates_counts_and_advances_watermark():
    s = _store()
    s.docket_upsert(_docket())
    api = MagicMock()
    api.iter_comments.return_value = [_c(1, "2026-09-01T00:00:00Z"), _c(2, "2026-09-02T00:00:00Z")]
    r = walk_docket(s, api, s.docket_get(D), today=date(2026, 9, 8))
    assert r == WalkResult(created=2, updated=0, unchanged=0, watermark="2026-09-02T00:00:00Z", clean=True, sealed=False)
    d = s.docket_get(D)
    assert d.pull_watermark == "2026-09-02T00:00:00Z"
    assert d.last_pull_new == 2 and d.last_pull_at
    api.iter_comments.assert_called_once()
    assert api.iter_comments.call_args.kwargs["since"] == ""


def test_walk_passes_watermark_and_counts_unchanged():
    s = _store()
    s.docket_upsert(_docket())
    api = MagicMock()
    api.iter_comments.return_value = [_c(1, "2026-09-01T00:00:00Z")]
    walk_docket(s, api, s.docket_get(D), today=date(2026, 9, 8))
    api.iter_comments.return_value = [_c(1, "2026-09-01T00:00:00Z")]  # boundary re-yield
    r = walk_docket(s, api, s.docket_get(D), today=date(2026, 9, 8))
    assert api.iter_comments.call_args.kwargs["since"] == "2026-09-01T00:00:00Z"
    assert (r.created, r.unchanged) == (0, 1)
    assert s.docket_get(D).last_pull_new == 0


def test_unclean_walk_keeps_old_watermark():
    s = _store()
    s.docket_upsert(_docket(pull_watermark="2026-08-01T00:00:00Z"))
    api = MagicMock()

    def _iter(_obj, *, since="", on_error=None):
        yield _c(9, "2026-09-05T00:00:00Z")
        on_error(httpx.HTTPStatusError("boom", request=MagicMock(), response=MagicMock()))

    api.iter_comments.side_effect = _iter
    r = walk_docket(s, api, s.docket_get(D), today=date(2026, 9, 8))
    assert r.clean is False and r.created == 1
    assert s.docket_get(D).pull_watermark == "2026-08-01T00:00:00Z"


def test_force_walks_from_scratch_and_never_seals():
    s = _store()
    s.docket_upsert(_docket(pull_watermark="2026-08-01T00:00:00Z"))
    api = MagicMock()
    api.iter_comments.return_value = []
    r = walk_docket(s, api, s.docket_get(D), force=True, today=date(2027, 1, 1))
    assert api.iter_comments.call_args.kwargs["since"] == ""
    assert r.sealed is False and not s.docket_get(D).sealed


def test_auto_seal_after_quiet_empty_pull():
    s = _store()
    s.docket_upsert(_docket())
    api = MagicMock()
    api.iter_comments.return_value = []
    r = walk_docket(s, api, s.docket_get(D), today=date(2026, 10, 20), quiet_days=30)
    assert r.sealed is True
    d = s.docket_get(D)
    assert d.sealed and d.seal_reason == "auto"

Step 2: Run to verify failure

Run: uv run --no-sync pytest tests/bib/test_walk_docket.py -q -p no:cacheprovider Expected: FAIL — ImportError: cannot import name 'WalkResult'

Step 3: Implement

Append to src/bib/regulations_gov.py (after upsert_comment):

# ── Docket walks ───────────────────────────────────────────────


def discover_docket(client: Client, docket_id: str, *, rule_cms_id: str = "") -> Docket:
    """One ``/documents`` listing → the docket's commentable document.

    Picks the document that has an ``objectId`` and a ``commentEndDate``
    (latest close date wins when several qualify). Called once per
    docket; the result is persisted so later runs make no API call.
    """
    best: dict | None = None
    for fr_doc in client.find_documents_in_docket(docket_id):
        attrs = fr_doc.get("attributes") or {}
        if not attrs.get("objectId") or not attrs.get("commentEndDate"):
            continue
        if best is None or attrs["commentEndDate"] > (best.get("attributes") or {})["commentEndDate"]:
            best = fr_doc
    attrs = (best or {}).get("attributes") or {}
    return Docket(
        id=docket_id,
        rule_cms_id=rule_cms_id,
        fr_document_id=(best or {}).get("id", ""),
        fr_object_id=attrs.get("objectId", ""),
        comment_end_date=(attrs.get("commentEndDate") or "")[:10],
    )


@dataclass
class WalkResult:
    created: int = 0
    updated: int = 0
    unchanged: int = 0
    watermark: str = ""
    clean: bool = True
    sealed: bool = False


def walk_docket(
    store: Store,
    client: Client,
    docket: Docket,
    *,
    attachments: bool = False,
    limit: int = 0,
    force: bool = False,
    quiet_days: int = 30,
    today: date | None = None,
    scratch_root: Path = Path(".state/comments"),
    echo: Callable[[str], None] | None = None,
) -> WalkResult:
    """Pull *docket*'s comments from the stored watermark, upsert them,
    advance the watermark on a clean walk, and auto-seal when
    :func:`should_seal` says so. ``force`` walks from page 1 and never
    seals. Returns per-status counts.
    """
    from bib.dockets import should_seal

    res = WalkResult()
    since = "" if force else docket.pull_watermark
    max_lm = docket.pull_watermark if not force else ""
    n = 0

    def _err(_e: Exception) -> None:
        res.clean = False

    scratch = scratch_root / docket.id
    for c in client.iter_comments(docket.fr_object_id, since=since, on_error=_err):
        if limit and n >= limit:
            res.clean = False  # a capped walk is not a complete one
            break
        key, status = upsert_comment_status(
            store, c, cms_id=docket.rule_cms_id, extra_tags=[f"reg-docket:{docket.id}"]
        )
        setattr(res, status, getattr(res, status) + 1)
        if attachments and c.attachment_count and status != "unchanged":
            for att in client.attachments_for(c.id):
                path = client.download_attachment(att.url, scratch / c.id)
                if path:
                    store.attach_file(key, path, title=att.filename)
        lm = (c.raw.get("attributes") or {}).get("lastModifiedDate") or ""
        if lm > max_lm:
            max_lm = lm
        n += 1
        if n % 50 == 0:
            store._con().commit()  # noqa: SLF001
            if echo:
                echo(f"    {n} comments")
    store._con().commit()  # noqa: SLF001
    res.watermark = max_lm

    if res.clean and not force:
        from datetime import datetime, timezone

        now = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
        updated = replace(
            docket, pull_watermark=max_lm, last_pull_at=now, last_pull_new=res.created
        )
        store.docket_upsert(updated)
        if should_seal(updated, today or date.today(), quiet_days):
            store.docket_seal(docket.id, reason="auto", counts=docket_counts(store, docket.id))
            res.sealed = True
    return res


def docket_counts(store: Store, docket_id: str) -> dict[str, int]:
    """``{"comments": n, "enriched": n}`` from SQL only (no filesystem)."""
    con = store._con()  # noqa: SLF001
    like = f"https://www.regulations.gov/comment/{docket_id}-%"
    comments = con.execute("SELECT count(*) FROM items WHERE url LIKE ?", (like,)).fetchone()[0]
    enriched = con.execute(
        """SELECT count(*) FROM items i WHERE i.url LIKE ? AND i.id IN (
             SELECT item_id FROM item_tags WHERE tag_id IN (SELECT id FROM tags WHERE name='enriched:ok'))""",
        (like,),
    ).fetchone()[0]
    return {"comments": comments, "enriched": enriched}

And split upsert_comment so the status is available:

def upsert_comment(store, comment, *, cms_id="", extra_tags=None) -> str:
    """Upsert the comment as a Source item, return bib key."""
    return upsert_comment_status(store, comment, cms_id=cms_id, extra_tags=extra_tags)[0]


def upsert_comment_status(store, comment, *, cms_id="", extra_tags=None) -> tuple[str, str]:
    """Like :func:`upsert_comment`, also returning created/updated/unchanged."""
    ...  # the existing body, ending with:
    return store.upsert_status(item)

Imports to add at the top of the module: from dataclasses import dataclass, field, replace, from datetime import date, from bib.dockets import Docket.

Step 4: Run to verify pass

Run: uv run --no-sync pytest tests/bib/test_walk_docket.py tests/bib -q -p no:cacheprovider Expected: pass.

Step 5: Commit

git add src/bib/regulations_gov.py tests/bib/test_walk_docket.py
git commit -m "feat(bib): discover_docket + walk_docket — watermark, counts, auto-seal (refs #615)"

Task 8: Fetch CLI rewired — sealed skip before any API call

Files:

Modify: src/cli/bib.py:129-298 (fetch_docket_comments, fetch_pfs_comments)
Test: tests/cli/test_bib_fetch_sealed.py; update tests/cli/test_bib_exercise.py::TestFetchDocketComments / TestFetchPfsComments to the new call shapes.

Interfaces:

Consumes: Store.docket_get/docket_for_rule/docket_upsert, discover_docket, walk_docket, bib.dockets.quiet_days.
Produces: stack bib fetch-pfs-comments [--docket ID] [--force] [--attachments] [--per-docket-limit N] [--since] [--until] [--sleep] and stack bib fetch-docket-comments <ID> [--cms-id] [--limit] [--attachments] [--sleep] [--force].
Step 1: Write the failing tests

# tests/cli/test_bib_fetch_sealed.py
"""fetch-pfs-comments / fetch-docket-comments must not call reg.gov for
sealed dockets and must reuse stored document ids for known ones."""

from __future__ import annotations

from unittest.mock import MagicMock, patch

from typer.testing import CliRunner

from bib.dockets import Docket
from bib.item import Rule
from bib.regulations_gov import WalkResult
from bib.store import Store
from cli.bib import app

runner = CliRunner()


def _store() -> Store:
    return Store(":memory:", storage_dir="/tmp/nope")


def _rule_doc():
    doc = MagicMock()
    doc.type = "Proposed Rule"
    doc.publication_date = "2026-07-16"
    doc.dockets = ["CMS-1848-P"]
    doc.html_url = "https://example.com"
    doc.document_number = "2026-1"
    return doc


def _api():
    api = MagicMock()
    api.__enter__ = MagicMock(return_value=api)
    api.__exit__ = MagicMock(return_value=False)
    return api


@patch("bib.regulations_gov.walk_docket", return_value=WalkResult())
@patch("bib.regulations_gov.Client")
@patch("bib.translate.federal_register")
@patch("bib.federalregister.split_docket_ids", return_value=["CMS-1848-P"])
@patch("bib.federalregister.pfs_rules")
@patch("bib.connect")
def test_sealed_docket_skipped_before_any_call(mc_connect, mc_pfs, _split, mc_fr, mc_client, mc_walk):
    s = _store()
    s.docket_upsert(Docket(id="CMS-2026-2377", rule_cms_id="CMS-1848-P", fr_object_id="o", sealed_at="2026-10-20T00:00:00Z", seal_reason="auto"))
    mc_connect.return_value = s
    mc_pfs.return_value = [_rule_doc()]
    api = _api()
    mc_client.return_value = api

    result = runner.invoke(app, ["fetch-pfs-comments"])

    assert result.exit_code == 0, result.output
    assert "sealed" in result.output
    mc_fr.assert_not_called()               # no rule-metadata fetch
    api.resolve_docket.assert_not_called()
    api.find_documents_in_docket.assert_not_called()
    mc_walk.assert_not_called()


@patch("bib.regulations_gov.walk_docket", return_value=WalkResult(created=2))
@patch("bib.regulations_gov.Client")
@patch("bib.translate.federal_register")
@patch("bib.federalregister.split_docket_ids", return_value=["CMS-1848-P"])
@patch("bib.federalregister.pfs_rules")
@patch("bib.connect")
def test_known_open_docket_walks_without_resolve(mc_connect, mc_pfs, _split, mc_fr, mc_client, mc_walk):
    s = _store()
    s.docket_upsert(Docket(id="CMS-2026-2377", rule_cms_id="CMS-1848-P", fr_object_id="o", comment_end_date="2026-09-14"))
    mc_connect.return_value = s
    mc_pfs.return_value = [_rule_doc()]
    mc_fr.return_value = Rule(title="CY2027 PFS NPRM", url="https://www.federalregister.gov/d/2026-1")
    api = _api()
    mc_client.return_value = api

    result = runner.invoke(app, ["fetch-pfs-comments"])

    assert result.exit_code == 0, result.output
    api.resolve_docket.assert_not_called()
    api.find_documents_in_docket.assert_not_called()
    mc_walk.assert_called_once()
    assert mc_walk.call_args.args[2].id == "CMS-2026-2377"
    assert "created=2" in result.output


@patch("bib.regulations_gov.walk_docket", return_value=WalkResult())
@patch("bib.regulations_gov.discover_docket")
@patch("bib.regulations_gov.Client")
@patch("bib.translate.federal_register")
@patch("bib.federalregister.split_docket_ids", return_value=["CMS-1848-P"])
@patch("bib.federalregister.pfs_rules")
@patch("bib.connect")
def test_unknown_docket_is_resolved_once_and_stored(mc_connect, mc_pfs, _split, mc_fr, mc_client, mc_disc, mc_walk):
    s = _store()
    mc_connect.return_value = s
    mc_pfs.return_value = [_rule_doc()]
    mc_fr.return_value = Rule(title="CY2027 PFS NPRM", url="https://www.federalregister.gov/d/2026-1")
    api = _api()
    api.resolve_docket.return_value = "CMS-2026-2377"
    mc_client.return_value = api
    mc_disc.return_value = Docket(id="CMS-2026-2377", rule_cms_id="CMS-1848-P", fr_object_id="o", comment_end_date="2026-09-14")

    result = runner.invoke(app, ["fetch-pfs-comments"])

    assert result.exit_code == 0, result.output
    api.resolve_docket.assert_called_once_with("CMS-1848-P")
    mc_disc.assert_called_once()
    assert s.docket_get("CMS-2026-2377").fr_object_id == "o"


@patch("bib.regulations_gov.walk_docket", return_value=WalkResult())
@patch("bib.regulations_gov.Client")
@patch("bib.translate.federal_register")
@patch("bib.federalregister.split_docket_ids", return_value=["CMS-1848-P"])
@patch("bib.federalregister.pfs_rules")
@patch("bib.connect")
def test_docket_filter_skips_other_known_dockets_without_calls(mc_connect, mc_pfs, _split, mc_fr, mc_client, mc_walk):
    s = _store()
    s.docket_upsert(Docket(id="CMS-2026-2377", rule_cms_id="CMS-1848-P", fr_object_id="o"))
    mc_connect.return_value = s
    mc_pfs.return_value = [_rule_doc()]
    api = _api()
    mc_client.return_value = api

    result = runner.invoke(app, ["fetch-pfs-comments", "--docket", "CMS-2019-0111"])

    assert result.exit_code == 0, result.output
    mc_fr.assert_not_called()
    api.resolve_docket.assert_not_called()
    mc_walk.assert_not_called()


@patch("bib.regulations_gov.walk_docket", return_value=WalkResult())
@patch("bib.regulations_gov.Client")
@patch("bib.translate.federal_register")
@patch("bib.federalregister.split_docket_ids", return_value=["CMS-1848-P"])
@patch("bib.federalregister.pfs_rules")
@patch("bib.connect")
def test_force_walks_sealed_docket(mc_connect, mc_pfs, _split, mc_fr, mc_client, mc_walk):
    s = _store()
    s.docket_upsert(Docket(id="CMS-2026-2377", rule_cms_id="CMS-1848-P", fr_object_id="o", sealed_at="2026-10-20T00:00:00Z"))
    mc_connect.return_value = s
    mc_pfs.return_value = [_rule_doc()]
    mc_fr.return_value = Rule(title="CY2027 PFS NPRM", url="https://www.federalregister.gov/d/2026-1")
    mc_client.return_value = _api()

    result = runner.invoke(app, ["fetch-pfs-comments", "--force"])

    assert result.exit_code == 0, result.output
    mc_walk.assert_called_once()
    assert mc_walk.call_args.kwargs["force"] is True


@patch("bib.regulations_gov.walk_docket", return_value=WalkResult(created=1))
@patch("bib.regulations_gov.discover_docket")
@patch("bib.regulations_gov.Client")
@patch("bib.connect")
def test_fetch_docket_comments_discovers_then_walks(mc_connect, mc_client, mc_disc, mc_walk):
    s = _store()
    mc_connect.return_value = s
    mc_client.return_value = _api()
    mc_disc.return_value = Docket(id="CMS-2026-2377", rule_cms_id="CMS-1848-P", fr_object_id="o", comment_end_date="2026-09-14")

    result = runner.invoke(app, ["fetch-docket-comments", "CMS-2026-2377", "--cms-id", "CMS-1848-P"])

    assert result.exit_code == 0, result.output
    mc_disc.assert_called_once()
    mc_walk.assert_called_once()
    assert s.docket_get("CMS-2026-2377") is not None


@patch("bib.regulations_gov.walk_docket")
@patch("bib.regulations_gov.Client")
@patch("bib.connect")
def test_fetch_docket_comments_sealed_notice(mc_connect, mc_client, mc_walk):
    s = _store()
    s.docket_upsert(Docket(id="CMS-2019-0111", fr_object_id="o", sealed_at="2020-01-01T00:00:00Z"))
    mc_connect.return_value = s
    mc_client.return_value = _api()

    result = runner.invoke(app, ["fetch-docket-comments", "CMS-2019-0111"])

    assert result.exit_code == 0
    assert "sealed" in result.output
    mc_walk.assert_not_called()

Step 2: Run to verify failure

Run: uv run --no-sync pytest tests/cli/test_bib_fetch_sealed.py -q -p no:cacheprovider Expected: FAIL (no --force, walk_docket never called, resolve called).

Step 3: Implement

Replace fetch_docket_comments and fetch_pfs_comments in src/cli/bib.py:

def _walk_and_report(store, api, docket, *, attachments, limit, force, echo) -> None:
    from bib.dockets import quiet_days
    from bib.regulations_gov import walk_docket

    r = walk_docket(
        store, api, docket,
        attachments=attachments, limit=limit, force=force,
        quiet_days=quiet_days(), echo=echo,
    )
    echo(
        f"    {docket.id}: created={r.created} updated={r.updated} "
        f"unchanged={r.unchanged} clean={r.clean}"
        + (" — sealed" if r.sealed else "")
    )


@app.command(name="fetch-docket-comments")
def fetch_docket_comments(
    docket: str = typer.Argument(..., help="reg.gov docket id, e.g. CMS-2026-2377"),
    cms_id: str = typer.Option("", "--cms-id", help="Associate comments with a specific CMS rule tag."),
    limit: int = typer.Option(0, "--limit", "-n", help="Stop after N comments. 0 = all."),
    attachments: bool = typer.Option(False, "--attachments", help="Also download each comment's PDF/DOCX attachments."),
    sleep: float = typer.Option(1.3, "--sleep", help="Seconds between API calls (rate budget)."),
    force: bool = typer.Option(False, "--force", help="Walk from page 1 even if sealed / watermarked. Never unseals."),
) -> None:
    """Walk one docket's comments from its stored watermark; upsert as Source items.

    The first run discovers the docket's commentable FR document (one
    listing call) and stores it; later runs make no discovery calls.
    Sealed dockets are skipped unless --force.
    """
    from bib import connect
    from bib.regulations_gov import Client, discover_docket

    store = connect()
    with Client(sleep=sleep) as api:
        d = store.docket_get(docket)
        if d is None:
            d = discover_docket(api, docket, rule_cms_id=cms_id)
            store.docket_upsert(d)
        elif cms_id and not d.rule_cms_id:
            from dataclasses import replace

            d = replace(d, rule_cms_id=cms_id)
            store.docket_upsert(d)
        if d.sealed and not force:
            typer.echo(f"  {docket}: sealed {d.sealed_at[:10]} ({d.seal_reason}); skipping — use --force to re-walk")
            return
        if not d.fr_object_id:
            typer.echo(f"  {docket}: no commentable FR document found")
            return
        _walk_and_report(store, api, d, attachments=attachments, limit=limit, force=force, echo=typer.echo)


@app.command(name="fetch-pfs-comments")
def fetch_pfs_comments(
    since: str = typer.Option("2017-01-01", "--since"),
    until: str = typer.Option("", "--until"),
    attachments: bool = typer.Option(False, "--attachments"),
    per_docket_limit: int = typer.Option(0, "--per-docket-limit", help="Cap comments fetched per docket. 0 = unlimited."),
    sleep: float = typer.Option(1.3, "--sleep"),
    docket: str = typer.Option("", "--docket", help="Only pull this reg.gov docket (e.g. CMS-2026-2377); other dockets are skipped before any API call once known."),
    force: bool = typer.Option(False, "--force", help="Walk every docket from page 1, sealed or not. Never unseals."),
) -> None:
    """Discover every PFS proposed rule since *since* and pull new comments
    on each docket from its stored watermark.

    Sealed dockets (comment period closed + quiet period + an empty pull)
    cost nothing: no rule-metadata fetch, no resolve call, no walk.
    """
    from bib import connect
    from bib.federalregister import pfs_rules, split_docket_ids
    from bib.regulations_gov import Client, discover_docket
    from bib.tag import Tag
    from bib.translate import federal_register

    store = connect()
    rules = pfs_rules(since=since, until=until or None)
    typer.echo(f"==> {len(rules)} PFS rules since {since}")
    proposed = [r for r in rules if r.type == "Proposed Rule"]
    typer.echo(f"  {len(proposed)} proposed rules (the ones with comments)")

    with Client(sleep=sleep) as api:
        for doc in proposed:
            cms_ids = split_docket_ids(doc.dockets)
            known = {cid: store.docket_for_rule(cid) for cid in cms_ids}

            # Decide what this rule needs BEFORE touching the network.
            def _wanted(cid: str) -> bool:
                d = known[cid]
                if docket and d is not None and d.id != docket:
                    return False
                if d is not None and d.sealed and not force:
                    typer.echo(f"  {d.id}: sealed {d.sealed_at[:10]} ({d.seal_reason}); skipping")
                    return False
                return True

            todo = [cid for cid in cms_ids if _wanted(cid)]
            if not todo:
                continue

            try:
                rule = federal_register(
                    doc.html_url or f"https://www.federalregister.gov/documents/{doc.document_number}"
                )
            except Exception as e:  # noqa: BLE001
                typer.echo(f"  skipped rule meta: {e}")
                continue
            rule.add_tag(Tag.source("federal-register").label)
            rule.add_tag("module:pfs")
            for cid in cms_ids:
                rule.add_tag(f"cms-rule:{cid}")

            for cms_id in todo:
                d = known[cms_id]
                if d is None:
                    reg_docket = api.resolve_docket(cms_id)
                    if not reg_docket:
                        typer.echo(f"  skip {cms_id}: no reg.gov docket found")
                        continue
                    if docket and reg_docket != docket:
                        continue
                    d = discover_docket(api, reg_docket, rule_cms_id=cms_id)
                    store.docket_upsert(d)
                typer.echo(f"  {cms_id} → {d.id} ({doc.publication_date})")
                rule.add_tag(f"reg-docket:{d.id}")
                store.upsert(rule)
                if not d.fr_object_id:
                    typer.echo(f"    {d.id}: no commentable FR document; skipping")
                    continue
                _walk_and_report(
                    store, api, d,
                    attachments=attachments, limit=per_docket_limit, force=force, echo=typer.echo,
                )

Update tests/cli/test_bib_exercise.py: TestFetchDocketComments and TestFetchPfsComments.test_full_loop currently drive api.iter_comments directly through a MagicMock store. Point them at the new shape: patch bib.regulations_gov.walk_docket (return WalkResult()) and bib.regulations_gov.discover_docket, and use a real in-memory Store for bib.connect. Keep test_no_rules, test_skip_bad_rule, test_no_docket (they still pass: resolve_docket returning None prints "skip").

Step 4: Run to verify pass

Run: uv run --no-sync pytest tests/cli/test_bib_fetch_sealed.py tests/cli/test_bib_exercise.py tests/cli/test_bib_deep.py -q -p no:cacheprovider Expected: pass.

Step 5: Commit

git add src/cli/bib.py tests/cli/test_bib_fetch_sealed.py tests/cli/test_bib_exercise.py
git commit -m "feat(cli): fetch commands walk from watermark, skip sealed dockets before any API call (refs #615)"

Task 9: Backfill skips sealed dockets (API + mirror)

Files:

Modify: src/bib/regulations_gov.py (backfill_details, backfill_from_mirror)
Modify: src/cli/bib.py (backfill_comments gains --force)
Test: tests/bib/test_backfill_sealed.py

Interfaces:

Produces: both backfills accept force: bool = False; when docket is sealed and not forced they return {"skipped_sealed": 1, "seen": 0, ...} before any listing or API call.
Step 1: Write the failing tests

# tests/bib/test_backfill_sealed.py
from __future__ import annotations

from unittest.mock import MagicMock

from bib.dockets import Docket
from bib.regulations_gov import backfill_details, backfill_from_mirror
from bib.store import Store

D = "CMS-2019-0111"


def _sealed_store() -> Store:
    s = Store(":memory:", storage_dir="/tmp/nope")
    s.docket_upsert(Docket(id=D, sealed_at="2020-06-01T00:00:00Z", seal_reason="manual"))
    return s


def test_api_backfill_skips_sealed_without_calls():
    s = _sealed_store()
    api = MagicMock()
    stats = backfill_details(s, api, docket=D)
    assert stats["skipped_sealed"] == 1 and stats["seen"] == 0
    api.get_comment_detail.assert_not_called()


def test_mirror_backfill_skips_sealed_without_listing():
    s = _sealed_store()
    m = MagicMock()
    stats = backfill_from_mirror(s, m, docket=D)
    assert stats["skipped_sealed"] == 1 and stats["seen"] == 0
    m.comment_ids.assert_not_called()
    m.attachment_keys.assert_not_called()


def test_force_runs_sealed_mirror_backfill():
    s = _sealed_store()
    m = MagicMock()
    m.comment_ids.return_value = []
    m.attachment_keys.return_value = {}
    stats = backfill_from_mirror(s, m, docket=D, force=True)
    assert "skipped_sealed" not in stats
    m.comment_ids.assert_called_once_with(D)

Step 2: Run to verify failure

Run: uv run --no-sync pytest tests/bib/test_backfill_sealed.py -q -p no:cacheprovider Expected: FAIL — KeyError: 'skipped_sealed' / TypeError on force.

Step 3: Implement

In both functions add force: bool = False to the keyword parameters and, as the first statement after the docstring:

    if docket and not force:
        d = store.docket_get(docket)
        if d is not None and d.sealed:
            log.info("%s sealed %s (%s); backfill skipped", docket, d.sealed_at[:10], d.seal_reason)
            print(f"{docket}: sealed {d.sealed_at[:10]} ({d.seal_reason}); skipped — use --force", flush=True)
            return {"skipped_sealed": 1, "seen": 0, "enriched": 0, "created": 0, "attached": 0, "errors": 0}

In cli/bib.py::backfill_comments add force: bool = typer.Option(False, "--force", help="Run even if the docket is sealed.") and pass force=force to both calls.

Step 4: Run to verify pass

Run: uv run --no-sync pytest tests/bib/test_backfill_sealed.py tests/bib/test_regulations_gov_mirror.py tests/bib/test_regulations_gov_exercise.py tests/cli/test_bib_exercise.py -q -p no:cacheprovider Expected: pass.

Step 5: Commit

git add src/bib/regulations_gov.py src/cli/bib.py tests/bib/test_backfill_sealed.py
git commit -m "feat(bib): backfill-comments skips sealed dockets; --force override (refs #615)"

Task 10: Extract currency by mtime — `source_paths`, `is_current`, walker skip without reads

Files:

Modify: src/rex/comments/combine.py (add source_paths, is_current)
Modify: src/rex/comments/walker.py (walk_and_extract, _collect_dirs, _one)
Test: tests/rex/comments/test_currency.py; update tests/rex/comments/test_walker.py::test_on_extracted_called_for_skipped_dirs

Interfaces:

Produces: combine.source_paths(comment_dir) -> list[Path] (files that feed extraction: not combined.md, not *.md, not *.tmp, not dotfiles); combine.is_current(comment_dir) -> bool (combined.md exists and its mtime ≥ every source file's mtime); walker.walk_and_extract(..., skip_dockets: set[str] = frozenset(), reattach: bool = False). Skipped dirs are never read; on_extracted fires for them only when reattach=True.
Step 1: Write the failing tests

# tests/rex/comments/test_currency.py
from __future__ import annotations

import os
from pathlib import Path

from rex.comments.combine import is_current, source_paths
from rex.comments.walker import walk_and_extract


def _dir(tmp_path: Path, docket="CMS-2024-0001", cid="CMS-2024-0001-0001") -> Path:
    d = tmp_path / docket / cid
    d.mkdir(parents=True)
    return d


def test_source_paths_excludes_outputs(tmp_path: Path):
    d = _dir(tmp_path)
    (d / "attachment_1.pdf").write_bytes(b"x")
    (d / "attachment_1.pdf.md").write_text("sibling")
    (d / "combined.md").write_text("c")
    (d / "combined.md.tmp").write_text("t")
    (d / ".hidden").write_text("h")
    assert [p.name for p in source_paths(d)] == ["attachment_1.pdf"]


def test_is_current_false_without_combined(tmp_path: Path):
    d = _dir(tmp_path)
    (d / "attachment_1.pdf").write_bytes(b"x")
    assert is_current(d) is False


def test_is_current_true_when_combined_newer(tmp_path: Path):
    d = _dir(tmp_path)
    (d / "attachment_1.pdf").write_bytes(b"x")
    os.utime(d / "attachment_1.pdf", ns=(1_000, 1_000))
    (d / "combined.md").write_text("c")
    assert is_current(d) is True


def test_is_current_false_when_source_newer(tmp_path: Path):
    d = _dir(tmp_path)
    (d / "combined.md").write_text("c")
    os.utime(d / "combined.md", ns=(1_000, 1_000))
    (d / "attachment_2.pdf").write_bytes(b"new")
    assert is_current(d) is False


def test_walker_reextracts_stale_dir(tmp_path: Path, monkeypatch):
    d = _dir(tmp_path)
    (d / "combined.md").write_text("---\ncomment_id: x\n---\n\nold\n")
    os.utime(d / "combined.md", ns=(1_000, 1_000))
    (d / "attachment_1.pdf").write_bytes(b"%PDF")  # newer than combined.md
    called = []
    monkeypatch.setattr("rex.comments.walker.extract_comment", lambda cdir, **kw: called.append(cdir) or (cdir / "combined.md"))
    stats = walk_and_extract(tmp_path, inline_body_lookup=lambda _c: "", workers=1)
    assert stats["written"] == 1 and called == [d]


def test_walker_skips_current_dir_without_reading(tmp_path: Path, monkeypatch):
    d = _dir(tmp_path)
    (d / "attachment_1.pdf").write_bytes(b"%PDF")
    os.utime(d / "attachment_1.pdf", ns=(1_000, 1_000))
    (d / "combined.md").write_text("---\ncomment_id: x\n---\n\nbody\n")
    reads = []
    real_read_text = Path.read_text

    def spy(self, *a, **kw):
        if self.name == "combined.md":
            reads.append(self)
        return real_read_text(self, *a, **kw)

    monkeypatch.setattr(Path, "read_text", spy)
    seen = []
    stats = walk_and_extract(tmp_path, inline_body_lookup=lambda _c: "", workers=1, on_extracted=lambda cid, _d: seen.append(cid))
    assert stats == {"written": 0, "skipped": 1, "failed": 0}
    assert reads == []          # skipped dirs are not read
    assert seen == []           # and the callback does not fire without --reattach


def test_walker_reattach_fires_callback_for_skipped(tmp_path: Path):
    d = _dir(tmp_path)
    (d / "attachment_1.pdf").write_bytes(b"%PDF")
    os.utime(d / "attachment_1.pdf", ns=(1_000, 1_000))
    (d / "combined.md").write_text("---\ncomment_id: x\ndocket_id: y\n---\n\n## attachment_1.pdf\n\nbody\n")
    seen = []
    walk_and_extract(tmp_path, inline_body_lookup=lambda _c: "", workers=1, reattach=True, on_extracted=lambda cid, _d: seen.append(cid))
    assert seen == [d.name]
    assert (d / "attachment_1.pdf.md").is_file()  # siblings derived on reattach


def test_walker_skip_dockets(tmp_path: Path):
    d = _dir(tmp_path)
    (d / "attachment_1.pdf").write_bytes(b"%PDF")
    stats = walk_and_extract(tmp_path, inline_body_lookup=lambda _c: "", workers=1, skip_dockets={"CMS-2024-0001"})
    assert stats == {"written": 0, "skipped": 0, "failed": 0, "skipped_sealed": 1}
    assert not (d / "combined.md").exists()

Update tests/rex/comments/test_walker.py::test_on_extracted_called_for_skipped_dirs to pass reattach=True (its docstring becomes "with reattach=True, pre-existing combined.md still gets the callback…"). Also in test_walk_and_extract_skips_existing, after writing the pre-existing combined.md, add os.utime(a / "attachment_1.pdf", ns=(1_000, 1_000)) so the pre-existing combined.md counts as current (import os).

Step 2: Run to verify failure

Run: uv run --no-sync pytest tests/rex/comments/test_currency.py -q -p no:cacheprovider Expected: FAIL — ImportError: cannot import name 'is_current'.

Step 3: Implement

In src/rex/comments/combine.py (public, after parse_combined):

def source_paths(comment_dir: Path) -> list[Path]:
    """Files that feed extraction: attachments and bodies, never our own
    outputs (``combined.md``, ``*.md`` siblings, ``*.tmp``) or dotfiles."""
    out: list[Path] = []
    for p in comment_dir.iterdir():
        n = p.name
        if not p.is_file() or n.startswith(".") or n == _FILENAME:
            continue
        if n.endswith(".md") or n.endswith(".tmp"):
            continue
        out.append(p)
    return sorted(out)


def is_current(comment_dir: Path) -> bool:
    """True when ``combined.md`` exists and is at least as new as every
    source file — a stat-only check, no reads."""
    out = comment_dir / _FILENAME
    try:
        out_m = out.stat().st_mtime_ns
    except FileNotFoundError:
        return False
    for p in source_paths(comment_dir):
        if p.stat().st_mtime_ns > out_m:
            return False
    return True

Make _attachment_paths delegate: return source_paths(comment_dir) (this also stops force re-extraction from feeding *.md siblings back into the extractor).

In src/rex/comments/walker.py:

from rex.comments.combine import derive_siblings_from_combined, extract_comment, is_current


def walk_and_extract(
    root: Path,
    *,
    inline_body_lookup: Callable[[str], str],
    docket: str | None = None,
    limit: int | None = None,
    workers: int = 8,
    force: bool = False,
    on_extracted: Callable[[str, Path], None] | None = None,
    skip_dockets: set[str] | frozenset[str] = frozenset(),
    reattach: bool = False,
) -> dict[str, int]:
    """...(existing docstring, amended:)

    A dir is *current* when its combined.md is newer than every source
    file (``combine.is_current``); current dirs are skipped without being
    read. *on_extracted* fires for newly written dirs; with *reattach*
    it also fires for skipped dirs (after deriving any missing sibling
    MDs), which is the repair path for bib notes. *skip_dockets* names
    docket dirs to ignore entirely (sealed dockets), unless *force*.
    """
    candidate_dirs, skipped_dirs, sealed = _collect_dirs(
        root, docket=docket, limit=limit, force=force, skip_dockets=skip_dockets
    )
    stats = {"written": 0, "skipped": len(skipped_dirs), "failed": 0}
    if sealed:
        stats["skipped_sealed"] = sealed
    ...  # pool section unchanged
    if on_extracted and reattach:
        for cdir in skipped_dirs:
            ...  # unchanged body
    return stats


def _collect_dirs(root, *, docket, limit, force, skip_dockets=frozenset()):
    todo, skipped, sealed = [], [], 0
    for docket_dir in sorted(root.iterdir()):
        if not docket_dir.is_dir():
            continue
        if docket and docket_dir.name != docket:
            continue
        if docket_dir.name in skip_dockets and not force:
            sealed += 1
            continue
        for cdir in sorted(docket_dir.iterdir()):
            if not cdir.is_dir():
                continue
            if not force and is_current(cdir):
                skipped.append(cdir)
                continue
            todo.append(cdir)
            if limit and len(todo) >= limit:
                return todo, skipped, sealed
    return todo, skipped, sealed


def _one(comment_dir, inline_body_lookup, force):
    if not force and is_current(comment_dir):
        return "skipped"
    ...  # unchanged; but call extract_comment(comment_dir, inline_body=body, force=True)

_one must pass force=True to extract_comment, because the walker has already decided the dir is stale and extract_comment's own "exists → return" guard would otherwise refuse to rewrite it.

Step 4: Run to verify pass

Run: uv run --no-sync pytest tests/rex/comments -q -p no:cacheprovider Expected: pass.

Step 5: Commit

git add src/rex/comments/combine.py src/rex/comments/walker.py tests/rex/comments/test_currency.py tests/rex/comments/test_walker.py
git commit -m "feat(comments): mtime-based extract currency; skipped dirs are never read; skip_dockets + reattach (refs #615)"

Task 11: `stack comments` — extract with sealed skip + lazy body lookup; `dockets` / `seal` / `unseal`

Files:

Modify: src/cli/comments.py
Test: tests/cli/test_comments_dockets.py; extend tests/cli/test_comments.py

Interfaces:

Consumes: Store.dockets/docket_get/docket_upsert/docket_seal/docket_unseal/sealed_dockets, walk_and_extract(skip_dockets=, reattach=), discover_docket, docket_counts, _rule_tag_for_docket.
Produces: stack comments extract [--reattach] [--force] (sealed dockets skipped, note printed); stack comments dockets [--discover]; stack comments seal <ID> [--reason TEXT]; stack comments unseal <ID>.
Step 1: Write the failing tests

# tests/cli/test_comments_dockets.py
from __future__ import annotations

from pathlib import Path
from unittest.mock import MagicMock, patch

import fitz
from typer.testing import CliRunner

from bib.dockets import Docket
from bib.store import Store
from cli.comments import app

runner = CliRunner()


def _store() -> Store:
    return Store(":memory:", storage_dir="/tmp/nope")


def _seed_dir(root: Path, docket: str) -> Path:
    cdir = root / docket / f"{docket}-0001"
    cdir.mkdir(parents=True)
    doc = fitz.open()
    doc.new_page().insert_text((50, 72), "Long body content " * 10)
    doc.save(str(cdir / "attachment_1.pdf"))
    doc.close()
    return cdir


@patch("bib.connect")
def test_dockets_table_lists_rows(mc_connect):
    s = _store()
    s.docket_upsert(Docket(id="CMS-2019-0111", rule_cms_id="CMS-1693-P", comment_end_date="2019-09-27", sealed_at="2020-01-01T00:00:00Z", seal_reason="manual"))
    s.docket_upsert(Docket(id="CMS-2026-2377", rule_cms_id="CMS-1848-P", comment_end_date="2026-09-14"))
    mc_connect.return_value = s
    result = runner.invoke(app, ["dockets"])
    assert result.exit_code == 0, result.output
    assert "CMS-2019-0111" in result.output and "sealed" in result.output
    assert "CMS-2026-2377" in result.output and "open" in result.output


@patch("bib.connect")
def test_seal_and_unseal(mc_connect):
    s = _store()
    s.docket_upsert(Docket(id="CMS-2019-0111"))
    mc_connect.return_value = s
    assert runner.invoke(app, ["seal", "CMS-2019-0111", "--reason", "historical"]).exit_code == 0
    assert s.docket_get("CMS-2019-0111").seal_reason == "historical"
    assert runner.invoke(app, ["unseal", "CMS-2019-0111"]).exit_code == 0
    assert not s.docket_get("CMS-2019-0111").sealed


@patch("bib.connect")
def test_seal_unknown_docket_fails(mc_connect):
    mc_connect.return_value = _store()
    result = runner.invoke(app, ["seal", "CMS-0000-0000"])
    assert result.exit_code == 1
    assert "unknown docket" in result.output


@patch("bib.regulations_gov.discover_docket")
@patch("bib.regulations_gov.Client")
@patch("bib.connect")
def test_dockets_discover_populates_from_tags(mc_connect, mc_client, mc_disc):
    from bib.item import Source

    s = _store()
    it = Source(title="c", url="https://www.regulations.gov/comment/CMS-2019-0111-1")
    it.add_tag("reg-docket:CMS-2019-0111")
    it.add_tag("rule:CMS-1693-P")
    s.create(it)
    mc_connect.return_value = s
    api = MagicMock(); api.__enter__ = MagicMock(return_value=api); api.__exit__ = MagicMock(return_value=False)
    mc_client.return_value = api
    mc_disc.return_value = Docket(id="CMS-2019-0111", rule_cms_id="CMS-1693-P", fr_object_id="o", comment_end_date="2019-09-27")

    result = runner.invoke(app, ["dockets", "--discover"])

    assert result.exit_code == 0, result.output
    mc_disc.assert_called_once_with(api, "CMS-2019-0111", rule_cms_id="CMS-1693-P")
    assert s.docket_get("CMS-2019-0111").comment_end_date == "2019-09-27"


@patch("bib.connect")
def test_extract_skips_sealed_docket(mc_connect, tmp_path: Path):
    s = _store()
    s.docket_upsert(Docket(id="CMS-2024-0001", sealed_at="2025-01-01T00:00:00Z", seal_reason="manual"))
    mc_connect.return_value = s
    cdir = _seed_dir(tmp_path, "CMS-2024-0001")
    result = runner.invoke(app, ["extract", "--root", str(tmp_path), "--workers", "1", "--no-attach"])
    assert result.exit_code == 0, result.output
    assert "skipped_sealed: 1" in result.output
    assert not (cdir / "combined.md").exists()


@patch("bib.connect")
def test_extract_force_runs_sealed_docket(mc_connect, tmp_path: Path):
    s = _store()
    s.docket_upsert(Docket(id="CMS-2024-0001", sealed_at="2025-01-01T00:00:00Z"))
    mc_connect.return_value = s
    cdir = _seed_dir(tmp_path, "CMS-2024-0001")
    result = runner.invoke(app, ["extract", "--root", str(tmp_path), "--workers", "1", "--no-attach", "--force"])
    assert result.exit_code == 0, result.output
    assert (cdir / "combined.md").is_file()

Step 2: Run to verify failure

Run: uv run --no-sync pytest tests/cli/test_comments_dockets.py -q -p no:cacheprovider Expected: FAIL — No such command 'dockets'.

Step 3: Implement

Rewrite _build_bib_helpers to look bodies up lazily (the URL index from Task 2 makes per-call lookups cheap) and return the store so extract can read seals:

def _build_bib_helpers(use_bib: bool, attach: bool):
    """Return ``(body_lookup, attach_callback, store_or_None)``.

    ``body_lookup`` queries bib per comment id (``items.url`` is indexed)
    under a lock — worker threads share one connection. Nothing is
    loaded up front, so a run that extracts nothing reads nothing.
    """
    if not use_bib:
        return (lambda _cid: ""), None, None

    import threading

    from bib import connect

    store = connect()
    con = store._con()  # noqa: SLF001
    lock = threading.Lock()

    def _row(comment_id: str):
        with lock:
            return con.execute(
                "SELECT key, abstract FROM items WHERE url = ?",
                (f"https://www.regulations.gov/comment/{comment_id}",),
            ).fetchone()

    def body_lookup(comment_id: str) -> str:
        row = _row(comment_id)
        return (row["abstract"] or "") if row else ""

    if not attach:
        return body_lookup, None, store

    from rex.comments.render import render_combined_md

    items_with_note: set[str] = {
        row[0]
        for row in con.execute(
            "SELECT i.key FROM notes n JOIN items i ON n.item_id = i.id WHERE n.title = ?",
            (NOTE_TITLE,),
        )
    }

    def attach_callback(comment_id: str, comment_dir: Path) -> None:
        row = _row(comment_id)
        if not row:
            return
        item_key = row["key"]
        if item_key in items_with_note:
            return
        combined = comment_dir / "combined.md"
        if not combined.is_file():
            return
        try:
            html = render_combined_md(combined.read_text(encoding="utf-8"))
            with lock:
                store.attach_note(item_key, html, title=NOTE_TITLE)
            items_with_note.add(item_key)
        except Exception as e:  # noqa: BLE001
            log.warning("attach_note failed for %s: %s", item_key, e)

    return body_lookup, attach_callback, store

Note: Store._con opens sqlite with the default check_same_thread=True, so worker threads calling body_lookup would raise. For file-backed stores open a second, thread-tolerant read connection for the lookups:

    if store._db_path == ":memory:":
        con = store._con()  # tests only — a fresh connection would be an empty DB
    else:
        con = sqlite3.connect(store._db_path, check_same_thread=False)
        con.row_factory = sqlite3.Row

(import sqlite3; attach_note still goes through store, which is main-thread only.) Keep _bib_lookup_factory returning body from the new triple. In the in-memory case a worker-thread lookup raises ProgrammingError, which _one already catches and logs (body falls back to ""), so the extract tests keep passing.

extract command: add reattach: bool = typer.Option(False, "--reattach", help="Also attach combined.md notes for dirs that were already current (repair)."); then:

    lookup, attach_cb, store = _build_bib_helpers(use_bib, attach=use_bib and attach)
    skip = set(store.sealed_dockets()) if (store is not None and not force) else set()
    stats = walk_and_extract(
        root, inline_body_lookup=lookup, docket=docket, limit=limit or None,
        workers=workers, force=force, on_extracted=attach_cb,
        skip_dockets=skip, reattach=reattach,
    )
    if docket and docket in skip:
        typer.echo(f"  {docket}: sealed; skipped — use --force")
    for k, v in stats.items():
        typer.echo(f"  {k}: {v}")

New commands:

@app.command()
def dockets(
    discover: bool = typer.Option(False, "--discover", help="Populate rows for every reg-docket: tag in bib that has no dockets row (one reg.gov listing call each)."),
) -> None:
    """List dockets: id, rule, close date, watermark, counts, seal."""
    from bib import connect
    from bib.regulations_gov import docket_counts

    store = connect()
    if discover:
        from bib.regulations_gov import Client, _rule_tag_for_docket, discover_docket

        tagged = [
            r[0].split(":", 1)[1]
            for r in store._con().execute("SELECT name FROM tags WHERE name LIKE 'reg-docket:%' ORDER BY name")  # noqa: SLF001
        ]
        missing = [d for d in tagged if store.docket_get(d) is None]
        with Client(sleep=1.3) as api:
            for d in missing:
                row = discover_docket(api, d, rule_cms_id=_rule_tag_for_docket(store, d))
                store.docket_upsert(row)
                typer.echo(f"  discovered {d}: closes {row.comment_end_date or '?'} rule {row.rule_cms_id or '?'}")
    rows = store.dockets()
    if not rows:
        typer.echo("no dockets recorded — run `stack comments dockets --discover`")
        return
    typer.echo(f"{'docket':<16}{'rule':<12}{'closes':<12}{'comments':>9}{'enriched':>9}  {'watermark':<20} state")
    for d in rows:
        c = docket_counts(store, d.id)
        state = f"sealed {d.sealed_at[:10]} ({d.seal_reason})" if d.sealed else "open"
        typer.echo(
            f"{d.id:<16}{d.rule_cms_id:<12}{d.comment_end_date:<12}"
            f"{c['comments']:>9}{c['enriched']:>9}  {d.pull_watermark[:19]:<20} {state}"
        )


@app.command()
def seal(
    docket: str = typer.Argument(..., help="reg.gov docket id"),
    reason: str = typer.Option("manual", "--reason"),
) -> None:
    """Mark a docket complete: every stage skips it until unsealed or --force."""
    from bib import connect
    from bib.regulations_gov import docket_counts

    store = connect()
    if store.docket_get(docket) is None:
        typer.echo(f"unknown docket {docket} — run `stack comments dockets --discover` first")
        raise typer.Exit(1)
    store.docket_seal(docket, reason=reason, counts=docket_counts(store, docket))
    typer.echo(f"sealed {docket} ({reason})")


@app.command()
def unseal(docket: str = typer.Argument(..., help="reg.gov docket id")) -> None:
    """Reopen a sealed docket."""
    from bib import connect

    store = connect()
    if store.docket_get(docket) is None:
        typer.echo(f"unknown docket {docket}")
        raise typer.Exit(1)
    store.docket_unseal(docket)
    typer.echo(f"unsealed {docket}")

Update the module docstring's command list.

Step 4: Run to verify pass

Run: uv run --no-sync pytest tests/cli/test_comments_dockets.py tests/cli/test_comments.py -q -p no:cacheprovider Expected: pass.

Step 5: Commit

git add src/cli/comments.py tests/cli/test_comments_dockets.py tests/cli/test_comments.py
git commit -m "feat(cli): stack comments dockets/seal/unseal; extract skips sealed dockets, lazy bib lookup, --reattach (refs #615)"

Task 12: llm migrations — `index_state.fingerprint`, `index_docket_state`

Files:

Modify: src/llm/migrate.py
Test: tests/llm/test_migrate.py

Interfaces:

Produces: INDEX_STATE_DDL gains fingerprint TEXT NOT NULL DEFAULT ''; INDEX_STATE_ALTER = "ALTER TABLE index_state ADD COLUMN IF NOT EXISTS fingerprint TEXT NOT NULL DEFAULT ''"; INDEX_DOCKET_STATE_DDL; migrate(engine) runs all three.
Step 1: Write the failing tests (append to tests/llm/test_migrate.py::TestDdl)

    def test_fingerprint_column_and_docket_state(self):
        assert "fingerprint" in migrate.INDEX_STATE_DDL
        assert "ADD COLUMN IF NOT EXISTS fingerprint" in migrate.INDEX_STATE_ALTER
        assert "CREATE TABLE IF NOT EXISTS index_docket_state" in migrate.INDEX_DOCKET_STATE_DDL
        assert "PRIMARY KEY (collection, docket)" in migrate.INDEX_DOCKET_STATE_DDL

    def test_migrate_runs_alter_and_docket_state(self):
        engine = MagicMock()
        conn = engine.begin.return_value.__enter__.return_value
        migrate.migrate(engine)
        executed = " ".join(str(call.args[0]) for call in conn.execute.call_args_list)
        assert "ADD COLUMN IF NOT EXISTS fingerprint" in executed
        assert "index_docket_state" in executed

Step 2: Run to verify failure

Run: uv run --no-sync pytest tests/llm/test_migrate.py -q -p no:cacheprovider Expected: FAIL — AttributeError: module 'llm.migrate' has no attribute 'INDEX_STATE_ALTER'.

Step 3: Implement

INDEX_STATE_DDL = """
CREATE TABLE IF NOT EXISTS index_state (
    item_key     TEXT        NOT NULL,
    collection   TEXT        NOT NULL,
    content_hash TEXT        NOT NULL,
    chunk_count  INTEGER     NOT NULL DEFAULT 0,
    indexed_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
    fingerprint  TEXT        NOT NULL DEFAULT '',
    PRIMARY KEY (item_key, collection)
)
"""

# Existing databases predate the column.
INDEX_STATE_ALTER = (
    "ALTER TABLE index_state ADD COLUMN IF NOT EXISTS fingerprint TEXT NOT NULL DEFAULT ''"
)

# A sealed docket that was fully indexed under a given seal: the iterator
# does not even list its items until the seal changes (unseal/reseal
# writes a new sealed_at in bib, which no longer matches).
INDEX_DOCKET_STATE_DDL = """
CREATE TABLE IF NOT EXISTS index_docket_state (
    collection TEXT        NOT NULL,
    docket     TEXT        NOT NULL,
    sealed_at  TEXT        NOT NULL,
    indexed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (collection, docket)
)
"""


def migrate(engine: Engine) -> None:
    """Create llm-owned tables/columns. Safe to run on every start."""
    with engine.begin() as conn:
        conn.execute(text(INDEX_STATE_DDL))
        conn.execute(text(INDEX_STATE_ALTER))
        conn.execute(text(INDEX_DOCKET_STATE_DDL))

Step 4: Run to verify pass — uv run --no-sync pytest tests/llm/test_migrate.py -q -p no:cacheprovider → pass.
Step 5: Commit

git add src/llm/migrate.py tests/llm/test_migrate.py
git commit -m "feat(llm): index_state.fingerprint + index_docket_state migrations (refs #615)"

Task 13: `DocRef` + lazy iterators + lazy Zotero snapshot

Files:

Modify: src/llm/source.py
Test: tests/llm/test_source_refs.py; extend tests/llm/test_source.py::TestCommentDocs only if a helper signature changes (it should not — iter_comment_docs/iter_corpus_docs/iter_rule_docs stay and become ref.load() loops).

Interfaces:

Consumes: bib.dockets.fingerprint_files, Store.sealed_dockets.
Produces:
- DocRef(key, collection, docket, fingerprint, load: Callable[[], Doc | None]) (frozen dataclass; load() returns None when the item has no text).
- iter_comment_refs(store, *, docket="", root=None, skip_dockets=frozenset()) -> Iterator[DocRef] — one SQL query for all rows (key, url, date, title, updated_at, year via correlated subquery), newest first; fingerprint = fingerprint_files(combined.md + attachment files) + "|" + updated_at.
- iter_rule_refs(store, *, keys=(), tag="") -> Iterator[DocRef] — fingerprint = fr_anchor_docs.sha256 when present else fingerprint_files(attachment paths) + "|" + updated_at.
- iter_corpus_refs(store, *, tag="", zotero=None) -> Iterator[DocRef] — keys selected in SQL excluding doctype:comment; fingerprint = updated_at + "|" + fingerprint_files(bib attachment paths).
- ZoteroPdfIndex.lazy(sqlite_path, storage_dir, tmp_dir) -> ZoteroPdfIndex — no copy until the first pdfs_for(); copies only when the source mtime is newer than an existing snapshot.
- iter_comment_docs / iter_rule_docs / iter_corpus_docs keep their signatures and yield ref.load() for each ref (skipping None).
Step 1: Write the failing tests

# tests/llm/test_source_refs.py
"""llm.source — lazy DocRefs: fingerprints without loads, docket skips,
lazy Zotero snapshot."""

from __future__ import annotations

import os
from pathlib import Path

import pytest

from bib.item import Item, Rule
from bib.store import Store
from llm.source import DocRef, ZoteroPdfIndex, iter_comment_refs, iter_corpus_refs, iter_rule_refs

DOCKET = "CMS-2019-0111"
CID = f"{DOCKET}-0042"
COMBINED = f"---\ncomment_id: {CID}\ndocket_id: {DOCKET}\n---\n\nWe object.\n"


@pytest.fixture
def store(tmp_path):
    s = Store(":memory:", storage_dir=tmp_path / "storage")
    key = s.create(Item(item_type="report", title="A comment", url=f"https://www.regulations.gov/comment/{CID}", abstract="Inline.", date_published="2019-09-27"))
    for tag in ("doctype:comment", "year:2019", f"reg-docket:{DOCKET}"):
        s.add_tag(key, tag)
    s._comment_key = key
    return s


@pytest.fixture
def root(tmp_path):
    d = tmp_path / DOCKET / CID
    d.mkdir(parents=True)
    (d / "combined.md").write_text(COMBINED)
    return tmp_path


class TestCommentRefs:
    def test_ref_has_fingerprint_and_lazy_load(self, store, root, monkeypatch):
        reads = []
        real = Path.read_text
        monkeypatch.setattr(Path, "read_text", lambda self, *a, **k: reads.append(self) or real(self, *a, **k))
        refs = list(iter_comment_refs(store, docket=DOCKET, root=root))
        assert len(refs) == 1
        r = refs[0]
        assert isinstance(r, DocRef)
        assert (r.key, r.collection, r.docket) == (store._comment_key, "comments", DOCKET)
        assert len(r.fingerprint) > 64 and "|" in r.fingerprint
        assert reads == []  # nothing read yet
        doc = r.load()
        assert "We object" in doc.text
        assert doc.metadata["year"] == "2019" and doc.metadata["docket"] == DOCKET
        assert reads  # load() read combined.md

    def test_fingerprint_changes_with_new_attachment(self, store, root):
        f1 = next(iter_comment_refs(store, docket=DOCKET, root=root)).fingerprint
        (root / DOCKET / CID / "attachment_1.pdf").write_bytes(b"%PDF")
        f2 = next(iter_comment_refs(store, docket=DOCKET, root=root)).fingerprint
        assert f1 != f2

    def test_fingerprint_changes_with_updated_at(self, store, root):
        f1 = next(iter_comment_refs(store, docket=DOCKET, root=root)).fingerprint
        store._con().execute("UPDATE items SET updated_at='2030-01-01T00:00:00Z' WHERE key=?", (store._comment_key,))
        f2 = next(iter_comment_refs(store, docket=DOCKET, root=root)).fingerprint
        assert f1 != f2

    def test_skip_dockets_excludes_rows(self, store, root):
        assert list(iter_comment_refs(store, root=root, skip_dockets={DOCKET})) == []

    def test_unextracted_load_falls_back_to_abstract(self, store, tmp_path):
        r = next(iter_comment_refs(store, docket=DOCKET, root=tmp_path))
        assert r.load().text == "Inline."

    def test_empty_comment_load_returns_none(self, store, tmp_path):
        store._con().execute("UPDATE items SET abstract='' WHERE key=?", (store._comment_key,))
        r = next(iter_comment_refs(store, docket=DOCKET, root=tmp_path))
        assert r.load() is None

    def test_single_query_for_year(self, store, root, monkeypatch):
        """No per-comment year lookups: exactly one SELECT on items for the listing."""
        calls: list[str] = []
        store._con().set_trace_callback(calls.append)
        list(iter_comment_refs(store, docket=DOCKET, root=root))
        store._con().set_trace_callback(None)
        selects = [c for c in calls if c.lstrip().upper().startswith("SELECT")]
        assert len(selects) == 1


class TestRuleRefs:
    def test_fingerprint_from_anchor_sha(self, store):
        key = store.create(Rule(title="R", url="https://fr/1", document_number="2019-1"))
        store._con().execute(
            "INSERT INTO fr_anchor_docs (item_key, document_number, html_url, start_page, end_page, fr_volume, sha256) VALUES (?,?,?,?,?,?,?)",
            (key, "2019-1", "https://fr/1", 1, 2, 84, "abc"),
        )
        store._con().execute("INSERT INTO fr_anchors (item_key, p_id, page, ordinal, text) VALUES (?,?,?,?,?)", (key, 1, 1, 1, "Para one."))
        refs = list(iter_rule_refs(store))
        assert [r.key for r in refs] == [key]
        assert refs[0].fingerprint == "anchors:abc"
        assert refs[0].collection == "rules" and refs[0].docket is None
        assert refs[0].load().text == "Para one."


class TestCorpusRefs:
    def test_excludes_comments_and_is_lazy(self, store):
        key = store.create(Item(item_type="report", title="Report", url="https://x/r", abstract="Body."))
        refs = list(iter_corpus_refs(store))
        assert [r.key for r in refs] == [key]
        assert refs[0].collection == "corpus"
        assert refs[0].load().metadata["kind"] == "corpus"

    def test_zotero_not_consulted_until_load(self, store, tmp_path):
        store.create(Item(item_type="report", title="Report", url="https://x/r", abstract="Body."))
        calls = []

        class Z(ZoteroPdfIndex):
            def pdfs_for(self, key):
                calls.append(key)
                return []

        refs = list(iter_corpus_refs(store, zotero=Z({})))
        assert calls == []
        refs[0].load()
        assert calls


class TestLazyZotero:
    def test_no_copy_until_first_lookup(self, tmp_path):
        src = tmp_path / "zotero.sqlite"
        import sqlite3

        con = sqlite3.connect(src)
        con.executescript(
            "CREATE TABLE items(itemID INTEGER, key TEXT); CREATE TABLE itemAttachments(itemID INTEGER, parentItemID INTEGER, path TEXT); CREATE TABLE deletedItems(itemID INTEGER);"
        )
        con.close()
        snap_dir = tmp_path / "snap"
        z = ZoteroPdfIndex.lazy(src, tmp_path / "storage", snap_dir)
        assert not (snap_dir / "zotero.sqlite").exists()
        assert z.pdfs_for("ABCD1234") == []
        assert (snap_dir / "zotero.sqlite").exists()
        m1 = (snap_dir / "zotero.sqlite").stat().st_mtime_ns
        z2 = ZoteroPdfIndex.lazy(src, tmp_path / "storage", snap_dir)
        z2.pdfs_for("ABCD1234")
        assert (snap_dir / "zotero.sqlite").stat().st_mtime_ns == m1  # source unchanged → no recopy
        bump = src.stat().st_mtime_ns + 10**9
        os.utime(src, ns=(bump, bump))  # source now newer than the snapshot
        z3 = ZoteroPdfIndex.lazy(src, tmp_path / "storage", snap_dir)
        z3.pdfs_for("ABCD1234")
        assert (snap_dir / "zotero.sqlite").stat().st_mtime_ns != m1

Step 2: Run to verify failure

Run: uv run --no-sync pytest tests/llm/test_source_refs.py -q -p no:cacheprovider Expected: FAIL — ImportError: cannot import name 'DocRef'.

Step 3: Implement

Add to src/llm/source.py:

from dataclasses import dataclass
from typing import Callable

from bib.dockets import fingerprint_files


@dataclass(frozen=True)
class DocRef:
    """A document the indexer *may* need: enough to decide (key +
    fingerprint) without building it. ``load()`` does the expensive part
    and returns ``None`` when the item has no text."""

    key: str
    collection: str
    docket: str | None
    fingerprint: str
    load: Callable[[], "Doc | None"]


# ── comments ──


def _comment_listing(store: Store, docket: str = "") -> list[sqlite3.Row]:
    """One query: key, comment_id, date, title, updated_at, year — newest first."""
    pattern = f"{_COMMENT_URL_PREFIX}{docket}-%" if docket else f"{_COMMENT_URL_PREFIX}%"
    return (
        store._con()
        .execute(
            "SELECT i.key, i.url, COALESCE(i.date_published,'') AS date, "
            "COALESCE(i.title,'') AS title, COALESCE(i.updated_at,'') AS updated_at, "
            "COALESCE((SELECT t.name FROM tags t JOIN item_tags it ON it.tag_id = t.id "
            "  WHERE it.item_id = i.id AND t.name LIKE 'year:%' LIMIT 1), '') AS year "
            "FROM items i WHERE i.url LIKE ? "
            "ORDER BY i.date_published DESC, i.key",
            (pattern,),
        )
        .fetchall()
    )


def iter_comment_refs(
    store: Store,
    *,
    docket: str = "",
    root: Path | None = None,
    skip_dockets: set[str] | frozenset[str] = frozenset(),
) -> Iterator[DocRef]:
    """One DocRef per comment, newest first. Fingerprint = file stats of
    combined.md + attachments + the item's updated_at; no file is read."""
    from rex.comments.combine import parse_combined

    root = root if root is not None else _default_root()
    for row in _comment_listing(store, docket):
        comment_id = row["url"].rsplit("/", 1)[-1]
        dk = docket or comment_id.rsplit("-", 1)[0]
        if dk in skip_dockets:
            continue
        key, date, title, updated_at, year = row["key"], row["date"], row["title"], row["updated_at"], row["year"].split(":", 1)[-1]
        comment_dir = root / dk / comment_id
        combined = comment_dir / "combined.md"
        files = _comment_files(comment_dir)
        fp = fingerprint_files([combined, *(Path(p) for _, p in files)]) + "|" + updated_at
        meta = {"docket": dk, "comment_id": comment_id, "doctype": "comment", "kind": "comment", "year": year, "date": date[:10], "title": title}

        def _load(key=key, meta=meta, combined=combined, files=files) -> Doc | None:
            if combined.exists():
                _, body = parse_combined(combined.read_text())
                if body.strip():
                    return Doc(key=key, text=body, metadata=meta, files=files)
            row = store._con().execute("SELECT abstract FROM items WHERE key = ?", (key,)).fetchone()
            abstract = (row["abstract"] if row else "") or ""
            return Doc(key=key, text=abstract, metadata=meta) if abstract.strip() else None

        yield DocRef(key=key, collection="comments", docket=dk, fingerprint=fp, load=_load)


def iter_comment_docs(store: Store, *, docket: str = "", root: Path | None = None) -> Iterator[Doc]:
    """One Doc per comment, newest first: extraction body, else abstract."""
    for ref in iter_comment_refs(store, docket=docket, root=root):
        doc = ref.load()
        if doc is not None:
            yield doc

comment_key_map becomes {cid: (key, year)} over _comment_listing (drop _year_of use there; keep _year_of for the rule/corpus loaders).

Rules:

def _attachment_paths(store: Store, item_key: str) -> list[Path]:
    rows = store._con().execute(
        "SELECT a.storage_path FROM attachments a JOIN items i ON i.id = a.item_id WHERE i.key = ?",
        (item_key,),
    ).fetchall()
    return [Path(r[0]) for r in rows]


def iter_rule_refs(store: Store, *, keys: tuple[str, ...] = (), tag: str = "") -> Iterator[DocRef]:
    rows = store._con().execute(
        "SELECT i.key, COALESCE(i.updated_at,'') AS updated_at, "
        "(SELECT d.sha256 FROM fr_anchor_docs d WHERE d.item_key = i.key) AS anchor_sha "
        "FROM items i WHERE i.item_type = 'rule'"
        + (" AND i.id IN (SELECT item_id FROM item_tags WHERE tag_id IN (SELECT id FROM tags WHERE name = ?))" if tag else "")
        + " ORDER BY i.id",
        (tag,) if tag else (),
    ).fetchall()
    for row in rows:
        key = row["key"]
        if keys and key not in keys:
            continue
        if row["anchor_sha"]:
            fp = f"anchors:{row['anchor_sha']}"
        else:
            fp = fingerprint_files(_attachment_paths(store, key)) + "|" + row["updated_at"]

        def _load(key=key) -> Doc | None:
            return _build_rule_doc(store, store.get(key))

        yield DocRef(key=key, collection="rules", docket=None, fingerprint=fp, load=_load)


def _build_rule_doc(store: Store, item) -> Doc | None:
    ...  # the current body of iter_rule_docs's loop for one item; return None when text is empty


def iter_rule_docs(store, *, keys=(), tag="") -> Iterator[Doc]:
    for ref in iter_rule_refs(store, keys=keys, tag=tag):
        doc = ref.load()
        if doc is not None:
            yield doc

Corpus:

def iter_corpus_refs(store: Store, *, tag: str = "", zotero: "ZoteroPdfIndex | None" = None) -> Iterator[DocRef]:
    sql = (
        "SELECT i.key, COALESCE(i.updated_at,'') AS updated_at FROM items i "
        "WHERE i.id NOT IN (SELECT item_id FROM item_tags WHERE tag_id IN (SELECT id FROM tags WHERE name = 'doctype:comment'))"
        + (" AND i.id IN (SELECT item_id FROM item_tags WHERE tag_id IN (SELECT id FROM tags WHERE name = ?))" if tag else "")
        + " ORDER BY i.id"
    )
    for row in store._con().execute(sql, (tag,) if tag else ()).fetchall():
        key = row["key"]
        fp = row["updated_at"] + "|" + fingerprint_files(_attachment_paths(store, key))

        def _load(key=key) -> Doc | None:
            return _build_corpus_doc(store, store.get(key), zotero)

        yield DocRef(key=key, collection="corpus", docket=None, fingerprint=fp, load=_load)


def _build_corpus_doc(store, item, zotero) -> Doc | None:
    ...  # the current body of iter_corpus_docs's loop for one item; return None when text is empty


def iter_corpus_docs(store, *, tag="", zotero=None) -> Iterator[Doc]:
    for ref in iter_corpus_refs(store, tag=tag, zotero=zotero):
        doc = ref.load()
        if doc is not None:
            yield doc

Lazy Zotero:

class ZoteroPdfIndex:
    def __init__(self, by_key: dict[str, list[Path]]) -> None:
        self._by_key = by_key
        self._loader: Callable[[], dict[str, list[Path]]] | None = None

    @classmethod
    def lazy(cls, sqlite_path: Path, storage_dir: Path, tmp_dir: Path) -> "ZoteroPdfIndex":
        """Snapshot on first ``pdfs_for``; skip the copy when the existing
        snapshot is already as new as the source."""
        inst = cls({})

        def _load() -> dict[str, list[Path]]:
            snap = Path(tmp_dir) / "zotero.sqlite"
            src = Path(sqlite_path)
            if snap.exists() and src.exists() and snap.stat().st_mtime_ns >= src.stat().st_mtime_ns:
                return cls._read(snap, storage_dir)
            return cls.snapshot(src, storage_dir, Path(tmp_dir))._by_key

        inst._loader = _load
        return inst

    @classmethod
    def snapshot(cls, sqlite_path, storage_dir, tmp_dir):
        ...  # unchanged, except the SELECT moves into cls._read(snap, storage_dir)

    @classmethod
    def _read(cls, snap: Path, storage_dir: Path) -> dict[str, list[Path]]:
        ...  # the SELECT + by_key build from the current snapshot(), returning the dict; {} on sqlite3.Error

    def pdfs_for(self, key: str) -> list[Path]:
        if self._loader is not None:
            self._by_key = self._loader()
            self._loader = None
        return list(self._by_key.get(key, []))

snapshot() must os.utime(snap) after copying is not needed: copy2 preserves the source mtime, so snap.mtime >= src.mtime holds until the source changes.

Step 4: Run to verify pass

Run: uv run --no-sync pytest tests/llm/test_source_refs.py tests/llm/test_source.py -q -p no:cacheprovider Expected: pass (existing TestCommentDocs, TestCorpusDocs, TestRuleDocs still green through the wrappers).

Step 5: Commit

git add src/llm/source.py tests/llm/test_source_refs.py
git commit -m "feat(llm): lazy DocRefs with fingerprints, single-query listings, lazy Zotero snapshot (refs #615)"

Task 14: `index_refs` — fingerprint-first loop, PDFs only on the embed path, docket completion

Files:

Modify: src/llm/index.py
Test: tests/llm/test_index_refs.py; update tests/llm/test_index.py::TestPageEnrichmentHook

Interfaces:

Consumes: DocRef, migrate, index_docket_state.
Produces:
- index_refs(refs, *, collection, cfg, pool, force=False, sealed: dict[str, str] | None = None, mark_complete=True) -> dict with stats keys indexed, skipped, chunks, fingerprint_skipped, hash_skipped, docket_complete.
- index_docs(docs, ...) unchanged signature, implemented as index_refs over DocRef(key, collection, docket=metadata.get("docket"), fingerprint="", load=lambda: doc).
- docket_complete(engine, collection) -> dict[str, str] (docket → sealed_at recorded).
- _state(engine, collection) -> dict[str, tuple[str, str]] (key → (content_hash, fingerprint)).
Step 1: Write the failing tests

# tests/llm/test_index_refs.py
from __future__ import annotations

from unittest.mock import MagicMock, patch

from llm.chunk import Doc, content_hash
from llm.config import LlmConfig
from llm.index import index_refs
from llm.source import DocRef

CFG = LlmConfig(ollama_hosts=("http://h1:11434",), embed_model="m", instruct_model="g", embed_dim=768, build_ann_index=False, pg_host="x", pg_port=5432, pg_db="llm", pg_user="llm")
DOC = Doc(key="K1", text="Some body text.", metadata={"docket": "D"})


def _ref(fp="fp1", docket="D", loads=None, doc=DOC):
    def _load():
        if loads is not None:
            loads.append(doc.key)
        return doc
    return DocRef(key=doc.key, collection="comments", docket=docket, fingerprint=fp, load=_load)


def _run(refs, state_rows, *, force=False, sealed=None, mark_complete=True, complete_rows=()):
    store = MagicMock()
    engine = MagicMock()
    conn = engine.begin.return_value.__enter__.return_value

    def fake_execute(clause, *a, **k):
        sql = str(clause)
        r = MagicMock()
        if "FROM index_docket_state" in sql:
            r.fetchall.return_value = list(complete_rows)
        elif "FROM index_state" in sql:
            r.fetchall.return_value = state_rows
        else:
            r.fetchall.return_value = []
        return r

    conn.execute.side_effect = fake_execute
    with (
        patch("llm.index._engine", return_value=engine),
        patch("llm.index.vectorstore", return_value=store),
        patch("llm.index.embed_texts", return_value=[[0.0] * 3]),
        patch("llm.index.ensure_hnsw"),
        patch("llm.index.enrich_pdf_pages", side_effect=lambda d, c: c) as enrich,
        patch("llm.index.HostPool") as MockPool,
    ):
        MockPool.return_value.check.return_value = ["http://h1:11434"]
        stats = index_refs(refs, collection="comments", cfg=CFG, pool=MockPool.return_value, force=force, sealed=sealed, mark_complete=mark_complete)
    return stats, store, conn, enrich


def test_fingerprint_match_skips_without_load():
    loads = []
    stats, store, _, enrich = _run([_ref("fp1", loads=loads)], [("K1", "h-old", "fp1")])
    assert stats["fingerprint_skipped"] == 1 and stats["indexed"] == 0
    assert loads == []
    store.add_embeddings.assert_not_called()
    enrich.assert_not_called()


def test_hash_match_updates_fingerprint_without_embedding():
    h = content_hash(DOC.text)
    stats, store, conn, enrich = _run([_ref("fp2")], [("K1", h, "fp1")])
    assert stats["hash_skipped"] == 1 and stats["indexed"] == 0
    store.add_embeddings.assert_not_called()
    enrich.assert_not_called()
    upd = [c for c in conn.execute.call_args_list if "UPDATE index_state SET fingerprint" in str(c.args[0])]
    assert len(upd) == 1 and upd[0].args[1]["f"] == "fp2"


def test_changed_doc_embeds_and_records_fingerprint():
    stats, store, conn, enrich = _run([_ref("fp2")], [("K1", "stale", "fp1")])
    assert stats["indexed"] == 1 and stats["chunks"] == 1
    store.add_embeddings.assert_called_once()
    enrich.assert_called_once()
    ins = [c for c in conn.execute.call_args_list if "INSERT INTO index_state" in str(c.args[0])]
    assert ins[0].args[1]["f"] == "fp2"


def test_empty_fingerprint_never_matches():
    stats, store, _, _ = _run([_ref("")], [("K1", "stale", "")])
    assert stats["indexed"] == 1


def test_force_ignores_fingerprint_and_hash():
    h = content_hash(DOC.text)
    stats, store, _, _ = _run([_ref("fp1")], [("K1", h, "fp1")], force=True)
    assert stats["indexed"] == 1


def test_load_none_counts_skipped():
    ref = DocRef(key="K9", collection="comments", docket="D", fingerprint="x", load=lambda: None)
    stats, *_ = _run([ref], [])
    assert stats["skipped"] == 1 and stats["indexed"] == 0


def test_sealed_docket_marked_complete_after_clean_run():
    stats, _, conn, _ = _run([_ref("fp1")], [], sealed={"D": "2026-10-20T00:00:00Z"})
    assert stats["docket_complete"] == 1
    ins = [c for c in conn.execute.call_args_list if "INSERT INTO index_docket_state" in str(c.args[0])]
    assert ins[0].args[1] == {"c": "comments", "d": "D", "s": "2026-10-20T00:00:00Z"}


def test_not_marked_when_mark_complete_false_or_unsealed():
    stats, _, conn, _ = _run([_ref("fp1")], [], sealed={"D": "s"}, mark_complete=False)
    assert stats["docket_complete"] == 0
    stats, _, conn, _ = _run([_ref("fp1")], [], sealed={})
    assert stats["docket_complete"] == 0

Update tests/llm/test_index.py::TestPageEnrichmentHook::test_enriches_chunks_before_add → rename to test_enriches_only_docs_that_embed and assert that a doc whose hash matches state is not enriched:

    def test_enriches_only_docs_that_embed(self, monkeypatch):
        from llm import index as index_mod
        seen = []
        monkeypatch.setattr(index_mod, "enrich_pdf_pages", lambda doc, chunks: seen.append(doc.key) or chunks)
        _run([DOC], [])
        assert seen == ["K1"]
        seen.clear()
        _run([DOC], [("K1", content_hash(DOC.text), "")])
        assert seen == []

_run in that file feeds state_rows as 2-tuples; change them to 3-tuples (key, hash, fingerprint) throughout (("K1", h) → ("K1", h, "")).

Step 2: Run to verify failure

Run: uv run --no-sync pytest tests/llm/test_index_refs.py -q -p no:cacheprovider Expected: FAIL — ImportError: cannot import name 'index_refs'.

Step 3: Implement

def _state(engine: Engine, collection: str) -> dict[str, tuple[str, str]]:
    with engine.begin() as conn:
        rows = conn.execute(
            text("SELECT item_key, content_hash, fingerprint FROM index_state WHERE collection = :c"),
            {"c": collection},
        ).fetchall()
    return {r[0]: (r[1], r[2] or "") for r in rows}


def docket_complete(engine: Engine, collection: str) -> dict[str, str]:
    """docket → sealed_at for dockets fully indexed under that seal."""
    with engine.begin() as conn:
        rows = conn.execute(
            text("SELECT docket, sealed_at FROM index_docket_state WHERE collection = :c"),
            {"c": collection},
        ).fetchall()
    return {r[0]: r[1] for r in rows}


def index_refs(
    refs: Iterable[DocRef],
    *,
    collection: str,
    cfg: LlmConfig,
    pool: HostPool,
    force: bool = False,
    sealed: dict[str, str] | None = None,
    mark_complete: bool = True,
) -> dict:
    """Fingerprint-first incremental indexing.

    Per ref: (1) fingerprint equal to the stored one → skip without
    loading; (2) load, hash the text; hash equal → record the new
    fingerprint, skip embedding; (3) chunk, locate PDF pages, embed,
    write. PDFs are opened only on path (3). After the loop, every
    docket in *sealed* that was iterated is recorded in
    ``index_docket_state`` so the next run does not list it at all —
    unless *mark_complete* is False (a ``--limit`` run is never complete).
    """
    engine = _engine(cfg)
    migrate(engine)
    pool.check(cfg.embed_model)
    store = vectorstore(collection, cfg, pool)
    seen = _state(engine, collection)
    stats = {"indexed": 0, "skipped": 0, "chunks": 0, "fingerprint_skipped": 0, "hash_skipped": 0, "docket_complete": 0}
    dockets_seen: set[str] = set()

    for ref in refs:
        if ref.docket:
            dockets_seen.add(ref.docket)
        prev_hash, prev_fp = seen.get(ref.key, ("", ""))
        if not force and ref.fingerprint and ref.fingerprint == prev_fp:
            stats["fingerprint_skipped"] += 1
            continue
        doc = ref.load()
        if doc is None or not doc.text.strip():
            stats["skipped"] += 1
            continue
        h = content_hash(doc.text)
        if not force and prev_hash == h:
            with engine.begin() as conn:
                conn.execute(
                    text("UPDATE index_state SET fingerprint = :f WHERE item_key = :k AND collection = :c"),
                    {"f": ref.fingerprint, "k": ref.key, "c": collection},
                )
            stats["hash_skipped"] += 1
            continue
        chunks = enrich_pdf_pages(doc, chunk_doc(doc))
        if not chunks:
            stats["skipped"] += 1
            continue
        vectors = embed_texts(pool, cfg.embed_model, [c.text for c in chunks])
        _delete_old_chunks(engine, ref.key, collection)
        store.add_embeddings(texts=[c.text for c in chunks], embeddings=vectors, metadatas=[c.metadata for c in chunks], ids=[c.id for c in chunks])
        with engine.begin() as conn:
            conn.execute(
                text(
                    "INSERT INTO index_state (item_key, collection, content_hash, chunk_count, fingerprint) "
                    "VALUES (:k, :c, :h, :n, :f) "
                    "ON CONFLICT (item_key, collection) DO UPDATE SET "
                    "content_hash = :h, chunk_count = :n, fingerprint = :f, indexed_at = now()"
                ),
                {"k": ref.key, "c": collection, "h": h, "n": len(chunks), "f": ref.fingerprint},
            )
        stats["indexed"] += 1
        stats["chunks"] += len(chunks)
        if stats["indexed"] % 100 == 0:
            log.info("indexed %(indexed)s (+%(chunks)s)", stats)  # pragma: no cover

    if mark_complete and sealed:
        for d in sorted(dockets_seen & set(sealed)):
            with engine.begin() as conn:
                conn.execute(
                    text(
                        "INSERT INTO index_docket_state (collection, docket, sealed_at) VALUES (:c, :d, :s) "
                        "ON CONFLICT (collection, docket) DO UPDATE SET sealed_at = :s, indexed_at = now()"
                    ),
                    {"c": collection, "d": d, "s": sealed[d]},
                )
            stats["docket_complete"] += 1

    if cfg.build_ann_index:
        ensure_hnsw(engine, cfg.embed_dim)
    else:
        log.info("ANN index skipped (build_ann_index=false); using exact search")
    return stats


def index_docs(docs: Iterable[Doc], *, collection: str, cfg: LlmConfig, pool: HostPool, force: bool = False) -> dict:
    """Back-compat wrapper: Docs without fingerprints (never fingerprint-skipped)."""
    from llm.source import DocRef

    refs = (
        DocRef(key=d.key, collection=collection, docket=d.metadata.get("docket") or None, fingerprint="", load=(lambda d=d: d))
        for d in docs
    )
    stats = index_refs(refs, collection=collection, cfg=cfg, pool=pool, force=force, sealed=None)
    return {k: stats[k] for k in ("indexed", "skipped", "chunks")}

Import DocRef lazily inside index_docs (as shown) to avoid an import cycle (llm.source imports llm.chunk, not llm.index, so a top-level from llm.source import DocRef is also fine; either works).

Step 4: Run to verify pass

Run: uv run --no-sync pytest tests/llm -q -p no:cacheprovider Expected: pass.

Step 5: Commit

git add src/llm/index.py tests/llm/test_index_refs.py tests/llm/test_index.py
git commit -m "feat(llm): fingerprint-first index_refs; PDFs opened only when embedding; docket completion (refs #615)"

Task 15: `stack llm index` wiring

Files:

Modify: src/cli/llm.py
Test: tests/cli/test_llm_exercise.py (update TestIndexComments, TestIndexCorpus, TestIndexAll, TestIndexLimit to the ref iterators), add tests/cli/test_llm_index_sealed.py

Interfaces:

Consumes: iter_comment_refs/iter_rule_refs/iter_corpus_refs, ZoteroPdfIndex.lazy, index_refs, docket_complete, Store.sealed_dockets.
Produces: stack llm index unchanged flags; output line adds fp_skipped= hash_skipped= docket_complete=.
Step 1: Write the failing tests

# tests/cli/test_llm_index_sealed.py
from __future__ import annotations

from unittest.mock import MagicMock, patch

from typer.testing import CliRunner

from cli.llm import app

runner = CliRunner()
_STATS = {"indexed": 0, "skipped": 0, "chunks": 0, "fingerprint_skipped": 5, "hash_skipped": 0, "docket_complete": 0}


@patch("llm.index._engine")
@patch("llm.index.docket_complete", return_value={"CMS-2019-0111": "s1", "CMS-2020-0088": "old"})
@patch("llm.source.iter_comment_refs")
@patch("llm.pool.HostPool.from_config")
@patch("llm.index.index_refs")
@patch("conf.connect.bib")
@patch("llm.config.load")
def test_complete_sealed_dockets_are_not_listed(mock_load, mock_bib, mock_index, mock_pool, mock_iter, mock_complete, _engine):
    mock_load.return_value = MagicMock()
    store = MagicMock()
    store.sealed_dockets.return_value = {"CMS-2019-0111": "s1", "CMS-2020-0088": "s2"}
    mock_bib.return_value = store
    mock_iter.return_value = iter([])
    mock_index.return_value = _STATS

    result = runner.invoke(app, ["index"])

    assert result.exit_code == 0, result.output
    kwargs = mock_iter.call_args.kwargs
    assert kwargs["skip_dockets"] == {"CMS-2019-0111"}      # seal matches → skipped
    ikw = mock_index.call_args.kwargs
    assert ikw["sealed"] == {"CMS-2020-0088": "s2"}          # re-sealed docket will be re-marked
    assert ikw["mark_complete"] is True
    assert "fp_skipped=5" in result.output


@patch("llm.index._engine")
@patch("llm.index.docket_complete", return_value={})
@patch("llm.source.iter_comment_refs")
@patch("llm.pool.HostPool.from_config")
@patch("llm.index.index_refs")
@patch("conf.connect.bib")
@patch("llm.config.load")
def test_force_and_limit_disable_skips_and_completion(mock_load, mock_bib, mock_index, mock_pool, mock_iter, mock_complete, _engine):
    mock_load.return_value = MagicMock()
    store = MagicMock()
    store.sealed_dockets.return_value = {"CMS-2019-0111": "s1"}
    mock_bib.return_value = store
    mock_iter.return_value = iter([])
    mock_index.return_value = _STATS

    result = runner.invoke(app, ["index", "--force", "--limit", "5"])

    assert result.exit_code == 0, result.output
    assert mock_iter.call_args.kwargs["skip_dockets"] == set()
    assert mock_index.call_args.kwargs["mark_complete"] is False

Step 2: Run to verify failure

Run: uv run --no-sync pytest tests/cli/test_llm_index_sealed.py -q -p no:cacheprovider Expected: FAIL — iter_comment_refs not called / index_refs not called.

Step 3: Implement

def _refs_for(collection: str, store, docket: str, keys: tuple[str, ...], skip_dockets: set[str]):
    from llm.source import ZoteroPdfIndex, iter_comment_refs, iter_corpus_refs, iter_rule_refs

    if collection == "comments":
        return iter_comment_refs(store, docket=docket, skip_dockets=skip_dockets)
    if collection == "rules":
        return iter_rule_refs(store, keys=keys)
    from conf import ROOT, path

    zotero = ZoteroPdfIndex.lazy(path("db.zotero"), path("storage.zotero"), ROOT / ".state" / "llm")
    return iter_corpus_refs(store, zotero=zotero)


@app.command()
def index(...same options...) -> None:
    import itertools

    from conf.connect import bib
    from llm import config as llm_config
    from llm.index import _engine, docket_complete, index_refs
    from llm.pool import HostPool

    targets = _COLLECTIONS if collection == "all" else (collection,)
    if any(t not in _COLLECTIONS for t in targets):
        raise typer.BadParameter("collection must be comments, rules, corpus or all")
    cfg = llm_config.load()
    store = bib()
    sealed_all = {} if force else store.sealed_dockets()
    for target in targets:
        complete = docket_complete(_engine(cfg), target) if (sealed_all and target == "comments") else {}
        skip = {d for d, s in sealed_all.items() if complete.get(d) == s}
        pending_seals = {d: s for d, s in sealed_all.items() if d not in skip}
        refs = _refs_for(target, store, docket, tuple(key), skip)
        if limit:
            refs = itertools.islice(refs, limit)
        stats = index_refs(
            refs, collection=target, cfg=cfg, pool=HostPool.from_config(cfg),
            force=force, sealed=pending_seals if target == "comments" else None,
            mark_complete=not limit,
        )
        if skip:
            typer.echo(f"{target}: {len(skip)} sealed docket(s) already complete — not listed")
        typer.echo(
            f"{target}: indexed={stats['indexed']} skipped={stats['skipped']} chunks={stats['chunks']} "
            f"fp_skipped={stats['fingerprint_skipped']} hash_skipped={stats['hash_skipped']} "
            f"docket_complete={stats['docket_complete']}"
        )

Update tests/cli/test_llm_exercise.py: patch llm.source.iter_comment_refs / iter_rule_refs / iter_corpus_refs and llm.source.ZoteroPdfIndex.lazy instead of the _docs/snapshot names; patch llm.index.index_refs and llm.index.docket_complete (return {}) and llm.index._engine; _STATS gains the three new keys; the comments assertion becomes mock_iter.assert_called_once_with(store, docket="", skip_dockets=set()) and the corpus one mock_iter.assert_called_once_with(store, zotero=zot).

Step 4: Run to verify pass

Run: uv run --no-sync pytest tests/cli/test_llm_index_sealed.py tests/cli/test_llm_exercise.py -q -p no:cacheprovider Expected: pass.

Step 5: Commit

git add src/cli/llm.py tests/cli/test_llm_index_sealed.py tests/cli/test_llm_exercise.py
git commit -m "feat(cli): stack llm index lists only unsealed/incomplete dockets, lazy Zotero snapshot (refs #615)"

Task 16: Re-farm script to the plain chain; full test run

Files:

Modify: dev/scripts/refarm_cms_2026_2377.sh
Step 1: Simplify the chain

Replace the two-phase block with the plain chain (every step is now cheap when idle), keeping the step helper:

# Every step is incremental: sealed dockets are skipped outright, open
# dockets pull from the stored watermark, extract/index compare cheap
# fingerprints before doing any work. Mirror first (bulk, no cap), then
# the API for anything the mirror lags on.
step "mirror backfill" uv run stack bib backfill-comments --docket CMS-2026-2377 --mirror
step "api fetch"       uv run stack bib fetch-pfs-comments --docket CMS-2026-2377
step "api backfill"    uv run stack bib backfill-comments --docket CMS-2026-2377
step "extract"         uv run stack comments extract --docket CMS-2026-2377
step "index"           uv run stack llm index --collection comments --docket CMS-2026-2377

Step 2: Run the whole suite

Run: uv run --no-sync pytest tests -q -p no:cacheprovider -x Expected: all pass. Fix anything the earlier tasks left behind before committing.

Step 3: Commit

git add dev/scripts/refarm_cms_2026_2377.sh
git commit -m "chore(comments): re-farm script is the plain chain now that every stage is incremental (refs #615)"

Task 17: Rollout on the live data

Run from /home/kert/stack with set -a; . ./.env; set +a first. Record every number in a comment on Gitea #615.

Step 1: Schema + discover

uv run stack comments dockets --discover     # 12 rows, ≤12 API calls, once
uv run stack comments dockets

Expected: 12 rows with close dates; CMS-2026-2377 closes 2026-09-14.

Step 2: Attachment dedupe

uv run python dev/scripts/dedupe_attachments.py             # dry run: ~33,379 groups
uv run python dev/scripts/dedupe_attachments.py --apply
uv run stack comments dockets                               # reopens the store → unique index created
sqlite3 data/bib.sqlite "SELECT name FROM sqlite_master WHERE name='idx_attachments_item_filename'"

Step 2b: Recover the abstracts blanked on 2026-09-08

The old upsert overwrote 17,607 enriched bodies in CMS-2026-2377 and 1,115 in CMS-2017-0092 with '' (incident in the SDD ledger). They still carry enriched:ok, so backfill skips them. Clear the tag on those rows and re-enrich from the mirror (idempotent attach + data-preserving upsert are in place by now):

for d in CMS-2026-2377 CMS-2017-0092; do
  sqlite3 data/bib.sqlite "DELETE FROM item_tags WHERE tag_id=(SELECT id FROM tags WHERE name='enriched:ok') AND item_id IN (SELECT id FROM items WHERE url LIKE 'https://www.regulations.gov/comment/$d-%' AND coalesce(abstract,'')='')"
  uv run stack bib backfill-comments --docket $d --mirror
done
sqlite3 data/bib.sqlite "SELECT substr(url,37,13), count(*), sum(coalesce(abstract,'')<>'') FROM items WHERE url LIKE 'https://www.regulations.gov/comment/CMS-2026-2377-%' OR url LIKE 'https://www.regulations.gov/comment/CMS-2017-0092-%' GROUP BY 1"

Expected: non-empty abstracts ≈ total for both dockets (a few genuinely empty bodies are fine). Then re-run stack comments extract and stack llm index --collection comments so any comment whose text was lost from combined.md/index is restored (the index still holds pre-wipe text; fingerprints change because updated_at moved, hash-skip absorbs unchanged text).

Step 3: Seal the historical dockets

for d in CMS-2017-0092 CMS-2018-0076 CMS-2019-0111 CMS-2020-0088 CMS-2021-0119 CMS-2022-0113 CMS-2023-0121 CMS-2024-0256 CMS-2025-0304; do
  uv run stack comments seal $d --reason historical
done

(Check stack comments dockets for any docket id outside this list — there were 12 tags on 2026-09-08; seal every one except CMS-2026-2377.)

Step 4: Last full walks (stamp fingerprints, mark completion)

time uv run stack llm index --collection all      # stat-only over 196k items; expect hash_skipped≈196k, indexed≈0
time uv run stack llm index --collection all      # second run: fp_skipped only for CMS-2026-2377; 11 dockets "already complete"
time uv run stack comments extract                # skipped_sealed: 11
time uv run stack bib fetch-pfs-comments          # 11 "sealed … skipping" lines, one walk from watermark

Acceptance: second index run < 60 s with no pdf pages log lines and no change to .state/llm/zotero.sqlite mtime; fetch makes ≤ 4 API calls (count _get log lines or watch the run time: ≤ 4 × sleep).

Step 5: Report + memory

Post the timings and counts to #615. Update the memory note palliative_rfi_project.md (re-farm section) with: sealed dockets, the dockets/seal/unseal commands, and that --force is the only way to redo sealed work.

137 KiB Raw Blame History Unescape Escape

Comment Pipeline Seal-and-Skip Implementation Plan

Global Constraints

File Structure

Task 1: bib/dockets.py — Docket, should_seal, fingerprint_files, quiet_days

Task 2: dockets table + Store docket CRUD + url index

Task 3: No-op upsert — Store.upsert_status

Task 3b: Data-preserving upsert — never blank a stored value

Task 4: Idempotent attach_file

Task 5: Dedupe migration script + guarded unique index

Task 6: iter_comments(since=, on_error=)

Task 7: discover_docket + walk_docket (watermark, counts, auto-seal)

Task 8: Fetch CLI rewired — sealed skip before any API call

Task 9: Backfill skips sealed dockets (API + mirror)

Task 10: Extract currency by mtime — source_paths, is_current, walker skip without reads

Task 11: stack comments — extract with sealed skip + lazy body lookup; dockets / seal / unseal

Task 12: llm migrations — index_state.fingerprint, index_docket_state

Task 13: DocRef + lazy iterators + lazy Zotero snapshot

Task 14: index_refs — fingerprint-first loop, PDFs only on the embed path, docket completion

Task 15: stack llm index wiring

Task 16: Re-farm script to the plain chain; full test run

Task 17: Rollout on the live data

137 KiB

Raw Blame History

Task 1: `bib/dockets.py` — Docket, should_seal, fingerprint_files, quiet_days

Task 2: `dockets` table + Store docket CRUD + url index

Task 3: No-op upsert — `Store.upsert_status`

Task 4: Idempotent `attach_file`

Task 6: `iter_comments(since=, on_error=)`

Task 7: `discover_docket` + `walk_docket` (watermark, counts, auto-seal)

Task 10: Extract currency by mtime — `source_paths`, `is_current`, walker skip without reads

Task 11: `stack comments` — extract with sealed skip + lazy body lookup; `dockets` / `seal` / `unseal`

Task 12: llm migrations — `index_state.fingerprint`, `index_docket_state`

Task 13: `DocRef` + lazy iterators + lazy Zotero snapshot

Task 14: `index_refs` — fingerprint-first loop, PDFs only on the embed path, docket completion

Task 15: `stack llm index` wiring