stack/docs/superpowers/plans/2026-08-17-zotero-rule-sync-p38.md

P38: Zotero Rule Sync Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Land the CY2027 PFS NPRM (bib 5ITGVDJV) in the Zotero library and make the whole rule-sync path unable to silently fail again.

Architecture: Four small moves, strictly ordered: (1) fix Store.upsert so the URL-dedupe update path merges tags/collections with what is already stored instead of wipe-and-replace; (2) repair the wiped data by re-running the existing idempotent fetch_nprm_addenda --register for 2026 and 2027; (3) run the tag-scoped sync-zotero that P37 skipped; (4) add a source:federal-register-scoped step to the generated nightly zotero-sync workflow. The fix must land before the repair, or the next comment-farm/mail-poller upsert re-wipes the repair (mail-poller mounts ./src live, so the host commit takes effect on its next 600s loop; the 2026-09-15 refarm cron also runs from this checkout).

Tech Stack: Python (sqlite3 stdlib, typer CLI), pytest, Gitea Actions workflows generated by dev/scripts/backends/gitea.py via dev/scripts/gen_config.py.

Spec: No standalone spec — the incident diagnosis and requirements live in the Gitea milestone and issues below (real links):

• Milestone: P38: Zotero Rule Sync — upsert tag merge, tag repair, nightly rules scope
• #624 — bib: Store.upsert clobbers curated tags/collections on URL-dedupe update
• #625 — bib,pfs: restore sup:*_PFS_NPRM / file:rvu / year tags wiped from 2KVJ2HKX and 5ITGVDJV
• #626 — bib: CY2027 NPRM (5ITGVDJV) never synced to Zotero — run the tag-scoped sync P37 skipped
• #627 — ci: nightly zotero-sync only covers source:email — add a rules scope
• Background: #623 (docs tag-drift symptom of the same clobber), P37 plan docs/superpowers/plans/2026-08-13-cy2027-pfs-proposed-rule-p37.md, P36 plan Step 5 (the sync step P37 skipped).

Global Constraints

• Commit messages: conventional-commit style, reference issues as (closes #N) / (refs #N); no Co-Authored-By trailer.
• Task order is load-bearing: Task 1 (merge fix) MUST be committed before Task 3 (data repair) runs.
• Never run an unscoped sync-zotero — it hydrates ~180k bib items (N+1). Always --tag.
• sync-zotero writes into data/zotero/data/zotero.sqlite, which the running zotero container holds an exclusive lock on. The CLI's default --hold-zotero stops/starts the container itself when run from the host (docker required).
• Store.update(tags=...) keeps its explicit replace semantics; only Store.upsert's dedupe path changes to merge. Deliberate tag removal stays Store.remove_tag.
• Zotero item keys mirror bib keys 1:1 (verified: all prior rules exist in Zotero under their bib key).

---

Task 1: Store.upsert merges tags/collections on the dedupe path (#624)

Files:

• Modify: src/bib/store.py:207-239 (Store.upsert)
• Test: tests/bib/test_store.py (extend TestUpsertDedup, ~line 678)

Interfaces:

• Consumes: Store.get(key) -> Item (existing; returns item with .tags: list[str], .collections: list[str]), Store.update(key, **fields) (existing; tags=/collections= are replace-semantics).

• Produces: Store.upsert(item, *, tags=None, collection="") -> str — unchanged signature; on the existing-URL path the stored row ends up with the union (existing ∪ incoming, order: existing first, dupes dropped) of tags and of collections. Tasks 3–4 rely on curated tags surviving re-ingest.

• Step 1: Write the failing tests — append to TestUpsertDedup in tests/bib/test_store.py:

    def test_upsert_preserves_curated_tags_on_existing(self) -> None:
        """Regression #624: a re-ingest upsert must not clobber tags
        added to the stored item since the last ingest (the CY2027 NPRM
        lost sup:2027_PFS_NPRM this way and never reached Zotero)."""
        s = Store(":memory:")
        item1 = Rule(title="First", url="https://example.com/keep")
        key = s.upsert(item1, tags=["source:federal-register"])
        s.add_tag(key, "sup:2027_PFS_NPRM")  # curated after ingest

        item2 = Rule(title="Re-ingested", url="https://example.com/keep")
        item2.add_tag("source:federal-register")
        key2 = s.upsert(item2)
        assert key2 == key
        got = s.get(key)
        assert "sup:2027_PFS_NPRM" in got.tags
        assert "source:federal-register" in got.tags
        assert got.tags.count("source:federal-register") == 1
        s.close()

    def test_upsert_preserves_existing_collections(self) -> None:
        """Regression #624: same wipe hazard via _sync_collections."""
        s = Store(":memory:")
        coll = s.ensure_collections({"Rules": {}})["Rules"]
        item1 = Rule(title="First", url="https://example.com/keepc")
        key = s.upsert(item1, collection=coll)

        item2 = Rule(title="Re-ingested", url="https://example.com/keepc")
        key2 = s.upsert(item2)
        assert key2 == key
        assert coll in s.get(key).collections
        s.close()

• Step 2: Run tests to verify they fail

Run: uv run pytest tests/bib/test_store.py::TestUpsertDedup -v Expected: the two new tests FAIL (sup:2027_PFS_NPRM / coll missing after second upsert); the three existing tests PASS.

• Step 3: Implement the merge — replace the existing-URL branch of Store.upsert (src/bib/store.py:220-237). Note the current code computes row = item.to_row() twice (line 222 is dead); the replacement drops it:

            if existing:
                ekey = existing["key"]
                if tags:
                    for tag in tags:
                        label = tag.label if hasattr(tag, "label") else str(tag)
                        item.add_tag(label)
                if collection and collection not in item.collections:
                    item.collections.append(collection)
                item.stamp_access()
                # Merge with what's already stored — a re-ingest must
                # never clobber tags/collections curated on the row
                # since the last ingest (#624: the CY2027 NPRM lost its
                # sup: registration tag to this and silently vanished
                # from the tag-scoped Zotero sync). Deliberate removal
                # goes through remove_tag, not upsert.
                current = self.get(ekey)
                row = item.to_row()
                row.pop("key", None)
                row["tags"] = list(dict.fromkeys([*current.tags, *item.tags]))
                row["collections"] = list(
                    dict.fromkeys([*current.collections, *item.collections])
                )
                self.update(ekey, **row)
                return ekey

• Step 4: Run the full store + sync test files

Run: uv run pytest tests/bib/test_store.py tests/bib/test_sync.py tests/bib/test_sync_full.py tests/bib/test_sync_deeper.py tests/bib/test_sync_exercise.py -q Expected: all PASS (no existing test asserts wipe semantics — verified before planning).

• Step 5: Commit

git add src/bib/store.py tests/bib/test_store.py
git commit -m "fix(bib): upsert merges tags/collections with stored row on URL dedupe (closes #624)"

---

Task 2: Nightly zotero-sync gains a rules scope (#627)

Files:

• Modify: dev/scripts/backends/gitea.py:584-634 (_gen_zotero_sync)
• Generated: .gitea/workflows/zotero-sync.yml (via uv run python dev/scripts/gen_config.py — never hand-edit)

Interfaces:

• Consumes: stack bib sync-zotero --tag <t> --no-hold CLI (existing; idempotent — existing Zotero items are skipped/refreshed, not duplicated).

• Produces: nightly workflow with a second sync step scoped --tag source:federal-register (78 items today), between the email-sync step and the restart step.

• Step 1: Add the step to the generator — in _gen_zotero_sync, insert after the email-sync step (before Restart zotero):

      - name: Sync Federal Register rule items into Zotero
        # Rule items only ever reached Zotero via one-off manual syncs
        # (P36 step 5); P37 skipped that step and the CY2027 NPRM never
        # arrived (#626/#627). Tag-scoped: a full unscoped sync would
        # hydrate ~180k bib items.
        run: |
          docker exec api uv run --no-sync \\
            stack bib sync-zotero --tag source:federal-register --no-hold

(inside the f-string content, matching the surrounding indentation and \\ line-continuation style of the email step)

• Step 2: Regenerate and inspect

Run: uv run python dev/scripts/gen_config.py && git diff .gitea/workflows/ Expected: only zotero-sync.yml changes, gaining exactly the new step; header still says generated.

• Step 3: Determinism check

Run: uv run python dev/scripts/gen_config.py && git diff --stat Expected: identical diff (no churn on re-run).

• Step 4: Commit

git add dev/scripts/backends/gitea.py .gitea/workflows/zotero-sync.yml
git commit -m "feat(ci): nightly zotero-sync also pushes source:federal-register rule items (closes #627)"

---

Task 3: Repair the wiped registration tags (#625)

Files:

• No source changes. Runs dev/scripts/fetch_nprm_addenda.py (existing, idempotent: attach is guarded by title-dedupe, tags via add_tag's INSERT OR IGNORE). Data: data/bib.sqlite.

Interfaces:

• Consumes: fetch_nprm_addenda.py --year {2026,2027} --register; CONFIGS[year].register_tags = sup:<year>_PFS_NPRM, module:pfs, file:rvu, year:<year>.
• Produces: 2KVJ2HKX and 5ITGVDJV carry their full register tag sets again; pfs.nprm._addendum_b_path(year) resolves for both years. Task 4 keys its sync on these tags.

Precondition: Task 1 is committed (mail-poller picks the fixed code up live; the repair must not be re-wipeable).

• Step 1: Re-run registration for both years (ZIPs already on disk → download step self-skips; attach guard prints SKIP attach):

Run: uv run python dev/scripts/fetch_nprm_addenda.py --year 2026 --register && uv run python dev/scripts/fetch_nprm_addenda.py --year 2027 --register Expected: both print SKIP attach — ... already has an attachment and Current tags on <key>: [...] including the sup: tag. If either attaches instead of skipping, STOP — the guard regressed; investigate before continuing.

• Step 2: Verify tag restoration + loader discovery

uv run python - <<'EOF'
from conf import connect
from pfs import nprm
store = connect.bib()
for year, tag, key in ((2026, "sup:2026_PFS_NPRM", "2KVJ2HKX"), (2027, "sup:2027_PFS_NPRM", "5ITGVDJV")):
    items = store.list_items(tag=tag)
    assert [i.key for i in items] == [key], (tag, [i.key for i in items])
    path = nprm._addendum_b_path(year)
    print(year, "OK:", key, "->", path)
EOF

Expected: 2026 OK: ... and 2027 OK: ... with real xlsx paths.

• Step 3: Simulate the wiper to prove the fix holds (read-only on real data — run the exact upsert the comment farm performs against a copy of bib.sqlite, assert sup:2027_PFS_NPRM survives):

cp data/bib.sqlite /tmp/claude-1000/-home-kert-stack/e01452a7-3395-4fb6-bd66-ed6fd8c06398/scratchpad/bib_copy.sqlite
uv run python - <<'EOF'
from bib.store import Store
from bib.translate import federal_register
s = Store("/tmp/claude-1000/-home-kert-stack/e01452a7-3395-4fb6-bd66-ed6fd8c06398/scratchpad/bib_copy.sqlite")
rule = federal_register("https://www.federalregister.gov/documents/2026-14327")
rule.add_tag("source:federal-register")
rule.add_tag("module:pfs")
key = s.upsert(rule)
assert key == "5ITGVDJV", key
assert "sup:2027_PFS_NPRM" in s.get(key).tags
print("survives re-ingest OK")
EOF

Expected: survives re-ingest OK. (Network hiccup on the FR fetch → retry; this step is diagnostic only.)

• Step 4: Comment the verification output on #625 and close it (data-only task, nothing to commit).

---

Task 4: Sync the CY2027 NPRM into Zotero (#626)

Files:

• No source changes. Data: data/zotero/data/zotero.sqlite. Requires docker on the host (CLI default --hold-zotero stops/starts the zotero container around the write).

Interfaces:

• Consumes: restored sup: tags from Task 3; stack bib sync-zotero --tag <t>.

• Produces: Zotero item 5ITGVDJV with 3 child attachments; refreshed tag set on Zotero's 2KVJ2HKX.

• Step 1: Confirm the zotero container is running and no notebooks-integration run is active (docker ps --format '{{.Names}}' | grep zotero; the nightly runs at 03:30/04:15 — any daytime run is clear).

• Step 2: Push both years' tag scopes

Run: uv run stack bib sync-zotero --tag sup:2027_PFS_NPRM && uv run stack bib sync-zotero --tag sup:2026_PFS_NPRM Expected: each prints sync stats (created: 1 … for 2027; skipped: 1 + tag refresh for 2026); zotero container restarted after each (CLI handles it).

• Step 3: Verify in zotero.sqlite

uv run python - <<'EOF'
import sqlite3
z = sqlite3.connect("file:data/zotero/data/zotero.sqlite?mode=ro&immutable=1", uri=True)
iid = z.execute("SELECT itemID FROM items WHERE key='5ITGVDJV'").fetchone()
assert iid, "5ITGVDJV missing from Zotero"
kids = list(z.execute("SELECT path FROM itemAttachments WHERE parentItemID=?", (iid[0],)))
assert len(kids) == 3, kids
tags = {r[0] for r in z.execute("SELECT t.name FROM itemTags it JOIN tags t ON t.tagID=it.tagID WHERE it.itemID=?", (iid[0],))}
assert "sup:2027_PFS_NPRM" in tags, tags
t26 = {r[0] for r in z.execute("SELECT t.name FROM itemTags it JOIN tags t ON t.tagID=it.tagID JOIN items i ON i.itemID=it.itemID WHERE i.key='2KVJ2HKX'")}
assert "sup:2026_PFS_NPRM" in t26 and "file:rvu" in t26, t26
print("Zotero verified: 5ITGVDJV +", len(kids), "attachments; tags OK both years")
EOF

Expected: Zotero verified: 5ITGVDJV + 3 attachments; tags OK both years. Also confirm zotero container is back up (docker ps | grep zotero).

• Step 4: Comment verification on #626 and close it.

---

Task 5: Close the loop

Files:

• Modify: docs/superpowers/plans/2026-08-17-zotero-rule-sync-p38.md (check boxes)

• Memory: ~/.claude/projects/-home-kert-stack/memory/ (update cms_mail_pipeline.md; new incident memory)

• Step 1: Push commits (git push), confirm CI green on the two commits (or file/watch per gitea-tracker-ops; the auto-filer leaves stale failure issues — check latest run on HEAD).

• Step 2: Close #624 and #627 if the closes #N footers didn't auto-close them (Gitea closes on push to main), each with a comment linking the commit SHA.

• Step 3: Close the P38 milestone.

• Step 4: Append a "P38 build outcomes" section to this plan (what shipped, SHAs, verification results — pattern from P36/P37 specs), commit as docs(plan): P38 outcomes (refs #624-#627).

• Step 5: Update memory — cms_mail_pipeline.md gains the rules-scope nightly step; new memory zotero-rule-sync-incident.md (upsert used to clobber curated tags; sup: tags are load-bearing for pfs.nprm discovery; sync coverage now email + federal-register).

P38 build outcomes (2026-08-17)

Issues #624–#628, milestone P38. Executed same-day on main (deliberate worktree deviation: mail-poller live-mounts this checkout's ./src, and Tasks 3–4 operate on the live databases here).

• #624 fixed (91c3306): Store.upsert now unions existing ∪ incoming tags and collections on the URL-dedupe path; two regression tests (both red pre-fix). 137 store+sync tests green.
• #627 fixed (c9e8e1e): _gen_zotero_sync emits a second nightly step, sync-zotero --tag source:federal-register --no-hold (78 items); regenerated zotero-sync.yml; generator deterministic.
• #625 repaired (data-only, closed with verification): re-ran fetch_nprm_addenda --year {2026,2027} --register — attach guards held (SKIP attach), full tag sets restored on 2KVJ2HKX and 5ITGVDJV; _discover_addendum_b resolves for both years; wiper simulation on a bib copy (comment-farm-style upsert, same URL + canonical FR tag set) confirms sup:/file:rvu survive.
• #628 found en route + fixed (18c1066): P37's worktree run left a dead .worktrees/p37/... storage_path on the CY2027 Addendum B attachment (attach_file stored the unresolved path; file itself was safe in canonical storage). attach_file now stores dest.resolve() (regression test with symlinked storage dir); the single affected row rewritten. Audit: no other .worktrees paths.
• #626 done (closed with verification): sync-zotero --tag sup:2027_PFS_NPRM → created 1 item + 9 tags + 3 attachments; --tag sup:2026_PFS_NPRM → skipped (Zotero's snapshot already matched). zotero.sqlite now holds 5ITGVDJV (statute-typed, 91 FR 43842, field parity with 2KVJ2HKX) with all 3 files physically in storage (39 MB PDF, 2.6 MB TXT, 1.2 MB Addendum B xlsx). Zotero container held during sync, restarted healthy.
• Plan deviations: the Task 3 wiper simulation ran offline (the FR API 404s on the bare-document-number URL form; same upsert, locally constructed item). nature.csl working-tree drift observed (Zotero app self-update, container-owned) — left uncommitted.

Self-Review

• Spec coverage: #624→Task 1, #625→Task 3, #626→Task 4, #627→Task 2, closure/tracking→Task 5. No gaps.
• Placeholder scan: none — every step has runnable code/commands and expected output.
• Type consistency: Store.get returns Item with .tags/.collections lists (verified in src); list_items(tag=) verified; nprm._addendum_b_path(year) verified private helper exists.
• Ordering hazard called out twice (Task 1 before Task 3) since mail-poller runs live-mounted source every 600s.