stack/docs/superpowers/plans/2026-08-17-zotero-rule-sync-p38.md

# P38: Zotero Rule Sync Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Land the CY2027 PFS NPRM (bib `5ITGVDJV`) in the Zotero library and make the whole rule-sync path unable to silently fail again.

**Architecture:** Four small moves, strictly ordered: (1) fix `Store.upsert` so the URL-dedupe update path *merges* tags/collections with what is already stored instead of wipe-and-replace; (2) repair the wiped data by re-running the existing idempotent `fetch_nprm_addenda --register` for 2026 and 2027; (3) run the tag-scoped `sync-zotero` that P37 skipped; (4) add a `source:federal-register`-scoped step to the generated nightly zotero-sync workflow. The fix must land before the repair, or the next comment-farm/mail-poller upsert re-wipes the repair (mail-poller mounts `./src` live, so the host commit takes effect on its next 600s loop; the 2026-09-15 refarm cron also runs from this checkout).

**Tech Stack:** Python (sqlite3 stdlib, typer CLI), pytest, Gitea Actions workflows generated by `dev/scripts/backends/gitea.py` via `dev/scripts/gen_config.py`.

**Spec:** No standalone spec — the incident diagnosis and requirements live in the Gitea milestone and issues below (real links):

- Milestone: [P38: Zotero Rule Sync — upsert tag merge, tag repair, nightly rules scope](https://git.fhirworx.io/homelab/stack/milestone/39)
- [#624 — bib: Store.upsert clobbers curated tags/collections on URL-dedupe update](https://git.fhirworx.io/homelab/stack/issues/624)
- [#625 — bib,pfs: restore sup:*_PFS_NPRM / file:rvu / year tags wiped from 2KVJ2HKX and 5ITGVDJV](https://git.fhirworx.io/homelab/stack/issues/625)
- [#626 — bib: CY2027 NPRM (5ITGVDJV) never synced to Zotero — run the tag-scoped sync P37 skipped](https://git.fhirworx.io/homelab/stack/issues/626)
- [#627 — ci: nightly zotero-sync only covers source:email — add a rules scope](https://git.fhirworx.io/homelab/stack/issues/627)
- Background: [#623](https://git.fhirworx.io/homelab/stack/issues/623) (docs tag-drift symptom of the same clobber), P37 plan `docs/superpowers/plans/2026-08-13-cy2027-pfs-proposed-rule-p37.md`, P36 plan Step 5 (the sync step P37 skipped).

## Global Constraints

- Commit messages: conventional-commit style, reference issues as `(closes #N)` / `(refs #N)`; **no Co-Authored-By trailer**.
- Task order is load-bearing: Task 1 (merge fix) MUST be committed before Task 3 (data repair) runs.
- Never run an unscoped `sync-zotero` — it hydrates ~180k bib items (N+1). Always `--tag`.
- `sync-zotero` writes into `data/zotero/data/zotero.sqlite`, which the running `zotero` container holds an exclusive lock on. The CLI's default `--hold-zotero` stops/starts the container itself when run from the host (docker required).
- `Store.update(tags=...)` keeps its explicit replace semantics; only `Store.upsert`'s dedupe path changes to merge. Deliberate tag removal stays `Store.remove_tag`.
- Zotero item keys mirror bib keys 1:1 (verified: all prior rules exist in Zotero under their bib key).

---

### Task 1: `Store.upsert` merges tags/collections on the dedupe path ([#624](https://git.fhirworx.io/homelab/stack/issues/624))

**Files:**
- Modify: `src/bib/store.py:207-239` (`Store.upsert`)
- Test: `tests/bib/test_store.py` (extend `TestUpsertDedup`, ~line 678)

**Interfaces:**
- Consumes: `Store.get(key) -> Item` (existing; returns item with `.tags: list[str]`, `.collections: list[str]`), `Store.update(key, **fields)` (existing; `tags=`/`collections=` are replace-semantics).
- Produces: `Store.upsert(item, *, tags=None, collection="") -> str` — unchanged signature; on the existing-URL path the stored row ends up with the **union** (existing ∪ incoming, order: existing first, dupes dropped) of tags and of collections. Tasks 3–4 rely on curated tags surviving re-ingest.

- [x] **Step 1: Write the failing tests** — append to `TestUpsertDedup` in `tests/bib/test_store.py`:

```python
    def test_upsert_preserves_curated_tags_on_existing(self) -> None:
        """Regression #624: a re-ingest upsert must not clobber tags
        added to the stored item since the last ingest (the CY2027 NPRM
        lost sup:2027_PFS_NPRM this way and never reached Zotero)."""
        s = Store(":memory:")
        item1 = Rule(title="First", url="https://example.com/keep")
        key = s.upsert(item1, tags=["source:federal-register"])
        s.add_tag(key, "sup:2027_PFS_NPRM")  # curated after ingest

        item2 = Rule(title="Re-ingested", url="https://example.com/keep")
        item2.add_tag("source:federal-register")
        key2 = s.upsert(item2)
        assert key2 == key
        got = s.get(key)
        assert "sup:2027_PFS_NPRM" in got.tags
        assert "source:federal-register" in got.tags
        assert got.tags.count("source:federal-register") == 1
        s.close()

    def test_upsert_preserves_existing_collections(self) -> None:
        """Regression #624: same wipe hazard via _sync_collections."""
        s = Store(":memory:")
        coll = s.ensure_collections({"Rules": {}})["Rules"]
        item1 = Rule(title="First", url="https://example.com/keepc")
        key = s.upsert(item1, collection=coll)

        item2 = Rule(title="Re-ingested", url="https://example.com/keepc")
        key2 = s.upsert(item2)
        assert key2 == key
        assert coll in s.get(key).collections
        s.close()
```

- [x] **Step 2: Run tests to verify they fail**

Run: `uv run pytest tests/bib/test_store.py::TestUpsertDedup -v`
Expected: the two new tests FAIL (`sup:2027_PFS_NPRM` / `coll` missing after second upsert); the three existing tests PASS.

- [x] **Step 3: Implement the merge** — replace the existing-URL branch of `Store.upsert` (`src/bib/store.py:220-237`). Note the current code computes `row = item.to_row()` twice (line 222 is dead); the replacement drops it:

```python
            if existing:
                ekey = existing["key"]
                if tags:
                    for tag in tags:
                        label = tag.label if hasattr(tag, "label") else str(tag)
                        item.add_tag(label)
                if collection and collection not in item.collections:
                    item.collections.append(collection)
                item.stamp_access()
                # Merge with what's already stored — a re-ingest must
                # never clobber tags/collections curated on the row
                # since the last ingest (#624: the CY2027 NPRM lost its
                # sup: registration tag to this and silently vanished
                # from the tag-scoped Zotero sync). Deliberate removal
                # goes through remove_tag, not upsert.
                current = self.get(ekey)
                row = item.to_row()
                row.pop("key", None)
                row["tags"] = list(dict.fromkeys([*current.tags, *item.tags]))
                row["collections"] = list(
                    dict.fromkeys([*current.collections, *item.collections])
                )
                self.update(ekey, **row)
                return ekey
```

- [x] **Step 4: Run the full store + sync test files**

Run: `uv run pytest tests/bib/test_store.py tests/bib/test_sync.py tests/bib/test_sync_full.py tests/bib/test_sync_deeper.py tests/bib/test_sync_exercise.py -q`
Expected: all PASS (no existing test asserts wipe semantics — verified before planning).

- [x] **Step 5: Commit**

```bash
git add src/bib/store.py tests/bib/test_store.py
git commit -m "fix(bib): upsert merges tags/collections with stored row on URL dedupe (closes #624)"
```

---

### Task 2: Nightly zotero-sync gains a rules scope ([#627](https://git.fhirworx.io/homelab/stack/issues/627))

**Files:**
- Modify: `dev/scripts/backends/gitea.py:584-634` (`_gen_zotero_sync`)
- Generated: `.gitea/workflows/zotero-sync.yml` (via `uv run python dev/scripts/gen_config.py` — never hand-edit)

**Interfaces:**
- Consumes: `stack bib sync-zotero --tag <t> --no-hold` CLI (existing; idempotent — existing Zotero items are skipped/refreshed, not duplicated).
- Produces: nightly workflow with a second sync step scoped `--tag source:federal-register` (78 items today), between the email-sync step and the restart step.

- [x] **Step 1: Add the step to the generator** — in `_gen_zotero_sync`, insert after the email-sync step (before `Restart zotero`):

```python
      - name: Sync Federal Register rule items into Zotero
        # Rule items only ever reached Zotero via one-off manual syncs
        # (P36 step 5); P37 skipped that step and the CY2027 NPRM never
        # arrived (#626/#627). Tag-scoped: a full unscoped sync would
        # hydrate ~180k bib items.
        run: |
          docker exec api uv run --no-sync \\
            stack bib sync-zotero --tag source:federal-register --no-hold
```

(inside the f-string `content`, matching the surrounding indentation and `\\` line-continuation style of the email step)

- [x] **Step 2: Regenerate and inspect**

Run: `uv run python dev/scripts/gen_config.py && git diff .gitea/workflows/`
Expected: only `zotero-sync.yml` changes, gaining exactly the new step; header still says generated.

- [x] **Step 3: Determinism check**

Run: `uv run python dev/scripts/gen_config.py && git diff --stat`
Expected: identical diff (no churn on re-run).

- [x] **Step 4: Commit**

```bash
git add dev/scripts/backends/gitea.py .gitea/workflows/zotero-sync.yml
git commit -m "feat(ci): nightly zotero-sync also pushes source:federal-register rule items (closes #627)"
```

---

### Task 3: Repair the wiped registration tags ([#625](https://git.fhirworx.io/homelab/stack/issues/625))

**Files:**
- No source changes. Runs `dev/scripts/fetch_nprm_addenda.py` (existing, idempotent: attach is guarded by title-dedupe, tags via `add_tag`'s INSERT OR IGNORE). Data: `data/bib.sqlite`.

**Interfaces:**
- Consumes: `fetch_nprm_addenda.py --year {2026,2027} --register`; `CONFIGS[year].register_tags` = `sup:<year>_PFS_NPRM`, `module:pfs`, `file:rvu`, `year:<year>`.
- Produces: `2KVJ2HKX` and `5ITGVDJV` carry their full register tag sets again; `pfs.nprm._addendum_b_path(year)` resolves for both years. Task 4 keys its sync on these tags.

**Precondition: Task 1 is committed** (mail-poller picks the fixed code up live; the repair must not be re-wipeable).

- [x] **Step 1: Re-run registration for both years** (ZIPs already on disk → download step self-skips; attach guard prints `SKIP attach`):

Run: `uv run python dev/scripts/fetch_nprm_addenda.py --year 2026 --register && uv run python dev/scripts/fetch_nprm_addenda.py --year 2027 --register`
Expected: both print `SKIP attach — ... already has an attachment` and `Current tags on <key>: [...]` including the `sup:` tag. If either *attaches* instead of skipping, STOP — the guard regressed; investigate before continuing.

- [x] **Step 2: Verify tag restoration + loader discovery**

```bash
uv run python - <<'EOF'
from conf import connect
from pfs import nprm
store = connect.bib()
for year, tag, key in ((2026, "sup:2026_PFS_NPRM", "2KVJ2HKX"), (2027, "sup:2027_PFS_NPRM", "5ITGVDJV")):
    items = store.list_items(tag=tag)
    assert [i.key for i in items] == [key], (tag, [i.key for i in items])
    path = nprm._addendum_b_path(year)
    print(year, "OK:", key, "->", path)
EOF
```
Expected: `2026 OK: ...` and `2027 OK: ...` with real xlsx paths.

- [x] **Step 3: Simulate the wiper to prove the fix holds** (read-only on real data — run the exact upsert the comment farm performs against a *copy* of bib.sqlite, assert `sup:2027_PFS_NPRM` survives):

```bash
cp data/bib.sqlite /tmp/claude-1000/-home-kert-stack/e01452a7-3395-4fb6-bd66-ed6fd8c06398/scratchpad/bib_copy.sqlite
uv run python - <<'EOF'
from bib.store import Store
from bib.translate import federal_register
s = Store("/tmp/claude-1000/-home-kert-stack/e01452a7-3395-4fb6-bd66-ed6fd8c06398/scratchpad/bib_copy.sqlite")
rule = federal_register("https://www.federalregister.gov/documents/2026-14327")
rule.add_tag("source:federal-register")
rule.add_tag("module:pfs")
key = s.upsert(rule)
assert key == "5ITGVDJV", key
assert "sup:2027_PFS_NPRM" in s.get(key).tags
print("survives re-ingest OK")
EOF
```
Expected: `survives re-ingest OK`. (Network hiccup on the FR fetch → retry; this step is diagnostic only.)

- [x] **Step 4: Comment the verification output on [#625](https://git.fhirworx.io/homelab/stack/issues/625) and close it** (data-only task, nothing to commit).

---

### Task 4: Sync the CY2027 NPRM into Zotero ([#626](https://git.fhirworx.io/homelab/stack/issues/626))

**Files:**
- No source changes. Data: `data/zotero/data/zotero.sqlite`. Requires docker on the host (CLI default `--hold-zotero` stops/starts the `zotero` container around the write).

**Interfaces:**
- Consumes: restored `sup:` tags from Task 3; `stack bib sync-zotero --tag <t>`.
- Produces: Zotero item `5ITGVDJV` with 3 child attachments; refreshed tag set on Zotero's `2KVJ2HKX`.

- [x] **Step 1: Confirm the zotero container is running and no notebooks-integration run is active** (`docker ps --format '{{.Names}}' | grep zotero`; the nightly runs at 03:30/04:15 — any daytime run is clear).

- [x] **Step 2: Push both years' tag scopes**

Run: `uv run stack bib sync-zotero --tag sup:2027_PFS_NPRM && uv run stack bib sync-zotero --tag sup:2026_PFS_NPRM`
Expected: each prints sync stats (`created: 1 …` for 2027; `skipped: 1` + tag refresh for 2026); zotero container restarted after each (CLI handles it).

- [x] **Step 3: Verify in zotero.sqlite**

```bash
uv run python - <<'EOF'
import sqlite3
z = sqlite3.connect("file:data/zotero/data/zotero.sqlite?mode=ro&immutable=1", uri=True)
iid = z.execute("SELECT itemID FROM items WHERE key='5ITGVDJV'").fetchone()
assert iid, "5ITGVDJV missing from Zotero"
kids = list(z.execute("SELECT path FROM itemAttachments WHERE parentItemID=?", (iid[0],)))
assert len(kids) == 3, kids
tags = {r[0] for r in z.execute("SELECT t.name FROM itemTags it JOIN tags t ON t.tagID=it.tagID WHERE it.itemID=?", (iid[0],))}
assert "sup:2027_PFS_NPRM" in tags, tags
t26 = {r[0] for r in z.execute("SELECT t.name FROM itemTags it JOIN tags t ON t.tagID=it.tagID JOIN items i ON i.itemID=it.itemID WHERE i.key='2KVJ2HKX'")}
assert "sup:2026_PFS_NPRM" in t26 and "file:rvu" in t26, t26
print("Zotero verified: 5ITGVDJV +", len(kids), "attachments; tags OK both years")
EOF
```
Expected: `Zotero verified: 5ITGVDJV + 3 attachments; tags OK both years`. Also confirm zotero container is back up (`docker ps | grep zotero`).

- [x] **Step 4: Comment verification on [#626](https://git.fhirworx.io/homelab/stack/issues/626) and close it.**

---

### Task 5: Close the loop

**Files:**
- Modify: `docs/superpowers/plans/2026-08-17-zotero-rule-sync-p38.md` (check boxes)
- Memory: `~/.claude/projects/-home-kert-stack/memory/` (update `cms_mail_pipeline.md`; new incident memory)

- [x] **Step 1: Push commits** (`git push`), confirm CI green on the two commits (or file/watch per gitea-tracker-ops; the auto-filer leaves stale failure issues — check latest run on HEAD).
- [x] **Step 2: Close [#624](https://git.fhirworx.io/homelab/stack/issues/624) and [#627](https://git.fhirworx.io/homelab/stack/issues/627)** if the `closes #N` footers didn't auto-close them (Gitea closes on push to main), each with a comment linking the commit SHA.
- [x] **Step 3: Close the [P38 milestone](https://git.fhirworx.io/homelab/stack/milestone/39).**
- [x] **Step 4: Append a "P38 build outcomes" section** to this plan (what shipped, SHAs, verification results — pattern from P36/P37 specs), commit as `docs(plan): P38 outcomes (refs #624-#627)`.
- [x] **Step 5: Update memory** — `cms_mail_pipeline.md` gains the rules-scope nightly step; new memory `zotero-rule-sync-incident.md` (upsert used to clobber curated tags; sup: tags are load-bearing for pfs.nprm discovery; sync coverage now email + federal-register).

## P38 build outcomes (2026-08-17)

Issues [#624](https://git.fhirworx.io/homelab/stack/issues/624)–[#628](https://git.fhirworx.io/homelab/stack/issues/628), milestone [P38](https://git.fhirworx.io/homelab/stack/milestone/39). Executed same-day on main (deliberate worktree deviation: mail-poller live-mounts this checkout's `./src`, and Tasks 3–4 operate on the live databases here).

- **#624 fixed (91c3306)**: `Store.upsert` now unions existing ∪ incoming tags and collections on the URL-dedupe path; two regression tests (both red pre-fix). 137 store+sync tests green.
- **#627 fixed (c9e8e1e)**: `_gen_zotero_sync` emits a second nightly step, `sync-zotero --tag source:federal-register --no-hold` (78 items); regenerated `zotero-sync.yml`; generator deterministic.
- **#625 repaired (data-only, closed with verification)**: re-ran `fetch_nprm_addenda --year {2026,2027} --register` — attach guards held (`SKIP attach`), full tag sets restored on `2KVJ2HKX` and `5ITGVDJV`; `_discover_addendum_b` resolves for both years; wiper simulation on a bib copy (comment-farm-style upsert, same URL + canonical FR tag set) confirms `sup:`/`file:rvu` survive.
- **#628 found en route + fixed (18c1066)**: P37's worktree run left a dead `.worktrees/p37/...` `storage_path` on the CY2027 Addendum B attachment (`attach_file` stored the unresolved path; file itself was safe in canonical storage). `attach_file` now stores `dest.resolve()` (regression test with symlinked storage dir); the single affected row rewritten. Audit: no other `.worktrees` paths.
- **#626 done (closed with verification)**: `sync-zotero --tag sup:2027_PFS_NPRM` → created 1 item + 9 tags + 3 attachments; `--tag sup:2026_PFS_NPRM` → skipped (Zotero's snapshot already matched). zotero.sqlite now holds `5ITGVDJV` (statute-typed, 91 FR 43842, field parity with 2KVJ2HKX) with all 3 files physically in storage (39 MB PDF, 2.6 MB TXT, 1.2 MB Addendum B xlsx). Zotero container held during sync, restarted healthy.
- Plan deviations: the Task 3 wiper simulation ran offline (the FR API 404s on the bare-document-number URL form; same upsert, locally constructed item). `nature.csl` working-tree drift observed (Zotero app self-update, container-owned) — left uncommitted.

## Self-Review

- Spec coverage: #624→Task 1, #625→Task 3, #626→Task 4, #627→Task 2, closure/tracking→Task 5. No gaps.
- Placeholder scan: none — every step has runnable code/commands and expected output.
- Type consistency: `Store.get` returns `Item` with `.tags`/`.collections` lists (verified in src); `list_items(tag=)` verified; `nprm._addendum_b_path(year)` verified private helper exists.
- Ordering hazard called out twice (Task 1 before Task 3) since mail-poller runs live-mounted source every 600s.