stack/docs/superpowers/plans/2026-08-26-zotero-dedupe.md

# Zotero Library Dedupe (fidelity-preserving) Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Remove every duplicate item from the Zotero library by *merging* (tags, collections, notes, attachments, relations) into a canonical copy — never by blind delete — and close the three ingest/ops holes that created them.

**Architecture:** A new `zot.merge` module provides a single `merge_into(db, keeper_id, loser_id)` primitive plus a URL-cluster driver. The nightly zotero-sync workflow calls the driver library-wide (replacing the lossy title+date `dedupe_collection` pass). Mail ingest gains content-hash aliasing so listserv re-sends stop creating new items. A one-off runbook (Task 5) executes the historical cleanup against the live DB inside the usual zotero-stop window, dry-run first, with a full backup.

**Tech Stack:** Python 3 / sqlite3, existing `zot.db.Db` wrapper, `bib.Store`, pytest (`tests/zot`, `tests/bib`), Gitea workflow `zotero-sync.yml`.

**Spec:** Root-cause findings below (this section *is* the spec; no separate doc).

## Root-cause findings (evidence, 2026-08-26)

Measured on a snapshot of `data/zotero/data/zotero.sqlite` (287,735 items; 199,139 top-level non-deleted) and `data/bib.sqlite` (219,215 items):

- **Class A — restore twins, ~8,981 surplus items in 8,955 URL clusters.** Live copies' keys contain no `L`; 8,938 twins' keys do; 8,947 twins were physically inserted 2026-05-14 13:41–13:46 (clientDateModified) with **original dateAdded preserved** and high itemIDs (~371k–376k vs 3k–48k for the live copies). Timeline: `zotero.sqlite.pre-rekey.bak` created 2026-05-13 16:03; commit `8149da5` (2026-05-13 16:40) dropped `L` from the Zotero key charset; live rows were rekeyed in place; then an uncommitted one-off on 05-14 restored rows from the pre-rekey backup **matching by key** — every rekeyed row looked "missing" and was re-inserted under its old L-key, children included (22,734 child rows carry the 05-14 stamp). `bib.sync` matches by URL and updates only the first-found copy, so twins hold stale tags (avg 5.1 vs 15.5) **but ~23% of sampled clusters have a twin with MORE child notes than the keeper, and a few have more attachments** — blind deletion loses data. Cluster types: 6,374 document, 2,202 journalArticle, 182 statute, 152 webpage, 44 report, 1 preprint.
- **Class B — listserv re-sends (bib-level): 17 title clusters / 34 surplus rows in bib** (e.g. "Upcoming iQIES Service Center Hold Times" ×8). Mail ingest keys identity on `email:<Message-ID>`; CMS re-sends identical content with a fresh Message-ID on different days. The nightly `--dedupe-collection Cmsupdates` pass groups on (title, **date**), so cross-day re-sends never group; it is also scoped to one collection and deletes without merging.
- **Class C — legacy ingest URL dups in bib: 3 clusters (~17 surplus)** (`aco-reach` ×12, `county-level-aggregate…` ×5, one ×3), created 2026-03-04 before `91c3306` made `Store.upsert` merge by URL. Mirrored into Zotero as same-URL rows (a Class A-shaped symptom with a different origin — the URL-cluster merge handles both).
- **Class D — same-DOI pairs: 2,080 clusters / 2,081 surplus**, zero PMID collisions — i.e. PubMed's own dual records (different PMIDs, one DOI) plus Class A twins. After Class A/C merging, the residue is a *scientific-record* judgment call → report-only triage artifact, no automated merge.
- **Protected, NOT duplicates:** (1) ~11k Zotero-only personal/archive items from the host-library migration (audioRecording 1,018, dictionaryEntry 744, artwork 70, plus personal journalArticles) — "absent from bib" must never be treated as a dup signal; (2) same-title different-PMID records (e.g. Cochrane review updates titled "Skin grafting for venous leg ulcers.") — title is not identity; (3) journalArticle rows whose `url` field holds page-range junk ("997-998") — only `http*`/`email:*` URLs count as identity.

## Global Constraints

- Zotero DB writes only inside a stop/start window: `docker stop zotero` … `docker start zotero` (same discipline as `bib.sync`; see `zotero-sync.yml`).
- Every live-DB mutation is preceded by a timestamped `.bak` copy and a dry-run report; merge decisions are logged to `.state/zotero-dedupe/` as JSON before any delete.
- Identity for clustering = the `url` itemData field, only when it starts with `http` or `email:`.
- Keeper = lowest `itemID` in a cluster (what `find_item_by_url` returns once made deterministic — Task 2).
- Clusters whose members differ in `itemTypeID` are skipped and reported, never auto-merged.
- Run everything with `uv run`; tests with targeted files (`uv run pytest tests/zot/test_merge.py -v`), not the full suite.
- Before every commit: `git status --short` (concurrent sessions share this worktree). No Co-Authored-By trailer.

---

### Task 1: `zot.merge` — merge primitive + URL-cluster driver

**Files:**
- Create: `src/zot/merge.py`
- Test: `tests/zot/test_merge.py`

**Interfaces:**
- Consumes: `zot.db.Db`, `zot.schema.create_db`, `zot.db.TYPE_MAP`.
- Produces:
  - `merge_into(db: Db, keeper_id: int, loser_id: int) -> dict[str, int]` — unions tags/collections/relations, reparents missing child notes (dedup by note-text hash) and attachments (dedup by path/filename), then hard-deletes the loser row. Returns per-facet counts.
  - `find_url_clusters(db: Db) -> list[list[tuple[int, str, int]]]` — clusters of `(itemID, key, itemTypeID)` sharing an `http*`/`email:*` url, sorted by itemID.
  - `merge_url_duplicates(db_path, *, dry_run: bool = True, log_dir: str | Path = ".state/zotero-dedupe") -> dict` — driver; stats `{clusters, merged, skipped_type_mismatch, notes_adopted, attachments_adopted, tags_added, collections_added}` and writes `merge-log-<runid>.json`.

- [ ] **Step 1: Write the failing tests**

```python
# tests/zot/test_merge.py
"""Merge-based dedupe: fidelity facets union into the keeper."""
import json
import sqlite3
from pathlib import Path

import pytest

from zot.db import TYPE_MAP, Db
from zot.schema import create_db


@pytest.fixture
def dup_db(tmp_path: Path) -> Path:
    p = tmp_path / "zotero.sqlite"
    create_db(str(p)).close()
    with Db(str(p)) as db:
        keeper = db.create_item(TYPE_MAP["document"], key="AAAAAAAA")
        loser = db.create_item(TYPE_MAP["document"], key="BBBBBBBB")
        for iid in (keeper, loser):
            db.set_field(iid, "url", "https://example.org/x")
        db.set_field(keeper, "title", "X")
        db.set_field(loser, "title", "X (stale)")
        db.sync_tags(keeper, ["shared", "keeper-only"])
        db.sync_tags(loser, ["shared", "loser-only"])
        # loser-only child note
        nid = db.create_item(TYPE_MAP["note"], key="CCCCCCCC")
        db.con.execute(
            "INSERT INTO itemNotes (itemID, parentItemID, note, title) VALUES (?,?,?,?)",
            (nid, loser, "<p>loser note</p>", ""),
        )
        # identical note on both — must NOT be doubled
        for parent, key in ((keeper, "DDDDDDDD"), (loser, "EEEEEEEE")):
            dn = db.create_item(TYPE_MAP["note"], key=key)
            db.con.execute(
                "INSERT INTO itemNotes (itemID, parentItemID, note, title) VALUES (?,?,?,?)",
                (dn, parent, "<p>same on both</p>", ""),
            )
        db.commit()
    return p


def _tags(db: Db, item_id: int) -> set[str]:
    return {
        r[0]
        for r in db.con.execute(
            "SELECT t.name FROM itemTags it JOIN tags t ON t.tagID=it.tagID"
            " WHERE it.itemID=?",
            (item_id,),
        )
    }


def _notes(db: Db, item_id: int) -> list[str]:
    return [
        r[0]
        for r in db.con.execute(
            "SELECT note FROM itemNotes WHERE parentItemID=?", (item_id,)
        )
    ]


class TestMergeInto:
    def test_unions_tags_and_adopts_missing_notes(self, dup_db):
        from zot.merge import merge_into

        with Db(str(dup_db)) as db:
            keeper = db.find_item_by_key("AAAAAAAA")
            loser = db.find_item_by_key("BBBBBBBB")
            merge_into(db, keeper, loser)
            db.commit()
            assert _tags(db, keeper) >= {"shared", "keeper-only", "loser-only"}
            notes = _notes(db, keeper)
            assert "<p>loser note</p>" in notes
            assert notes.count("<p>same on both</p>") == 1
            assert db.find_item_by_key("BBBBBBBB") is None

    def test_keeper_fields_win(self, dup_db):
        from zot.merge import merge_into

        with Db(str(dup_db)) as db:
            keeper = db.find_item_by_key("AAAAAAAA")
            loser = db.find_item_by_key("BBBBBBBB")
            merge_into(db, keeper, loser)
            row = db.con.execute(
                "SELECT idv.value FROM itemData id"
                " JOIN fields f ON f.fieldID=id.fieldID"
                " JOIN itemDataValues idv ON idv.valueID=id.valueID"
                " WHERE id.itemID=? AND f.fieldName='title'",
                (keeper,),
            ).fetchone()
            assert row[0] == "X"


class TestDriver:
    def test_dry_run_reports_without_deleting(self, dup_db, tmp_path):
        from zot.merge import merge_url_duplicates

        stats = merge_url_duplicates(dup_db, dry_run=True, log_dir=tmp_path)
        assert stats["clusters"] == 1
        with Db(str(dup_db)) as db:
            assert db.find_item_by_key("BBBBBBBB") is not None
        logs = list(tmp_path.glob("merge-log-*.json"))
        assert logs and json.loads(logs[0].read_text())[0]["keeper_key"] == "AAAAAAAA"

    def test_live_run_merges(self, dup_db, tmp_path):
        from zot.merge import merge_url_duplicates

        stats = merge_url_duplicates(dup_db, dry_run=False, log_dir=tmp_path)
        assert stats["merged"] == 1
        with Db(str(dup_db)) as db:
            assert db.find_item_by_key("BBBBBBBB") is None

    def test_type_mismatch_skipped(self, tmp_path):
        from zot.merge import merge_url_duplicates

        p = tmp_path / "z.sqlite"
        create_db(str(p)).close()
        with Db(str(p)) as db:
            a = db.create_item(TYPE_MAP["document"], key="AAAAAAAA")
            b = db.create_item(TYPE_MAP["journalArticle"], key="BBBBBBBB")
            for iid in (a, b):
                db.set_field(iid, "url", "https://example.org/y")
            db.commit()
        stats = merge_url_duplicates(p, dry_run=False, log_dir=tmp_path)
        assert stats["skipped_type_mismatch"] == 1
        with Db(str(p)) as db:
            assert db.find_item_by_key("BBBBBBBB") is not None

    def test_junk_urls_ignored(self, tmp_path):
        from zot.merge import merge_url_duplicates

        p = tmp_path / "z.sqlite"
        create_db(str(p)).close()
        with Db(str(p)) as db:
            a = db.create_item(TYPE_MAP["journalArticle"], key="AAAAAAAA")
            b = db.create_item(TYPE_MAP["journalArticle"], key="BBBBBBBB")
            for iid in (a, b):
                db.set_field(iid, "url", "997-998")
            db.commit()
        stats = merge_url_duplicates(p, dry_run=False, log_dir=tmp_path)
        assert stats["clusters"] == 0
```

- [ ] **Step 2: Run tests to verify they fail**

Run: `uv run pytest tests/zot/test_merge.py -v`
Expected: FAIL — `ModuleNotFoundError: No module named 'zot.merge'`

- [ ] **Step 3: Implement `src/zot/merge.py`**

```python
"""Merge-based Zotero dedupe.

Identity = the ``url`` itemData field (only ``http*``/``email:*``
values). The keeper is the lowest ``itemID`` — the row a deterministic
``find_item_by_url`` returns, i.e. the copy ``bib.sync`` has been
updating. Losers are merged facet-by-facet into the keeper, then
hard-deleted; every decision is logged as JSON before any delete.

Born from the 2026-05-14 incident where a key-matched restore from
``pre-rekey.bak`` re-inserted ~9k rekeyed rows as URL-twins (issue in
tracker; plan: docs/superpowers/plans/2026-08-26-zotero-dedupe.md).
"""

from __future__ import annotations

import hashlib
import json
from collections import defaultdict
from pathlib import Path

from zot.db import Db, now_iso

URL_FIELD_PREFIXES = ("http", "email:")


def _note_hash(text: str) -> str:
    return hashlib.sha1(" ".join((text or "").split()).encode()).hexdigest()


def merge_into(db: Db, keeper_id: int, loser_id: int) -> dict[str, int]:
    """Union loser's facets into keeper, then delete the loser row."""
    out = {"tags_added": 0, "collections_added": 0, "notes_adopted": 0,
           "attachments_adopted": 0, "relations_moved": 0}
    con = db.con

    for (tag_id,) in con.execute(
        "SELECT tagID FROM itemTags WHERE itemID=?", (loser_id,)
    ).fetchall():
        cur = con.execute(
            "INSERT OR IGNORE INTO itemTags (itemID, tagID, type) VALUES (?,?,0)",
            (keeper_id, tag_id),
        )
        out["tags_added"] += cur.rowcount

    for (coll_id,) in con.execute(
        "SELECT collectionID FROM collectionItems WHERE itemID=?", (loser_id,)
    ).fetchall():
        cur = con.execute(
            "INSERT OR IGNORE INTO collectionItems (collectionID, itemID)"
            " VALUES (?,?)",
            (coll_id, keeper_id),
        )
        out["collections_added"] += cur.rowcount

    keeper_notes = {
        _note_hash(r[0])
        for r in con.execute(
            "SELECT note FROM itemNotes WHERE parentItemID=?", (keeper_id,)
        )
    }
    for note_item, note in con.execute(
        "SELECT itemID, note FROM itemNotes WHERE parentItemID=?", (loser_id,)
    ).fetchall():
        if _note_hash(note) in keeper_notes:
            db.delete_item(note_item)
            continue
        con.execute(
            "UPDATE itemNotes SET parentItemID=? WHERE itemID=?",
            (keeper_id, note_item),
        )
        out["notes_adopted"] += 1

    keeper_atts = {
        r[0]
        for r in con.execute(
            "SELECT COALESCE(path,'') FROM itemAttachments WHERE parentItemID=?",
            (keeper_id,),
        )
    }
    for att_item, path in con.execute(
        "SELECT itemID, COALESCE(path,'') FROM itemAttachments"
        " WHERE parentItemID=?",
        (loser_id,),
    ).fetchall():
        if path and path in keeper_atts:
            db.delete_item(att_item)
            continue
        con.execute(
            "UPDATE itemAttachments SET parentItemID=? WHERE itemID=?",
            (keeper_id, att_item),
        )
        out["attachments_adopted"] += 1

    cur = con.execute(
        "UPDATE OR IGNORE itemRelations SET itemID=? WHERE itemID=?",
        (keeper_id, loser_id),
    )
    out["relations_moved"] = cur.rowcount

    # Keeper has no creators but loser does → adopt them wholesale.
    has_keeper = con.execute(
        "SELECT 1 FROM itemCreators WHERE itemID=? LIMIT 1", (keeper_id,)
    ).fetchone()
    if not has_keeper:
        con.execute(
            "UPDATE OR IGNORE itemCreators SET itemID=? WHERE itemID=?",
            (keeper_id, loser_id),
        )

    db.delete_item(loser_id)
    return out


def find_url_clusters(db: Db) -> list[list[tuple[int, str, int]]]:
    rows = db.con.execute(
        """
        SELECT i.itemID, i.key, i.itemTypeID, idv.value
        FROM items i
        JOIN itemData id ON id.itemID = i.itemID
        JOIN fields f ON f.fieldID = id.fieldID AND f.fieldName = 'url'
        JOIN itemDataValues idv ON idv.valueID = id.valueID
        WHERE i.itemID NOT IN (SELECT itemID FROM deletedItems)
          AND i.itemID NOT IN (SELECT itemID FROM itemAttachments)
          AND i.itemID NOT IN (SELECT itemID FROM itemNotes)
        ORDER BY i.itemID
        """
    ).fetchall()
    by_url: dict[str, list[tuple[int, str, int]]] = defaultdict(list)
    for item_id, key, type_id, url in rows:
        if url and url.startswith(URL_FIELD_PREFIXES):
            by_url[url].append((item_id, key, type_id))
    return [v for v in by_url.values() if len(v) > 1]


def merge_url_duplicates(
    db_path: str | Path,
    *,
    dry_run: bool = True,
    log_dir: str | Path = ".state/zotero-dedupe",
) -> dict:
    stats = {"clusters": 0, "merged": 0, "skipped_type_mismatch": 0,
             "tags_added": 0, "collections_added": 0, "notes_adopted": 0,
             "attachments_adopted": 0, "relations_moved": 0}
    log: list[dict] = []
    log_dir = Path(log_dir)
    log_dir.mkdir(parents=True, exist_ok=True)
    run_id = now_iso().replace(" ", "T").replace(":", "")

    with Db(str(db_path)) as db:
        clusters = find_url_clusters(db)
        for members in clusters:
            keeper = members[0]
            losers = members[1:]
            if any(m[2] != keeper[2] for m in losers):
                stats["skipped_type_mismatch"] += 1
                log.append({"keeper_key": keeper[1], "action": "skip-type-mismatch",
                            "loser_keys": [m[1] for m in losers]})
                continue
            stats["clusters"] += 1
            entry = {"keeper_key": keeper[1],
                     "loser_keys": [m[1] for m in losers],
                     "action": "dry-run" if dry_run else "merged"}
            if not dry_run:
                for m in losers:
                    for k, v in merge_into(db, keeper[0], m[0]).items():
                        stats[k] += v
                    stats["merged"] += 1
            log.append(entry)
        if not dry_run:
            db.commit()

    (log_dir / f"merge-log-{run_id}.json").write_text(json.dumps(log, indent=1))
    return stats
```

Note: if `Db.delete_item` does not already cascade a top-level item's remaining children (check `src/zot/db.py`), extend `merge_into` to delete leftover child rows of the loser before `db.delete_item(loser_id)` — the tests' `find_item_by_key` assertion will catch orphans via FK errors.

- [ ] **Step 4: Run tests to verify they pass**

Run: `uv run pytest tests/zot/test_merge.py -v`
Expected: 6 PASS

- [ ] **Step 5: Commit**

```bash
git status --short   # foreign staged work check
git add src/zot/merge.py tests/zot/test_merge.py
git commit -m "feat(zot): merge-based URL dedupe engine — union facets, keep-earliest (refs #665)"
```

---

### Task 2: Deterministic URL lookup + nightly library-wide merge

**Files:**
- Modify: `src/zot/db.py:391-400` (`find_item_by_url`)
- Modify: `src/cli/bib.py:506-543` (zotero-sync command options)
- Modify: `.gitea/workflows/zotero-sync.yml:27-34`
- Test: `tests/zot/test_db.py`, `tests/cli/` (existing zotero-sync CLI test file if present, else add asserts in `tests/zot/test_merge.py`)

**Interfaces:**
- Consumes: `zot.merge.merge_url_duplicates` (Task 1 signature).
- Produces: `find_item_by_url` now `ORDER BY i.itemID` (bib.sync keeps updating the merge keeper); CLI flag `--merge-dupes/--no-merge-dupes` (default off) on `stack bib zotero-sync`, running the library-wide merge after the push, inside the same stop window. `--dedupe-collection` stays but is documented as deprecated in its help string.

- [ ] **Step 1: Write the failing test**

```python
# append to tests/zot/test_db.py
class TestFindItemByUrlDeterministic:
    def test_returns_lowest_item_id(self, tmp_path):
        from zot.db import TYPE_MAP, Db
        from zot.schema import create_db

        p = tmp_path / "z.sqlite"
        create_db(str(p)).close()
        with Db(str(p)) as db:
            a = db.create_item(TYPE_MAP["document"], key="AAAAAAAA")
            b = db.create_item(TYPE_MAP["document"], key="BBBBBBBB")
            # insert the URL row for the HIGHER itemID first so a
            # scan-order query would return it
            db.set_field(b, "url", "https://example.org/det")
            db.set_field(a, "url", "https://example.org/det")
            db.commit()
            assert db.find_item_by_url("https://example.org/det") == a
```

- [ ] **Step 2: Run test to verify it fails**

Run: `uv run pytest tests/zot/test_db.py -k Deterministic -v`
Expected: FAIL (returns `b`, the first row in scan order)

- [ ] **Step 3: Add `ORDER BY i.itemID` to the query in `find_item_by_url`**

```python
            """SELECT i.itemID FROM items i
               JOIN itemData id ON i.itemID = id.itemID
               JOIN itemDataValues idv ON id.valueID = idv.valueID
               WHERE id.fieldID = ? AND idv.value = ?
               ORDER BY i.itemID""",
```

- [ ] **Step 4: Run test to verify it passes**

Run: `uv run pytest tests/zot/test_db.py -k Deterministic -v` → PASS

- [ ] **Step 5: Wire `--merge-dupes` into the CLI and workflow**

In `src/cli/bib.py` next to the existing `dedupe_collection` option (line ~506):

```python
    merge_dupes: bool = typer.Option(
        False,
        "--merge-dupes/--no-merge-dupes",
        help="After the push, merge same-URL duplicate items library-wide "
        "(union tags/collections/notes/attachments; keep-earliest). "
        "Supersedes --dedupe-collection, which deletes without merging.",
    ),
```

and after the existing `if dedupe_collection:` block (line ~542):

```python
        if merge_dupes:
            from zot.merge import merge_url_duplicates

            stats |= {f"merge_{k}": v for k, v in merge_url_duplicates(
                zotero_db_path, dry_run=False
            ).items()}
```

(resolve `zotero_db_path` the same way the surrounding command does — it already computes the Zotero DB path for the push; reuse that variable name.)

In `.gitea/workflows/zotero-sync.yml`, replace `--dedupe-collection Cmsupdates` with `--merge-dupes` and update the comment block (lines 27–34) to say: listserv re-sends are now aliased at ingest (Task 3) and any same-URL twins are merged, not deleted.

- [ ] **Step 6: Run the CLI help to verify wiring**

Run: `uv run stack bib zotero-sync --help`
Expected: `--merge-dupes` listed; exit 0.

- [ ] **Step 7: Commit**

```bash
git status --short
git add src/zot/db.py tests/zot/test_db.py src/cli/bib.py .gitea/workflows/zotero-sync.yml
git commit -m "feat(bib,zot): deterministic URL lookup + nightly library-wide merge-dupes (refs #665)"
```

---

### Task 3: Alias listserv re-sends at mail ingest (content-hash identity)

**Files:**
- Modify: `src/bib/email_ingest.py:108-150` (`_ingest_message`)
- Test: `tests/bib/test_email_ingest.py` (extend the existing file)

**Interfaces:**
- Consumes: `bib.Store.upsert(item) -> str`, `Source(title=..., url=...)` with `extra_json` packing via `to_row()`.
- Produces: new items carry `extra_json["body_sha1"]`; a re-send whose `(title, body_sha1)` matches an existing `source:email` item does NOT create a new item — its Message-ID is appended to the keeper's `extra_json["alias_mids"]` list and its tags are merged via the normal upsert-by-url path (we upsert with the *keeper's* url).

- [ ] **Step 1: Write the failing test**

```python
# append to tests/bib/test_email_ingest.py (reuse that file's existing
# fixtures for Store and for building EmailMessage objects; the ones
# below assume a `store` fixture and a `make_msg(subject, body, mid, date)`
# helper — add the helper if the file lacks one)
import email.message
import json


def make_msg(subject: str, body: str, mid: str, date: str):
    m = email.message.EmailMessage()
    m["Subject"] = subject
    m["From"] = "CMS Updates <cmslists@subscriptions.cms.hhs.gov>"
    m["Message-ID"] = f"<{mid}>"
    m["Date"] = date
    m.set_content(body)
    return m


class TestResendAliasing:
    def test_same_body_new_mid_aliases_instead_of_new_item(
        self, store, mailbox, tmp_path
    ):
        from bib.email_ingest import _ingest_message

        m1 = make_msg("Upcoming iQIES Hold Times", "Same body.",
                      "aaa@x.example", "Mon, 27 Apr 2026 10:00:00 -0400")
        m2 = make_msg("Upcoming iQIES Hold Times", "Same body.",
                      "bbb@x.example", "Fri, 10 Jul 2026 10:00:00 -0400")
        _ingest_message(store, mailbox, m1, tmp_path)
        _ingest_message(store, mailbox, m2, tmp_path)
        items = store.list_items(query="iQIES")
        assert len(items) == 1
        ej = json.loads(items[0].to_row()["extra_json"])
        assert "bbb@x.example" in ej.get("alias_mids", [])

    def test_different_body_same_subject_stays_separate(
        self, store, mailbox, tmp_path
    ):
        from bib.email_ingest import _ingest_message

        m1 = make_msg("Weekly Digest", "Body one.",
                      "ccc@x.example", "Mon, 27 Apr 2026 10:00:00 -0400")
        m2 = make_msg("Weekly Digest", "Body two.",
                      "ddd@x.example", "Fri, 10 Jul 2026 10:00:00 -0400")
        _ingest_message(store, mailbox, m1, tmp_path)
        _ingest_message(store, mailbox, m2, tmp_path)
        assert len(store.list_items(query="Weekly Digest")) == 2
```

(If `tests/bib/test_email_ingest.py` has no `store`/`mailbox` fixtures, copy the setup its existing tests use — the file already exercises `_ingest_message`.)

- [ ] **Step 2: Run tests to verify they fail**

Run: `uv run pytest tests/bib/test_email_ingest.py -k Resend -v`
Expected: FAIL — two items created, no `alias_mids`.

- [ ] **Step 3: Implement content-hash aliasing in `_ingest_message`**

After `body = _extract_body(msg)` and before `item = Source(...)`:

```python
    body_sha1 = hashlib.sha1(" ".join((body or "").split()).encode()).hexdigest()

    # Listservs re-send identical content under a fresh Message-ID
    # (Class B in docs/superpowers/plans/2026-08-26-zotero-dedupe.md).
    # If an item with this exact title+body already exists, treat this
    # message as an alias of it rather than a new Source.
    con = store._con()
    dup = con.execute(
        "SELECT key, url, extra_json FROM items"
        " WHERE title = ? AND extra_json LIKE ?",
        (subject[:255], f'%"body_sha1": "{body_sha1}"%'),
    ).fetchone()
    if dup:
        ej = json.loads(dup["extra_json"] or "{}")
        aliases = ej.get("alias_mids", [])
        if raw_mid not in aliases:
            aliases.append(raw_mid)
            ej["alias_mids"] = aliases
            store.update(dup["key"], extra_json=json.dumps(ej))
        return 0
```

and record the hash on new items (next to the existing `item.doc_type = "Email"` block):

```python
    item.extra_json_fields = {"body_sha1": body_sha1}
```

— use whatever mechanism `Source`/`Item.to_row()` already packs unknown fields into `extra_json` (see `src/bib/item.py:81`: "are packed into `extra_json`"); if the model expects direct attribute assignment (`item.body_sha1 = body_sha1`), do that instead. Add `import hashlib`, `import json` at the top of `email_ingest.py` if absent.

- [ ] **Step 4: Run tests to verify they pass**

Run: `uv run pytest tests/bib/test_email_ingest.py -v`
Expected: new tests PASS, existing tests still PASS.

- [ ] **Step 5: Commit**

```bash
git status --short
git add src/bib/email_ingest.py tests/bib/test_email_ingest.py
git commit -m "feat(bib): alias listserv re-sends by body hash — one item per content (refs #665)"
```

---

### Task 4: bib-side legacy cleanup — Class B (email re-sends) + Class C (URL dups)

**Files:**
- Create: `dev/scripts/dedupe_bib_legacy.py`
- Test: `tests/bib/test_dedupe_legacy.py`

**Interfaces:**
- Consumes: `bib.Store` (`.get`, `.update`, `.delete`, `._con()`), `zot.merge.merge_into`, `zot.db.Db`.
- Produces: `dedupe_bib(store, *, dry_run=True) -> dict` — importable from the script; groups bib rows (a) by exact `url` (Class C) and (b) by `(title, sha1(abstract))` among `source:email` items (Class B); keep-earliest by `created_at` then rowid; union tags/collections onto the keeper via `store.update`; for email groups append surplus Message-IDs (parsed from each loser's `email:<mid>` url) to keeper `extra_json["alias_mids"]`; delete surplus bib rows. Returns `{url_groups, email_groups, removed, aliased}` plus a `pairs` list of `(keeper_key, loser_key)` for the Zotero-side pass in Task 5.

- [ ] **Step 1: Write the failing test**

```python
# tests/bib/test_dedupe_legacy.py
import json

from bib import Store
from bib.item import Source


def _mk_store(tmp_path):
    return Store(str(tmp_path / "bib.sqlite"))


def test_url_group_keeps_earliest_and_unions_tags(tmp_path):
    from dev.scripts.dedupe_bib_legacy import dedupe_bib

    store = _mk_store(tmp_path)
    a = store.create(Source(title="ACO REACH", url="https://x.test/reach"),
                     tags=["module:aco", "keep-me"])
    # second row with the same url — simulate the pre-#624 upsert
    con = store._con()
    con.execute(
        "INSERT INTO items (key, item_type, title, url, extra_json)"
        " VALUES ('ZZZZZZZ2','source','ACO REACH','https://x.test/reach','{}')"
    )
    con.commit()
    store.add_tag("ZZZZZZZ2", "loser-tag")
    out = dedupe_bib(store, dry_run=False)
    assert out["removed"] == 1
    kept = store.get(a)
    assert "loser-tag" in kept.tags and "keep-me" in kept.tags


def test_email_group_aliases_mids(tmp_path):
    from dev.scripts.dedupe_bib_legacy import dedupe_bib

    store = _mk_store(tmp_path)
    s1 = Source(title="iQIES Hold Times", url="email:aaa@x")
    s1.abstract = "Same body."
    s2 = Source(title="iQIES Hold Times", url="email:bbb@x")
    s2.abstract = "Same body."
    k1 = store.create(s1, tags=["source:email"])
    store.create(s2, tags=["source:email"])
    out = dedupe_bib(store, dry_run=False)
    assert out["removed"] == 1
    ej = json.loads(store.get(k1).to_row()["extra_json"])
    assert "bbb@x" in ej["alias_mids"]


def test_dry_run_touches_nothing(tmp_path):
    from dev.scripts.dedupe_bib_legacy import dedupe_bib

    store = _mk_store(tmp_path)
    s1 = Source(title="T", url="email:a@x"); s1.abstract = "B"
    s2 = Source(title="T", url="email:b@x"); s2.abstract = "B"
    store.create(s1, tags=["source:email"])
    store.create(s2, tags=["source:email"])
    out = dedupe_bib(store, dry_run=True)
    assert out["removed"] == 0 and out["email_groups"] == 1
    assert len(store.list_items(query="T")) == 2
```

(Adjust `Store`/`Source` constructor details to match `tests/bib`'s existing idiom — the suite already builds throwaway stores; mirror it. `store.create` signature is at `src/bib/store.py:98`.)

- [ ] **Step 2: Run tests to verify they fail**

Run: `uv run pytest tests/bib/test_dedupe_legacy.py -v`
Expected: FAIL — module not found.

- [ ] **Step 3: Implement `dev/scripts/dedupe_bib_legacy.py`**

```python
"""One-off: merge legacy duplicate rows inside bib.sqlite.

Class C — same-url rows created before Store.upsert deduped by URL
(91c3306). Class B — listserv re-sends: same (title, body-hash) among
source:email items, each under a distinct email:<Message-ID> url.

Keep-earliest; union tags/collections into the keeper; alias surplus
Message-IDs; delete surplus rows. Prints stats; --dry-run default.
Returns keeper/loser key pairs so the Zotero-side merge (zot.merge)
can be driven off the same decisions.

Usage: uv run python dev/scripts/dedupe_bib_legacy.py [--live]
"""

from __future__ import annotations

import hashlib
import json
import sys
from collections import defaultdict


def _body_hash(abstract: str) -> str:
    return hashlib.sha1(" ".join((abstract or "").split()).encode()).hexdigest()


def dedupe_bib(store, *, dry_run: bool = True) -> dict:
    con = store._con()
    out = {"url_groups": 0, "email_groups": 0, "removed": 0, "aliased": 0,
           "pairs": []}

    groups: list[list[str]] = []  # each: [keeper_key, loser_key, ...]

    # Class C: exact-url groups (http only; email urls are unique per mid)
    by_url = defaultdict(list)
    for key, url in con.execute(
        "SELECT key, url FROM items WHERE url LIKE 'http%'"
        " ORDER BY created_at, id"
    ):
        by_url[url].append(key)
    for keys in by_url.values():
        if len(keys) > 1:
            out["url_groups"] += 1
            groups.append(keys)

    # Class B: (title, body-hash) among email items
    by_content = defaultdict(list)
    for key, title, abstract in con.execute(
        "SELECT i.key, i.title, COALESCE(i.abstract,'') FROM items i"
        " JOIN item_tags it ON it.item_id = i.id"
        " JOIN tags t ON t.id = it.tag_id AND t.name = 'source:email'"
        " ORDER BY i.created_at, i.id"
    ):
        by_content[(title, _body_hash(abstract))].append(key)
    for keys in by_content.values():
        if len(keys) > 1:
            out["email_groups"] += 1
            groups.append(keys)

    for keys in groups:
        keeper_key, losers = keys[0], keys[1:]
        if dry_run:
            continue
        keeper = store.get(keeper_key)
        ej = json.loads(keeper.to_row().get("extra_json", "{}") or "{}")
        aliases = ej.get("alias_mids", [])
        tags = list(keeper.tags)
        collections = list(keeper.collections)
        for lk in losers:
            loser = store.get(lk)
            tags += [t for t in loser.tags if t not in tags]
            collections += [c for c in loser.collections if c not in collections]
            if loser.url.startswith("email:"):
                mid = loser.url.removeprefix("email:")
                if mid not in aliases:
                    aliases.append(mid)
                    out["aliased"] += 1
            out["pairs"].append((keeper_key, lk))
            store.delete(lk)
            out["removed"] += 1
        if aliases:
            ej["alias_mids"] = aliases
        store.update(keeper_key, tags=tags, collections=collections,
                     extra_json=json.dumps(ej))
    return out


if __name__ == "__main__":
    from bib import Store

    store = Store()
    live = "--live" in sys.argv
    out = dedupe_bib(store, dry_run=not live)
    pairs = out.pop("pairs")
    print(json.dumps(out, indent=1))
    for k, l in pairs:
        print(f"pair {k} <- {l}")
```

Check the real bib schema names before running (`src/bib/schema.sql`): the tag join above assumes `item_tags(item_id, tag_id)` / `tags(id, name)`; fix to match the actual DDL. `store.update`'s accepted kwargs are at `src/bib/store.py:169` — if it does not take `collections`/`extra_json` directly, route through the same row dict `upsert` uses (`item.to_row()` keys).

- [ ] **Step 4: Run tests to verify they pass**

Run: `uv run pytest tests/bib/test_dedupe_legacy.py -v` → 3 PASS

- [ ] **Step 5: Commit**

```bash
git status --short
git add dev/scripts/dedupe_bib_legacy.py tests/bib/test_dedupe_legacy.py
git commit -m "feat(bib): legacy dedupe script — merge same-url and re-sent-email rows (refs #665)"
```

---

### Task 5: Execute the cleanup on the live library (runbook)

**Files:**
- Modify: live `data/bib.sqlite`, `data/zotero/data/zotero.sqlite` (with backups)
- Create: `.state/zotero-dedupe/` logs, `docs/superpowers/plans/2026-08-26-zotero-dedupe.md` outcome section, memory update

**Interfaces:**
- Consumes: everything above.

- [ ] **Step 1: Rehearse on a snapshot.** Copy `data/zotero/data/zotero.sqlite{,-wal}` and `data/bib.sqlite` to the scratchpad; run `merge_url_duplicates(snapshot, dry_run=True)` then `dry_run=False`, and `dedupe_bib(Store(snapshot_bib), dry_run=False)`. Verify against the measured baseline: ~8,955 URL clusters merged (± the type-mismatch skips), bib removes ~17 (Class C) + ~34 (Class B) rows; audioRecording/dictionaryEntry/artwork counts unchanged before vs after; the iQIES cluster collapses to 1 item whose `alias_mids` holds 7 Message-IDs; PMID 17443510 and 23440784 appear exactly once each.

- [ ] **Step 2: Sanity-check merged snapshot with the app-side reader.** `uv run python -c "from zot.duck import ..."`-style smoke or the existing `tests/zot` schema-parity tests pointed at the snapshot: `uv run pytest tests/zot -v` with `LIVE_DB` env/monkeypatch if the conftest supports it (see `tests/zot/conftest.py`) — schema intact, no FK orphans: `PRAGMA foreign_key_check` returns no rows.

- [ ] **Step 3: Live run, inside the stop window.**

```bash
docker stop zotero
cp data/zotero/data/zotero.sqlite data/zotero/data/zotero.sqlite.pre-dedupe-$(date +%Y%m%d).bak
cp data/bib.sqlite data/bib.sqlite.pre-dedupe-$(date +%Y%m%d).bak
uv run python dev/scripts/dedupe_bib_legacy.py --live | tee .state/zotero-dedupe/bib-legacy.log
uv run python -c "
from zot.merge import merge_url_duplicates
print(merge_url_duplicates('data/zotero/data/zotero.sqlite', dry_run=False))
" | tee .state/zotero-dedupe/zotero-merge.log
docker start zotero
```

Zotero-side rows for the bib pairs printed by the legacy script are same-URL (Class C — already merged by the URL pass) or email items whose bib row is gone; delete those email losers in Zotero by key with `merge_into` against the keeper's Zotero item (look both up with `find_item_by_key`), inside the same window.

- [ ] **Step 4: Post-verification.** Re-run the cluster census (zero `http*`/`email:*` URL clusters of size >1 among non-deleted top-level items); spot-check in the Zotero UI (Cmsupdates, Skin Substitutes, ACO collections); `uv run pytest tests/zot tests/bib -x -q`; confirm `stack prisma flow palliative-rfi` counts are unchanged (twins were never in bib, so PRISMA counts must not move — if they do, stop and investigate before proceeding).

- [ ] **Step 5: Class D triage artifact (report only).** Emit remaining same-DOI clusters to `.state/zotero-dedupe/doi-clusters.md` (DOI, member keys, titles, PMIDs) for human review — same-DOI different-PMID records are PubMed's own dual records; do NOT auto-merge. Post the count to the tracker issue.

- [ ] **Step 6: Close out.** Append an outcome section to this plan (counts, log paths, backup names); update the auto-memory (new file `zotero-dedupe-2026-08` linking [[zotero-rule-sync-incident]] and [[cms-mail-pipeline]]; correct any stale claims); comment + close the tracker issue; commit docs/memory changes.

```bash
git status --short
git add docs/superpowers/plans/2026-08-26-zotero-dedupe.md
git commit -m "docs(plan): zotero dedupe executed — outcome record (closes #665)"
```

---

## Self-review notes

- Spec coverage: Class A+C → Tasks 1/2/5; Class B → Tasks 3/4/5; Class D → Task 5 step 5; recurrence prevention → Tasks 2 (deterministic lookup + nightly merge) and 3 (ingest aliasing); fidelity → merge-not-delete, dry-run, logs, backups, protected-item checks (Task 5 steps 1/4).
- Known verify-before-run items are called out inline where the plan depends on code not fully read here: `Db.delete_item` cascade behavior (Task 1), bib schema tag-table names and `Store.update` kwargs (Task 4), test fixture idioms (Tasks 3/4). Executors must check those files first; the tests will catch mismatches.
- The uncommitted 05-14 one-off restore script was never found in the repo — prevention therefore targets the *conditions* (key-matched merges against a rekeyed DB, non-deterministic URL lookup, no library-wide invariant) rather than the lost script. Any future restore work must match by URL, not key, and must run the Task 1 dry-run census afterward.

---

## Outcome (executed 2026-08-26)

All five tasks complete; live run executed and verified the same day. Commits
d4e3018, 5320662, 73407bf, b0a168f, 45dedcc, 41dd77f, cab9828 (+ sidebar
a632908 fixing pre-existing prisma test-fixture drift that blocked the hook).

**Live run** (zotero stopped ~15:14–15:29 EDT; backups
`*.pre-dedupe-20260826.bak` for zotero.sqlite/-wal and bib.sqlite):
- bib legacy dedupe: 3 url groups + 9 email groups → 31 rows removed,
  14 Message-IDs aliased.
- Zotero URL merge: 8,954 clusters, **8,980 duplicate items merged away**
  in 326 s, preserving 1,373 child notes, 1,779 attachments, 822 collection
  memberships, 7 tags, 3 relations that lived only on the duplicates.
- Email-pair Zotero pass: 14 more merged.
- Post-run: 199,420 → 190,426 top-level items; **zero merge-eligible
  same-URL clusters remain** (one intentional type-mismatch skip:
  GTVEBJ3T document / 76PA8PRE webpage @ qpp.cms.gov resource-library);
  protected personal/archive types and PRISMA `stage:` tags unchanged;
  foreign_key_check clean; pubmed 17443510/23440784 unique.
- body_sha1 backfill on legacy email keepers: 217 backfilled,
  185 skipped (truncated abstracts — hash parity can't be guaranteed).
- Class-D triage: 3 residual same-DOI clusters (1 genuine PubMed dual
  record, 2 DOI-field noise) in `.state/zotero-dedupe/doi-clusters.md`.

**Known residuals** (accepted, documented in #665): the type-mismatch pair
above (re-reported nightly as `skipped_type_mismatch: 1` — expected
baseline); up to one new item per re-send of a legacy notice whose abstract
was truncated (185 candidates); Class-D clusters are human-decision-only.
Standing assumption of the nightly `--merge-dupes` run: URL is identity —
two genuinely distinct same-type items sharing one URL will be merged
(bib enforces one-item-per-URL; Zotero-only personal items are outside
that invariant).

Rollback: restore the three `.bak` files inside a zotero-stop window.
Run logs: `.state/zotero-dedupe/`; nightly logs now land in
`data/zotero/data/dedupe-logs/` (host-durable).