stack/docs/superpowers/plans/2026-08-26-zotero-dedupe.md

Zotero Library Dedupe (fidelity-preserving) Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Remove every duplicate item from the Zotero library by merging (tags, collections, notes, attachments, relations) into a canonical copy — never by blind delete — and close the three ingest/ops holes that created them.

Architecture: A new zot.merge module provides a single merge_into(db, keeper_id, loser_id) primitive plus a URL-cluster driver. The nightly zotero-sync workflow calls the driver library-wide (replacing the lossy title+date dedupe_collection pass). Mail ingest gains content-hash aliasing so listserv re-sends stop creating new items. A one-off runbook (Task 5) executes the historical cleanup against the live DB inside the usual zotero-stop window, dry-run first, with a full backup.

Tech Stack: Python 3 / sqlite3, existing zot.db.Db wrapper, bib.Store, pytest (tests/zot, tests/bib), Gitea workflow zotero-sync.yml.

Spec: Root-cause findings below (this section is the spec; no separate doc).

Root-cause findings (evidence, 2026-08-26)

Measured on a snapshot of data/zotero/data/zotero.sqlite (287,735 items; 199,139 top-level non-deleted) and data/bib.sqlite (219,215 items):

• Class A — restore twins, ~8,981 surplus items in 8,955 URL clusters. Live copies' keys contain no L; 8,938 twins' keys do; 8,947 twins were physically inserted 2026-05-14 13:41–13:46 (clientDateModified) with original dateAdded preserved and high itemIDs (~371k–376k vs 3k–48k for the live copies). Timeline: zotero.sqlite.pre-rekey.bak created 2026-05-13 16:03; commit 8149da5 (2026-05-13 16:40) dropped L from the Zotero key charset; live rows were rekeyed in place; then an uncommitted one-off on 05-14 restored rows from the pre-rekey backup matching by key — every rekeyed row looked "missing" and was re-inserted under its old L-key, children included (22,734 child rows carry the 05-14 stamp). bib.sync matches by URL and updates only the first-found copy, so twins hold stale tags (avg 5.1 vs 15.5) but ~23% of sampled clusters have a twin with MORE child notes than the keeper, and a few have more attachments — blind deletion loses data. Cluster types: 6,374 document, 2,202 journalArticle, 182 statute, 152 webpage, 44 report, 1 preprint.
• Class B — listserv re-sends (bib-level): 17 title clusters / 34 surplus rows in bib (e.g. "Upcoming iQIES Service Center Hold Times" ×8). Mail ingest keys identity on email:<Message-ID>; CMS re-sends identical content with a fresh Message-ID on different days. The nightly --dedupe-collection Cmsupdates pass groups on (title, date), so cross-day re-sends never group; it is also scoped to one collection and deletes without merging.
• Class C — legacy ingest URL dups in bib: 3 clusters (~17 surplus) (aco-reach ×12, county-level-aggregate… ×5, one ×3), created 2026-03-04 before 91c3306 made Store.upsert merge by URL. Mirrored into Zotero as same-URL rows (a Class A-shaped symptom with a different origin — the URL-cluster merge handles both).
• Class D — same-DOI pairs: 2,080 clusters / 2,081 surplus, zero PMID collisions — i.e. PubMed's own dual records (different PMIDs, one DOI) plus Class A twins. After Class A/C merging, the residue is a scientific-record judgment call → report-only triage artifact, no automated merge.
• Protected, NOT duplicates: (1) ~11k Zotero-only personal/archive items from the host-library migration (audioRecording 1,018, dictionaryEntry 744, artwork 70, plus personal journalArticles) — "absent from bib" must never be treated as a dup signal; (2) same-title different-PMID records (e.g. Cochrane review updates titled "Skin grafting for venous leg ulcers.") — title is not identity; (3) journalArticle rows whose url field holds page-range junk ("997-998") — only http*/email:* URLs count as identity.

Global Constraints

• Zotero DB writes only inside a stop/start window: docker stop zotero … docker start zotero (same discipline as bib.sync; see zotero-sync.yml).
• Every live-DB mutation is preceded by a timestamped .bak copy and a dry-run report; merge decisions are logged to .state/zotero-dedupe/ as JSON before any delete.
• Identity for clustering = the url itemData field, only when it starts with http or email:.
• Keeper = lowest itemID in a cluster (what find_item_by_url returns once made deterministic — Task 2).
• Clusters whose members differ in itemTypeID are skipped and reported, never auto-merged.
• Run everything with uv run; tests with targeted files (uv run pytest tests/zot/test_merge.py -v), not the full suite.
• Before every commit: git status --short (concurrent sessions share this worktree). No Co-Authored-By trailer.

---

Task 1: zot.merge — merge primitive + URL-cluster driver

Files:

• Create: src/zot/merge.py
• Test: tests/zot/test_merge.py

Interfaces:

• Consumes: zot.db.Db, zot.schema.create_db, zot.db.TYPE_MAP.

• Produces:

  • merge_into(db: Db, keeper_id: int, loser_id: int) -> dict[str, int] — unions tags/collections/relations, reparents missing child notes (dedup by note-text hash) and attachments (dedup by path/filename), then hard-deletes the loser row. Returns per-facet counts.
  • find_url_clusters(db: Db) -> list[list[tuple[int, str, int]]] — clusters of (itemID, key, itemTypeID) sharing an http*/email:* url, sorted by itemID.
  • merge_url_duplicates(db_path, *, dry_run: bool = True, log_dir: str | Path = ".state/zotero-dedupe") -> dict — driver; stats {clusters, merged, skipped_type_mismatch, notes_adopted, attachments_adopted, tags_added, collections_added} and writes merge-log-<runid>.json.

• Step 1: Write the failing tests

# tests/zot/test_merge.py
"""Merge-based dedupe: fidelity facets union into the keeper."""
import json
import sqlite3
from pathlib import Path

import pytest

from zot.db import TYPE_MAP, Db
from zot.schema import create_db


@pytest.fixture
def dup_db(tmp_path: Path) -> Path:
    p = tmp_path / "zotero.sqlite"
    create_db(str(p)).close()
    with Db(str(p)) as db:
        keeper = db.create_item(TYPE_MAP["document"], key="AAAAAAAA")
        loser = db.create_item(TYPE_MAP["document"], key="BBBBBBBB")
        for iid in (keeper, loser):
            db.set_field(iid, "url", "https://example.org/x")
        db.set_field(keeper, "title", "X")
        db.set_field(loser, "title", "X (stale)")
        db.sync_tags(keeper, ["shared", "keeper-only"])
        db.sync_tags(loser, ["shared", "loser-only"])
        # loser-only child note
        nid = db.create_item(TYPE_MAP["note"], key="CCCCCCCC")
        db.con.execute(
            "INSERT INTO itemNotes (itemID, parentItemID, note, title) VALUES (?,?,?,?)",
            (nid, loser, "<p>loser note</p>", ""),
        )
        # identical note on both — must NOT be doubled
        for parent, key in ((keeper, "DDDDDDDD"), (loser, "EEEEEEEE")):
            dn = db.create_item(TYPE_MAP["note"], key=key)
            db.con.execute(
                "INSERT INTO itemNotes (itemID, parentItemID, note, title) VALUES (?,?,?,?)",
                (dn, parent, "<p>same on both</p>", ""),
            )
        db.commit()
    return p


def _tags(db: Db, item_id: int) -> set[str]:
    return {
        r[0]
        for r in db.con.execute(
            "SELECT t.name FROM itemTags it JOIN tags t ON t.tagID=it.tagID"
            " WHERE it.itemID=?",
            (item_id,),
        )
    }


def _notes(db: Db, item_id: int) -> list[str]:
    return [
        r[0]
        for r in db.con.execute(
            "SELECT note FROM itemNotes WHERE parentItemID=?", (item_id,)
        )
    ]


class TestMergeInto:
    def test_unions_tags_and_adopts_missing_notes(self, dup_db):
        from zot.merge import merge_into

        with Db(str(dup_db)) as db:
            keeper = db.find_item_by_key("AAAAAAAA")
            loser = db.find_item_by_key("BBBBBBBB")
            merge_into(db, keeper, loser)
            db.commit()
            assert _tags(db, keeper) >= {"shared", "keeper-only", "loser-only"}
            notes = _notes(db, keeper)
            assert "<p>loser note</p>" in notes
            assert notes.count("<p>same on both</p>") == 1
            assert db.find_item_by_key("BBBBBBBB") is None

    def test_keeper_fields_win(self, dup_db):
        from zot.merge import merge_into

        with Db(str(dup_db)) as db:
            keeper = db.find_item_by_key("AAAAAAAA")
            loser = db.find_item_by_key("BBBBBBBB")
            merge_into(db, keeper, loser)
            row = db.con.execute(
                "SELECT idv.value FROM itemData id"
                " JOIN fields f ON f.fieldID=id.fieldID"
                " JOIN itemDataValues idv ON idv.valueID=id.valueID"
                " WHERE id.itemID=? AND f.fieldName='title'",
                (keeper,),
            ).fetchone()
            assert row[0] == "X"


class TestDriver:
    def test_dry_run_reports_without_deleting(self, dup_db, tmp_path):
        from zot.merge import merge_url_duplicates

        stats = merge_url_duplicates(dup_db, dry_run=True, log_dir=tmp_path)
        assert stats["clusters"] == 1
        with Db(str(dup_db)) as db:
            assert db.find_item_by_key("BBBBBBBB") is not None
        logs = list(tmp_path.glob("merge-log-*.json"))
        assert logs and json.loads(logs[0].read_text())[0]["keeper_key"] == "AAAAAAAA"

    def test_live_run_merges(self, dup_db, tmp_path):
        from zot.merge import merge_url_duplicates

        stats = merge_url_duplicates(dup_db, dry_run=False, log_dir=tmp_path)
        assert stats["merged"] == 1
        with Db(str(dup_db)) as db:
            assert db.find_item_by_key("BBBBBBBB") is None

    def test_type_mismatch_skipped(self, tmp_path):
        from zot.merge import merge_url_duplicates

        p = tmp_path / "z.sqlite"
        create_db(str(p)).close()
        with Db(str(p)) as db:
            a = db.create_item(TYPE_MAP["document"], key="AAAAAAAA")
            b = db.create_item(TYPE_MAP["journalArticle"], key="BBBBBBBB")
            for iid in (a, b):
                db.set_field(iid, "url", "https://example.org/y")
            db.commit()
        stats = merge_url_duplicates(p, dry_run=False, log_dir=tmp_path)
        assert stats["skipped_type_mismatch"] == 1
        with Db(str(p)) as db:
            assert db.find_item_by_key("BBBBBBBB") is not None

    def test_junk_urls_ignored(self, tmp_path):
        from zot.merge import merge_url_duplicates

        p = tmp_path / "z.sqlite"
        create_db(str(p)).close()
        with Db(str(p)) as db:
            a = db.create_item(TYPE_MAP["journalArticle"], key="AAAAAAAA")
            b = db.create_item(TYPE_MAP["journalArticle"], key="BBBBBBBB")
            for iid in (a, b):
                db.set_field(iid, "url", "997-998")
            db.commit()
        stats = merge_url_duplicates(p, dry_run=False, log_dir=tmp_path)
        assert stats["clusters"] == 0

• Step 2: Run tests to verify they fail

Run: uv run pytest tests/zot/test_merge.py -v Expected: FAIL — ModuleNotFoundError: No module named 'zot.merge'

• Step 3: Implement src/zot/merge.py

"""Merge-based Zotero dedupe.

Identity = the ``url`` itemData field (only ``http*``/``email:*``
values). The keeper is the lowest ``itemID`` — the row a deterministic
``find_item_by_url`` returns, i.e. the copy ``bib.sync`` has been
updating. Losers are merged facet-by-facet into the keeper, then
hard-deleted; every decision is logged as JSON before any delete.

Born from the 2026-05-14 incident where a key-matched restore from
``pre-rekey.bak`` re-inserted ~9k rekeyed rows as URL-twins (issue in
tracker; plan: docs/superpowers/plans/2026-08-26-zotero-dedupe.md).
"""

from __future__ import annotations

import hashlib
import json
from collections import defaultdict
from pathlib import Path

from zot.db import Db, now_iso

URL_FIELD_PREFIXES = ("http", "email:")


def _note_hash(text: str) -> str:
    return hashlib.sha1(" ".join((text or "").split()).encode()).hexdigest()


def merge_into(db: Db, keeper_id: int, loser_id: int) -> dict[str, int]:
    """Union loser's facets into keeper, then delete the loser row."""
    out = {"tags_added": 0, "collections_added": 0, "notes_adopted": 0,
           "attachments_adopted": 0, "relations_moved": 0}
    con = db.con

    for (tag_id,) in con.execute(
        "SELECT tagID FROM itemTags WHERE itemID=?", (loser_id,)
    ).fetchall():
        cur = con.execute(
            "INSERT OR IGNORE INTO itemTags (itemID, tagID, type) VALUES (?,?,0)",
            (keeper_id, tag_id),
        )
        out["tags_added"] += cur.rowcount

    for (coll_id,) in con.execute(
        "SELECT collectionID FROM collectionItems WHERE itemID=?", (loser_id,)
    ).fetchall():
        cur = con.execute(
            "INSERT OR IGNORE INTO collectionItems (collectionID, itemID)"
            " VALUES (?,?)",
            (coll_id, keeper_id),
        )
        out["collections_added"] += cur.rowcount

    keeper_notes = {
        _note_hash(r[0])
        for r in con.execute(
            "SELECT note FROM itemNotes WHERE parentItemID=?", (keeper_id,)
        )
    }
    for note_item, note in con.execute(
        "SELECT itemID, note FROM itemNotes WHERE parentItemID=?", (loser_id,)
    ).fetchall():
        if _note_hash(note) in keeper_notes:
            db.delete_item(note_item)
            continue
        con.execute(
            "UPDATE itemNotes SET parentItemID=? WHERE itemID=?",
            (keeper_id, note_item),
        )
        out["notes_adopted"] += 1

    keeper_atts = {
        r[0]
        for r in con.execute(
            "SELECT COALESCE(path,'') FROM itemAttachments WHERE parentItemID=?",
            (keeper_id,),
        )
    }
    for att_item, path in con.execute(
        "SELECT itemID, COALESCE(path,'') FROM itemAttachments"
        " WHERE parentItemID=?",
        (loser_id,),
    ).fetchall():
        if path and path in keeper_atts:
            db.delete_item(att_item)
            continue
        con.execute(
            "UPDATE itemAttachments SET parentItemID=? WHERE itemID=?",
            (keeper_id, att_item),
        )
        out["attachments_adopted"] += 1

    cur = con.execute(
        "UPDATE OR IGNORE itemRelations SET itemID=? WHERE itemID=?",
        (keeper_id, loser_id),
    )
    out["relations_moved"] = cur.rowcount

    # Keeper has no creators but loser does → adopt them wholesale.
    has_keeper = con.execute(
        "SELECT 1 FROM itemCreators WHERE itemID=? LIMIT 1", (keeper_id,)
    ).fetchone()
    if not has_keeper:
        con.execute(
            "UPDATE OR IGNORE itemCreators SET itemID=? WHERE itemID=?",
            (keeper_id, loser_id),
        )

    db.delete_item(loser_id)
    return out


def find_url_clusters(db: Db) -> list[list[tuple[int, str, int]]]:
    rows = db.con.execute(
        """
        SELECT i.itemID, i.key, i.itemTypeID, idv.value
        FROM items i
        JOIN itemData id ON id.itemID = i.itemID
        JOIN fields f ON f.fieldID = id.fieldID AND f.fieldName = 'url'
        JOIN itemDataValues idv ON idv.valueID = id.valueID
        WHERE i.itemID NOT IN (SELECT itemID FROM deletedItems)
          AND i.itemID NOT IN (SELECT itemID FROM itemAttachments)
          AND i.itemID NOT IN (SELECT itemID FROM itemNotes)
        ORDER BY i.itemID
        """
    ).fetchall()
    by_url: dict[str, list[tuple[int, str, int]]] = defaultdict(list)
    for item_id, key, type_id, url in rows:
        if url and url.startswith(URL_FIELD_PREFIXES):
            by_url[url].append((item_id, key, type_id))
    return [v for v in by_url.values() if len(v) > 1]


def merge_url_duplicates(
    db_path: str | Path,
    *,
    dry_run: bool = True,
    log_dir: str | Path = ".state/zotero-dedupe",
) -> dict:
    stats = {"clusters": 0, "merged": 0, "skipped_type_mismatch": 0,
             "tags_added": 0, "collections_added": 0, "notes_adopted": 0,
             "attachments_adopted": 0, "relations_moved": 0}
    log: list[dict] = []
    log_dir = Path(log_dir)
    log_dir.mkdir(parents=True, exist_ok=True)
    run_id = now_iso().replace(" ", "T").replace(":", "")

    with Db(str(db_path)) as db:
        clusters = find_url_clusters(db)
        for members in clusters:
            keeper = members[0]
            losers = members[1:]
            if any(m[2] != keeper[2] for m in losers):
                stats["skipped_type_mismatch"] += 1
                log.append({"keeper_key": keeper[1], "action": "skip-type-mismatch",
                            "loser_keys": [m[1] for m in losers]})
                continue
            stats["clusters"] += 1
            entry = {"keeper_key": keeper[1],
                     "loser_keys": [m[1] for m in losers],
                     "action": "dry-run" if dry_run else "merged"}
            if not dry_run:
                for m in losers:
                    for k, v in merge_into(db, keeper[0], m[0]).items():
                        stats[k] += v
                    stats["merged"] += 1
            log.append(entry)
        if not dry_run:
            db.commit()

    (log_dir / f"merge-log-{run_id}.json").write_text(json.dumps(log, indent=1))
    return stats

Note: if Db.delete_item does not already cascade a top-level item's remaining children (check src/zot/db.py), extend merge_into to delete leftover child rows of the loser before db.delete_item(loser_id) — the tests' find_item_by_key assertion will catch orphans via FK errors.

• Step 4: Run tests to verify they pass

Run: uv run pytest tests/zot/test_merge.py -v Expected: 6 PASS

• Step 5: Commit

git status --short   # foreign staged work check
git add src/zot/merge.py tests/zot/test_merge.py
git commit -m "feat(zot): merge-based URL dedupe engine — union facets, keep-earliest (refs #665)"

---

Task 2: Deterministic URL lookup + nightly library-wide merge

Files:

• Modify: src/zot/db.py:391-400 (find_item_by_url)
• Modify: src/cli/bib.py:506-543 (zotero-sync command options)
• Modify: .gitea/workflows/zotero-sync.yml:27-34
• Test: tests/zot/test_db.py, tests/cli/ (existing zotero-sync CLI test file if present, else add asserts in tests/zot/test_merge.py)

Interfaces:

• Consumes: zot.merge.merge_url_duplicates (Task 1 signature).

• Produces: find_item_by_url now ORDER BY i.itemID (bib.sync keeps updating the merge keeper); CLI flag --merge-dupes/--no-merge-dupes (default off) on stack bib zotero-sync, running the library-wide merge after the push, inside the same stop window. --dedupe-collection stays but is documented as deprecated in its help string.

• Step 1: Write the failing test

# append to tests/zot/test_db.py
class TestFindItemByUrlDeterministic:
    def test_returns_lowest_item_id(self, tmp_path):
        from zot.db import TYPE_MAP, Db
        from zot.schema import create_db

        p = tmp_path / "z.sqlite"
        create_db(str(p)).close()
        with Db(str(p)) as db:
            a = db.create_item(TYPE_MAP["document"], key="AAAAAAAA")
            b = db.create_item(TYPE_MAP["document"], key="BBBBBBBB")
            # insert the URL row for the HIGHER itemID first so a
            # scan-order query would return it
            db.set_field(b, "url", "https://example.org/det")
            db.set_field(a, "url", "https://example.org/det")
            db.commit()
            assert db.find_item_by_url("https://example.org/det") == a

• Step 2: Run test to verify it fails

Run: uv run pytest tests/zot/test_db.py -k Deterministic -v Expected: FAIL (returns b, the first row in scan order)

• Step 3: Add ORDER BY i.itemID to the query in find_item_by_url

            """SELECT i.itemID FROM items i
               JOIN itemData id ON i.itemID = id.itemID
               JOIN itemDataValues idv ON id.valueID = idv.valueID
               WHERE id.fieldID = ? AND idv.value = ?
               ORDER BY i.itemID""",

• Step 4: Run test to verify it passes

Run: uv run pytest tests/zot/test_db.py -k Deterministic -v → PASS

• Step 5: Wire --merge-dupes into the CLI and workflow

In src/cli/bib.py next to the existing dedupe_collection option (line ~506):

    merge_dupes: bool = typer.Option(
        False,
        "--merge-dupes/--no-merge-dupes",
        help="After the push, merge same-URL duplicate items library-wide "
        "(union tags/collections/notes/attachments; keep-earliest). "
        "Supersedes --dedupe-collection, which deletes without merging.",
    ),

and after the existing if dedupe_collection: block (line ~542):

        if merge_dupes:
            from zot.merge import merge_url_duplicates

            stats |= {f"merge_{k}": v for k, v in merge_url_duplicates(
                zotero_db_path, dry_run=False
            ).items()}

(resolve zotero_db_path the same way the surrounding command does — it already computes the Zotero DB path for the push; reuse that variable name.)

In .gitea/workflows/zotero-sync.yml, replace --dedupe-collection Cmsupdates with --merge-dupes and update the comment block (lines 27–34) to say: listserv re-sends are now aliased at ingest (Task 3) and any same-URL twins are merged, not deleted.

• Step 6: Run the CLI help to verify wiring

Run: uv run stack bib zotero-sync --help Expected: --merge-dupes listed; exit 0.

• Step 7: Commit

git status --short
git add src/zot/db.py tests/zot/test_db.py src/cli/bib.py .gitea/workflows/zotero-sync.yml
git commit -m "feat(bib,zot): deterministic URL lookup + nightly library-wide merge-dupes (refs #665)"

---

Task 3: Alias listserv re-sends at mail ingest (content-hash identity)

Files:

• Modify: src/bib/email_ingest.py:108-150 (_ingest_message)
• Test: tests/bib/test_email_ingest.py (extend the existing file)

Interfaces:

• Consumes: bib.Store.upsert(item) -> str, Source(title=..., url=...) with extra_json packing via to_row().

• Produces: new items carry extra_json["body_sha1"]; a re-send whose (title, body_sha1) matches an existing source:email item does NOT create a new item — its Message-ID is appended to the keeper's extra_json["alias_mids"] list and its tags are merged via the normal upsert-by-url path (we upsert with the keeper's url).

• Step 1: Write the failing test

# append to tests/bib/test_email_ingest.py (reuse that file's existing
# fixtures for Store and for building EmailMessage objects; the ones
# below assume a `store` fixture and a `make_msg(subject, body, mid, date)`
# helper — add the helper if the file lacks one)
import email.message
import json


def make_msg(subject: str, body: str, mid: str, date: str):
    m = email.message.EmailMessage()
    m["Subject"] = subject
    m["From"] = "CMS Updates <cmslists@subscriptions.cms.hhs.gov>"
    m["Message-ID"] = f"<{mid}>"
    m["Date"] = date
    m.set_content(body)
    return m


class TestResendAliasing:
    def test_same_body_new_mid_aliases_instead_of_new_item(
        self, store, mailbox, tmp_path
    ):
        from bib.email_ingest import _ingest_message

        m1 = make_msg("Upcoming iQIES Hold Times", "Same body.",
                      "aaa@x.example", "Mon, 27 Apr 2026 10:00:00 -0400")
        m2 = make_msg("Upcoming iQIES Hold Times", "Same body.",
                      "bbb@x.example", "Fri, 10 Jul 2026 10:00:00 -0400")
        _ingest_message(store, mailbox, m1, tmp_path)
        _ingest_message(store, mailbox, m2, tmp_path)
        items = store.list_items(query="iQIES")
        assert len(items) == 1
        ej = json.loads(items[0].to_row()["extra_json"])
        assert "bbb@x.example" in ej.get("alias_mids", [])

    def test_different_body_same_subject_stays_separate(
        self, store, mailbox, tmp_path
    ):
        from bib.email_ingest import _ingest_message

        m1 = make_msg("Weekly Digest", "Body one.",
                      "ccc@x.example", "Mon, 27 Apr 2026 10:00:00 -0400")
        m2 = make_msg("Weekly Digest", "Body two.",
                      "ddd@x.example", "Fri, 10 Jul 2026 10:00:00 -0400")
        _ingest_message(store, mailbox, m1, tmp_path)
        _ingest_message(store, mailbox, m2, tmp_path)
        assert len(store.list_items(query="Weekly Digest")) == 2

(If tests/bib/test_email_ingest.py has no store/mailbox fixtures, copy the setup its existing tests use — the file already exercises _ingest_message.)

• Step 2: Run tests to verify they fail

Run: uv run pytest tests/bib/test_email_ingest.py -k Resend -v Expected: FAIL — two items created, no alias_mids.

• Step 3: Implement content-hash aliasing in _ingest_message

After body = _extract_body(msg) and before item = Source(...):

    body_sha1 = hashlib.sha1(" ".join((body or "").split()).encode()).hexdigest()

    # Listservs re-send identical content under a fresh Message-ID
    # (Class B in docs/superpowers/plans/2026-08-26-zotero-dedupe.md).
    # If an item with this exact title+body already exists, treat this
    # message as an alias of it rather than a new Source.
    con = store._con()
    dup = con.execute(
        "SELECT key, url, extra_json FROM items"
        " WHERE title = ? AND extra_json LIKE ?",
        (subject[:255], f'%"body_sha1": "{body_sha1}"%'),
    ).fetchone()
    if dup:
        ej = json.loads(dup["extra_json"] or "{}")
        aliases = ej.get("alias_mids", [])
        if raw_mid not in aliases:
            aliases.append(raw_mid)
            ej["alias_mids"] = aliases
            store.update(dup["key"], extra_json=json.dumps(ej))
        return 0

and record the hash on new items (next to the existing item.doc_type = "Email" block):

    item.extra_json_fields = {"body_sha1": body_sha1}

— use whatever mechanism Source/Item.to_row() already packs unknown fields into extra_json (see src/bib/item.py:81: "are packed into extra_json"); if the model expects direct attribute assignment (item.body_sha1 = body_sha1), do that instead. Add import hashlib, import json at the top of email_ingest.py if absent.

• Step 4: Run tests to verify they pass

Run: uv run pytest tests/bib/test_email_ingest.py -v Expected: new tests PASS, existing tests still PASS.

• Step 5: Commit

git status --short
git add src/bib/email_ingest.py tests/bib/test_email_ingest.py
git commit -m "feat(bib): alias listserv re-sends by body hash — one item per content (refs #665)"

---

Task 4: bib-side legacy cleanup — Class B (email re-sends) + Class C (URL dups)

Files:

• Create: dev/scripts/dedupe_bib_legacy.py
• Test: tests/bib/test_dedupe_legacy.py

Interfaces:

• Consumes: bib.Store (.get, .update, .delete, ._con()), zot.merge.merge_into, zot.db.Db.

• Produces: dedupe_bib(store, *, dry_run=True) -> dict — importable from the script; groups bib rows (a) by exact url (Class C) and (b) by (title, sha1(abstract)) among source:email items (Class B); keep-earliest by created_at then rowid; union tags/collections onto the keeper via store.update; for email groups append surplus Message-IDs (parsed from each loser's email:<mid> url) to keeper extra_json["alias_mids"]; delete surplus bib rows. Returns {url_groups, email_groups, removed, aliased} plus a pairs list of (keeper_key, loser_key) for the Zotero-side pass in Task 5.

• Step 1: Write the failing test

# tests/bib/test_dedupe_legacy.py
import json

from bib import Store
from bib.item import Source


def _mk_store(tmp_path):
    return Store(str(tmp_path / "bib.sqlite"))


def test_url_group_keeps_earliest_and_unions_tags(tmp_path):
    from dev.scripts.dedupe_bib_legacy import dedupe_bib

    store = _mk_store(tmp_path)
    a = store.create(Source(title="ACO REACH", url="https://x.test/reach"),
                     tags=["module:aco", "keep-me"])
    # second row with the same url — simulate the pre-#624 upsert
    con = store._con()
    con.execute(
        "INSERT INTO items (key, item_type, title, url, extra_json)"
        " VALUES ('ZZZZZZZ2','source','ACO REACH','https://x.test/reach','{}')"
    )
    con.commit()
    store.add_tag("ZZZZZZZ2", "loser-tag")
    out = dedupe_bib(store, dry_run=False)
    assert out["removed"] == 1
    kept = store.get(a)
    assert "loser-tag" in kept.tags and "keep-me" in kept.tags


def test_email_group_aliases_mids(tmp_path):
    from dev.scripts.dedupe_bib_legacy import dedupe_bib

    store = _mk_store(tmp_path)
    s1 = Source(title="iQIES Hold Times", url="email:aaa@x")
    s1.abstract = "Same body."
    s2 = Source(title="iQIES Hold Times", url="email:bbb@x")
    s2.abstract = "Same body."
    k1 = store.create(s1, tags=["source:email"])
    store.create(s2, tags=["source:email"])
    out = dedupe_bib(store, dry_run=False)
    assert out["removed"] == 1
    ej = json.loads(store.get(k1).to_row()["extra_json"])
    assert "bbb@x" in ej["alias_mids"]


def test_dry_run_touches_nothing(tmp_path):
    from dev.scripts.dedupe_bib_legacy import dedupe_bib

    store = _mk_store(tmp_path)
    s1 = Source(title="T", url="email:a@x"); s1.abstract = "B"
    s2 = Source(title="T", url="email:b@x"); s2.abstract = "B"
    store.create(s1, tags=["source:email"])
    store.create(s2, tags=["source:email"])
    out = dedupe_bib(store, dry_run=True)
    assert out["removed"] == 0 and out["email_groups"] == 1
    assert len(store.list_items(query="T")) == 2

(Adjust Store/Source constructor details to match tests/bib's existing idiom — the suite already builds throwaway stores; mirror it. store.create signature is at src/bib/store.py:98.)

• Step 2: Run tests to verify they fail

Run: uv run pytest tests/bib/test_dedupe_legacy.py -v Expected: FAIL — module not found.

• Step 3: Implement dev/scripts/dedupe_bib_legacy.py

"""One-off: merge legacy duplicate rows inside bib.sqlite.

Class C — same-url rows created before Store.upsert deduped by URL
(91c3306). Class B — listserv re-sends: same (title, body-hash) among
source:email items, each under a distinct email:<Message-ID> url.

Keep-earliest; union tags/collections into the keeper; alias surplus
Message-IDs; delete surplus rows. Prints stats; --dry-run default.
Returns keeper/loser key pairs so the Zotero-side merge (zot.merge)
can be driven off the same decisions.

Usage: uv run python dev/scripts/dedupe_bib_legacy.py [--live]
"""

from __future__ import annotations

import hashlib
import json
import sys
from collections import defaultdict


def _body_hash(abstract: str) -> str:
    return hashlib.sha1(" ".join((abstract or "").split()).encode()).hexdigest()


def dedupe_bib(store, *, dry_run: bool = True) -> dict:
    con = store._con()
    out = {"url_groups": 0, "email_groups": 0, "removed": 0, "aliased": 0,
           "pairs": []}

    groups: list[list[str]] = []  # each: [keeper_key, loser_key, ...]

    # Class C: exact-url groups (http only; email urls are unique per mid)
    by_url = defaultdict(list)
    for key, url in con.execute(
        "SELECT key, url FROM items WHERE url LIKE 'http%'"
        " ORDER BY created_at, id"
    ):
        by_url[url].append(key)
    for keys in by_url.values():
        if len(keys) > 1:
            out["url_groups"] += 1
            groups.append(keys)

    # Class B: (title, body-hash) among email items
    by_content = defaultdict(list)
    for key, title, abstract in con.execute(
        "SELECT i.key, i.title, COALESCE(i.abstract,'') FROM items i"
        " JOIN item_tags it ON it.item_id = i.id"
        " JOIN tags t ON t.id = it.tag_id AND t.name = 'source:email'"
        " ORDER BY i.created_at, i.id"
    ):
        by_content[(title, _body_hash(abstract))].append(key)
    for keys in by_content.values():
        if len(keys) > 1:
            out["email_groups"] += 1
            groups.append(keys)

    for keys in groups:
        keeper_key, losers = keys[0], keys[1:]
        if dry_run:
            continue
        keeper = store.get(keeper_key)
        ej = json.loads(keeper.to_row().get("extra_json", "{}") or "{}")
        aliases = ej.get("alias_mids", [])
        tags = list(keeper.tags)
        collections = list(keeper.collections)
        for lk in losers:
            loser = store.get(lk)
            tags += [t for t in loser.tags if t not in tags]
            collections += [c for c in loser.collections if c not in collections]
            if loser.url.startswith("email:"):
                mid = loser.url.removeprefix("email:")
                if mid not in aliases:
                    aliases.append(mid)
                    out["aliased"] += 1
            out["pairs"].append((keeper_key, lk))
            store.delete(lk)
            out["removed"] += 1
        if aliases:
            ej["alias_mids"] = aliases
        store.update(keeper_key, tags=tags, collections=collections,
                     extra_json=json.dumps(ej))
    return out


if __name__ == "__main__":
    from bib import Store

    store = Store()
    live = "--live" in sys.argv
    out = dedupe_bib(store, dry_run=not live)
    pairs = out.pop("pairs")
    print(json.dumps(out, indent=1))
    for k, l in pairs:
        print(f"pair {k} <- {l}")

Check the real bib schema names before running (src/bib/schema.sql): the tag join above assumes item_tags(item_id, tag_id) / tags(id, name); fix to match the actual DDL. store.update's accepted kwargs are at src/bib/store.py:169 — if it does not take collections/extra_json directly, route through the same row dict upsert uses (item.to_row() keys).

• Step 4: Run tests to verify they pass

Run: uv run pytest tests/bib/test_dedupe_legacy.py -v → 3 PASS

• Step 5: Commit

git status --short
git add dev/scripts/dedupe_bib_legacy.py tests/bib/test_dedupe_legacy.py
git commit -m "feat(bib): legacy dedupe script — merge same-url and re-sent-email rows (refs #665)"

---

Task 5: Execute the cleanup on the live library (runbook)

Files:

• Modify: live data/bib.sqlite, data/zotero/data/zotero.sqlite (with backups)
• Create: .state/zotero-dedupe/ logs, docs/superpowers/plans/2026-08-26-zotero-dedupe.md outcome section, memory update

Interfaces:

• Consumes: everything above.

• Step 1: Rehearse on a snapshot. Copy data/zotero/data/zotero.sqlite{,-wal} and data/bib.sqlite to the scratchpad; run merge_url_duplicates(snapshot, dry_run=True) then dry_run=False, and dedupe_bib(Store(snapshot_bib), dry_run=False). Verify against the measured baseline: ~8,955 URL clusters merged (± the type-mismatch skips), bib removes ~17 (Class C) + ~34 (Class B) rows; audioRecording/dictionaryEntry/artwork counts unchanged before vs after; the iQIES cluster collapses to 1 item whose alias_mids holds 7 Message-IDs; PMID 17443510 and 23440784 appear exactly once each.

• Step 2: Sanity-check merged snapshot with the app-side reader. uv run python -c "from zot.duck import ..."-style smoke or the existing tests/zot schema-parity tests pointed at the snapshot: uv run pytest tests/zot -v with LIVE_DB env/monkeypatch if the conftest supports it (see tests/zot/conftest.py) — schema intact, no FK orphans: PRAGMA foreign_key_check returns no rows.

• Step 3: Live run, inside the stop window.

docker stop zotero
cp data/zotero/data/zotero.sqlite data/zotero/data/zotero.sqlite.pre-dedupe-$(date +%Y%m%d).bak
cp data/bib.sqlite data/bib.sqlite.pre-dedupe-$(date +%Y%m%d).bak
uv run python dev/scripts/dedupe_bib_legacy.py --live | tee .state/zotero-dedupe/bib-legacy.log
uv run python -c "
from zot.merge import merge_url_duplicates
print(merge_url_duplicates('data/zotero/data/zotero.sqlite', dry_run=False))
" | tee .state/zotero-dedupe/zotero-merge.log
docker start zotero

Zotero-side rows for the bib pairs printed by the legacy script are same-URL (Class C — already merged by the URL pass) or email items whose bib row is gone; delete those email losers in Zotero by key with merge_into against the keeper's Zotero item (look both up with find_item_by_key), inside the same window.

• Step 4: Post-verification. Re-run the cluster census (zero http*/email:* URL clusters of size >1 among non-deleted top-level items); spot-check in the Zotero UI (Cmsupdates, Skin Substitutes, ACO collections); uv run pytest tests/zot tests/bib -x -q; confirm stack prisma flow palliative-rfi counts are unchanged (twins were never in bib, so PRISMA counts must not move — if they do, stop and investigate before proceeding).

• Step 5: Class D triage artifact (report only). Emit remaining same-DOI clusters to .state/zotero-dedupe/doi-clusters.md (DOI, member keys, titles, PMIDs) for human review — same-DOI different-PMID records are PubMed's own dual records; do NOT auto-merge. Post the count to the tracker issue.

• Step 6: Close out. Append an outcome section to this plan (counts, log paths, backup names); update the auto-memory (new file zotero-dedupe-2026-08 linking zotero-rule-sync-incident and cms-mail-pipeline; correct any stale claims); comment + close the tracker issue; commit docs/memory changes.

git status --short
git add docs/superpowers/plans/2026-08-26-zotero-dedupe.md
git commit -m "docs(plan): zotero dedupe executed — outcome record (closes #665)"

---

Self-review notes

• Spec coverage: Class A+C → Tasks 1/2/5; Class B → Tasks 3/4/5; Class D → Task 5 step 5; recurrence prevention → Tasks 2 (deterministic lookup + nightly merge) and 3 (ingest aliasing); fidelity → merge-not-delete, dry-run, logs, backups, protected-item checks (Task 5 steps 1/4).
• Known verify-before-run items are called out inline where the plan depends on code not fully read here: Db.delete_item cascade behavior (Task 1), bib schema tag-table names and Store.update kwargs (Task 4), test fixture idioms (Tasks 3/4). Executors must check those files first; the tests will catch mismatches.
• The uncommitted 05-14 one-off restore script was never found in the repo — prevention therefore targets the conditions (key-matched merges against a rekeyed DB, non-deterministic URL lookup, no library-wide invariant) rather than the lost script. Any future restore work must match by URL, not key, and must run the Task 1 dry-run census afterward.

---

Outcome (executed 2026-08-26)

All five tasks complete; live run executed and verified the same day. Commits d4e3018, 5320662, 73407bf, b0a168f, 45dedcc, 41dd77f, cab9828 (+ sidebar a632908 fixing pre-existing prisma test-fixture drift that blocked the hook).

Live run (zotero stopped ~15:14–15:29 EDT; backups *.pre-dedupe-20260826.bak for zotero.sqlite/-wal and bib.sqlite):

• bib legacy dedupe: 3 url groups + 9 email groups → 31 rows removed, 14 Message-IDs aliased.
• Zotero URL merge: 8,954 clusters, 8,980 duplicate items merged away in 326 s, preserving 1,373 child notes, 1,779 attachments, 822 collection memberships, 7 tags, 3 relations that lived only on the duplicates.
• Email-pair Zotero pass: 14 more merged.
• Post-run: 199,420 → 190,426 top-level items; zero merge-eligible same-URL clusters remain (one intentional type-mismatch skip: GTVEBJ3T document / 76PA8PRE webpage @ qpp.cms.gov resource-library); protected personal/archive types and PRISMA stage: tags unchanged; foreign_key_check clean; pubmed 17443510/23440784 unique.
• body_sha1 backfill on legacy email keepers: 217 backfilled, 185 skipped (truncated abstracts — hash parity can't be guaranteed).
• Class-D triage: 3 residual same-DOI clusters (1 genuine PubMed dual record, 2 DOI-field noise) in .state/zotero-dedupe/doi-clusters.md.

Known residuals (accepted, documented in #665): the type-mismatch pair above (re-reported nightly as skipped_type_mismatch: 1 — expected baseline); up to one new item per re-send of a legacy notice whose abstract was truncated (185 candidates); Class-D clusters are human-decision-only. Standing assumption of the nightly --merge-dupes run: URL is identity — two genuinely distinct same-type items sharing one URL will be merged (bib enforces one-item-per-URL; Zotero-only personal items are outside that invariant).

Rollback: restore the three .bak files inside a zotero-stop window. Run logs: .state/zotero-dedupe/; nightly logs now land in data/zotero/data/dedupe-logs/ (host-durable).