stack/.claude/plans/2026-04-28-comments-zotero-notes-plan.md

Comments → Zotero Notes Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Replace the current .md file attachment output (33,351 attachments / 23,605 items) with rendered-HTML Zotero notes, so extracted comment text is readable directly in Zotero's reader pane. New code path produces notes; one-shot migration retires the old file attachments.

Architecture: Three new code units — a markdown→HTML renderer (rex/comments/render.py), note-syncing in bib/sync.py, and a flipped attach_callback in cli/comments.py. Plus a one-shot migration script that backfills notes for existing comment dirs and destructively cleans up the old .md attachment rows + storage dirs in both bib and Zotero.

Tech Stack: Python 3.12, markdown-it-py + mdit-py-plugins (rendering), SQLite (bib + Zotero), pytest. Spec: .claude/specs/2026-04-28-comments-zotero-notes-design.md.

---

Task 1: Add markdown rendering deps

Files:

• Modify: pyproject.toml

• Step 1: Add deps to the bib extra

In pyproject.toml, update the bib extra:

bib = [
    "stack[conf]",
    "pydantic>=2.0.0",
    "markdown-it-py>=4.0",
    "mdit-py-plugins>=0.4",
]

• Step 2: Sync deps

Run: uv sync --extra bib Expected: succeeds; markdown-it-py and mdit-py-plugins move from transitive to direct.

• Step 3: Smoke-import

Run: uv run python -c "from markdown_it import MarkdownIt; from mdit_py_plugins.front_matter import front_matter_plugin; print('ok')" Expected: ok

• Step 4: Commit

git add pyproject.toml uv.lock
git commit -m "$(cat <<'EOF'
deps(bib): promote markdown-it-py + mdit-py-plugins to direct deps

Used by upcoming rex.comments.render to convert combined.md to HTML
for Zotero notes. Both were already transitive in uv.lock.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"

---

Task 2: Renderer module + tests

Files:

• Create: src/rex/comments/render.py

• Create: tests/rex/comments/test_render.py

• Step 1: Write failing tests

Create tests/rex/comments/test_render.py:

"""rex.comments.render — combined.md → Zotero-edible HTML."""

from __future__ import annotations

from rex.comments.render import render_combined_md


def test_strips_frontmatter():
    md = (
        "---\n"
        "comment_id: CMS-2024-0001-0001\n"
        "status: ok\n"
        "---\n"
        "\n"
        "# Heading\n"
        "\n"
        "body\n"
    )
    html = render_combined_md(md)
    assert "comment_id" not in html
    assert "status: ok" not in html
    assert "<h1>Heading</h1>" in html
    assert "<p>body</p>" in html


def test_renders_basic_markdown():
    md = (
        "---\nstatus: ok\n---\n\n"
        "## Section\n\n"
        "Text with **bold** and *italic*.\n"
    )
    html = render_combined_md(md)
    assert "<h2>Section</h2>" in html
    assert "<strong>bold</strong>" in html
    assert "<em>italic</em>" in html


def test_renders_tables():
    md = (
        "---\nstatus: ok\n---\n\n"
        "| Col A | Col B |\n"
        "| --- | --- |\n"
        "| a1 | b1 |\n"
        "| a2 | b2 |\n"
    )
    html = render_combined_md(md)
    assert "<table>" in html
    assert "<th>Col A</th>" in html
    assert "<td>a1</td>" in html


def test_handles_empty_body():
    md = "---\nstatus: ok\n---\n\n"
    html = render_combined_md(md)
    assert html == "" or html.strip() == ""


def test_raises_on_missing_frontmatter():
    """combined.md is always written with frontmatter — guard against
    silently ingesting malformed input."""
    import pytest
    with pytest.raises(ValueError, match="frontmatter"):
        render_combined_md("# No frontmatter here\n")

• Step 2: Run tests to verify they fail

Run: uv run pytest tests/rex/comments/test_render.py -v Expected: ImportError / "No module named 'rex.comments.render'" — all fail.

• Step 3: Implement the renderer

Create src/rex/comments/render.py:

"""Render combined.md → HTML for Zotero notes.

combined.md is the per-comment aggregate written by ``combine.extract_comment``:
a YAML frontmatter block followed by an inline-body section and one H2 section
per attachment. Zotero notes accept HTML, so we strip the frontmatter and run
the body through markdown-it-py.
"""

from __future__ import annotations

from markdown_it import MarkdownIt

from rex.comments.combine import parse_combined

_MD = MarkdownIt("commonmark", {"html": False, "breaks": False, "linkify": True})
_MD.enable("table")


def render_combined_md(text: str) -> str:
    """Strip combined.md's YAML frontmatter, render body to HTML.

    Raises ValueError if the input is missing frontmatter — combined.md
    always has it, so a missing block means the input isn't combined.md.
    """
    _, body = parse_combined(text)
    return _MD.render(body)

• Step 4: Run tests to verify they pass

Run: uv run pytest tests/rex/comments/test_render.py -v Expected: all 5 tests pass.

• Step 5: Commit

git add src/rex/comments/render.py tests/rex/comments/test_render.py
git commit -m "$(cat <<'EOF'
feat(rex/comments): render combined.md → HTML for Zotero notes

Pure function over (str)→str. Reuses combine.parse_combined to peel
off the YAML frontmatter, runs the body through markdown-it-py with
table support enabled (CMS comments lean on tabular data).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"

---

Task 3: Note-syncing in bib/sync.py

Files:

• Modify: src/bib/sync.py

• Modify: tests/bib/test_sync.py:17-103 (ZOTERO_SCHEMA — add itemNotes table)

• Modify: tests/bib/test_sync.py (add new test class TestSyncNotes)

• Step 1: Extend test schema with itemNotes

In tests/bib/test_sync.py, append to ZOTERO_SCHEMA (after the creatorTypes block, around line 102):

CREATE TABLE IF NOT EXISTS itemNotes (
    itemID INTEGER PRIMARY KEY,
    parentItemID INT,
    note TEXT,
    title TEXT
);
CREATE TABLE IF NOT EXISTS itemAttachments (
    itemID INTEGER PRIMARY KEY,
    parentItemID INT,
    linkMode INT,
    contentType TEXT,
    charsetID INT,
    path TEXT,
    syncState INT DEFAULT 0,
    storageModTime INT,
    storageHash TEXT
);

(Tests for sync up to now didn't exercise children — itemAttachments is added here too because the new branch in _sync_attachments also references it during update flow.)

• Step 2: Write failing tests

Append to tests/bib/test_sync.py:

class TestSyncNotes:
    """bib notes → Zotero itemNotes (non-attachment children)."""

    def _setup_bib_with_note(self, tmp_path, *, html: str, title: str):
        from bib import connect
        from bib.item import Source

        bib_db = tmp_path / "bib.sqlite"
        store = connect(str(bib_db))
        item = Source(
            title="Comment CMS-2024-0001-0001",
            url="https://www.regulations.gov/comment/CMS-2024-0001-0001",
        )
        item_key = store.upsert(item)
        store.attach_note(item_key, html, title=title)
        store.close()
        return bib_db, item_key

    def test_note_creates_zotero_itemnote(self, tmp_path):
        bib_db, item_key = self._setup_bib_with_note(
            tmp_path, html="<h1>Body</h1><p>text</p>", title="Comment text"
        )

        zot_db = str(tmp_path / "zotero.sqlite")
        con = _make_zotero_db(zot_db)
        con.close()

        from bib import connect
        store = connect(str(bib_db))
        items = store.list_items(tag=None)
        stats = push_to_zotero(items, store=store, zotero_db=zot_db)
        store.close()

        assert stats["notes"] == 1

        con = sqlite3.connect(zot_db)
        rows = con.execute(
            "SELECT title, note FROM itemNotes"
        ).fetchall()
        con.close()
        assert rows == [("Comment text", "<h1>Body</h1><p>text</p>")]

    def test_resync_is_idempotent(self, tmp_path):
        """Second sync must not duplicate the note (dedup by parent + title)."""
        bib_db, item_key = self._setup_bib_with_note(
            tmp_path, html="<p>x</p>", title="Comment text"
        )

        zot_db = str(tmp_path / "zotero.sqlite")
        con = _make_zotero_db(zot_db)
        con.close()

        from bib import connect
        store = connect(str(bib_db))
        items = store.list_items(tag=None)

        push_to_zotero(items, store=store, zotero_db=zot_db)
        stats = push_to_zotero(items, store=store, zotero_db=zot_db)
        store.close()

        assert stats["notes"] == 0  # second pass, nothing new

        con = sqlite3.connect(zot_db)
        n = con.execute("SELECT COUNT(*) FROM itemNotes").fetchone()[0]
        con.close()
        assert n == 1

• Step 3: Run tests to verify they fail

Run: uv run pytest tests/bib/test_sync.py::TestSyncNotes -v Expected: failures — KeyError: 'notes' (stats dict has no notes key) and assertion failures.

• Step 4: Implement _sync_notes in bib/sync.py

Add at the end of src/bib/sync.py (after _sync_attachments):

def _sync_notes(
    db: Db,
    store: Store,
    bib_item: Item,
    zot_parent_id: int,
) -> int:
    """Push every bib note for ``bib_item`` into Zotero's ``itemNotes``.

    Idempotent on (parentItemID, title): an existing same-title child
    note is left alone. Notes are emitted as child items of
    ``zot_parent_id``."""
    con = store._con()  # noqa: SLF001
    rows = con.execute(
        """SELECT n.title, n.content
           FROM notes n
           JOIN items i ON n.item_id = i.id
           WHERE i.key = ?""",
        (bib_item.key,),
    ).fetchall()
    if not rows:
        return 0

    existing_titles = {
        r[0]
        for r in db.con.execute(
            "SELECT title FROM itemNotes WHERE parentItemID = ?",
            (zot_parent_id,),
        ).fetchall()
    }

    count = 0
    for row in rows:
        title = row["title"] or ""
        if title in existing_titles:
            continue
        db.add_note(zot_parent_id, row["content"], title=title)
        existing_titles.add(title)
        count += 1
    return count

• Step 5: Wire _sync_notes into both branches of push_to_zotero

In src/bib/sync.py:

(a) Add notes to the stats dict initializer (around line 258):

        stats: dict[str, int] = {
            "created": 0,
            "skipped": 0,
            "tags": 0,
            "collections": 0,
            "attachments": 0,
            "notes": 0,
        }

(b) In the update branch (after _sync_attachments call around line 301):

                    if store is not None:
                        stats["attachments"] += _sync_attachments(
                            db, store, item, existing_id, storage_dir,
                        )
                        stats["notes"] += _sync_notes(
                            db, store, item, existing_id,
                        )

(c) In the create branch (after _sync_attachments call around line 367):

            if store is not None:
                stats["attachments"] += _sync_attachments(
                    db, store, item, item_id, storage_dir,
                )
                stats["notes"] += _sync_notes(
                    db, store, item, item_id,
                )

• Step 6: Run tests to verify they pass

Run: uv run pytest tests/bib/test_sync.py -v Expected: all sync tests pass, including the two new TestSyncNotes tests. Run the full sync test file (not just the new class) to catch regressions in the dict initialization.

• Step 7: Commit

git add src/bib/sync.py tests/bib/test_sync.py
git commit -m "$(cat <<'EOF'
feat(bib/sync): push bib notes into Zotero itemNotes

Mirrors _sync_attachments. Idempotent dedup keyed on
(parentItemID, title) so re-syncing is a no-op. Wired into both
the create-item and update-existing-item branches of push_to_zotero.
Adds 'notes' counter to the stats dict.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"

---

Task 4: Flip cli/comments.py from attach_file to attach_note

Files:

• Modify: src/cli/comments.py:60-100 (_build_bib_helpers — replace per-file glob with single-note attach)

• Modify: tests/cli/test_comments.py:62-107 (rename + rewrite the existing attach test)

• Step 1: Rewrite the existing attach test

Replace test_extract_attaches_only_sibling_mds_to_bib in tests/cli/test_comments.py (lines 62–107) with:

def test_extract_attaches_combined_md_as_note(tmp_path: Path, monkeypatch):
    """Extract writes combined.md and per-attachment sibling MDs to disk.
    Only one Zotero-bound artifact: a single bib note titled 'Comment text'
    whose content is the rendered-HTML of combined.md. No file attachments."""
    from bib import connect
    from bib.item import Source

    bib_db = tmp_path / "bib.sqlite"
    monkeypatch.setattr("conf.path", lambda _: bib_db)

    cdir = _seed(tmp_path)
    comment_id = cdir.name  # CMS-2024-0001-0001

    store = connect(str(bib_db))
    item = Source(
        title=f"Comment {comment_id}",
        url=f"https://www.regulations.gov/comment/{comment_id}",
    )
    item_key = store.upsert(item)
    store.close()

    result = runner.invoke(app, ["extract", "--root", str(tmp_path), "--workers", "1"])
    assert result.exit_code == 0, result.output

    # Disk siblings still written (kept for non-Zotero corpus consumers)
    assert (cdir / "combined.md").is_file()
    assert (cdir / "attachment_1.pdf.md").is_file()

    def md_attachments():
        store = connect(str(bib_db))
        rows = list(
            store._con().execute(
                "SELECT a.filename FROM attachments a "
                "JOIN items i ON a.item_id=i.id "
                "WHERE i.key=? AND a.filename LIKE '%.md'",
                (item_key,),
            )
        )
        store.close()
        return rows

    def notes():
        store = connect(str(bib_db))
        rows = list(
            store._con().execute(
                "SELECT n.title, n.content FROM notes n "
                "JOIN items i ON n.item_id=i.id WHERE i.key=?",
                (item_key,),
            )
        )
        store.close()
        return rows

    # No file attachments — old behavior is gone.
    assert md_attachments() == []

    # Exactly one note, rendered HTML.
    rows = notes()
    assert len(rows) == 1
    title, content = rows[0]
    assert title == "Comment text"
    assert "<h1>" in content or "<h2>" in content
    assert "Long body content" in content  # the seeded PDF text shows up

    # Idempotent on re-run
    runner.invoke(app, ["extract", "--root", str(tmp_path), "--workers", "1"])
    assert len(notes()) == 1
    assert md_attachments() == []

• Step 2: Run test to verify it fails

Run: uv run pytest tests/cli/test_comments.py::test_extract_attaches_combined_md_as_note -v Expected: failure — current code path still calls attach_file, so md_attachments() returns 1 row and notes() returns 0 rows.

• Step 3: Update _build_bib_helpers in cli/comments.py

Replace the attach_callback block in src/cli/comments.py (lines 60–100). The full new shape of the function (lines 38 onward):

    if not use_bib:
        return lambda _cid: "", None

    import sqlite3

    from conf import path

    db = str(path("db.bib"))
    con = sqlite3.connect(db, check_same_thread=False)

    cache: dict[str, tuple[str, str]] = {}
    for row in con.execute(
        "SELECT key, url, abstract FROM items "
        "WHERE url LIKE 'https://www.regulations.gov/comment/%'"
    ):
        cid = (row[1] or "").rsplit("/", 1)[-1]
        if cid:
            cache[cid] = (row[0], row[2] or "")

    def body_lookup(comment_id: str) -> str:
        return cache.get(comment_id, ("", ""))[1]

    if not attach:
        return body_lookup, None

    from bib import connect
    from rex.comments.render import render_combined_md

    store = connect()
    # Items that already have a "Comment text" note — keeps re-runs idempotent.
    NOTE_TITLE = "Comment text"
    items_with_note: set[str] = {
        row[0]
        for row in store._con().execute(
            "SELECT i.key FROM notes n "
            "JOIN items i ON n.item_id = i.id "
            "WHERE n.title = ?",
            (NOTE_TITLE,),
        )
    }

    def attach_callback(comment_id: str, comment_dir: Path) -> None:
        info = cache.get(comment_id)
        if not info:
            return  # comment not in bib
        item_key = info[0]
        if item_key in items_with_note:
            return
        combined = comment_dir / "combined.md"
        if not combined.is_file():
            return  # extraction must have failed; nothing to attach
        try:
            html = render_combined_md(combined.read_text(encoding="utf-8"))
            store.attach_note(item_key, html, title=NOTE_TITLE)
            items_with_note.add(item_key)
        except Exception as e:  # noqa: BLE001
            log.warning("attach_note failed for %s: %s", item_key, e)

    return body_lookup, attach_callback

(Note: the old docstring lines 25–37 stay, but the part that mentions "registers combined.md as a bib attachment" needs an update — replace those four sentences with: "attach_callback(comment_id, comment_dir) -> None`` runs on the main thread and registers a single ``"Comment text"`` note on the bib item containing combined.md rendered to HTML. Skips items that already have one. Returns None when --no-bib or --no-attach.")

• Step 4: Run test to verify it passes

Run: uv run pytest tests/cli/test_comments.py -v Expected: all comments tests pass, especially the rewritten note assertion.

• Step 5: Commit

git add src/cli/comments.py tests/cli/test_comments.py
git commit -m "$(cat <<'EOF'
feat(comments): attach combined.md as a Zotero note (HTML), not files

Replaces the per-attachment .md attach_file loop with a single
attach_note call per comment item: combined.md rendered to HTML via
rex.comments.render. Sibling .md files keep being written to disk
for non-Zotero corpus consumers.

Idempotency cache shifts from (item_key, filename) to a per-item
"Comment text" title check.

Migration of the 33,351 existing .md attachments lands in a
follow-up one-shot script.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"

---

Task 5: Migration script — Phase A (note backfill)

Files:

• Create: scripts/migrate_comment_md_to_notes.py

• Step 1: Write the script (Phase A only for now — destructive bits land in Task 6)

Create scripts/migrate_comment_md_to_notes.py:

#!/usr/bin/env python
"""One-shot: comment .md attachments → Zotero notes.

Phases:
    A (default):  for every .state/comments/<docket>/<id>/combined.md,
                  attach a single 'Comment text' note (rendered HTML)
                  to the bib item. Idempotent; safe to retry.

    C (--cleanup): destructively delete every .md row from bib.attachments,
                  matching itemAttachments + items rows in Zotero, and the
                  per-attachment storage dirs. Prompts before destruction.

Phase B (push notes to Zotero) lives outside this script — operator runs
'stack bib sync' between A and C.

After a successful migration, this file should be removed in a follow-up
commit.
"""

from __future__ import annotations

import argparse
import logging
import shutil
import sqlite3
import sys
from pathlib import Path

logging.basicConfig(format="%(asctime)s %(levelname)s %(message)s", level=logging.INFO)
log = logging.getLogger("migrate-comments-md-to-notes")


COMMENTS_ROOT = Path(".state/comments")
NOTE_TITLE = "Comment text"


def _safety_preflight() -> None:
    """Refuse to run unless _sync_notes is on disk — sanity check that
    re-syncing won't recreate the same problem we're cleaning up."""
    sync_py = Path("src/bib/sync.py")
    if not sync_py.is_file():
        sys.exit("FATAL: src/bib/sync.py missing — run from repo root.")
    if "_sync_notes" not in sync_py.read_text(encoding="utf-8"):
        sys.exit(
            "FATAL: src/bib/sync.py has no _sync_notes — apply Task 3 first."
        )


def phase_a_backfill(root: Path) -> dict[str, int]:
    """Walk comment dirs, attach combined.md as a 'Comment text' note."""
    from bib import connect
    from rex.comments.render import render_combined_md

    store = connect()
    con = store._con()  # noqa: SLF001

    # comment_id → bib item_key
    keys: dict[str, str] = {
        (row[1] or "").rsplit("/", 1)[-1]: row[0]
        for row in con.execute(
            "SELECT key, url FROM items "
            "WHERE url LIKE 'https://www.regulations.gov/comment/%'"
        )
    }

    # item_keys that already have a 'Comment text' note → skip
    have_note: set[str] = {
        row[0]
        for row in con.execute(
            "SELECT i.key FROM notes n "
            "JOIN items i ON n.item_id = i.id "
            "WHERE n.title = ?",
            (NOTE_TITLE,),
        )
    }

    stats = {"scanned": 0, "attached": 0, "skipped_have_note": 0,
             "skipped_no_combined": 0, "skipped_unknown_item": 0,
             "errors": 0}

    for combined in root.rglob("combined.md"):
        stats["scanned"] += 1
        comment_id = combined.parent.name
        item_key = keys.get(comment_id)
        if not item_key:
            stats["skipped_unknown_item"] += 1
            continue
        if item_key in have_note:
            stats["skipped_have_note"] += 1
            continue
        try:
            html = render_combined_md(combined.read_text(encoding="utf-8"))
            store.attach_note(item_key, html, title=NOTE_TITLE)
            have_note.add(item_key)
            stats["attached"] += 1
        except Exception as e:  # noqa: BLE001
            log.warning("attach_note failed for %s: %s", item_key, e)
            stats["errors"] += 1

        if stats["scanned"] % 1000 == 0:
            log.info("phase A progress: %s", stats)

    return stats


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--root", type=Path, default=COMMENTS_ROOT,
        help="Comments root (default: .state/comments)",
    )
    parser.add_argument(
        "--cleanup", action="store_true",
        help="After Phase A, also run Phase C (destructive: deletes all "
             "*.md rows from bib + Zotero and removes their storage dirs).",
    )
    args = parser.parse_args()

    _safety_preflight()

    if not args.root.is_dir():
        sys.exit(f"FATAL: comments root not found: {args.root}")

    log.info("Phase A: backfilling notes from %s", args.root)
    stats_a = phase_a_backfill(args.root)
    log.info("Phase A complete: %s", stats_a)

    if not args.cleanup:
        log.info(
            "Done. Run 'stack bib sync' to push the notes to Zotero, "
            "then re-run with --cleanup to delete the old .md attachments."
        )
        return

    # Phase C lands in Task 6.
    log.warning("--cleanup not yet implemented; landing in Task 6")


if __name__ == "__main__":
    main()

• Step 2: Smoke-test against a tiny dummy fixture

Build a tmp comments dir with one combined.md, run Phase A against a tmp bib, verify a note exists.

uv run python -c "
import sqlite3, tempfile, pathlib, os
from bib import connect
from bib.item import Source

# tmp bib with one comment item
tmp = pathlib.Path(tempfile.mkdtemp())
db = tmp / 'bib.sqlite'
os.environ['BIB_DB'] = str(db)
store = connect(str(db))
key = store.upsert(Source(
    title='Comment CMS-2024-0001-0001',
    url='https://www.regulations.gov/comment/CMS-2024-0001-0001',
))
store.close()

# tmp comments dir with one combined.md
cdir = tmp / 'comments' / 'CMS-2024-0001' / 'CMS-2024-0001-0001'
cdir.mkdir(parents=True)
(cdir / 'combined.md').write_text(
    '---\nstatus: ok\n---\n\n# Body\n\nText.\n'
)

# Run Phase A directly
from scripts.migrate_comment_md_to_notes import phase_a_backfill
stats = phase_a_backfill(tmp / 'comments')
print('STATS:', stats)
assert stats['attached'] == 1, stats

# Verify note exists
con = sqlite3.connect(str(db))
title, content = con.execute('SELECT title, content FROM notes').fetchone()
print('NOTE:', title)
assert title == 'Comment text'
assert '<h1>Body</h1>' in content
print('OK')
"

Expected: STATS: {'scanned': 1, 'attached': 1, ...} and NOTE: Comment text and OK. (Path import via scripts. may need a sys.path.insert; if the import fails, run with PYTHONPATH=. prefix.)

• Step 3: Commit

git add scripts/migrate_comment_md_to_notes.py
git commit -m "$(cat <<'EOF'
chore(scripts): migration Phase A — backfill comment notes

Walk .state/comments/, attach combined.md (rendered HTML) as a
single 'Comment text' bib note per item. Idempotent; --cleanup
flag for the destructive Phase C lands next.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"

---

Task 6: Migration script — Phase C (destructive cleanup)

Files:

• Modify: scripts/migrate_comment_md_to_notes.py — replace the Phase C placeholder

• Step 1: Add Phase C implementation

In scripts/migrate_comment_md_to_notes.py, replace the placeholder block:

    # Phase C lands in Task 6.
    log.warning("--cleanup not yet implemented; landing in Task 6")

with:

    log.info("Phase C: destructive cleanup of .md file attachments")
    counts = _phase_c_preview()
    log.info("Will delete: %s", counts)
    if not _confirm("Proceed with destructive cleanup?"):
        log.info("Aborted.")
        return
    stats_c = phase_c_cleanup()
    log.info("Phase C complete: %s", stats_c)

And add these helpers above main():

def _confirm(msg: str) -> bool:
    """Prompt y/N. Defaults to N."""
    reply = input(f"{msg} [y/N] ").strip().lower()
    return reply == "y"


def _phase_c_preview() -> dict[str, int]:
    from conf import path
    bib_db = sqlite3.connect(str(path("db.bib")))
    n_bib = bib_db.execute(
        "SELECT COUNT(*) FROM attachments WHERE filename LIKE '%.md'"
    ).fetchone()[0]
    bib_db.close()
    zot_db = sqlite3.connect(str(path("db.zotero")))
    n_zot = zot_db.execute(
        "SELECT COUNT(*) FROM itemAttachments WHERE path LIKE 'storage:%.md'"
    ).fetchone()[0]
    zot_db.close()
    return {"bib_md_attachments": n_bib, "zotero_md_attachments": n_zot}


def phase_c_cleanup() -> dict[str, int]:
    """Delete every .md attachment row + storage dir on bib and Zotero."""
    from conf import path

    stats = {"bib_rows_deleted": 0, "bib_dirs_removed": 0,
             "zot_rows_deleted": 0, "zot_dirs_removed": 0,
             "zot_dirs_skipped_nonempty": 0}

    # ── bib ──────────────────────────────────────────
    bib_con = sqlite3.connect(str(path("db.bib")))
    bib_rows = bib_con.execute(
        "SELECT id, storage_path FROM attachments WHERE filename LIKE '%.md'"
    ).fetchall()
    for _att_id, storage_path in bib_rows:
        if storage_path:
            d = Path(storage_path).parent
            if d.is_dir():
                shutil.rmtree(d, ignore_errors=True)
                stats["bib_dirs_removed"] += 1
    bib_con.execute("DELETE FROM attachments WHERE filename LIKE '%.md'")
    stats["bib_rows_deleted"] = bib_con.total_changes
    bib_con.commit()
    bib_con.close()

    # ── Zotero ───────────────────────────────────────
    zot_db_path = str(path("db.zotero"))
    zot_storage = Path(zot_db_path).parent / "storage"
    zot_con = sqlite3.connect(zot_db_path)
    zot_rows = zot_con.execute(
        "SELECT i.itemID, i.key FROM items i "
        "JOIN itemAttachments ia ON ia.itemID = i.itemID "
        "WHERE ia.path LIKE 'storage:%.md'"
    ).fetchall()

    item_ids_to_delete: list[int] = []
    for item_id, item_key in zot_rows:
        d = zot_storage / item_key
        if d.is_dir():
            non_md = [p for p in d.iterdir() if not p.name.endswith(".md")]
            if non_md:
                log.warning("skip storage dir %s (contains non-md: %s)",
                            d, [p.name for p in non_md])
                stats["zot_dirs_skipped_nonempty"] += 1
                # We still drop the rows — Zotero will show a missing
                # attachment, easier to clean than a phantom row.
            else:
                shutil.rmtree(d, ignore_errors=True)
                stats["zot_dirs_removed"] += 1
        item_ids_to_delete.append(item_id)

    if item_ids_to_delete:
        placeholders = ",".join("?" * len(item_ids_to_delete))
        zot_con.execute(
            f"DELETE FROM itemAttachments WHERE itemID IN ({placeholders})",
            item_ids_to_delete,
        )
        zot_con.execute(
            f"DELETE FROM items WHERE itemID IN ({placeholders})",
            item_ids_to_delete,
        )
        stats["zot_rows_deleted"] = len(item_ids_to_delete)
    zot_con.commit()
    zot_con.close()

    return stats

• Step 2: Smoke-test the safety pre-flight only

Don't run against real data yet — just confirm the safety guard fires when expected. Comment out _sync_notes momentarily, run, restore:

# Save current sync.py
cp src/bib/sync.py /tmp/sync.py.bak

# Strip the helper temporarily
sed -i 's/_sync_notes/_disabled_sync_notes/g' src/bib/sync.py
uv run python scripts/migrate_comment_md_to_notes.py --cleanup 2>&1 | head -3
# Expected: "FATAL: src/bib/sync.py has no _sync_notes — apply Task 3 first."

# Restore
cp /tmp/sync.py.bak src/bib/sync.py
rm /tmp/sync.py.bak

Expected: the FATAL message appears; non-zero exit. Restore leaves git status clean.

• Step 3: Commit

git add scripts/migrate_comment_md_to_notes.py
git commit -m "$(cat <<'EOF'
chore(scripts): migration Phase C — destructive .md attachment cleanup

Gated by --cleanup flag and y/N confirmation. Walks bib.attachments
and Zotero itemAttachments + items, removes storage dirs that
contain only the .md file (non-empty dirs with foreign content are
logged and the dir is skipped, but the SQL rows still drop).

Safety pre-flight refuses to run unless src/bib/sync.py contains
_sync_notes — sanity check that re-syncing won't recreate the
attachments we're deleting.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"

---

Task 7: Execute the migration against real data

Not a code change — operator-driven steps. Stop and confirm with the user before each phase.

• Step 1: Snapshot current counts

uv run python -c "
import sqlite3
from conf import path
b = sqlite3.connect(str(path('db.bib')))
print('bib .md:', b.execute(\"SELECT COUNT(*) FROM attachments WHERE filename LIKE '%.md'\").fetchone()[0])
print('bib notes:', b.execute(\"SELECT COUNT(*) FROM notes WHERE title='Comment text'\").fetchone()[0])
b.close()
z = sqlite3.connect(str(path('db.zotero')))
print('zot .md attachments:', z.execute(\"SELECT COUNT(*) FROM itemAttachments WHERE path LIKE 'storage:%.md'\").fetchone()[0])
print('zot Comment text notes:', z.execute(\"SELECT COUNT(*) FROM itemNotes WHERE title='Comment text'\").fetchone()[0])
z.close()
"

Expected starting state: bib .md ≈ 33,351; bib notes 0; zot .md attachments ≈ 33,351; zot Comment text notes 0.

• Step 2: Run Phase A

uv run python scripts/migrate_comment_md_to_notes.py

Expected: log lines every 1000 dirs; final stats line with attached ≈ 23,605.

Re-run the snapshot from Step 1. bib notes should now ≈ 23,605.

• Step 3: Push to Zotero (Phase B)

uv run stack bib sync

Expected: stats line includes notes: ~23605.

Re-run snapshot. zot Comment text notes should now ≈ 23,605. (Zot .md attachments still ≈ 33,351 at this point — Phase C handles those.)

• Step 4: Sanity check in Zotero UI before destructive cleanup

Open Zotero, find a CMS-2024 comment, confirm:

• The "Comment text" child note exists.
• Opening it in the right pane shows rendered HTML (headings, paragraphs, tables) — NOT raw markdown source.

If anything looks off (note empty, looks like raw markdown, formatting busted), STOP. Don't proceed to Phase C.

• Step 5: Run Phase C with confirmation

uv run python scripts/migrate_comment_md_to_notes.py --cleanup

Expected:

• Phase A re-runs as no-op (attached: 0, skipped_have_note: ~23605).

• "Will delete: {'bib_md_attachments': ~33351, 'zotero_md_attachments': ~33351}".

• Confirmation prompt — review counts, type y.

• Phase C completes with bib_rows_deleted: ~33351, zot_rows_deleted: ~33351, etc.

• Step 6: Verify final state

Re-run the snapshot from Step 1. Expected:

• bib .md: 0
• bib notes: ~23,605
• zot .md attachments: 0
• zot Comment text notes: ~23,605

Spot-check Zotero UI: pick a few items, confirm child shows note (not file attachment).

---

Task 8: Remove the migration script

Files:

• Delete: scripts/migrate_comment_md_to_notes.py

• Step 1: Delete the script

git rm scripts/migrate_comment_md_to_notes.py

• Step 2: Commit

git commit -m "$(cat <<'EOF'
chore(scripts): remove one-shot comment-md→notes migration

Run successfully against the 23,605 comment items / 33,351 .md
file attachments. bib + Zotero now hold only the rendered-HTML
'Comment text' notes; old file attachments and storage dirs are
gone. Restoring this script for re-runs is one git revert away.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"

---

Self-Review Notes

After writing this plan I checked:

1. Spec coverage: every section of the spec maps to a task — renderer (T2), write path (T4), sync path (T3), migration phases A/B/C (T5/T7/T6), siblings-stay-on-disk (T4 keeps the assertion), deps (T1), code touch list (covered across tasks).
2. Type consistency: NOTE_TITLE = "Comment text" is the same string in cli/comments.py, the migration script, and the test assertions. _sync_notes signature matches _sync_attachments. Stats key notes consistent across sync.py, tests, and migration steps.
3. Placeholders: none of the forbidden patterns. Every code step includes the actual code.
4. Out-of-order safety: Tasks 1-6 are merge-safe in any order within their phase but the listed order is the strict dependency chain — T1 unlocks T2, T2 unlocks T4 and T5, T3 unlocks T7 step 3.