stack/.claude/specs/2026-04-28-comments-zotero-notes-design.md

Comments → Zotero Notes (instead of file attachments)

Status: approved 2026-04-28 Owner: kert Related: #253 (comments extraction), commits 684fb7f / fafe661 / c53571c

Problem

Recent comments extraction work (commits 684fb7f, fafe661, c53571c) attaches per-attachment .md sibling files back to bib via store.attach_file. After bib sync these land in Zotero as file attachments. Zotero displays file attachments as opaque rows that require an external editor to open — the markdown text never renders in Zotero's reader pane.

Concrete state: 33,351 .md rows in bib.attachments across 23,605 parent comment items, with matching itemAttachments rows and storage dirs in Zotero. None of them are readable from the Zotero UI.

Goal

Make the extracted comment text directly readable inside Zotero by storing it as Zotero notes (HTML, rendered in the right pane) rather than file attachments.

Decisions (locked in via brainstorming)

1. Granularity: one consolidated note per comment item, body = combined.md (the inline body + every attachment's text concatenated, with H2 attachment-name anchors). One child per item rather than 1+N.
2. Rendering: render markdown → HTML before storing. # Heading becomes <h1>Heading</h1>, tables render as HTML tables, etc. Zotero displays a properly formatted document, not literal **bold** source.
3. Migration: one-shot — delete every existing .md attachment from bib.attachments, bib storage, Zotero itemAttachments, Zotero items, and Zotero storage dirs. Then re-attach as notes.
4. On-disk siblings: keep them. combined.md and per-attachment <name>.md continue to be written under .state/comments/<docket>/<id>/. They're cheap, useful for non-Zotero tooling, and avoid a re-parse round-trip later.
5. Storage shape: bib.notes.content stores rendered HTML, not markdown source. Zotero is the only consumer; doing markdown → HTML once at write time keeps bib/sync.py ignorant of markdown.

Architecture

Three moving parts:

1. Renderer

New module src/rex/comments/render.py:

def render_combined_md(text: str) -> str:
    """Strip YAML frontmatter, render the body to Zotero-edible HTML."""

• Strips the ---\n…\n---\n frontmatter that combine.extract_comment writes at the top of combined.md.
• Runs the body through markdown-it-py with the table plugin enabled (CMS comments lean on tabular data).
• Returns sanitized HTML suitable for Zotero's itemNotes.note column.

2. Write path

src/cli/comments.py's attach_callback (currently lines 78–98) changes shape:

• Stops globbing *.md and calling store.attach_file per file.
• Reads combined.md from the comment dir, calls render_combined_md, calls store.attach_note(item_key, html, title="Comment text").
• Idempotency cache key changes from (item_key, filename) to item_key — single note per item, fixed title "Comment text".
• combined.md continues to be written by combine.extract_comment; per-attachment <name>.md siblings continue to be written for on-disk corpus consumers.

3. Sync path

src/bib/sync.py gains a _sync_notes helper mirroring _sync_attachments:

def _sync_notes(db, store, bib_item, zot_parent_id) -> int:
    """Push every bib note for this item into Zotero's itemNotes.
    Idempotent on (parentItemID, title)."""

Wired into both branches of push_to_zotero (the create branch around line 367 and the update branch around line 300). The stats dict gains a notes counter.

zot.db.Db.add_note (already present at db.py:682) handles the actual insert.

Migration

One-shot script: scripts/migrate_comment_md_to_notes.py. Three phases, each idempotent:

Phase A — backfill notes (additive, safe to retry)

For every comment dir under .state/comments/<docket>/<id>/ that has combined.md:

1. Look up item_key via the existing comment_id → item_key cache (same query as _build_bib_helpers in cli/comments.py).
2. Skip if a note titled "Comment text" already exists for that item.
3. render_combined_md(combined_md) → HTML.
4. store.attach_note(item_key, html, title="Comment text").

Populates bib.notes for all 23,605 items. Idempotent by title-dedup.

Phase B — push to Zotero (operator step, not part of the migration script)

After Phase A completes, the operator runs stack bib sync (the same command they'd run after any bib mutation). The new _sync_notes plumbing turns each bib note into a child Zotero note item. Idempotent by (parentItemID, title) dedup. Kept outside the migration script because re-running bib sync is a normal day-to-day op, not a one-shot.

Phase C — destructive cleanup (gated by --cleanup flag + confirmation)

Three deletes in order:

1. bib side:

   • SELECT key, storage_path FROM attachments WHERE filename LIKE '%.md' → list.
   • For each: rm -rf the parent dir of storage_path.
   • DELETE FROM attachments WHERE filename LIKE '%.md'.

2. Zotero side:

   SELECT i.itemID, i.key, ia.path
   FROM items i
   JOIN itemAttachments ia ON ia.itemID = i.itemID
   WHERE ia.path LIKE 'storage:%.md'
   

   For each row:

   • Inspect data/zotero/data/storage/<i.key>/. If anything other than the .md file is present, log and skip — don't nuke unrelated content.
   • Otherwise rm -rf the dir.
   • DELETE FROM itemAttachments WHERE itemID = ?.
   • DELETE FROM items WHERE itemID = ?.

3. Verify: re-run the count queries from the Problem section above. Expect 0 in both.

Safety pre-flight

• Script refuses to run unless _sync_notes exists in src/bib/sync.py (sanity check that re-syncing won't recreate the same problem).
• Phase C prints a row count and prompts for y/N confirmation before any destructive action.
• All three phases are independently restartable: A and B by their dedup logic, C because the SQL deletes are no-ops once rows are gone.

After the migration runs successfully, the script is deleted in a follow-up commit (same pattern as the c53571c retitle migration).

Code touch list

File	Change
src/rex/comments/render.py	NEW — render_combined_md(text) -> str
src/cli/comments.py	Replace *.md attach_file loop with single attach_note call. Idempotency cache key drops the filename component.
src/bib/sync.py	Add _sync_notes. Wire into create + update paths in push_to_zotero. Add notes counter to stats.
pyproject.toml	Add markdown-it-py>=4.0 and mdit-py-plugins>=0.4 to the bib extra (already in uv.lock as transitive).
scripts/migrate_comment_md_to_notes.py	NEW one-shot migration; deleted in a follow-up commit.
tests/rex/comments/test_render.py	NEW — frontmatter strip, basic markdown elements, table rendering, empty body
tests/cli/test_comments.py	Update attach-tests: assert attach_note called with rendered HTML and "Comment text" title; idempotency dedups by item, not filename
tests/bib/test_sync.py	New cases: bib note → Zotero itemNotes row on first sync; second sync is a no-op

Out of scope

• Per-attachment notes. Single consolidated note per item only.
• Re-rendering after extractor improvements. If OCR / parsing improves later, re-rendering 23k notes is a separate task. Title-keyed dedup will quietly skip them; a --force-replace flag would be added then.
• attach_note / Db.add_note plumbing. Already exists in src/bib/store.py:500 and src/zot/db.py:682. This design uses them, doesn't change them.

Risks / unknowns

1. Zotero TinyMCE normalization — Zotero's note editor rewrites HTML on first edit (drops unknown attrs, normalizes inline styles). Read-only viewing is unaffected. Round-tripped notes won't byte-match renderer output.
2. Note size — combined.md for the largest comments runs into hundreds of KB. Zotero handles it; list-view note-preview hover may be sluggish on those items. Acceptable.
3. Storage dir edge case — if a Zotero attachment's storage dir contains files we didn't write, the Phase C cleanup logs and skips rather than deleting them. Operator cleans by hand.
4. storage:%.md LIKE match — verified against _sync_attachments (sync.py:476) which writes path = "storage:<filename>". Migration verifies one sample row before bulk delete.