stack/.claude/specs/2026-04-28-comments-zotero-notes-design.md

# Comments → Zotero Notes (instead of file attachments)

**Status:** approved 2026-04-28
**Owner:** kert
**Related:** #253 (comments extraction), commits 684fb7f / fafe661 / c53571c

## Problem

Recent comments extraction work (commits 684fb7f, fafe661, c53571c) attaches per-attachment `.md` sibling files back to bib via `store.attach_file`. After `bib sync` these land in Zotero as **file attachments**. Zotero displays file attachments as opaque rows that require an external editor to open — the markdown text never renders in Zotero's reader pane.

Concrete state: 33,351 `.md` rows in `bib.attachments` across 23,605 parent comment items, with matching `itemAttachments` rows and storage dirs in Zotero. None of them are readable from the Zotero UI.

## Goal

Make the extracted comment text directly readable inside Zotero by storing it as **Zotero notes** (HTML, rendered in the right pane) rather than file attachments.

## Decisions (locked in via brainstorming)

1. **Granularity:** one consolidated note per comment item, body = `combined.md` (the inline body + every attachment's text concatenated, with H2 attachment-name anchors). One child per item rather than 1+N.
2. **Rendering:** render markdown → HTML before storing. `# Heading` becomes `<h1>Heading</h1>`, tables render as HTML tables, etc. Zotero displays a properly formatted document, not literal `**bold**` source.
3. **Migration:** one-shot — delete every existing `.md` attachment from `bib.attachments`, `bib` storage, Zotero `itemAttachments`, Zotero `items`, and Zotero storage dirs. Then re-attach as notes.
4. **On-disk siblings:** keep them. `combined.md` and per-attachment `<name>.md` continue to be written under `.state/comments/<docket>/<id>/`. They're cheap, useful for non-Zotero tooling, and avoid a re-parse round-trip later.
5. **Storage shape:** `bib.notes.content` stores rendered HTML, not markdown source. Zotero is the only consumer; doing markdown → HTML once at write time keeps `bib/sync.py` ignorant of markdown.

## Architecture

Three moving parts:

### 1. Renderer

New module `src/rex/comments/render.py`:

```python
def render_combined_md(text: str) -> str:
    """Strip YAML frontmatter, render the body to Zotero-edible HTML."""
```

- Strips the `---\n…\n---\n` frontmatter that `combine.extract_comment` writes at the top of `combined.md`.
- Runs the body through `markdown-it-py` with the `table` plugin enabled (CMS comments lean on tabular data).
- Returns sanitized HTML suitable for Zotero's `itemNotes.note` column.

### 2. Write path

`src/cli/comments.py`'s `attach_callback` (currently lines 78–98) changes shape:

- Stops globbing `*.md` and calling `store.attach_file` per file.
- Reads `combined.md` from the comment dir, calls `render_combined_md`, calls `store.attach_note(item_key, html, title="Comment text")`.
- Idempotency cache key changes from `(item_key, filename)` to `item_key` — single note per item, fixed title `"Comment text"`.
- `combined.md` continues to be written by `combine.extract_comment`; per-attachment `<name>.md` siblings continue to be written for on-disk corpus consumers.

### 3. Sync path

`src/bib/sync.py` gains a `_sync_notes` helper mirroring `_sync_attachments`:

```python
def _sync_notes(db, store, bib_item, zot_parent_id) -> int:
    """Push every bib note for this item into Zotero's itemNotes.
    Idempotent on (parentItemID, title)."""
```

Wired into both branches of `push_to_zotero` (the create branch around line 367 and the update branch around line 300). The stats dict gains a `notes` counter.

`zot.db.Db.add_note` (already present at db.py:682) handles the actual insert.

## Migration

One-shot script: `scripts/migrate_comment_md_to_notes.py`. Three phases, each idempotent:

### Phase A — backfill notes (additive, safe to retry)

For every comment dir under `.state/comments/<docket>/<id>/` that has `combined.md`:

1. Look up `item_key` via the existing comment_id → item_key cache (same query as `_build_bib_helpers` in cli/comments.py).
2. Skip if a note titled `"Comment text"` already exists for that item.
3. `render_combined_md(combined_md)` → HTML.
4. `store.attach_note(item_key, html, title="Comment text")`.

Populates `bib.notes` for all 23,605 items. Idempotent by title-dedup.

### Phase B — push to Zotero (operator step, not part of the migration script)

After Phase A completes, the operator runs `stack bib sync` (the same command they'd run after any bib mutation). The new `_sync_notes` plumbing turns each bib note into a child Zotero note item. Idempotent by `(parentItemID, title)` dedup. Kept outside the migration script because re-running `bib sync` is a normal day-to-day op, not a one-shot.

### Phase C — destructive cleanup (gated by `--cleanup` flag + confirmation)

Three deletes in order:

1. **bib side:**
   - `SELECT key, storage_path FROM attachments WHERE filename LIKE '%.md'` → list.
   - For each: `rm -rf` the parent dir of `storage_path`.
   - `DELETE FROM attachments WHERE filename LIKE '%.md'`.

2. **Zotero side:**
   ```sql
   SELECT i.itemID, i.key, ia.path
   FROM items i
   JOIN itemAttachments ia ON ia.itemID = i.itemID
   WHERE ia.path LIKE 'storage:%.md'
   ```
   For each row:
   - Inspect `data/zotero/data/storage/<i.key>/`. If anything other than the `.md` file is present, log and skip — don't nuke unrelated content.
   - Otherwise `rm -rf` the dir.
   - `DELETE FROM itemAttachments WHERE itemID = ?`.
   - `DELETE FROM items WHERE itemID = ?`.

3. **Verify:** re-run the count queries from the **Problem** section above. Expect `0` in both.

### Safety pre-flight

- Script refuses to run unless `_sync_notes` exists in `src/bib/sync.py` (sanity check that re-syncing won't recreate the same problem).
- Phase C prints a row count and prompts for `y/N` confirmation before any destructive action.
- All three phases are independently restartable: A and B by their dedup logic, C because the SQL deletes are no-ops once rows are gone.

After the migration runs successfully, the script is deleted in a follow-up commit (same pattern as the c53571c retitle migration).

## Code touch list

| File | Change |
|---|---|
| `src/rex/comments/render.py` | **NEW** — `render_combined_md(text) -> str` |
| `src/cli/comments.py` | Replace `*.md` `attach_file` loop with single `attach_note` call. Idempotency cache key drops the filename component. |
| `src/bib/sync.py` | Add `_sync_notes`. Wire into create + update paths in `push_to_zotero`. Add `notes` counter to stats. |
| `pyproject.toml` | Add `markdown-it-py>=4.0` and `mdit-py-plugins>=0.4` to the `bib` extra (already in `uv.lock` as transitive). |
| `scripts/migrate_comment_md_to_notes.py` | **NEW** one-shot migration; deleted in a follow-up commit. |
| `tests/rex/comments/test_render.py` | **NEW** — frontmatter strip, basic markdown elements, table rendering, empty body |
| `tests/cli/test_comments.py` | Update attach-tests: assert `attach_note` called with rendered HTML and `"Comment text"` title; idempotency dedups by item, not filename |
| `tests/bib/test_sync.py` | New cases: bib note → Zotero `itemNotes` row on first sync; second sync is a no-op |

## Out of scope

- **Per-attachment notes.** Single consolidated note per item only.
- **Re-rendering after extractor improvements.** If OCR / parsing improves later, re-rendering 23k notes is a separate task. Title-keyed dedup will quietly skip them; a `--force-replace` flag would be added then.
- **`attach_note` / `Db.add_note` plumbing.** Already exists in `src/bib/store.py:500` and `src/zot/db.py:682`. This design uses them, doesn't change them.

## Risks / unknowns

1. **Zotero TinyMCE normalization** — Zotero's note editor rewrites HTML on first edit (drops unknown attrs, normalizes inline styles). Read-only viewing is unaffected. Round-tripped notes won't byte-match renderer output.
2. **Note size** — combined.md for the largest comments runs into hundreds of KB. Zotero handles it; list-view note-preview hover may be sluggish on those items. Acceptable.
3. **Storage dir edge case** — if a Zotero attachment's storage dir contains files we didn't write, the Phase C cleanup logs and skips rather than deleting them. Operator cleans by hand.
4. **`storage:%.md` LIKE match** — verified against `_sync_attachments` (sync.py:476) which writes `path = "storage:<filename>"`. Migration verifies one sample row before bulk delete.