docs(spec): comments → Zotero notes migration design
Switch comment-extract output in Zotero from file attachments to notes (HTML rendered from combined.md). Spec covers: renderer, write-path change in cli/comments.py, new note-sync plumbing in bib/sync.py, and a one-shot migration to retire the existing 33,351 *.md attachments across 23,605 items. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
kert
140
.claude/specs/2026-04-28-comments-zotero-notes-design.md
Normal file
140
.claude/specs/2026-04-28-comments-zotero-notes-design.md
Normal file
@@ -0,0 +1,140 @@
|
||||
# Comments → Zotero Notes (instead of file attachments)
|
||||
|
||||
**Status:** approved 2026-04-28
|
||||
**Owner:** kert
|
||||
**Related:** #253 (comments extraction), commits 684fb7f / fafe661 / c53571c
|
||||
|
||||
## Problem
|
||||
|
||||
Recent comments extraction work (commits 684fb7f, fafe661, c53571c) attaches per-attachment `.md` sibling files back to bib via `store.attach_file`. After `bib sync` these land in Zotero as **file attachments**. Zotero displays file attachments as opaque rows that require an external editor to open — the markdown text never renders in Zotero's reader pane.
|
||||
|
||||
Concrete state: 33,351 `.md` rows in `bib.attachments` across 23,605 parent comment items, with matching `itemAttachments` rows and storage dirs in Zotero. None of them are readable from the Zotero UI.
|
||||
|
||||
## Goal
|
||||
|
||||
Make the extracted comment text directly readable inside Zotero by storing it as **Zotero notes** (HTML, rendered in the right pane) rather than file attachments.
|
||||
|
||||
## Decisions (locked in via brainstorming)
|
||||
|
||||
1. **Granularity:** one consolidated note per comment item, body = `combined.md` (the inline body + every attachment's text concatenated, with H2 attachment-name anchors). One child per item rather than 1+N.
|
||||
2. **Rendering:** render markdown → HTML before storing. `# Heading` becomes `<h1>Heading</h1>`, tables render as HTML tables, etc. Zotero displays a properly formatted document, not literal `**bold**` source.
|
||||
3. **Migration:** one-shot — delete every existing `.md` attachment from `bib.attachments`, `bib` storage, Zotero `itemAttachments`, Zotero `items`, and Zotero storage dirs. Then re-attach as notes.
|
||||
4. **On-disk siblings:** keep them. `combined.md` and per-attachment `<name>.md` continue to be written under `.state/comments/<docket>/<id>/`. They're cheap, useful for non-Zotero tooling, and avoid a re-parse round-trip later.
|
||||
5. **Storage shape:** `bib.notes.content` stores rendered HTML, not markdown source. Zotero is the only consumer; doing markdown → HTML once at write time keeps `bib/sync.py` ignorant of markdown.
|
||||
|
||||
## Architecture
|
||||
|
||||
Three moving parts:
|
||||
|
||||
### 1. Renderer
|
||||
|
||||
New module `src/rex/comments/render.py`:
|
||||
|
||||
```python
|
||||
def render_combined_md(text: str) -> str:
|
||||
"""Strip YAML frontmatter, render the body to Zotero-edible HTML."""
|
||||
```
|
||||
|
||||
- Strips the `---\n…\n---\n` frontmatter that `combine.extract_comment` writes at the top of `combined.md`.
|
||||
- Runs the body through `markdown-it-py` with the `table` plugin enabled (CMS comments lean on tabular data).
|
||||
- Returns sanitized HTML suitable for Zotero's `itemNotes.note` column.
|
||||
|
||||
### 2. Write path
|
||||
|
||||
`src/cli/comments.py`'s `attach_callback` (currently lines 78–98) changes shape:
|
||||
|
||||
- Stops globbing `*.md` and calling `store.attach_file` per file.
|
||||
- Reads `combined.md` from the comment dir, calls `render_combined_md`, calls `store.attach_note(item_key, html, title="Comment text")`.
|
||||
- Idempotency cache key changes from `(item_key, filename)` to `item_key` — single note per item, fixed title `"Comment text"`.
|
||||
- `combined.md` continues to be written by `combine.extract_comment`; per-attachment `<name>.md` siblings continue to be written for on-disk corpus consumers.
|
||||
|
||||
### 3. Sync path
|
||||
|
||||
`src/bib/sync.py` gains a `_sync_notes` helper mirroring `_sync_attachments`:
|
||||
|
||||
```python
|
||||
def _sync_notes(db, store, bib_item, zot_parent_id) -> int:
|
||||
"""Push every bib note for this item into Zotero's itemNotes.
|
||||
Idempotent on (parentItemID, title)."""
|
||||
```
|
||||
|
||||
Wired into both branches of `push_to_zotero` (the create branch around line 367 and the update branch around line 300). The stats dict gains a `notes` counter.
|
||||
|
||||
`zot.db.Db.add_note` (already present at db.py:682) handles the actual insert.
|
||||
|
||||
## Migration
|
||||
|
||||
One-shot script: `scripts/migrate_comment_md_to_notes.py`. Three phases, each idempotent:
|
||||
|
||||
### Phase A — backfill notes (additive, safe to retry)
|
||||
|
||||
For every comment dir under `.state/comments/<docket>/<id>/` that has `combined.md`:
|
||||
|
||||
1. Look up `item_key` via the existing comment_id → item_key cache (same query as `_build_bib_helpers` in cli/comments.py).
|
||||
2. Skip if a note titled `"Comment text"` already exists for that item.
|
||||
3. `render_combined_md(combined_md)` → HTML.
|
||||
4. `store.attach_note(item_key, html, title="Comment text")`.
|
||||
|
||||
Populates `bib.notes` for all 23,605 items. Idempotent by title-dedup.
|
||||
|
||||
### Phase B — push to Zotero (operator step, not part of the migration script)
|
||||
|
||||
After Phase A completes, the operator runs `stack bib sync` (the same command they'd run after any bib mutation). The new `_sync_notes` plumbing turns each bib note into a child Zotero note item. Idempotent by `(parentItemID, title)` dedup. Kept outside the migration script because re-running `bib sync` is a normal day-to-day op, not a one-shot.
|
||||
|
||||
### Phase C — destructive cleanup (gated by `--cleanup` flag + confirmation)
|
||||
|
||||
Three deletes in order:
|
||||
|
||||
1. **bib side:**
|
||||
- `SELECT key, storage_path FROM attachments WHERE filename LIKE '%.md'` → list.
|
||||
- For each: `rm -rf` the parent dir of `storage_path`.
|
||||
- `DELETE FROM attachments WHERE filename LIKE '%.md'`.
|
||||
|
||||
2. **Zotero side:**
|
||||
```sql
|
||||
SELECT i.itemID, i.key, ia.path
|
||||
FROM items i
|
||||
JOIN itemAttachments ia ON ia.itemID = i.itemID
|
||||
WHERE ia.path LIKE 'storage:%.md'
|
||||
```
|
||||
For each row:
|
||||
- Inspect `data/zotero/data/storage/<i.key>/`. If anything other than the `.md` file is present, log and skip — don't nuke unrelated content.
|
||||
- Otherwise `rm -rf` the dir.
|
||||
- `DELETE FROM itemAttachments WHERE itemID = ?`.
|
||||
- `DELETE FROM items WHERE itemID = ?`.
|
||||
|
||||
3. **Verify:** re-run the count queries from the **Problem** section above. Expect `0` in both.
|
||||
|
||||
### Safety pre-flight
|
||||
|
||||
- Script refuses to run unless `_sync_notes` exists in `src/bib/sync.py` (sanity check that re-syncing won't recreate the same problem).
|
||||
- Phase C prints a row count and prompts for `y/N` confirmation before any destructive action.
|
||||
- All three phases are independently restartable: A and B by their dedup logic, C because the SQL deletes are no-ops once rows are gone.
|
||||
|
||||
After the migration runs successfully, the script is deleted in a follow-up commit (same pattern as the c53571c retitle migration).
|
||||
|
||||
## Code touch list
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `src/rex/comments/render.py` | **NEW** — `render_combined_md(text) -> str` |
|
||||
| `src/cli/comments.py` | Replace `*.md` `attach_file` loop with single `attach_note` call. Idempotency cache key drops the filename component. |
|
||||
| `src/bib/sync.py` | Add `_sync_notes`. Wire into create + update paths in `push_to_zotero`. Add `notes` counter to stats. |
|
||||
| `pyproject.toml` | Add `markdown-it-py>=4.0` and `mdit-py-plugins>=0.4` to the `bib` extra (already in `uv.lock` as transitive). |
|
||||
| `scripts/migrate_comment_md_to_notes.py` | **NEW** one-shot migration; deleted in a follow-up commit. |
|
||||
| `tests/rex/comments/test_render.py` | **NEW** — frontmatter strip, basic markdown elements, table rendering, empty body |
|
||||
| `tests/cli/test_comments.py` | Update attach-tests: assert `attach_note` called with rendered HTML and `"Comment text"` title; idempotency dedups by item, not filename |
|
||||
| `tests/bib/test_sync.py` | New cases: bib note → Zotero `itemNotes` row on first sync; second sync is a no-op |
|
||||
|
||||
## Out of scope
|
||||
|
||||
- **Per-attachment notes.** Single consolidated note per item only.
|
||||
- **Re-rendering after extractor improvements.** If OCR / parsing improves later, re-rendering 23k notes is a separate task. Title-keyed dedup will quietly skip them; a `--force-replace` flag would be added then.
|
||||
- **`attach_note` / `Db.add_note` plumbing.** Already exists in `src/bib/store.py:500` and `src/zot/db.py:682`. This design uses them, doesn't change them.
|
||||
|
||||
## Risks / unknowns
|
||||
|
||||
1. **Zotero TinyMCE normalization** — Zotero's note editor rewrites HTML on first edit (drops unknown attrs, normalizes inline styles). Read-only viewing is unaffected. Round-tripped notes won't byte-match renderer output.
|
||||
2. **Note size** — combined.md for the largest comments runs into hundreds of KB. Zotero handles it; list-view note-preview hover may be sluggish on those items. Acceptable.
|
||||
3. **Storage dir edge case** — if a Zotero attachment's storage dir contains files we didn't write, the Phase C cleanup logs and skips rather than deleting them. Operator cleans by hand.
|
||||
4. **`storage:%.md` LIKE match** — verified against `_sync_attachments` (sync.py:476) which writes `path = "storage:<filename>"`. Migration verifies one sample row before bulk delete.
|
||||
Reference in New Issue
Block a user