comments: implement extract-ocr — 2,393 comments have image-only attachments invisible to all text exhibits #663
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: homelab/stack#663
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
stack comments extract-ocrhas been a phase-2 stub since #253. As of 2026-08-19, 2,393 comments across the ten PFS dockets haveocr_neededattachments (image-only PDFs and JPG/PNG scans) whose text is invisible to every downstream consumer — the RAG index, the palliative sweep (#660), and any docket exhibit. The count will grow as #662's backfill pulls the remaining ~160k comment details.The GPU fleet (P45, #653-#655: rack 3060 + laptop 5070 Ti + rig 4090) is live and the distributed-GPU OCR notes are already in the spec (
76c7288phase-2 marker). Implement the OCR pass (tesseract or a vision model on the fleet), update each comment'scombined.mdin place (attachment statusocr_needed→ok), and re-attach the "Comment text" bib note.Implemented (2026-08-20):
stack comments extract-ocr [--docket] [--limit]— RapidOCR (onnxruntime, pip-only, no system packages:uv run --with rapidocr-onnxruntime), PDFs rasterized per page through PyMuPDF at 200 dpi (no poppler), images OCR'd directly. Hooked intoextract_attachment(ocr_engine=...)so a re-extract with the engine rewrites combined.md + sibling transcripts and clears theocr_neededflag only when OCR clears the threshold. TDD'd (tests/rex/comments/test_extract_ocr.py); smoke: 3 scanned letters → 5-10k chars each. Full run over the 2,738-dir backlog launched (niced,.state/comments/ocr.log), chained with an idempotent palliative re-sweep so #660's exhibit picks up newly readable text. Leaving open until the run completes; note the bib 'Comment text' notes are not yet refreshed for OCR'd dirs (attach is skip-if-exists) — follow-up if Zotero-side transcripts matter.Run complete (2026-08-20 18:38 ET): all 2,738 queued comment dirs re-extracted with OCR, 0 errors (~6.5 hrs CPU, niced). The chained palliative re-sweep found +1 new hit (topic:palliative 417 → 418) — the scanned-attachment population was largely not palliative-relevant, a useful negative result now disclosed by the live notebook exhibit. Remaining known gap: bib 'Comment text' notes are not refreshed for OCR'd dirs (attach is skip-if-exists) — cosmetic for Zotero browsing, not for any exhibit, so closing.