docs(spec): comments extraction pipeline design
Tracks #253. Captures Q1-Q6 design decisions: filesystem markdown + DuckDB view, PyMuPDF (fitz, AGPL-flagged), lazy OCR via separate phase-2 command, filesystem-only resume tracking, new stack comments CLI namespace, YAML frontmatter format. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
kert
139
.claude/specs/2026-04-23-comments-extraction-design.md
Normal file
139
.claude/specs/2026-04-23-comments-extraction-design.md
Normal file
@@ -0,0 +1,139 @@
|
||||
# Comments Extraction Pipeline — Design
|
||||
|
||||
**Tracks:** [#253](https://git.fhirworx.io/homelab/stack/issues/253)
|
||||
**Date:** 2026-04-23
|
||||
**Status:** Approved, ready for implementation plan
|
||||
|
||||
## Context
|
||||
|
||||
The reg.gov backfill (`bib backfill-comments`) downloads CMS rulemaking comments + their PDF/DOCX attachments into `.state/comments/{docket_id}/{comment_id}/`. As of 2026-04-23, ~142K of 164K reg-gov items are enriched, with ~18.5K PDFs and ~9.6K DOCX files on disk.
|
||||
|
||||
Per the hti5 reference project, **70%+ of substantive comment content lives in attachments, not the inline body**. Without text extraction, downstream pipelines (#254 position classification, #255 stakeholder/coordination detection, #256 dashboard) operate on the unsubstantive 30%.
|
||||
|
||||
## Decisions
|
||||
|
||||
| # | Question | Decision |
|
||||
|---|---|---|
|
||||
| Q1 | Where does extracted text live? | **Filesystem markdown + DuckDB view** over the markdown index |
|
||||
| Q2 | PDF library? | **PyMuPDF (fitz)** — fastest + best quality. AGPL-3.0 (flag if/when this code ships outside internal use) |
|
||||
| Q3 | OCR scope? | **Lazy OCR** — primary extraction stays fast, scanned PDFs tagged `ocr_needed`, separate `extract-ocr` command processes them |
|
||||
| Q4 | Resume tracking? | **Filesystem-only** — `combined.md` exists ⇒ done. No bib.sqlite tag writes (avoids contention with running backfill) |
|
||||
| Q5 | CLI namespace? | **New `cli/comments.py`** with `stack comments {extract,extract-ocr,stats}` — opens namespace for #254/#255/#256 commands |
|
||||
| Q6 | `combined.md` format? | **YAML frontmatter + markdown body** — frontmatter holds attachment status (used by `extract-ocr` to find candidates) |
|
||||
|
||||
## Module layout
|
||||
|
||||
```
|
||||
src/rex/comments/
|
||||
├── __init__.py # public API: extract_comment(), extract_attachment()
|
||||
├── extract.py # core: PyMuPDF + python-docx, per-attachment extraction
|
||||
├── combine.py # writes combined.md (frontmatter + body) per comment dir
|
||||
└── view.py # DuckDB view: read combined.md frontmatter + body
|
||||
|
||||
# Phase 2 (separate PR, not this work):
|
||||
# src/rex/comments/ocr.py — tesseract pass for ocr_needed attachments
|
||||
|
||||
src/cli/comments.py # NEW: stack comments {extract, extract-ocr, stats}
|
||||
tests/rex/comments/
|
||||
├── test_extract.py
|
||||
├── test_combine.py
|
||||
└── fixtures/ # 3-5 real .state/comments/ dirs as committed fixtures
|
||||
```
|
||||
|
||||
`prisma/export.py:_extract_one` is **not** refactored — out of scope.
|
||||
|
||||
## Data flow
|
||||
|
||||
For each `.state/comments/{docket}/{cid}/`:
|
||||
|
||||
1. Skip if `combined.md` exists (resume rule).
|
||||
2. For each `attachment_N.{pdf,docx,...}` in the dir:
|
||||
- **PDF**: `fitz.open(p)` → `"\n\n".join(page.get_text() for page in doc)`
|
||||
- **DOCX**: `docx.Document(p)` → join paragraph text
|
||||
- **Other** (`.txt`, `.html`): plain read or basic strip
|
||||
- Status:
|
||||
- `ok` — text > 50 chars
|
||||
- `ocr_needed` — text ≤ 50 chars (likely scanned image PDF)
|
||||
- `failed` — exception caught and logged
|
||||
- `unsupported` — file type we don't handle
|
||||
3. Pull inline body from `bib.sqlite` (`items.abstract` for that `cid`).
|
||||
4. Write `combined.md`:
|
||||
|
||||
```markdown
|
||||
---
|
||||
comment_id: CMS-2017-0092-0009
|
||||
docket_id: CMS-2017-0092
|
||||
extracted_at: 2026-04-23T13:45:00Z
|
||||
chars: 12453
|
||||
attachments:
|
||||
- file: attachment_1.pdf
|
||||
status: ok
|
||||
chars: 12300
|
||||
- file: attachment_2.pdf
|
||||
status: ocr_needed
|
||||
chars: 0
|
||||
---
|
||||
|
||||
## Inline comment
|
||||
|
||||
[body from bib.sqlite items.abstract]
|
||||
|
||||
## attachment_1.pdf
|
||||
|
||||
[extracted text]
|
||||
|
||||
## attachment_2.pdf
|
||||
|
||||
_(ocr_needed — 0 chars extracted)_
|
||||
```
|
||||
|
||||
## CLI surface
|
||||
|
||||
```
|
||||
stack comments extract [--docket CMS-2023-0121] [--limit N] [--workers 8]
|
||||
stack comments extract-ocr [--docket ...] [--limit N] # phase 2
|
||||
stack comments stats # counts by status
|
||||
```
|
||||
|
||||
`extract` walks `.state/comments/`, parallelizes per-comment extraction across `--workers` threads. PyMuPDF releases the GIL during decoding, so threads are effective. Default `--workers 8`. Resumable: existing `combined.md` skipped.
|
||||
|
||||
## DuckDB view
|
||||
|
||||
`rex.comments.view.register(con)` registers a DuckDB view backed by `read_csv` over a generated `.state/comments/_index.csv` (one row per `combined.md`) with columns:
|
||||
|
||||
- `comment_id`, `docket_id`, `extracted_at`, `chars`
|
||||
- `n_attachments`, `n_ok`, `n_ocr_needed`, `n_failed`
|
||||
|
||||
Index is regenerated by `stack comments stats` (which prints the aggregate counts as a side-effect). Downstream consumers should call `stack comments stats` (or its underlying `view.rebuild_index()`) before relying on the view; we don't auto-rebuild on every extract to keep extraction fast.
|
||||
|
||||
Bodies stay on disk. The view is for filtering/aggregation; downstream classifiers (#254/#255) read body text from disk via the path convention `.state/comments/{docket_id}/{comment_id}/combined.md`. A helper `view.body_path(comment_id, docket_id) -> Path` lives in `view.py` for this lookup.
|
||||
|
||||
## Error handling
|
||||
|
||||
- Per-attachment exceptions → log warning, mark `status: failed`, continue. Never abort the whole run.
|
||||
- Unexpected file types → mark `status: unsupported` and continue.
|
||||
- Missing inline body in bib.sqlite → write combined.md with empty inline section (don't fail).
|
||||
- Out-of-disk → bubble up; loop is restartable.
|
||||
|
||||
## Testing
|
||||
|
||||
- **Unit:** synthetic PDFs (PyMuPDF can create them) + DOCX fixtures via python-docx.
|
||||
- **Integration:** 3-5 real `.state/comments/` dirs committed under `tests/rex/comments/fixtures/`.
|
||||
- **Property:** `extract_comment(dir)` is idempotent — running on a dir with existing `combined.md` is a no-op.
|
||||
- **Negative:** corrupted PDF, password-protected PDF, zero-byte file, unknown extension.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- OCR execution (phase 2, separate command, separate dependency `pytesseract` + system `tesseract`)
|
||||
- Position classification (#254)
|
||||
- Coordination detection (#255)
|
||||
- Dashboard (#256)
|
||||
- Re-extracting already-completed `combined.md` files (force re-run = `rm combined.md` then re-run)
|
||||
- Refactoring `prisma/export.py` PDF extraction
|
||||
|
||||
## Dependencies added
|
||||
|
||||
- `pymupdf>=1.24` (AGPL-3.0)
|
||||
- `python-docx>=1.1` (MIT)
|
||||
|
||||
`pytesseract` is added in phase 2 (OCR), not now.
|
||||
Reference in New Issue
Block a user