homelab/stack
1
0
Fork 0
Code Issues 70 Pull Requests 1 Actions Packages Projects 1 Releases Wiki Activity

comments: PDF/DOCX text extraction pipeline #253

New Issue
Closed
opened 2026-03-25 19:30:23 +00:00 by kert · 3 comments
kert commented 2026-03-25 19:30:23 +00:00
Owner
Copy Link

Objective

Extract full text from comment attachments.

Tasks

  • pypdf extraction for PDF attachments
  • python-docx extraction for DOCX attachments
  • OCR fallback for scanned-image PDFs (tesseract or similar)
  • Markdown output per attachment
  • Merge inline comment text with extracted attachment text
  • Handle edge cases: password-protected PDFs, corrupted files, multi-file attachments
  • Store extracted text: data/cms/comments/{docket_id}/extracted/

Key insight from hti5

70%+ of substantive comments are in attachments, not inline text. Initial analysis using only inline text misses the real content.

## Objective Extract full text from comment attachments. ## Tasks - [ ] pypdf extraction for PDF attachments - [ ] python-docx extraction for DOCX attachments - [ ] OCR fallback for scanned-image PDFs (tesseract or similar) - [ ] Markdown output per attachment - [ ] Merge inline comment text with extracted attachment text - [ ] Handle edge cases: password-protected PDFs, corrupted files, multi-file attachments - [ ] Store extracted text: `data/cms/comments/{docket_id}/extracted/` ## Key insight from hti5 70%+ of substantive comments are in attachments, not inline text. Initial analysis using only inline text misses the real content.
kert added this to the P30: CMS Rulemaking Comment Analysis — regulations.gov sentiment + coordination detection milestone 2026-03-25 19:31:53 +00:00
kert commented 2026-04-23 15:28:19 +00:00
Author
Owner
Copy Link

FYI: download-side throughput improvement landed in 22cbc74 — attachments are now fetched concurrently from the CDN (4 workers, no per-request sleep). They stream into .state/comments/{docket_id}/{comment_id}/ and are ready for extraction. Backfill is at ~62% across the 56K-comment corpus; by the time the extraction pipeline kicks off, the attachment store should be substantially complete.

FYI: download-side throughput improvement landed in 22cbc74 — attachments are now fetched concurrently from the CDN (4 workers, no per-request sleep). They stream into `.state/comments/{docket_id}/{comment_id}/` and are ready for extraction. Backfill is at ~62% across the 56K-comment corpus; by the time the extraction pipeline kicks off, the attachment store should be substantially complete.
kert commented 2026-04-23 20:13:36 +00:00
Author
Owner
Copy Link

Phase 1 complete — landed in 6 commits on main.

What shipped

  • src/rex/comments/ — extract.py (PyMuPDF + python-docx + cheap HTML strip + image→ocr_needed routing), combine.py (atomic writer for combined.md with YAML frontmatter), walker.py (parallel ThreadPoolExecutor), view.py (DuckDB view over _index.csv)
  • stack comments {extract, extract-ocr, stats} CLI
  • 29 tests, including a real-comment integration smoke

Live smoke run on CMS-2024-0256 (smallest docket)

count %
comments 124 —
attachments 162 —
ok 149 92%
ocr_needed 8 5%
failed 0 0%

Full docket processed in 3.3 seconds at 8 workers. Idempotent re-run is a no-op.

Notes

  • Combined.md output spot-checked: clean text extraction, frontmatter parses cleanly. Example: a 39K-char comment with two PDF attachments rendered to a single readable markdown file.
  • Image attachments (.jpg/.png/.tif/.tiff/.gif/.bmp) auto-route to ocr_needed so phase-2 can pick them up.
  • combined.md writes are atomic (.tmp + os.replace) — safe under the planned distributed-GPU phase-2 fan-out.
  • extract-ocr is a stub that exits 2 with a clear message — phase-2 work tracked in spec.

Phase-2 (deferred)

Will use marker (GPU-accelerated PDF→markdown) instead of tesseract. ~1.8K OCR-needed comments across the 56K corpus → ~3-5 hr on a single 4090, vs multi-day on CPU.

Ready for #254/#255

Public API: body_path(), parse_combined(), comments_index DuckDB view. Downstream classifiers can consume directly.

Commits: d82bc48 skeleton+PDF, b2ad400 DOCX+combine, 1f60f8c walker+view, da831c0 CLI+integration, 7281012 review fixes (cross-thread sqlite + atomic writes + image classification + index freshness).

**Phase 1 complete** — landed in 6 commits on main. ## What shipped - `src/rex/comments/` — `extract.py` (PyMuPDF + python-docx + cheap HTML strip + image→ocr_needed routing), `combine.py` (atomic writer for `combined.md` with YAML frontmatter), `walker.py` (parallel `ThreadPoolExecutor`), `view.py` (DuckDB view over `_index.csv`) - `stack comments {extract, extract-ocr, stats}` CLI - 29 tests, including a real-comment integration smoke ## Live smoke run on CMS-2024-0256 (smallest docket) | | count | % | |---|---|---| | comments | 124 | — | | attachments | 162 | — | | ok | 149 | 92% | | ocr_needed | 8 | 5% | | failed | 0 | 0% | Full docket processed in **3.3 seconds** at 8 workers. Idempotent re-run is a no-op. ## Notes - Combined.md output spot-checked: clean text extraction, frontmatter parses cleanly. Example: a 39K-char comment with two PDF attachments rendered to a single readable markdown file. - Image attachments (`.jpg/.png/.tif/.tiff/.gif/.bmp`) auto-route to `ocr_needed` so phase-2 can pick them up. - `combined.md` writes are atomic (`.tmp` + `os.replace`) — safe under the planned distributed-GPU phase-2 fan-out. - `extract-ocr` is a stub that exits 2 with a clear message — phase-2 work tracked in spec. ## Phase-2 (deferred) Will use **[marker](https://github.com/datalab-to/marker)** (GPU-accelerated PDF→markdown) instead of tesseract. ~1.8K OCR-needed comments across the 56K corpus → ~3-5 hr on a single 4090, vs multi-day on CPU. ## Ready for #254/#255 Public API: `body_path()`, `parse_combined()`, `comments_index` DuckDB view. Downstream classifiers can consume directly. **Commits:** `d82bc48` skeleton+PDF, `b2ad400` DOCX+combine, `1f60f8c` walker+view, `da831c0` CLI+integration, `7281012` review fixes (cross-thread sqlite + atomic writes + image classification + index freshness).
kert commented 2026-07-10 20:49:05 +00:00
Author
Owner
Copy Link

Shipped. The extraction pipeline lives in src/rex/comments/ with the stack comments CLI (extract / extract-ocr / stats — https://git.fhirworx.io/homelab/stack/commit/da831c0 refs this issue):

Ran at scale: every regulations.gov comment item in Zotero carries the combined extracted text (see #416, which builds on this).

Shipped. The extraction pipeline lives in `src/rex/comments/` with the `stack comments` CLI (extract / extract-ocr / stats — https://git.fhirworx.io/homelab/stack/commit/da831c0 refs this issue): - [x] PDF extraction (pypdf) + [x] DOCX (python-docx) + txt/html fallbacks (`extract.py`) - [x] OCR fallback for scanned PDFs (`stack comments extract-ocr`) - [x] Markdown output + [x] inline/attachment merge: one `combined.md` per comment with YAML frontmatter recording each attachment's extraction status; resumable (`combine.py`) - [x] Edge cases: per-file failures are logged and recorded in frontmatter rather than aborting the run - Storage evolved from the planned `data/cms/comments/{docket}/extracted/` to bib/Zotero: per-attachment `.md` siblings (https://git.fhirworx.io/homelab/stack/commit/684a8740, https://git.fhirworx.io/homelab/stack/commit/fafe661) and `combined.md` as a Zotero note (https://git.fhirworx.io/homelab/stack/commit/2f6e2d4), which is where the corpus is actually consumed. Ran at scale: every regulations.gov comment item in Zotero carries the combined extracted text (see #416, which builds on this).
kert closed this issue 2026-07-10 20:49:05 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
api
REST API (FastAPI)
 auth
Authentication and authorization
 bib
Bibliography, Zotero, citations
 ci
CI/CD pipelines and Woodpecker plugins
 cli
CLI commands and UX
 cloud
Cloud provider abstraction (AWS, GCP, Azure, self-hosted)
 config
Configuration and wiring
 data
Data loading, CCLF, BCDA, seeds
 databricks
Databricks jobs and deployment
 docs
Documentation and generated references
 image:api
image:docs
image:mc
image:notebooks
image:zotero
infra
lakehouse
Nessie, Polaris, Trino, Iceberg
 llm
llm module — local RAG over comments + Zotero corpus
 notebooks
marimo notebook errors (auto-filed by nb-watcher/nb-integration)
 packaging
Python packaging, optional deps, build system
 pfs
pipeline
Pipeline execution and validation
 pipeline:ci
pipeline:deploy
pipeline:harden
pipeline:infra-ci
pipeline:rebuild-all
pipeline:release
qpp
quality
Testing, coverage, error handling
 step:build
step:deploy
step:hadolint
step:lint
step:publish
step:scan
step:test
step:validate
styles
Charts, figures, visual theming
 tree
Project directory structure and file organization
No Label
Milestone
No items
No Milestone P30: CMS Rulemaking Comment Analysis — regulations.gov sentiment + coordination detection
Projects
Clear projects
No project
Assignees
Clear assignees
kert renovate-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: homelab/stack#253
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?