comments: extract author/org metadata from attachment text #416

New Issue

kert · 2026-04-27T22:19:59Z

kert commented

2026-04-27 22:19:59 +00:00

Background

After the regulations.gov backfill + extraction pipeline (commits 8a09925 → dfa2590 → sync to Zotero), every comment item in the Zotero library has a combined.md attachment containing the inline body and full extracted text of every uploaded attachment.

Item titles default to Comment on <rule_id> (e.g. Comment on CMS-2025-0304-0009) because the regulations.gov API firstName/lastName/organization fields are unpopulated for the bulk of submissions. upsert_comment already prepends a byline when those fields are present (src/bib/regulations_gov.py), so the API path works — the data just isn't there.

Problem

For most comments the real author and organization live inside the attachment itself: letterhead, signature line, "Submitted by:" preamble, the From: field on PDFs, etc. Pulling these out turns ~23k anonymous-looking Comment on ... records into properly attributed items, which makes the corpus actually browsable in Zotero and citable downstream.

Scope

Source: the per-attachment sections in each combined.md (~23,610 items under data/zotero/data/storage/<key>/combined.md).
Targets: firstName, lastName, organization on each items row in data/bib.sqlite.
Sync: existing stack bib sync-zotero already carries those fields through upsert_comment's byline logic, so once bib is updated the Zotero titles refresh on next sync.

Approach options

Heuristic / regex. Cheap, free, ~70% accuracy. Catches the obvious patterns (Sincerely,\n<name>, On behalf of <org>, common letterhead structures). Misses scanned-image letterheads, image-only signatures, free-form prose.
LLM extraction (Haiku 4.5). Process each combined.md once, ask for {first_name, last_name, organization, role}. ~95% accuracy. Ballpark cost: 23k items × ~3k input tokens × $0.80/Mtok + ~50 output tokens × $4/Mtok ≈ $60. One-shot batch (messages.batches) → 50% off ≈ $30.
Hybrid: regex pass first (cheap fast wins), LLM only on the ones regex couldn't extract. Probably the right answer.

Acceptance

New CLI: stack comments extract-metadata walks every Zotero combined.md, extracts {first_name, last_name, organization}, writes them to bib's items table.
Idempotent: skip items where first_name or organization is already set (or use a tag like meta:extracted to mark processed items).
stack bib sync-zotero afterwards repopulates the byline-prefixed titles.
Add the same metadata to combined.md frontmatter on next extract pass so the data lives next to the corpus.

Notes / open questions

Some commenters submit on behalf of multiple orgs (e.g. American Hospital Association and 47 state hospital associations). Capture the lead org and stash the full list in extra metadata?
Roles (MD, CEO, Senator, patient) are valuable but Zotero has no native field — could go into extra_json or as tags.
Some attachments are pure data tables (HCPCS code lists submitted as DOCX); they have no author info and the LLM should be allowed to return null.

## Background After the regulations.gov backfill + extraction pipeline (commits `8a09925` → `dfa2590` → sync to Zotero), every comment item in the Zotero library has a `combined.md` attachment containing the inline body and full extracted text of every uploaded attachment. Item titles default to `Comment on <rule_id>` (e.g. `Comment on CMS-2025-0304-0009`) because the regulations.gov API `firstName`/`lastName`/`organization` fields are unpopulated for the bulk of submissions. `upsert_comment` already prepends a byline when those fields are present ([`src/bib/regulations_gov.py`](https://git.fhirworx.io/homelab/stack/src/branch/main/src/bib/regulations_gov.py)), so the API path works — the data just isn't there. ## Problem For most comments the *real* author and organization live inside the attachment itself: letterhead, signature line, "Submitted by:" preamble, the `From:` field on PDFs, etc. Pulling these out turns ~23k anonymous-looking `Comment on ...` records into properly attributed items, which makes the corpus actually browsable in Zotero and citable downstream. ## Scope - **Source:** the per-attachment sections in each `combined.md` (~23,610 items under `data/zotero/data/storage/<key>/combined.md`). - **Targets:** `firstName`, `lastName`, `organization` on each `items` row in `data/bib.sqlite`. - **Sync:** existing `stack bib sync-zotero` already carries those fields through `upsert_comment`'s byline logic, so once bib is updated the Zotero titles refresh on next sync. ## Approach options 1. **Heuristic / regex.** Cheap, free, ~70% accuracy. Catches the obvious patterns (`Sincerely,\n<name>`, `On behalf of <org>`, common letterhead structures). Misses scanned-image letterheads, image-only signatures, free-form prose. 2. **LLM extraction** (Haiku 4.5). Process each `combined.md` once, ask for `{first_name, last_name, organization, role}`. ~95% accuracy. Ballpark cost: 23k items × ~3k input tokens × $0.80/Mtok + ~50 output tokens × $4/Mtok ≈ $60. One-shot batch (`messages.batches`) → 50% off ≈ $30. 3. **Hybrid:** regex pass first (cheap fast wins), LLM only on the ones regex couldn't extract. Probably the right answer. ## Acceptance - New CLI: `stack comments extract-metadata` walks every Zotero combined.md, extracts `{first_name, last_name, organization}`, writes them to bib's `items` table. - Idempotent: skip items where `first_name` or `organization` is already set (or use a tag like `meta:extracted` to mark processed items). - `stack bib sync-zotero` afterwards repopulates the byline-prefixed titles. - Add the same metadata to `combined.md` frontmatter on next extract pass so the data lives next to the corpus. ## Notes / open questions - Some commenters submit on behalf of multiple orgs (e.g. *American Hospital Association and 47 state hospital associations*). Capture the lead org and stash the full list in extra metadata? - Roles (MD, CEO, Senator, patient) are valuable but Zotero has no native field — could go into `extra_json` or as tags. - Some attachments are pure data tables (HCPCS code lists submitted as DOCX); they have no author info and the LLM should be allowed to return null.

kert referenced this issue from a commit

2026-05-18 00:23:57 +00:00

feat(comments): write per-attachment .md siblings + attach to bib

kert referenced this issue

2026-07-10 20:49:05 +00:00

comments: PDF/DOCX text extraction pipeline #253

kert commented

2026-07-10 20:49:07 +00:00

Status update: all prerequisites are now in place — every comment item in Zotero carries its combined.md note (#253 closed), the corpus is complete (#251 closed), and skin_subs.rulemaking_comments has commenter_name/organization columns waiting for backfilled values. The extraction heuristics (letterhead / signature / 'Submitted by:' parsing over ~23k attachments) are still to be built.

Status update: all prerequisites are now in place — every comment item in Zotero carries its `combined.md` note (#253 closed), the corpus is complete (#251 closed), and `skin_subs.rulemaking_comments` has `commenter_name`/`organization` columns waiting for backfilled values. The extraction heuristics (letterhead / signature / 'Submitted by:' parsing over ~23k attachments) are still to be built.

kert referenced this issue from a commit

2026-07-11 02:06:25 +00:00

feat(comments): LLM position/theme classifier for rulemaking comments (refs #254)

kert referenced this issue

2026-07-11 02:06:54 +00:00

comments: position classification + thematic tagging #254

kert commented

2026-08-13 23:46:55 +00:00

Status audit (2026-08-13, repo-verified): NOT DONE. Only author/org signal is the LLM classifier's side fields (rex/comments/classify.py:126-137) scoped to one docket's analytics table; nothing writes first/last/organization back to bib.sqlite, no frontmatter enrichment, no extract-metadata CLI.

**Status audit (2026-08-13, repo-verified):** NOT DONE. Only author/org signal is the LLM classifier's side fields ([`rex/comments/classify.py:126`](https://git.fhirworx.io/homelab/stack/src/branch/main/src/rex/comments/classify.py#L126)-137) scoped to one docket's analytics table; nothing writes first/last/organization back to bib.sqlite, no frontmatter enrichment, no extract-metadata CLI.

kert commented

2026-09-11 22:45:13 +00:00

Premise check (2026-09-11), before anyone builds this: the data model has moved since this was filed.

213,614 comment items in bib carry a docket: tag; 0 are titled 'Comment on …' (P47 enrichment rewrote titles), and 0 have creators rows — so the attribution gap is real, but the source text is no longer a per-item combined.md: there are none under data/zotero/data/storage (40,901 dirs, 0 *.md). Comment text now lives as abstracts in bib plus attachment files (28,659 PDF, 12,995 DOCX, a few hundred images/xlsx/rtf/txt across 42,249 attachments) and as pgvector chunks in the llm comments collection.
The cloud-LLM cost estimate no longer applies: the llm module is self-hosted only. At ~2 s per comment on one card, 213k items is ~5 GPU-days; the #655 fleet driver (prisma.fleet.run_pool) spreads that across hosts.

Re-scoped plan when this is picked up: (1) regex/heuristic pass over the abstract and the first ~2k chars of extracted attachment text (letterhead, 'Sincerely,' signature blocks, 'on behalf of', 'From:'), writing first_name/last_name/organization to items and a meta:extracted tag; (2) local-LLM pass (qwen on the fleet, structured output, null allowed) only for items the regex left empty; (3) sync-zotero already carries the byline. Golden set of ~200 hand-checked items to measure both passes before the corpus-wide run — same gate shape as #577.

Premise check (2026-09-11), before anyone builds this: the data model has moved since this was filed. - 213,614 comment items in bib carry a docket: tag; **0** are titled 'Comment on …' (P47 enrichment rewrote titles), and **0** have creators rows — so the attribution gap is real, but the source text is no longer a per-item combined.md: there are none under data/zotero/data/storage (40,901 dirs, 0 *.md). Comment text now lives as abstracts in bib plus attachment files (28,659 PDF, 12,995 DOCX, a few hundred images/xlsx/rtf/txt across 42,249 attachments) and as pgvector chunks in the llm comments collection. - The cloud-LLM cost estimate no longer applies: the llm module is self-hosted only. At ~2 s per comment on one card, 213k items is ~5 GPU-days; the #655 fleet driver (prisma.fleet.run_pool) spreads that across hosts. Re-scoped plan when this is picked up: (1) regex/heuristic pass over the abstract and the first ~2k chars of extracted attachment text (letterhead, 'Sincerely,' signature blocks, 'on behalf of', 'From:'), writing first_name/last_name/organization to items and a meta:extracted tag; (2) local-LLM pass (qwen on the fleet, structured output, null allowed) only for items the regex left empty; (3) sync-zotero already carries the byline. Golden set of ~200 hand-checked items to measure both passes before the corpus-wide run — same gate shape as #577.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#416