llm: tag vocabulary curation from corpus tags #574

New Issue

kert · 2026-07-16T19:26:09Z

kert commented

2026-07-16 19:26:09 +00:00

Closed vocabulary derived from the existing Zotero/bib corpus tags (7,795 tags — most are mechanical prefixes like docket:/year:/rule: to exclude).

Scope

extract candidate topical tags from corpus; curate to a working set (target ~50–150)
versioned vocab artifact (config or table) the tagging chain loads
mapping rules: which existing tag families are excluded (mechanical prefixes)

Acceptance

vocab reviewed by hand; loading is versioned so tag runs record which vocab produced them

Spec: docs/superpowers/specs/2026-07-16-llm-module-design.md

Closed vocabulary derived from the existing Zotero/bib corpus tags (7,795 tags — most are mechanical prefixes like docket:/year:/rule: to exclude). **Scope** - [ ] extract candidate topical tags from corpus; curate to a working set (target ~50–150) - [ ] versioned vocab artifact (config or table) the tagging chain loads - [ ] mapping rules: which existing tag families are excluded (mechanical prefixes) **Acceptance** - vocab reviewed by hand; loading is versioned so tag runs record which vocab produced them Spec: [`docs/superpowers/specs/2026-07-16-llm-module-design.md`](https://git.fhirworx.io/homelab/stack/src/branch/main/docs/superpowers/specs/2026-07-16-llm-module-design.md)

kert added this to the P35: LLM Tagging — closed-vocab RAG tagging into bib/Zotero milestone 2026-07-16 19:26:09 +00:00

kert added the bib llm labels 2026-07-16 19:26:09 +00:00

kert commented

2026-08-13 23:46:54 +00:00

Status audit (2026-08-13, repo-verified): NOT DONE. No vocab extraction/curation artifact; src/bib/tag.py is the mechanical ns:value vocabulary, not a curated topical set; dev/scripts/llm_bakeoff.py explicitly defers this to P35.

**Status audit (2026-08-13, repo-verified):** NOT DONE. No vocab extraction/curation artifact; [`src/bib/tag.py`](https://git.fhirworx.io/homelab/stack/src/branch/main/src/bib/tag.py) is the mechanical ns:value vocabulary, not a curated topical set; [`dev/scripts/llm_bakeoff.py`](https://git.fhirworx.io/homelab/stack/src/branch/main/dev/scripts/llm_bakeoff.py) explicitly defers this to P35.

kert referenced this issue from a commit

2026-09-22 20:55:01 +00:00

feat(llm): closed theme vocabulary for comment tagging — themes.yaml v1, llm.vocab loader, stack llm vocab, P35 design spec (refs #574)

kert referenced this issue from a commit

2026-09-22 20:55:01 +00:00

merge: #574/#575 — P35 theme vocabulary + tagging chain, runner and write-back (refs #574, #575, #576)

kert referenced this issue from a commit

2026-09-22 21:08:07 +00:00

merge: #577 — golden-set gate, eval-tags, vocab v2 and tuned tagger defaults (refs #574, #575, #576, #577)

kert commented

2026-09-22 21:09:22 +00:00

Done — merged to main as 5e37641 (commit 7693e97). Design: docs/superpowers/specs/2026-09-22-llm-tagging-design.md.

src/llm/vocab/themes.yaml — the closed vocabulary, v2, 57 themes (v1 had 53; the CY2027 letters forced modifier-25-same-day, lactation-services, health-coaching, radiation-therapy). Each theme: kebab-case slug, label, a one-sentence definition the judge sees, letter-language synonyms, and the FR section stems it was seeded from. Mechanical namespaces (docket:, year:, rule:, reg-docket:, enriched:, code:, family:, mesh:) are excluded by construction — they are provenance, not themes; the corpus's mesh: tags are the skin-substitute literature, not rulemaking themes.
Seeding: dev/scripts/llm_vocab_candidates.py mines heading-like paragraphs from fr_anchors for rules with payment year ≥ 2018 and ranks the stems by how many rule years carry them. Hand-curated from there (by me — please skim the YAML; every slug is stable once published, retire via retired: and bump version).
llm.vocab.load() validates (unique slugs, non-empty definitions, version ≥ 1, retired ≠ active); Theme.card is the text embedded per theme; split_tags() keeps the write-back away from hand tags. stack llm vocab [--path] [--slug].
Versioned: every tagging run records the vocab version + model + content hash on the item, so a vocabulary change re-tags rather than silently re-labelling. 13 unit tests.

**Done — merged to main as 5e37641** (commit 7693e97). Design: `docs/superpowers/specs/2026-09-22-llm-tagging-design.md`. - `src/llm/vocab/themes.yaml` — the closed vocabulary, **v2, 57 themes** (v1 had 53; the CY2027 letters forced `modifier-25-same-day`, `lactation-services`, `health-coaching`, `radiation-therapy`). Each theme: kebab-case slug, label, a one-sentence definition the judge sees, letter-language synonyms, and the FR section stems it was seeded from. Mechanical namespaces (`docket:`, `year:`, `rule:`, `reg-docket:`, `enriched:`, `code:`, `family:`, `mesh:`) are excluded by construction — they are provenance, not themes; the corpus's `mesh:` tags are the skin-substitute literature, not rulemaking themes. - Seeding: `dev/scripts/llm_vocab_candidates.py` mines heading-like paragraphs from `fr_anchors` for rules with payment year ≥ 2018 and ranks the stems by how many rule years carry them. Hand-curated from there (by me — please skim the YAML; every slug is stable once published, retire via `retired:` and bump `version`). - `llm.vocab.load()` validates (unique slugs, non-empty definitions, version ≥ 1, retired ≠ active); `Theme.card` is the text embedded per theme; `split_tags()` keeps the write-back away from hand tags. `stack llm vocab [--path] [--slug]`. - Versioned: every tagging run records the vocab version + model + content hash on the item, so a vocabulary change re-tags rather than silently re-labelling. 13 unit tests.

kert closed this issue

2026-09-22 21:09:22 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#574