llm: golden-set eval — precision/recall gate #577

New Issue

kert · 2026-07-16T19:26:09Z

kert commented

2026-07-16 19:26:09 +00:00

Quality gate before corpus-wide fan-out.

Scope

hand-labeled golden set (~200 comments across ≥3 dockets)
eval harness: precision/recall/abstain-rate per tag + overall; report artifact
runs mocked in CI, real on demand (stack llm eval)
gate: agreed threshold must pass before tagging beyond the pilot docket

Acceptance

eval numbers recorded in the spec; corpus-wide run blocked on the gate

Spec: docs/superpowers/specs/2026-07-16-llm-module-design.md

Quality gate before corpus-wide fan-out. **Scope** - [ ] hand-labeled golden set (~200 comments across ≥3 dockets) - [ ] eval harness: precision/recall/abstain-rate per tag + overall; report artifact - [ ] runs mocked in CI, real on demand (`stack llm eval`) - [ ] gate: agreed threshold must pass before tagging beyond the pilot docket **Acceptance** - eval numbers recorded in the spec; corpus-wide run blocked on the gate Spec: [`docs/superpowers/specs/2026-07-16-llm-module-design.md`](https://git.fhirworx.io/homelab/stack/src/branch/main/docs/superpowers/specs/2026-07-16-llm-module-design.md)

kert added this to the P35: LLM Tagging — closed-vocab RAG tagging into bib/Zotero milestone 2026-07-16 19:26:09 +00:00

kert added the quality llm labels 2026-07-16 19:26:09 +00:00

kert commented

2026-08-13 23:46:55 +00:00

Status audit (2026-08-13, repo-verified): NOT DONE. No golden-set eval harness in src/llm or tests (P35 not started).

**Status audit (2026-08-13, repo-verified):** NOT DONE. No golden-set eval harness in src/llm or tests (P35 not started).

kert referenced this issue

2026-08-18 19:24:20 +00:00

llm: extend HostPool fan-out from embeddings to chat/generation #654

kert referenced this issue

2026-09-11 22:45:13 +00:00

comments: extract author/org metadata from attachment text #416

kert referenced this issue from a commit

2026-09-22 21:08:07 +00:00

feat(llm): golden-set gate for theme tagging — llm.tageval, tests/llm/golden_themes.yaml, stack llm eval-tags; vocab v2 (refs #577)

kert referenced this issue from a commit

2026-09-22 21:08:07 +00:00

feat(llm): eval-tags --dump writes every golden comment's shortlist scores and verdicts for offline threshold analysis (refs #577)

kert referenced this issue from a commit

2026-09-22 21:08:07 +00:00

feat(llm): tagger defaults from the golden sweep — 3 tags, min confidence 0.60 (refs #577)

kert referenced this issue from a commit

2026-09-22 21:08:07 +00:00

fix(llm): stack llm tag --max-tags default is 3, matching the tagger and eval-tags (refs #577)

kert referenced this issue from a commit

2026-09-22 21:08:07 +00:00

merge: #577 — golden-set gate, eval-tags, vocab v2 and tuned tagger defaults (refs #574, #575, #576, #577)

kert referenced this issue

2026-09-22 21:09:22 +00:00

llm: RAG tagging chain + batch runner #575

kert commented

2026-09-22 21:09:23 +00:00

Done — merged to main as 48bbc5f (commits c354047, 7000d27, c5e060a, 5b… spec). Results are recorded in the spec's Results section.

Golden set: tests/llm/golden_themes.yaml, 39 CMS-2026-2377 comments hand-labelled from their opening text — expected (must carry) and acceptable (allowed, not required). Smaller than the ~200 across ≥3 dockets the issue asked for; it is one docket and one reader, stated as such.
llm.tageval: per-slug tp / fp (outside expected ∪ acceptable) / fn, micro precision/recall/F1, abstain rate; gate() = micro-F1 ≥ 0.70 and no theme with support ≥ 5 under 0.5 precision. stack llm eval-tags [--golden] [--live] [--limit] [--dump] — default scores what the last run stored on the items (no model calls, what CI can run), --live re-tags through the chain, --dump writes every shortlist score + verdict for offline sweeps. Exit 1 on gate failure. 18 unit + 2 CLI tests.
Live numbers (qwen2.5:14b, laptop 5070 Ti): 4 tags / no threshold → P 0.60 R 0.76 F1 0.67 (fail); 3 tags / ≥ 0.60 → P 0.70 R 0.76 F1 0.73 (pass) — now the defaults; 2 tags / ≥ 0.60 → F1 0.79. Recall is the judge's ceiling (11 expected themes never answered yes). Caveat: thresholds were tuned on the same 39 comments they are scored on; the next check is a held-out sample from the pilot docket once fan-out reaches it.
Corpus-wide fan-out stays gated: only the CMS-2026-2377 pilot is running.

**Done — merged to main as 48bbc5f** (commits c354047, 7000d27, c5e060a, 5b… spec). Results are recorded in the spec's *Results* section. - Golden set: `tests/llm/golden_themes.yaml`, **39 CMS-2026-2377 comments** hand-labelled from their opening text — `expected` (must carry) and `acceptable` (allowed, not required). Smaller than the ~200 across ≥3 dockets the issue asked for; it is one docket and one reader, stated as such. - `llm.tageval`: per-slug tp / fp (outside expected ∪ acceptable) / fn, micro precision/recall/F1, abstain rate; `gate()` = micro-F1 ≥ 0.70 and no theme with support ≥ 5 under 0.5 precision. `stack llm eval-tags [--golden] [--live] [--limit] [--dump]` — default scores what the last run stored on the items (no model calls, what CI can run), `--live` re-tags through the chain, `--dump` writes every shortlist score + verdict for offline sweeps. Exit 1 on gate failure. 18 unit + 2 CLI tests. - Live numbers (qwen2.5:14b, laptop 5070 Ti): 4 tags / no threshold → P 0.60 R 0.76 **F1 0.67 (fail)**; **3 tags / ≥ 0.60 → P 0.70 R 0.76 F1 0.73 (pass)** — now the defaults; 2 tags / ≥ 0.60 → F1 0.79. Recall is the judge's ceiling (11 expected themes never answered yes). Caveat: thresholds were tuned on the same 39 comments they are scored on; the next check is a held-out sample from the pilot docket once fan-out reaches it. - Corpus-wide fan-out stays gated: only the CMS-2026-2377 pilot is running.

kert closed this issue

2026-09-22 21:09:23 +00:00

kert referenced this issue from a commit

2026-09-22 21:09:36 +00:00

docs(llm): record the P35 golden-gate results and the Zotero scope caveat in the tagging spec (refs #577, #576)

kert referenced this issue from a commit

2026-09-22 21:09:36 +00:00

merge: P35 spec results section (refs #577, #576)

Sign in to join this conversation.