llm: evaluation set — golden longitudinal questions with expected anchors (CCM, audio-only E/M, APCM, G2211, telehealth steps) #692

New Issue

kert · 2026-09-09T14:30:05Z

kert commented

2026-09-09 14:30:05 +00:00

Spec docs/superpowers/specs/2026-09-09-code-family-longitudinal-design.md (§Evaluation).

What. tests/llm/golden_lineage.yaml: question → expected (item_key, p_id) anchors that must appear in sources, expected event labels that must be cited in prose, and forbidden claims (e.g. no payment figure without a [PFS CY… Addendum B] label).

Seed questions and anchors:

History of CCM coding and payment → DE2VH9PD ¶1249–1251; YBM4IZUS ¶1578; JJ6AM5HJ ¶1163.
What replaced G2058 and why → YBM4IZUS ¶1578, ¶2369.
How do APCM's elements differ from CCM's → JJ6AM5HJ ¶1164–1185 vs DE2VH9PD ¶1245–1247.
When were audio-only E/M codes payable and why did 99441–99443 end → XFGGRBDH ¶489; pfs.rvu 99441 last year 2024.
What did commenters say about G2211 in 2023 vs 2025 → CMS-2023-0121 (3,353 abstract hits) vs CMS-2025-0304 (146).
Does 99490 pass telehealth Steps 1–3 → 2KVJ2HKX ¶394/¶396/¶398 and 99490's elements.
How did G2064/G2065 → 99424/99426 change the RHC/FQHC G0511 rate → JE7KYBW3 ¶1100/¶1111; MZ24MX5S ¶1305.

Done when the set runs in CI against the live chat (nightly, like notebooks-integration) and files an issue on regression through the existing filer.

Spec docs/superpowers/specs/2026-09-09-code-family-longitudinal-design.md (§Evaluation). **What.** `tests/llm/golden_lineage.yaml`: question → expected `(item_key, p_id)` anchors that must appear in `sources`, expected event labels that must be cited in prose, and forbidden claims (e.g. no payment figure without a `[PFS CY… Addendum B]` label). Seed questions and anchors: 1. History of CCM coding and payment → `DE2VH9PD` ¶1249–1251; `YBM4IZUS` ¶1578; `JJ6AM5HJ` ¶1163. 2. What replaced G2058 and why → `YBM4IZUS` ¶1578, ¶2369. 3. How do APCM's elements differ from CCM's → `JJ6AM5HJ` ¶1164–1185 vs `DE2VH9PD` ¶1245–1247. 4. When were audio-only E/M codes payable and why did 99441–99443 end → `XFGGRBDH` ¶489; `pfs.rvu` 99441 last year 2024. 5. What did commenters say about G2211 in 2023 vs 2025 → CMS-2023-0121 (3,353 abstract hits) vs CMS-2025-0304 (146). 6. Does 99490 pass telehealth Steps 1–3 → `2KVJ2HKX` ¶394/¶396/¶398 and 99490's elements. 7. How did G2064/G2065 → 99424/99426 change the RHC/FQHC G0511 rate → `JE7KYBW3` ¶1100/¶1111; `MZ24MX5S` ¶1305. **Done when** the set runs in CI against the live chat (nightly, like notebooks-integration) and files an issue on regression through the existing filer.

kert added this to the P49: Code families as first-class objects — element model, lineage anchors, longitudinal chat milestone 2026-09-09 14:30:05 +00:00

kert added the llm quality labels 2026-09-09 14:30:05 +00:00

kert referenced this issue from a commit

2026-09-10 18:14:18 +00:00

docs(plans): P49 slice 3 — longitudinal chat, era-balanced retrieval, CPT manual source, golden evaluation (refs #691 #692 #699 #705)

kert referenced this issue from a commit

2026-09-10 18:14:19 +00:00

feat(llm): golden longitudinal evaluation — yaml set, runner with anchor/label/forbidden checks, nightly workflow + issue filer (refs #692)

kert referenced this issue from a commit

2026-09-10 18:14:19 +00:00

feat(llm): lineage-cited FR paragraphs as sources; per-docket comment window in timeline mode (refs #691 #692)

kert referenced this issue from a commit

2026-09-10 18:14:19 +00:00

fix(llm): lineage sources prefer the question's own codes; element-diff anchors; cap 8 (refs #691 #692)

kert referenced this issue from a commit

2026-09-10 18:14:19 +00:00

fix(ci,llm): golden runner exits 0 when filing issues; workflow keeps the report on failure; cmd_run tests (refs #692)

kert referenced this issue from a commit

2026-09-10 18:14:19 +00:00

fix(llm): thread-local bib Store; deterministic collapse; unique labels; era coverage; system prompt; prompt budget (refs #691 #692)

kert referenced this issue from a commit

2026-09-10 18:14:19 +00:00

fix(pfs): refresh_from never clears FAMILIES mid-swap; fix its stale docstring (refs #691 #692)

kert referenced this issue from a commit

2026-09-10 18:14:19 +00:00

fix(cli): _codes_for's slow-SQL warning is elements-only, not lineage --all-payable (refs #691 #692)

kert referenced this issue from a commit

2026-09-10 18:14:19 +00:00

fix(ci,llm): golden workflow survives a report-less crash; pyyaml ships in the llm extra (refs #691 #692)

kert referenced this issue from a commit

2026-09-10 18:14:19 +00:00

fix(llm): per-code priority cap only under budget pressure (refs #691 #692)

kert referenced this issue from a commit

2026-09-10 18:14:19 +00:00

test(llm): golden set exercises expect_labels and forbid — audio-only final-rule label, no unlabelled dollar amounts (refs #692)

kert referenced this issue from a commit

2026-09-10 18:14:19 +00:00

merge: P49 slice 3 — longitudinal chat: derived families, lineage for every payable code, timeline SSE event + cited prompt block, era-balanced retrieval, CPT manual source, chat UI timeline, golden evaluation (refs #691 #692 #699 #705)

kert commented

2026-09-10 18:15:26 +00:00

Merged to main as da345bc. tests/llm/golden_lineage.yaml (7 questions: anchors, labels, forbidden claims, events, dockets, era counts) + dev/scripts/llm_golden.py (streams /chat, six pure checkers, exit 0 when filing issues) + nightly llm-golden workflow generated from dev/scripts/backends/gitea.py (03:40, runs inside the llm container, files/sweeps through nb_issue_filer with label llm). In-process against the branch code: 4/7 pass (ccm-history, audio-only-em, g2211 2023-vs-2025, g2064→99424 rate). Three documented gaps stay red on purpose: g2058-replacement misses YBM4IZUS ¶1578 (attached to 99439's replaces event, outside the 8-paragraph cap), apcm-vs-ccm-elements misses the element paragraphs, 99490-telehealth-steps needs the Step 1–3 paragraphs retrieved. NOTE: the deployed llm image predates this branch (no lineage event, sources without item_key/p_id), so the nightly will be red until the image is rebuilt and rolled (COMMIT_SHA in .env + compose build/up).

Merged to main as da345bc. tests/llm/golden_lineage.yaml (7 questions: anchors, labels, forbidden claims, events, dockets, era counts) + dev/scripts/llm_golden.py (streams /chat, six pure checkers, exit 0 when filing issues) + nightly llm-golden workflow generated from dev/scripts/backends/gitea.py (03:40, runs inside the llm container, files/sweeps through nb_issue_filer with label llm). In-process against the branch code: 4/7 pass (ccm-history, audio-only-em, g2211 2023-vs-2025, g2064→99424 rate). Three documented gaps stay red on purpose: g2058-replacement misses YBM4IZUS ¶1578 (attached to 99439's replaces event, outside the 8-paragraph cap), apcm-vs-ccm-elements misses the element paragraphs, 99490-telehealth-steps needs the Step 1–3 paragraphs retrieved. NOTE: the deployed llm image predates this branch (no lineage event, sources without item_key/p_id), so the nightly will be red until the image is rebuilt and rolled (COMMIT_SHA in .env + compose build/up).

kert closed this issue

2026-09-10 18:15:26 +00:00

kert referenced this issue

2026-09-10 18:15:27 +00:00

P49 anchors follow-ups: CPT manual as a chat source, IOM page locating, MLN resolver, shared code-or-family SQL predicate #705

kert referenced this issue from a commit

2026-09-10 19:27:14 +00:00

fix(bib,llm): rule kind from the rule's own text — proposed rules were labelled 'final' because CMS titles since CY2018 carry no kind word (refs #691 #692)

kert referenced this issue from a commit

2026-09-10 19:49:01 +00:00

fix(llm): the chat cites what it is told to — lineage lines as sentences ending with their label, junk page numbers filtered from event code lists, an end-of-prompt citation rule, temperature 0.2; valuation labels link in the page (refs #691 #692)

Sign in to join this conversation.