196 lines
28 KiB
Markdown
196 lines
28 KiB
Markdown
# Code-family longitudinal chat + golden evaluation — Implementation Plan (P49 slice 3)
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||
|
||
**Goal:** A "history of X" question in the LLM chat gets a dated, anchored timeline for the detected codes and families (a `lineage` SSE event and a prompt block the model must cite), era-balanced retrieval so CY2015 and CY2027 both surface, the CPT manual's own heading text as a source, and a golden question set that runs nightly against the live chat and files an issue on regression — for ANY PFS-payable code, not only the five hand families.
|
||
|
||
**Architecture:** Mirrors P48's valuation flow. `llm/evidence.py` gains `lineage_evidence(question, cfg)` which reads `pfs.code_event`, `pfs.code_element` and `pfs.code_guidance` from the cached read-only replica handle (falling back to on-demand `pfs.lineage.lineage(...)` for codes with no precomputed rows), returns a payload for the SSE `lineage` event and a "Lineage" prompt block whose rows carry bracketed labels (`[CY2021 PFS final ¶1578]`). `rag.retrieve(..., mode="timeline")` groups hits by rule year and caps per era. `chat.html` renders the timeline between the answer and the sources drawer. `stack pfs lineage --all-payable --write` precomputes events for every A/R/T code with one inverted pass over `fr_anchors`. `tests/llm/golden_lineage.yaml` + `dev/scripts/llm_golden.py` check anchors, cited labels and forbidden claims against the live `/chat` endpoint, nightly, through the existing issue filer.
|
||
|
||
**Tech Stack:** Python 3.13, FastAPI SSE, DuckDB replica (read-only cached handle, per-call cursor), SQLite bib (`fr_anchors`, `items`), pgvector via SQLAlchemy `text()`, Ollama self-hosted pool, marimo notebook, Gitea Actions workflows generated by `dev/scripts/gen_config.py` from `stack.toml`.
|
||
|
||
**Spec:** `docs/superpowers/specs/2026-09-09-code-family-longitudinal-design.md` (§Decisions 5–6, §Components "longitudinal chat", §Evaluation). Tracker: #691 (chat), #692 (eval), #699 items 1–2 (derived families in chat), #705 item 1 (CPT manual as a chat source, Ruling A9 of the anchors slice).
|
||
|
||
## Global Constraints
|
||
|
||
- Self-hosted inference only (Ollama pool from config); no cloud LLM API anywhere in `src/llm` or `src/pfs`.
|
||
- `pfs.*` modules never touch DuckDB at import time; the chat reads the replica through `llm/evidence.py`'s cached handle (`_connect`, mtime-keyed reopen, per-call `.cursor()`); writes only through `conf.connect.duckdb_batch` + `publish_replica`, never inside slow work (build first, then open the batch).
|
||
- **Control question is byte-identical**: for a question with no detected code, the SSE event sequence and every event payload are unchanged from main 4616d36 (`token* → sources → done`); for a coded question without history words the sequence is `valuation? → token* → sources → done` exactly as today plus the new `lineage` event first: `lineage? → valuation? → token* → sources → done`.
|
||
- Evidence budget: `lineage_evidence` adds < 300 ms wall-clock on the live replica for ≤ 3 codes with precomputed rows; on-demand fallback runs for at most 3 codes per question and is skipped (logged) beyond that.
|
||
- SSE payloads are JSON-serialisable dicts with a `type` key; the API wraps them as `data: {json}\n\n` (no `event:` field).
|
||
- Bracketed labels in prompt blocks and payloads use the exact same string; the system prompt tells the model to cite them verbatim.
|
||
- Every FR anchor link is built from `(item_key, p_id)` through `bib.frlink` (`_para_link` via `resolve`/`place`), eCFR links through `bib.cfrlink.url`.
|
||
- Tests: no live DB or Ollama required (fakes as in `tests/llm/test_rag.py` and `tests/llm/test_evidence.py`); live checks skip without `LLM_DB_PASSWORD` / `LLM_CHAT_URL`.
|
||
- Commit messages: conventional prefix, `(refs #NNN)`; **never add a Co-Authored-By or any trailer**.
|
||
- Docs: regenerate `docs/docs/cli/` fully after CLI changes (`uv run python docs/scripts/extract_cli.py`), commit every changed page.
|
||
|
||
---
|
||
|
||
### Task 1: Derived families in the chat (#699 items 1–2)
|
||
|
||
**Files:**
|
||
- Modify: `src/pfs/families.py` (`detect_codes`, `family_of`, a compiled index built by `refresh_from` and on `FAMILIES` mutation), `src/llm/evidence.py` (`_connect` → call `refresh_from` when the replica (re)opens)
|
||
- Test: `tests/pfs/test_families.py`, `tests/llm/test_evidence.py`
|
||
|
||
**Interfaces:**
|
||
- Consumes: `pfs.families.FAMILIES`, `Family(key, name, codes, synonyms)`, `refresh_from(con) -> int`, `detect_codes(text) -> Detection(codes, families, explicit)`, `code_family_index(families)` in `pfs.anchors`.
|
||
- Produces: `pfs.families.rebuild_index() -> None` (called by `refresh_from` and at import after `HAND_FAMILIES` seeding); `detect_codes` unchanged signature, same results for the five hand families (regression fixture), now also detecting derived families by member code and by *name phrase* (see rule below); `family_of(code) -> Family | None` O(1) via the code→family index (first key in sorted order when a code belongs to several families, hand families first); `llm.evidence._connect(path)` refreshes `FAMILIES` from `pfs.code_family` whenever it opens or reopens the replica (mtime change), logging the family count.
|
||
|
||
**Rules (binding):**
|
||
- Family detection by synonym stays for hand families. A derived family is detected by name only when its `name` is ≥ 12 characters and ≥ 2 words and appears as a case-insensitive whole-phrase match; single-word CPT headings (`GENE`, `ADM`, `REVERSE`, `Introduction`) never match by name. Detection by member code applies to every family.
|
||
- One compiled alternation regex over all synonym/name phrases (longest first, word-bounded) plus a `dict[code, tuple[key, ...]]` index, rebuilt in `rebuild_index()`; `detect_codes` must run in < 5 ms on a 300-character question with the live 8,518-key registry (measure in a test marked `live`, skipped without the replica).
|
||
- `refresh_from` is idempotent and thread-safe enough for the chat: it builds the new dict/index first and swaps them under `_REGISTRY_LOCK`.
|
||
|
||
- [ ] **Step 1: Failing tests** — `tests/pfs/test_families.py`: `test_detect_derived_family_by_member_code` (a fake `FAMILIES` entry `SUTURE-REMOVAL` with codes `15850 15851`; "removal of sutures 15850" → families contains `SUTURE-REMOVAL`), `test_derived_family_name_phrase_requires_two_words` (`GENE` name never matches; "Chronic Care Management Services" matches by name), `test_family_of_is_indexed` (monkeypatch `FAMILIES` with 5,000 synthetic families; 10,000 `family_of` calls < 0.2 s), `test_hand_families_regression` (the existing five-family detections unchanged — reuse existing assertions). `tests/llm/test_evidence.py`: `test_connect_refreshes_families_on_open_and_reopen` (patch `llm.evidence.refresh_from`; assert called once on first `_connect`, not on a cached call, again after `_mtime` changes).
|
||
- [ ] **Step 2: Run, confirm failures.**
|
||
- [ ] **Step 3: Implement** `rebuild_index`, index-backed `family_of`, phrase regex in `detect_codes`, the `refresh_from` swap, and the `_connect` hook (`refresh_from(con)` inside the lock after a successful open; exceptions logged, never raised).
|
||
- [ ] **Step 4: Run** `uv run pytest tests/pfs/test_families.py tests/llm -q -p no:cacheprovider`; ruff.
|
||
- [ ] **Step 5: Live check** (env wrapper): `uv run python -c "from pfs.families import detect_codes, FAMILIES, refresh_from; import duckdb; refresh_from(duckdb.connect('data/replica/aco.ro.duckdb', read_only=True)); import time; t=time.perf_counter(); d=detect_codes('history of chronic care management services and 99490'); print(len(FAMILIES), d, (time.perf_counter()-t)*1000, 'ms')"` — report the ms.
|
||
- [ ] **Step 6: Commit** `feat(pfs,llm): derived families reach the chat — indexed detect_codes/family_of, refresh on replica open (refs #699)`.
|
||
|
||
---
|
||
|
||
### Task 2: Precompute lineage for every payable code — `stack pfs lineage --all-payable --write`
|
||
|
||
**Files:**
|
||
- Modify: `src/pfs/lineage.py` (inverted FR pass), `src/cli/pfs.py` (`lineage` command gains `--all-payable`), `src/pfs/codetables.py` (`write_events_many(con, by_code)` bulk writer if not present)
|
||
- Test: `tests/pfs/test_lineage.py`, `tests/cli/test_pfs_cli.py`
|
||
|
||
**Interfaces:**
|
||
- Consumes: `lineage(con, store, code) -> list[EventRow]`, `rvu_events`, `fr_events(store, code)`, `cpt_events`, `pfs.families.find_codes`, `codes_in/code_pattern`, `pfs.valuation`/`pfs.rvu` for the A/R/T universe (the same query `stack pfs elements --all-payable` uses — reuse its helper), `write_events`/`read_all_events`.
|
||
- Produces: `fr_events_bucketed(store, codes: Sequence[str]) -> dict[str, list[EventRow]]` — ONE pass over `fr_anchors` joined to `items` (SQL prefilter: text LIKE any target code is too wide for 8.7k codes → instead stream all rows ordered by item_key, p_id and bucket with `find_codes(text)` ∩ targets), then the existing per-code event-verb logic applied to each code's bucket (factor the verb/anchor logic out of `fr_events` into `_events_for(code, rows)` so `fr_events(store, code)` == `fr_events_bucketed(store, [code])[code]`); `lineage_all(con, store, codes) -> dict[str, list[EventRow]]` merging rvu/fr/cpt per code with the same ±1-year anchoring; CLI `stack pfs lineage --all-payable [--write] [--limit N]` printing counts per source and unanchored count, writing per code (delete-then-insert per code inside one batch, opened only after the build — Ruling A13).
|
||
|
||
**Rules:** equality test `fr_events_bucketed(store, [c])[c] == fr_events(store, c)` for the seven fixture codes on the sqlite fixture; the full pass must finish in < 10 minutes on the live bib (193k paragraphs) — report the time; `--limit N` takes the first N target codes for smoke runs.
|
||
|
||
- [ ] **Step 1: Failing tests** — bucketed == per-code equality on a fixture store with three rules and four codes; `lineage_all` marks rvu events anchored when an FR/CPT event is within ±1 year; CLI `--all-payable --limit 3 --write` calls the writer once per code after the build and publishes (indirections monkeypatched, call order asserted); `--all-payable` without `--write` never opens the batch.
|
||
- [ ] **Step 2: Run, confirm failures.**
|
||
- [ ] **Step 3: Implement.**
|
||
- [ ] **Step 4: Run** `uv run pytest tests/pfs/test_lineage.py tests/cli/test_pfs_cli.py -q -p no:cacheprovider`; ruff.
|
||
- [ ] **Step 5: Live** (env wrapper): `stack pfs lineage --all-payable --limit 50` (dry) to time the pass, then `stack pfs lineage --all-payable --write` — report codes written, events per source, unanchored share, wall time; confirm `pfs.code_event` now has rows for G2211 and 99441 on the replica.
|
||
- [ ] **Step 6: Commit** `feat(pfs): lineage for every payable code — one inverted pass over fr_anchors, stack pfs lineage --all-payable (refs #691 #698)`.
|
||
|
||
---
|
||
|
||
### Task 3: `lineage_evidence` — events, element diffs, guidance; SSE payload and prompt block
|
||
|
||
**Files:**
|
||
- Create: `src/llm/lineage.py`
|
||
- Modify: `src/llm/evidence.py` (export), `src/llm/rag.py` (`stream_answer`, `build_messages`, `_SYSTEM`), `src/llm/config.py` + `stack.toml` (`lineage_max_rows = 25`, `lineage_on_demand_max = 3`)
|
||
- Test: `tests/llm/test_lineage.py`, `tests/llm/test_rag.py`
|
||
|
||
**Interfaces:**
|
||
- Consumes: `detect_codes`, `_connect(path)`/`_replica_path(cfg)` from `llm/evidence.py`, `read_events`/`read_all_events`, `read_elements`, `read_guidance`, `pfs.lineage.lineage(con, store, code)` (on-demand fallback; the bib `Store` from `conf.connect.bib()`), `pfs.descriptors.rule_year_of`, `bib.frlink.resolve(ref, store=..., item_key=...)` with `ref=f"p-{p_id}"` for the FR URL, `bib.cfrlink.url(parse_cite(locator))` for CFR guidance rows, `Family.name`.
|
||
- Produces:
|
||
- `@dataclass(frozen=True) LineageEvent(code, year, kind, from_codes, to_codes, label, item_key, p_id, page, url, source, anchored, note)`.
|
||
- `@dataclass(frozen=True) LineageEvidence(codes, families, events: tuple[LineageEvent,...], element_diffs: tuple[ElementDiff,...], guidance: tuple[GuidanceRef,...])` with `.payload() -> dict` (`{"type": "lineage", "codes", "families", "events": [asdict...], "element_diffs": [...], "guidance": [...]}`) and `.prompt_block() -> str`.
|
||
- `ElementDiff(type, value, in_codes: tuple[str,...], not_in_codes: tuple[str,...], label, item_key, p_id)` — for a detected family (or ≥ 2 detected codes) the element values present for some codes and absent for others, from `pfs.code_element` (newest year per code).
|
||
- `GuidanceRef(kind, locator, url, label, item_key_src, p_id_src)` from `pfs.code_guidance` for the detected families (≤ 8 rows, CFR first, deduped by locator).
|
||
- `rule_label(title: str, date_published: str, p_id: int) -> str` → `"CY2021 PFS final ¶1578"` / `"CY2027 PFS proposed ¶394"` (proposed when the title contains "Proposed"; year via `rule_year_of`); CPT events label `"CPT Changes 2022"`; rvu-only events `"PFS CY2020 RVU file"`.
|
||
- `lineage_evidence(question, cfg) -> LineageEvidence | None` — `None` when no codes detected; events from `read_events` per code (≤ `lineage_on_demand_max` codes fall back to `pfs.lineage.lineage(...)` when a code has no rows, cached in-process by `(replica mtime, code)`); events collapsed to one row per `(year, kind, from_codes, to_codes, code)` preferring anchored FR rows, sorted by year then code, capped at `lineage_max_rows` for the prompt block (payload carries all collapsed rows); never raises (log + `None`).
|
||
- `build_messages(question, sources, evidence=None, lineage=None)` inserts the lineage block after the excerpts and before the valuation block; `_SYSTEM` gains one paragraph: the Lineage section is authoritative for dates/predecessors/successors; cite its bracketed label for every dated claim; never invent a year without a label.
|
||
- `stream_answer` yields `lineage.payload()` first (before valuation) when `lineage_evidence` returns non-None; when the lineage detected families, its codes are unioned into the `code_cited_sources` call's `codes`.
|
||
|
||
**Rules:** payload/prompt labels identical strings; the prompt block format is one line per event `[label] YEAR KIND CODE (from → to) — note` and one line per element diff `[label] element type=value: in 99490, 99491; not in G0556`; guidance lines `[label] 42 CFR 410.78(a)(3) — cited by …`.
|
||
|
||
- [ ] **Step 1: Failing tests** — `tests/llm/test_lineage.py`: build events from a real temp DuckDB (`pfs.code_event`/`code_element`/`code_guidance` DDL via `ensure_tables`) with the 99490/G2058/99439 fixture rows above → collapsed row order, labels (`CY2021 PFS final ¶1578` from a fake `Store` returning the rule title/date), element diff for CCM vs APCM fixture elements, guidance refs with CFR URLs; on-demand fallback called only for codes without rows and at most `lineage_on_demand_max`; `payload()` JSON-serialisable; `prompt_block()` exact text on a two-event fixture; `None` for a control question. `tests/llm/test_rag.py`: event order `lineage → valuation → token* → sources → done`; control question events byte-identical to a recorded baseline (json.dumps of every event with a fixed fake pool, compared against the same run with `lineage_evidence` patched to `None`); `build_messages` places the block correctly.
|
||
- [ ] **Step 2: Run, confirm failures.**
|
||
- [ ] **Step 3: Implement.**
|
||
- [ ] **Step 4: Run** `uv run pytest tests/llm -q -p no:cacheprovider`; ruff.
|
||
- [ ] **Step 5: Live** (env wrapper): time `lineage_evidence("history of CCM coding and payment", cfg)` ×5 on the live replica — report ms and the collapsed event rows; run `"what replaced G2058 and why"` and `"does 99490 pass telehealth step 3"` (elements present?).
|
||
- [ ] **Step 6: Commit** `feat(llm): lineage evidence — timeline events, element diffs and guidance as a lineage SSE event + cited prompt block (refs #691)`.
|
||
|
||
---
|
||
|
||
### Task 4: Era-balanced retrieval — `retrieve(..., mode="timeline")`, trigger, API `mode`
|
||
|
||
**Files:**
|
||
- Modify: `src/llm/rag.py` (`retrieve`, `stream_answer(mode=...)`, `is_history_question`), `src/llm/api.py` (`ChatRequest.mode`), `src/llm/config.py` + `stack.toml` (`timeline_per_era = 2`, `timeline_overfetch = 6`)
|
||
- Test: `tests/llm/test_rag.py`, `tests/llm/test_api.py`
|
||
|
||
**Interfaces:**
|
||
- Consumes: `_hits`, `filter_since`, `blend`, `Hit` metadata (`kind`, `title`, `date`, `docket`), `rule_year_of`.
|
||
- Produces: `HISTORY_RE` (history, evolution, "when did", "when was", replaced, predecessor, successor, timeline, "over the years", "since 20"); `is_history_question(q) -> bool`; `era_of(hit) -> int` (rules: `rule_year_of(title, date)`; comments: docket year from `docket` → the rule year the docket belongs to when `Store.dockets()` maps it, else `date[:4]`; corpus: `date[:4]`, 0 when unknown); `era_balance(hits, *, per_era, top_n) -> list[Hit]` — group by era, take the best `per_era` per era (score order), then fill to `top_n` by score from the remainder; `retrieve(..., mode="auto"|"timeline"|"recent")` — `timeline` overfetches `timeline_overfetch × k` per kind, skips the recency blend and applies `era_balance`; `auto` = `timeline` when `is_history_question(question)` else `recent` (today's behaviour); `stream_answer(..., mode="auto")`; `ChatRequest.mode: Literal["auto","timeline","recent"] = "auto"`, validated (400 otherwise).
|
||
|
||
**Rules:** with `mode="recent"` the output is byte-identical to main for every question; `era_balance` is deterministic (stable sort by (-score, label)).
|
||
|
||
- [ ] **Step 1: Failing tests** — `is_history_question` positives/negatives; `era_of` for the three kinds; `era_balance` picks CY2015 and CY2027 hits over three CY2026 hits with higher scores; `retrieve(mode="timeline")` uses the overfetch and skips `blend` (patch and assert); `mode="recent"` path unchanged (existing tests untouched); API accepts `mode`, rejects `mode="x"` with 400, forwards to `stream_answer`.
|
||
- [ ] **Step 2–4:** run/implement/run; ruff.
|
||
- [ ] **Step 5: Live** (env wrapper): `retrieve("history of chronic care management payment", cfg=..., pool=..., mode="timeline")` — report the eras represented vs `mode="recent"`.
|
||
- [ ] **Step 6: Commit** `feat(llm): era-balanced retrieval for history questions — retrieve(mode="timeline"), auto trigger, ChatRequest.mode (refs #691)`.
|
||
|
||
---
|
||
|
||
### Task 5: The CPT manual as a source; item_key/p_id on source dicts
|
||
|
||
**Files:**
|
||
- Modify: `src/llm/evidence.py` (`manual_sources`), `src/llm/links.py` (`as_source` adds `item_key`, `p_id`, `seq`, `section`), `src/llm/rag.py` (merge after cited sources)
|
||
- Test: `tests/llm/test_evidence.py`, `tests/llm/test_links.py`
|
||
|
||
**Interfaces:**
|
||
- Consumes: `FamilyRow.note` = the CPT heading path (via `pfs.codetables.read_family_rows` or the `pfs.code_family` table: `select distinct note from pfs.code_family where key = ?`), `pfs.cpt_years(con)` newest edition and its `item_key` (`pfs.cpt_section.item_key`), corpus chunk metadata `section` (markdown heading) and `item_key`; `_collect`/`text()` binding style from `code_cited_sources`.
|
||
- Produces: `manual_sources(engine, con, families, *, per_family=1) -> list[dict]` — for each detected family with a CPT heading, the corpus chunk(s) of the newest edition whose `cmetadata->>'section'` equals the heading's last path segment (case-insensitive; fall back to `ILIKE '%<title>%'`), first `seq`, labelled by `as_source` with `title` = `"CPT <year> — <heading title>"`; merged after the cited sources in `stream_answer` (dedupe by label; counts toward `code_cited_max`? **No** — manual rows are added after the cap, at most one per family); `as_source` carries `item_key`, `p_id` (rules), `seq` (others) and `section` so the eval can match anchors.
|
||
|
||
**Rules:** the P48 `test_serves_chat_page…` style assertions and every existing `as_source` test keep passing (new keys are additive); a family without a CPT heading yields nothing; SQL parameterised.
|
||
|
||
- [ ] **Step 1: Failing tests** — fake engine returns a corpus row for `CCM`'s heading → one source with `kind="corpus"`, `title` starting `CPT 2024 — `; no heading → empty; `as_source` new keys; `stream_answer` merges manual rows after cited rows (order asserted).
|
||
- [ ] **Step 2–4:** run/implement/run; ruff.
|
||
- [ ] **Step 5: Live** (env wrapper): `manual_sources(engine, con, ["CCM"])` returns the GQGTPGYV chunk whose section is "Chronic Care Management Services" — print label/section/seq.
|
||
- [ ] **Step 6: Commit** `feat(llm): the CPT manual's heading text as a chat source; item_key/p_id on sources (refs #691 #705)`.
|
||
|
||
---
|
||
|
||
### Task 6: UI — timeline table, element diffs, guidance, mode toggle
|
||
|
||
**Files:**
|
||
- Modify: `src/llm/web/chat.html`
|
||
- Test: `tests/llm/test_api.py` (`TestIndex` marker strings)
|
||
|
||
**Interfaces:**
|
||
- Consumes: the `lineage` payload (Task 3), `ChatRequest.mode` (Task 4), the `renderValuation`/`renderSources` DOM pattern and `.cite` painting.
|
||
- Produces: `renderLineage(wrap, ev)` — `<table class="lineage">` (Year, Event, Codes, Source) one row per event, `Source` cell = `<a href=url>[label]</a>` (span when `url` empty), rows with `anchored=false` get class `unanchored` and a title tooltip; below it a compact `<ul class="elements">` of element diffs (`type=value — in …; not in …`) and a `<ul class="guidance">` with CFR/IOM links; a `<label class="mode"><select id="mode">auto/timeline/recent</select></label>` beside the "last 12 months" checkbox, sent as `mode` in the POST body; the `ask()` dispatch gains `else if (ev.type === 'lineage') renderLineage(wrap, ev)` before the valuation branch; CSS `table.lineage`, `.lineage-wrap`, `tr.unanchored`.
|
||
|
||
- [ ] **Step 1: Failing test** — `test_serves_chat_page_with_lineage_renderer`: `"function renderLineage"`, `"ev.type === 'lineage'"`, `"table.lineage"`, `"id=\"mode\""`, `"tr.unanchored"` in the served HTML.
|
||
- [ ] **Step 2–4:** run/implement/run.
|
||
- [ ] **Step 5: Headless check** — `uv run python -c` that serves the page through the FastAPI TestClient and greps the markers; if `playwright` is available in the venv, load the page and post a canned SSE (skip otherwise, say so).
|
||
- [ ] **Step 6: Commit** `feat(llm): timeline table, element diffs and guidance in the chat UI; mode selector (refs #691)`.
|
||
|
||
---
|
||
|
||
### Task 7: Golden evaluation — `tests/llm/golden_lineage.yaml`, runner, nightly workflow, filer
|
||
|
||
**Files:**
|
||
- Create: `tests/llm/golden_lineage.yaml`, `dev/scripts/llm_golden.py`, `tests/llm/test_golden.py`
|
||
- Modify: `stack.toml` (`[ci.llm_golden]` or the existing ci section the generator reads), `dev/scripts/gen_config.py` (emit `.gitea/workflows/llm-golden.yml` mirroring `notebooks-integration.yml`: `docker exec llm …`, `--file-issues`, report artifact, failure filer), regenerated `.gitea/workflows/llm-golden.yml`
|
||
- Test: `tests/llm/test_golden.py`, `tests/dev/test_gen_config.py` (if such tests exist; otherwise a snapshot test that the generated workflow contains the docker exec line)
|
||
|
||
**Interfaces:**
|
||
- Consumes: `/chat` SSE (`data: {json}` lines; events `lineage`, `valuation`, `token`, `sources`, `done`, `error`), source dicts with `item_key`/`p_id`/`seq`/`label` (Task 5), `dev/scripts/nb_issue_filer.py` (`signature`, `cmd_report`, `MARKER`, `decide`) — import it, do not copy it.
|
||
- Produces:
|
||
- YAML schema, one entry per question: `id`, `question`, `mode` (default auto), `expect_anchors: [{item_key, p_id}]` (must appear in `sources`), `expect_labels: [regex]` (must be cited in the answer prose, e.g. `CY2021 PFS final ¶1578`), `forbid: [{pattern, unless_label: regex}]` (e.g. `pattern: "\\$\\d"`, `unless_label: "Addendum B"`), `expect_events: [{code, kind, year}]` (must be in the `lineage` payload), `min_eras: 3` (distinct rule years among sources).
|
||
- Seed entries (from #692, anchors verified on the corpus): (1) history of CCM coding and payment → `DE2VH9PD` ¶1249–1251, `YBM4IZUS` ¶1578, `JJ6AM5HJ` ¶1163; events `99490 created 2015`, `G2058 replaced_by 2021`, `G0556 created 2025`; (2) what replaced G2058 and why → `YBM4IZUS` ¶1578, ¶2369; (3) how do APCM's elements differ from CCM's → `JJ6AM5HJ` ¶1164–1185 vs `DE2VH9PD` ¶1245–1247; (4) when were audio-only E/M codes payable and why did 99441–99443 end → `XFGGRBDH` ¶489, event `99441 deleted|disappeared 2025`; (5) what did commenters say about G2211 in 2023 vs 2025 → sources from dockets CMS-2023-0121 and CMS-2025-0304 (`expect_dockets`); (6) does 99490 pass telehealth Steps 1–3 → `2KVJ2HKX` ¶394/¶396/¶398; (7) how did G2064/G2065 → 99424/99426 change the RHC/FQHC G0511 rate → `JE7KYBW3` ¶1100/¶1111, `MZ24MX5S` ¶1305.
|
||
- `dev/scripts/llm_golden.py run --url http://llm:8000 --set tests/llm/golden_lineage.yaml --report report.json [--file-issues --source nightly-llm-golden] [--only id]` — streams each question, evaluates the checks (`check_anchors`, `check_labels`, `check_forbidden`, `check_events`, `check_eras`, `check_dockets` — pure functions over the collected events, unit-tested), prints a per-question table, exit 1 on any failure, files/sweeps issues through `nb_issue_filer` with `signature(question_id, check_name, detail)`.
|
||
- `tests/llm/test_golden.py`: the checkers against a canned SSE transcript (pass and fail cases); YAML loads and every entry has the required keys; a live test skipped unless `LLM_CHAT_URL` is set that runs entry (2) and asserts pass.
|
||
- Workflow: nightly `40 3 * * *` after notebooks-integration, `docker exec llm uv run python /tmp/golden/llm_golden.py run --url http://localhost:8000 …` (copy the script and YAML in with `docker cp` like the notebooks job), `GITEA_TOKEN` from `secrets.DEPLOY_TOKEN`, failure filer step identical to the notebooks job.
|
||
|
||
- [ ] **Step 1: Failing tests** (checkers, YAML shape, gen_config snapshot).
|
||
- [ ] **Step 2–4:** run/implement/run; `uv run python dev/scripts/gen_config.py` regenerates workflows (commit them).
|
||
- [ ] **Step 5: Live** (env wrapper; the chat service must be reachable — check `LLM_CHAT_URL` or the compose service `http://localhost:8000`; if unreachable, run the checkers against a transcript captured with `stream_answer` directly and say so): `uv run python dev/scripts/llm_golden.py run --url … --set tests/llm/golden_lineage.yaml --report scratch/golden-report.json` — report pass/fail per question with the failing checks; do NOT file issues from the worktree run.
|
||
- [ ] **Step 6: Commit** `feat(llm): golden longitudinal evaluation — yaml set, runner with anchor/label/forbidden checks, nightly workflow + issue filer (refs #692)`.
|
||
|
||
---
|
||
|
||
### Task 8: Notebook section, docs, tracker
|
||
|
||
**Files:**
|
||
- Modify: `notebooks/code_families.py` (new section "7b. What the chat sees" — `lineage_evidence` events/labels for the picked code, rendered as a table with the same labels the chat cites, plus the `prompt_block()` text in a collapsible), `docs/docs/cli/*` (regen), `tests/notebooks/test_code_families_nb.py` (banner assertion for the new section)
|
||
- Steps: implement the cell (guarded like sections 5–7; uses the replica only — `lineage_evidence` with a config pointing at `data/replica/aco.ro.duckdb`), headless export with and without env, `uv run pytest tests/notebooks -q`, `uv run python docs/scripts/extract_cli.py`, commit `feat(notebooks,docs): code_families 7b — the chat's lineage view; CLI docs (refs #691)`. The controller posts the live numbers on #691/#692 and closes #699 items 1–2 with a comment.
|
||
|
||
---
|
||
|
||
## Self-review
|
||
|
||
**Spec coverage.** Decision 5 (era-balanced) → T4; Decision 6 (timeline like valuation) → T3 + T6; "lineage_evidence → events + element diffs" → T3; "system prompt: cite the event label" → T3; UI links (P40/P41) → T3 labels/urls + T6; Evaluation section → T7; "ANY PFS-payable code" (user) → T1 (families) + T2 (events for all codes); Ruling A9 (manual source) → T5; #699 items 1–2 → T1. Not in this slice: #699 item 3 (synonym curation beyond the name-phrase rule), #698 elements inversion (T2 inverts events only; elements stay 20 codes — the chat's element diffs therefore cover the fixture families until #698 lands; say so in the notebook cell), #703, #700.
|
||
|
||
**Placeholders.** Tasks give signatures, rules, test intents and live checks; implementers are mid-tier models working from prose (allowed by the SDD model-selection rule). Exact SQL for the manual source and era grouping is left to the implementer with the binding style references.
|
||
|
||
**Type consistency.** `LineageEvidence.payload()["type"] == "lineage"` (T3) is what T6 dispatches on and T7 reads; `as_source` keys added in T5 (`item_key`, `p_id`, `seq`) are what T7's `check_anchors` matches; `ChatRequest.mode` (T4) is what T6 posts and T7's YAML `mode` sends; `rule_label` (T3) strings are what T7's `expect_labels` regexes target.
|