28 KiB
Code-family longitudinal chat + golden evaluation — Implementation Plan (P49 slice 3)
For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (
- [ ]) syntax for tracking.
Goal: A "history of X" question in the LLM chat gets a dated, anchored timeline for the detected codes and families (a lineage SSE event and a prompt block the model must cite), era-balanced retrieval so CY2015 and CY2027 both surface, the CPT manual's own heading text as a source, and a golden question set that runs nightly against the live chat and files an issue on regression — for ANY PFS-payable code, not only the five hand families.
Architecture: Mirrors P48's valuation flow. llm/evidence.py gains lineage_evidence(question, cfg) which reads pfs.code_event, pfs.code_element and pfs.code_guidance from the cached read-only replica handle (falling back to on-demand pfs.lineage.lineage(...) for codes with no precomputed rows), returns a payload for the SSE lineage event and a "Lineage" prompt block whose rows carry bracketed labels ([CY2021 PFS final ¶1578]). rag.retrieve(..., mode="timeline") groups hits by rule year and caps per era. chat.html renders the timeline between the answer and the sources drawer. stack pfs lineage --all-payable --write precomputes events for every A/R/T code with one inverted pass over fr_anchors. tests/llm/golden_lineage.yaml + dev/scripts/llm_golden.py check anchors, cited labels and forbidden claims against the live /chat endpoint, nightly, through the existing issue filer.
Tech Stack: Python 3.13, FastAPI SSE, DuckDB replica (read-only cached handle, per-call cursor), SQLite bib (fr_anchors, items), pgvector via SQLAlchemy text(), Ollama self-hosted pool, marimo notebook, Gitea Actions workflows generated by dev/scripts/gen_config.py from stack.toml.
Spec: docs/superpowers/specs/2026-09-09-code-family-longitudinal-design.md (§Decisions 5–6, §Components "longitudinal chat", §Evaluation). Tracker: #691 (chat), #692 (eval), #699 items 1–2 (derived families in chat), #705 item 1 (CPT manual as a chat source, Ruling A9 of the anchors slice).
Global Constraints
- Self-hosted inference only (Ollama pool from config); no cloud LLM API anywhere in
src/llmorsrc/pfs. pfs.*modules never touch DuckDB at import time; the chat reads the replica throughllm/evidence.py's cached handle (_connect, mtime-keyed reopen, per-call.cursor()); writes only throughconf.connect.duckdb_batch+publish_replica, never inside slow work (build first, then open the batch).- Control question is byte-identical: for a question with no detected code, the SSE event sequence and every event payload are unchanged from main
4616d36(token* → sources → done); for a coded question without history words the sequence isvaluation? → token* → sources → doneexactly as today plus the newlineageevent first:lineage? → valuation? → token* → sources → done. - Evidence budget:
lineage_evidenceadds < 300 ms wall-clock on the live replica for ≤ 3 codes with precomputed rows; on-demand fallback runs for at most 3 codes per question and is skipped (logged) beyond that. - SSE payloads are JSON-serialisable dicts with a
typekey; the API wraps them asdata: {json}\n\n(noevent:field). - Bracketed labels in prompt blocks and payloads use the exact same string; the system prompt tells the model to cite them verbatim.
- Every FR anchor link is built from
(item_key, p_id)throughbib.frlink(_para_linkviaresolve/place), eCFR links throughbib.cfrlink.url. - Tests: no live DB or Ollama required (fakes as in
tests/llm/test_rag.pyandtests/llm/test_evidence.py); live checks skip withoutLLM_DB_PASSWORD/LLM_CHAT_URL. - Commit messages: conventional prefix,
(refs #NNN); never add a Co-Authored-By or any trailer. - Docs: regenerate
docs/docs/cli/fully after CLI changes (uv run python docs/scripts/extract_cli.py), commit every changed page.
Task 1: Derived families in the chat (#699 items 1–2)
Files:
- Modify:
src/pfs/families.py(detect_codes,family_of, a compiled index built byrefresh_fromand onFAMILIESmutation),src/llm/evidence.py(_connect→ callrefresh_fromwhen the replica (re)opens) - Test:
tests/pfs/test_families.py,tests/llm/test_evidence.py
Interfaces:
- Consumes:
pfs.families.FAMILIES,Family(key, name, codes, synonyms),refresh_from(con) -> int,detect_codes(text) -> Detection(codes, families, explicit),code_family_index(families)inpfs.anchors. - Produces:
pfs.families.rebuild_index() -> None(called byrefresh_fromand at import afterHAND_FAMILIESseeding);detect_codesunchanged signature, same results for the five hand families (regression fixture), now also detecting derived families by member code and by name phrase (see rule below);family_of(code) -> Family | NoneO(1) via the code→family index (first key in sorted order when a code belongs to several families, hand families first);llm.evidence._connect(path)refreshesFAMILIESfrompfs.code_familywhenever it opens or reopens the replica (mtime change), logging the family count.
Rules (binding):
-
Family detection by synonym stays for hand families. A derived family is detected by name only when its
nameis ≥ 12 characters and ≥ 2 words and appears as a case-insensitive whole-phrase match; single-word CPT headings (GENE,ADM,REVERSE,Introduction) never match by name. Detection by member code applies to every family. -
One compiled alternation regex over all synonym/name phrases (longest first, word-bounded) plus a
dict[code, tuple[key, ...]]index, rebuilt inrebuild_index();detect_codesmust run in < 5 ms on a 300-character question with the live 8,518-key registry (measure in a test markedlive, skipped without the replica). -
refresh_fromis idempotent and thread-safe enough for the chat: it builds the new dict/index first and swaps them under_REGISTRY_LOCK. -
Step 1: Failing tests —
tests/pfs/test_families.py:test_detect_derived_family_by_member_code(a fakeFAMILIESentrySUTURE-REMOVALwith codes15850 15851; "removal of sutures 15850" → families containsSUTURE-REMOVAL),test_derived_family_name_phrase_requires_two_words(GENEname never matches; "Chronic Care Management Services" matches by name),test_family_of_is_indexed(monkeypatchFAMILIESwith 5,000 synthetic families; 10,000family_ofcalls < 0.2 s),test_hand_families_regression(the existing five-family detections unchanged — reuse existing assertions).tests/llm/test_evidence.py:test_connect_refreshes_families_on_open_and_reopen(patchllm.evidence.refresh_from; assert called once on first_connect, not on a cached call, again after_mtimechanges). -
Step 2: Run, confirm failures.
-
Step 3: Implement
rebuild_index, index-backedfamily_of, phrase regex indetect_codes, therefresh_fromswap, and the_connecthook (refresh_from(con)inside the lock after a successful open; exceptions logged, never raised). -
Step 4: Run
uv run pytest tests/pfs/test_families.py tests/llm -q -p no:cacheprovider; ruff. -
Step 5: Live check (env wrapper):
uv run python -c "from pfs.families import detect_codes, FAMILIES, refresh_from; import duckdb; refresh_from(duckdb.connect('data/replica/aco.ro.duckdb', read_only=True)); import time; t=time.perf_counter(); d=detect_codes('history of chronic care management services and 99490'); print(len(FAMILIES), d, (time.perf_counter()-t)*1000, 'ms')"— report the ms. -
Step 6: Commit
feat(pfs,llm): derived families reach the chat — indexed detect_codes/family_of, refresh on replica open (refs #699).
Task 2: Precompute lineage for every payable code — stack pfs lineage --all-payable --write
Files:
- Modify:
src/pfs/lineage.py(inverted FR pass),src/cli/pfs.py(lineagecommand gains--all-payable),src/pfs/codetables.py(write_events_many(con, by_code)bulk writer if not present) - Test:
tests/pfs/test_lineage.py,tests/cli/test_pfs_cli.py
Interfaces:
- Consumes:
lineage(con, store, code) -> list[EventRow],rvu_events,fr_events(store, code),cpt_events,pfs.families.find_codes,codes_in/code_pattern,pfs.valuation/pfs.rvufor the A/R/T universe (the same querystack pfs elements --all-payableuses — reuse its helper),write_events/read_all_events. - Produces:
fr_events_bucketed(store, codes: Sequence[str]) -> dict[str, list[EventRow]]— ONE pass overfr_anchorsjoined toitems(SQL prefilter: text LIKE any target code is too wide for 8.7k codes → instead stream all rows ordered by item_key, p_id and bucket withfind_codes(text)∩ targets), then the existing per-code event-verb logic applied to each code's bucket (factor the verb/anchor logic out offr_eventsinto_events_for(code, rows)sofr_events(store, code)==fr_events_bucketed(store, [code])[code]);lineage_all(con, store, codes) -> dict[str, list[EventRow]]merging rvu/fr/cpt per code with the same ±1-year anchoring; CLIstack pfs lineage --all-payable [--write] [--limit N]printing counts per source and unanchored count, writing per code (delete-then-insert per code inside one batch, opened only after the build — Ruling A13).
Rules: equality test fr_events_bucketed(store, [c])[c] == fr_events(store, c) for the seven fixture codes on the sqlite fixture; the full pass must finish in < 10 minutes on the live bib (193k paragraphs) — report the time; --limit N takes the first N target codes for smoke runs.
- Step 1: Failing tests — bucketed == per-code equality on a fixture store with three rules and four codes;
lineage_allmarks rvu events anchored when an FR/CPT event is within ±1 year; CLI--all-payable --limit 3 --writecalls the writer once per code after the build and publishes (indirections monkeypatched, call order asserted);--all-payablewithout--writenever opens the batch. - Step 2: Run, confirm failures.
- Step 3: Implement.
- Step 4: Run
uv run pytest tests/pfs/test_lineage.py tests/cli/test_pfs_cli.py -q -p no:cacheprovider; ruff. - Step 5: Live (env wrapper):
stack pfs lineage --all-payable --limit 50(dry) to time the pass, thenstack pfs lineage --all-payable --write— report codes written, events per source, unanchored share, wall time; confirmpfs.code_eventnow has rows for G2211 and 99441 on the replica. - Step 6: Commit
feat(pfs): lineage for every payable code — one inverted pass over fr_anchors, stack pfs lineage --all-payable (refs #691 #698).
Task 3: lineage_evidence — events, element diffs, guidance; SSE payload and prompt block
Files:
- Create:
src/llm/lineage.py - Modify:
src/llm/evidence.py(export),src/llm/rag.py(stream_answer,build_messages,_SYSTEM),src/llm/config.py+stack.toml(lineage_max_rows = 25,lineage_on_demand_max = 3) - Test:
tests/llm/test_lineage.py,tests/llm/test_rag.py
Interfaces:
- Consumes:
detect_codes,_connect(path)/_replica_path(cfg)fromllm/evidence.py,read_events/read_all_events,read_elements,read_guidance,pfs.lineage.lineage(con, store, code)(on-demand fallback; the bibStorefromconf.connect.bib()),pfs.descriptors.rule_year_of,bib.frlink.resolve(ref, store=..., item_key=...)withref=f"p-{p_id}"for the FR URL,bib.cfrlink.url(parse_cite(locator))for CFR guidance rows,Family.name. - Produces:
@dataclass(frozen=True) LineageEvent(code, year, kind, from_codes, to_codes, label, item_key, p_id, page, url, source, anchored, note).@dataclass(frozen=True) LineageEvidence(codes, families, events: tuple[LineageEvent,...], element_diffs: tuple[ElementDiff,...], guidance: tuple[GuidanceRef,...])with.payload() -> dict({"type": "lineage", "codes", "families", "events": [asdict...], "element_diffs": [...], "guidance": [...]}) and.prompt_block() -> str.ElementDiff(type, value, in_codes: tuple[str,...], not_in_codes: tuple[str,...], label, item_key, p_id)— for a detected family (or ≥ 2 detected codes) the element values present for some codes and absent for others, frompfs.code_element(newest year per code).GuidanceRef(kind, locator, url, label, item_key_src, p_id_src)frompfs.code_guidancefor the detected families (≤ 8 rows, CFR first, deduped by locator).rule_label(title: str, date_published: str, p_id: int) -> str→"CY2021 PFS final ¶1578"/"CY2027 PFS proposed ¶394"(proposed when the title contains "Proposed"; year viarule_year_of); CPT events label"CPT Changes 2022"; rvu-only events"PFS CY2020 RVU file".lineage_evidence(question, cfg) -> LineageEvidence | None—Nonewhen no codes detected; events fromread_eventsper code (≤lineage_on_demand_maxcodes fall back topfs.lineage.lineage(...)when a code has no rows, cached in-process by(replica mtime, code)); events collapsed to one row per(year, kind, from_codes, to_codes, code)preferring anchored FR rows, sorted by year then code, capped atlineage_max_rowsfor the prompt block (payload carries all collapsed rows); never raises (log +None).build_messages(question, sources, evidence=None, lineage=None)inserts the lineage block after the excerpts and before the valuation block;_SYSTEMgains one paragraph: the Lineage section is authoritative for dates/predecessors/successors; cite its bracketed label for every dated claim; never invent a year without a label.stream_answeryieldslineage.payload()first (before valuation) whenlineage_evidencereturns non-None; when the lineage detected families, its codes are unioned into thecode_cited_sourcescall'scodes.
Rules: payload/prompt labels identical strings; the prompt block format is one line per event [label] YEAR KIND CODE (from → to) — note and one line per element diff [label] element type=value: in 99490, 99491; not in G0556; guidance lines [label] 42 CFR 410.78(a)(3) — cited by ….
- Step 1: Failing tests —
tests/llm/test_lineage.py: build events from a real temp DuckDB (pfs.code_event/code_element/code_guidanceDDL viaensure_tables) with the 99490/G2058/99439 fixture rows above → collapsed row order, labels (CY2021 PFS final ¶1578from a fakeStorereturning the rule title/date), element diff for CCM vs APCM fixture elements, guidance refs with CFR URLs; on-demand fallback called only for codes without rows and at mostlineage_on_demand_max;payload()JSON-serialisable;prompt_block()exact text on a two-event fixture;Nonefor a control question.tests/llm/test_rag.py: event orderlineage → valuation → token* → sources → done; control question events byte-identical to a recorded baseline (json.dumps of every event with a fixed fake pool, compared against the same run withlineage_evidencepatched toNone);build_messagesplaces the block correctly. - Step 2: Run, confirm failures.
- Step 3: Implement.
- Step 4: Run
uv run pytest tests/llm -q -p no:cacheprovider; ruff. - Step 5: Live (env wrapper): time
lineage_evidence("history of CCM coding and payment", cfg)×5 on the live replica — report ms and the collapsed event rows; run"what replaced G2058 and why"and"does 99490 pass telehealth step 3"(elements present?). - Step 6: Commit
feat(llm): lineage evidence — timeline events, element diffs and guidance as a lineage SSE event + cited prompt block (refs #691).
Task 4: Era-balanced retrieval — retrieve(..., mode="timeline"), trigger, API mode
Files:
- Modify:
src/llm/rag.py(retrieve,stream_answer(mode=...),is_history_question),src/llm/api.py(ChatRequest.mode),src/llm/config.py+stack.toml(timeline_per_era = 2,timeline_overfetch = 6) - Test:
tests/llm/test_rag.py,tests/llm/test_api.py
Interfaces:
- Consumes:
_hits,filter_since,blend,Hitmetadata (kind,title,date,docket),rule_year_of. - Produces:
HISTORY_RE(history, evolution, "when did", "when was", replaced, predecessor, successor, timeline, "over the years", "since 20");is_history_question(q) -> bool;era_of(hit) -> int(rules:rule_year_of(title, date); comments: docket year fromdocket→ the rule year the docket belongs to whenStore.dockets()maps it, elsedate[:4]; corpus:date[:4], 0 when unknown);era_balance(hits, *, per_era, top_n) -> list[Hit]— group by era, take the bestper_eraper era (score order), then fill totop_nby score from the remainder;retrieve(..., mode="auto"|"timeline"|"recent")—timelineoverfetchestimeline_overfetch × kper kind, skips the recency blend and appliesera_balance;auto=timelinewhenis_history_question(question)elserecent(today's behaviour);stream_answer(..., mode="auto");ChatRequest.mode: Literal["auto","timeline","recent"] = "auto", validated (400 otherwise).
Rules: with mode="recent" the output is byte-identical to main for every question; era_balance is deterministic (stable sort by (-score, label)).
- Step 1: Failing tests —
is_history_questionpositives/negatives;era_offor the three kinds;era_balancepicks CY2015 and CY2027 hits over three CY2026 hits with higher scores;retrieve(mode="timeline")uses the overfetch and skipsblend(patch and assert);mode="recent"path unchanged (existing tests untouched); API acceptsmode, rejectsmode="x"with 400, forwards tostream_answer. - Step 2–4: run/implement/run; ruff.
- Step 5: Live (env wrapper):
retrieve("history of chronic care management payment", cfg=..., pool=..., mode="timeline")— report the eras represented vsmode="recent". - Step 6: Commit
feat(llm): era-balanced retrieval for history questions — retrieve(mode="timeline"), auto trigger, ChatRequest.mode (refs #691).
Task 5: The CPT manual as a source; item_key/p_id on source dicts
Files:
- Modify:
src/llm/evidence.py(manual_sources),src/llm/links.py(as_sourceaddsitem_key,p_id,seq,section),src/llm/rag.py(merge after cited sources) - Test:
tests/llm/test_evidence.py,tests/llm/test_links.py
Interfaces:
- Consumes:
FamilyRow.note= the CPT heading path (viapfs.codetables.read_family_rowsor thepfs.code_familytable:select distinct note from pfs.code_family where key = ?),pfs.cpt_years(con)newest edition and itsitem_key(pfs.cpt_section.item_key), corpus chunk metadatasection(markdown heading) anditem_key;_collect/text()binding style fromcode_cited_sources. - Produces:
manual_sources(engine, con, families, *, per_family=1) -> list[dict]— for each detected family with a CPT heading, the corpus chunk(s) of the newest edition whosecmetadata->>'section'equals the heading's last path segment (case-insensitive; fall back toILIKE '%<title>%'), firstseq, labelled byas_sourcewithtitle="CPT <year> — <heading title>"; merged after the cited sources instream_answer(dedupe by label; counts towardcode_cited_max? No — manual rows are added after the cap, at most one per family);as_sourcecarriesitem_key,p_id(rules),seq(others) andsectionso the eval can match anchors.
Rules: the P48 test_serves_chat_page… style assertions and every existing as_source test keep passing (new keys are additive); a family without a CPT heading yields nothing; SQL parameterised.
- Step 1: Failing tests — fake engine returns a corpus row for
CCM's heading → one source withkind="corpus",titlestartingCPT 2024 —; no heading → empty;as_sourcenew keys;stream_answermerges manual rows after cited rows (order asserted). - Step 2–4: run/implement/run; ruff.
- Step 5: Live (env wrapper):
manual_sources(engine, con, ["CCM"])returns the GQGTPGYV chunk whose section is "Chronic Care Management Services" — print label/section/seq. - Step 6: Commit
feat(llm): the CPT manual's heading text as a chat source; item_key/p_id on sources (refs #691 #705).
Task 6: UI — timeline table, element diffs, guidance, mode toggle
Files:
- Modify:
src/llm/web/chat.html - Test:
tests/llm/test_api.py(TestIndexmarker strings)
Interfaces:
-
Consumes: the
lineagepayload (Task 3),ChatRequest.mode(Task 4), therenderValuation/renderSourcesDOM pattern and.citepainting. -
Produces:
renderLineage(wrap, ev)—<table class="lineage">(Year, Event, Codes, Source) one row per event,Sourcecell =<a href=url>[label]</a>(span whenurlempty), rows withanchored=falseget classunanchoredand a title tooltip; below it a compact<ul class="elements">of element diffs (type=value — in …; not in …) and a<ul class="guidance">with CFR/IOM links; a<label class="mode"><select id="mode">auto/timeline/recent</select></label>beside the "last 12 months" checkbox, sent asmodein the POST body; theask()dispatch gainselse if (ev.type === 'lineage') renderLineage(wrap, ev)before the valuation branch; CSStable.lineage,.lineage-wrap,tr.unanchored. -
Step 1: Failing test —
test_serves_chat_page_with_lineage_renderer:"function renderLineage","ev.type === 'lineage'","table.lineage","id=\"mode\"","tr.unanchored"in the served HTML. -
Step 2–4: run/implement/run.
-
Step 5: Headless check —
uv run python -cthat serves the page through the FastAPI TestClient and greps the markers; ifplaywrightis available in the venv, load the page and post a canned SSE (skip otherwise, say so). -
Step 6: Commit
feat(llm): timeline table, element diffs and guidance in the chat UI; mode selector (refs #691).
Task 7: Golden evaluation — tests/llm/golden_lineage.yaml, runner, nightly workflow, filer
Files:
- Create:
tests/llm/golden_lineage.yaml,dev/scripts/llm_golden.py,tests/llm/test_golden.py - Modify:
stack.toml([ci.llm_golden]or the existing ci section the generator reads),dev/scripts/gen_config.py(emit.gitea/workflows/llm-golden.ymlmirroringnotebooks-integration.yml:docker exec llm …,--file-issues, report artifact, failure filer), regenerated.gitea/workflows/llm-golden.yml - Test:
tests/llm/test_golden.py,tests/dev/test_gen_config.py(if such tests exist; otherwise a snapshot test that the generated workflow contains the docker exec line)
Interfaces:
-
Consumes:
/chatSSE (data: {json}lines; eventslineage,valuation,token,sources,done,error), source dicts withitem_key/p_id/seq/label(Task 5),dev/scripts/nb_issue_filer.py(signature,cmd_report,MARKER,decide) — import it, do not copy it. -
Produces:
- YAML schema, one entry per question:
id,question,mode(default auto),expect_anchors: [{item_key, p_id}](must appear insources),expect_labels: [regex](must be cited in the answer prose, e.g.CY2021 PFS final ¶1578),forbid: [{pattern, unless_label: regex}](e.g.pattern: "\\$\\d",unless_label: "Addendum B"),expect_events: [{code, kind, year}](must be in thelineagepayload),min_eras: 3(distinct rule years among sources). - Seed entries (from #692, anchors verified on the corpus): (1) history of CCM coding and payment →
DE2VH9PD¶1249–1251,YBM4IZUS¶1578,JJ6AM5HJ¶1163; events99490 created 2015,G2058 replaced_by 2021,G0556 created 2025; (2) what replaced G2058 and why →YBM4IZUS¶1578, ¶2369; (3) how do APCM's elements differ from CCM's →JJ6AM5HJ¶1164–1185 vsDE2VH9PD¶1245–1247; (4) when were audio-only E/M codes payable and why did 99441–99443 end →XFGGRBDH¶489, event99441 deleted|disappeared 2025; (5) what did commenters say about G2211 in 2023 vs 2025 → sources from dockets CMS-2023-0121 and CMS-2025-0304 (expect_dockets); (6) does 99490 pass telehealth Steps 1–3 →2KVJ2HKX¶394/¶396/¶398; (7) how did G2064/G2065 → 99424/99426 change the RHC/FQHC G0511 rate →JE7KYBW3¶1100/¶1111,MZ24MX5S¶1305. dev/scripts/llm_golden.py run --url http://llm:8000 --set tests/llm/golden_lineage.yaml --report report.json [--file-issues --source nightly-llm-golden] [--only id]— streams each question, evaluates the checks (check_anchors,check_labels,check_forbidden,check_events,check_eras,check_dockets— pure functions over the collected events, unit-tested), prints a per-question table, exit 1 on any failure, files/sweeps issues throughnb_issue_filerwithsignature(question_id, check_name, detail).tests/llm/test_golden.py: the checkers against a canned SSE transcript (pass and fail cases); YAML loads and every entry has the required keys; a live test skipped unlessLLM_CHAT_URLis set that runs entry (2) and asserts pass.- Workflow: nightly
40 3 * * *after notebooks-integration,docker exec llm uv run python /tmp/golden/llm_golden.py run --url http://localhost:8000 …(copy the script and YAML in withdocker cplike the notebooks job),GITEA_TOKENfromsecrets.DEPLOY_TOKEN, failure filer step identical to the notebooks job.
- YAML schema, one entry per question:
-
Step 1: Failing tests (checkers, YAML shape, gen_config snapshot).
-
Step 2–4: run/implement/run;
uv run python dev/scripts/gen_config.pyregenerates workflows (commit them). -
Step 5: Live (env wrapper; the chat service must be reachable — check
LLM_CHAT_URLor the compose servicehttp://localhost:8000; if unreachable, run the checkers against a transcript captured withstream_answerdirectly and say so):uv run python dev/scripts/llm_golden.py run --url … --set tests/llm/golden_lineage.yaml --report scratch/golden-report.json— report pass/fail per question with the failing checks; do NOT file issues from the worktree run. -
Step 6: Commit
feat(llm): golden longitudinal evaluation — yaml set, runner with anchor/label/forbidden checks, nightly workflow + issue filer (refs #692).
Task 8: Notebook section, docs, tracker
Files:
- Modify:
notebooks/code_families.py(new section "7b. What the chat sees" —lineage_evidenceevents/labels for the picked code, rendered as a table with the same labels the chat cites, plus theprompt_block()text in a collapsible),docs/docs/cli/*(regen),tests/notebooks/test_code_families_nb.py(banner assertion for the new section) - Steps: implement the cell (guarded like sections 5–7; uses the replica only —
lineage_evidencewith a config pointing atdata/replica/aco.ro.duckdb), headless export with and without env,uv run pytest tests/notebooks -q,uv run python docs/scripts/extract_cli.py, commitfeat(notebooks,docs): code_families 7b — the chat's lineage view; CLI docs (refs #691). The controller posts the live numbers on #691/#692 and closes #699 items 1–2 with a comment.
Self-review
Spec coverage. Decision 5 (era-balanced) → T4; Decision 6 (timeline like valuation) → T3 + T6; "lineage_evidence → events + element diffs" → T3; "system prompt: cite the event label" → T3; UI links (P40/P41) → T3 labels/urls + T6; Evaluation section → T7; "ANY PFS-payable code" (user) → T1 (families) + T2 (events for all codes); Ruling A9 (manual source) → T5; #699 items 1–2 → T1. Not in this slice: #699 item 3 (synonym curation beyond the name-phrase rule), #698 elements inversion (T2 inverts events only; elements stay 20 codes — the chat's element diffs therefore cover the fixture families until #698 lands; say so in the notebook cell), #703, #700.
Placeholders. Tasks give signatures, rules, test intents and live checks; implementers are mid-tier models working from prose (allowed by the SDD model-selection rule). Exact SQL for the manual source and era grouping is left to the implementer with the binding style references.
Type consistency. LineageEvidence.payload()["type"] == "lineage" (T3) is what T6 dispatches on and T7 reads; as_source keys added in T5 (item_key, p_id, seq) are what T7's check_anchors matches; ChatRequest.mode (T4) is what T6 posts and T7's YAML mode sends; rule_label (T3) strings are what T7's expect_labels regexes target.