homelab/stack
1
0
Fork 0
Code Issues 20 Pull Requests Actions Packages Projects 1 Releases Wiki Activity
Files
2de806de3206b01afec7a9d138e656c8582ae5f1
stack/docs/superpowers/plans/2026-09-08-llm-chat-code-valuation.md

1254 lines
52 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# LLM Chat Code Valuation Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** When a chat question names a HCPCS/CPT code or a registered code family, the answer keeps its retrieved citations and additionally presents each code's RVUs, conversion factor and national payment by vintage, as a cited prompt block, a `valuation` SSE event, and a rendered table.
**Architecture:** Deterministic detection (`pfs/families.py`) runs before generation; `pfs/valuation.py` reads the DuckDB read replica and attaches conversion factors from `pfs.rules`; `llm/evidence.py` turns rows into a prompt block, a UI payload, and pulls "code-cited" rule chunks from pgvector via a `codes` metadata list stamped at index time; `llm/rag.py` merges those sources, adds the block to the prompt and emits the event; `chat.html` renders the table. The `llm` container gets a read-only `./data` mount.
**Tech Stack:** Python 3.13, `uv run --no-sync`, pytest, DuckDB (in-memory fixtures in tests), SQLAlchemy `text()` against pgvector's `langchain_pg_embedding`, FastAPI SSE, vanilla JS in `chat.html`.
**Spec:** `docs/superpowers/specs/2026-09-08-llm-chat-code-valuation-design.md`
## Global Constraints
- Tests: `uv run --no-sync pytest <path> -q -p no:cacheprovider`; output pristine. The pre-commit hook runs ruff + the cli suite.
- Commit messages: conventional prefix, `(refs P48)`; **no `Co-Authored-By` trailer**. `git status --short` before every commit; stage only your own files (shared worktree).
- Nothing under `src/pfs/families.py`, `src/pfs/valuation.py`, `src/llm/evidence.py` may import `narwhals`, `pfs.calcs`, `pfs.pipe`, `aco`, or `conf.connect.ducklake` — the chat image lacks narwhals. `pfs/__init__.py` has no imports, so `pfs.rules` / `pfs.nprm` / `pfs.families` / `pfs.valuation` are safe.
- Chunk metadata stays `dict[str, str]`: the new `codes` value is a space-joined string (`"G0556 G0557"`), `""` when none.
- Citation labels are square-bracketed: final `[PFS CY2026 Addendum B]`, proposed `[CY2027 NPRM Addendum B]`. FR citation URL: `https://www.federalregister.gov/citation/<vol>-FR-<page>` built from a `"90 FR 49266"` string.
- Conversion factors come only from `pfs.rules.RULES[year].conversion_factor` (final) and `pfs.rules.proposed_for(year).conversion_factor` (proposed); never from `pfs.rvu.conv_factor`.
- National unadjusted payment = `round(total_rvu * cf, 2)`; no GPCI, no OPPS.
- Event order: `valuation` (only when codes detected) → `token`* → `sources` → `done`. With no codes detected, the prompt and event sequence are byte-identical to today.
- Failures in evidence (replica missing, pgvector error) log a warning and drop the evidence; the chat never fails because of it.
## File Structure
| File | Responsibility |
|---|---|
| `src/pfs/families.py` (new) | `Family`, `FAMILIES`, `CODE_RE`, `find_codes(text)`, `detect_codes(text) -> Detection` — pure |
| `src/pfs/valuation.py` (new) | `ValuationRow`, `valuation(con, codes, *, years)`, `vintage_label/citation/url` helpers — DuckDB SQL + `pfs.rules` |
| `src/llm/links.py` | `as_source(md, text, score) -> dict` (moved from `rag._source`) |
| `src/llm/chunk.py` | stamp `codes` metadata on every chunk |
| `src/llm/config.py` + `stack.toml` | `duckdb_replica`, `valuation_years`, `code_cited_per_code` |
| `src/llm/evidence.py` (new) | `ValuationEvidence`, `valuation_evidence(question, cfg)`, `code_cited_sources(engine, codes, *, per_code, collection)` |
| `src/llm/rag.py` | system prompt addendum, `build_messages(..., evidence=)`, `stream_answer` merge + event |
| `src/llm/web/chat.html` | `renderValuation` + styles |
| `compose.yml` | `llm` mount + env |
| `tests/pfs/test_families.py`, `tests/pfs/test_valuation.py`, `tests/llm/test_evidence.py` (new); `tests/llm/test_chunk.py`, `test_config.py`, `test_rag.py`, `test_api.py`, `test_links.py` (extend) | |
---
### Task 1: `pfs/families.py` — registry and detection
**Files:**
- Create: `src/pfs/families.py`
- Test: `tests/pfs/test_families.py`
**Interfaces:**
- Produces: `CODE_RE` (compiled), `find_codes(text) -> tuple[str, ...]` (sorted unique codes literally present), `Family(key, name, codes, synonyms)`, `FAMILIES: dict[str, Family]`, `Detection(codes, families, explicit)`, `detect_codes(text) -> Detection`, `family_of(code) -> Family | None`.
- [ ] **Step 1: Write the failing tests**
```python
# tests/pfs/test_families.py
"""pfs.families — code-family registry + deterministic code detection."""
from __future__ import annotations
from pfs.families import FAMILIES, Detection, detect_codes, family_of, find_codes
class TestFindCodes:
def test_hcpcs_and_cpt(self):
assert find_codes("Codes G0556 and 99490 apply; see g0557.") == ("99490", "G0556")
def test_no_false_hits_on_years_and_fr_pages(self):
# 2026 (4 digits) and 43842 (5 digits) — the CPT regex will match 43842;
# validation against the fee schedule happens at lookup, not here.
assert find_codes("91 FR 43842, CY2026") == ("43842",)
def test_dedupes_and_sorts(self):
assert find_codes("G0558 G0556 G0558") == ("G0556", "G0558")
def test_empty(self):
assert find_codes("") == ()
class TestRegistry:
def test_apcm_family(self):
f = FAMILIES["APCM"]
assert f.codes == ("G0556", "G0557", "G0558")
assert "advanced primary care management" in f.synonyms
assert family_of("G0557") is f
assert family_of("00000") is None
def test_all_families_present(self):
assert set(FAMILIES) == {"ACP", "CCM", "PCM", "TCM", "APCM"}
class TestDetectCodes:
def test_family_name_expands_to_codes(self):
d = detect_codes("What is APCM and how is it valued?")
assert d == Detection(codes=("G0556", "G0557", "G0558"), families=("APCM",), explicit=())
def test_synonym_case_insensitive_word_boundary(self):
d = detect_codes("Tell me about Advanced Primary Care Management.")
assert d.families == ("APCM",)
assert detect_codes("the apcmx code").families == ()
def test_single_member_code_expands_family(self):
d = detect_codes("How much does G0557 pay?")
assert d.explicit == ("G0557",)
assert d.families == ("APCM",)
assert d.codes == ("G0556", "G0557", "G0558")
def test_unregistered_code_stays_alone(self):
d = detect_codes("value of 99213")
assert d == Detection(codes=("99213",), families=(), explicit=("99213",))
def test_multiple_families(self):
d = detect_codes("compare CCM and TCM")
assert d.families == ("CCM", "TCM")
assert d.codes == tuple(sorted(FAMILIES["CCM"].codes + FAMILIES["TCM"].codes))
def test_nothing(self):
assert detect_codes("what did commenters say about telehealth?") == Detection((), (), ())
```
- [ ] **Step 2: Run to verify failure**
Run: `uv run --no-sync pytest tests/pfs/test_families.py -q -p no:cacheprovider`
Expected: FAIL — `ModuleNotFoundError: No module named 'pfs.families'`
- [ ] **Step 3: Implement**
```python
# src/pfs/families.py
"""Code families and deterministic code detection for chat questions.
The registry used to live in a notebook cell
(``notebooks/palliative_care_rfi.py``); this is its home in ``src`` so
the chat, notebooks and scripts share one list. Pure: no I/O, no
DuckDB, no narwhals — it must import inside the ``llm`` container.
"""
from __future__ import annotations
import re
from dataclasses import dataclass
# HCPCS level II (letter + 4 digits) or CPT (5 digits). Validation against
# the fee schedule happens at lookup time (pfs.valuation), not here.
CODE_RE = re.compile(r"\b(?:[A-Z]\d{4}|\d{5})\b")
def find_codes(text: str) -> tuple[str, ...]:
"""Sorted unique codes literally present in *text* (case-normalised)."""
return tuple(sorted({m.group(0).upper() for m in CODE_RE.finditer(text.upper())}))
@dataclass(frozen=True)
class Family:
key: str
name: str
codes: tuple[str, ...]
synonyms: tuple[str, ...] # lower-case; matched on word boundaries
FAMILIES: dict[str, Family] = {
"ACP": Family(
"ACP", "Advance Care Planning", ("99497", "99498"),
("acp", "advance care planning"),
),
"CCM": Family(
"CCM", "Chronic Care Management",
("99437", "99439", "99487", "99489", "99490", "99491"),
("ccm", "chronic care management"),
),
"PCM": Family(
"PCM", "Principal Care Management", ("99424", "99425", "99426", "99427"),
("pcm", "principal care management"),
),
"TCM": Family(
"TCM", "Transitional Care Management", ("99495", "99496"),
("tcm", "transitional care management"),
),
"APCM": Family(
"APCM", "Advanced Primary Care Management", ("G0556", "G0557", "G0558"),
("apcm", "advanced primary care management"),
),
}
_CODE_TO_FAMILY: dict[str, Family] = {c: f for f in FAMILIES.values() for c in f.codes}
def family_of(code: str) -> Family | None:
return _CODE_TO_FAMILY.get(code.upper())
@dataclass(frozen=True)
class Detection:
codes: tuple[str, ...] # sorted unique: explicit + family-expanded
families: tuple[str, ...] # family keys, sorted
explicit: tuple[str, ...] # codes literally present in the text
def detect_codes(text: str) -> Detection:
"""Codes a question is about: explicit codes plus every code of any
family named (by synonym) or touched (by one member code)."""
explicit = find_codes(text)
lowered = text.lower()
families: set[str] = set()
for fam in FAMILIES.values():
if any(re.search(rf"\b{re.escape(s)}\b", lowered) for s in fam.synonyms):
families.add(fam.key)
for code in explicit:
fam = family_of(code)
if fam is not None:
families.add(fam.key)
codes = set(explicit)
for key in families:
codes.update(FAMILIES[key].codes)
return Detection(
codes=tuple(sorted(codes)),
families=tuple(sorted(families)),
explicit=explicit,
)
```
- [ ] **Step 4: Run to verify pass**
Run: `uv run --no-sync pytest tests/pfs/test_families.py -q -p no:cacheprovider`
Expected: 12 passed
- [ ] **Step 5: Commit**
```bash
git add src/pfs/families.py tests/pfs/test_families.py
git commit -m "feat(pfs): code-family registry + deterministic code detection (refs P48)"
```
---
### Task 2: `pfs/valuation.py` — rows by vintage from the replica
**Files:**
- Create: `src/pfs/valuation.py`
- Test: `tests/pfs/test_valuation.py`
**Interfaces:**
- Consumes: `pfs.rules.RULES`, `pfs.rules.proposed_for`, `pfs.nprm.NPRM_SOURCES`.
- Produces: `ValuationRow` (frozen dataclass; fields below), `valuation(con, codes, *, years=4) -> tuple[list[ValuationRow], list[str]]` (rows ordered by code then year, proposed last per code; second element = codes with no rows), `fr_citation_url(cite) -> str`, `vintage_label(year, proposed) -> str`.
- [ ] **Step 1: Write the failing tests**
```python
# tests/pfs/test_valuation.py
"""pfs.valuation — RVUs + national payment by vintage from a DuckDB replica."""
from __future__ import annotations
import duckdb
import pytest
from pfs.rules import RULES, proposed_for
from pfs.valuation import ValuationRow, fr_citation_url, valuation, vintage_label
RVU_COLS = (
"hcpcs VARCHAR, mod VARCHAR, description VARCHAR, status_code VARCHAR, "
"work_rvu DOUBLE, non_fac_pe_rvu DOUBLE, fac_pe_rvu DOUBLE, mp_rvu DOUBLE, "
"non_fac_total DOUBLE, fac_total DOUBLE, conv_factor DOUBLE, year INTEGER"
)
PROPOSED_COLS = (
"hcpcs VARCHAR, mod VARCHAR, description VARCHAR, status_code VARCHAR, "
"work_rvu DOUBLE, non_fac_pe_rvu DOUBLE, fac_pe_rvu DOUBLE, mp_rvu DOUBLE, cms_rule_id VARCHAR"
)
@pytest.fixture
def con():
c = duckdb.connect(":memory:")
c.execute("CREATE SCHEMA pfs")
c.execute(f"CREATE TABLE pfs.rvu ({RVU_COLS})")
c.execute(f"CREATE TABLE pfs.rvu_proposed ({PROPOSED_COLS})")
rows = [
# G0556: 2025 + 2026 final, with a modifier row that must be ignored
("G0556", None, "Adv prim care mgmt lvl 1", "A", 0.25, 0.20, 0.10, 0.02, 0.47, 0.37, 32.3465, 2025),
("G0556", None, "Adv prim care mgmt lvl 1", "A", 0.25, 0.22, 0.06, 0.02, 0.49, 0.33, 33.4009, 2026),
("G0556", "26", "Adv prim care mgmt lvl 1", "A", 9.0, 9.0, 9.0, 9.0, 27.0, 27.0, 33.4009, 2026),
# 99490: six years so the last-N cut applies (2021..2026)
*[("99490", "", "Chrnc care mgmt srvc 20 min", "A", 1.0, 1.0, 0.5, 0.05, 2.05, 1.55, 30.0, y) for y in range(2021, 2027)],
# I-status code with NULL components
("G9999", None, "Not priced", "I", None, None, None, None, None, None, 33.4009, 2026),
]
c.executemany("INSERT INTO pfs.rvu VALUES (?,?,?,?,?,?,?,?,?,?,?,?)", rows)
c.executemany(
"INSERT INTO pfs.rvu_proposed VALUES (?,?,?,?,?,?,?,?,?)",
[
("G0556", None, "Adv prim care mgmt lvl 1", "A", 0.25, 0.23, 0.07, 0.02, "CMS-1848-P"),
("G0556", "26", "Adv prim care mgmt lvl 1", "A", 9.0, 9.0, 9.0, 9.0, "CMS-1848-P"),
("G0556", None, "older nprm", "A", 0.1, 0.1, 0.1, 0.1, "CMS-1832-P"),
],
)
yield c
c.close()
class TestHelpers:
def test_vintage_label(self):
assert vintage_label(2026, False) == "[PFS CY2026 Addendum B]"
assert vintage_label(2027, True) == "[CY2027 NPRM Addendum B]"
def test_fr_citation_url(self):
assert fr_citation_url("90 FR 49266") == "https://www.federalregister.gov/citation/90-FR-49266"
assert fr_citation_url("") == ""
class TestValuation:
def test_final_and_proposed_rows_with_cf_and_payment(self, con):
rows, unpriced = valuation(con, ["G0556"], years=4)
assert unpriced == []
assert [(r.year, r.proposed) for r in rows] == [(2025, False), (2026, False), (2027, True)]
r26 = rows[1]
assert isinstance(r26, ValuationRow)
assert (r26.code, r26.status, r26.work, r26.pe_nf, r26.pe_f, r26.mp) == ("G0556", "A", 0.25, 0.22, 0.06, 0.02)
assert (r26.total_nf, r26.total_f) == (0.49, 0.33)
assert r26.cf == RULES[2026].conversion_factor
assert r26.pay_nf == round(0.49 * RULES[2026].conversion_factor, 2)
assert r26.pay_f == round(0.33 * RULES[2026].conversion_factor, 2)
assert r26.label == "[PFS CY2026 Addendum B]"
assert r26.citation == RULES[2026].federal_register_citation
assert r26.url == fr_citation_url(RULES[2026].federal_register_citation)
assert r26.vintage == "CY2026 final"
p = rows[2]
assert p.label == "[CY2027 NPRM Addendum B]" and p.vintage == "CY2027 proposed"
assert p.cf == proposed_for(2027).conversion_factor
assert p.total_nf == round(0.25 + 0.23 + 0.02, 4) and p.total_f == round(0.25 + 0.07 + 0.02, 4)
assert p.pay_nf == round(p.total_nf * p.cf, 2)
assert p.citation == proposed_for(2027).federal_register_citation
def test_modifier_rows_are_ignored(self, con):
rows, _ = valuation(con, ["G0556"])
assert all(r.work == 0.25 for r in rows)
def test_last_n_years_only(self, con):
rows, _ = valuation(con, ["99490"], years=4)
assert [r.year for r in rows if not r.proposed] == [2023, 2024, 2025, 2026]
def test_unpriced_codes_reported(self, con):
rows, unpriced = valuation(con, ["G0556", "Z9999"])
assert unpriced == ["Z9999"]
assert {r.code for r in rows} == {"G0556"}
def test_null_components_pass_through(self, con):
rows, unpriced = valuation(con, ["G9999"])
assert unpriced == []
(r,) = rows
assert r.status == "I" and r.work is None and r.pay_nf is None
def test_order_is_code_then_year(self, con):
rows, _ = valuation(con, ["G0556", "99490"], years=2)
assert [(r.code, r.year) for r in rows] == [
("99490", 2025), ("99490", 2026), ("G0556", 2025), ("G0556", 2026), ("G0556", 2027)
]
```
- [ ] **Step 2: Run to verify failure**
Run: `uv run --no-sync pytest tests/pfs/test_valuation.py -q -p no:cacheprovider`
Expected: FAIL — `ModuleNotFoundError: No module named 'pfs.valuation'`
- [ ] **Step 3: Implement**
```python
# src/pfs/valuation.py
"""RVUs, conversion factor and national unadjusted payment per code and
vintage, read from the DuckDB replica (``pfs.rvu`` final years,
``pfs.rvu_proposed`` for the newest NPRM). Conversion factors come from
``pfs.rules`` — never from ``pfs.rvu.conv_factor``. Pure DuckDB SQL; no
narwhals (this module runs inside the ``llm`` container).
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Any, Sequence
from pfs.nprm import NPRM_SOURCES
from pfs.rules import RULES, proposed_for
@dataclass(frozen=True)
class ValuationRow:
code: str
description: str
vintage: str # "CY2026 final" | "CY2027 proposed"
year: int
proposed: bool
status: str
work: float | None
pe_nf: float | None
pe_f: float | None
mp: float | None
total_nf: float | None
total_f: float | None
cf: float
pay_nf: float | None
pay_f: float | None
label: str # "[PFS CY2026 Addendum B]"
citation: str # "90 FR 49266"
url: str
def vintage_label(year: int, proposed: bool) -> str:
return f"[CY{year} NPRM Addendum B]" if proposed else f"[PFS CY{year} Addendum B]"
def fr_citation_url(cite: str) -> str:
parts = cite.split()
if len(parts) != 3 or parts[1].upper() != "FR":
return ""
return f"https://www.federalregister.gov/citation/{parts[0]}-FR-{parts[2]}"
def _pay(total: float | None, cf: float) -> float | None:
return None if total is None else round(total * cf, 2)
_FINAL_SQL = """
WITH base AS (
SELECT hcpcs, year, description, status_code, work_rvu, non_fac_pe_rvu,
fac_pe_rvu, mp_rvu, non_fac_total, fac_total
FROM pfs.rvu
WHERE hcpcs IN (SELECT UNNEST(?::VARCHAR[]))
QUALIFY row_number() OVER (
PARTITION BY hcpcs, year ORDER BY (mod IS NULL OR mod = '') DESC, mod
) = 1
)
SELECT * FROM base
QUALIFY dense_rank() OVER (PARTITION BY hcpcs ORDER BY year DESC) <= ?
ORDER BY hcpcs, year
"""
_PROPOSED_SQL = """
SELECT hcpcs, description, status_code, work_rvu, non_fac_pe_rvu, fac_pe_rvu, mp_rvu
FROM pfs.rvu_proposed
WHERE cms_rule_id = ? AND hcpcs IN (SELECT UNNEST(?::VARCHAR[]))
QUALIFY row_number() OVER (
PARTITION BY hcpcs ORDER BY (mod IS NULL OR mod = '') DESC, mod
) = 1
ORDER BY hcpcs
"""
def _sum(*vals: float | None) -> float | None:
if any(v is None for v in vals):
return None
return round(sum(vals), 4) # type: ignore[arg-type]
def valuation(
con: Any, codes: Sequence[str], *, years: int = 4
) -> tuple[list[ValuationRow], list[str]]:
"""Rows for *codes* (last *years* final vintages + the newest NPRM),
ordered by code then year with the proposed row last per code, and the
codes that produced no rows at all."""
wanted = sorted({c.upper() for c in codes})
if not wanted:
return [], []
by_code: dict[str, list[ValuationRow]] = {c: [] for c in wanted}
for hcpcs, year, desc, status, work, pe_nf, pe_f, mp, tot_nf, tot_f in con.execute(
_FINAL_SQL, [wanted, years]
).fetchall():
rule = RULES.get(int(year))
if rule is None:
continue
cf = rule.conversion_factor
cite = rule.federal_register_citation
by_code[hcpcs].append(
ValuationRow(
code=hcpcs, description=desc or "", vintage=f"CY{year} final",
year=int(year), proposed=False, status=status or "",
work=work, pe_nf=pe_nf, pe_f=pe_f, mp=mp,
total_nf=tot_nf, total_f=tot_f, cf=cf,
pay_nf=_pay(tot_nf, cf), pay_f=_pay(tot_f, cf),
label=vintage_label(int(year), False), citation=cite,
url=fr_citation_url(cite),
)
)
nprm_year, _tag, rule_id, _pin = max(NPRM_SOURCES, key=lambda s: s[0])
prop = proposed_for(nprm_year)
if prop is not None:
cf = prop.conversion_factor
cite = prop.federal_register_citation
for hcpcs, desc, status, work, pe_nf, pe_f, mp in con.execute(
_PROPOSED_SQL, [rule_id, wanted]
).fetchall():
tot_nf, tot_f = _sum(work, pe_nf, mp), _sum(work, pe_f, mp)
by_code[hcpcs].append(
ValuationRow(
code=hcpcs, description=desc or "", vintage=f"CY{nprm_year} proposed",
year=nprm_year, proposed=True, status=status or "",
work=work, pe_nf=pe_nf, pe_f=pe_f, mp=mp,
total_nf=tot_nf, total_f=tot_f, cf=cf,
pay_nf=_pay(tot_nf, cf), pay_f=_pay(tot_f, cf),
label=vintage_label(nprm_year, True), citation=cite,
url=fr_citation_url(cite),
)
)
rows = [r for c in wanted for r in sorted(by_code[c], key=lambda r: (r.year, r.proposed))]
unpriced = [c for c in wanted if not by_code[c]]
return rows, unpriced
```
Note: DuckDB accepts a Python list for a `?::VARCHAR[]` parameter; `UNNEST` turns it into rows for `IN`.
- [ ] **Step 4: Run to verify pass**
Run: `uv run --no-sync pytest tests/pfs/test_valuation.py -q -p no:cacheprovider`
Expected: 8 passed
- [ ] **Step 5: Commit**
```bash
git add src/pfs/valuation.py tests/pfs/test_valuation.py
git commit -m "feat(pfs): valuation rows by vintage from the DuckDB replica with pfs.rules conversion factors (refs P48)"
```
---
### Task 3: `codes` chunk metadata + `links.as_source`
**Files:**
- Modify: `src/llm/chunk.py` (`chunk_doc`, `_chunk_paragraphs`)
- Modify: `src/llm/links.py` (add `as_source`), `src/llm/rag.py` (`_source` delegates)
- Test: `tests/llm/test_chunk.py`, `tests/llm/test_links.py`
**Interfaces:**
- Consumes: `pfs.families.find_codes`.
- Produces: every `Chunk.metadata["codes"]` = space-joined sorted codes in that chunk's text (`""` when none); `llm.links.as_source(md: dict[str,str], text: str, score: float) -> dict` returning the same dict shape `rag._source` returns today (`id,label,kind,url,title,date,docket,comment_id,snippet,score`).
- [ ] **Step 1: Write the failing tests**
Append to `tests/llm/test_chunk.py`:
```python
class TestCodesMetadata:
def test_codes_stamped_per_chunk(self):
text = "We propose G0556 and G0557.\n\n## Other\n\nNo codes here.\n\n## More\n\n99490 applies."
chunks = chunk_doc(_doc(text), target_chars=60, overlap_chars=10)
found = {c.metadata["codes"] for c in chunks}
assert "G0556 G0557" in found
assert "99490" in found
assert "" in found
def test_rule_paragraph_chunks_get_codes(self):
from llm.chunk import Paragraph
doc = Doc(
key="R1", text="x",
metadata={"kind": "rule"},
paragraphs=(Paragraph(1, 100, 1, "APCM code G0556 is valued at 0.25 work RVUs."),),
)
(chunk,) = chunk_doc(doc)
assert chunk.metadata["codes"] == "G0556"
```
Append to `tests/llm/test_links.py`:
```python
class TestAsSource:
def test_shape_matches_rag_source(self):
from llm.links import as_source
s = as_source({"kind": "comment", "comment_id": "CMS-2026-2377-1", "date": "2026-08-19", "title": "t", "docket": "CMS-2026-2377"}, "body " * 200, 0.123456)
assert set(s) == {"id", "label", "kind", "url", "title", "date", "docket", "comment_id", "snippet", "score"}
assert s["label"] == "CMS-2026-2377-1" and s["kind"] == "comment"
assert len(s["snippet"]) <= 500 and s["score"] == 0.1235
```
- [ ] **Step 2: Run to verify failure**
Run: `uv run --no-sync pytest tests/llm/test_chunk.py tests/llm/test_links.py -q -p no:cacheprovider`
Expected: FAIL — `KeyError: 'codes'` and `ImportError: cannot import name 'as_source'`.
- [ ] **Step 3: Implement**
In `src/llm/chunk.py`: add `from pfs.families import find_codes` at the top; in both chunk builders add `"codes": " ".join(find_codes(piece))` (comment path, using `piece`) and `"codes": " ".join(find_codes(text))` (paragraph path) to the metadata dicts after `"section"`.
In `src/llm/links.py` add:
```python
_SNIPPET_CHARS = 500
def as_source(md: dict[str, str], text: str, score: float) -> dict:
"""The source dict the chat sends to the prompt and the UI."""
snippet = text[:_SNIPPET_CHARS].strip()
url, label = for_source(md, snippet)
return {
"id": label,
"label": label,
"kind": md.get("kind", ""),
"url": url,
"title": md.get("title", ""),
"date": md.get("date", ""),
"docket": md.get("docket", ""),
"comment_id": md.get("comment_id", ""),
"snippet": snippet,
"score": round(score, 4),
}
```
In `src/llm/rag.py`: `_source(hit)` becomes `return as_source(hit.metadata, hit.text, hit.score)` (import `as_source` from `llm.links`; delete the local `_SNIPPET_CHARS` if unused).
- [ ] **Step 4: Run to verify pass**
Run: `uv run --no-sync pytest tests/llm -q -p no:cacheprovider`
Expected: pass (existing `test_rag` retrieve tests still assert the same source shape).
- [ ] **Step 5: Commit**
```bash
git add src/llm/chunk.py src/llm/links.py src/llm/rag.py tests/llm/test_chunk.py tests/llm/test_links.py
git commit -m "feat(llm): stamp codes metadata on every chunk; links.as_source shared source dict (refs P48)"
```
---
### Task 4: Config knobs
**Files:**
- Modify: `src/llm/config.py` (`LlmConfig`, `load`), `stack.toml` (`[llm]`)
- Test: `tests/llm/test_config.py`
**Interfaces:**
- Produces: `LlmConfig.duckdb_replica: str` (default `"data/aco.ro.duckdb"`, env `LLM_DUCKDB_REPLICA` overrides), `valuation_years: int` (default 4), `code_cited_per_code: int` (default 3).
- [ ] **Step 1: Write the failing tests** (append to `TestLoad`)
```python
def test_valuation_knobs_defaults(self, monkeypatch):
monkeypatch.delenv("LLM_DUCKDB_REPLICA", raising=False)
cfg = llm_config.load()
assert cfg.duckdb_replica == "data/aco.ro.duckdb"
assert cfg.valuation_years == 4
assert cfg.code_cited_per_code == 3
def test_duckdb_replica_env_override(self, monkeypatch):
monkeypatch.setenv("LLM_DUCKDB_REPLICA", "/app/data/aco.ro.duckdb")
assert llm_config.load().duckdb_replica == "/app/data/aco.ro.duckdb"
```
- [ ] **Step 2: Run to verify failure** — `uv run --no-sync pytest tests/llm/test_config.py -q -p no:cacheprovider` → FAIL (`AttributeError: duckdb_replica`).
- [ ] **Step 3: Implement**
`LlmConfig` gains (after `top_n`):
```python
duckdb_replica: str = "data/aco.ro.duckdb"
valuation_years: int = 4
code_cited_per_code: int = 3
```
`load()` passes:
```python
duckdb_replica=os.environ.get("LLM_DUCKDB_REPLICA") or str(_opt(section, "duckdb_replica", "data/aco.ro.duckdb")),
valuation_years=int(_opt(section, "valuation_years", 4)),
code_cited_per_code=int(_opt(section, "code_cited_per_code", 3)),
```
Add to the module docstring env contract: `LLM_DUCKDB_REPLICA beats [llm].duckdb_replica (containers set /app/data/aco.ro.duckdb)`. In `stack.toml` `[llm]` add:
```toml
duckdb_replica = "data/aco.ro.duckdb" # read-only DuckDB replica the chat reads valuations from
valuation_years = 4 # final-rule vintages shown per code (plus the newest NPRM)
code_cited_per_code = 3 # rule paragraphs literally citing each detected code
```
Also update `tests/llm/test_rag.py`'s `CFG` and any other `LlmConfig(...)` literal in tests only if they break (they use defaults, so they should not).
- [ ] **Step 4: Run to verify pass** — `uv run --no-sync pytest tests/llm/test_config.py tests/llm/test_rag.py -q -p no:cacheprovider` → pass.
- [ ] **Step 5: Commit**
```bash
git add src/llm/config.py stack.toml tests/llm/test_config.py
git commit -m "feat(llm): duckdb_replica / valuation_years / code_cited_per_code config (refs P48)"
```
---
### Task 5: `llm/evidence.py` — evidence block, payload, code-cited sources
**Files:**
- Create: `src/llm/evidence.py`
- Test: `tests/llm/test_evidence.py`
**Interfaces:**
- Consumes: `pfs.families.detect_codes`, `pfs.valuation.valuation/ValuationRow`, `llm.links.as_source`, `llm.index._engine`, `LlmConfig.duckdb_replica/valuation_years`.
- Produces:
- `ValuationEvidence(codes, families, rows, unpriced)` with `prompt_block() -> str` and `payload() -> dict`.
- `valuation_evidence(question: str, cfg: LlmConfig) -> ValuationEvidence | None` (None when no codes detected or the replica cannot be opened).
- `code_cited_sources(engine, codes: Sequence[str], *, per_code: int, collection: str = "rules") -> list[dict]`.
- `merge_sources(retrieved: list[dict], extra: list[dict]) -> list[dict]` (retrieved order kept; extras appended unless their `label` already present).
- [ ] **Step 1: Write the failing tests**
```python
# tests/llm/test_evidence.py
"""llm.evidence — valuation prompt block, SSE payload, code-cited rule sources."""
from __future__ import annotations
from unittest.mock import MagicMock, patch
from llm.config import LlmConfig
from llm.evidence import ValuationEvidence, code_cited_sources, merge_sources, valuation_evidence
from pfs.valuation import ValuationRow
CFG = LlmConfig(
ollama_hosts=("http://h1:11434",), embed_model="e", instruct_model="c", embed_dim=768,
build_ann_index=False, pg_host="x", pg_port=5432, pg_db="llm", pg_user="llm",
duckdb_replica="/nonexistent/aco.ro.duckdb", valuation_years=4, code_cited_per_code=2,
)
def _row(**kw) -> ValuationRow:
base = dict(
code="G0556", description="Adv prim care mgmt lvl 1", vintage="CY2026 final", year=2026,
proposed=False, status="A", work=0.25, pe_nf=0.22, pe_f=0.06, mp=0.02, total_nf=0.49,
total_f=0.33, cf=33.4009, pay_nf=16.37, pay_f=11.02, label="[PFS CY2026 Addendum B]",
citation="90 FR 49266", url="https://www.federalregister.gov/citation/90-FR-49266",
)
base.update(kw)
return ValuationRow(**base)
class TestEvidenceText:
def test_prompt_block_exact(self):
ev = ValuationEvidence(codes=("G0556",), families=("APCM",), rows=(_row(),), unpriced=("Z9999",))
assert ev.prompt_block() == (
"Valuation (authoritative for RVUs, conversion factors and payments; "
"national unadjusted, no GPCI):\n"
'[PFS CY2026 Addendum B] G0556 "Adv prim care mgmt lvl 1" — status A; work 0.25; '
"PE 0.22 non-fac / 0.06 fac; MP 0.02; total 0.49 non-fac / 0.33 fac; "
"CF $33.4009 → payment $16.37 non-fac / $11.02 fac\n"
"Not priced in the indexed fee schedule: Z9999"
)
def test_prompt_block_null_components(self):
ev = ValuationEvidence(("G9999",), (), (_row(code="G9999", status="I", work=None, pe_nf=None, pe_f=None, mp=None, total_nf=None, total_f=None, pay_nf=None, pay_f=None),), ())
line = ev.prompt_block().splitlines()[1]
assert "status I; work n/a; PE n/a non-fac / n/a fac" in line and "payment n/a" in line
def test_payload_shape(self):
ev = ValuationEvidence(("G0556",), ("APCM",), (_row(), _row(year=2027, proposed=True, vintage="CY2027 proposed", label="[CY2027 NPRM Addendum B]", citation="91 FR 43842", url="https://www.federalregister.gov/citation/91-FR-43842")), ())
p = ev.payload()
assert p["type"] == "valuation" and p["codes"] == ["G0556"] and p["families"] == ["APCM"]
assert p["rows"][0]["pay_nf"] == 16.37 and p["rows"][1]["proposed"] is True
assert p["provenance"] == [
{"label": "[PFS CY2026 Addendum B]", "vintage": "CY2026 final", "citation": "90 FR 49266", "url": "https://www.federalregister.gov/citation/90-FR-49266"},
{"label": "[CY2027 NPRM Addendum B]", "vintage": "CY2027 proposed", "citation": "91 FR 43842", "url": "https://www.federalregister.gov/citation/91-FR-43842"},
]
assert p["unpriced"] == []
class TestValuationEvidence:
def test_none_when_no_codes(self):
assert valuation_evidence("what did commenters say about telehealth?", CFG) is None
@patch("llm.evidence.valuation", return_value=([_row()], []))
@patch("llm.evidence.duckdb.connect")
def test_detects_and_queries_replica_read_only(self, mock_connect, mock_val):
ev = valuation_evidence("How is APCM valued?", CFG)
mock_connect.assert_called_once_with("/nonexistent/aco.ro.duckdb", read_only=True)
assert mock_val.call_args.args[1] == ["G0556", "G0557", "G0558"]
assert mock_val.call_args.kwargs == {"years": 4}
assert ev is not None and ev.codes == ("G0556", "G0557", "G0558") and ev.families == ("APCM",)
mock_connect.return_value.close.assert_called_once()
@patch("llm.evidence.duckdb.connect", side_effect=OSError("no replica"))
def test_unopenable_replica_yields_none(self, _c, caplog):
assert valuation_evidence("G0556?", CFG) is None
assert "valuation evidence skipped" in caplog.text
def _engine(rows):
engine = MagicMock()
conn = engine.begin.return_value.__enter__.return_value
conn.execute.return_value.fetchall.return_value = rows
return engine, conn
def _rule_md(p_id, codes, date="2024-12-09"):
return {"kind": "rule", "item_key": "R1", "p_id": str(p_id), "page": "97710", "ordinal": "1",
"html_url": "https://www.federalregister.gov/d/2024-25382", "fr_volume": "89",
"date": date, "title": "CY2025 PFS final rule", "codes": codes}
class TestCodeCitedSources:
def test_sql_and_per_code_cap_and_dedupe(self):
engine, conn = _engine([
("para one G0556", _rule_md(1, "G0556")),
("para two G0556", _rule_md(2, "G0556")),
("para three G0556", _rule_md(3, "G0556")), # over the cap for G0556
("para four G0557", _rule_md(4, "G0557 G0556")), # G0557 still under cap
("para one again", _rule_md(1, "G0556")), # duplicate (item_key, p_id)
])
out = code_cited_sources(engine, ["G0556", "G0557"], per_code=2)
sql = str(conn.execute.call_args.args[0])
assert "string_to_array" in sql and "langchain_pg_collection" in sql
assert conn.execute.call_args.args[1]["collection"] == "rules"
assert conn.execute.call_args.args[1]["codes"] == ["G0556", "G0557"]
assert [s["snippet"] for s in out] == ["para one G0556", "para two G0556", "para four G0557"]
assert all(s["kind"] == "rule" and s["score"] == 0.0 for s in out)
def test_engine_error_yields_empty(self, caplog):
engine = MagicMock()
engine.begin.side_effect = RuntimeError("pg down")
assert code_cited_sources(engine, ["G0556"], per_code=2) == []
assert "code-cited sources skipped" in caplog.text
class TestMergeSources:
def test_keeps_order_and_dedupes_by_label(self):
a = [{"label": "X", "score": 0.9}, {"label": "Y", "score": 0.8}]
b = [{"label": "Y", "score": 0.0}, {"label": "Z", "score": 0.0}]
assert [s["label"] for s in merge_sources(a, b)] == ["X", "Y", "Z"]
assert merge_sources(a, b)[1]["score"] == 0.8
```
- [ ] **Step 2: Run to verify failure**
Run: `uv run --no-sync pytest tests/llm/test_evidence.py -q -p no:cacheprovider`
Expected: FAIL — `ModuleNotFoundError: No module named 'llm.evidence'`
- [ ] **Step 3: Implement**
```python
# src/llm/evidence.py
"""Structured evidence for the chat: code valuations beside the excerpts.
``valuation_evidence`` detects HCPCS/CPT codes (and registered families)
in the question, reads their RVUs from the DuckDB read replica and
attaches conversion factors from ``pfs.rules``. ``code_cited_sources``
pulls rule chunks whose ``codes`` metadata mention those codes straight
from pgvector so the answer can cite where CMS valued them. Both degrade
to "no evidence" with a warning — the chat never fails because of them.
"""
from __future__ import annotations
import logging
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Any, Sequence
import duckdb
from sqlalchemy import text
from llm.config import LlmConfig
from llm.links import as_source
from pfs.families import detect_codes
from pfs.valuation import ValuationRow, valuation
log = logging.getLogger(__name__)
_HEADER = (
"Valuation (authoritative for RVUs, conversion factors and payments; "
"national unadjusted, no GPCI):"
)
def _n(v: float | None, digits: int = 2, money: bool = False) -> str:
if v is None:
return "n/a"
s = f"{v:.{digits}f}"
return f"${s}" if money else s
@dataclass(frozen=True)
class ValuationEvidence:
codes: tuple[str, ...]
families: tuple[str, ...]
rows: tuple[ValuationRow, ...]
unpriced: tuple[str, ...]
def prompt_block(self) -> str:
lines = [_HEADER]
for r in self.rows:
lines.append(
f'{r.label} {r.code} "{r.description}" — status {r.status or "?"}; '
f"work {_n(r.work)}; PE {_n(r.pe_nf)} non-fac / {_n(r.pe_f)} fac; "
f"MP {_n(r.mp)}; total {_n(r.total_nf)} non-fac / {_n(r.total_f)} fac; "
f"CF {_n(r.cf, 4, money=True)} → payment {_n(r.pay_nf, money=True)} non-fac / "
f"{_n(r.pay_f, money=True)} fac"
)
if self.unpriced:
lines.append("Not priced in the indexed fee schedule: " + ", ".join(self.unpriced))
return "\n".join(lines)
def payload(self) -> dict[str, Any]:
provenance: list[dict[str, str]] = []
seen: set[str] = set()
for r in self.rows:
if r.label not in seen:
seen.add(r.label)
provenance.append({"label": r.label, "vintage": r.vintage, "citation": r.citation, "url": r.url})
return {
"type": "valuation",
"codes": list(self.codes),
"families": list(self.families),
"rows": [asdict(r) for r in self.rows],
"provenance": provenance,
"unpriced": list(self.unpriced),
}
def _replica_path(cfg: LlmConfig) -> str:
p = Path(cfg.duckdb_replica)
if p.is_absolute():
return str(p)
from conf import ROOT
return str(ROOT / p)
def valuation_evidence(question: str, cfg: LlmConfig) -> ValuationEvidence | None:
det = detect_codes(question)
if not det.codes:
return None
try:
con = duckdb.connect(_replica_path(cfg), read_only=True)
except Exception as e: # noqa: BLE001 — duckdb raises several types
log.warning("valuation evidence skipped (%s): %s", cfg.duckdb_replica, e)
return None
try:
rows, unpriced = valuation(con, list(det.codes), years=cfg.valuation_years)
except Exception as e: # noqa: BLE001
log.warning("valuation evidence skipped (query): %s", e)
return None
finally:
con.close()
return ValuationEvidence(det.codes, det.families, tuple(rows), tuple(unpriced))
_CITED_SQL = text(
"SELECT e.document, e.cmetadata FROM langchain_pg_embedding e "
"JOIN langchain_pg_collection c ON c.uuid = e.collection_id "
"WHERE c.name = :collection "
"AND string_to_array(COALESCE(e.cmetadata->>'codes', ''), ' ') && CAST(:codes AS text[]) "
"ORDER BY e.cmetadata->>'date' DESC NULLS LAST, e.cmetadata->>'item_key', e.cmetadata->>'p_id' "
"LIMIT :lim"
)
def code_cited_sources(
engine: Any, codes: Sequence[str], *, per_code: int, collection: str = "rules"
) -> list[dict]:
"""Chunks in *collection* whose ``codes`` metadata mention any of
*codes* — at most *per_code* per code, newest rule first, deduped on
(item_key, p_id). Scores are 0.0: these are additive, not ranked."""
wanted = [c.upper() for c in codes]
try:
with engine.begin() as conn:
rows = conn.execute(
_CITED_SQL,
{"collection": collection, "codes": wanted, "lim": per_code * len(wanted) * 4},
).fetchall()
except Exception as e: # noqa: BLE001
log.warning("code-cited sources skipped: %s", e)
return []
out: list[dict] = []
seen: set[tuple[str, str]] = set()
count: dict[str, int] = {c: 0 for c in wanted}
for document, md in rows:
md = {k: str(v) for k, v in (md or {}).items()}
key = (md.get("item_key", ""), md.get("p_id", ""))
if key in seen:
continue
hit_codes = [c for c in md.get("codes", "").split() if c in count and count[c] < per_code]
if not hit_codes:
continue
seen.add(key)
for c in hit_codes:
count[c] += 1
out.append(as_source(md, document, 0.0))
return out
def merge_sources(retrieved: list[dict], extra: list[dict]) -> list[dict]:
labels = {s["label"] for s in retrieved}
return retrieved + [s for s in extra if s["label"] not in labels]
```
- [ ] **Step 4: Run to verify pass** — `uv run --no-sync pytest tests/llm/test_evidence.py -q -p no:cacheprovider` → 9 passed.
- [ ] **Step 5: Commit**
```bash
git add src/llm/evidence.py tests/llm/test_evidence.py
git commit -m "feat(llm): valuation evidence block, SSE payload and code-cited rule sources (refs P48)"
```
---
### Task 6: `rag.py` — prompt block, merged sources, `valuation` event
**Files:**
- Modify: `src/llm/rag.py` (`_SYSTEM`, `build_messages`, `stream_answer`)
- Test: `tests/llm/test_rag.py`
**Interfaces:**
- Consumes: `llm.evidence.valuation_evidence/code_cited_sources/merge_sources/ValuationEvidence`, `llm.index._engine`.
- Produces: `build_messages(question, sources, evidence: ValuationEvidence | None = None)`; `stream_answer` yields `valuation` first when evidence exists.
- [ ] **Step 1: Write the failing tests**
In `tests/llm/test_rag.py` add to `TestBuildMessages`:
```python
def test_evidence_block_between_excerpts_and_question(self):
from llm.evidence import ValuationEvidence
from pfs.valuation import ValuationRow
row = ValuationRow(code="G0556", description="d", vintage="CY2026 final", year=2026, proposed=False,
status="A", work=0.25, pe_nf=0.22, pe_f=0.06, mp=0.02, total_nf=0.49, total_f=0.33,
cf=33.4009, pay_nf=16.37, pay_f=11.02, label="[PFS CY2026 Addendum B]",
citation="90 FR 49266", url="u")
ev = ValuationEvidence(("G0556",), ("APCM",), (row,), ())
msgs = build_messages("how much?", [], evidence=ev)
user = msgs[1]["content"]
assert user.index("Excerpts:") < user.index("Valuation (authoritative") < user.index("Question: how much?")
assert "[PFS CY2026 Addendum B] G0556" in user
assert "valuation" in msgs[0]["content"].lower()
assert "do not compute" in msgs[0]["content"].lower()
def test_no_evidence_prompt_unchanged(self):
assert build_messages("q", []) == build_messages("q", [], evidence=None)
assert "Valuation" not in build_messages("q", [])[1]["content"]
```
In `TestStreamAnswer`, patch `llm.rag.valuation_evidence` to return `None` in the existing four tests (add `@patch("llm.rag.valuation_evidence", return_value=None)` as the outermost decorator and a leading `_ev` parameter) so they stay deterministic, and add:
```python
@patch("llm.rag._engine")
@patch("llm.rag.code_cited_sources")
@patch("llm.rag.valuation_evidence")
@patch("llm.rag.httpx.Client")
@patch("llm.rag.retrieve")
def test_valuation_event_first_and_sources_merged(self, mock_retrieve, MockClient, mock_ev, mock_cited, mock_engine):
from llm.evidence import ValuationEvidence
src = {"id": "C1", "label": "C1", "kind": "comment", "snippet": "s", "score": 0.1}
cited = {"id": "89 FR 97710 ¶3", "label": "89 FR 97710 ¶3", "kind": "rule", "snippet": "G0556", "score": 0.0}
mock_retrieve.return_value = [src]
mock_cited.return_value = [cited]
mock_ev.return_value = ValuationEvidence(("G0556",), ("APCM",), (), ())
client = MockClient.return_value.__enter__.return_value
resp = client.stream.return_value.__enter__.return_value
resp.iter_lines.return_value = iter(['{"message":{"content":"x"},"done":true}'])
events = list(stream_answer("APCM?", cfg=CFG, pool=self._pool()))
assert events[0]["type"] == "valuation" and events[0]["codes"] == ["G0556"]
assert events[1] == {"type": "token", "text": "x"}
assert events[-2]["sources"] == [src, cited]
mock_cited.assert_called_once_with(mock_engine.return_value, ("G0556",), per_code=CFG.code_cited_per_code)
body = client.stream.call_args.kwargs["json"]
assert "Valuation (authoritative" in body["messages"][1]["content"]
```
- [ ] **Step 2: Run to verify failure** — `uv run --no-sync pytest tests/llm/test_rag.py -q -p no:cacheprovider` → FAIL (`TypeError: build_messages() got an unexpected keyword argument 'evidence'`, `AttributeError: llm.rag has no attribute valuation_evidence`).
- [ ] **Step 3: Implement**
`_SYSTEM` +=
```python
" A Valuation section may follow the excerpts. Its rows are "
"authoritative for RVUs, conversion factors and payment amounts; when "
"you state any of those numbers, cite the row's bracketed label exactly, "
"e.g. [PFS CY2026 Addendum B]. Do not compute, extrapolate or convert "
"numbers beyond what the rows show; if a code is listed as not priced, "
"say so."
```
Imports: `from llm.evidence import ValuationEvidence, code_cited_sources, merge_sources, valuation_evidence` and `from llm.index import _engine`.
```python
def build_messages(
question: str, sources: list[dict], evidence: ValuationEvidence | None = None
) -> list[dict]:
... # context as today
parts = [f"Excerpts:\n\n{context}"]
if evidence is not None:
parts.append(evidence.prompt_block())
parts.append(f"Question: {question}")
user = "\n\n".join(parts)
...
def stream_answer(question, *, cfg, pool, since=""):
sources = retrieve(question, cfg=cfg, pool=pool, since=since)
evidence = valuation_evidence(question, cfg)
if evidence is not None:
cited = code_cited_sources(_engine(cfg), evidence.codes, per_code=cfg.code_cited_per_code)
sources = merge_sources(sources, cited)
yield evidence.payload()
pool.check(cfg.instruct_model)
messages = build_messages(question, sources, evidence)
... # unchanged
```
Update the module docstring's event list.
- [ ] **Step 4: Run to verify pass** — `uv run --no-sync pytest tests/llm -q -p no:cacheprovider` → pass.
- [ ] **Step 5: Commit**
```bash
git add src/llm/rag.py tests/llm/test_rag.py
git commit -m "feat(llm): chat cites code valuations — Valuation prompt block, merged code-cited sources, valuation SSE event (refs P48)"
```
---
### Task 7: `chat.html` — render the valuation table
**Files:**
- Modify: `src/llm/web/chat.html`
- Test: `tests/llm/test_api.py`
- [ ] **Step 1: Write the failing test** (extend `test_serves_chat_page`)
```python
def test_serves_chat_page_with_valuation_renderer(self):
r = client.get("/")
assert r.status_code == 200
html = r.text
assert "function renderValuation" in html
assert "ev.type === 'valuation'" in html
assert "table.valuation" in html
```
- [ ] **Step 2: Run to verify failure** — `uv run --no-sync pytest tests/llm/test_api.py -q -p no:cacheprovider` → FAIL on the new assertions.
- [ ] **Step 3: Implement**
CSS (next to `details.sources`):
```css
table.valuation {
border-collapse: collapse; width: 100%; font-size: 13px; margin-top: 4px;
font-variant-numeric: tabular-nums;
}
table.valuation th, table.valuation td {
border: 1px solid var(--border); padding: 4px 6px; text-align: right; white-space: nowrap;
}
table.valuation th { font-family: var(--font-mono); font-size: 10px; letter-spacing: .08em;
text-transform: uppercase; color: var(--muted-fg); }
table.valuation td:nth-child(-n+3), table.valuation th:nth-child(-n+3) { text-align: left; }
table.valuation tr.proposed td { font-style: italic; opacity: .85; }
.valuation-wrap { overflow-x: auto; }
.provenance { font-size: 12px; color: var(--muted-fg); margin-top: 4px; }
.provenance .cite { margin-right: 4px; }
```
JS (next to `renderSources`):
```js
const VAL_COLS = [
['code', 'Code'], ['description', 'Description'], ['vintage', 'Vintage'], ['status', 'St'],
['work', 'Work'], ['pe_nf', 'PE NF'], ['pe_f', 'PE F'], ['mp', 'MP'],
['total_nf', 'Total NF'], ['total_f', 'Total F'], ['cf', 'CF'], ['pay_nf', 'Pay NF'], ['pay_f', 'Pay F'],
];
function fmt(key, v) {
if (v === null || v === undefined) return '—';
if (key === 'cf') return '$' + Number(v).toFixed(4);
if (key === 'pay_nf' || key === 'pay_f') return '$' + Number(v).toFixed(2);
if (typeof v === 'number') return v.toFixed(2);
return String(v);
}
function renderValuation(wrap, ev) {
const rows = ev.rows || [];
const box = document.createElement('div');
box.className = 'valuation-wrap';
if (rows.length) {
const t = document.createElement('table');
t.className = 'valuation';
const thead = t.createTHead().insertRow();
for (const [, name] of VAL_COLS) { const th = document.createElement('th'); th.textContent = name; thead.appendChild(th); }
const tb = t.createTBody();
for (const r of rows) {
const tr = tb.insertRow();
if (r.proposed) tr.className = 'proposed';
for (const [key] of VAL_COLS) tr.insertCell().textContent = fmt(key, r[key]);
}
box.appendChild(t);
}
const prov = document.createElement('div');
prov.className = 'provenance';
for (const p of ev.provenance || []) {
const c = document.createElement('span'); c.className = 'cite'; c.textContent = p.label;
prov.appendChild(c);
const a = document.createElement('a'); a.href = p.url; a.target = '_blank'; a.rel = 'noopener';
a.textContent = p.citation || p.vintage;
prov.appendChild(a);
prov.appendChild(document.createTextNode(' '));
}
if ((ev.unpriced || []).length) {
prov.appendChild(document.createTextNode('not priced: ' + ev.unpriced.join(', ')));
}
box.appendChild(prov);
wrap.appendChild(box);
log.scrollTop = log.scrollHeight;
}
```
Dispatch: add `else if (ev.type === 'valuation') renderValuation(wrap, ev);` after the `token` branch. The table lands under the bubble (before the sources drawer that arrives later).
- [ ] **Step 4: Run to verify pass** — `uv run --no-sync pytest tests/llm/test_api.py -q -p no:cacheprovider` → pass. Also open the page locally (`uv run stack llm serve --port 8010`, then `curl -s localhost:8010/ | grep -c renderValuation`) to be sure the HTML is well-formed (no JS syntax error in the browser console if you can open it).
- [ ] **Step 5: Commit**
```bash
git add src/llm/web/chat.html tests/llm/test_api.py
git commit -m "feat(llm): chat UI renders the valuation table with FR provenance links (refs P48)"
```
---
### Task 8: compose mount, rollout, live probes
**Files:**
- Modify: `compose.yml` (`llm` service)
- [ ] **Step 1: compose**
Add to the `llm` service:
```yaml
volumes:
- ./data:/app/data:ro # DuckDB read replica for code valuations (directory mount survives replica re-publish)
environment:
...
- LLM_DUCKDB_REPLICA=/app/data/aco.ro.duckdb
```
Run `docker compose config --services >/dev/null && echo ok`. Commit: `chore(compose): llm reads the DuckDB replica read-only (refs P48)`.
- [ ] **Step 2: Full suite** — `uv run --no-sync pytest tests -q -p no:cacheprovider -n auto` → all pass.
- [ ] **Step 3: Rollout (from /home/kert/stack on main, after merge)**
```bash
set -a; . ./.env; set +a
uv run stack llm index --collection rules --force # stamp codes metadata on the 80 rules (~15 min)
# roll the container: COMMIT_SHA in .env → HEAD, then
docker compose build llm && docker compose up -d llm && docker compose ps llm
docker exec llm curl -s http://localhost:8000/health
# probes
docker exec llm curl -s -N -X POST http://localhost:8000/chat -H 'Content-Type: application/json' \
-d '{"question":"What is APCM and how is it valued?"}' | tee .state/llm/probe-apcm.log | grep -c '"type": "valuation"'
grep -o '"code": "G055[678]"' .state/llm/probe-apcm.log | sort | uniq -c # 3 codes × 3 vintages
grep -o '\[PFS CY2026 Addendum B\]\|\[CY2027 NPRM Addendum B\]' .state/llm/probe-apcm.log | sort | uniq -c # cited in prose
grep -c '"kind": "rule"' .state/llm/probe-apcm.log # code-cited paragraphs present
docker exec llm curl -s -N -X POST http://localhost:8000/chat -H 'Content-Type: application/json' \
-d '{"question":"What did commenters say about telehealth?"}' | grep -c '"type": "valuation"' # must be 0
```
Record the probe results (counts + the first prose paragraph) on the tracker issue for P48 and update the memory note `llm_service_whole_library.md` with the valuation feature and the `LLM_DUCKDB_REPLICA` contract.
Reference in New Issue View Git Blame Copy Permalink