Files

kert b8beb97b08 docs(plan): llm chat code valuation implementation plan

2026-09-08 23:40:22 -04:00

52 KiB

Raw Blame History

LLM Chat Code Valuation Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: When a chat question names a HCPCS/CPT code or a registered code family, the answer keeps its retrieved citations and additionally presents each code's RVUs, conversion factor and national payment by vintage, as a cited prompt block, a valuation SSE event, and a rendered table.

Architecture: Deterministic detection (pfs/families.py) runs before generation; pfs/valuation.py reads the DuckDB read replica and attaches conversion factors from pfs.rules; llm/evidence.py turns rows into a prompt block, a UI payload, and pulls "code-cited" rule chunks from pgvector via a codes metadata list stamped at index time; llm/rag.py merges those sources, adds the block to the prompt and emits the event; chat.html renders the table. The llm container gets a read-only ./data mount.

Tech Stack: Python 3.13, uv run --no-sync, pytest, DuckDB (in-memory fixtures in tests), SQLAlchemy text() against pgvector's langchain_pg_embedding, FastAPI SSE, vanilla JS in chat.html.

Spec: docs/superpowers/specs/2026-09-08-llm-chat-code-valuation-design.md

Global Constraints

Tests: uv run --no-sync pytest <path> -q -p no:cacheprovider; output pristine. The pre-commit hook runs ruff + the cli suite.
Commit messages: conventional prefix, (refs P48); no Co-Authored-By trailer. git status --short before every commit; stage only your own files (shared worktree).
Nothing under src/pfs/families.py, src/pfs/valuation.py, src/llm/evidence.py may import narwhals, pfs.calcs, pfs.pipe, aco, or conf.connect.ducklake — the chat image lacks narwhals. pfs/__init__.py has no imports, so pfs.rules / pfs.nprm / pfs.families / pfs.valuation are safe.
Chunk metadata stays dict[str, str]: the new codes value is a space-joined string ("G0556 G0557"), "" when none.
Citation labels are square-bracketed: final [PFS CY2026 Addendum B], proposed [CY2027 NPRM Addendum B]. FR citation URL: https://www.federalregister.gov/citation/<vol>-FR-<page> built from a "90 FR 49266" string.
Conversion factors come only from pfs.rules.RULES[year].conversion_factor (final) and pfs.rules.proposed_for(year).conversion_factor (proposed); never from pfs.rvu.conv_factor.
National unadjusted payment = round(total_rvu * cf, 2); no GPCI, no OPPS.
Event order: valuation (only when codes detected) → token* → sources → done. With no codes detected, the prompt and event sequence are byte-identical to today.
Failures in evidence (replica missing, pgvector error) log a warning and drop the evidence; the chat never fails because of it.

File Structure

File	Responsibility
`src/pfs/families.py` (new)	`Family`, `FAMILIES`, `CODE_RE`, `find_codes(text)`, `detect_codes(text) -> Detection` — pure
`src/pfs/valuation.py` (new)	`ValuationRow`, `valuation(con, codes, *, years)`, `vintage_label/citation/url` helpers — DuckDB SQL + `pfs.rules`
`src/llm/links.py`	`as_source(md, text, score) -> dict` (moved from `rag._source`)
`src/llm/chunk.py`	stamp `codes` metadata on every chunk
`src/llm/config.py` + `stack.toml`	`duckdb_replica`, `valuation_years`, `code_cited_per_code`
`src/llm/evidence.py` (new)	`ValuationEvidence`, `valuation_evidence(question, cfg)`, `code_cited_sources(engine, codes, *, per_code, collection)`
`src/llm/rag.py`	system prompt addendum, `build_messages(..., evidence=)`, `stream_answer` merge + event
`src/llm/web/chat.html`	`renderValuation` + styles
`compose.yml`	`llm` mount + env
`tests/pfs/test_families.py`, `tests/pfs/test_valuation.py`, `tests/llm/test_evidence.py` (new); `tests/llm/test_chunk.py`, `test_config.py`, `test_rag.py`, `test_api.py`, `test_links.py` (extend)

Task 1: `pfs/families.py` — registry and detection

Files:

Create: src/pfs/families.py
Test: tests/pfs/test_families.py

Interfaces:

Produces: CODE_RE (compiled), find_codes(text) -> tuple[str, ...] (sorted unique codes literally present), Family(key, name, codes, synonyms), FAMILIES: dict[str, Family], Detection(codes, families, explicit), detect_codes(text) -> Detection, family_of(code) -> Family | None.
Step 1: Write the failing tests

# tests/pfs/test_families.py
"""pfs.families — code-family registry + deterministic code detection."""

from __future__ import annotations

from pfs.families import FAMILIES, Detection, detect_codes, family_of, find_codes


class TestFindCodes:
    def test_hcpcs_and_cpt(self):
        assert find_codes("Codes G0556 and 99490 apply; see g0557.") == ("99490", "G0556")

    def test_no_false_hits_on_years_and_fr_pages(self):
        # 2026 (4 digits) and 43842 (5 digits) — the CPT regex will match 43842;
        # validation against the fee schedule happens at lookup, not here.
        assert find_codes("91 FR 43842, CY2026") == ("43842",)

    def test_dedupes_and_sorts(self):
        assert find_codes("G0558 G0556 G0558") == ("G0556", "G0558")

    def test_empty(self):
        assert find_codes("") == ()


class TestRegistry:
    def test_apcm_family(self):
        f = FAMILIES["APCM"]
        assert f.codes == ("G0556", "G0557", "G0558")
        assert "advanced primary care management" in f.synonyms
        assert family_of("G0557") is f
        assert family_of("00000") is None

    def test_all_families_present(self):
        assert set(FAMILIES) == {"ACP", "CCM", "PCM", "TCM", "APCM"}


class TestDetectCodes:
    def test_family_name_expands_to_codes(self):
        d = detect_codes("What is APCM and how is it valued?")
        assert d == Detection(codes=("G0556", "G0557", "G0558"), families=("APCM",), explicit=())

    def test_synonym_case_insensitive_word_boundary(self):
        d = detect_codes("Tell me about Advanced Primary Care Management.")
        assert d.families == ("APCM",)
        assert detect_codes("the apcmx code").families == ()

    def test_single_member_code_expands_family(self):
        d = detect_codes("How much does G0557 pay?")
        assert d.explicit == ("G0557",)
        assert d.families == ("APCM",)
        assert d.codes == ("G0556", "G0557", "G0558")

    def test_unregistered_code_stays_alone(self):
        d = detect_codes("value of 99213")
        assert d == Detection(codes=("99213",), families=(), explicit=("99213",))

    def test_multiple_families(self):
        d = detect_codes("compare CCM and TCM")
        assert d.families == ("CCM", "TCM")
        assert d.codes == tuple(sorted(FAMILIES["CCM"].codes + FAMILIES["TCM"].codes))

    def test_nothing(self):
        assert detect_codes("what did commenters say about telehealth?") == Detection((), (), ())

Step 2: Run to verify failure

Run: uv run --no-sync pytest tests/pfs/test_families.py -q -p no:cacheprovider Expected: FAIL — ModuleNotFoundError: No module named 'pfs.families'

Step 3: Implement

# src/pfs/families.py
"""Code families and deterministic code detection for chat questions.

The registry used to live in a notebook cell
(``notebooks/palliative_care_rfi.py``); this is its home in ``src`` so
the chat, notebooks and scripts share one list. Pure: no I/O, no
DuckDB, no narwhals — it must import inside the ``llm`` container.
"""

from __future__ import annotations

import re
from dataclasses import dataclass

# HCPCS level II (letter + 4 digits) or CPT (5 digits). Validation against
# the fee schedule happens at lookup time (pfs.valuation), not here.
CODE_RE = re.compile(r"\b(?:[A-Z]\d{4}|\d{5})\b")


def find_codes(text: str) -> tuple[str, ...]:
    """Sorted unique codes literally present in *text* (case-normalised)."""
    return tuple(sorted({m.group(0).upper() for m in CODE_RE.finditer(text.upper())}))


@dataclass(frozen=True)
class Family:
    key: str
    name: str
    codes: tuple[str, ...]
    synonyms: tuple[str, ...]  # lower-case; matched on word boundaries


FAMILIES: dict[str, Family] = {
    "ACP": Family(
        "ACP", "Advance Care Planning", ("99497", "99498"),
        ("acp", "advance care planning"),
    ),
    "CCM": Family(
        "CCM", "Chronic Care Management",
        ("99437", "99439", "99487", "99489", "99490", "99491"),
        ("ccm", "chronic care management"),
    ),
    "PCM": Family(
        "PCM", "Principal Care Management", ("99424", "99425", "99426", "99427"),
        ("pcm", "principal care management"),
    ),
    "TCM": Family(
        "TCM", "Transitional Care Management", ("99495", "99496"),
        ("tcm", "transitional care management"),
    ),
    "APCM": Family(
        "APCM", "Advanced Primary Care Management", ("G0556", "G0557", "G0558"),
        ("apcm", "advanced primary care management"),
    ),
}

_CODE_TO_FAMILY: dict[str, Family] = {c: f for f in FAMILIES.values() for c in f.codes}


def family_of(code: str) -> Family | None:
    return _CODE_TO_FAMILY.get(code.upper())


@dataclass(frozen=True)
class Detection:
    codes: tuple[str, ...]      # sorted unique: explicit + family-expanded
    families: tuple[str, ...]   # family keys, sorted
    explicit: tuple[str, ...]   # codes literally present in the text


def detect_codes(text: str) -> Detection:
    """Codes a question is about: explicit codes plus every code of any
    family named (by synonym) or touched (by one member code)."""
    explicit = find_codes(text)
    lowered = text.lower()
    families: set[str] = set()
    for fam in FAMILIES.values():
        if any(re.search(rf"\b{re.escape(s)}\b", lowered) for s in fam.synonyms):
            families.add(fam.key)
    for code in explicit:
        fam = family_of(code)
        if fam is not None:
            families.add(fam.key)
    codes = set(explicit)
    for key in families:
        codes.update(FAMILIES[key].codes)
    return Detection(
        codes=tuple(sorted(codes)),
        families=tuple(sorted(families)),
        explicit=explicit,
    )

Step 4: Run to verify pass

Run: uv run --no-sync pytest tests/pfs/test_families.py -q -p no:cacheprovider Expected: 12 passed

Step 5: Commit

git add src/pfs/families.py tests/pfs/test_families.py
git commit -m "feat(pfs): code-family registry + deterministic code detection (refs P48)"

Task 2: `pfs/valuation.py` — rows by vintage from the replica

Files:

Create: src/pfs/valuation.py
Test: tests/pfs/test_valuation.py

Interfaces:

Consumes: pfs.rules.RULES, pfs.rules.proposed_for, pfs.nprm.NPRM_SOURCES.
Produces: ValuationRow (frozen dataclass; fields below), valuation(con, codes, *, years=4) -> tuple[list[ValuationRow], list[str]] (rows ordered by code then year, proposed last per code; second element = codes with no rows), fr_citation_url(cite) -> str, vintage_label(year, proposed) -> str.
Step 1: Write the failing tests

# tests/pfs/test_valuation.py
"""pfs.valuation — RVUs + national payment by vintage from a DuckDB replica."""

from __future__ import annotations

import duckdb
import pytest

from pfs.rules import RULES, proposed_for
from pfs.valuation import ValuationRow, fr_citation_url, valuation, vintage_label

RVU_COLS = (
    "hcpcs VARCHAR, mod VARCHAR, description VARCHAR, status_code VARCHAR, "
    "work_rvu DOUBLE, non_fac_pe_rvu DOUBLE, fac_pe_rvu DOUBLE, mp_rvu DOUBLE, "
    "non_fac_total DOUBLE, fac_total DOUBLE, conv_factor DOUBLE, year INTEGER"
)
PROPOSED_COLS = (
    "hcpcs VARCHAR, mod VARCHAR, description VARCHAR, status_code VARCHAR, "
    "work_rvu DOUBLE, non_fac_pe_rvu DOUBLE, fac_pe_rvu DOUBLE, mp_rvu DOUBLE, cms_rule_id VARCHAR"
)


@pytest.fixture
def con():
    c = duckdb.connect(":memory:")
    c.execute("CREATE SCHEMA pfs")
    c.execute(f"CREATE TABLE pfs.rvu ({RVU_COLS})")
    c.execute(f"CREATE TABLE pfs.rvu_proposed ({PROPOSED_COLS})")
    rows = [
        # G0556: 2025 + 2026 final, with a modifier row that must be ignored
        ("G0556", None, "Adv prim care mgmt lvl 1", "A", 0.25, 0.20, 0.10, 0.02, 0.47, 0.37, 32.3465, 2025),
        ("G0556", None, "Adv prim care mgmt lvl 1", "A", 0.25, 0.22, 0.06, 0.02, 0.49, 0.33, 33.4009, 2026),
        ("G0556", "26", "Adv prim care mgmt lvl 1", "A", 9.0, 9.0, 9.0, 9.0, 27.0, 27.0, 33.4009, 2026),
        # 99490: six years so the last-N cut applies (2021..2026)
        *[("99490", "", "Chrnc care mgmt srvc 20 min", "A", 1.0, 1.0, 0.5, 0.05, 2.05, 1.55, 30.0, y) for y in range(2021, 2027)],
        # I-status code with NULL components
        ("G9999", None, "Not priced", "I", None, None, None, None, None, None, 33.4009, 2026),
    ]
    c.executemany("INSERT INTO pfs.rvu VALUES (?,?,?,?,?,?,?,?,?,?,?,?)", rows)
    c.executemany(
        "INSERT INTO pfs.rvu_proposed VALUES (?,?,?,?,?,?,?,?,?)",
        [
            ("G0556", None, "Adv prim care mgmt lvl 1", "A", 0.25, 0.23, 0.07, 0.02, "CMS-1848-P"),
            ("G0556", "26", "Adv prim care mgmt lvl 1", "A", 9.0, 9.0, 9.0, 9.0, "CMS-1848-P"),
            ("G0556", None, "older nprm", "A", 0.1, 0.1, 0.1, 0.1, "CMS-1832-P"),
        ],
    )
    yield c
    c.close()


class TestHelpers:
    def test_vintage_label(self):
        assert vintage_label(2026, False) == "[PFS CY2026 Addendum B]"
        assert vintage_label(2027, True) == "[CY2027 NPRM Addendum B]"

    def test_fr_citation_url(self):
        assert fr_citation_url("90 FR 49266") == "https://www.federalregister.gov/citation/90-FR-49266"
        assert fr_citation_url("") == ""


class TestValuation:
    def test_final_and_proposed_rows_with_cf_and_payment(self, con):
        rows, unpriced = valuation(con, ["G0556"], years=4)
        assert unpriced == []
        assert [(r.year, r.proposed) for r in rows] == [(2025, False), (2026, False), (2027, True)]
        r26 = rows[1]
        assert isinstance(r26, ValuationRow)
        assert (r26.code, r26.status, r26.work, r26.pe_nf, r26.pe_f, r26.mp) == ("G0556", "A", 0.25, 0.22, 0.06, 0.02)
        assert (r26.total_nf, r26.total_f) == (0.49, 0.33)
        assert r26.cf == RULES[2026].conversion_factor
        assert r26.pay_nf == round(0.49 * RULES[2026].conversion_factor, 2)
        assert r26.pay_f == round(0.33 * RULES[2026].conversion_factor, 2)
        assert r26.label == "[PFS CY2026 Addendum B]"
        assert r26.citation == RULES[2026].federal_register_citation
        assert r26.url == fr_citation_url(RULES[2026].federal_register_citation)
        assert r26.vintage == "CY2026 final"
        p = rows[2]
        assert p.label == "[CY2027 NPRM Addendum B]" and p.vintage == "CY2027 proposed"
        assert p.cf == proposed_for(2027).conversion_factor
        assert p.total_nf == round(0.25 + 0.23 + 0.02, 4) and p.total_f == round(0.25 + 0.07 + 0.02, 4)
        assert p.pay_nf == round(p.total_nf * p.cf, 2)
        assert p.citation == proposed_for(2027).federal_register_citation

    def test_modifier_rows_are_ignored(self, con):
        rows, _ = valuation(con, ["G0556"])
        assert all(r.work == 0.25 for r in rows)

    def test_last_n_years_only(self, con):
        rows, _ = valuation(con, ["99490"], years=4)
        assert [r.year for r in rows if not r.proposed] == [2023, 2024, 2025, 2026]

    def test_unpriced_codes_reported(self, con):
        rows, unpriced = valuation(con, ["G0556", "Z9999"])
        assert unpriced == ["Z9999"]
        assert {r.code for r in rows} == {"G0556"}

    def test_null_components_pass_through(self, con):
        rows, unpriced = valuation(con, ["G9999"])
        assert unpriced == []
        (r,) = rows
        assert r.status == "I" and r.work is None and r.pay_nf is None

    def test_order_is_code_then_year(self, con):
        rows, _ = valuation(con, ["G0556", "99490"], years=2)
        assert [(r.code, r.year) for r in rows] == [
            ("99490", 2025), ("99490", 2026), ("G0556", 2025), ("G0556", 2026), ("G0556", 2027)
        ]

Step 2: Run to verify failure

Run: uv run --no-sync pytest tests/pfs/test_valuation.py -q -p no:cacheprovider Expected: FAIL — ModuleNotFoundError: No module named 'pfs.valuation'

Step 3: Implement

# src/pfs/valuation.py
"""RVUs, conversion factor and national unadjusted payment per code and
vintage, read from the DuckDB replica (``pfs.rvu`` final years,
``pfs.rvu_proposed`` for the newest NPRM). Conversion factors come from
``pfs.rules`` — never from ``pfs.rvu.conv_factor``. Pure DuckDB SQL; no
narwhals (this module runs inside the ``llm`` container).
"""

from __future__ import annotations

from dataclasses import dataclass
from typing import Any, Sequence

from pfs.nprm import NPRM_SOURCES
from pfs.rules import RULES, proposed_for


@dataclass(frozen=True)
class ValuationRow:
    code: str
    description: str
    vintage: str          # "CY2026 final" | "CY2027 proposed"
    year: int
    proposed: bool
    status: str
    work: float | None
    pe_nf: float | None
    pe_f: float | None
    mp: float | None
    total_nf: float | None
    total_f: float | None
    cf: float
    pay_nf: float | None
    pay_f: float | None
    label: str            # "[PFS CY2026 Addendum B]"
    citation: str         # "90 FR 49266"
    url: str


def vintage_label(year: int, proposed: bool) -> str:
    return f"[CY{year} NPRM Addendum B]" if proposed else f"[PFS CY{year} Addendum B]"


def fr_citation_url(cite: str) -> str:
    parts = cite.split()
    if len(parts) != 3 or parts[1].upper() != "FR":
        return ""
    return f"https://www.federalregister.gov/citation/{parts[0]}-FR-{parts[2]}"


def _pay(total: float | None, cf: float) -> float | None:
    return None if total is None else round(total * cf, 2)


_FINAL_SQL = """
WITH base AS (
    SELECT hcpcs, year, description, status_code, work_rvu, non_fac_pe_rvu,
           fac_pe_rvu, mp_rvu, non_fac_total, fac_total
    FROM pfs.rvu
    WHERE hcpcs IN (SELECT UNNEST(?::VARCHAR[]))
    QUALIFY row_number() OVER (
        PARTITION BY hcpcs, year ORDER BY (mod IS NULL OR mod = '') DESC, mod
    ) = 1
)
SELECT * FROM base
QUALIFY dense_rank() OVER (PARTITION BY hcpcs ORDER BY year DESC) <= ?
ORDER BY hcpcs, year
"""

_PROPOSED_SQL = """
SELECT hcpcs, description, status_code, work_rvu, non_fac_pe_rvu, fac_pe_rvu, mp_rvu
FROM pfs.rvu_proposed
WHERE cms_rule_id = ? AND hcpcs IN (SELECT UNNEST(?::VARCHAR[]))
QUALIFY row_number() OVER (
    PARTITION BY hcpcs ORDER BY (mod IS NULL OR mod = '') DESC, mod
) = 1
ORDER BY hcpcs
"""


def _sum(*vals: float | None) -> float | None:
    if any(v is None for v in vals):
        return None
    return round(sum(vals), 4)  # type: ignore[arg-type]


def valuation(
    con: Any, codes: Sequence[str], *, years: int = 4
) -> tuple[list[ValuationRow], list[str]]:
    """Rows for *codes* (last *years* final vintages + the newest NPRM),
    ordered by code then year with the proposed row last per code, and the
    codes that produced no rows at all."""
    wanted = sorted({c.upper() for c in codes})
    if not wanted:
        return [], []
    by_code: dict[str, list[ValuationRow]] = {c: [] for c in wanted}

    for hcpcs, year, desc, status, work, pe_nf, pe_f, mp, tot_nf, tot_f in con.execute(
        _FINAL_SQL, [wanted, years]
    ).fetchall():
        rule = RULES.get(int(year))
        if rule is None:
            continue
        cf = rule.conversion_factor
        cite = rule.federal_register_citation
        by_code[hcpcs].append(
            ValuationRow(
                code=hcpcs, description=desc or "", vintage=f"CY{year} final",
                year=int(year), proposed=False, status=status or "",
                work=work, pe_nf=pe_nf, pe_f=pe_f, mp=mp,
                total_nf=tot_nf, total_f=tot_f, cf=cf,
                pay_nf=_pay(tot_nf, cf), pay_f=_pay(tot_f, cf),
                label=vintage_label(int(year), False), citation=cite,
                url=fr_citation_url(cite),
            )
        )

    nprm_year, _tag, rule_id, _pin = max(NPRM_SOURCES, key=lambda s: s[0])
    prop = proposed_for(nprm_year)
    if prop is not None:
        cf = prop.conversion_factor
        cite = prop.federal_register_citation
        for hcpcs, desc, status, work, pe_nf, pe_f, mp in con.execute(
            _PROPOSED_SQL, [rule_id, wanted]
        ).fetchall():
            tot_nf, tot_f = _sum(work, pe_nf, mp), _sum(work, pe_f, mp)
            by_code[hcpcs].append(
                ValuationRow(
                    code=hcpcs, description=desc or "", vintage=f"CY{nprm_year} proposed",
                    year=nprm_year, proposed=True, status=status or "",
                    work=work, pe_nf=pe_nf, pe_f=pe_f, mp=mp,
                    total_nf=tot_nf, total_f=tot_f, cf=cf,
                    pay_nf=_pay(tot_nf, cf), pay_f=_pay(tot_f, cf),
                    label=vintage_label(nprm_year, True), citation=cite,
                    url=fr_citation_url(cite),
                )
            )

    rows = [r for c in wanted for r in sorted(by_code[c], key=lambda r: (r.year, r.proposed))]
    unpriced = [c for c in wanted if not by_code[c]]
    return rows, unpriced

Note: DuckDB accepts a Python list for a ?::VARCHAR[] parameter; UNNEST turns it into rows for IN.

Step 4: Run to verify pass

Run: uv run --no-sync pytest tests/pfs/test_valuation.py -q -p no:cacheprovider Expected: 8 passed

Step 5: Commit

git add src/pfs/valuation.py tests/pfs/test_valuation.py
git commit -m "feat(pfs): valuation rows by vintage from the DuckDB replica with pfs.rules conversion factors (refs P48)"

Task 3: `codes` chunk metadata + `links.as_source`

Files:

Modify: src/llm/chunk.py (chunk_doc, _chunk_paragraphs)
Modify: src/llm/links.py (add as_source), src/llm/rag.py (_source delegates)
Test: tests/llm/test_chunk.py, tests/llm/test_links.py

Interfaces:

Consumes: pfs.families.find_codes.
Produces: every Chunk.metadata["codes"] = space-joined sorted codes in that chunk's text ("" when none); llm.links.as_source(md: dict[str,str], text: str, score: float) -> dict returning the same dict shape rag._source returns today (id,label,kind,url,title,date,docket,comment_id,snippet,score).
Step 1: Write the failing tests

Append to tests/llm/test_chunk.py:

class TestCodesMetadata:
    def test_codes_stamped_per_chunk(self):
        text = "We propose G0556 and G0557.\n\n## Other\n\nNo codes here.\n\n## More\n\n99490 applies."
        chunks = chunk_doc(_doc(text), target_chars=60, overlap_chars=10)
        found = {c.metadata["codes"] for c in chunks}
        assert "G0556 G0557" in found
        assert "99490" in found
        assert "" in found

    def test_rule_paragraph_chunks_get_codes(self):
        from llm.chunk import Paragraph

        doc = Doc(
            key="R1", text="x",
            metadata={"kind": "rule"},
            paragraphs=(Paragraph(1, 100, 1, "APCM code G0556 is valued at 0.25 work RVUs."),),
        )
        (chunk,) = chunk_doc(doc)
        assert chunk.metadata["codes"] == "G0556"

Append to tests/llm/test_links.py:

class TestAsSource:
    def test_shape_matches_rag_source(self):
        from llm.links import as_source

        s = as_source({"kind": "comment", "comment_id": "CMS-2026-2377-1", "date": "2026-08-19", "title": "t", "docket": "CMS-2026-2377"}, "body " * 200, 0.123456)
        assert set(s) == {"id", "label", "kind", "url", "title", "date", "docket", "comment_id", "snippet", "score"}
        assert s["label"] == "CMS-2026-2377-1" and s["kind"] == "comment"
        assert len(s["snippet"]) <= 500 and s["score"] == 0.1235

Step 2: Run to verify failure

Run: uv run --no-sync pytest tests/llm/test_chunk.py tests/llm/test_links.py -q -p no:cacheprovider Expected: FAIL — KeyError: 'codes' and ImportError: cannot import name 'as_source'.

Step 3: Implement

In src/llm/chunk.py: add from pfs.families import find_codes at the top; in both chunk builders add "codes": " ".join(find_codes(piece)) (comment path, using piece) and "codes": " ".join(find_codes(text)) (paragraph path) to the metadata dicts after "section".

In src/llm/links.py add:

_SNIPPET_CHARS = 500


def as_source(md: dict[str, str], text: str, score: float) -> dict:
    """The source dict the chat sends to the prompt and the UI."""
    snippet = text[:_SNIPPET_CHARS].strip()
    url, label = for_source(md, snippet)
    return {
        "id": label,
        "label": label,
        "kind": md.get("kind", ""),
        "url": url,
        "title": md.get("title", ""),
        "date": md.get("date", ""),
        "docket": md.get("docket", ""),
        "comment_id": md.get("comment_id", ""),
        "snippet": snippet,
        "score": round(score, 4),
    }

In src/llm/rag.py: _source(hit) becomes return as_source(hit.metadata, hit.text, hit.score) (import as_source from llm.links; delete the local _SNIPPET_CHARS if unused).

Step 4: Run to verify pass

Run: uv run --no-sync pytest tests/llm -q -p no:cacheprovider Expected: pass (existing test_rag retrieve tests still assert the same source shape).

Step 5: Commit

git add src/llm/chunk.py src/llm/links.py src/llm/rag.py tests/llm/test_chunk.py tests/llm/test_links.py
git commit -m "feat(llm): stamp codes metadata on every chunk; links.as_source shared source dict (refs P48)"

Task 4: Config knobs

Files:

Modify: src/llm/config.py (LlmConfig, load), stack.toml ([llm])
Test: tests/llm/test_config.py

Interfaces:

Produces: LlmConfig.duckdb_replica: str (default "data/aco.ro.duckdb", env LLM_DUCKDB_REPLICA overrides), valuation_years: int (default 4), code_cited_per_code: int (default 3).
Step 1: Write the failing tests (append to TestLoad)

    def test_valuation_knobs_defaults(self, monkeypatch):
        monkeypatch.delenv("LLM_DUCKDB_REPLICA", raising=False)
        cfg = llm_config.load()
        assert cfg.duckdb_replica == "data/aco.ro.duckdb"
        assert cfg.valuation_years == 4
        assert cfg.code_cited_per_code == 3

    def test_duckdb_replica_env_override(self, monkeypatch):
        monkeypatch.setenv("LLM_DUCKDB_REPLICA", "/app/data/aco.ro.duckdb")
        assert llm_config.load().duckdb_replica == "/app/data/aco.ro.duckdb"

Step 2: Run to verify failure — uv run --no-sync pytest tests/llm/test_config.py -q -p no:cacheprovider → FAIL (AttributeError: duckdb_replica).
Step 3: Implement

LlmConfig gains (after top_n):

    duckdb_replica: str = "data/aco.ro.duckdb"
    valuation_years: int = 4
    code_cited_per_code: int = 3

load() passes:

        duckdb_replica=os.environ.get("LLM_DUCKDB_REPLICA") or str(_opt(section, "duckdb_replica", "data/aco.ro.duckdb")),
        valuation_years=int(_opt(section, "valuation_years", 4)),
        code_cited_per_code=int(_opt(section, "code_cited_per_code", 3)),

Add to the module docstring env contract: LLM_DUCKDB_REPLICA beats [llm].duckdb_replica (containers set /app/data/aco.ro.duckdb). In stack.toml [llm] add:

duckdb_replica = "data/aco.ro.duckdb"   # read-only DuckDB replica the chat reads valuations from
valuation_years = 4                      # final-rule vintages shown per code (plus the newest NPRM)
code_cited_per_code = 3                  # rule paragraphs literally citing each detected code

Also update tests/llm/test_rag.py's CFG and any other LlmConfig(...) literal in tests only if they break (they use defaults, so they should not).

Step 4: Run to verify pass — uv run --no-sync pytest tests/llm/test_config.py tests/llm/test_rag.py -q -p no:cacheprovider → pass.
Step 5: Commit

git add src/llm/config.py stack.toml tests/llm/test_config.py
git commit -m "feat(llm): duckdb_replica / valuation_years / code_cited_per_code config (refs P48)"

Task 5: `llm/evidence.py` — evidence block, payload, code-cited sources

Files:

Create: src/llm/evidence.py
Test: tests/llm/test_evidence.py

Interfaces:

Consumes: pfs.families.detect_codes, pfs.valuation.valuation/ValuationRow, llm.links.as_source, llm.index._engine, LlmConfig.duckdb_replica/valuation_years.
Produces:
- ValuationEvidence(codes, families, rows, unpriced) with prompt_block() -> str and payload() -> dict.
- valuation_evidence(question: str, cfg: LlmConfig) -> ValuationEvidence | None (None when no codes detected or the replica cannot be opened).
- code_cited_sources(engine, codes: Sequence[str], *, per_code: int, collection: str = "rules") -> list[dict].
- merge_sources(retrieved: list[dict], extra: list[dict]) -> list[dict] (retrieved order kept; extras appended unless their label already present).
Step 1: Write the failing tests

# tests/llm/test_evidence.py
"""llm.evidence — valuation prompt block, SSE payload, code-cited rule sources."""

from __future__ import annotations

from unittest.mock import MagicMock, patch

from llm.config import LlmConfig
from llm.evidence import ValuationEvidence, code_cited_sources, merge_sources, valuation_evidence
from pfs.valuation import ValuationRow

CFG = LlmConfig(
    ollama_hosts=("http://h1:11434",), embed_model="e", instruct_model="c", embed_dim=768,
    build_ann_index=False, pg_host="x", pg_port=5432, pg_db="llm", pg_user="llm",
    duckdb_replica="/nonexistent/aco.ro.duckdb", valuation_years=4, code_cited_per_code=2,
)


def _row(**kw) -> ValuationRow:
    base = dict(
        code="G0556", description="Adv prim care mgmt lvl 1", vintage="CY2026 final", year=2026,
        proposed=False, status="A", work=0.25, pe_nf=0.22, pe_f=0.06, mp=0.02, total_nf=0.49,
        total_f=0.33, cf=33.4009, pay_nf=16.37, pay_f=11.02, label="[PFS CY2026 Addendum B]",
        citation="90 FR 49266", url="https://www.federalregister.gov/citation/90-FR-49266",
    )
    base.update(kw)
    return ValuationRow(**base)


class TestEvidenceText:
    def test_prompt_block_exact(self):
        ev = ValuationEvidence(codes=("G0556",), families=("APCM",), rows=(_row(),), unpriced=("Z9999",))
        assert ev.prompt_block() == (
            "Valuation (authoritative for RVUs, conversion factors and payments; "
            "national unadjusted, no GPCI):\n"
            '[PFS CY2026 Addendum B] G0556 "Adv prim care mgmt lvl 1" — status A; work 0.25; '
            "PE 0.22 non-fac / 0.06 fac; MP 0.02; total 0.49 non-fac / 0.33 fac; "
            "CF $33.4009 → payment $16.37 non-fac / $11.02 fac\n"
            "Not priced in the indexed fee schedule: Z9999"
        )

    def test_prompt_block_null_components(self):
        ev = ValuationEvidence(("G9999",), (), (_row(code="G9999", status="I", work=None, pe_nf=None, pe_f=None, mp=None, total_nf=None, total_f=None, pay_nf=None, pay_f=None),), ())
        line = ev.prompt_block().splitlines()[1]
        assert "status I; work n/a; PE n/a non-fac / n/a fac" in line and "payment n/a" in line

    def test_payload_shape(self):
        ev = ValuationEvidence(("G0556",), ("APCM",), (_row(), _row(year=2027, proposed=True, vintage="CY2027 proposed", label="[CY2027 NPRM Addendum B]", citation="91 FR 43842", url="https://www.federalregister.gov/citation/91-FR-43842")), ())
        p = ev.payload()
        assert p["type"] == "valuation" and p["codes"] == ["G0556"] and p["families"] == ["APCM"]
        assert p["rows"][0]["pay_nf"] == 16.37 and p["rows"][1]["proposed"] is True
        assert p["provenance"] == [
            {"label": "[PFS CY2026 Addendum B]", "vintage": "CY2026 final", "citation": "90 FR 49266", "url": "https://www.federalregister.gov/citation/90-FR-49266"},
            {"label": "[CY2027 NPRM Addendum B]", "vintage": "CY2027 proposed", "citation": "91 FR 43842", "url": "https://www.federalregister.gov/citation/91-FR-43842"},
        ]
        assert p["unpriced"] == []


class TestValuationEvidence:
    def test_none_when_no_codes(self):
        assert valuation_evidence("what did commenters say about telehealth?", CFG) is None

    @patch("llm.evidence.valuation", return_value=([_row()], []))
    @patch("llm.evidence.duckdb.connect")
    def test_detects_and_queries_replica_read_only(self, mock_connect, mock_val):
        ev = valuation_evidence("How is APCM valued?", CFG)
        mock_connect.assert_called_once_with("/nonexistent/aco.ro.duckdb", read_only=True)
        assert mock_val.call_args.args[1] == ["G0556", "G0557", "G0558"]
        assert mock_val.call_args.kwargs == {"years": 4}
        assert ev is not None and ev.codes == ("G0556", "G0557", "G0558") and ev.families == ("APCM",)
        mock_connect.return_value.close.assert_called_once()

    @patch("llm.evidence.duckdb.connect", side_effect=OSError("no replica"))
    def test_unopenable_replica_yields_none(self, _c, caplog):
        assert valuation_evidence("G0556?", CFG) is None
        assert "valuation evidence skipped" in caplog.text


def _engine(rows):
    engine = MagicMock()
    conn = engine.begin.return_value.__enter__.return_value
    conn.execute.return_value.fetchall.return_value = rows
    return engine, conn


def _rule_md(p_id, codes, date="2024-12-09"):
    return {"kind": "rule", "item_key": "R1", "p_id": str(p_id), "page": "97710", "ordinal": "1",
            "html_url": "https://www.federalregister.gov/d/2024-25382", "fr_volume": "89",
            "date": date, "title": "CY2025 PFS final rule", "codes": codes}


class TestCodeCitedSources:
    def test_sql_and_per_code_cap_and_dedupe(self):
        engine, conn = _engine([
            ("para one G0556", _rule_md(1, "G0556")),
            ("para two G0556", _rule_md(2, "G0556")),
            ("para three G0556", _rule_md(3, "G0556")),          # over the cap for G0556
            ("para four G0557", _rule_md(4, "G0557 G0556")),      # G0557 still under cap
            ("para one again", _rule_md(1, "G0556")),             # duplicate (item_key, p_id)
        ])
        out = code_cited_sources(engine, ["G0556", "G0557"], per_code=2)
        sql = str(conn.execute.call_args.args[0])
        assert "string_to_array" in sql and "langchain_pg_collection" in sql
        assert conn.execute.call_args.args[1]["collection"] == "rules"
        assert conn.execute.call_args.args[1]["codes"] == ["G0556", "G0557"]
        assert [s["snippet"] for s in out] == ["para one G0556", "para two G0556", "para four G0557"]
        assert all(s["kind"] == "rule" and s["score"] == 0.0 for s in out)

    def test_engine_error_yields_empty(self, caplog):
        engine = MagicMock()
        engine.begin.side_effect = RuntimeError("pg down")
        assert code_cited_sources(engine, ["G0556"], per_code=2) == []
        assert "code-cited sources skipped" in caplog.text


class TestMergeSources:
    def test_keeps_order_and_dedupes_by_label(self):
        a = [{"label": "X", "score": 0.9}, {"label": "Y", "score": 0.8}]
        b = [{"label": "Y", "score": 0.0}, {"label": "Z", "score": 0.0}]
        assert [s["label"] for s in merge_sources(a, b)] == ["X", "Y", "Z"]
        assert merge_sources(a, b)[1]["score"] == 0.8

Step 2: Run to verify failure

Run: uv run --no-sync pytest tests/llm/test_evidence.py -q -p no:cacheprovider Expected: FAIL — ModuleNotFoundError: No module named 'llm.evidence'

Step 3: Implement

# src/llm/evidence.py
"""Structured evidence for the chat: code valuations beside the excerpts.

``valuation_evidence`` detects HCPCS/CPT codes (and registered families)
in the question, reads their RVUs from the DuckDB read replica and
attaches conversion factors from ``pfs.rules``. ``code_cited_sources``
pulls rule chunks whose ``codes`` metadata mention those codes straight
from pgvector so the answer can cite where CMS valued them. Both degrade
to "no evidence" with a warning — the chat never fails because of them.
"""

from __future__ import annotations

import logging
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Any, Sequence

import duckdb
from sqlalchemy import text

from llm.config import LlmConfig
from llm.links import as_source
from pfs.families import detect_codes
from pfs.valuation import ValuationRow, valuation

log = logging.getLogger(__name__)

_HEADER = (
    "Valuation (authoritative for RVUs, conversion factors and payments; "
    "national unadjusted, no GPCI):"
)


def _n(v: float | None, digits: int = 2, money: bool = False) -> str:
    if v is None:
        return "n/a"
    s = f"{v:.{digits}f}"
    return f"${s}" if money else s


@dataclass(frozen=True)
class ValuationEvidence:
    codes: tuple[str, ...]
    families: tuple[str, ...]
    rows: tuple[ValuationRow, ...]
    unpriced: tuple[str, ...]

    def prompt_block(self) -> str:
        lines = [_HEADER]
        for r in self.rows:
            lines.append(
                f'{r.label} {r.code} "{r.description}" — status {r.status or "?"}; '
                f"work {_n(r.work)}; PE {_n(r.pe_nf)} non-fac / {_n(r.pe_f)} fac; "
                f"MP {_n(r.mp)}; total {_n(r.total_nf)} non-fac / {_n(r.total_f)} fac; "
                f"CF {_n(r.cf, 4, money=True)} → payment {_n(r.pay_nf, money=True)} non-fac / "
                f"{_n(r.pay_f, money=True)} fac"
            )
        if self.unpriced:
            lines.append("Not priced in the indexed fee schedule: " + ", ".join(self.unpriced))
        return "\n".join(lines)

    def payload(self) -> dict[str, Any]:
        provenance: list[dict[str, str]] = []
        seen: set[str] = set()
        for r in self.rows:
            if r.label not in seen:
                seen.add(r.label)
                provenance.append({"label": r.label, "vintage": r.vintage, "citation": r.citation, "url": r.url})
        return {
            "type": "valuation",
            "codes": list(self.codes),
            "families": list(self.families),
            "rows": [asdict(r) for r in self.rows],
            "provenance": provenance,
            "unpriced": list(self.unpriced),
        }


def _replica_path(cfg: LlmConfig) -> str:
    p = Path(cfg.duckdb_replica)
    if p.is_absolute():
        return str(p)
    from conf import ROOT

    return str(ROOT / p)


def valuation_evidence(question: str, cfg: LlmConfig) -> ValuationEvidence | None:
    det = detect_codes(question)
    if not det.codes:
        return None
    try:
        con = duckdb.connect(_replica_path(cfg), read_only=True)
    except Exception as e:  # noqa: BLE001 — duckdb raises several types
        log.warning("valuation evidence skipped (%s): %s", cfg.duckdb_replica, e)
        return None
    try:
        rows, unpriced = valuation(con, list(det.codes), years=cfg.valuation_years)
    except Exception as e:  # noqa: BLE001
        log.warning("valuation evidence skipped (query): %s", e)
        return None
    finally:
        con.close()
    return ValuationEvidence(det.codes, det.families, tuple(rows), tuple(unpriced))


_CITED_SQL = text(
    "SELECT e.document, e.cmetadata FROM langchain_pg_embedding e "
    "JOIN langchain_pg_collection c ON c.uuid = e.collection_id "
    "WHERE c.name = :collection "
    "AND string_to_array(COALESCE(e.cmetadata->>'codes', ''), ' ') && CAST(:codes AS text[]) "
    "ORDER BY e.cmetadata->>'date' DESC NULLS LAST, e.cmetadata->>'item_key', e.cmetadata->>'p_id' "
    "LIMIT :lim"
)


def code_cited_sources(
    engine: Any, codes: Sequence[str], *, per_code: int, collection: str = "rules"
) -> list[dict]:
    """Chunks in *collection* whose ``codes`` metadata mention any of
    *codes* — at most *per_code* per code, newest rule first, deduped on
    (item_key, p_id). Scores are 0.0: these are additive, not ranked."""
    wanted = [c.upper() for c in codes]
    try:
        with engine.begin() as conn:
            rows = conn.execute(
                _CITED_SQL,
                {"collection": collection, "codes": wanted, "lim": per_code * len(wanted) * 4},
            ).fetchall()
    except Exception as e:  # noqa: BLE001
        log.warning("code-cited sources skipped: %s", e)
        return []
    out: list[dict] = []
    seen: set[tuple[str, str]] = set()
    count: dict[str, int] = {c: 0 for c in wanted}
    for document, md in rows:
        md = {k: str(v) for k, v in (md or {}).items()}
        key = (md.get("item_key", ""), md.get("p_id", ""))
        if key in seen:
            continue
        hit_codes = [c for c in md.get("codes", "").split() if c in count and count[c] < per_code]
        if not hit_codes:
            continue
        seen.add(key)
        for c in hit_codes:
            count[c] += 1
        out.append(as_source(md, document, 0.0))
    return out


def merge_sources(retrieved: list[dict], extra: list[dict]) -> list[dict]:
    labels = {s["label"] for s in retrieved}
    return retrieved + [s for s in extra if s["label"] not in labels]

Step 4: Run to verify pass — uv run --no-sync pytest tests/llm/test_evidence.py -q -p no:cacheprovider → 9 passed.
Step 5: Commit

git add src/llm/evidence.py tests/llm/test_evidence.py
git commit -m "feat(llm): valuation evidence block, SSE payload and code-cited rule sources (refs P48)"

Task 6: `rag.py` — prompt block, merged sources, `valuation` event

Files:

Modify: src/llm/rag.py (_SYSTEM, build_messages, stream_answer)
Test: tests/llm/test_rag.py

Interfaces:

Consumes: llm.evidence.valuation_evidence/code_cited_sources/merge_sources/ValuationEvidence, llm.index._engine.
Produces: build_messages(question, sources, evidence: ValuationEvidence | None = None); stream_answer yields valuation first when evidence exists.
Step 1: Write the failing tests

In tests/llm/test_rag.py add to TestBuildMessages:

    def test_evidence_block_between_excerpts_and_question(self):
        from llm.evidence import ValuationEvidence
        from pfs.valuation import ValuationRow

        row = ValuationRow(code="G0556", description="d", vintage="CY2026 final", year=2026, proposed=False,
                           status="A", work=0.25, pe_nf=0.22, pe_f=0.06, mp=0.02, total_nf=0.49, total_f=0.33,
                           cf=33.4009, pay_nf=16.37, pay_f=11.02, label="[PFS CY2026 Addendum B]",
                           citation="90 FR 49266", url="u")
        ev = ValuationEvidence(("G0556",), ("APCM",), (row,), ())
        msgs = build_messages("how much?", [], evidence=ev)
        user = msgs[1]["content"]
        assert user.index("Excerpts:") < user.index("Valuation (authoritative") < user.index("Question: how much?")
        assert "[PFS CY2026 Addendum B] G0556" in user
        assert "valuation" in msgs[0]["content"].lower()
        assert "do not compute" in msgs[0]["content"].lower()

    def test_no_evidence_prompt_unchanged(self):
        assert build_messages("q", []) == build_messages("q", [], evidence=None)
        assert "Valuation" not in build_messages("q", [])[1]["content"]

In TestStreamAnswer, patch llm.rag.valuation_evidence to return None in the existing four tests (add @patch("llm.rag.valuation_evidence", return_value=None) as the outermost decorator and a leading _ev parameter) so they stay deterministic, and add:

    @patch("llm.rag._engine")
    @patch("llm.rag.code_cited_sources")
    @patch("llm.rag.valuation_evidence")
    @patch("llm.rag.httpx.Client")
    @patch("llm.rag.retrieve")
    def test_valuation_event_first_and_sources_merged(self, mock_retrieve, MockClient, mock_ev, mock_cited, mock_engine):
        from llm.evidence import ValuationEvidence

        src = {"id": "C1", "label": "C1", "kind": "comment", "snippet": "s", "score": 0.1}
        cited = {"id": "89 FR 97710 ¶3", "label": "89 FR 97710 ¶3", "kind": "rule", "snippet": "G0556", "score": 0.0}
        mock_retrieve.return_value = [src]
        mock_cited.return_value = [cited]
        mock_ev.return_value = ValuationEvidence(("G0556",), ("APCM",), (), ())
        client = MockClient.return_value.__enter__.return_value
        resp = client.stream.return_value.__enter__.return_value
        resp.iter_lines.return_value = iter(['{"message":{"content":"x"},"done":true}'])

        events = list(stream_answer("APCM?", cfg=CFG, pool=self._pool()))

        assert events[0]["type"] == "valuation" and events[0]["codes"] == ["G0556"]
        assert events[1] == {"type": "token", "text": "x"}
        assert events[-2]["sources"] == [src, cited]
        mock_cited.assert_called_once_with(mock_engine.return_value, ("G0556",), per_code=CFG.code_cited_per_code)
        body = client.stream.call_args.kwargs["json"]
        assert "Valuation (authoritative" in body["messages"][1]["content"]

Step 2: Run to verify failure — uv run --no-sync pytest tests/llm/test_rag.py -q -p no:cacheprovider → FAIL (TypeError: build_messages() got an unexpected keyword argument 'evidence', AttributeError: llm.rag has no attribute valuation_evidence).
Step 3: Implement

_SYSTEM +=

    " A Valuation section may follow the excerpts. Its rows are "
    "authoritative for RVUs, conversion factors and payment amounts; when "
    "you state any of those numbers, cite the row's bracketed label exactly, "
    "e.g. [PFS CY2026 Addendum B]. Do not compute, extrapolate or convert "
    "numbers beyond what the rows show; if a code is listed as not priced, "
    "say so."

Imports: from llm.evidence import ValuationEvidence, code_cited_sources, merge_sources, valuation_evidence and from llm.index import _engine.

def build_messages(
    question: str, sources: list[dict], evidence: ValuationEvidence | None = None
) -> list[dict]:
    ...  # context as today
    parts = [f"Excerpts:\n\n{context}"]
    if evidence is not None:
        parts.append(evidence.prompt_block())
    parts.append(f"Question: {question}")
    user = "\n\n".join(parts)
    ...


def stream_answer(question, *, cfg, pool, since=""):
    sources = retrieve(question, cfg=cfg, pool=pool, since=since)
    evidence = valuation_evidence(question, cfg)
    if evidence is not None:
        cited = code_cited_sources(_engine(cfg), evidence.codes, per_code=cfg.code_cited_per_code)
        sources = merge_sources(sources, cited)
        yield evidence.payload()
    pool.check(cfg.instruct_model)
    messages = build_messages(question, sources, evidence)
    ...  # unchanged

Update the module docstring's event list.

Step 4: Run to verify pass — uv run --no-sync pytest tests/llm -q -p no:cacheprovider → pass.
Step 5: Commit

git add src/llm/rag.py tests/llm/test_rag.py
git commit -m "feat(llm): chat cites code valuations — Valuation prompt block, merged code-cited sources, valuation SSE event (refs P48)"

Task 7: `chat.html` — render the valuation table

Files:

Modify: src/llm/web/chat.html
Test: tests/llm/test_api.py
Step 1: Write the failing test (extend test_serves_chat_page)

    def test_serves_chat_page_with_valuation_renderer(self):
        r = client.get("/")
        assert r.status_code == 200
        html = r.text
        assert "function renderValuation" in html
        assert "ev.type === 'valuation'" in html
        assert "table.valuation" in html

Step 2: Run to verify failure — uv run --no-sync pytest tests/llm/test_api.py -q -p no:cacheprovider → FAIL on the new assertions.
Step 3: Implement

CSS (next to details.sources):

    table.valuation {
      border-collapse: collapse; width: 100%; font-size: 13px; margin-top: 4px;
      font-variant-numeric: tabular-nums;
    }
    table.valuation th, table.valuation td {
      border: 1px solid var(--border); padding: 4px 6px; text-align: right; white-space: nowrap;
    }
    table.valuation th { font-family: var(--font-mono); font-size: 10px; letter-spacing: .08em;
      text-transform: uppercase; color: var(--muted-fg); }
    table.valuation td:nth-child(-n+3), table.valuation th:nth-child(-n+3) { text-align: left; }
    table.valuation tr.proposed td { font-style: italic; opacity: .85; }
    .valuation-wrap { overflow-x: auto; }
    .provenance { font-size: 12px; color: var(--muted-fg); margin-top: 4px; }
    .provenance .cite { margin-right: 4px; }

JS (next to renderSources):

    const VAL_COLS = [
      ['code', 'Code'], ['description', 'Description'], ['vintage', 'Vintage'], ['status', 'St'],
      ['work', 'Work'], ['pe_nf', 'PE NF'], ['pe_f', 'PE F'], ['mp', 'MP'],
      ['total_nf', 'Total NF'], ['total_f', 'Total F'], ['cf', 'CF'], ['pay_nf', 'Pay NF'], ['pay_f', 'Pay F'],
    ];
    function fmt(key, v) {
      if (v === null || v === undefined) return '—';
      if (key === 'cf') return '$' + Number(v).toFixed(4);
      if (key === 'pay_nf' || key === 'pay_f') return '$' + Number(v).toFixed(2);
      if (typeof v === 'number') return v.toFixed(2);
      return String(v);
    }
    function renderValuation(wrap, ev) {
      const rows = ev.rows || [];
      const box = document.createElement('div');
      box.className = 'valuation-wrap';
      if (rows.length) {
        const t = document.createElement('table');
        t.className = 'valuation';
        const thead = t.createTHead().insertRow();
        for (const [, name] of VAL_COLS) { const th = document.createElement('th'); th.textContent = name; thead.appendChild(th); }
        const tb = t.createTBody();
        for (const r of rows) {
          const tr = tb.insertRow();
          if (r.proposed) tr.className = 'proposed';
          for (const [key] of VAL_COLS) tr.insertCell().textContent = fmt(key, r[key]);
        }
        box.appendChild(t);
      }
      const prov = document.createElement('div');
      prov.className = 'provenance';
      for (const p of ev.provenance || []) {
        const c = document.createElement('span'); c.className = 'cite'; c.textContent = p.label;
        prov.appendChild(c);
        const a = document.createElement('a'); a.href = p.url; a.target = '_blank'; a.rel = 'noopener';
        a.textContent = p.citation || p.vintage;
        prov.appendChild(a);
        prov.appendChild(document.createTextNode(' '));
      }
      if ((ev.unpriced || []).length) {
        prov.appendChild(document.createTextNode('not priced: ' + ev.unpriced.join(', ')));
      }
      box.appendChild(prov);
      wrap.appendChild(box);
      log.scrollTop = log.scrollHeight;
    }

Dispatch: add else if (ev.type === 'valuation') renderValuation(wrap, ev); after the token branch. The table lands under the bubble (before the sources drawer that arrives later).

Step 4: Run to verify pass — uv run --no-sync pytest tests/llm/test_api.py -q -p no:cacheprovider → pass. Also open the page locally (uv run stack llm serve --port 8010, then curl -s localhost:8010/ | grep -c renderValuation) to be sure the HTML is well-formed (no JS syntax error in the browser console if you can open it).
Step 5: Commit

git add src/llm/web/chat.html tests/llm/test_api.py
git commit -m "feat(llm): chat UI renders the valuation table with FR provenance links (refs P48)"

Task 8: compose mount, rollout, live probes

Files:

Modify: compose.yml (llm service)
Step 1: compose

Add to the llm service:

    volumes:
      - ./data:/app/data:ro   # DuckDB read replica for code valuations (directory mount survives replica re-publish)
    environment:
      ...
      - LLM_DUCKDB_REPLICA=/app/data/aco.ro.duckdb

Run docker compose config --services >/dev/null && echo ok. Commit: chore(compose): llm reads the DuckDB replica read-only (refs P48).

Step 2: Full suite — uv run --no-sync pytest tests -q -p no:cacheprovider -n auto → all pass.
Step 3: Rollout (from /home/kert/stack on main, after merge)

set -a; . ./.env; set +a
uv run stack llm index --collection rules --force          # stamp codes metadata on the 80 rules (~15 min)
# roll the container: COMMIT_SHA in .env → HEAD, then
docker compose build llm && docker compose up -d llm && docker compose ps llm
docker exec llm curl -s http://localhost:8000/health
# probes
docker exec llm curl -s -N -X POST http://localhost:8000/chat -H 'Content-Type: application/json' \
  -d '{"question":"What is APCM and how is it valued?"}' | tee .state/llm/probe-apcm.log | grep -c '"type": "valuation"'
grep -o '"code": "G055[678]"' .state/llm/probe-apcm.log | sort | uniq -c            # 3 codes × 3 vintages
grep -o '\[PFS CY2026 Addendum B\]\|\[CY2027 NPRM Addendum B\]' .state/llm/probe-apcm.log | sort | uniq -c   # cited in prose
grep -c '"kind": "rule"' .state/llm/probe-apcm.log                                # code-cited paragraphs present
docker exec llm curl -s -N -X POST http://localhost:8000/chat -H 'Content-Type: application/json' \
  -d '{"question":"What did commenters say about telehealth?"}' | grep -c '"type": "valuation"'   # must be 0

Record the probe results (counts + the first prose paragraph) on the tracker issue for P48 and update the memory note llm_service_whole_library.md with the valuation feature and the LLM_DUCKDB_REPLICA contract.

52 KiB Raw Blame History Unescape Escape

LLM Chat Code Valuation Implementation Plan

Global Constraints

File Structure

Task 1: pfs/families.py — registry and detection

Task 2: pfs/valuation.py — rows by vintage from the replica

Task 3: codes chunk metadata + links.as_source

Task 4: Config knobs

Task 5: llm/evidence.py — evidence block, payload, code-cited sources

Task 6: rag.py — prompt block, merged sources, valuation event

Task 7: chat.html — render the valuation table

Task 8: compose mount, rollout, live probes

52 KiB

Raw Blame History

Task 1: `pfs/families.py` — registry and detection

Task 2: `pfs/valuation.py` — rows by vintage from the replica

Task 3: `codes` chunk metadata + `links.as_source`

Task 5: `llm/evidence.py` — evidence block, payload, code-cited sources

Task 6: `rag.py` — prompt block, merged sources, `valuation` event

Task 7: `chat.html` — render the valuation table