llm: code valuation evidence in chat (APCM etc.) — registry, replica lookup, codes metadata, valuation event + table #682

New Issue

kert · 2026-09-09T02:24:14Z

kert commented

2026-09-09 02:24:14 +00:00

Implements docs/superpowers/specs/2026-09-08-llm-chat-code-valuation-design.md per docs/superpowers/plans/2026-09-08-llm-chat-code-valuation.md. Rollout: force re-index rules, mount ./data ro into llm, live APCM probe + control.

kert added this to the P48: LLM chat code valuation — RVUs and payments beside the references milestone 2026-09-09 02:24:14 +00:00

kert commented

2026-09-09 03:37:21 +00:00

P48 residuals parked from the final re-review (minor):

llm/evidence.py replica cache: on a republish (mtime change) the parent connection is closed while another thread may still hold a cursor; that one turn logs "valuation evidence skipped" — needs refcounting or a grace period
the codes GIN + collection_id indexes ride in _HNSW_DDL, so build_ann_index = false would also skip them; split into their own ensure_indexes
_UNKNOWN_YEARS_SQL runs one extra pfs.rvu scan per code-bearing chat turn (silent today)
notebooks/palliative_care_rfi.py still carries its own family registry copy — import pfs.families.FAMILIES
pfs/valuation.py round() is banker's rounding (CMS rounds half-up); unreachable with float RVUs, add a comment
stream_answer docstring should mention the silent degrade-to-no-evidence path

P48 residuals parked from the final re-review (minor): - [ ] `llm/evidence.py` replica cache: on a republish (mtime change) the parent connection is closed while another thread may still hold a cursor; that one turn logs "valuation evidence skipped" — needs refcounting or a grace period - [ ] the `codes` GIN + collection_id indexes ride in `_HNSW_DDL`, so `build_ann_index = false` would also skip them; split into their own `ensure_indexes` - [ ] `_UNKNOWN_YEARS_SQL` runs one extra `pfs.rvu` scan per code-bearing chat turn (silent today) - [ ] `notebooks/palliative_care_rfi.py` still carries its own family registry copy — import `pfs.families.FAMILIES` - [ ] `pfs/valuation.py` `round()` is banker's rounding (CMS rounds half-up); unreachable with float RVUs, add a comment - [ ] `stream_answer` docstring should mention the silent degrade-to-no-evidence path

kert commented

2026-09-09 03:55:06 +00:00

P48 rolled out (main @ d7ba609; llm container fhirworx/llm:ac06d7d, rolled 2026-09-08 23:51 ET).

Live probes (SSE, in-container):

"What is APCM and how is it valued?" → 1 valuation event: G0556/G0557/G0558 × {CY2025 final, CY2026 final, CY2027 proposed} = 9 rows; prose cites [PFS CY2025 Addendum B], [PFS CY2026 Addendum B], [CY2027 NPRM Addendum B] (4 each) plus [89 FR 97897 ¶10] and a docket comment; 5 code-cited rule paragraphs merged into the 11 sources; 0 errors.
Control "What did commenters say about telehealth?" → 0 valuation events, normal answer.

Browser probe (Playwright on the gateway network, real page): table rendered with 9 rows, 3 proposed rows flagged, caption present, 3 FR provenance links (89-FR-98452, 90-FR-49266, 91-FR-43842), 11 sources (5 rule), no page errors.

Rollout steps: rules re-indexed with codes metadata (80 rules, 67,340 chunks); DuckDB replica relocated to data/replica/ (only that directory is mounted read-only into the container); GIN + collection_id indexes created by ensure_hnsw.

Final-review fixes landed before merge: payments only for status A/R/T with status notes (bundled/carrier-priced/anesthesia…), FR citations never read as CPT codes, case-insensitive codes, CF labelled non-APM vs QP and CY2024 split, table caption, per-code cited-source window, collection-filtered query, cached replica handle with per-call cursor (thread-safe), prompt-size caps. Residual minors are in the comment above.

**P48 rolled out (main @ d7ba609; llm container `fhirworx/llm:ac06d7d`, rolled 2026-09-08 23:51 ET).** Live probes (SSE, in-container): - "What is APCM and how is it valued?" → 1 `valuation` event: G0556/G0557/G0558 × {CY2025 final, CY2026 final, CY2027 proposed} = 9 rows; prose cites `[PFS CY2025 Addendum B]`, `[PFS CY2026 Addendum B]`, `[CY2027 NPRM Addendum B]` (4 each) plus `[89 FR 97897 ¶10]` and a docket comment; 5 code-cited rule paragraphs merged into the 11 sources; 0 errors. - Control "What did commenters say about telehealth?" → 0 valuation events, normal answer. Browser probe (Playwright on the gateway network, real page): table rendered with 9 rows, 3 proposed rows flagged, caption present, 3 FR provenance links (89-FR-98452, 90-FR-49266, 91-FR-43842), 11 sources (5 rule), no page errors. Rollout steps: rules re-indexed with `codes` metadata (80 rules, 67,340 chunks); DuckDB replica relocated to `data/replica/` (only that directory is mounted read-only into the container); GIN + collection_id indexes created by `ensure_hnsw`. Final-review fixes landed before merge: payments only for status A/R/T with status notes (bundled/carrier-priced/anesthesia…), FR citations never read as CPT codes, case-insensitive codes, CF labelled non-APM vs QP and CY2024 split, table caption, per-code cited-source window, collection-filtered query, cached replica handle with per-call cursor (thread-safe), prompt-size caps. Residual minors are in the comment above.

kert referenced this issue from a commit

2026-09-09 14:32:23 +00:00

merge: P48 LLM chat code valuation — RVUs, CFs and national payments beside the references (refs #682)

kert referenced this issue from a commit

2026-09-09 14:38:37 +00:00

merge: P48 LLM chat code valuation — RVUs, CFs and national payments beside the references (refs #682)

kert commented

2026-09-11 18:55:18 +00:00

Delivered and merged as ac06d7d (P48): valuation SSE event + table, codes chunk metadata, cached replica handle; rolled to the llm image. Closing.

Delivered and merged as ac06d7d (P48): valuation SSE event + table, codes chunk metadata, cached replica handle; rolled to the llm image. Closing.

kert closed this issue

2026-09-11 18:55:18 +00:00