104 KiB
104 KiB
Code-Family Foundation (P49 slice 1: #684–#687) Implementation Plan
For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (
- [ ]) syntax for tracking.
Goal: Turn a PFS code into typed logical elements, dated lineage events and derived families — three DuckDB tables in pfs, every row anchored to a Federal Register paragraph or an RVU-file year — reachable from stack pfs elements|lineage|families.
Architecture: Pure parsers in pfs/elements.py (closed vocabulary, deterministic descriptor parse) and pfs/lineage.py (event detection over pfs.rvu diffs and fr_anchors text) feed thin writers in pfs/codetables.py (DDL + delete-then-insert per code, replica republished by the CLI). The local model is only consulted through an injected classify(text, choices) callable (llm/classify.py) so pfs stays importable without Ollama. pfs/families.py keeps its P48 API and gains derive_families plus an in-place refresh_from(con).
Tech Stack: Python 3.13, DuckDB (write via conf.connect.duckdb_batch, read via the replica), SQLite bib.Store for fr_anchors, typer CLI, pytest, Ollama /api/chat via llm.pool.HostPool (self-hosted only).
Spec: docs/superpowers/specs/2026-09-09-code-family-longitudinal-design.md (§Decisions 1–3, 7; §Components pfs/elements.py, pfs/extract.py, pfs/lineage.py, pfs/families.py).
Global Constraints
pfs/elements.py,pfs/extract.py,pfs/lineage.py,pfs/families.pymust import inside thellmcontainer: no narwhals, no DuckDB at import time, nobibimport at module level (function-level imports only).- Element types are a fixed enum; element values are a closed list; an unknown value goes to
pfs.code_element_review, never intopfs.code_element. - Every
pfs.code_element/pfs.code_eventrow carriesitem_key, p_id, page(FR anchor) orsource='rvu'with ayear; an event without an FR anchor within ±1 rule year isanchored = false. - Inference is self-hosted: the classifier calls
HostPool.acquire_generation()+{host}/api/chatexactly asllm/rag.py:158-170does; no cloud API. - DuckDB writes go through
conf.connect.duckdb_batch("aco"); the CLI callsconf.connect.publish_replica("aco")after writing (single-writer rule). pfs.families.FAMILIES,detect_codes,family_of,find_codeskeep their P48 signatures (the chat depends on them).- Commit after every task; never add a Co-Authored-By trailer.
File map
| File | Responsibility |
|---|---|
src/pfs/elements.py (new) |
ElementType, Element, VOCAB, parse_descriptor() — pure |
src/pfs/descriptors.py (new) |
Where descriptor text comes from: FR paragraph runs (fr_anchors), terminology.hcpcs_level_2, pfs.rvu.description |
src/pfs/extract.py (new) |
extract() = deterministic parse + injected classifier → ElementRow / ReviewRow |
src/llm/classify.py (new) |
closed_vocab_classifier(cfg, pool) → classify(text, choices) -> str | None over Ollama |
src/pfs/codetables.py (new) |
DDL for pfs.code_element, pfs.code_element_review, pfs.code_event, pfs.code_family; write_* helpers |
src/pfs/lineage.py (new) |
rvu_events(), fr_events(), lineage() with cross-check |
src/pfs/families.py (modify) |
derive_families(), load_families(), refresh_from() |
src/cli/pfs.py (new) + src/cli/__init__.py (modify) |
stack pfs elements | lineage | families |
tests/pfs/test_elements.py, test_descriptors.py, test_extract.py, test_codetables.py, test_lineage.py, test_families.py (extend), tests/llm/test_classify.py, tests/cli/test_pfs_cli.py |
one test file per module |
Task 1: Element model and deterministic descriptor parser
Files:
- Create:
src/pfs/elements.py - Test:
tests/pfs/test_elements.py
Interfaces:
-
Produces:
class ElementType(str, Enum): ACTOR="actor"; TIME="time"; PERIOD="period"; POPULATION="population"; ACTIVITY="activity"; MODALITY="modality"; RELATION="relation"; SETTING="setting"; BILLING="billing" @dataclass(frozen=True) class Element: type: ElementType; value: str; detail: str = "" # detail: "20" for time, "99490" for relation targets VOCAB: dict[ElementType, tuple[str, ...]] def parse_descriptor(text: str) -> tuple[Element, ...] # deterministic; sorted, unique def is_element_paragraph(text: str) -> bool # FR element-line heuristic def slug(e: Element) -> str # "time=first-20" / "relation=addon-of:99490" -
Step 1: Write the failing tests
# tests/pfs/test_elements.py
"""pfs.elements — closed element vocabulary + deterministic descriptor parse."""
from __future__ import annotations
from pfs.elements import VOCAB, Element, ElementType, is_element_paragraph, parse_descriptor, slug
CCM_99490 = (
"Chronic care management services, at least 20 minutes of clinical staff time "
"directed by a physician or other qualified health care professional, per calendar "
"month, with the following required elements: multiple (two or more) chronic "
"conditions expected to last at least 12 months, or until the death of the patient; "
"chronic conditions place the patient at significant risk of death, acute "
"exacerbation/decompensation, or functional decline; comprehensive care plan "
"established, implemented, revised, or monitored."
)
G2058 = (
"Chronic care management services, each additional 20 minutes of clinical staff time "
"directed by a physician or other qualified health care professional, per calendar month "
"(List separately in addition to code for primary procedure). (Use G2058 in conjunction "
"with 99490). (Do not report 99490, G2058 in the same calendar month as 99487, 99489, 99491)"
)
APCM_G0556_STEM = (
"Advanced primary care management services provided by clinical staff and directed by a "
"physician or other qualified health care professional who is responsible for all primary "
"care and serves as the continuing focal point for all needed health care services, per "
"calendar month, with the following elements, as appropriate:"
)
def _has(elements, type_, value, detail=None):
return any(
e.type == type_ and e.value == value and (detail is None or e.detail == detail)
for e in elements
)
class TestVocab:
def test_every_type_has_values(self):
for t in ElementType:
assert VOCAB[t], t
def test_values_are_slugs(self):
for values in VOCAB.values():
for v in values:
assert v == v.lower() and " " not in v, v
class TestTimeAndPeriod:
def test_first_20_minutes_per_calendar_month(self):
els = parse_descriptor(CCM_99490)
assert _has(els, ElementType.TIME, "first", "20")
assert _has(els, ElementType.PERIOD, "calendar-month")
def test_each_additional(self):
els = parse_descriptor(G2058)
assert _has(els, ElementType.TIME, "each-additional", "20")
def test_per_30_days(self):
els = parse_descriptor("Chronic care management, at least 20 minutes, per 30 days")
assert _has(els, ElementType.PERIOD, "30-days")
class TestActorAndPopulation:
def test_clinical_staff_directed(self):
els = parse_descriptor(CCM_99490)
assert _has(els, ElementType.ACTOR, "clinical-staff-directed")
def test_personally_by_physician(self):
els = parse_descriptor(
"Chronic care management services, provided personally by a physician or other "
"qualified health care professional, at least 30 minutes, per calendar month"
)
assert _has(els, ElementType.ACTOR, "physician-or-qhp-personally")
def test_two_or_more_chronic_conditions(self):
els = parse_descriptor(CCM_99490)
assert _has(els, ElementType.POPULATION, "multiple-chronic-conditions-12-months")
assert _has(els, ElementType.POPULATION, "significant-risk")
class TestActivityAndRelation:
def test_care_plan_activity(self):
els = parse_descriptor(CCM_99490)
assert _has(els, ElementType.ACTIVITY, "comprehensive-care-plan")
def test_addon_and_exclusions(self):
els = parse_descriptor(G2058)
assert _has(els, ElementType.RELATION, "addon-of", "99490")
for code in ("99487", "99489", "99491"):
assert _has(els, ElementType.RELATION, "not-with", code)
assert _has(els, ElementType.BILLING, "list-separately")
def test_apcm_stem_billing_and_actor(self):
els = parse_descriptor(APCM_G0556_STEM)
assert _has(els, ElementType.ACTOR, "clinical-staff-directed")
assert _has(els, ElementType.PERIOD, "calendar-month")
assert _has(els, ElementType.BILLING, "focal-point-all-primary-care")
class TestModality:
def test_telecommunications(self):
els = parse_descriptor(
"Synchronous audio-video visit for the evaluation and management of a new patient, "
"furnished using an interactive telecommunications system"
)
assert _has(els, ElementType.MODALITY, "interactive-telecommunications")
assert _has(els, ElementType.MODALITY, "audio-video")
def test_audio_only(self):
els = parse_descriptor("Telephone evaluation and management service, audio-only")
assert _has(els, ElementType.MODALITY, "audio-only")
class TestHelpers:
def test_sorted_unique(self):
els = parse_descriptor(G2058 + " " + G2058)
assert list(els) == sorted(set(els), key=lambda e: (e.type.value, e.value, e.detail))
def test_element_paragraph_heuristic(self):
assert is_element_paragraph("Consent;")
assert is_element_paragraph("++ Document in patient's medical record that consent was obtained.")
assert is_element_paragraph("Comprehensive care plan established, implemented, revised, or monitored).")
assert not is_element_paragraph("Comment: Several commenters noted that the CPT Editorial Panel created a new code")
assert not is_element_paragraph("Response: It is our preference to use CPT codes unless Medicare has a programmatic need")
assert not is_element_paragraph("We proposed that HCPCS codes G0556 through G0558 would describe APCM services furnished per calendar month.")
def test_slug(self):
assert slug(Element(ElementType.TIME, "first", "20")) == "time=first:20"
assert slug(Element(ElementType.ACTIVITY, "consent")) == "activity=consent"
- Step 2: Run the tests to verify they fail
Run: uv run pytest tests/pfs/test_elements.py -q
Expected: FAIL — ModuleNotFoundError: No module named 'pfs.elements'
- Step 3: Implement
src/pfs/elements.py
"""Closed, typed vocabulary for the logical elements of a PFS code
descriptor, and a deterministic parser for the parts of a descriptor that
regular expressions can read (minutes, periods, code references, the
recurring actor / population / activity / modality phrases).
CMS treats a code as a bundle of elements — telehealth Step 3 is "Review
the elements of the service as described by the HCPCS code" (90 FR 32389).
Types are fixed here; *values* are a curated list that grows only by
review (an unknown phrase is queued, see ``pfs.extract``). Pure: no I/O,
importable inside the ``llm`` container.
"""
from __future__ import annotations
import re
from dataclasses import dataclass
from enum import Enum
from pfs.families import find_codes
class ElementType(str, Enum):
ACTOR = "actor"
TIME = "time"
PERIOD = "period"
POPULATION = "population"
ACTIVITY = "activity"
MODALITY = "modality"
RELATION = "relation"
SETTING = "setting"
BILLING = "billing"
@dataclass(frozen=True, order=True)
class Element:
type: ElementType
value: str
detail: str = "" # "20" for TIME, the target code for RELATION, "" otherwise
#: Closed value lists. Seeded from the descriptors the FR prints for
#: 99490/99487/99489/99491/99437/99439, 99495/99496, 99497/99498,
#: 99424–99427, G0556–G0558, 99441–99443/98008–98016, G2211, 99457/99458.
VOCAB: dict[ElementType, tuple[str, ...]] = {
ElementType.ACTOR: (
"clinical-staff-directed",
"physician-or-qhp-personally",
"rhc-fqhc",
"care-team",
),
ElementType.TIME: ("first", "each-additional", "at-least", "total"),
ElementType.PERIOD: (
"calendar-month",
"30-days",
"per-visit",
"14-days-post-discharge",
"7-days-post-discharge",
"per-day",
),
ElementType.POPULATION: (
"multiple-chronic-conditions-12-months",
"single-high-risk-condition",
"significant-risk",
"qualified-medicare-beneficiary",
"new-patient",
"established-patient",
"one-or-fewer-chronic-conditions",
),
ElementType.ACTIVITY: (
"comprehensive-care-plan",
"consent",
"initiating-visit",
"24-7-access",
"continuity-of-care",
"alternative-care-delivery",
"comprehensive-care-management",
"needs-assessment",
"preventive-services",
"medication-reconciliation",
"care-transitions",
"community-coordination",
"asynchronous-communication",
"population-data-analysis",
"risk-stratification",
"performance-measurement",
"medical-decision-making",
"interactive-contact-post-discharge",
"advance-directive-discussion",
"device-data-review",
),
ElementType.MODALITY: (
"face-to-face",
"interactive-telecommunications",
"audio-video",
"audio-only",
"asynchronous",
"non-face-to-face",
),
ElementType.RELATION: (
"addon-of",
"not-with",
"replaces",
"replaced-by",
"defined-by-reference-to",
"crosswalk-valued-to",
),
ElementType.SETTING: ("non-facility", "facility", "home", "office"),
ElementType.BILLING: (
"list-separately",
"one-practitioner-per-month",
"focal-point-all-primary-care",
"once-per-period",
),
}
_MIN = r"(\d{1,3})\s*(?:minutes|min)"
_RE_FIRST = re.compile(rf"\b(?:first|initial)\s+{_MIN}", re.I)
_RE_AT_LEAST = re.compile(rf"\bat least\s+{_MIN}", re.I)
_RE_EACH_ADDL = re.compile(rf"\beach additional\s+{_MIN}", re.I)
_RE_TOTAL = re.compile(rf"\btotal (?:time )?(?:of )?{_MIN}", re.I)
_PERIOD_PATTERNS: tuple[tuple[re.Pattern[str], str], ...] = (
(re.compile(r"\bper calendar month\b", re.I), "calendar-month"),
(re.compile(r"\bper 30 days\b", re.I), "30-days"),
(re.compile(r"\bwithin 14 (?:calendar )?days of discharge\b", re.I), "14-days-post-discharge"),
(re.compile(r"\bwithin 7 (?:calendar )?days of discharge\b", re.I), "7-days-post-discharge"),
(re.compile(r"\bper day\b", re.I), "per-day"),
(re.compile(r"\bper visit\b", re.I), "per-visit"),
)
_PHRASES: tuple[tuple[ElementType, str, re.Pattern[str]], ...] = (
(ElementType.ACTOR, "clinical-staff-directed", re.compile(r"clinical staff(?: time)?[^.;]{0,40}directed by a physician", re.I)),
(ElementType.ACTOR, "physician-or-qhp-personally", re.compile(r"(?:provided|furnished) personally by a physician", re.I)),
(ElementType.ACTOR, "rhc-fqhc", re.compile(r"\b(?:RHC|FQHC)s?\b")),
(ElementType.POPULATION, "multiple-chronic-conditions-12-months", re.compile(r"multiple \(two or more\) chronic conditions", re.I)),
(ElementType.POPULATION, "one-or-fewer-chronic-conditions", re.compile(r"one chronic condition[^;]{0,120}or fewer", re.I)),
(ElementType.POPULATION, "significant-risk", re.compile(r"significant risk of death", re.I)),
(ElementType.POPULATION, "qualified-medicare-beneficiary", re.compile(r"qualified medicare beneficiary", re.I)),
(ElementType.POPULATION, "new-patient", re.compile(r"\bnew patient\b", re.I)),
(ElementType.POPULATION, "established-patient", re.compile(r"\bestablished patient\b", re.I)),
(ElementType.ACTIVITY, "comprehensive-care-plan", re.compile(r"comprehensive care plan", re.I)),
(ElementType.ACTIVITY, "consent", re.compile(r"\bconsent\b", re.I)),
(ElementType.ACTIVITY, "initiating-visit", re.compile(r"initiat\w+ (?:during a )?(?:qualifying )?visit", re.I)),
(ElementType.ACTIVITY, "24-7-access", re.compile(r"24/7 access", re.I)),
(ElementType.ACTIVITY, "continuity-of-care", re.compile(r"continuity of care", re.I)),
(ElementType.ACTIVITY, "alternative-care-delivery", re.compile(r"alternative ways to traditional office visits", re.I)),
(ElementType.ACTIVITY, "comprehensive-care-management", re.compile(r"comprehensive care management", re.I)),
(ElementType.ACTIVITY, "needs-assessment", re.compile(r"needs assessment", re.I)),
(ElementType.ACTIVITY, "preventive-services", re.compile(r"preventive services", re.I)),
(ElementType.ACTIVITY, "medication-reconciliation", re.compile(r"medication reconciliation", re.I)),
(ElementType.ACTIVITY, "care-transitions", re.compile(r"care transitions|transitions? of care", re.I)),
(ElementType.ACTIVITY, "community-coordination", re.compile(r"community-based", re.I)),
(ElementType.ACTIVITY, "asynchronous-communication", re.compile(r"asynchronous", re.I)),
(ElementType.ACTIVITY, "population-data-analysis", re.compile(r"patient population data", re.I)),
(ElementType.ACTIVITY, "risk-stratification", re.compile(r"risk stratif", re.I)),
(ElementType.ACTIVITY, "performance-measurement", re.compile(r"performance measurement", re.I)),
(ElementType.ACTIVITY, "medical-decision-making", re.compile(r"medical decision making", re.I)),
(ElementType.ACTIVITY, "interactive-contact-post-discharge", re.compile(r"interactive contact", re.I)),
(ElementType.ACTIVITY, "advance-directive-discussion", re.compile(r"advance directive", re.I)),
(ElementType.ACTIVITY, "device-data-review", re.compile(r"physiologic (?:parameter|data)", re.I)),
(ElementType.MODALITY, "interactive-telecommunications", re.compile(r"interactive telecommunications", re.I)),
(ElementType.MODALITY, "audio-video", re.compile(r"audio-video|audio and video", re.I)),
(ElementType.MODALITY, "audio-only", re.compile(r"audio-only|telephone", re.I)),
(ElementType.MODALITY, "face-to-face", re.compile(r"(?<!non-)face-to-face", re.I)),
(ElementType.MODALITY, "non-face-to-face", re.compile(r"non-face-to-face", re.I)),
(ElementType.SETTING, "home", re.compile(r"\bhome visits?\b", re.I)),
(ElementType.BILLING, "list-separately", re.compile(r"list separately in addition", re.I)),
(ElementType.BILLING, "one-practitioner-per-month", re.compile(r"only one practitioner can furnish", re.I)),
(ElementType.BILLING, "focal-point-all-primary-care", re.compile(r"continuing focal point", re.I)),
)
_RE_IN_CONJUNCTION = re.compile(r"in conjunction with\s+([^)]+)", re.I)
_RE_NOT_WITH = re.compile(r"do not report[^)]*?(?:same (?:calendar )?month|in conjunction|with)\s+(?:as\s+)?([^)]+)", re.I)
_RE_ELEMENTS_OF = re.compile(r"with the elements included in\s+([A-Z]\d{4}|\d{5})", re.I)
def parse_descriptor(text: str) -> tuple[Element, ...]:
"""Every element a regex can read from *text*, sorted and unique."""
found: set[Element] = set()
for pat, value in ((_RE_FIRST, "first"), (_RE_EACH_ADDL, "each-additional"), (_RE_TOTAL, "total")):
for m in pat.finditer(text):
found.add(Element(ElementType.TIME, value, m.group(1)))
for m in _RE_AT_LEAST.finditer(text):
# "at least N minutes" is the first-unit threshold unless an
# each-additional phrase already claims N.
value = "first" if not _RE_EACH_ADDL.search(text) else "at-least"
found.add(Element(ElementType.TIME, value, m.group(1)))
for pat, value in _PERIOD_PATTERNS:
if pat.search(text):
found.add(Element(ElementType.PERIOD, value))
for type_, value, pat in _PHRASES:
if pat.search(text):
found.add(Element(type_, value))
for m in _RE_IN_CONJUNCTION.finditer(text):
for code in find_codes(m.group(1)):
found.add(Element(ElementType.RELATION, "addon-of", code))
for m in _RE_NOT_WITH.finditer(text):
for code in find_codes(m.group(1)):
found.add(Element(ElementType.RELATION, "not-with", code))
for m in _RE_ELEMENTS_OF.finditer(text):
found.add(Element(ElementType.RELATION, "defined-by-reference-to", m.group(1).upper()))
return tuple(sorted(found, key=lambda e: (e.type.value, e.value, e.detail)))
_RE_PROSE_START = re.compile(r"^(?:Comment|Response|We|In the|After|For|As|The|CMS|Section)\b")
def is_element_paragraph(text: str) -> bool:
"""True for the short, fragment-like lines the FR prints inside a
descriptor (``Consent;``, ``++ Document …``), false for prose."""
t = text.strip()
if not t or len(t) > 600:
return False
if t.startswith("++"):
return True
if _RE_PROSE_START.match(t):
return False
return t.endswith((";", ")", ").", ":")) or (t[0].islower() is False and t.count(". ") <= 1 and len(t) < 300)
def slug(e: Element) -> str:
return f"{e.type.value}={e.value}" + (f":{e.detail}" if e.detail else "")
- Step 4: Run the tests to verify they pass
Run: uv run pytest tests/pfs/test_elements.py -q
Expected: all PASS. If test_element_paragraph_heuristic fails on the "Comprehensive care plan … monitored)." line, the endswith tuple already covers ")." — check the prose-start regex did not match; fix the regex, not the test.
- Step 5: Commit
git add src/pfs/elements.py tests/pfs/test_elements.py
git commit -m "feat(pfs): closed element vocabulary + deterministic descriptor parser (refs #684)"
Task 2: Descriptor sources — FR paragraph runs, HCPCS long descriptors, RVU short descriptors
Files:
- Create:
src/pfs/descriptors.py - Test:
tests/pfs/test_descriptors.py
Interfaces:
-
Consumes:
pfs.elements.is_element_paragraph,bib.Store._con()(SQLitefr_anchors,fr_anchor_docs,items), DuckDBterminology.hcpcs_level_2,pfs.rvu. -
Produces:
@dataclass(frozen=True) class Para: item_key: str; p_id: int; page: int; text: str @dataclass(frozen=True) class DescriptorRun: code: str; item_key: str; rule_year: int; stem: Para; elements: tuple[Para, ...] @property def text(self) -> str # stem + " " + "; ".join(element texts) def rule_year_of(title: str, date_published: str) -> int def descriptor_runs(store, code: str, *, max_elements: int = 40) -> list[DescriptorRun] # every FR paragraph that prints "<code> (" / "code <code> (" as a stem def hcpcs_long_description(con, code: str) -> str # terminology.hcpcs_level_2 or "" def rvu_descriptions(con, code: str) -> list[tuple[int, str, str]] # (year, status_code, description) base rows ascending -
Step 1: Write the failing tests
# tests/pfs/test_descriptors.py
"""pfs.descriptors — where descriptor text comes from."""
from __future__ import annotations
import sqlite3
import duckdb
import pytest
from pfs.descriptors import (
DescriptorRun,
descriptor_runs,
hcpcs_long_description,
rule_year_of,
rvu_descriptions,
)
class _Store:
"""Minimal stand-in for bib.Store: only ``_con()`` is used."""
def __init__(self) -> None:
self.con = sqlite3.connect(":memory:")
self.con.row_factory = sqlite3.Row
self.con.executescript(
"""
CREATE TABLE items (key TEXT PRIMARY KEY, title TEXT, date_published TEXT);
CREATE TABLE fr_anchors (item_key TEXT, p_id INTEGER, page INTEGER, ordinal INTEGER, text TEXT);
"""
)
def _con(self):
return self.con
@pytest.fixture
def store():
s = _Store()
s.con.execute("INSERT INTO items VALUES ('DE2VH9PD', 'Medicare Program; CY 2015 PFS Final Rule', '2014-11-13')")
rows = [
("DE2VH9PD", 1243, 67716, "Comment: commenters noted the CPT panel created a code."),
("DE2VH9PD", 1244, 67716, "These commenters suggested that we use the new CPT code 99490 (Chronic care management services, at least 20 minutes of clinical staff time directed by a physician or other qualified health care professional, per calendar month, with the following required elements:"),
("DE2VH9PD", 1245, 67716, "Multiple (two or more) chronic conditions expected to last at least 12 months, or until the death of the patient;"),
("DE2VH9PD", 1246, 67716, "Chronic conditions place the patient at significant risk of death, acute exacerbation/decompensation, or functional decline;"),
("DE2VH9PD", 1247, 67716, "Comprehensive care plan established, implemented, revised, or monitored)."),
("DE2VH9PD", 1248, 67716, "Many of these commenters expressed a preference for the per calendar month period."),
("DE2VH9PD", 1249, 67716, "Response: It is our preference to use CPT codes."),
]
s.con.executemany("INSERT INTO fr_anchors VALUES (?,?,?,?,?)", [(k, p, pg, p, t) for k, p, pg, t in rows])
return s
@pytest.fixture
def con():
c = duckdb.connect(":memory:")
c.execute("CREATE SCHEMA terminology; CREATE SCHEMA pfs")
c.execute("CREATE TABLE terminology.hcpcs_level_2 (hcpcs VARCHAR, seqnum VARCHAR, recid VARCHAR, long_description VARCHAR, short_description VARCHAR)")
c.execute("INSERT INTO terminology.hcpcs_level_2 VALUES ('G0556','10','3','Advanced primary care management services … per calendar month','Adv prim care mgmt lvl 1')")
c.execute("CREATE TABLE pfs.rvu (hcpcs VARCHAR, mod VARCHAR, description VARCHAR, status_code VARCHAR, year INTEGER)")
c.executemany(
"INSERT INTO pfs.rvu VALUES (?,?,?,?,?)",
[
("99490", None, "Chron care mgmt srvc 20 min", "A", 2015),
("99490", "26", "ignored modifier row", "A", 2015),
("99490", "", "Chrnc care mgmt staff 1st 20", "A", 2022),
],
)
return c
class TestRuleYear:
def test_from_title(self):
assert rule_year_of("Medicare Program; CY 2015 PFS Final Rule", "2014-11-13") == 2015
assert rule_year_of("Medicare and Medicaid Programs; CY 2027 Payment Policies", "2026-07-16") == 2027
def test_fallback_to_date_plus_one(self):
assert rule_year_of("Revisions to Payment Policies", "2017-07-21") == 2018
class TestDescriptorRuns:
def test_stem_and_following_elements(self, store):
runs = descriptor_runs(store, "99490")
assert len(runs) == 1
run = runs[0]
assert isinstance(run, DescriptorRun)
assert run.item_key == "DE2VH9PD" and run.rule_year == 2015
assert run.stem.p_id == 1244
assert [p.p_id for p in run.elements] == [1245, 1246, 1247]
assert "per calendar month" in run.text and "Comprehensive care plan" in run.text
def test_no_stem_no_run(self, store):
assert descriptor_runs(store, "99491") == []
def test_max_elements_cap(self, store):
run = descriptor_runs(store, "99490", max_elements=2)[0]
assert [p.p_id for p in run.elements] == [1245, 1246]
class TestDuckDBSources:
def test_long_description(self, con):
assert hcpcs_long_description(con, "G0556").startswith("Advanced primary care")
assert hcpcs_long_description(con, "99490") == ""
def test_rvu_descriptions_base_rows_only(self, con):
assert rvu_descriptions(con, "99490") == [
(2015, "A", "Chron care mgmt srvc 20 min"),
(2022, "A", "Chrnc care mgmt staff 1st 20"),
]
- Step 2: Run the tests to verify they fail
Run: uv run pytest tests/pfs/test_descriptors.py -q
Expected: FAIL — ModuleNotFoundError: No module named 'pfs.descriptors'
- Step 3: Implement
src/pfs/descriptors.py
"""Where a code's descriptor text comes from.
1. The Federal Register prints a descriptor as a *stem* paragraph
("…CPT code 99490 (Chronic care management services … with the
following required elements:") followed by one paragraph per element
(``fr_anchors`` keeps them as consecutive ``p_id`` rows — CY2015 final
rule ¶1244–1247 for 99490, CY2025 final rule ¶1163–1185 for G0556).
2. ``terminology.hcpcs_level_2.long_description`` for HCPCS level II
codes (G-codes carry the full element list; CPT long descriptors are
not distributed).
3. ``pfs.rvu.description`` — the short descriptor per year (tracks
renames such as "Chron care mgmt srvc 20 min" → "Chrnc care mgmt
staff 1st 20").
Function-level ``bib`` access only (a ``Store``-like object exposing
``_con()``) so this module imports without SQLite files present.
"""
from __future__ import annotations
import re
from dataclasses import dataclass
from typing import Any
from pfs.elements import is_element_paragraph
@dataclass(frozen=True)
class Para:
item_key: str
p_id: int
page: int
text: str
@dataclass(frozen=True)
class DescriptorRun:
code: str
item_key: str
rule_year: int
stem: Para
elements: tuple[Para, ...]
@property
def text(self) -> str:
return " ".join([self.stem.text, *(p.text for p in self.elements)])
_RE_CY = re.compile(r"\bCY\s?(\d{4})\b")
def rule_year_of(title: str, date_published: str) -> int:
"""The payment year a rule governs: ``CY nnnn`` in the title, else the
publication year + 1 (PFS rules publish in the second half of the
prior year)."""
m = _RE_CY.search(title or "")
if m:
return int(m.group(1))
return int((date_published or "1970")[:4]) + 1
def _stem_pattern(code: str) -> re.Pattern[str]:
# "code 99490 (Chronic…", "CPT code 99490 (", "HCPCS code G0556 ( Advanced…"
return re.compile(rf"\b{re.escape(code)}\s*\(\s*[A-Z]", re.I)
def descriptor_runs(store: Any, code: str, *, max_elements: int = 40) -> list[DescriptorRun]:
"""Every FR paragraph that opens *code*'s descriptor, with the element
paragraphs that follow it (stops at the first prose paragraph)."""
con = store._con()
pat = _stem_pattern(code.upper())
stems = con.execute(
"SELECT a.item_key, a.p_id, a.page, a.text, i.title, i.date_published "
"FROM fr_anchors a JOIN items i ON i.key = a.item_key "
"WHERE a.text LIKE ? ORDER BY i.date_published, a.p_id",
(f"%{code.upper()}%",),
).fetchall()
runs: list[DescriptorRun] = []
for row in stems:
if not pat.search(row["text"]):
continue
stem = Para(row["item_key"], row["p_id"], row["page"], row["text"])
following = con.execute(
"SELECT p_id, page, text FROM fr_anchors WHERE item_key = ? AND p_id > ? "
"ORDER BY p_id LIMIT ?",
(stem.item_key, stem.p_id, max_elements),
).fetchall()
elements: list[Para] = []
for f in following:
if not is_element_paragraph(f["text"]):
break
elements.append(Para(stem.item_key, f["p_id"], f["page"], f["text"]))
if f["text"].rstrip().endswith((")", ").")):
break # closing parenthesis ends the descriptor
runs.append(
DescriptorRun(
code=code.upper(),
item_key=stem.item_key,
rule_year=rule_year_of(row["title"], row["date_published"]),
stem=stem,
elements=tuple(elements),
)
)
return runs
def hcpcs_long_description(con: Any, code: str) -> str:
row = con.execute(
"SELECT long_description FROM terminology.hcpcs_level_2 WHERE hcpcs = ? LIMIT 1",
[code.upper()],
).fetchone()
return (row[0] or "") if row else ""
def rvu_descriptions(con: Any, code: str) -> list[tuple[int, str, str]]:
"""(year, status_code, description) for the base (no-modifier) row of
each year, ascending."""
rows = con.execute(
"SELECT year, status_code, description FROM pfs.rvu "
"WHERE hcpcs = ? AND (mod IS NULL OR mod = '') "
"QUALIFY row_number() OVER (PARTITION BY year ORDER BY mod NULLS FIRST) = 1 "
"ORDER BY year",
[code.upper()],
).fetchall()
return [(int(y), s or "", d or "") for y, s, d in rows]
- Step 4: Run the tests to verify they pass
Run: uv run pytest tests/pfs/test_descriptors.py -q
Expected: all PASS. Note test_max_elements_cap expects the cap to stop at 2 elements even though ¶1247 would end the run — LIMIT ? on the SQL handles it.
- Step 5: Commit
git add src/pfs/descriptors.py tests/pfs/test_descriptors.py
git commit -m "feat(pfs): descriptor sources — FR stem+element paragraph runs, HCPCS long and RVU short descriptors (refs #685)"
Task 3: Closed-vocabulary classifier over the local model
Files:
- Create:
src/llm/classify.py - Test:
tests/llm/test_classify.py
Interfaces:
-
Consumes:
llm.config.LlmConfig(instruct_model,chat_num_ctx),llm.pool.HostPool(acquire_generation(),check()),llm.pool.pick_model. -
Produces:
Classifier = Callable[[str, Sequence[str]], str | None] def build_prompt(text: str, choices: Sequence[str]) -> list[dict] # system+user messages def parse_choice(reply: str, choices: Sequence[str]) -> str | None # exact slug or None ("none"/unknown) def closed_vocab_classifier(cfg: LlmConfig, pool: HostPool, *, post=None) -> Classifier # post: injectable HTTP post for tests -
Step 1: Write the failing tests
# tests/llm/test_classify.py
"""llm.classify — one-of-N choice from the local model, never free text."""
from __future__ import annotations
from llm.classify import build_prompt, closed_vocab_classifier, parse_choice
class TestParseChoice:
def test_exact(self):
assert parse_choice("consent", ["consent", "24-7-access"]) == "consent"
def test_quoted_and_cased(self):
assert parse_choice(' "24-7-Access".\n', ["consent", "24-7-access"]) == "24-7-access"
def test_none_and_garbage(self):
assert parse_choice("none", ["consent"]) is None
assert parse_choice("I think it is about consent and access", ["consent", "24-7-access"]) is None
class TestPrompt:
def test_prompt_lists_choices_and_none(self):
msgs = build_prompt("Provide 24/7 access for urgent needs", ["consent", "24-7-access"])
assert msgs[0]["role"] == "system"
user = msgs[1]["content"]
assert "consent" in user and "24-7-access" in user and "none" in user
assert "Provide 24/7 access" in user
class _Pool:
def __init__(self):
self.checked = []
def check(self, model):
self.checked.append(model)
return ["http://h"]
class _Ctx:
def __enter__(self):
return "http://h"
def __exit__(self, *a):
return False
def acquire_generation(self):
return self._Ctx()
class _Cfg:
instruct_model = "qwen2.5:14b"
instruct_model_large = ""
large_min_vram_gb = 20.0
chat_num_ctx = 8192
host_vram = {}
class TestClassifier:
def test_returns_choice_from_model_reply(self):
calls = []
def post(url, json):
calls.append((url, json))
return {"message": {"content": "24-7-access"}}
classify = closed_vocab_classifier(_Cfg(), _Pool(), post=post)
assert classify("Provide 24/7 access for urgent needs", ["consent", "24-7-access"]) == "24-7-access"
url, body = calls[0]
assert url == "http://h/api/chat" and body["stream"] is False
assert body["options"]["temperature"] == 0
def test_unparseable_reply_is_none(self):
classify = closed_vocab_classifier(_Cfg(), _Pool(), post=lambda url, json: {"message": {"content": "maybe consent?"}})
assert classify("x", ["consent"]) is None
- Step 2: Run the tests to verify they fail
Run: uv run pytest tests/llm/test_classify.py -q
Expected: FAIL — ModuleNotFoundError: No module named 'llm.classify'
- Step 3: Implement
src/llm/classify.py
"""Closed-vocabulary classification on the local model pool.
Given a text and a list of allowed slugs, the model must answer with
exactly one slug or ``none``. Anything else is treated as ``None`` so a
caller can queue the text for human review instead of inventing a value.
Self-hosted inference only (Ollama ``/api/chat`` on the largest live
host, the same route ``llm.rag.stream_answer`` takes).
"""
from __future__ import annotations
import logging
from typing import Any, Callable, Sequence
import httpx
from llm.pool import pick_model
log = logging.getLogger(__name__)
Classifier = Callable[[str, Sequence[str]], str | None]
_SYSTEM = (
"You label short healthcare-policy text with exactly one slug from a "
"closed list. Reply with the slug only — no words, no punctuation. If no "
"slug fits, reply: none"
)
_TIMEOUT = httpx.Timeout(120.0, connect=10.0)
def build_prompt(text: str, choices: Sequence[str]) -> list[dict]:
listing = "\n".join(f"- {c}" for c in choices)
return [
{"role": "system", "content": _SYSTEM},
{
"role": "user",
"content": f"Allowed slugs:\n{listing}\n- none\n\nText:\n{text.strip()}\n\nSlug:",
},
]
def parse_choice(reply: str, choices: Sequence[str]) -> str | None:
token = (reply or "").strip().strip("\"'`.").strip().lower()
if token == "none":
return None
return token if token in {c.lower() for c in choices} else None
def _default_post(url: str, json: dict) -> dict:
with httpx.Client(timeout=_TIMEOUT) as client:
resp = client.post(url, json=json)
resp.raise_for_status()
return resp.json()
def closed_vocab_classifier(cfg: Any, pool: Any, *, post: Callable[..., dict] | None = None) -> Classifier:
"""Return ``classify(text, choices) -> slug | None`` bound to *pool*."""
send = post or _default_post
def classify(text: str, choices: Sequence[str]) -> str | None:
pool.check(cfg.instruct_model)
with pool.acquire_generation() as host:
model = pick_model(cfg, pool, host)
data = send(
f"{host}/api/chat",
json={
"model": model,
"messages": build_prompt(text, choices),
"stream": False,
"options": {"num_ctx": cfg.chat_num_ctx, "temperature": 0},
},
)
reply = data.get("message", {}).get("content", "")
choice = parse_choice(reply, choices)
if choice is None and reply.strip().lower() != "none":
log.info("classify: unparseable reply %r for %r", reply[:60], text[:60])
return choice
return classify
- Step 4: Run the tests to verify they pass
Run: uv run pytest tests/llm/test_classify.py -q
Expected: all PASS (pick_model reads cfg.instruct_model_large/large_min_vram_gb/host_vram — the _Cfg stub provides them; if pick_model needs another attribute, add it to the stub rather than bypassing pick_model).
- Step 5: Commit
git add src/llm/classify.py tests/llm/test_classify.py
git commit -m "feat(llm): closed-vocabulary classifier on the local model pool (refs #685)"
Task 4: DuckDB tables for elements, review queue, events, families
Files:
- Create:
src/pfs/codetables.py - Test:
tests/pfs/test_codetables.py
Interfaces:
-
Produces:
@dataclass(frozen=True) class ElementRow: code: str; year: int; type: str; value: str; detail: str; text: str; item_key: str; p_id: int; page: int; source: str # source ∈ {"fr","hcpcs","rvu"} @dataclass(frozen=True) class ReviewRow: code: str; text: str; proposed_type: str; proposed_value: str; item_key: str; p_id: int @dataclass(frozen=True) class EventRow: code: str; year: int; kind: str; from_codes: str; to_codes: str; item_key: str; p_id: int; page: int; source: str; anchored: bool; note: str @dataclass(frozen=True) class FamilyRow: key: str; name: str; code: str; role: str; since: int | None; until: int | None; item_key: str; p_id: int def ensure_tables(con) -> None def write_elements(con, code: str, rows: Sequence[ElementRow], reviews: Sequence[ReviewRow]) -> int # delete code's rows, insert; returns rows written def write_events(con, code: str, rows: Sequence[EventRow]) -> int def write_families(con, rows: Sequence[FamilyRow]) -> int # full replace def read_elements(con, code: str) -> list[ElementRow] def read_events(con, code: str) -> list[EventRow] def read_families(con) -> list[FamilyRow] -
Step 1: Write the failing tests
# tests/pfs/test_codetables.py
"""pfs.codetables — DDL + writers for the code element/event/family tables."""
from __future__ import annotations
import duckdb
import pytest
from pfs.codetables import (
ElementRow,
EventRow,
FamilyRow,
ReviewRow,
ensure_tables,
read_elements,
read_events,
read_families,
write_elements,
write_events,
write_families,
)
@pytest.fixture
def con():
c = duckdb.connect(":memory:")
ensure_tables(c)
return c
def _el(code="99490", value="consent", year=2015):
return ElementRow(code, year, "activity", value, "", "Consent;", "DE2VH9PD", 1245, 67716, "fr")
class TestDDL:
def test_idempotent(self, con):
ensure_tables(con)
names = {r[0] for r in con.execute("SELECT table_name FROM information_schema.tables WHERE table_schema='pfs'").fetchall()}
assert {"code_element", "code_element_review", "code_event", "code_family"} <= names
class TestElements:
def test_write_replaces_per_code(self, con):
assert write_elements(con, "99490", [_el(), _el(value="24-7-access")], [ReviewRow("99490", "odd line", "activity", "", "DE2VH9PD", 1246)]) == 2
assert write_elements(con, "99490", [_el()], []) == 1
rows = read_elements(con, "99490")
assert [r.value for r in rows] == ["consent"]
assert con.execute("SELECT count(*) FROM pfs.code_element_review WHERE code='99490'").fetchone()[0] == 0
def test_other_codes_untouched(self, con):
write_elements(con, "99490", [_el()], [])
write_elements(con, "G0556", [_el(code="G0556")], [])
write_elements(con, "99490", [], [])
assert read_elements(con, "99490") == []
assert len(read_elements(con, "G0556")) == 1
class TestEvents:
def test_roundtrip_sorted_by_year(self, con):
rows = [
EventRow("99439", 2021, "replaces", "G2058", "99439", "YBM4IZUS", 1578, 84639, "fr", True, ""),
EventRow("99439", 2021, "appeared", "", "", "", 0, 0, "rvu", True, ""),
]
assert write_events(con, "99439", rows) == 2
got = read_events(con, "99439")
assert [e.kind for e in got] == ["appeared", "replaces"]
assert got[1].anchored is True
class TestFamilies:
def test_full_replace(self, con):
write_families(con, [FamilyRow("CCM", "Chronic Care Management", "99490", "base", 2015, None, "", 0)])
write_families(con, [FamilyRow("CCM", "Chronic Care Management", "99439", "add-on", 2021, None, "YBM4IZUS", 1578)])
fams = read_families(con)
assert len(fams) == 1 and fams[0].code == "99439" and fams[0].role == "add-on"
- Step 2: Run the tests to verify they fail
Run: uv run pytest tests/pfs/test_codetables.py -q
Expected: FAIL — ModuleNotFoundError: No module named 'pfs.codetables'
- Step 3: Implement
src/pfs/codetables.py
"""DDL and writers for the code-family tables in the ``pfs`` schema.
``pfs.code_element`` one row per (code, element) with its FR anchor
``pfs.code_element_review`` element lines the classifier could not place
``pfs.code_event`` dated lineage events (created, replaced, …)
``pfs.code_family`` derived family membership with roles
Writers are delete-then-insert per code (events, elements) or full
replace (families) so a re-run is idempotent. Callers own the
connection (``conf.connect.duckdb_batch`` for writes) and republish the
replica afterwards; nothing here opens a database.
"""
from __future__ import annotations
from dataclasses import astuple, dataclass
from typing import Any, Sequence
@dataclass(frozen=True)
class ElementRow:
code: str
year: int
type: str
value: str
detail: str
text: str
item_key: str
p_id: int
page: int
source: str # "fr" | "hcpcs" | "rvu"
@dataclass(frozen=True)
class ReviewRow:
code: str
text: str
proposed_type: str
proposed_value: str
item_key: str
p_id: int
@dataclass(frozen=True)
class EventRow:
code: str
year: int
kind: str
from_codes: str # space-joined
to_codes: str
item_key: str
p_id: int
page: int
source: str # "fr" | "rvu"
anchored: bool
note: str
@dataclass(frozen=True)
class FamilyRow:
key: str
name: str
code: str
role: str # base | add-on | predecessor | successor | bundle | member
since: int | None
until: int | None
item_key: str
p_id: int
_DDL = """
CREATE SCHEMA IF NOT EXISTS pfs;
CREATE TABLE IF NOT EXISTS pfs.code_element (
code VARCHAR, year INTEGER, type VARCHAR, value VARCHAR, detail VARCHAR,
text VARCHAR, item_key VARCHAR, p_id INTEGER, page INTEGER, source VARCHAR);
CREATE TABLE IF NOT EXISTS pfs.code_element_review (
code VARCHAR, text VARCHAR, proposed_type VARCHAR, proposed_value VARCHAR,
item_key VARCHAR, p_id INTEGER);
CREATE TABLE IF NOT EXISTS pfs.code_event (
code VARCHAR, year INTEGER, kind VARCHAR, from_codes VARCHAR, to_codes VARCHAR,
item_key VARCHAR, p_id INTEGER, page INTEGER, source VARCHAR, anchored BOOLEAN, note VARCHAR);
CREATE TABLE IF NOT EXISTS pfs.code_family (
key VARCHAR, name VARCHAR, code VARCHAR, role VARCHAR, since INTEGER, until INTEGER,
item_key VARCHAR, p_id INTEGER);
"""
def ensure_tables(con: Any) -> None:
for stmt in _DDL.strip().split(";"):
if stmt.strip():
con.execute(stmt)
def _insert(con: Any, table: str, rows: Sequence[Any]) -> int:
if not rows:
return 0
width = len(astuple(rows[0]))
marks = ",".join("?" * width)
con.executemany(f"INSERT INTO {table} VALUES ({marks})", [astuple(r) for r in rows])
return len(rows)
def write_elements(con: Any, code: str, rows: Sequence[ElementRow], reviews: Sequence[ReviewRow]) -> int:
ensure_tables(con)
con.execute("DELETE FROM pfs.code_element WHERE code = ?", [code])
con.execute("DELETE FROM pfs.code_element_review WHERE code = ?", [code])
_insert(con, "pfs.code_element_review", reviews)
return _insert(con, "pfs.code_element", rows)
def write_events(con: Any, code: str, rows: Sequence[EventRow]) -> int:
ensure_tables(con)
con.execute("DELETE FROM pfs.code_event WHERE code = ?", [code])
return _insert(con, "pfs.code_event", rows)
def write_families(con: Any, rows: Sequence[FamilyRow]) -> int:
ensure_tables(con)
con.execute("DELETE FROM pfs.code_family")
return _insert(con, "pfs.code_family", rows)
def read_elements(con: Any, code: str) -> list[ElementRow]:
rows = con.execute(
"SELECT * FROM pfs.code_element WHERE code = ? ORDER BY year, type, value, detail", [code]
).fetchall()
return [ElementRow(*r) for r in rows]
def read_events(con: Any, code: str) -> list[EventRow]:
rows = con.execute(
"SELECT * FROM pfs.code_event WHERE code = ? ORDER BY year, kind", [code]
).fetchall()
return [EventRow(*r) for r in rows]
def read_families(con: Any) -> list[FamilyRow]:
rows = con.execute("SELECT * FROM pfs.code_family ORDER BY key, role, code").fetchall()
return [FamilyRow(*r) for r in rows]
- Step 4: Run the tests to verify they pass
Run: uv run pytest tests/pfs/test_codetables.py -q
Expected: all PASS.
- Step 5: Commit
git add src/pfs/codetables.py tests/pfs/test_codetables.py
git commit -m "feat(pfs): code_element / code_element_review / code_event / code_family tables + writers (refs #684 #686 #687)"
Task 5: Extractor — deterministic parse, then classifier, then review queue
Files:
- Create:
src/pfs/extract.py - Test:
tests/pfs/test_extract.py
Interfaces:
-
Consumes:
pfs.elements.parse_descriptor,pfs.elements.VOCAB,pfs.descriptors.DescriptorRun/Para,pfs.codetables.ElementRow/ReviewRow,Classifierfromllm.classify(type only — injected callable). -
Produces:
@dataclass(frozen=True) class Extraction: code: str; rows: tuple[ElementRow, ...]; reviews: tuple[ReviewRow, ...] def extract_run(run: DescriptorRun, *, classify: Classifier | None = None) -> Extraction def extract_text(code: str, text: str, *, year: int, source: str, classify=None) -> Extraction # hcpcs long description / rvu short def extract_code(store, con, code: str, *, classify=None) -> Extraction # all sources for a code, merged (FR rows win on duplicates) -
Rules: every element the deterministic parser finds on the whole run text becomes a row anchored to the stem paragraph (or to the element paragraph that contains it, when exactly one does). Each element paragraph that yields no deterministic element is sent to
classify(text, choices)withchoices= all ACTIVITY ∪ POPULATION ∪ BILLING values; a returned slug becomes a row (type looked up fromVOCAB),Nonebecomes aReviewRow. Without a classifier, such paragraphs go straight to review. -
Step 1: Write the failing tests
# tests/pfs/test_extract.py
"""pfs.extract — deterministic parse + injected classifier → rows and review queue."""
from __future__ import annotations
from pfs.descriptors import DescriptorRun, Para
from pfs.extract import Extraction, extract_run, extract_text
STEM = Para("JJ6AM5HJ", 1163, 97864,
"HCPCS code G0556 ( Advanced primary care management services provided by clinical staff and directed by a "
"physician or other qualified health care professional who is responsible for all primary care and serves as "
"the continuing focal point for all needed health care services, per calendar month, with the following elements, as appropriate:")
ELS = (
Para("JJ6AM5HJ", 1164, 97864, "Consent;"),
Para("JJ6AM5HJ", 1168, 97864, "Provide 24/7 access for urgent needs to care team/practitioner;"),
Para("JJ6AM5HJ", 1170, 97864, "Deliver care in alternative ways to traditional office visits to best meet the patient's needs, such as home visits and/or expanded hours;"),
Para("JJ6AM5HJ", 1185, 97865, "Be assessed through performance measurement of primary care quality, total cost of care, and meaningful use of Certified EHR Technology)."),
Para("JJ6AM5HJ", 1199, 97865, "Something the vocabulary does not know about at all;"),
)
RUN = DescriptorRun("G0556", "JJ6AM5HJ", 2025, STEM, ELS)
def _vals(x, type_):
return sorted(r.value for r in x.rows if r.type == type_)
class TestDeterministic:
def test_rows_anchor_to_the_paragraph_that_contains_them(self):
x = extract_run(RUN)
assert isinstance(x, Extraction) and x.code == "G0556"
by_value = {r.value: r for r in x.rows}
assert by_value["consent"].p_id == 1164
assert by_value["24-7-access"].p_id == 1168
assert by_value["performance-measurement"].p_id == 1185
assert by_value["clinical-staff-directed"].p_id == 1163 # stem-level
assert by_value["calendar-month"].source == "fr" and by_value["calendar-month"].year == 2025
def test_unknown_paragraph_goes_to_review_without_classifier(self):
x = extract_run(RUN)
assert [r.p_id for r in x.reviews] == [1199]
assert x.reviews[0].proposed_value == ""
def test_home_setting_from_element_line(self):
assert "home" in _vals(extract_run(RUN), "setting")
class TestClassifier:
def test_classifier_places_unknown_line(self):
seen = []
def classify(text, choices):
seen.append((text, tuple(choices)))
return "community-coordination" if "vocabulary" in text else None
x = extract_run(RUN, classify=classify)
assert x.reviews == ()
row = next(r for r in x.rows if r.value == "community-coordination")
assert row.type == "activity" and row.p_id == 1199
assert seen and "consent" in seen[0][1] and "significant-risk" in seen[0][1]
def test_classifier_none_means_review_with_no_proposal(self):
x = extract_run(RUN, classify=lambda t, c: None)
assert len(x.reviews) == 1 and x.reviews[0].proposed_value == ""
def test_classifier_not_called_for_deterministic_lines(self):
calls = []
extract_run(RUN, classify=lambda t, c: calls.append(t) or None)
assert calls == ["Something the vocabulary does not know about at all;"]
class TestText:
def test_hcpcs_long_description(self):
x = extract_text("G0556", "… per calendar month, with the following elements: consent; 24/7 access …", year=2025, source="hcpcs")
assert {r.value for r in x.rows} >= {"calendar-month", "consent", "24-7-access"}
assert all(r.item_key == "" and r.p_id == 0 and r.source == "hcpcs" for r in x.rows)
- Step 2: Run the tests to verify they fail
Run: uv run pytest tests/pfs/test_extract.py -q
Expected: FAIL — ModuleNotFoundError: No module named 'pfs.extract'
- Step 3: Implement
src/pfs/extract.py
"""Element extraction for one code: deterministic parse first, the local
model second (closed vocabulary only), a review queue for the rest.
``extract_run`` works on an FR descriptor run (stem + element
paragraphs); ``extract_text`` on a flat descriptor (HCPCS long
description, RVU short description); ``extract_code`` gathers every
source for a code. No I/O here beyond what the caller hands in.
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Any, Callable, Sequence
from pfs.codetables import ElementRow, ReviewRow
from pfs.descriptors import DescriptorRun, Para, descriptor_runs, hcpcs_long_description, rvu_descriptions
from pfs.elements import VOCAB, Element, ElementType, parse_descriptor
Classifier = Callable[[str, Sequence[str]], "str | None"]
#: Types the classifier may assign to a free-standing element line.
_CLASSIFIABLE = (ElementType.ACTIVITY, ElementType.POPULATION, ElementType.BILLING)
_CHOICES: tuple[str, ...] = tuple(v for t in _CLASSIFIABLE for v in VOCAB[t])
_TYPE_OF: dict[str, ElementType] = {v: t for t in _CLASSIFIABLE for v in VOCAB[t]}
@dataclass(frozen=True)
class Extraction:
code: str
rows: tuple[ElementRow, ...]
reviews: tuple[ReviewRow, ...]
def _row(code: str, e: Element, para: Para, *, year: int, source: str) -> ElementRow:
return ElementRow(code, year, e.type.value, e.value, e.detail, para.text, para.item_key, para.p_id, para.page, source)
def extract_run(run: DescriptorRun, *, classify: Classifier | None = None) -> Extraction:
rows: dict[Element, ElementRow] = {}
reviews: list[ReviewRow] = []
# 1. deterministic, per element paragraph (anchors to that paragraph)
for para in run.elements:
found = parse_descriptor(para.text)
for e in found:
rows.setdefault(e, _row(run.code, e, para, year=run.rule_year, source="fr"))
if found:
continue
# 2. classifier for lines nothing matched
choice = classify(para.text, _CHOICES) if classify else None
if choice is not None and choice in _TYPE_OF:
e = Element(_TYPE_OF[choice], choice)
rows.setdefault(e, _row(run.code, e, para, year=run.rule_year, source="fr"))
else:
reviews.append(ReviewRow(run.code, para.text, "", "", para.item_key, para.p_id))
# 3. deterministic on the stem (and anything the whole text reveals)
for e in parse_descriptor(run.text):
rows.setdefault(e, _row(run.code, e, run.stem, year=run.rule_year, source="fr"))
ordered = sorted(rows.values(), key=lambda r: (r.type, r.value, r.detail))
return Extraction(run.code, tuple(ordered), tuple(reviews))
def extract_text(code: str, text: str, *, year: int, source: str, classify: Classifier | None = None) -> Extraction:
para = Para("", 0, 0, text)
rows = [ElementRow(code, year, e.type.value, e.value, e.detail, text, "", 0, 0, source) for e in parse_descriptor(text)]
return Extraction(code, tuple(rows), ())
def extract_code(store: Any, con: Any, code: str, *, classify: Classifier | None = None) -> Extraction:
"""All sources for *code*; FR rows win over HCPCS/RVU rows for the same
element, and each source contributes its own year."""
code = code.upper()
merged: dict[tuple[str, str, str], ElementRow] = {}
reviews: list[ReviewRow] = []
for run in descriptor_runs(store, code):
x = extract_run(run, classify=classify)
for r in x.rows:
merged.setdefault((r.type, r.value, r.detail), r)
reviews.extend(x.reviews)
long_desc = hcpcs_long_description(con, code)
years = rvu_descriptions(con, code)
if long_desc:
y = years[-1][0] if years else 0
for r in extract_text(code, long_desc, year=y, source="hcpcs").rows:
merged.setdefault((r.type, r.value, r.detail), r)
for year, _status, desc in years:
for r in extract_text(code, desc, year=year, source="rvu").rows:
merged.setdefault((r.type, r.value, r.detail), r)
ordered = sorted(merged.values(), key=lambda r: (r.type, r.value, r.detail))
return Extraction(code, tuple(ordered), tuple(reviews))
- Step 4: Run the tests to verify they pass
Run: uv run pytest tests/pfs/test_extract.py tests/pfs/test_elements.py -q
Expected: all PASS. test_home_setting_from_element_line depends on the SETTING home phrase in Task 1's _PHRASES; test_rows_anchor_to_the_paragraph_that_contains_them requires element-paragraph rows to be inserted before stem rows (step 1 before step 3 — setdefault keeps the first).
- Step 5: Commit
git add src/pfs/extract.py tests/pfs/test_extract.py
git commit -m "feat(pfs): element extractor — deterministic parse, closed-vocab classifier, review queue (refs #685)"
Task 6: Lineage events from RVU-file diffs
Files:
- Create:
src/pfs/lineage.py(part 1) - Test:
tests/pfs/test_lineage.py(part 1)
Interfaces:
-
Consumes: DuckDB
pfs.rvu(base rows),pfs.codetables.EventRow. -
Produces:
RVU_KINDS = ("appeared", "disappeared", "status_change", "descriptor_change", "revalued") def rvu_events(con, code: str, *, revalue_threshold: float = 0.10) -> list[EventRow] # source="rvu", anchored=False, item_key="" (cross-check fills anchored later) def rvu_year_span(con) -> tuple[int, int] -
Rules:
appearedwhen the first year > table min year;disappearedin (last year + 1) when last year < table max year;status_changein the year the base row's status differs from the prior year (note="B→A");descriptor_changelikewise (note="old → new");revaluedwhen|Δ non_fac_total| / prior > threshold(note="+23.4%"). -
Step 1: Write the failing tests
# tests/pfs/test_lineage.py
"""pfs.lineage — dated events from RVU diffs and FR paragraphs, cross-checked."""
from __future__ import annotations
import sqlite3
import duckdb
import pytest
from pfs.lineage import rvu_events, rvu_year_span
RVU_COLS = "hcpcs VARCHAR, mod VARCHAR, description VARCHAR, status_code VARCHAR, non_fac_total DOUBLE, year INTEGER"
@pytest.fixture
def con():
c = duckdb.connect(":memory:")
c.execute("CREATE SCHEMA pfs")
c.execute(f"CREATE TABLE pfs.rvu ({RVU_COLS})")
rows = [
# table spans 2015..2026 (filler code keeps the span honest)
*[("00000", None, "filler", "A", 1.0, y) for y in range(2015, 2027)],
# 99487: B in 2015-2016, A from 2017; description rename 2021; +25% revalue 2022
("99487", None, "Cmplx chron care w/o pt vsit", "B", 0.0, 2015),
("99487", None, "Cmplx chron care w/o pt vsit", "B", 0.0, 2016),
("99487", None, "Cmplx chron care w/o pt vsit", "A", 2.00, 2017),
("99487", None, "Cmplx chron care w/o pt vsit", "A", 2.02, 2018),
("99487", None, "Cmplx chron care w/o pt vsit", "A", 2.05, 2019),
("99487", None, "Cmplx chron care w/o pt vsit", "A", 2.10, 2020),
("99487", None, "Cplx chrnc care 1st 60 min", "A", 2.12, 2021),
("99487", None, "Cplx chrnc care 1st 60 min", "A", 2.65, 2022),
*[("99487", None, "Cplx chrnc care 1st 60 min", "A", 2.65, y) for y in range(2023, 2027)],
("99487", "26", "modifier row ignored", "A", 99.0, 2022),
# G2058: 2020 only
("G2058", None, "Ccm add 20min", "A", 1.0, 2020),
# 99439: 2021 onward
*[("99439", "", "Chrnc care mgmt svc ea addl", "A", 1.0, y) for y in range(2021, 2027)],
]
c.executemany("INSERT INTO pfs.rvu VALUES (?,?,?,?,?,?)", rows)
return c
class TestSpan:
def test_span(self, con):
assert rvu_year_span(con) == (2015, 2026)
class TestRvuEvents:
def test_status_descriptor_and_revalue(self, con):
ev = rvu_events(con, "99487")
kinds = [(e.year, e.kind, e.note) for e in ev]
assert (2017, "status_change", "B→A") in kinds
assert (2021, "descriptor_change", "Cmplx chron care w/o pt vsit → Cplx chrnc care 1st 60 min") in kinds
assert (2022, "revalued", "+25.0%") in kinds
assert not any(k == "appeared" for _, k, _ in kinds) # present from the first table year
assert all(e.source == "rvu" and e.anchored is False and e.code == "99487" for e in ev)
def test_appeared_and_disappeared(self, con):
ev = rvu_events(con, "G2058")
assert [(e.year, e.kind) for e in ev] == [(2020, "appeared"), (2021, "disappeared")]
def test_appeared_only(self, con):
ev = rvu_events(con, "99439")
assert [(e.year, e.kind) for e in ev] == [(2021, "appeared")]
def test_unknown_code(self, con):
assert rvu_events(con, "99999") == []
- Step 2: Run the tests to verify they fail
Run: uv run pytest tests/pfs/test_lineage.py -q
Expected: FAIL — ModuleNotFoundError: No module named 'pfs.lineage'
- Step 3: Implement
src/pfs/lineage.py(RVU half)
"""Lineage events for a code: what happened, when, and where the record
says so.
Two independent sources: the RVU files (``pfs.rvu`` base rows year over
year — appeared / disappeared / status_change / descriptor_change /
revalued) and the Federal Register (``fr_anchors`` paragraphs that name
the code next to an event verb — created / adopted_cpt / replaces /
replaced_by / deleted / crosswalk / bundled / telehealth_list).
``lineage`` merges them and marks each RVU event ``anchored`` when an FR
event for the same code lies within ±1 rule year; the rest are
``unanchored`` for review, never silently accepted.
"""
from __future__ import annotations
import re
from typing import Any
from pfs.codetables import EventRow
RVU_KINDS = ("appeared", "disappeared", "status_change", "descriptor_change", "revalued")
def rvu_year_span(con: Any) -> tuple[int, int]:
lo, hi = con.execute("SELECT min(year), max(year) FROM pfs.rvu").fetchone()
return int(lo), int(hi)
def _base_rows(con: Any, code: str) -> list[tuple[int, str, str, float | None]]:
return [
(int(y), s or "", d or "", t)
for y, s, d, t in con.execute(
"SELECT year, status_code, description, non_fac_total FROM pfs.rvu "
"WHERE hcpcs = ? AND (mod IS NULL OR mod = '') "
"QUALIFY row_number() OVER (PARTITION BY year ORDER BY mod NULLS FIRST) = 1 "
"ORDER BY year",
[code.upper()],
).fetchall()
]
def _ev(code: str, year: int, kind: str, note: str = "") -> EventRow:
return EventRow(code.upper(), year, kind, "", "", "", 0, 0, "rvu", False, note)
def rvu_events(con: Any, code: str, *, revalue_threshold: float = 0.10) -> list[EventRow]:
rows = _base_rows(con, code)
if not rows:
return []
lo, hi = rvu_year_span(con)
out: list[EventRow] = []
first, last = rows[0][0], rows[-1][0]
if first > lo:
out.append(_ev(code, first, "appeared"))
if last < hi:
out.append(_ev(code, last + 1, "disappeared"))
for (py, ps, pd, pt), (y, s, d, t) in zip(rows, rows[1:]):
if s != ps:
out.append(_ev(code, y, "status_change", f"{ps}→{s}"))
if d != pd:
out.append(_ev(code, y, "descriptor_change", f"{pd} → {d}"))
if pt and t is not None and pt > 0:
delta = (t - pt) / pt
if abs(delta) > revalue_threshold:
out.append(_ev(code, y, "revalued", f"{delta:+.1%}"))
return sorted(out, key=lambda e: (e.year, RVU_KINDS.index(e.kind)))
- Step 4: Run the tests to verify they pass
Run: uv run pytest tests/pfs/test_lineage.py -q
Expected: all PASS.
- Step 5: Commit
git add src/pfs/lineage.py tests/pfs/test_lineage.py
git commit -m "feat(pfs): lineage events from RVU-file diffs — appeared/disappeared/status/descriptor/revalued (refs #686)"
Task 7: Lineage events from FR paragraphs + cross-check
Files:
- Modify:
src/pfs/lineage.py - Modify:
tests/pfs/test_lineage.py
Interfaces:
- Consumes:
pfs.descriptors.rule_year_of,pfs.families.find_codes, aStore-like with_con(). - Produces:
FR_KINDS = ("created", "adopted_cpt", "replaces", "replaced_by", "deleted", "crosswalk", "bundled", "telehealth_list") def fr_events(store, code: str) -> list[EventRow] # source="fr", anchored=True, from/to codes filled from the paragraph def lineage(con, store, code: str) -> list[EventRow] # rvu_events ⊕ fr_events, RVU rows anchored when an FR event is within ±1 year - Patterns (case-insensitive, applied to paragraphs whose text contains the code):
| kind | regex on the paragraph |
|---|---|
created |
`\b(?:creat |
adopted_cpt |
`adopt(?:ed |
replaces |
`[^.]{0,120}\b(?:replace |
replaced_by |
`\b(?:replace |
deleted |
`\bdelet(?:ed |
crosswalk |
\bcrosswalk\w*\b[^.]{0,160}<code> or <code>[^.]{0,160}\bcrosswalk |
bundled |
`[^.]{0,120}\b(?:bundled |
telehealth_list |
Medicare telehealth services list[^.]{0,200}<code> or <code>[^.]{0,200}telehealth services list |
One paragraph may yield several kinds; identical (year, kind, item_key, p_id) rows are deduped.
- Step 1: Add the failing tests
Append to tests/pfs/test_lineage.py:
from pfs.lineage import fr_events, lineage
class _Store:
def __init__(self):
self.con = sqlite3.connect(":memory:")
self.con.row_factory = sqlite3.Row
self.con.executescript(
"CREATE TABLE items (key TEXT PRIMARY KEY, title TEXT, date_published TEXT);"
"CREATE TABLE fr_anchors (item_key TEXT, p_id INTEGER, page INTEGER, ordinal INTEGER, text TEXT);"
)
def _con(self):
return self.con
@pytest.fixture
def store():
s = _Store()
s.con.executemany(
"INSERT INTO items VALUES (?,?,?)",
[
("DE2VH9PD", "Medicare Program; CY 2015 PFS Final Rule", "2014-11-13"),
("YBM4IZUS", "Medicare Program; CY 2021 Payment Policies Under the PFS", "2020-12-28"),
("XFGGRBDH", "Medicare and Medicaid Programs; CY 2025 Payment Policies", "2024-07-31"),
],
)
paras = [
("DE2VH9PD", 1251, 67716, "Accordingly, we will adopt CPT code 99490 for Medicare CCM services, effective January 1, 2015 instead of the G code."),
("YBM4IZUS", 686, 84547, "We also are finalizing our proposal to allow HCPCS code G2058 (which we are finalizing in this rule as new CPT code 99439, see the codes in section II.H.) to be billed concurrently with TCM."),
("YBM4IZUS", 1578, 84639, "At the January 2020 RUC meeting, specialty societies requested a temporary crosswalk through CY 2021 between the value established by CMS for HCPCS code G2058 and the value of new CPT code 99439 (with a descriptor identical to G2058)."),
("XFGGRBDH", 489, 61652, "The CPT Editorial Panel also deleted three codes (99441-99443) for reporting telephone E/M services. We note that CPT codes 99441, 99442, and 99443, each are assigned provisional status on the Medicare telehealth services list, and would return to bundled status when the telehealth flexibilities expire."),
]
s.con.executemany("INSERT INTO fr_anchors VALUES (?,?,?,?,?)", [(k, p, pg, p, t) for k, p, pg, t in paras])
return s
class TestFrEvents:
def test_adopted_cpt(self, store):
ev = fr_events(store, "99490")
assert [(e.year, e.kind, e.item_key, e.p_id) for e in ev] == [(2015, "adopted_cpt", "DE2VH9PD", 1251)]
assert ev[0].anchored is True and ev[0].source == "fr" and ev[0].page == 67716
def test_replaced_by_and_crosswalk(self, store):
ev = fr_events(store, "G2058")
kinds = {(e.year, e.kind, e.to_codes) for e in ev}
assert (2021, "replaced_by", "99439") in kinds
assert (2021, "crosswalk", "") in kinds or any(e.kind == "crosswalk" for e in ev)
def test_replaces_from_the_successor_side(self, store):
ev = fr_events(store, "99439")
rep = [e for e in ev if e.kind == "replaces"]
assert rep and rep[0].from_codes == "G2058"
def test_deleted_bundled_and_telehealth(self, store):
kinds = {e.kind for e in fr_events(store, "99441")}
assert {"deleted", "bundled", "telehealth_list"} <= kinds
class TestCrossCheck:
def test_rvu_events_anchored_by_nearby_fr_event(self, con, store):
ev = lineage(con, store, "G2058")
by_kind = {e.kind: e for e in ev}
assert by_kind["appeared"].year == 2020 and by_kind["appeared"].anchored is False # nothing in FR for 2020 in fixture
assert by_kind["disappeared"].year == 2021 and by_kind["disappeared"].anchored is True # 2021 replaced_by
assert "replaced_by" in by_kind
def test_sorted_by_year_then_source(self, con, store):
ev = lineage(con, store, "99439")
assert [e.year for e in ev] == sorted(e.year for e in ev)
- Step 2: Run the tests to verify they fail
Run: uv run pytest tests/pfs/test_lineage.py -q
Expected: FAIL — ImportError: cannot import name 'fr_events'
- Step 3: Add the FR half and
lineage()tosrc/pfs/lineage.py
from pfs.descriptors import rule_year_of
from pfs.families import find_codes
FR_KINDS = ("created", "adopted_cpt", "replaces", "replaced_by", "deleted", "crosswalk", "bundled", "telehealth_list")
_CODE = r"(?:[A-Z]\d{4}|\d{5})"
def _patterns(code: str) -> list[tuple[str, re.Pattern[str], str]]:
c = re.escape(code)
return [
("created", re.compile(rf"(?:\b(?:creat|establish)\w*\b.{{0,80}}\b(?:HCPCS|CPT|G-?)\s*codes?\b[^.]{{0,80}}{c})|(?:new (?:HCPCS|CPT) code\s+{c})", re.I), ""),
("adopted_cpt", re.compile(rf"adopt(?:ed|ing)?\s+CPT code\s+{c}", re.I), ""),
("replaces", re.compile(rf"{c}[^.]{{0,120}}\b(?:replace|replacing|replacement for|instead of)\b", re.I), "from"),
("replaced_by", re.compile(rf"(?:\b(?:replace\w*|replacement)\b[^.]{{0,120}}{c})|(?:{c}[^.]{{0,80}}\b(?:as|is being|will be|which we are) (?:finaliz|replac)\w+ (?:in this rule )?as new (?:CPT|HCPCS) code\s+{_CODE})", re.I), "to"),
("deleted", re.compile(rf"(?:\bdelet(?:ed|ion|es)\b[^.]{{0,120}}{c})|(?:{c}[^.]{{0,60}}\bdeleted\b)", re.I), ""),
("crosswalk", re.compile(rf"(?:\bcrosswalk\w*\b[^.]{{0,160}}{c})|(?:{c}[^.]{{0,160}}\bcrosswalk)", re.I), ""),
("bundled", re.compile(rf"{c}[^.]{{0,200}}\bbundled\b", re.I), ""),
("telehealth_list", re.compile(rf"(?:telehealth services list[^.]{{0,200}}{c})|(?:{c}[^.]{{0,200}}telehealth services list)", re.I), ""),
]
_RE_RANGE = re.compile(r"\b(\d{5})-(\d{5})\b")
def _expand_ranges(text: str) -> str:
""""99441-99443" → "99441 99442 99443" so a range mentions each code."""
def rep(m: re.Match[str]) -> str:
a, b = int(m.group(1)), int(m.group(2))
return " ".join(str(n) for n in range(a, b + 1)) if 0 < b - a <= 20 else m.group(0)
return _RE_RANGE.sub(rep, text)
def fr_events(store: Any, code: str) -> list[EventRow]:
code = code.upper()
con = store._con()
rows = con.execute(
"SELECT a.item_key, a.p_id, a.page, a.text, i.title, i.date_published "
"FROM fr_anchors a JOIN items i ON i.key = a.item_key WHERE a.text LIKE ? OR a.text LIKE ?",
(f"%{code}%", f"%{code[:5]}-%"),
).fetchall()
seen: set[tuple[int, str, str, int]] = set()
out: list[EventRow] = []
for r in rows:
text = _expand_ranges(r["text"])
if code not in text.upper():
continue
year = rule_year_of(r["title"], r["date_published"])
others = tuple(c for c in find_codes(text) if c != code)
for kind, pat, direction in _patterns(code):
m = pat.search(text)
if not m:
continue
key = (year, kind, r["item_key"], r["p_id"])
if key in seen:
continue
seen.add(key)
frm = " ".join(others) if direction == "from" else ""
to = ""
if direction == "to":
tail = find_codes(text[m.start():m.end()])
to = " ".join(c for c in tail if c != code) or " ".join(others)
out.append(EventRow(code, year, kind, frm, to, r["item_key"], r["p_id"], r["page"], "fr", True, ""))
return sorted(out, key=lambda e: (e.year, FR_KINDS.index(e.kind), e.p_id))
def lineage(con: Any, store: Any, code: str) -> list[EventRow]:
fr = fr_events(store, code)
fr_years = {e.year for e in fr}
rvu = [
EventRow(*e[:9], any(abs(e.year - y) <= 1 for y in fr_years), e.note)
for e in (tuple(ev.__dict__.values()) for ev in rvu_events(con, code))
]
return sorted([*rvu, *fr], key=lambda e: (e.year, e.source != "fr", e.kind))
EventRow(*e[:9], anchored, note) rebuilds the dataclass with anchored replaced — the field order is code, year, kind, from_codes, to_codes, item_key, p_id, page, source, anchored, note.
- Step 4: Run the tests to verify they pass
Run: uv run pytest tests/pfs/test_lineage.py -q
Expected: all PASS. If test_replaces_from_the_successor_side fails, the "replaces" pattern must see "99439 … replac" in "…as new CPT code 99439, see the codes…" — it does not; it matches on ¶1578 "new CPT code 99439 (with a descriptor identical to G2058)" only if you add \bidentical to\b to the replaces alternation: (?:replace|replacing|replacement for|instead of|with a descriptor identical to). Add it.
- Step 5: Commit
git add src/pfs/lineage.py tests/pfs/test_lineage.py
git commit -m "feat(pfs): lineage events from FR paragraphs (created/adopted/replaced/deleted/crosswalk/bundled/telehealth) + RVU cross-check (refs #686)"
Task 8: Derived families with the hand registry as fixture
Files:
- Modify:
src/pfs/families.py - Modify:
tests/pfs/test_families.py
Interfaces:
-
Consumes:
pfs.codetables.ElementRow/EventRow/FamilyRow/read_*. -
Produces:
HAND_FAMILIES: dict[str, Family] # the P48 list, unchanged content FAMILIES: dict[str, Family] # starts as a copy of HAND_FAMILIES; refresh_from() mutates in place def stem_tokens(description: str) -> frozenset[str] # short-descriptor tokens minus time/actor/number noise def derive_families(elements: Mapping[str, Sequence[ElementRow]], events: Mapping[str, Sequence[EventRow]], descriptions: Mapping[str, str]) -> list[FamilyRow] def load_families(con) -> dict[str, Family] # from pfs.code_family; {} when the table is empty/missing def refresh_from(con) -> int # replace FAMILIES contents with load_families(con) if non-empty; returns count -
Derivation: build an undirected graph over codes. Edge when (a) a
relationelement links two codes (addon-of,not-with,defined-by-reference-to,replaces,replaced-by,crosswalk-valued-to), (b) areplaces/replaced_by/crosswalkevent links them, or (c) both havestem_tokenswith Jaccard ≥ 0.6 and the sameactivityset. Components become families. Key = the stem tokens of the component's lowest-numbered code with the longest tenure, joined with-and upper-cased, unless a hand family contains any member — then the hand key and name are used. Roles:add-onwhen the code has anaddon-ofelement;predecessorwhen it has areplaced_byevent and noappearedafter the successor;successorwhen it hasreplaces; elsebase.since/untilfrom the code'sappeared/disappearedevents (None when absent). -
Step 1: Add the failing tests
Append to tests/pfs/test_families.py:
import duckdb
from pfs.codetables import ElementRow, EventRow, ensure_tables, write_families
from pfs.families import HAND_FAMILIES, derive_families, load_families, refresh_from, stem_tokens
def _el(code, type_, value, detail=""):
return ElementRow(code, 2025, type_, value, detail, "", "", 0, 0, "fr")
def _ev(code, year, kind, frm="", to=""):
return EventRow(code, year, kind, frm, to, "", 0, 0, "fr", True, "")
class TestStemTokens:
def test_strips_time_and_actor_noise(self):
assert stem_tokens("Chrnc care mgmt staff 1st 20") == frozenset({"chrnc", "care", "mgmt"})
assert stem_tokens("Chrnc care mgmt phys ea addl") == frozenset({"chrnc", "care", "mgmt"})
assert stem_tokens("Adv prim care mgmt lvl 1") == frozenset({"adv", "prim", "care", "mgmt"})
class TestDerive:
def test_ccm_reproduced_with_predecessor_and_addon_roles(self):
elements = {
"99490": [_el("99490", "actor", "clinical-staff-directed"), _el("99490", "activity", "comprehensive-care-plan")],
"99439": [_el("99439", "relation", "addon-of", "99490"), _el("99439", "relation", "not-with", "99487")],
"99487": [_el("99487", "activity", "comprehensive-care-plan")],
"99489": [_el("99489", "relation", "addon-of", "99487")],
"99491": [_el("99491", "actor", "physician-or-qhp-personally"), _el("99491", "activity", "comprehensive-care-plan")],
"99437": [_el("99437", "relation", "addon-of", "99491")],
"G2058": [_el("G2058", "relation", "addon-of", "99490")],
"99497": [_el("99497", "activity", "advance-directive-discussion")],
"99498": [_el("99498", "relation", "addon-of", "99497")],
}
events = {
"G2058": [_ev("G2058", 2020, "appeared"), _ev("G2058", 2021, "disappeared"), _ev("G2058", 2021, "replaced_by", to="99439")],
"99439": [_ev("99439", 2021, "appeared"), _ev("99439", 2021, "replaces", frm="G2058")],
}
descriptions = {
"99490": "Chrnc care mgmt staff 1st 20", "99439": "Chrnc care mgmt staf ea addl", "99487": "Cplx chrnc care 1st 60 min",
"99489": "Cplx chrnc care ea addl 30", "99491": "Chrnc care mgmt phys 1st 30", "99437": "Chrnc care mgmt phys ea addl",
"G2058": "Ccm add 20min", "99497": "Advncd care plan 30 min", "99498": "Advncd care plan addl 30 min",
}
rows = derive_families(elements, events, descriptions)
ccm = {r.code: r for r in rows if r.key == "CCM"}
assert set(ccm) == {"99490", "99439", "99487", "99489", "99491", "99437", "G2058"}
assert ccm["99439"].role == "add-on" and ccm["G2058"].role == "predecessor"
assert ccm["G2058"].since == 2020 and ccm["G2058"].until == 2021
assert ccm["99490"].role == "base"
acp = {r.code for r in rows if r.key == "ACP"}
assert acp == {"99497", "99498"}
assert all(r.name == HAND_FAMILIES[r.key].name for r in rows if r.key in HAND_FAMILIES)
def test_unknown_family_gets_stem_key(self):
elements = {"99453": [_el("99453", "activity", "device-data-review")], "99454": [_el("99454", "activity", "device-data-review")]}
rows = derive_families(elements, {}, {"99453": "Rem mntr physiol param setup", "99454": "Rem mntr physiol param dev supl"})
keys = {r.key for r in rows}
assert len(keys) == 1 and next(iter(keys)).startswith("REM-MNTR")
class TestLoadAndRefresh:
def test_load_and_refresh_in_place(self):
from pfs import families as mod
con = duckdb.connect(":memory:")
ensure_tables(con)
assert load_families(con) == {}
before = dict(mod.FAMILIES)
assert refresh_from(con) == 0 and mod.FAMILIES == before
write_families(con, [
__import__("pfs.codetables", fromlist=["FamilyRow"]).FamilyRow("CCM", "Chronic Care Management", "99490", "base", None, None, "", 0),
__import__("pfs.codetables", fromlist=["FamilyRow"]).FamilyRow("CCM", "Chronic Care Management", "G2058", "predecessor", 2020, 2021, "", 0),
])
n = refresh_from(con)
assert n == 1 and set(mod.FAMILIES) == {"CCM"} and "G2058" in mod.FAMILIES["CCM"].codes
# restore for other tests
mod.FAMILIES.clear(); mod.FAMILIES.update(before)
- Step 2: Run the tests to verify they fail
Run: uv run pytest tests/pfs/test_families.py -q
Expected: FAIL — ImportError: cannot import name 'HAND_FAMILIES'
- Step 3: Extend
src/pfs/families.py
Rename the existing FAMILIES = {...} literal to HAND_FAMILIES = {...} and add after it:
#: Live registry. Starts as the hand list; ``refresh_from(con)`` swaps in
#: the derived families (``pfs.code_family``) in place so every importer
#: (chat, notebooks) sees the same dict object.
FAMILIES: dict[str, Family] = dict(HAND_FAMILIES)
Change _CODE_TO_FAMILY into a function so it tracks the live dict:
def family_of(code: str) -> Family | None:
code = code.upper()
for fam in FAMILIES.values():
if code in fam.codes:
return fam
return None
Append:
from typing import Mapping, Sequence # at top with the other imports
_NOISE = frozenset({
"1st", "ea", "addl", "add", "min", "mo", "staff", "staf", "phys", "qhp", "lvl", "svc", "srvc", "svcs",
"w", "w/o", "pt", "per", "the", "a", "of", "&", "each", "first", "initial", "additional", "level",
})
_RE_TOKEN = re.compile(r"[a-z]+(?:/[a-z]+)?")
def stem_tokens(description: str) -> frozenset[str]:
"""Short-descriptor tokens minus time/actor/number noise — the part
of a description that names the service."""
return frozenset(t for t in _RE_TOKEN.findall(description.lower()) if t not in _NOISE and len(t) > 1)
_LINK_RELATIONS = {"addon-of", "not-with", "defined-by-reference-to", "replaces", "replaced-by", "crosswalk-valued-to"}
_LINK_EVENTS = {"replaces", "replaced_by", "crosswalk"}
def _jaccard(a: frozenset[str], b: frozenset[str]) -> float:
return len(a & b) / len(a | b) if a or b else 0.0
def derive_families(
elements: Mapping[str, Sequence[Any]],
events: Mapping[str, Sequence[Any]],
descriptions: Mapping[str, str],
) -> list[Any]:
"""Connected components over codes linked by relation elements,
lineage events, or a shared service stem + activity set."""
from pfs.codetables import FamilyRow
codes = sorted(set(elements) | set(events) | set(descriptions))
parent = {c: c for c in codes}
def find(c: str) -> str:
while parent[c] != c:
parent[c] = parent[parent[c]]
c = parent[c]
return c
def union(a: str, b: str) -> None:
if a in parent and b in parent:
parent[find(a)] = find(b)
for code, els in elements.items():
for e in els:
if e.type == "relation" and e.value in _LINK_RELATIONS and e.detail:
union(code, e.detail.upper())
for code, evs in events.items():
for ev in evs:
if ev.kind in _LINK_EVENTS:
for other in (ev.from_codes + " " + ev.to_codes).split():
union(code, other.upper())
stems = {c: stem_tokens(descriptions.get(c, "")) for c in codes}
acts = {c: frozenset(e.value for e in elements.get(c, ()) if e.type == "activity") for c in codes}
for i, a in enumerate(codes):
for b in codes[i + 1:]:
if stems[a] and stems[b] and _jaccard(stems[a], stems[b]) >= 0.6 and acts[a] == acts[b]:
union(a, b)
groups: dict[str, list[str]] = {}
for c in codes:
groups.setdefault(find(c), []).append(c)
def tenure(c: str) -> int:
since = next((e.year for e in events.get(c, ()) if e.kind == "appeared"), 0)
until = next((e.year for e in events.get(c, ()) if e.kind == "disappeared"), 9999)
return until - since
rows: list[FamilyRow] = []
for members in groups.values():
hand = next((k for k, f in HAND_FAMILIES.items() if set(f.codes) & set(members)), None)
if hand:
key, name = hand, HAND_FAMILIES[hand].name
else:
rep = sorted(members, key=lambda c: (-tenure(c), c))[0]
key = "-".join(sorted(stems[rep])).upper() or rep
name = descriptions.get(rep, rep)
for c in sorted(members):
els = elements.get(c, ())
evs = events.get(c, ())
kinds = {ev.kind for ev in evs}
if any(e.type == "relation" and e.value == "addon-of" for e in els):
role = "add-on"
elif "replaced_by" in kinds:
role = "predecessor"
elif "replaces" in kinds:
role = "successor"
else:
role = "base"
since = next((ev.year for ev in evs if ev.kind == "appeared"), None)
until = next((ev.year for ev in evs if ev.kind == "disappeared"), None)
anchor = next(((ev.item_key, ev.p_id) for ev in evs if ev.item_key), ("", 0))
rows.append(FamilyRow(key, name, c, role, since, until, anchor[0], anchor[1]))
return sorted(rows, key=lambda r: (r.key, r.code))
def load_families(con: Any) -> dict[str, Family]:
from pfs.codetables import read_families
try:
rows = read_families(con)
except Exception: # table missing on an old replica
return {}
out: dict[str, Family] = {}
for r in rows:
fam = out.get(r.key)
codes = (*(fam.codes if fam else ()), r.code)
syn = HAND_FAMILIES[r.key].synonyms if r.key in HAND_FAMILIES else (r.key.lower(), r.name.lower())
out[r.key] = Family(r.key, r.name, tuple(sorted(set(codes))), syn)
return out
def refresh_from(con: Any) -> int:
derived = load_families(con)
if not derived:
return 0
FAMILIES.clear()
FAMILIES.update(derived)
return len(FAMILIES)
Add from typing import Any to the imports. Keep detect_codes unchanged — it reads FAMILIES and calls family_of, both live.
- Step 4: Run the tests to verify they pass
Run: uv run pytest tests/pfs/test_families.py tests/llm/test_evidence.py tests/pfs/test_valuation.py -q
Expected: all PASS — the P48 tests exercise FAMILIES/detect_codes/family_of and must not change behaviour.
- Step 5: Commit
git add src/pfs/families.py tests/pfs/test_families.py
git commit -m "feat(pfs): derived code families (relation/lineage/stem components) with the hand registry as fixture; live refresh_from (refs #687)"
Task 9: stack pfs CLI — elements, lineage, families
Files:
- Create:
src/cli/pfs.py - Modify:
src/cli/__init__.py(register) - Test:
tests/cli/test_pfs_cli.py
Interfaces:
-
Consumes:
conf.connect.duckdb_batch,conf.connect.duckdb(read),conf.connect.publish_replica,conf.connect.bib,pfs.extract.extract_code,pfs.lineage.lineage,pfs.families.derive_families,pfs.codetables.*,llm.classify.closed_vocab_classifier,llm.config.load,llm.pool.HostPool. -
Produces commands:
stack pfs elements --code G0556 [--code …] | --family CCM | --all-payable [--no-llm] [--dry-run]— writespfs.code_element+ review rows, prints per-code counts, republishes the replica unless--dry-run.stack pfs lineage --code 99439 [--write]— prints the timeline (year kind from→to anchored item_key ¶p_id); with--writepersists topfs.code_eventand republishes.stack pfs families [--write]— derives from the tables, prints families,--writereplacespfs.code_familyand republishes.stack pfs review [--code X]— listspfs.code_element_reviewrows.
-
Helper
_codes_for(con, code, family, all_payable) -> list[str]:--familyexpands viapfs.families.FAMILIES(afterrefresh_from),--all-payable= distincthcpcswith status in('A','R','T')in the newestpfs.rvuyear. -
Step 1: Write the failing tests
# tests/cli/test_pfs_cli.py
"""stack pfs — elements / lineage / families / review."""
from __future__ import annotations
import duckdb
import pytest
from typer.testing import CliRunner
import cli.pfs as pfs_cli
from cli import app
from pfs.codetables import ElementRow, EventRow, ensure_tables, read_elements, read_events, read_families, write_elements, write_events
from pfs.extract import Extraction
runner = CliRunner()
@pytest.fixture
def con(monkeypatch):
c = duckdb.connect(":memory:")
ensure_tables(c)
c.execute("CREATE TABLE pfs.rvu (hcpcs VARCHAR, mod VARCHAR, description VARCHAR, status_code VARCHAR, non_fac_total DOUBLE, year INTEGER)")
c.executemany("INSERT INTO pfs.rvu VALUES (?,?,?,?,?,?)", [
("99490", None, "Chrnc care mgmt staff 1st 20", "A", 1.0, 2026),
("G2058", None, "Ccm add 20min", "A", 1.0, 2020),
("99999", None, "bundled thing", "B", 0.0, 2026),
])
class _Batch:
def __enter__(self):
return c
def __exit__(self, *a):
return False
monkeypatch.setattr(pfs_cli, "_batch", lambda: _Batch())
monkeypatch.setattr(pfs_cli, "_read", lambda: c)
monkeypatch.setattr(pfs_cli, "_store", lambda: object())
published = []
monkeypatch.setattr(pfs_cli, "_publish", lambda: published.append(True))
c.published = published
return c
class TestElements:
def test_writes_rows_and_publishes(self, con, monkeypatch):
def fake_extract(store, con_, code, *, classify=None):
return Extraction(code, (ElementRow(code, 2026, "activity", "consent", "", "Consent;", "K", 1, 2, "fr"),), ())
monkeypatch.setattr(pfs_cli, "extract_code", fake_extract)
res = runner.invoke(app, ["pfs", "elements", "--code", "g0556", "--no-llm"])
assert res.exit_code == 0, res.output
assert "G0556: 1 elements, 0 for review" in res.output
assert [r.value for r in read_elements(con, "G0556")] == ["consent"]
assert con.published == [True]
def test_dry_run_does_not_write(self, con, monkeypatch):
monkeypatch.setattr(pfs_cli, "extract_code", lambda s, c, code, *, classify=None: Extraction(code, (), ()))
res = runner.invoke(app, ["pfs", "elements", "--code", "G0556", "--no-llm", "--dry-run"])
assert res.exit_code == 0 and con.published == []
def test_all_payable_selects_art_status_in_newest_year(self, con, monkeypatch):
seen = []
monkeypatch.setattr(pfs_cli, "extract_code", lambda s, c, code, *, classify=None: seen.append(code) or Extraction(code, (), ()))
res = runner.invoke(app, ["pfs", "elements", "--all-payable", "--no-llm", "--dry-run"])
assert res.exit_code == 0 and seen == ["99490"]
class TestLineage:
def test_prints_and_writes(self, con, monkeypatch):
ev = EventRow("G2058", 2021, "replaced_by", "", "99439", "YBM4IZUS", 1578, 84639, "fr", True, "")
monkeypatch.setattr(pfs_cli, "lineage", lambda c, s, code: [ev])
res = runner.invoke(app, ["pfs", "lineage", "--code", "G2058", "--write"])
assert res.exit_code == 0, res.output
assert "2021 replaced_by" in res.output and "YBM4IZUS ¶1578" in res.output
assert read_events(con, "G2058")[0].to_codes == "99439"
assert con.published == [True]
class TestFamilies:
def test_derives_from_tables_and_writes(self, con):
write_elements(con, "99439", [ElementRow("99439", 2021, "relation", "addon-of", "99490", "", "K", 1, 1, "fr")], [])
write_events(con, "G2058", [EventRow("G2058", 2021, "replaced_by", "", "99439", "K", 2, 1, "fr", True, "")])
res = runner.invoke(app, ["pfs", "families", "--write"])
assert res.exit_code == 0, res.output
fams = read_families(con)
assert {r.code for r in fams if r.key == "CCM"} >= {"99490", "99439", "G2058"}
assert "CCM" in res.output
class TestReview:
def test_lists_queue(self, con):
from pfs.codetables import ReviewRow
write_elements(con, "G0556", [], [ReviewRow("G0556", "Odd line;", "", "", "K", 9)])
res = runner.invoke(app, ["pfs", "review"])
assert res.exit_code == 0 and "G0556" in res.output and "Odd line" in res.output
- Step 2: Run the tests to verify they fail
Run: uv run pytest tests/cli/test_pfs_cli.py -q
Expected: FAIL — ModuleNotFoundError: No module named 'cli.pfs'
- Step 3: Implement
src/cli/pfs.pyand register it
"""stack pfs — code elements, lineage events and derived families.
uv run stack pfs elements --code G0556 --code 99490
uv run stack pfs elements --family CCM
uv run stack pfs elements --all-payable --no-llm --dry-run
uv run stack pfs lineage --code G2058 [--write]
uv run stack pfs families [--write]
uv run stack pfs review [--code G0556]
Writes go through ``duckdb_batch`` (single-writer rule) and republish
the read-only replica so the chat and notebooks see them.
"""
from __future__ import annotations
from typing import Any
import typer
from pfs.codetables import (
ensure_tables,
read_elements,
read_events,
write_elements,
write_events,
write_families,
)
from pfs.extract import extract_code
from pfs.families import FAMILIES, derive_families, refresh_from
from pfs.lineage import lineage
app = typer.Typer(no_args_is_help=True)
# ── indirections the tests monkeypatch ───────────────────────────────
def _batch() -> Any:
from conf.connect import duckdb_batch
return duckdb_batch("aco")
def _read() -> Any:
from conf.connect import duckdb
return duckdb("aco", read_only=True)
def _store() -> Any:
from conf.connect import bib
return bib()
def _publish() -> None:
from conf.connect import publish_replica
typer.echo(f"replica → {publish_replica('aco')}")
def _classifier() -> Any:
from llm import config as llm_config
from llm.classify import closed_vocab_classifier
from llm.pool import HostPool
cfg = llm_config.load()
return closed_vocab_classifier(cfg, HostPool.from_config(cfg))
def _codes_for(con: Any, codes: list[str], family: str, all_payable: bool) -> list[str]:
out: list[str] = [c.upper() for c in codes]
if family:
refresh_from(con)
fam = FAMILIES.get(family.upper())
if fam is None:
raise typer.BadParameter(f"unknown family {family!r}; known: {', '.join(FAMILIES)}")
out.extend(fam.codes)
if all_payable:
rows = con.execute(
"SELECT DISTINCT hcpcs FROM pfs.rvu WHERE status_code IN ('A','R','T') "
"AND year = (SELECT max(year) FROM pfs.rvu) ORDER BY hcpcs"
).fetchall()
out.extend(r[0] for r in rows)
if not out:
raise typer.BadParameter("pass --code, --family or --all-payable")
return sorted(set(out))
@app.command()
def elements(
code: list[str] = typer.Option([], "--code", help="HCPCS/CPT code (repeatable)."),
family: str = typer.Option("", help="Expand a registered family (CCM, APCM, …)."),
all_payable: bool = typer.Option(False, "--all-payable", help="Every A/R/T code in the newest RVU year."),
no_llm: bool = typer.Option(False, "--no-llm", help="Skip the local-model classifier (unknown lines go to review)."),
dry_run: bool = typer.Option(False, "--dry-run", help="Extract and report; write nothing."),
) -> None:
"""Extract typed elements for codes into pfs.code_element (+ review queue)."""
store = _store()
classify = None if no_llm else _classifier()
with _batch() as con:
ensure_tables(con)
targets = _codes_for(con, code, family, all_payable)
for c in targets:
x = extract_code(store, con, c, classify=classify)
if not dry_run:
write_elements(con, c, x.rows, x.reviews)
typer.echo(f"{c}: {len(x.rows)} elements, {len(x.reviews)} for review")
if not dry_run:
_publish()
@app.command("lineage")
def lineage_cmd(
code: str = typer.Option(..., "--code"),
write: bool = typer.Option(False, "--write", help="Persist to pfs.code_event and republish."),
) -> None:
"""Timeline of a code: RVU-file diffs and FR paragraphs, cross-checked."""
store = _store()
with _batch() as con:
ensure_tables(con)
events = lineage(con, store, code.upper())
for e in events:
arrow = f"{e.from_codes or '·'}→{e.to_codes or '·'}"
anchor = f"{e.item_key} ¶{e.p_id}" if e.item_key else f"rvu {e.note}".strip()
flag = "" if e.anchored else " UNANCHORED"
typer.echo(f"{e.year} {e.kind:<18}{arrow:<16}{anchor}{flag}")
if write:
write_events(con, code.upper(), events)
if write:
_publish()
@app.command()
def families(write: bool = typer.Option(False, "--write")) -> None:
"""Derive families from pfs.code_element / pfs.code_event / pfs.rvu."""
with _batch() as con:
ensure_tables(con)
codes = [r[0] for r in con.execute(
"SELECT DISTINCT code FROM pfs.code_element UNION SELECT DISTINCT code FROM pfs.code_event "
"UNION SELECT DISTINCT hcpcs FROM pfs.rvu WHERE year = (SELECT max(year) FROM pfs.rvu) AND status_code IN ('A','R','T')"
).fetchall()]
elements = {c: read_elements(con, c) for c in codes}
events = {c: read_events(con, c) for c in codes}
descriptions = {
r[0]: r[1] for r in con.execute(
"SELECT hcpcs, arg_max(description, year) FROM pfs.rvu WHERE mod IS NULL OR mod = '' GROUP BY hcpcs"
).fetchall()
}
rows = derive_families(elements, events, descriptions)
by_key: dict[str, list[str]] = {}
for r in rows:
by_key.setdefault(r.key, []).append(f"{r.code}({r.role})")
for key, members in sorted(by_key.items()):
typer.echo(f"{key}: {' '.join(members)}")
if write:
write_families(con, rows)
refresh_from(con)
if write:
_publish()
@app.command()
def review(code: str = typer.Option("", "--code")) -> None:
"""Element lines the classifier could not place."""
con = _read()
sql = "SELECT code, text, proposed_value, item_key, p_id FROM pfs.code_element_review"
params: list[Any] = []
if code:
sql += " WHERE code = ?"
params.append(code.upper())
for c, text, proposed, key, p_id in con.execute(sql + " ORDER BY code, p_id", params).fetchall():
typer.echo(f"{c} {key} ¶{p_id} [{proposed or '?'}] {text[:120]}")
In src/cli/__init__.py add from cli.pfs import app as pfs_app with the other imports and, after the rec_app registration:
app.add_typer(pfs_app, name="pfs", help="Code elements, lineage events and derived families.")
- Step 4: Run the tests to verify they pass
Run: uv run pytest tests/cli/test_pfs_cli.py tests/cli -q
Expected: all PASS. test_all_payable_selects_art_status_in_newest_year relies on the fixture's 99999 being status B and G2058 being an old year — both excluded. TestFamilies runs the real derive_families on the tiny tables: 99490 (rvu, payable), 99439 (addon-of 99490), G2058 (replaced_by 99439) all land in CCM because 99490 is in HAND_FAMILIES["CCM"].
- Step 5: Commit
git add src/cli/pfs.py src/cli/__init__.py tests/cli/test_pfs_cli.py
git commit -m "feat(cli): stack pfs elements|lineage|families|review (refs #684 #685 #686 #687)"
Task 10: Live run on the fixture families and docs
Files:
-
Modify:
docs/docs/page that lists CLI commands (find withgrep -rl "stack rec pfs" docs/docs) — add the fourstack pfscommands. -
No new tests; this task is verification against the real corpus.
-
Step 1: Extract the fixture families without the model, dry run
Run (from the repo root, with the notebook kernels closed so the DuckDB writer lock is free):
set -a; . ./.env; set +a
uv run stack pfs elements --family CCM --family APCM --no-llm --dry-run
Expected: one line per code (99490, 99487, 99489, 99491, 99439, 99437, G0556, G0557, G0558) with element counts > 0 for every code that has an FR descriptor run; G0556 should report ≥ 15 elements and a handful of review lines (the ++ sub-elements the phrases do not name). If a code reports 0 elements, run uv run python -c "from conf.connect import bib; from pfs.descriptors import descriptor_runs; print(descriptor_runs(bib(), 'CODE'))" and fix the stem regex in pfs/descriptors.py before continuing.
- Step 2: Extract with the local model and write
uv run stack llm hosts # confirm a host serves the instruct model
uv run stack pfs elements --family CCM --family APCM --family ACP --family TCM --family PCM
uv run stack pfs review
Expected: review queue shrinks versus step 1; the replica path is printed. Anything still in review is expected — do not add vocabulary values in this task (that is the review loop from the spec).
- Step 3: Lineage for the six fixture lineages
for c in 99490 G2058 99439 G2064 99424 99441 G0556; do uv run stack pfs lineage --code $c --write; done
Expected, from the spec's evidence table: 99490 2015 adopted_cpt DE2VH9PD ¶1251; G2058 2020 appeared, 2021 replaced_by →99439 YBM4IZUS ¶1578, 2021 disappeared anchored; 99439 2021 replaces G2058→; G2064 2022 replaced_by →99424 (JE7KYBW3 ¶1100/¶1111); 99441 2025 deleted + telehealth_list + bundled (XFGGRBDH ¶489) and 2025 disappeared anchored; G0556 2025 appeared anchored by a 2025 created row from JJ6AM5HJ. Record any UNANCHORED line in issue #686 as a comment (they are the review list, not failures).
- Step 4: Derive families and check the P48 fixture reproduces
uv run stack pfs families --write
uv run python -c "from pfs.families import HAND_FAMILIES, refresh_from; from conf.connect import duckdb; con=duckdb('aco'); refresh_from(con); from pfs.families import FAMILIES; [print(k, sorted(set(HAND_FAMILIES[k].codes) - set(FAMILIES[k].codes))) for k in HAND_FAMILIES]"
Expected: every hand family prints [] (no hand code lost) and CCM now also lists G2058. Then uv run pytest tests/pfs tests/llm tests/cli -q — all green.
- Step 5: Docs + commit
Add the stack pfs commands to the CLI reference page found in step 0 of this task, then:
git add docs/docs
git commit -m "docs: stack pfs elements|lineage|families|review (refs #684-#687)"
Comment on issues #684–#687 with the counts from steps 1–4 (elements per code, review-queue size, unanchored events) and link the commits.
Self-review
Spec coverage (slice 1). Decision 1 (closed typed vocabulary) → Task 1; §pfs/extract.py (deterministic then local model, review queue, stack pfs elements --code|--family|--all-payable) → Tasks 2, 3, 5, 9; Decision 2 / §pfs/lineage.py (two sources, must agree or flag unanchored, anchors on every row, fixture lineages) → Tasks 6, 7, 10; Decision 3 / §pfs/families.py (derived, API kept, hand list as fixture, since/until, roles) → Task 8; Decision 7 (self-hosted only) → Task 3; single-writer + replica → Task 9. Not in this slice, by design: chunk/item anchors (#688), IOM/eCFR crosswalk (#689), reaction series (#690), chat (#691), eval (#692) — next plans.
Placeholders. None: every step carries code or an exact command. The _PHRASES table is the seed vocabulary and is expected to be extended through the review queue, not in this plan.
Type consistency. ElementRow/ReviewRow/EventRow/FamilyRow are defined once in Task 4 and used by Tasks 5–9 with the same field order; Extraction(code, rows, reviews) in Task 5 matches the CLI fakes in Task 9; lineage(con, store, code) argument order is identical in Task 7 and Task 9; EventRow.anchored is the 10th field, which the Task 7 rebuild relies on.