pfs: element extractor — deterministic parse + local-model closed-vocab mapping of descriptor paragraphs from the FR #685

New Issue

kert · 2026-09-09T14:30:04Z

kert commented

2026-09-09 14:30:04 +00:00

Spec docs/superpowers/specs/2026-09-09-code-family-longitudinal-design.md (pfs/extract.py).

Input. A descriptor and the FR paragraph run that prints it: stem paragraph followed by element paragraphs, as fr_anchors already stores them (DE2VH9PD ¶1244 stem, ¶1245–1247 elements; JJ6AM5HJ ¶1163 stem, ¶1164–1185 elements/sub-elements). Also pfs.rvu.description (short descriptor, 18,641 distinct HCPCS 2015–2026) and terminology.hcpcs_level_2 long descriptors for G-codes.

Pipeline. 1) deterministic parser for time/period/relation (code refs inside "Use in conjunction with…", "Do not report … in the same calendar month as 99487, 99489, 99491" — YBM4IZUS ¶2369). 2) local model (qwen2.5 on the GPU pool; llm module is self-hosted only) maps each element paragraph to a vocabulary value or proposes a new one into a review queue — never auto-added (P35 closed-vocab pattern). 3) rows → DuckDB pfs.code_element(code, year, type, value, item_key, p_id, page).

CLI. stack pfs elements --code G0556 | --family CCM | --all-payable (8,572 status A/R/T codes in CY2026).

Done when the extractor reproduces the seed set with ≥95 % element recall against the hand-curated fixture, every row carries an anchor, and unknown proposals land in the review queue rather than the table. Respect the DuckDB single-writer rule (write via the ingest path, publish the replica).

Spec docs/superpowers/specs/2026-09-09-code-family-longitudinal-design.md (`pfs/extract.py`). **Input.** A descriptor and the FR paragraph run that prints it: stem paragraph followed by element paragraphs, as `fr_anchors` already stores them (`DE2VH9PD` ¶1244 stem, ¶1245–1247 elements; `JJ6AM5HJ` ¶1163 stem, ¶1164–1185 elements/sub-elements). Also `pfs.rvu.description` (short descriptor, 18,641 distinct HCPCS 2015–2026) and `terminology.hcpcs_level_2` long descriptors for G-codes. **Pipeline.** 1) deterministic parser for `time`/`period`/`relation` (code refs inside "Use in conjunction with…", "Do not report … in the same calendar month as 99487, 99489, 99491" — `YBM4IZUS` ¶2369). 2) local model (qwen2.5 on the GPU pool; llm module is self-hosted only) maps each element paragraph to a vocabulary value or proposes a new one into a review queue — never auto-added (P35 closed-vocab pattern). 3) rows → DuckDB `pfs.code_element(code, year, type, value, item_key, p_id, page)`. **CLI.** `stack pfs elements --code G0556 | --family CCM | --all-payable` (8,572 status A/R/T codes in CY2026). **Done when** the extractor reproduces the seed set with ≥95 % element recall against the hand-curated fixture, every row carries an anchor, and unknown proposals land in the review queue rather than the table. Respect the DuckDB single-writer rule (write via the ingest path, publish the replica).

kert added this to the P49: Code families as first-class objects — element model, lineage anchors, longitudinal chat milestone 2026-09-09 14:30:04 +00:00

kert added the pfs llm labels 2026-09-09 14:30:04 +00:00

kert commented

2026-09-09 18:24:10 +00:00

Implemented on branch worktree-p49-code-family-foundation (Tasks 2, 3, 5): src/pfs/descriptors.py (FR stem + element-paragraph runs from fr_anchors, terminology.hcpcs_level_2 long descriptors, pfs.rvu short descriptors), src/llm/classify.py (closed-vocabulary one-of-N on the Ollama pool, temperature 0, pool.check once per classifier, fail-loud on transport errors), src/pfs/extract.py (deterministic parse → classifier for unplaced lines → review queue; unknown slugs never enter pfs.code_element). CLI: stack pfs elements --code … | --family … (repeatable) | --all-payable (experimental) [--no-llm] [--dry-run] — dry-run reads the replica only.

Effect of the model on the review queue (CCM + APCM): 61 lines with --no-llm → 21 after classification; after the final-review fix to the stem pattern, 9 lines across all 20 fixture codes.

Final-review fix that matters: _stem_pattern was case-insensitive on [A-Z], so FR cross-reference lists (( 9) 99439 (code for non-complex chronic care management).) were parsed as descriptors and produced elements anchored to paragraphs about other codes. Fixed ((?!codes?\b) lookahead + enumeration rejection in is_element_paragraph), regression tests added, pipeline re-run; the anchor audit for 99439/99490 shows 0 rows whose paragraph names a different code.

Remaining: a multi-thousand-character background paragraph can still be absorbed as a stem (new issue filed; G0502/G0503 case).

Implemented on branch `worktree-p49-code-family-foundation` (Tasks 2, 3, 5): `src/pfs/descriptors.py` (FR stem + element-paragraph runs from `fr_anchors`, `terminology.hcpcs_level_2` long descriptors, `pfs.rvu` short descriptors), `src/llm/classify.py` (closed-vocabulary one-of-N on the Ollama pool, `temperature 0`, `pool.check` once per classifier, fail-loud on transport errors), `src/pfs/extract.py` (deterministic parse → classifier for unplaced lines → review queue; unknown slugs never enter `pfs.code_element`). CLI: `stack pfs elements --code … | --family … (repeatable) | --all-payable (experimental) [--no-llm] [--dry-run]` — dry-run reads the replica only. **Effect of the model on the review queue (CCM + APCM):** 61 lines with `--no-llm` → 21 after classification; after the final-review fix to the stem pattern, 9 lines across all 20 fixture codes. **Final-review fix that matters:** `_stem_pattern` was case-insensitive on `[A-Z]`, so FR cross-reference lists (`( 9) 99439 (code for non-complex chronic care management).`) were parsed as descriptors and produced elements anchored to paragraphs about other codes. Fixed (`(?!codes?\b)` lookahead + enumeration rejection in `is_element_paragraph`), regression tests added, pipeline re-run; the anchor audit for 99439/99490 shows 0 rows whose paragraph names a different code. Remaining: a multi-thousand-character background paragraph can still be absorbed as a stem (new issue filed; G0502/G0503 case).

kert referenced this issue

2026-09-09 18:55:18 +00:00

P49 slice 1: code families as first-class objects — element model, lineage anchors, derived families, stack pfs CLI (refs #684 #685 #686 #687) #701