pfs: stack pfs elements --all-payable — invert the loop (one pass over fr_anchors for all codes) before it is usable at 8.7k codes #698
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: homelab/stack#698
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Follow-up from the P49 slice-1 whole-branch review (spec docs/superpowers/specs/2026-09-09-code-family-longitudinal-design.md, plan docs/superpowers/plans/2026-09-09-code-family-foundation.md).
Problem.
stack pfs elements --all-payabletargets every A/R/T code in the newest RVU year (8,692 in CY2026). As written it runs two full scans offr_anchors(193,220 rows, no index ontext) per code —descriptor_runs≈ 60 ms andfr_events≈ 80 ms measured — i.e. roughly 15–25 minutes of SQL before the first model call, plus one follow-up query per descriptor run and one classifier round-trip per unplaced element line. It has never been run end to end. Slice 1 ships it flagged experimental in the help text with a warning log naming the target count.Fix. Invert the loop: one pass over
fr_anchors(and one overpfs.rvu/terminology.hcpcs_level_2) matching all target codes withpfs.families.find_codes, bucketing paragraphs by code, then extracting per code from the bucket. Same forfr_events. Consider an FTS or trigram index onfr_anchors.textin bib.Done when
--all-payable --no-llm --dry-runcompletes in under 5 minutes on the current corpus and the per-code counts for the 17 fixture codes are unchanged versus the per-family run.Done in
e14f12e(merged to main ase149447). Measured on the live corpus: stack pfs elements --all-payable --no-llm --dry-run now takes ~12 s wall clock for 8,689 A/R/T codes (was ~15–25 min of SQL before the first model call). One streaming pass over fr_anchors in (item_key, p_id) order finds every target stem with a single regex (descriptor_runs_bucketed), keeping each run open exactly the way the old per-stem follow-up query walked; hcpcs_long_descriptions / rvu_descriptions_bucketed / _cpt_elements_bucketed are one DuckDB query each with QUALIFY row_number() for the newest-row picks. pfs.extract._assemble is now the single home of the FR > CPT > HCPCS/RVU precedence and confirmed_by logic, used by both extract_code and the new extract_codes, so the two paths cannot drift; tests assert bucketed[c] == per_code(c) across all four sources, and all 17 HAND_FAMILIES codes produced byte-identical rows and reviews on the real corpus. The 'experimental' / '~20 min' wording is gone. No FTS/trigram index: the inverted pass does no text LIKE, so an index would be write-path cost for nothing. Note for --write: the whole scan now runs inside the DuckDB batch before the first write_elements — less total lock time than before, but one contiguous hold rather than interleaved, so a notebook competing for the writer will wait once.