homelab/stack
1
0
Fork 0
Code Issues 20 Pull Requests Actions Packages Projects 1 Releases Wiki Activity
Files
c244cebba11359aa887c735490af8b6e3258584b
stack/docs/superpowers/plans/2026-09-09-cpt-organizing-principles.md

18 KiB
Raw Blame History

CPT Manual Organizing Principles (P49 slice 2a) Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Bring the AMA CPT manuals (2019–2024 EPUB editions in Zotero) into the bibliography and DuckDB as the CPT's own organizing structure — section hierarchy, code entries with symbols and required elements, parenthetical instructions, and per-code "CPT Changes" years — and make that structure the primary edge in family derivation, so every CPT code gets a family named the way the manual names it.

Architecture: A pure EPUB parser (pfs/cpt_epub.py) turns one edition into dataclasses; a loader (pfs/cpt_load.py) writes four pfs.cpt_* tables through duckdb_batch with the bib item key of the edition as provenance; pfs/families.py gains a "same CPT subsection" edge and takes family key/name from the CPT heading; pfs/lineage.py gains cpt_changed events from the per-code reference line; pfs/extract.py gains a CPT source for elements (the bulleted required-elements list is literally the element list). A small Zotero→bib import command brings the edition items and files into the bibliography so tables can cite them.

Tech Stack: Python 3.13 stdlib zipfile + html.parser (no new dependency; lxml/bs4 only if already in uv.lock — check), DuckDB via duckdb_batch, SQLite bib.Store, Zotero SQLite read-only via conf.connect.zotero(), typer, pytest.

Spec: docs/superpowers/specs/2026-09-09-code-family-longitudinal-design.md §Decisions 1–3 (elements, lineage, derived families). This slice supplies the organizing principle the spec's family derivation lacked (Ruling: the CPT's own hierarchy outranks stem-token similarity).

Global Constraints

  • Copyright. CPT content is AMA-licensed material the user owns. It is stored only in the local DuckDB and replica for the user's own analysis. It is never indexed into the chat's pgvector corpus in this slice (the CPT bib items carry a tag the corpus indexer skips), never printed into docs, the tracker, or commit messages, and the notebook shows at most one code entry at a time as an illustration. Tests use a synthetic fixture in the manual's markup style, not copied CPT text.
  • Files are read in place from Zotero storage (data/zotero/data/storage/<attKey>/<file>, owned by the Zotero container uid — read-only, never chown) or from the bib copy attach_file makes.
  • pfs/cpt_epub.py and pfs/cpt_model.py are pure (no DuckDB, no bib at import). Writes only inside duckdb_batch("aco"), publish_replica("aco") after.
  • Hand families remain a minimum (Ruling 17); CPT subsection edges are additive to the slice-1 edges.
  • Never add a Co-Authored-By trailer. Commit after every task. Test output pristine.

File map

File Responsibility
src/bib/zotero_import.py (new) + src/cli/bib.py (modify) stack bib import-zotero --collection "AMA Coding Publications" --with-files [--tag …]: Zotero items (+ attachments) → bib Source items tagged source:ama module:coding year:YYYY llm:skip
src/llm/source.py (modify) iter_corpus_refs skips items tagged llm:skip
src/pfs/cpt_model.py (new) dataclasses CptSection, CptCode, CptInstruction, CptReference, CptEdition
src/pfs/cpt_epub.py (new) parse_epub(path) -> CptEdition (2021+ template; 2019 template via the same table classes)
src/pfs/codetables.py (modify) DDL + writers/readers for pfs.cpt_section, pfs.cpt_code, pfs.cpt_instruction, pfs.cpt_reference
src/pfs/cpt_load.py (new) + src/cli/pfs.py (modify) `stack pfs cpt-ingest [--edition 2024 …
src/pfs/families.py (modify) CPT subsection edge; key/name from the CPT heading; add-on edges from "Use X in conjunction with Y"
src/pfs/lineage.py (modify) cpt_changed events from pfs.cpt_reference
src/pfs/extract.py (modify) extract_code also reads pfs.cpt_code elements (source="cpt")
notebooks/code_families.py (modify) section 1 shows the CPT heading path + CPT elements; section 4 shows the CPT subsection
tests: tests/bib/test_zotero_import.py, tests/pfs/test_cpt_epub.py (+ tests/pfs/fixtures/cpt_sample.xhtml), tests/pfs/test_cpt_load.py, tests/pfs/test_families.py (extend), tests/pfs/test_lineage.py (extend), tests/pfs/test_extract.py (extend), tests/cli/test_pfs_cli.py (extend)

Task 1: Zotero → bib import of the CPT editions (with files) and corpus skip tag

Files:

  • Create: src/bib/zotero_import.py; Modify: src/cli/bib.py, src/llm/source.py (iter_corpus_refs)
  • Test: tests/bib/test_zotero_import.py, tests/llm/test_source.py (extend)

Interfaces:

@dataclass(frozen=True) class ZoteroBook: key: str; title: str; url: str; year: str; publisher: str; attachments: tuple[Path, ...]
def list_collection(zcon, name: str, *, with_files: bool) -> list[ZoteroBook]      # sqlite3 connection from conf.connect.zotero(); resolves storage:<file> → storage/<attKey>/<file>
def import_books(store, books, *, tags: Sequence[str], copy_files: bool = True, dry_run=False) -> dict   # Source items upserted by url (fallback url = f"zotero://select/library/items/{key}"), tags + year:YYYY, attach_file per attachment; {"created","updated","attached"}

iter_corpus_refs (and any other corpus walker) skips items tagged llm:skip. CLI: stack bib import-zotero --collection "AMA Coding Publications" --with-files --tag source:ama --tag module:coding --tag llm:skip [--only "CPT"] [--dry-run] (--only = title substring filter).

  • Failing tests: list_collection against a tiny sqlite built with the Zotero schema subset (items, itemData, itemDataValues, fields, collections, collectionItems, itemAttachments, itemTypes) — two books, one with a storage: attachment; import_books into a tmp Store creates Source items with the tags and attaches the file (copy); rerun → updated/unchanged, no duplicate attachment; iter_corpus_refs skips an item tagged llm:skip.
  • Implement; run uv run pytest tests/bib tests/llm/test_source.py -q; live: uv run stack bib import-zotero --collection "AMA Coding Publications" --with-files --only CPT --tag source:ama --tag module:coding --tag llm:skip → report the bib keys of the six CPT editions (2018, 2019, 2021, 2022, 2023 Changes, 2024).
  • Commit: feat(bib): import Zotero collection items with files into bib; llm:skip tag honoured by the corpus indexer (refs #688).

Task 2: CPT EPUB parser (pure)

Files:

  • Create: src/pfs/cpt_model.py, src/pfs/cpt_epub.py, tests/pfs/fixtures/cpt_sample.xhtml (synthetic, in the 2024 template's markup style: two div.h1, two div.h2, one table.table1 with a primary code row with three table-slist elements, an add-on row with ✚, three table-para2 parentheticals (use-with, do-not-report-with, plain), and a table-RT "CPT Changes" line)
  • Test: tests/pfs/test_cpt_epub.py

Interfaces:

@dataclass(frozen=True) class CptSection: sec_id: str; level: int; title: str; path: tuple[str, ...]; code_lo: str; code_hi: str; guideline: str   # path = ancestors' titles + own; lo/hi from the TOC "(99490-99437)" when present
@dataclass(frozen=True) class CptCode: code: str; sec_id: str; descriptor: str; stem: str; elements: tuple[str, ...]; tail: str; addon: bool; resequenced: bool; new: bool; revised: bool; telemedicine: bool; parent: str
    # sub-rows ("each additional …") inherit stem/elements from the preceding primary row; parent = that primary code
@dataclass(frozen=True) class CptInstruction: code: str; kind: str; text: str; targets: tuple[str, ...]   # kind ∈ {"use-with","not-with","not-with-time","see","other"}; targets = codes (ranges expanded ≤ 20)
@dataclass(frozen=True) class CptReference: code: str; kind: str; years: tuple[int, ...]; text: str     # kind ∈ {"cpt-changes","cpt-assistant"}
@dataclass(frozen=True) class CptEdition: year: int; sections: tuple[CptSection, ...]; codes: tuple[CptCode, ...]; instructions: tuple[CptInstruction, ...]; references: tuple[CptReference, ...]
def parse_epub(path: Path, *, year: int | None = None) -> CptEdition
def parse_xhtml(text: str, *, year: int) -> tuple[list[CptSection], list[CptCode], list[CptInstruction], list[CptReference]]   # one body file

Parsing rules (from the 2024/2022/2021 template): headings are <div class="h1|h2|h3|h4" id="sec_N">TITLE</div> (the * and range are in the TOC files *_Toc.xhtml, whose div.hN-toc entries carry TITLE* (LO-HI) — read them for lo/hi); guideline text = the div.noindent paragraphs between a heading and the next table or heading; code rows = tr with td.td-w1[id=code_NNNNN] (2019: no id — fall back to the <b>NNNNN</b> in div.table-para); symbols in span.ama-en glyphs: # resequenced, ✚ add-on, ● new, ▲ revised, ★ telemedicine (also accept the 2019 glyph set — verify on the 2019 file and document what differs); descriptor = table-para1 (stem) + table-slist* items (elements) + first table-para2 that does not start with ( (tail, e.g. the time clause); sub-rows use table-para1-sub; parentheticals = table-para2 starting with (; use-with = (Use X in conjunction with …), not-with = (Do not report … in the same … with …), not-with-time = (Do not report … for service time reported with …), see = (For …, see …) / (… use NNNNN); references = table-RT* lines (CPT Changes: An Insider's View 2015, 2021, 2022 → years; CPT Assistant … kept as text). HTML entities and <span class="ssp"/> bullets are stripped; whitespace normalised; the <i> and <b> tags removed from text.

  • Failing tests on the synthetic fixture: two sections with correct levels/paths and lo/hi from a synthetic TOC snippet; primary code has 3 elements and the tail; the add-on row has addon=True, parent=<primary>, inherited elements; use-with targets (primary,); not-with targets expanded from a range 90951-90970 (20 codes) plus listed codes; cpt-changes years (2015, 2021, 2022); entities/whitespace cleaned. Plus an integration test @pytest.mark.skipif(not Path(EPUB_2024).exists()) that parse_epub on the real 2024 file yields ≥ 8,000 codes, the section titled "Chronic Care Management Services" contains 99490/99439/99491/99437, and 99439 is an add-on whose use-with target is 99490 — asserting only structure, never descriptor text.
  • Implement with html.parser.HTMLParser (stdlib) building a flat event list, then a second pass grouping rows; keep parse_xhtml independently testable.
  • Commit: feat(pfs): CPT EPUB parser — sections, code entries with symbols and elements, parenthetical instructions, CPT Changes references (refs #687).

Task 3: pfs.cpt_* tables and stack pfs cpt-ingest

Files:

  • Modify: src/pfs/codetables.py (four DDLs, write_cpt_edition(con, edition, item_key), read_cpt_codes(con, year), read_cpt_sections(con, year)), Create: src/pfs/cpt_load.py, Modify: src/cli/pfs.py
  • Test: tests/pfs/test_cpt_load.py, tests/cli/test_pfs_cli.py (extend)

Tables (all with edition_year INTEGER, item_key VARCHAR first): pfs.cpt_section(sec_id, level, title, path, code_lo, code_hi, guideline), pfs.cpt_code(code, sec_id, descriptor, stem, elements, tail, addon, resequenced, new, revised, telemedicine, parent) (elements space-joined? No — elements is a VARCHAR[] list; DuckDB supports it), pfs.cpt_instruction(code, kind, text, targets) (targets VARCHAR[]), pfs.cpt_reference(code, kind, years INTEGER[], text). Delete-then-insert per edition_year.

cpt_load.ingest(store, con, *, years: Sequence[int] | None) -> dict: finds bib items tagged source:ama whose title matches CPT (Professional )?(\d{4}) with an EPUB attachment, parses each, writes, returns rows per table per year. CLI stack pfs cpt-ingest [--edition 2024]… [--all] [--dry-run]; --dry-run parses and prints counts only.

  • Failing tests: DDL idempotent; write/read round trip for a two-code synthetic edition; ingest with a fake store/attachment path pointing at a tiny EPUB built in the test (zip the fixture xhtml + a minimal container) writes rows; CLI test with _batch/_read/_store/_publish monkeypatched as in slice 1.
  • Implement; live: stack pfs cpt-ingest --all (2019, 2021, 2022, 2024 EPUBs; 2023 Changes is a different book — skip it here, note for later) → report rows per table per year, and one sanity query per year: the section holding 99490 and the count of add-on codes.
  • Commit: feat(pfs): pfs.cpt_section/cpt_code/cpt_instruction/cpt_reference + stack pfs cpt-ingest (refs #687).

Task 4: Families from the CPT hierarchy

Files: Modify src/pfs/families.py, src/cli/pfs.py (families command feeds CPT data), Test tests/pfs/test_families.py, tests/cli/test_pfs_cli.py

Ruling (binding): for CPT codes the family is the lowest CPT heading that groups ≥ 2 codes in the newest ingested edition (e.g. "Chronic Care Management Services" for 99490/99439/99491/99437; "Complex Chronic Care Management Services" for 99487/99489). Its key is a slug of the heading title (CHRONIC-CARE-MANAGEMENT-SERVICES), its name the heading title, and since = the earliest edition year the code appears in pfs.cpt_code. Slice-1 edges still apply on top (so G2058 still joins via its single-target replacement, and use-with instructions from pfs.cpt_instruction count as add-on edges). Hand families keep their keys and names: when a CPT heading's group intersects a hand family, the hand key/name wins for that component (Ruling 17), and the CPT heading title is stored as the family's note. Codes not in any CPT edition (HCPCS G-codes) keep the slice-1 derivation.

Interfaces:

def cpt_groups(cpt_codes: Sequence[CptCodeRow], cpt_sections: Sequence[CptSectionRow]) -> dict[str, tuple[str, str, tuple[str, ...]]]   # sec_id -> (key, title, codes) for the lowest heading with ≥2 codes
def derive_families(elements, events, descriptions, *, cpt_codes=(), cpt_sections=(), cpt_instructions=()) -> list[FamilyRow]           # new keyword inputs; existing calls unchanged

FamilyRow gains note: str = "" (append to the DDL and dataclass — last field, default, so slice-1 readers keep working; read_families reads it).

  • Failing tests: a synthetic edition with "Care Management Services" (h1) → "Chronic Care Management Services" (h2, 99490 99439 99491 99437) and "Complex …" (h2, 99487 99489): cpt_groups returns the two h2 groups, not the h1; derive_families with those inputs and the slice-1 CCM fixture yields the hand key CCM for the merged component with note == "Chronic Care Management Services" (99487/99489 join via slice-1 stem+activity as before); a synthetic "Remote Physiologic Monitoring Treatment Management Services" heading with 99457/99458 and no elements yields key REMOTE-PHYSIOLOGIC-MONITORING-TREATMENT-MANAGEMENT-SERVICES with since = edition year.
  • Implement; live: stack pfs families --write → report: total families, families with ≥ 2 codes (expect thousands now), the five hand families' membership and notes, and five sample CPT-named families.
  • Commit: feat(pfs): families from the CPT manual hierarchy — lowest multi-code heading names the family; use-with edges (refs #687).

Task 5: CPT Changes years as lineage events and CPT elements in extraction

Files: Modify src/pfs/lineage.py (cpt_events(con, code) from pfs.cpt_reference kind cpt-changes → EventRow(kind="cpt_changed", year=Y, source="cpt", item_key=<edition bib key>, anchored=True); lineage() merges them and an RVU event is also anchored by a cpt_changed within ±1 year), src/pfs/extract.py (extract_code adds ElementRows from the newest pfs.cpt_code row: parse_descriptor(stem + tail) plus each bulleted element parsed and, when the regex finds nothing, a ReviewRow with source="cpt"; FR rows still win on duplicates), src/cli/pfs.py (lineage prints the new kind).

  • Failing tests for both; implement; live: stack pfs lineage --code 99490 --write shows cpt_changed 2015/2021/2022 anchored to the 2024 edition item; stack pfs elements --family CCM --no-llm review queue does not grow.
  • Commit: feat(pfs): CPT Changes years as lineage events; CPT required-elements lists as an extraction source (refs #685 #686).

Task 6: Notebook — the CPT manual's own organizing principles

Files: Modify notebooks/code_families.py (section 1 gains "Where the manual files this code": the CPT heading path for the picked code from pfs.cpt_section/pfs.cpt_code and its instructions; section 4 shows the family's CPT heading note and the sibling headings under the same parent; a new short callout in section 0 that CPT's hierarchy is the first organizing principle and the FR is the second), tests/notebooks/test_code_families_nb.py (banner count unchanged; cells still degrade).

  • Implement; headless export clean; commit feat(notebooks): code_families — CPT hierarchy, instructions and family headings (refs #687).

Self-review

Coverage. User request "bring in the CPT manuals from Zotero so we have a better sense of organizing principles" → T1 (into bib), T2–T3 (structure into tables), T4 (organizing principle drives families), T5 (lineage/elements benefit), T6 (explainer). Copyright handling is a global constraint with a concrete mechanism (llm:skip, no text in docs/tracker/tests).

Placeholders. T2's parsing rules are exact selectors observed on the 2024/2022/2021 files; T1/T3–T6 give signatures, rules and test intents (mid-tier implementers).

Types. FamilyRow.note is appended last with a default so slice-1 readers/writers keep working; EventRow.kind="cpt_changed" and source="cpt" reuse the existing dataclass; ElementRow.source="cpt" reuses the existing column.

Reference in New Issue View Git Blame Copy Permalink