merge: P49 slice 2a + anchors 1–3 — CPT manual organizing principles (parser, cpt tables, CPT-named families, lineage/extraction sources), chunk anchors + restamp, explainer notebook (refs #684 #685 #686 #687 #688)
Some checks failed
CI / lint (push) Successful in 31s
CI / notebooks-smoke (push) Successful in 1m30s
Deploy / notebooks (push) Has been skipped
Deploy / zotero (push) Has been skipped
Deploy / docs (push) Has been skipped
Deploy / api (push) Has been skipped
Infra CI / notebooks (push) Successful in 1m1s
Infra CI / docs (push) Successful in 1m38s
Deploy / llm (push) Has been skipped
Deploy / mc (push) Has been skipped
Infra CI / zotero (push) Successful in 15s
Infra CI / api (push) Successful in 1m16s
Infra CI / llm (push) Successful in 46s
Infra CI / mc (push) Successful in 22s
Deploy / report (push) Successful in 13s
CI / test (push) Successful in 14m1s
Notebooks Integration / notebooks-integration (push) Successful in 8m49s
Zotero Sync / zotero-sync (push) Failing after 29s
Some checks failed
CI / lint (push) Successful in 31s
CI / notebooks-smoke (push) Successful in 1m30s
Deploy / notebooks (push) Has been skipped
Deploy / zotero (push) Has been skipped
Deploy / docs (push) Has been skipped
Deploy / api (push) Has been skipped
Infra CI / notebooks (push) Successful in 1m1s
Infra CI / docs (push) Successful in 1m38s
Deploy / llm (push) Has been skipped
Deploy / mc (push) Has been skipped
Infra CI / zotero (push) Successful in 15s
Infra CI / api (push) Successful in 1m16s
Infra CI / llm (push) Successful in 46s
Infra CI / mc (push) Successful in 22s
Deploy / report (push) Successful in 13s
CI / test (push) Successful in 14m1s
Notebooks Integration / notebooks-integration (push) Successful in 8m49s
Zotero Sync / zotero-sync (push) Failing after 29s
Branch worktree-p49-slice2-anchors (34 commits). Plans docs/superpowers/plans/2026-09-09-cpt-organizing-principles.md and docs/superpowers/plans/2026-09-09-code-family-anchors.md (Tasks 1–3); schema spec docs/superpowers/specs/2026-09-09-cpt-canonical-schema-design.md. Follow-ups #698 #699 #700 #703. Claude-Session: https://claude.ai/code/session_01Aum3pEMAM3yQVdFSdVe6Gc
This commit is contained in:
kert
38
docs/docs/cli/bib-import-zotero.md
Normal file
38
docs/docs/cli/bib-import-zotero.md
Normal file
@@ -0,0 +1,38 @@
|
||||
---
|
||||
title: stack bib import-zotero
|
||||
sidebar_position: 23
|
||||
---
|
||||
|
||||
# `stack bib import-zotero`
|
||||
|
||||
```
|
||||
Usage: stack bib import-zotero [OPTIONS]
|
||||
|
||||
Pull a Zotero collection's book items (with files) into bib.
|
||||
|
||||
The reverse of ``sync-zotero``: reads the Zotero SQLite database
|
||||
read-only and upserts each book as a bib ``Source`` item (dedup by
|
||||
URL), copying its attachment file(s) into bib storage — an ordinary
|
||||
corpus document like any other, indexed like everything else in
|
||||
bib. ``llm:skip`` is available as a generic opt-out for material
|
||||
that should not be embedded (see ``llm.source.iter_corpus_refs``)
|
||||
— e.g. a scratch export or a duplicate scan — but it's never added
|
||||
automatically here; pass ``--tag llm:skip`` explicitly if it
|
||||
applies.
|
||||
|
||||
╭─ Options ────────────────────────────────────────────────────────────────────╮
|
||||
│ * --collection TEXT Zotero collection name to pull │
|
||||
│ from. │
|
||||
│ [required] │
|
||||
│ --with-files --no-with-files Only import items that have at │
|
||||
│ least one attachment file. │
|
||||
│ [default: no-with-files] │
|
||||
│ --only TEXT Title substring filter │
|
||||
│ (case-insensitive). │
|
||||
│ --tag TEXT Tag every imported item with │
|
||||
│ this. Repeat for multiple. │
|
||||
│ --dry-run Report what would happen without │
|
||||
│ writing. │
|
||||
│ --help Show this message and exit. │
|
||||
╰──────────────────────────────────────────────────────────────────────────────╯
|
||||
```
|
||||
@@ -14,8 +14,43 @@ Usage: stack bib [OPTIONS] COMMAND [ARGS]...
|
||||
│ --help Show this message and exit. │
|
||||
╰──────────────────────────────────────────────────────────────────────────────╯
|
||||
╭─ Commands ───────────────────────────────────────────────────────────────────╮
|
||||
│ sync Apply col: tags as DuckDB column comments. │
|
||||
│ tag List or search bibliography tags. │
|
||||
│ query Search the bibliography store. │
|
||||
│ sync Apply col: tags as DuckDB column comments. │
|
||||
│ tag List or search bibliography tags. │
|
||||
│ query Search the bibliography store. │
|
||||
│ discover-pfs-rules Walk the Federal Register API for every PFS rule │
|
||||
│ since a date. │
|
||||
│ fetch-docket-comments Walk one docket's comments from its stored watermark; │
|
||||
│ upsert as Source items. │
|
||||
│ fetch-pfs-comments Discover every PFS proposed rule since *since* and │
|
||||
│ pull new comments │
|
||||
│ on each docket from its stored watermark. │
|
||||
│ ingest-mail Pull UNSEEN mail from a mailbox via IMAPS, upsert as │
|
||||
│ bib Sources. │
|
||||
│ backfill-comments Enrich every reg-gov comment stub with body + │
|
||||
│ attachments. │
|
||||
│ ingest-iom Crawl the CMS IOM index and upsert every chapter as a │
|
||||
│ Manual. │
|
||||
│ attach-iom Download the PDF for every Manual and attach it to │
|
||||
│ the record. │
|
||||
│ watch-iom Poll the CMS futurepdf.pdf schedule; attach if │
|
||||
│ changed. │
|
||||
│ import-zotero Pull a Zotero collection's book items (with files) │
|
||||
│ into bib. │
|
||||
│ sync-zotero Push bib items (and their attachments) to the Zotero │
|
||||
│ database. │
|
||||
│ ingest-oig Crawl OIG Compliance Program Guidance + fraud alerts │
|
||||
│ indexes. │
|
||||
│ attach-oig Download and attach the PDF/HTML for every OIG │
|
||||
│ guidance record. │
|
||||
│ refresh-oig One-shot: ingest-oig → attach-oig → sync-zotero (OIG │
|
||||
│ tag). │
|
||||
│ refresh-iom One-shot: ingest-iom → watch-iom → attach-iom → │
|
||||
│ sync-zotero. │
|
||||
│ fr-grab Grab federalregister.gov paragraph/page anchor maps │
|
||||
│ into bib (#634). │
|
||||
│ fr-jump Resolve an FR citation to its web jump link (#637); │
|
||||
│ optionally place it. │
|
||||
│ cfr-jump Bidirectional CFR cite ↔ eCFR URL (#640); optionally │
|
||||
│ place the link. │
|
||||
╰──────────────────────────────────────────────────────────────────────────────╯
|
||||
```
|
||||
|
||||
21
docs/docs/cli/pfs-cpt-ingest.md
Normal file
21
docs/docs/cli/pfs-cpt-ingest.md
Normal file
@@ -0,0 +1,21 @@
|
||||
---
|
||||
title: stack pfs cpt-ingest
|
||||
sidebar_position: 69
|
||||
---
|
||||
|
||||
# `stack pfs cpt-ingest`
|
||||
|
||||
```
|
||||
Usage: stack pfs cpt-ingest [OPTIONS]
|
||||
|
||||
Parse CPT EPUB editions into pfs.cpt_section/cpt_code/cpt_instruction/
|
||||
cpt_reference/cpt_crosswalk/cpt_list (docs/superpowers/specs/
|
||||
2026-09-09-cpt-canonical-schema-design.md §3).
|
||||
|
||||
╭─ Options ────────────────────────────────────────────────────────────────────╮
|
||||
│ --edition INTEGER CPT edition year to ingest (repeatable). │
|
||||
│ --all Every CPT EPUB edition found in bib (source:ama). │
|
||||
│ --dry-run Parse and report counts; write nothing. │
|
||||
│ --help Show this message and exit. │
|
||||
╰──────────────────────────────────────────────────────────────────────────────╯
|
||||
```
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: stack pfs families
|
||||
sidebar_position: 65
|
||||
sidebar_position: 67
|
||||
---
|
||||
|
||||
# `stack pfs families`
|
||||
@@ -12,6 +12,7 @@ Usage: stack pfs families [OPTIONS]
|
||||
|
||||
╭─ Options ────────────────────────────────────────────────────────────────────╮
|
||||
│ --write │
|
||||
│ --help Show this message and exit. │
|
||||
│ --verbose Print every family's members, not just the summary. │
|
||||
│ --help Show this message and exit. │
|
||||
╰──────────────────────────────────────────────────────────────────────────────╯
|
||||
```
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: stack pfs
|
||||
sidebar_position: 62
|
||||
sidebar_position: 64
|
||||
---
|
||||
|
||||
# `stack pfs`
|
||||
@@ -14,11 +14,16 @@ Usage: stack pfs [OPTIONS] COMMAND [ARGS]...
|
||||
│ --help Show this message and exit. │
|
||||
╰──────────────────────────────────────────────────────────────────────────────╯
|
||||
╭─ Commands ───────────────────────────────────────────────────────────────────╮
|
||||
│ elements Extract typed elements for codes into pfs.code_element (+ review │
|
||||
│ queue). │
|
||||
│ lineage Timeline of a code: RVU-file diffs and FR paragraphs, │
|
||||
│ cross-checked. │
|
||||
│ families Derive families from pfs.code_element / pfs.code_event / pfs.rvu. │
|
||||
│ review Element lines the classifier could not place. │
|
||||
│ elements Extract typed elements for codes into pfs.code_element (+ review │
|
||||
│ queue). │
|
||||
│ lineage Timeline of a code: RVU-file diffs and FR paragraphs, │
|
||||
│ cross-checked. │
|
||||
│ families Derive families from pfs.code_element / pfs.code_event / │
|
||||
│ pfs.rvu. │
|
||||
│ review Element lines the classifier could not place. │
|
||||
│ cpt-ingest Parse CPT EPUB editions into │
|
||||
│ pfs.cpt_section/cpt_code/cpt_instruction/ │
|
||||
│ cpt_reference/cpt_crosswalk/cpt_list (docs/superpowers/specs/ │
|
||||
│ 2026-09-09-cpt-canonical-schema-design.md §3). │
|
||||
╰──────────────────────────────────────────────────────────────────────────────╯
|
||||
```
|
||||
|
||||
731
docs/superpowers/plans/2026-09-09-code-family-anchors.md
Normal file
731
docs/superpowers/plans/2026-09-09-code-family-anchors.md
Normal file
@@ -0,0 +1,731 @@
|
||||
# Code-Family Anchors, Crosswalk, Reaction + Explainer Notebook (P49 slice 2: #688–#690) Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Every chunk and item in the corpus carries `codes`/`families`/`elements` anchors without re-embedding; each family knows its IOM and CFR sections; each family has a per-docket public-reaction series; and one marimo notebook walks a reader through every capability of P49 step by step on live data.
|
||||
|
||||
**Architecture:** Slice 1's pure modules (`pfs.elements`, `pfs.families`, `pfs.lineage`, `llm.classify`) are reused, never re-implemented. Anchors are computed once from chunk text and written two ways: a metadata-only `restamp` on pgvector (SQL `UPDATE … cmetadata || jsonb`) and `code:`/`family:` item tags in bib aggregated from that metadata. The crosswalk and reaction series are two more DuckDB tables in `pfs`, written through `duckdb_batch` and republished, exactly like slice 1. The notebook reads the replica and bib only, and degrades to a "not built yet" note per section when a table is absent.
|
||||
|
||||
**Tech Stack:** Python 3.13, DuckDB (replica reads, `duckdb_batch` writes), SQLite `bib.Store`, pgvector via SQLAlchemy `text()` (the `llm` engine), Ollama through `llm.classify`, marimo 0.23 + polars + Altair, typer, pytest.
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-09-09-code-family-longitudinal-design.md` (§Decisions 4, 7; §Components `bib/tag.py + bib/sync.py`, `llm/chunk.py + llm/index.py`, `bib/reaction.py`; §Components crosswalk is issue #689). Slice 1 rulings in `docs/superpowers/plans/2026-09-09-code-family-foundation.md` apply (no DuckDB at import in `pfs.*`, hand families are a minimum, self-hosted inference only).
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Self-hosted inference only: stance classification goes through `llm.classify.closed_vocab_classifier`; no cloud API.
|
||||
- Never re-embed to add metadata: anchors are backfilled with `UPDATE langchain_pg_embedding SET cmetadata = cmetadata || :patch::jsonb WHERE id = :id`; `--force` re-index is not part of this slice.
|
||||
- Chunk metadata stays the all-string dict convention: `codes`, `families`, `elements` are space-joined, sorted, unique strings (`""` when none).
|
||||
- Item tags are additive: `code:<HCPCS>` and `family:<KEY>` are added with `Store.add_tag`, never removed by this slice (the nightly Zotero sync is an additive union).
|
||||
- DuckDB writes only inside `conf.connect.duckdb_batch("aco")`; `conf.connect.publish_replica("aco")` after every write command; read-only commands and notebooks open the replica.
|
||||
- The notebook must import inside the notebooks container: `src/` is on `PYTHONPATH`; `llm` (pgvector) access is optional and guarded — the notebook must render fully with only the replica and `bib.sqlite`.
|
||||
- Never add a Co-Authored-By trailer. Commit after every task.
|
||||
- Test output pristine (`-W error::ResourceWarning`); every DB fixture yields and closes.
|
||||
|
||||
---
|
||||
|
||||
## File map
|
||||
|
||||
| File | Responsibility |
|
||||
|---|---|
|
||||
| `notebooks/code_families.py` (new, T1, extended T8) | Explainer notebook: one section per capability, live data, graceful degradation |
|
||||
| `src/pfs/anchors.py` (new, T2) | Pure: `anchor_metadata(text, *, families) -> dict[str,str]` computing `codes`/`families`/`elements` |
|
||||
| `src/llm/chunk.py` (modify, T2) | stamp `families`/`elements` beside `codes` |
|
||||
| `src/llm/migrate.py` (modify, T2) | GIN indexes on `families` and `elements` |
|
||||
| `src/llm/restamp.py` (new, T3) + `src/cli/llm.py` (modify) | metadata-only backfill: `stack llm restamp --collection all|comments|rules|corpus [--batch N] [--dry-run]` |
|
||||
| `src/bib/codetags.py` (new, T4) + `src/cli/bib.py` (modify) | `code:`/`family:` item tags from pgvector metadata + `fr_anchors`: `stack bib code-tags [--collection …] [--dry-run]` |
|
||||
| `src/llm/evidence.py` (modify, T5) | `code_cited_sources(engine, codes, *, per_code, collections=("rules","comments","corpus"))` |
|
||||
| `src/pfs/guidance.py` (new, T6) + `src/pfs/codetables.py` (modify) + `src/cli/pfs.py` (modify) | CFR/IOM cross-references per family → `pfs.code_guidance`; `stack pfs guidance --family X [--write]` |
|
||||
| `src/pfs/reaction.py` (new, T7) + `src/pfs/codetables.py` + `src/cli/pfs.py` | per-(family, docket) counts + stance, FR Comment/Response pairs → `pfs.code_reaction`; `stack pfs reaction --family X [--stance-sample N] [--write]` |
|
||||
| tests: `tests/pfs/test_anchors.py`, `tests/llm/test_chunk.py` (extend), `tests/llm/test_restamp.py`, `tests/bib/test_codetags.py`, `tests/llm/test_evidence.py` (extend), `tests/pfs/test_guidance.py`, `tests/pfs/test_reaction.py`, `tests/cli/test_pfs_cli.py` (extend), `tests/notebooks/test_code_families_nb.py` | one per module |
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Explainer notebook — sections 0–4 on live data (elements, extraction, lineage, families)
|
||||
|
||||
**Files:**
|
||||
- Create: `notebooks/code_families.py`
|
||||
- Test: `tests/notebooks/test_code_families_nb.py` (new dir with `__init__.py`)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `pfs.elements.{VOCAB, parse_descriptor, slug}`, `pfs.descriptors.descriptor_runs`, `pfs.codetables.{read_elements, read_events, read_families}`, `pfs.families.{HAND_FAMILIES, load_families}`, `conf.connect.{duckdb, bib, theme}`, `bib.frlink.md_link`, `conf.display.plain_years`.
|
||||
- Produces: a marimo app whose cells each carry a `# ── N. Title ──` banner; every data-dependent section wrapped so a missing table renders `mo.md("_Not built yet — run `stack pfs …`_")` instead of raising.
|
||||
|
||||
Design (what the reader sees, in order):
|
||||
|
||||
0. **Why a code is a bundle of elements** — prose + the three telehealth steps quoted live from `fr_anchors` (item `2KVJ2HKX` ¶394/¶396/¶398) with FR jump links via `md_link`.
|
||||
1. **Reading a descriptor** — a code picker (default `99490`; options = hand-family codes); `descriptor_runs(store, code)` → the stem paragraph and its element paragraphs as a table (`p_id`, text); then `parse_descriptor(run.text)` → typed elements table (`type`, `value`, `detail`) and the closed `VOCAB` as a reference table.
|
||||
2. **What the extractor wrote** — `pfs.code_element` rows for the picked code (element, anchor `item_key ¶p_id`, source) and the `pfs.code_element_review` lines; a one-paragraph explanation of deterministic-first, model-second, review-third.
|
||||
3. **Lineage** — `pfs.code_event` rows for the picked code's whole family as an Altair timeline (x = year, y = kind, colour = source, tooltip = anchor/note), the table beneath with FR links (`md_link(f"p-{p_id}", store=store, item_key=item_key)` inside try/except → plain `item_key ¶p_id` fallback), and a callout on the two dating rules (RVU-file year vs mentioning-rule year).
|
||||
4. **Families** — hand list vs derived (`read_families` filtered to the picked family; roles; `since/until`), and how the edges work (add-on, by-reference, single-target replacement, stem + activity) in prose with the fixture example G2058 → 99439.
|
||||
5–7. Placeholders rendering "Not built yet" until T8 (anchors, guidance, reaction).
|
||||
8. **Provenance** — the replica path and mtime, `pfs.code_*` row counts, `cms.ingest_log` filtered to `pfs.rvu` (as the other notebooks do).
|
||||
|
||||
- [ ] **Step 1: Write the failing test** — the test imports the notebook module, checks it defines `app`, that every cell function is anonymous (`_`), and that a headless run with the replica absent does not raise (all guards trip).
|
||||
|
||||
```python
|
||||
# tests/notebooks/__init__.py (empty)
|
||||
|
||||
# tests/notebooks/test_code_families_nb.py
|
||||
"""notebooks/code_families.py — structure and headless degradation."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
import importlib.util
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
NB = Path(__file__).resolve().parents[2] / "notebooks" / "code_families.py"
|
||||
|
||||
|
||||
def _load():
|
||||
spec = importlib.util.spec_from_file_location("code_families_nb", NB)
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(mod)
|
||||
return mod
|
||||
|
||||
|
||||
def test_notebook_is_a_marimo_app():
|
||||
mod = _load()
|
||||
assert mod.app.__class__.__name__ == "App"
|
||||
|
||||
|
||||
def test_cells_are_anonymous_and_banners_present():
|
||||
src = NB.read_text()
|
||||
tree = ast.parse(src)
|
||||
names = [n.name for n in tree.body if isinstance(n, ast.FunctionDef)]
|
||||
assert names and set(names) == {"_"}
|
||||
banners = re.findall(r"# ── (\d)\. ", src)
|
||||
assert [int(b) for b in banners] == list(range(0, 9))
|
||||
|
||||
|
||||
def test_headless_run_degrades_without_data(monkeypatch, tmp_path):
|
||||
"""With no replica and no bib the guards render notes instead of raising."""
|
||||
monkeypatch.setenv("STACK_DUCKDB_REPLICA", "1")
|
||||
mod = _load()
|
||||
import conf.connect as cc
|
||||
|
||||
monkeypatch.setattr(cc, "duckdb", lambda *a, **k: (_ for _ in ()).throw(FileNotFoundError("no replica")))
|
||||
monkeypatch.setattr(cc, "bib", lambda *a, **k: (_ for _ in ()).throw(FileNotFoundError("no bib")))
|
||||
result = mod.app.run() # marimo runs all cells; mo.stop cascades are fine
|
||||
assert result is not None
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run it to verify it fails**
|
||||
|
||||
Run: `uv run pytest tests/notebooks/test_code_families_nb.py -q`
|
||||
Expected: FAIL — `FileNotFoundError` on `NB.read_text()` / `spec_from_file_location` (notebook missing).
|
||||
|
||||
- [ ] **Step 3: Write `notebooks/code_families.py`**
|
||||
|
||||
Follow `notebooks/pfs_calcs.py` for the header and `notebooks/cy2027_pfs_proposed_rule.py` for the banner/prose style. Skeleton (every cell shown; fill prose as indicated):
|
||||
|
||||
```python
|
||||
import marimo
|
||||
|
||||
__generated_with = "0.23.13"
|
||||
app = marimo.App(width="medium")
|
||||
|
||||
|
||||
@app.cell(hide_code=True)
|
||||
def _():
|
||||
import marimo as mo
|
||||
return (mo,)
|
||||
|
||||
|
||||
@app.cell(hide_code=True)
|
||||
def _(mo):
|
||||
mo.md(
|
||||
"""
|
||||
# Code families as first-class objects
|
||||
|
||||
A physician fee schedule code is not a number — it is a **bundle of logical elements**:
|
||||
who furnishes the service, for how long, per what period, to which patients, doing which
|
||||
activities, by which modality. This notebook walks through how the stack turns that idea
|
||||
into tables you can query and cite: elements → extraction → lineage → families → anchors
|
||||
→ guidance → public reaction. Every number on this page is read live from the replica and
|
||||
the bibliography; every claim links to the Federal Register paragraph it came from.
|
||||
"""
|
||||
)
|
||||
return
|
||||
|
||||
|
||||
@app.cell(hide_code=True)
|
||||
def _():
|
||||
# ── 0. Setup ──
|
||||
import altair as alt
|
||||
import polars as pl
|
||||
|
||||
from conf import connect
|
||||
from conf.display import plain_years
|
||||
|
||||
connect.theme()
|
||||
NOTES = {}
|
||||
|
||||
def _open_replica():
|
||||
try:
|
||||
return connect.duckdb("aco", read_only=True)
|
||||
except Exception as e: # noqa: BLE001 — degrade, never crash the page
|
||||
NOTES["replica"] = f"replica unavailable: {e}"
|
||||
return None
|
||||
|
||||
def _open_bib():
|
||||
try:
|
||||
return connect.bib()
|
||||
except Exception as e: # noqa: BLE001
|
||||
NOTES["bib"] = f"bibliography unavailable: {e}"
|
||||
return None
|
||||
|
||||
con = _open_replica()
|
||||
store = _open_bib()
|
||||
|
||||
def q(sql, params=()):
|
||||
if con is None:
|
||||
return pl.DataFrame()
|
||||
try:
|
||||
return con.execute(sql, list(params)).pl()
|
||||
except Exception as e: # noqa: BLE001 — a missing table is a "not built yet"
|
||||
NOTES[sql[:40]] = str(e)
|
||||
return pl.DataFrame()
|
||||
|
||||
def fr_md(item_key, p_id):
|
||||
if store is None:
|
||||
return f"{item_key} ¶{p_id}"
|
||||
try:
|
||||
from bib.frlink import md_link
|
||||
|
||||
return md_link(f"p-{p_id}", store=store, item_key=item_key, text=f"{item_key} ¶{p_id}")
|
||||
except Exception: # noqa: BLE001
|
||||
return f"{item_key} ¶{p_id}"
|
||||
|
||||
def not_built(cmd):
|
||||
return f"_Not built yet — run `{cmd}` and republish the replica._"
|
||||
|
||||
return NOTES, alt, con, fr_md, not_built, pl, plain_years, q, store
|
||||
```
|
||||
|
||||
Then, one cell per section, in this order (prose in `mo.md`, data in `mo.ui.table` / Altair):
|
||||
|
||||
```python
|
||||
@app.cell(hide_code=True)
|
||||
def _(fr_md, mo, store):
|
||||
# ── 0. Why a code is a bundle of elements ──
|
||||
_steps = []
|
||||
if store is not None:
|
||||
try:
|
||||
_con = store._con() # noqa: SLF001
|
||||
for _p in (394, 396, 398):
|
||||
_row = _con.execute(
|
||||
"SELECT text FROM fr_anchors WHERE item_key = ? AND p_id = ?", ("2KVJ2HKX", _p)
|
||||
).fetchone()
|
||||
if _row:
|
||||
_steps.append(f"> {_row[0][:400]}… — {fr_md('2KVJ2HKX', _p)}")
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
mo.md(
|
||||
"## 0. Why a code is a bundle of elements\n\n"
|
||||
"CMS says so itself. When it decides whether a service can be furnished by telehealth it "
|
||||
"walks three steps, and the third is literally *review the elements of the service as "
|
||||
"described by the HCPCS code* (CY2026 proposed rule, 90 FR 32389):\n\n"
|
||||
+ ("\n\n".join(_steps) if _steps else "_(bibliography unavailable — quotes omitted)_")
|
||||
+ "\n\nEverything below is built on that one sentence."
|
||||
)
|
||||
return
|
||||
```
|
||||
|
||||
```python
|
||||
@app.cell(hide_code=True)
|
||||
def _(mo):
|
||||
# ── 1. Reading a descriptor ──
|
||||
from pfs.families import HAND_FAMILIES
|
||||
|
||||
_codes = sorted({c for f in HAND_FAMILIES.values() for c in f.codes})
|
||||
code_picker = mo.ui.dropdown(options=_codes, value="99490", label="Code")
|
||||
mo.vstack([mo.md("## 1. Reading a descriptor"), code_picker])
|
||||
return (code_picker,)
|
||||
|
||||
|
||||
@app.cell(hide_code=True)
|
||||
def _(code_picker, fr_md, mo, pl, store):
|
||||
from pfs.descriptors import descriptor_runs
|
||||
from pfs.elements import VOCAB, parse_descriptor
|
||||
|
||||
code = code_picker.value
|
||||
runs = descriptor_runs(store, code) if store is not None else []
|
||||
if not runs:
|
||||
_view = mo.md(f"_No Federal Register descriptor run found for {code} (bibliography unavailable or code never printed as a stem)._")
|
||||
elements = ()
|
||||
else:
|
||||
run = runs[-1] # the latest rule that prints the descriptor
|
||||
_paras = pl.DataFrame(
|
||||
{
|
||||
"p_id": [run.stem.p_id, *[p.p_id for p in run.elements]],
|
||||
"role": ["stem", *["element"] * len(run.elements)],
|
||||
"text": [run.stem.text[:300], *[p.text[:300] for p in run.elements]],
|
||||
}
|
||||
)
|
||||
elements = parse_descriptor(run.text)
|
||||
_els = pl.DataFrame(
|
||||
{"type": [e.type.value for e in elements], "value": [e.value for e in elements], "detail": [e.detail for e in elements]}
|
||||
)
|
||||
_vocab = pl.DataFrame({"type": [t.value for t in VOCAB for _ in VOCAB[t]], "value": [v for t in VOCAB for v in VOCAB[t]]})
|
||||
_view = mo.vstack(
|
||||
[
|
||||
mo.md(
|
||||
f"**{code}** as printed in {run.item_key} (CY{run.rule_year}), stem {fr_md(run.item_key, run.stem.p_id)}: "
|
||||
"the stem paragraph opens the descriptor and each following paragraph is one element."
|
||||
),
|
||||
mo.ui.table(_paras, label="Descriptor paragraphs"),
|
||||
mo.md("The deterministic parser reads what a regex can read — minutes, periods, code references, and the recurring phrases:"),
|
||||
mo.ui.table(_els, label="Typed elements"),
|
||||
mo.accordion({"The closed vocabulary (values grow only by review)": mo.ui.table(_vocab)}),
|
||||
]
|
||||
)
|
||||
_view
|
||||
return code, elements
|
||||
```
|
||||
|
||||
Sections 2, 3, 4 read the replica via `q(...)`:
|
||||
|
||||
```python
|
||||
@app.cell(hide_code=True)
|
||||
def _(code, mo, not_built, q):
|
||||
# ── 2. What the extractor wrote ──
|
||||
_els = q("SELECT type, value, detail, source, item_key, p_id, page FROM pfs.code_element WHERE code = ? ORDER BY type, value", (code,))
|
||||
_rev = q("SELECT text, proposed_value, item_key, p_id FROM pfs.code_element_review WHERE code = ? ORDER BY p_id", (code,))
|
||||
if _els.is_empty():
|
||||
_view = mo.md("## 2. What the extractor wrote\n\n" + not_built(f"stack pfs elements --code {code}"))
|
||||
else:
|
||||
_view = mo.vstack(
|
||||
[
|
||||
mo.md(
|
||||
"## 2. What the extractor wrote\n\n"
|
||||
"Three passes, in order: the regex parser, then the local model choosing **one slug from the closed list or `none`**, "
|
||||
"then a review queue for what neither could place. Nothing enters `pfs.code_element` unless it is in the vocabulary."
|
||||
),
|
||||
mo.ui.table(_els, label=f"pfs.code_element — {code} ({_els.height} rows)"),
|
||||
mo.ui.table(_rev, label=f"pfs.code_element_review — {code} ({_rev.height} lines)") if not _rev.is_empty() else mo.md("_Review queue empty for this code._"),
|
||||
]
|
||||
)
|
||||
_view
|
||||
return
|
||||
```
|
||||
|
||||
```python
|
||||
@app.cell(hide_code=True)
|
||||
def _(alt, code, fr_md, mo, not_built, plain_years, q):
|
||||
# ── 3. Lineage ──
|
||||
from pfs.families import HAND_FAMILIES, family_of
|
||||
|
||||
_fam = family_of(code)
|
||||
_codes = list(_fam.codes) if _fam else [code]
|
||||
_ev = q(
|
||||
"SELECT code, year, kind, from_codes, to_codes, source, anchored, item_key, p_id, note FROM pfs.code_event "
|
||||
"WHERE code IN (" + ",".join("?" * len(_codes)) + ") ORDER BY year, code, kind",
|
||||
_codes,
|
||||
)
|
||||
if _ev.is_empty():
|
||||
_view = mo.md("## 3. Lineage\n\n" + not_built(f"stack pfs lineage --code {code} --write"))
|
||||
else:
|
||||
_chart = (
|
||||
alt.Chart(_ev.to_pandas())
|
||||
.mark_circle(size=90)
|
||||
.encode(
|
||||
x=alt.X("year:O", title="Rule year"),
|
||||
y=alt.Y("kind:N", title=None),
|
||||
color=alt.Color("source:N", title="Source"),
|
||||
shape=alt.Shape("anchored:N", title="Anchored"),
|
||||
tooltip=["code", "year", "kind", "from_codes", "to_codes", "item_key", "p_id", "note"],
|
||||
)
|
||||
.properties(height=260, width=640)
|
||||
)
|
||||
_links = _ev.with_columns(
|
||||
anchor=_ev.select("item_key", "p_id").map_rows(lambda r: fr_md(r[0], r[1]) if r[0] else "rvu")["map"]
|
||||
)
|
||||
_view = mo.vstack(
|
||||
[
|
||||
mo.md(
|
||||
"## 3. Lineage\n\n"
|
||||
f"Every dated event for the **{_fam.name if _fam else code}** codes. RVU-file events (`source=rvu`) are dated by the "
|
||||
"fee-schedule year; Federal Register events are dated by the **rule that mentions them** — a later rule recounting a "
|
||||
"code's creation adds a later `created` row, which is why the earliest anchored event is the origin. An RVU event is "
|
||||
"`anchored` when a Federal Register event for the same code lies within one rule year."
|
||||
),
|
||||
mo.ui.altair_chart(_chart),
|
||||
mo.ui.table(plain_years(_links.drop("item_key", "p_id")), label="pfs.code_event"),
|
||||
]
|
||||
)
|
||||
_view
|
||||
return
|
||||
```
|
||||
|
||||
```python
|
||||
@app.cell(hide_code=True)
|
||||
def _(code, mo, not_built, plain_years, q):
|
||||
# ── 4. Families ──
|
||||
from pfs.families import HAND_FAMILIES, family_of
|
||||
|
||||
_hand = family_of(code)
|
||||
_key = _hand.key if _hand else ""
|
||||
_rows = q("SELECT key, name, code, role, since, until, item_key, p_id FROM pfs.code_family WHERE key = ? ORDER BY code", (_key,)) if _key else None
|
||||
if _rows is None or _rows.is_empty():
|
||||
_view = mo.md("## 4. Families\n\n" + not_built("stack pfs families --write"))
|
||||
else:
|
||||
_view = mo.vstack(
|
||||
[
|
||||
mo.md(
|
||||
"## 4. Families\n\n"
|
||||
"A family is a connected component over four kinds of edge: an **add-on** relation, a **defined-by-reference** relation, "
|
||||
"a **single-target replacement** event, and **stem similarity with an identical activity set**. The hand-written registry "
|
||||
f"is a floor, never a ceiling: **{_key}** lists {len(_hand.codes)} hand codes; the derived table shows {_rows.height}."
|
||||
),
|
||||
mo.ui.table(plain_years(_rows), label=f"pfs.code_family — {_key}"),
|
||||
mo.md("Example: G2058 (2020 only) was replaced by CPT 99439 with an identical descriptor — a single-target `replaced_by` event, so G2058 joins CCM as a *predecessor*."),
|
||||
]
|
||||
)
|
||||
_view
|
||||
return
|
||||
```
|
||||
|
||||
Sections 5–7 are stubs in T1 (each a hidden cell with the banner and `mo.md(not_built("…"))`), and section 8:
|
||||
|
||||
```python
|
||||
@app.cell(hide_code=True)
|
||||
def _(NOTES, con, mo, q):
|
||||
# ── 8. Provenance ──
|
||||
_counts = q(
|
||||
"SELECT 'code_element' t, count(*) n FROM pfs.code_element UNION ALL SELECT 'code_event', count(*) FROM pfs.code_event "
|
||||
"UNION ALL SELECT 'code_family', count(*) FROM pfs.code_family"
|
||||
)
|
||||
_log = q("SELECT * FROM cms.ingest_log WHERE table_name LIKE 'pfs.%' ORDER BY ingested_at DESC LIMIT 20")
|
||||
mo.vstack(
|
||||
[
|
||||
mo.md("## 8. Provenance"),
|
||||
mo.ui.table(_counts, label="pfs.code_* row counts") if not _counts.is_empty() else mo.md("_replica unavailable_"),
|
||||
mo.ui.table(_log, label="Ingest log — pfs.*") if not _log.is_empty() else mo.md("_no ingest log rows_"),
|
||||
mo.md("\n".join(f"- {k}: {v}" for k, v in NOTES.items()) if NOTES else "_All sources available._"),
|
||||
]
|
||||
)
|
||||
return
|
||||
```
|
||||
|
||||
(`cms.ingest_log` column names: check `notebooks/cy2027_pfs_proposed_rule.py:1580-1591` and copy the exact query used there.)
|
||||
|
||||
- [ ] **Step 4: Run the notebook headlessly and the test**
|
||||
|
||||
Run: `uv run marimo export session notebooks/code_families.py --no-sandbox --force-overwrite --continue-on-error -o /tmp/claude-1000/-home-kert-stack/fe33b7f6-fbbd-4b5e-a2ef-aac20e2431fc/scratchpad/code_families.session.json` (with the data links present) and `uv run python dev/scripts/nb_integration.py --set code_families.py` if that entry point accepts a single notebook (read its `--help`); then `uv run pytest tests/notebooks -q`.
|
||||
Expected: no cell errors in the session JSON; sections 0–4 populated for 99490; 5–7 show "Not built yet"; test PASS.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add notebooks/code_families.py tests/notebooks/__init__.py tests/notebooks/test_code_families_nb.py
|
||||
git commit -m "feat(notebooks): code_families explainer — elements, extraction, lineage, families on live data (refs #684-#687)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Anchor metadata on chunks — `families` and `elements` beside `codes`
|
||||
|
||||
**Files:**
|
||||
- Create: `src/pfs/anchors.py`
|
||||
- Modify: `src/llm/chunk.py:151,193` (both metadata dicts), `src/llm/migrate.py` (`_HNSW_DDL`)
|
||||
- Test: `tests/pfs/test_anchors.py`, `tests/llm/test_chunk.py` (extend), `tests/llm/test_migrate.py` (extend)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces:
|
||||
```python
|
||||
def anchor_metadata(text: str, *, families: Mapping[str, Family] | None = None) -> dict[str, str]
|
||||
# {"codes": "99439 99490", "families": "CCM", "elements": "activity=consent time=first"}; families default = pfs.families.FAMILIES (live)
|
||||
```
|
||||
`elements` = sorted unique `f"{type}={value}"` (no detail) from `parse_descriptor(text)`; `families` = sorted unique keys of `family_of(code)` for each code.
|
||||
|
||||
- [ ] **Step 1: Failing tests**
|
||||
|
||||
```python
|
||||
# tests/pfs/test_anchors.py
|
||||
from pfs.anchors import anchor_metadata
|
||||
from pfs.families import Family
|
||||
|
||||
|
||||
def test_codes_families_elements():
|
||||
md = anchor_metadata("Use 99439 in conjunction with 99490; consent; per calendar month")
|
||||
assert md["codes"] == "99439 99490"
|
||||
assert md["families"] == "CCM"
|
||||
assert "activity=consent" in md["elements"].split() and "period=calendar-month" in md["elements"].split()
|
||||
assert "relation=addon-of" in md["elements"].split() # detail dropped
|
||||
|
||||
|
||||
def test_empty_and_unknown():
|
||||
assert anchor_metadata("nothing here") == {"codes": "", "families": "", "elements": ""}
|
||||
fams = {"X": Family("X", "X fam", ("12345",), ("x fam",))}
|
||||
assert anchor_metadata("code 12345", families=fams)["families"] == "X"
|
||||
```
|
||||
|
||||
Extend `tests/llm/test_chunk.py`: for a paragraph doc and a plain doc, every chunk's metadata has keys `codes`, `families`, `elements` and a chunk mentioning `G0556` has `families == "APCM"`. Extend `tests/llm/test_migrate.py`: `_HNSW_DDL` contains `ix_embedding_families` and `ix_embedding_elements` GIN statements shaped like `ix_embedding_codes`.
|
||||
|
||||
- [ ] **Step 2: Run to fail.** `uv run pytest tests/pfs/test_anchors.py tests/llm/test_chunk.py tests/llm/test_migrate.py -q` → ImportError / KeyError.
|
||||
|
||||
- [ ] **Step 3: Implement**
|
||||
|
||||
```python
|
||||
# src/pfs/anchors.py
|
||||
"""Anchor metadata for a piece of text: the codes it names, the families
|
||||
those codes belong to, and the element slugs its phrases match. One
|
||||
function, used by the indexer at chunk time and by the restamp backfill,
|
||||
so both write identical values. Pure."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Mapping
|
||||
|
||||
from pfs.elements import parse_descriptor
|
||||
from pfs.families import FAMILIES, Family, find_codes
|
||||
|
||||
|
||||
def anchor_metadata(text: str, *, families: Mapping[str, Family] | None = None) -> dict[str, str]:
|
||||
fams = FAMILIES if families is None else families
|
||||
codes = find_codes(text)
|
||||
keys = sorted({k for k, f in fams.items() for c in codes if c in f.codes})
|
||||
elements = sorted({f"{e.type.value}={e.value}" for e in parse_descriptor(text)})
|
||||
return {"codes": " ".join(codes), "families": " ".join(keys), "elements": " ".join(elements)}
|
||||
```
|
||||
|
||||
In `src/llm/chunk.py` replace both `"codes": " ".join(find_codes(...))` entries with `**anchor_metadata(f"{heading} {piece}")` / `**anchor_metadata(text)` (import `from pfs.anchors import anchor_metadata`; keep `find_codes` import only if still used). In `src/llm/migrate.py` append to `_HNSW_DDL`:
|
||||
|
||||
```python
|
||||
"CREATE INDEX IF NOT EXISTS ix_embedding_families "
|
||||
"ON langchain_pg_embedding USING gin "
|
||||
"(string_to_array(coalesce(cmetadata->>'families',''), ' '))",
|
||||
"CREATE INDEX IF NOT EXISTS ix_embedding_elements "
|
||||
"ON langchain_pg_embedding USING gin "
|
||||
"(string_to_array(coalesce(cmetadata->>'elements',''), ' '))",
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run to pass**, plus `uv run pytest tests/llm -q`.
|
||||
- [ ] **Step 5: Commit** — `feat(pfs,llm): anchor metadata — families and elements stamped beside codes; GIN indexes (refs #688)`.
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Metadata-only backfill — `stack llm restamp`
|
||||
|
||||
**Files:**
|
||||
- Create: `src/llm/restamp.py`
|
||||
- Modify: `src/cli/llm.py` (new command)
|
||||
- Test: `tests/llm/test_restamp.py`
|
||||
|
||||
**Interfaces:**
|
||||
```python
|
||||
def restamp(engine, *, collection: str, batch: int = 2000, dry_run: bool = False, families=None, log_every: int = 50_000) -> dict
|
||||
# {"scanned": n, "updated": n} — iterates id, document, cmetadata ordered by id in batches (keyset pagination on id),
|
||||
# computes anchor_metadata(document), skips rows whose three values already match, else
|
||||
# UPDATE langchain_pg_embedding SET cmetadata = cmetadata || :patch WHERE id = :id (executemany per batch, one transaction per batch)
|
||||
```
|
||||
CLI: `stack llm restamp --collection all|comments|rules|corpus [--batch 2000] [--dry-run]` — loads `.env`-dependent config the way `index` does, calls `refresh_from` on the replica when `cfg.duckdb_replica` exists (so derived families stamp), runs `migrate`/`ensure_hnsw` first so the new GIN indexes exist, prints per-collection counts and elapsed time.
|
||||
|
||||
- [ ] **Step 1: Failing test** — uses SQLite via SQLAlchemy? No: pgvector-specific `||` jsonb. Test the pure parts: `plan_patches(rows) -> list[tuple[id, patch_json]]` (skips unchanged) and the SQL text constants; and an integration test marked `@pytest.mark.skipif(not os.environ.get("LLM_DB_PASSWORD"), reason="needs pgvector")` that restamps `--collection rules --batch 50 --dry-run` against the live DB and asserts `scanned > 0`.
|
||||
|
||||
```python
|
||||
# tests/llm/test_restamp.py
|
||||
import json
|
||||
from llm.restamp import plan_patches, SELECT_SQL, UPDATE_SQL
|
||||
|
||||
|
||||
def test_plan_skips_unchanged_and_patches_changed():
|
||||
rows = [
|
||||
("a", "Use 99439 with 99490; consent", {"codes": "99439 99490", "families": "CCM", "elements": "activity=consent relation=addon-of"}),
|
||||
("b", "Use 99439 with 99490; consent", {"codes": "99439 99490"}),
|
||||
("c", "nothing", {}),
|
||||
]
|
||||
patches = plan_patches(rows)
|
||||
ids = [p[0] for p in patches]
|
||||
assert ids == ["b", "c"]
|
||||
assert json.loads(patches[0][1])["families"] == "CCM"
|
||||
assert json.loads(patches[1][1]) == {"codes": "", "families": "", "elements": ""}
|
||||
|
||||
|
||||
def test_sql_shapes():
|
||||
assert "ORDER BY e.id" in SELECT_SQL and ":after" in SELECT_SQL and ":batch" in SELECT_SQL
|
||||
assert "cmetadata || CAST(:patch AS jsonb)" in UPDATE_SQL
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run to fail.**
|
||||
- [ ] **Step 3: Implement**
|
||||
|
||||
```python
|
||||
# src/llm/restamp.py
|
||||
"""Backfill anchor metadata on already-indexed chunks without re-embedding.
|
||||
|
||||
Re-indexing 1.15 M chunks costs days of GPU time; the anchors are a pure
|
||||
function of the chunk text, so they are recomputed in Python and merged
|
||||
into ``cmetadata`` with a jsonb concatenation. Keyset pagination on the
|
||||
primary key keeps memory flat; one transaction per batch keeps the lock
|
||||
short. Idempotent: rows whose three values already match are skipped.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
from typing import Any, Iterable, Mapping
|
||||
|
||||
from sqlalchemy import text
|
||||
|
||||
from pfs.anchors import anchor_metadata
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
KEYS = ("codes", "families", "elements")
|
||||
|
||||
SELECT_SQL = (
|
||||
"SELECT e.id, e.document, e.cmetadata FROM langchain_pg_embedding e "
|
||||
"WHERE e.collection_id = (SELECT uuid FROM langchain_pg_collection WHERE name = :collection) "
|
||||
"AND e.id > :after ORDER BY e.id LIMIT :batch"
|
||||
)
|
||||
UPDATE_SQL = "UPDATE langchain_pg_embedding SET cmetadata = cmetadata || CAST(:patch AS jsonb) WHERE id = :id"
|
||||
|
||||
|
||||
def plan_patches(rows: Iterable[tuple[str, str, Mapping[str, Any] | None]], *, families=None) -> list[tuple[str, str]]:
|
||||
out: list[tuple[str, str]] = []
|
||||
for id_, document, md in rows:
|
||||
md = md or {}
|
||||
want = anchor_metadata(document or "", families=families)
|
||||
if all(str(md.get(k, "")) == want[k] for k in KEYS):
|
||||
continue
|
||||
out.append((id_, json.dumps(want)))
|
||||
return out
|
||||
|
||||
|
||||
def restamp(engine: Any, *, collection: str, batch: int = 2000, dry_run: bool = False, families=None, log_every: int = 50_000) -> dict:
|
||||
scanned = updated = 0
|
||||
after = ""
|
||||
t0 = time.perf_counter()
|
||||
while True:
|
||||
with engine.begin() as conn:
|
||||
rows = conn.execute(text(SELECT_SQL), {"collection": collection, "after": after, "batch": batch}).fetchall()
|
||||
if not rows:
|
||||
break
|
||||
patches = plan_patches(rows, families=families)
|
||||
if patches and not dry_run:
|
||||
conn.execute(text(UPDATE_SQL), [{"id": i, "patch": p} for i, p in patches])
|
||||
scanned += len(rows)
|
||||
updated += len(patches)
|
||||
after = rows[-1][0]
|
||||
if scanned % log_every < batch:
|
||||
log.info("restamp %s: scanned %d updated %d (%.0fs)", collection, scanned, updated, time.perf_counter() - t0)
|
||||
return {"collection": collection, "scanned": scanned, "updated": updated, "seconds": round(time.perf_counter() - t0, 1)}
|
||||
```
|
||||
|
||||
CLI in `src/cli/llm.py` (mirror `index` for config/engine setup; `_COLLECTIONS` fan-out for `all`; call `migrate(engine)` and `ensure_hnsw(engine, cfg.embed_dim)` before restamping; `refresh_from` guarded by `os.path.exists(cfg.duckdb_replica)`).
|
||||
|
||||
- [ ] **Step 4: Run to pass**; then live: `bash <withenv> uv run stack llm restamp --collection rules` (67k chunks, minutes), then `--collection corpus`, then `--collection comments` (918k — run in background with `nohup`, expect ~20–40 min; record the counts). Verify: `SELECT count(*) FROM langchain_pg_embedding WHERE cmetadata ? 'families'` equals the total.
|
||||
- [ ] **Step 5: Commit** — `feat(llm): stack llm restamp — metadata-only anchor backfill on pgvector (refs #688)`.
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Item tags `code:` / `family:` in bib — `stack bib code-tags`
|
||||
|
||||
**Files:**
|
||||
- Create: `src/bib/codetags.py`
|
||||
- Modify: `src/bib/tag.py` (docstring + `Tag.code`, `Tag.family` factories), `src/cli/bib.py`
|
||||
- Test: `tests/bib/test_codetags.py`
|
||||
|
||||
**Interfaces:**
|
||||
```python
|
||||
def item_codes_from_chunks(engine, *, collection: str) -> dict[str, tuple[set[str], set[str]]] # item_key -> (codes, families) via GROUP BY on cmetadata->>'item_key'
|
||||
def item_codes_from_anchors(store) -> dict[str, tuple[set[str], set[str]]] # rules: fr_anchors.text per item_key via anchor_metadata (no pgvector needed)
|
||||
def apply_code_tags(store, mapping, *, dry_run=False, max_codes_per_item=200) -> dict # {"items": n, "tags_added": n, "skipped_wide": n}
|
||||
```
|
||||
Rule: an item gets `code:X` for every code and `family:K` for every family; items with more than `max_codes_per_item` distinct codes (RVU files, code lists) get only `family:` tags plus `code:` for hand-family codes — a tag cloud of 8k codes helps nobody. `Store.add_tag` only stamps when new, so reruns are cheap.
|
||||
|
||||
CLI: `stack bib code-tags [--collection all|comments|rules|corpus] [--anchors-only] [--dry-run]`.
|
||||
|
||||
- [ ] **Step 1: Failing tests** — `apply_code_tags` against an in-memory `Store` (use `bib.Store(path=":memory:")` if supported — check `src/bib/store.py:51`; else `tmp_path / "bib.sqlite"`): two items, one wide (300 codes) → only family tags + hand codes; `item_codes_from_anchors` with two `fr_anchors` rows; `Tag.code("G0556").label == "code:G0556"`.
|
||||
- [ ] **Step 2: Run to fail.**
|
||||
- [ ] **Step 3: Implement** (SQL for chunks: `SELECT cmetadata->>'item_key', string_agg(cmetadata->>'codes',' '), string_agg(cmetadata->>'families',' ') FROM langchain_pg_embedding WHERE collection_id = (…) AND coalesce(cmetadata->>'codes','') <> '' GROUP BY 1`).
|
||||
- [ ] **Step 4: Run to pass**; live: `bash <withenv> uv run stack bib code-tags --collection all` and `uv run stack bib tag --namespace code | head` (or whatever `stack bib tag` lists) to show counts; the nightly `zotero-sync.yml` carries them (no change needed — confirm by reading the workflow's `--tag` scopes; if it only syncs `source:email`/`source:federal-register`, add a third invocation `--tag family:CCM`? No — ruling: leave the nightly scopes as they are; tags ride along on items already in scope).
|
||||
- [ ] **Step 5: Commit** — `feat(bib): code:/family: item tags from chunk anchors and FR paragraphs — stack bib code-tags (refs #688)`.
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Code-cited sources across collections in the chat
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/llm/evidence.py` (`code_cited_sources` signature + `_CITED_SQL` per collection loop), `src/llm/rag.py:150` call site, `src/llm/config.py` + `stack.toml` (`code_cited_collections = ["rules","comments","corpus"]`)
|
||||
- Test: `tests/llm/test_evidence.py` (extend), `tests/llm/test_config.py` (extend)
|
||||
|
||||
**Interfaces:** `code_cited_sources(engine, codes, *, per_code, collections=("rules",)) -> list[dict]` — runs the existing query once per collection, per-code cap applies per collection, results concatenated rules → comments → corpus; sources carry `kind` from metadata so `llm/links.py` builds the right link.
|
||||
|
||||
- [ ] Steps: failing test (fake engine returning rows per collection; assert order and caps), implement, run `tests/llm -q`, commit `feat(llm): code-cited sources from comments and corpus too (refs #688)`.
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Sub-regulatory crosswalk — `pfs.code_guidance` + `stack pfs guidance`
|
||||
|
||||
**Files:**
|
||||
- Create: `src/pfs/guidance.py`
|
||||
- Modify: `src/pfs/codetables.py` (`GuidanceRow`, DDL, `write_guidance`, `read_guidance`), `src/cli/pfs.py`
|
||||
- Test: `tests/pfs/test_guidance.py`, `tests/cli/test_pfs_cli.py` (extend)
|
||||
|
||||
**Interfaces:**
|
||||
```python
|
||||
@dataclass(frozen=True) class GuidanceRow: family: str; code: str; kind: str; locator: str; item_key: str; item_key_src: str; p_id_src: int; page_src: int
|
||||
# kind ∈ {"cfr","iom","mln"}; locator = "42 CFR 410.78" | "100-04 ch.12 §30.6.x" | title; item_key = resolved bib item ("" when unresolved)
|
||||
CFR_RE = re.compile(r"\b(\d{1,2})\s*CFR\s*(?:part\s*)?(\d{3,4})\.(\d+[a-z]?)", re.I) # + "§ 410.78" with title inferred as 42
|
||||
IOM_RE = re.compile(r"(?:Pub(?:lication|\.)?\s*100-0(\d)|(Claims Processing|Benefit Policy|Program Integrity) Manual)[^.]{0,80}?[Cc]hapter\s*(\d+)(?:[^.]{0,40}?[Ss]ection\s*([\d.]+))?")
|
||||
def harvest(store, codes: Sequence[str], *, family: str) -> list[GuidanceRow] # paragraphs whose text names any code → regex → resolve
|
||||
def resolve_cfr(store, title: str, section: str) -> str # bib.cfrlink.item_for(parse_cite(f"{title} CFR {section}"), store)
|
||||
def resolve_iom(store, pub: str, chapter: str) -> str # items tagged pub:100-0N whose title contains f"Chapter {chapter}:" (Store.list_items(tag=…))
|
||||
```
|
||||
CLI: `stack pfs guidance --family CCM [--family …] [--write]` (read-only without `--write`, same shape as `families`).
|
||||
|
||||
- [ ] Steps: failing tests (regexes on three real paragraph snippets: `2KVJ2HKX` ¶398 "§ 410.78(a)(3)", a "Medicare Claims Processing Manual, Chapter 12, Section 30.6.4" sentence, a "Pub. 100-04, chapter 12" sentence; resolver with a stub store), implement, live run for the five hand families and record rows per family, commit `feat(pfs): sub-regulatory crosswalk — CFR/IOM references per family with FR anchors (refs #689)`.
|
||||
|
||||
---
|
||||
|
||||
### Task 7: Public-reaction series — `pfs.code_reaction` + `stack pfs reaction`
|
||||
|
||||
**Files:**
|
||||
- Create: `src/pfs/reaction.py`
|
||||
- Modify: `src/pfs/codetables.py` (`ReactionRow`, DDL, writers/readers), `src/cli/pfs.py`
|
||||
- Test: `tests/pfs/test_reaction.py`, `tests/cli/test_pfs_cli.py` (extend)
|
||||
|
||||
**Interfaces:**
|
||||
```python
|
||||
@dataclass(frozen=True) class ReactionRow: family: str; period: str; period_kind: str; n_items: int; n_total: int; stance_support: int; stance_oppose: int; stance_modify: int; stance_unclear: int; sample_json: str
|
||||
# period = docket id ("CMS-2025-0304") or rule item_key for FR pairs; period_kind ∈ {"docket","fr-pairs"}
|
||||
STANCES = ("support", "oppose", "modify", "unclear")
|
||||
def docket_counts(engine, codes) -> list[tuple[str, int, int]] # (docket, n_items_with_code, n_items_total) from comments chunk metadata (docket, item_key)
|
||||
def stance_sample(engine, codes, docket, *, n) -> list[tuple[str, str]] # (item_key, first chunk text mentioning a code), newest first, capped
|
||||
def classify_stances(sample, classify) -> Counter # closed vocab via llm.classify; prompt text = chunk text[:1500]
|
||||
def fr_pairs(store, codes) -> list[tuple[str, int, int, int]] # (item_key, rule_year, n_comment_paras, n_response_paras) — "Comment:" paragraph followed by "Response:" where either names a code
|
||||
def series(engine, store, family, codes, *, classify=None, stance_sample_n=0) -> list[ReactionRow]
|
||||
```
|
||||
CLI: `stack pfs reaction --family CCM [--stance-sample 50] [--write]` (stance only when `--stance-sample > 0`; `--no-llm` implied otherwise).
|
||||
|
||||
- [ ] Steps: failing tests (fake engine rows; `fr_pairs` on a sqlite `_Store` fixture with Comment:/Response: paragraphs; `classify_stances` with a fake classifier), implement, live run for CCM/APCM/G2211-family? (G2211 has no hand family — run `--code G2211` too: accept `--code` producing a single-code "family"), commit `feat(pfs): public-reaction series per family — docket counts, stance sample, FR Comment/Response pairs (refs #690)`.
|
||||
|
||||
---
|
||||
|
||||
### Task 8: Notebook sections 5–7 on live data + docs + tracker
|
||||
|
||||
**Files:**
|
||||
- Modify: `notebooks/code_families.py` (replace the three stubs), `docs/docs/cli/*.md` (regenerate)
|
||||
- Test: `tests/notebooks/test_code_families_nb.py` (banner test already covers 0–8; add an assertion that the section-5 cell tolerates a missing `LLM_DB_PASSWORD`)
|
||||
|
||||
Section 5 **Anchors in the corpus**: from the replica? No — counts live in pgvector. Guarded: if `LLM_DB_PASSWORD` is set, query `SELECT c.name, count(*) FROM langchain_pg_embedding e JOIN langchain_pg_collection c … WHERE string_to_array(cmetadata->>'families',' ') && ARRAY[:key] GROUP BY 1` and show chunks per collection for the picked family plus the item-tag counts from bib (`Store.list_tags(namespace="family")`); else render the bib tag counts only with a note.
|
||||
Section 6 **Guidance**: `pfs.code_guidance` rows for the family with CFR links (`bib.cfrlink.md_link`) and IOM item titles.
|
||||
Section 7 **Reaction**: `pfs.code_reaction` for the family as an Altair bar chart (x = period, y = n_items, tooltip = stance counts) and the FR-pairs series as a second bar chart by rule year.
|
||||
|
||||
- [ ] Steps: implement the three cells, run headless export (no errors), `uv run pytest tests/notebooks -q`, `uv run python docs/scripts/extract_cli.py` → commit only `docs/docs/cli/{llm-restamp,bib-code-tags,pfs-guidance,pfs-reaction}.md` and changed index pages, commit `feat(notebooks,docs): code_families sections 5–7 — anchors, guidance, reaction; CLI docs (refs #688-#690)`. The controller posts the live numbers on #688–#690.
|
||||
|
||||
---
|
||||
|
||||
## Self-review
|
||||
|
||||
**Spec coverage.** Decision 4 (anchors on items and chunks) → T2–T4; "code_cited_sources can pull comment and IOM chunks" (#688 done-when) → T5; #689 crosswalk with FR-paragraph provenance → T6; #690 reaction incl. pre-2017 FR pairs → T7; the notebook the user asked for → T1 + T8. `stack llm index --force` re-index deliberately replaced by `restamp` (Global Constraints). Not in this slice: IOM section-level page locating inside chapter PDFs (#689 mentions `llm/pages.py`; T6 resolves to the chapter item and leaves `page_src` for the FR anchor only — noted for the tracker), `org:` top-organisations column (existing `org:` tags can be joined in the notebook later).
|
||||
|
||||
**Placeholders.** T5–T7 give interfaces, regexes, SQL shapes and test intents but not full code bodies — the implementers are mid-tier models working from signatures, which the SDD model-selection rule allows ("prose descriptions → mid tier"). Every other step has code.
|
||||
|
||||
**Type consistency.** `anchor_metadata` is the single producer of `codes/families/elements` (T2, T3, T4); `GuidanceRow`/`ReactionRow` are defined in T6/T7 and read by T8; `code_cited_sources(collections=…)` in T5 matches the `rag.py` call site.
|
||||
143
docs/superpowers/plans/2026-09-09-cpt-organizing-principles.md
Normal file
143
docs/superpowers/plans/2026-09-09-cpt-organizing-principles.md
Normal file
@@ -0,0 +1,143 @@
|
||||
# CPT Manual Organizing Principles (P49 slice 2a) Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Bring the AMA CPT manuals (2019–2024 EPUB editions in Zotero) into the bibliography and DuckDB as the CPT's own organizing structure — section hierarchy, code entries with symbols and required elements, parenthetical instructions, and per-code "CPT Changes" years — and make that structure the primary edge in family derivation, so every CPT code gets a family named the way the manual names it.
|
||||
|
||||
**Architecture:** A pure EPUB parser (`pfs/cpt_epub.py`) turns one edition into dataclasses; a loader (`pfs/cpt_load.py`) writes four `pfs.cpt_*` tables through `duckdb_batch` with the bib item key of the edition as provenance; `pfs/families.py` gains a "same CPT subsection" edge and takes family key/name from the CPT heading; `pfs/lineage.py` gains `cpt_changed` events from the per-code reference line; `pfs/extract.py` gains a CPT source for elements (the bulleted required-elements list is literally the element list). A small Zotero→bib import command brings the edition items and files into the bibliography so tables can cite them.
|
||||
|
||||
**Tech Stack:** Python 3.13 stdlib `zipfile` + `html.parser` (no new dependency; `lxml`/`bs4` only if already in `uv.lock` — check), DuckDB via `duckdb_batch`, SQLite `bib.Store`, Zotero SQLite read-only via `conf.connect.zotero()`, typer, pytest.
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-09-09-cpt-canonical-schema-design.md` (the books' structure and the derived schema) and `docs/superpowers/specs/2026-09-09-code-family-longitudinal-design.md` §Decisions 1–3. This slice supplies the organizing principle the family derivation lacked (Ruling: the CPT's own hierarchy outranks stem-token similarity).
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- **The CPT items are corpus documents like any other.** They are imported into bib with their files and indexed into the chat corpus; the `llm:skip` tag is a generic capability and is NOT applied to them (Ruling C4). Tests use a synthetic fixture in the manual's markup style.
|
||||
- Files are read in place from Zotero storage (`data/zotero/data/storage/<attKey>/<file>`, owned by the Zotero container uid — read-only, never chown) or from the bib copy `attach_file` makes.
|
||||
- `pfs/cpt_epub.py` and `pfs/cpt_model.py` are pure (no DuckDB, no bib at import). Writes only inside `duckdb_batch("aco")`, `publish_replica("aco")` after.
|
||||
- Hand families remain a minimum (Ruling 17); CPT subsection edges are additive to the slice-1 edges.
|
||||
- Never add a Co-Authored-By trailer. Commit after every task. Test output pristine.
|
||||
|
||||
---
|
||||
|
||||
## File map
|
||||
|
||||
| File | Responsibility |
|
||||
|---|---|
|
||||
| `src/bib/zotero_import.py` (new) + `src/cli/bib.py` (modify) | `stack bib import-zotero --collection "AMA Coding Publications" --with-files [--tag …]`: Zotero items (+ attachments) → bib `Source` items tagged `source:ama module:coding year:YYYY` |
|
||||
| `src/llm/source.py` (modify) | `iter_corpus_refs` skips items tagged `llm:skip` (generic capability; not used for AMA items) |
|
||||
| `src/pfs/cpt_model.py` (new) | dataclasses `CptSection`, `CptCode`, `CptInstruction`, `CptReference`, `CptEdition` |
|
||||
| `src/pfs/cpt_epub.py` (new) | `parse_epub(path) -> CptEdition` (2021+ template; 2019 template via the same table classes) |
|
||||
| `src/pfs/codetables.py` (modify) | DDL + writers/readers for `pfs.cpt_section`, `pfs.cpt_code`, `pfs.cpt_instruction`, `pfs.cpt_reference` |
|
||||
| `src/pfs/cpt_load.py` (new) + `src/cli/pfs.py` (modify) | `stack pfs cpt-ingest [--edition 2024 …|--all] [--dry-run]` |
|
||||
| `src/pfs/families.py` (modify) | CPT subsection edge; key/name from the CPT heading; add-on edges from "Use X in conjunction with Y" |
|
||||
| `src/pfs/lineage.py` (modify) | `cpt_changed` events from `pfs.cpt_reference` |
|
||||
| `src/pfs/extract.py` (modify) | `extract_code` also reads `pfs.cpt_code` elements (`source="cpt"`) |
|
||||
| `notebooks/code_families.py` (modify) | section 1 shows the CPT heading path + CPT elements; section 4 shows the CPT subsection |
|
||||
| tests: `tests/bib/test_zotero_import.py`, `tests/pfs/test_cpt_epub.py` (+ `tests/pfs/fixtures/cpt_sample.xhtml`), `tests/pfs/test_cpt_load.py`, `tests/pfs/test_families.py` (extend), `tests/pfs/test_lineage.py` (extend), `tests/pfs/test_extract.py` (extend), `tests/cli/test_pfs_cli.py` (extend) | |
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Zotero → bib import of the CPT editions (with files) and corpus skip tag
|
||||
|
||||
**Files:**
|
||||
- Create: `src/bib/zotero_import.py`; Modify: `src/cli/bib.py`, `src/llm/source.py` (`iter_corpus_refs`)
|
||||
- Test: `tests/bib/test_zotero_import.py`, `tests/llm/test_source.py` (extend)
|
||||
|
||||
**Interfaces:**
|
||||
```python
|
||||
@dataclass(frozen=True) class ZoteroBook: key: str; title: str; url: str; year: str; publisher: str; attachments: tuple[Path, ...]
|
||||
def list_collection(zcon, name: str, *, with_files: bool) -> list[ZoteroBook] # sqlite3 connection from conf.connect.zotero(); resolves storage:<file> → storage/<attKey>/<file>
|
||||
def import_books(store, books, *, tags: Sequence[str], copy_files: bool = True, dry_run=False) -> dict # Source items upserted by url (fallback url = f"zotero://select/library/items/{key}"), tags + year:YYYY, attach_file per attachment; {"created","updated","attached"}
|
||||
```
|
||||
`iter_corpus_refs` (and any other corpus walker) skips items tagged `llm:skip`. CLI: `stack bib import-zotero --collection "AMA Coding Publications" --with-files --tag source:ama --tag module:coding [--only "CPT"] [--dry-run]` (`--only` = title substring filter).
|
||||
|
||||
- [ ] Failing tests: `list_collection` against a tiny sqlite built with the Zotero schema subset (`items, itemData, itemDataValues, fields, collections, collectionItems, itemAttachments, itemTypes`) — two books, one with a `storage:` attachment; `import_books` into a tmp `Store` creates Source items with the tags and attaches the file (copy); rerun → `updated/unchanged`, no duplicate attachment; `iter_corpus_refs` skips an item tagged `llm:skip`.
|
||||
- [ ] Implement; run `uv run pytest tests/bib tests/llm/test_source.py -q`; live: `uv run stack bib import-zotero --collection "AMA Coding Publications" --with-files --only CPT --tag source:ama --tag module:coding` → report the bib keys of the six CPT editions (2018, 2019, 2021, 2022, 2023 Changes, 2024).
|
||||
- [ ] After import: `stack llm index --collection corpus` picks the CPT items up on its next run (no special handling).
|
||||
- [ ] Commit: `feat(bib): import Zotero collection items with files into bib; llm:skip tag honoured by the corpus indexer (refs #688)`.
|
||||
|
||||
---
|
||||
|
||||
### Task 2: CPT EPUB parser (pure)
|
||||
|
||||
**Files:**
|
||||
- Create: `src/pfs/cpt_model.py`, `src/pfs/cpt_epub.py`, `tests/pfs/fixtures/cpt_sample.xhtml` (synthetic, in the 2024 template's markup style: two `div.h1`, two `div.h2`, one `table.table1` with a primary code row with three `table-slist` elements, an add-on row with `✚`, three `table-para2` parentheticals (use-with, do-not-report-with, plain), and a `table-RT` "CPT Changes" line)
|
||||
- Test: `tests/pfs/test_cpt_epub.py`
|
||||
|
||||
**Interfaces:**
|
||||
```python
|
||||
@dataclass(frozen=True) class CptSection: sec_id: str; level: int; title: str; path: tuple[str, ...]; code_lo: str; code_hi: str; guideline: str # path = ancestors' titles + own; lo/hi from the TOC "(99490-99437)" when present
|
||||
@dataclass(frozen=True) class CptCode: code: str; sec_id: str; descriptor: str; stem: str; elements: tuple[str, ...]; tail: str; addon: bool; resequenced: bool; new: bool; revised: bool; telemedicine: bool; parent: str
|
||||
# sub-rows ("each additional …") inherit stem/elements from the preceding primary row; parent = that primary code
|
||||
@dataclass(frozen=True) class CptInstruction: code: str; kind: str; text: str; targets: tuple[str, ...] # kind ∈ {"use-with","not-with","not-with-time","see","other"}; targets = codes (ranges expanded ≤ 20)
|
||||
@dataclass(frozen=True) class CptReference: code: str; kind: str; years: tuple[int, ...]; text: str # kind ∈ {"cpt-changes","cpt-assistant"}
|
||||
@dataclass(frozen=True) class CptEdition: year: int; sections: tuple[CptSection, ...]; codes: tuple[CptCode, ...]; instructions: tuple[CptInstruction, ...]; references: tuple[CptReference, ...]
|
||||
def parse_epub(path: Path, *, year: int | None = None) -> CptEdition
|
||||
def parse_xhtml(text: str, *, year: int) -> tuple[list[CptSection], list[CptCode], list[CptInstruction], list[CptReference]] # one body file
|
||||
```
|
||||
Parsing rules (from the 2024/2022/2021 template): headings are `<div class="h1|h2|h3|h4" id="sec_N">TITLE</div>` (the `*` and range are in the TOC files `*_Toc.xhtml`, whose `div.hN-toc` entries carry `TITLE* (LO-HI)` — read them for lo/hi); guideline text = the `div.noindent` paragraphs between a heading and the next table or heading; code rows = `tr` with `td.td-w1[id=code_NNNNN]` (2019: no id — fall back to the `<b>NNNNN</b>` in `div.table-para`); symbols in `span.ama-en` glyphs: `#` resequenced, `✚` add-on, `●` new, `▲` revised, `★` telemedicine (also accept the 2019 glyph set — verify on the 2019 file and document what differs); descriptor = `table-para1` (stem) + `table-slist*` items (elements) + first `table-para2` that does not start with `(` (tail, e.g. the time clause); sub-rows use `table-para1-sub`; parentheticals = `table-para2` starting with `(`; `use-with` = `(Use X in conjunction with …)`, `not-with` = `(Do not report … in the same … with …)`, `not-with-time` = `(Do not report … for service time reported with …)`, `see` = `(For …, see …)` / `(… use NNNNN)`; references = `table-RT*` lines (`CPT Changes: An Insider's View 2015, 2021, 2022` → years; `CPT Assistant …` kept as text). HTML entities and `<span class="ssp"/>` bullets are stripped; whitespace normalised; the `<i>` and `<b>` tags removed from text.
|
||||
|
||||
- [ ] Failing tests on the synthetic fixture: two sections with correct levels/paths and lo/hi from a synthetic TOC snippet; primary code has 3 elements and the tail; the add-on row has `addon=True`, `parent=<primary>`, inherited elements; `use-with` targets `(primary,)`; `not-with` targets expanded from a range `90951-90970` (20 codes) plus listed codes; `cpt-changes` years `(2015, 2021, 2022)`; entities/whitespace cleaned. Plus an integration test `@pytest.mark.skipif(not Path(EPUB_2024).exists())` that `parse_epub` on the real 2024 file yields ≥ 8,000 codes, the section titled "Chronic Care Management Services" contains 99490/99439/99491/99437, and 99439 is an add-on whose `use-with` target is 99490 — asserting only structure, never descriptor text.
|
||||
- [ ] Implement with `html.parser.HTMLParser` (stdlib) building a flat event list, then a second pass grouping rows; keep `parse_xhtml` independently testable.
|
||||
- [ ] Commit: `feat(pfs): CPT EPUB parser — sections, code entries with symbols and elements, parenthetical instructions, CPT Changes references (refs #687)`.
|
||||
|
||||
---
|
||||
|
||||
### Task 3: `pfs.cpt_*` tables and `stack pfs cpt-ingest`
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/pfs/codetables.py` (four DDLs, `write_cpt_edition(con, edition, item_key)`, `read_cpt_codes(con, year)`, `read_cpt_sections(con, year)`), Create: `src/pfs/cpt_load.py`, Modify: `src/cli/pfs.py`
|
||||
- Test: `tests/pfs/test_cpt_load.py`, `tests/cli/test_pfs_cli.py` (extend)
|
||||
|
||||
Tables (all with `edition_year INTEGER, item_key VARCHAR` first): `pfs.cpt_section(sec_id, level, title, path, code_lo, code_hi, guideline)`, `pfs.cpt_code(code, sec_id, descriptor, stem, elements, tail, addon, resequenced, new, revised, telemedicine, parent)` (`elements` space-joined? No — `elements` is a `VARCHAR[]` list; DuckDB supports it), `pfs.cpt_instruction(code, kind, text, targets)` (`targets VARCHAR[]`), `pfs.cpt_reference(code, kind, years INTEGER[], text)`, plus the appendix-derived tables from the schema spec `docs/superpowers/specs/2026-09-09-cpt-canonical-schema-design.md` §3: `pfs.cpt_change(code, kind, old_text, new_text, section)` from Appendix B, `pfs.cpt_crosswalk(current_code, former_code, year_deleted, citations)` from Appendix M, `pfs.cpt_list(appendix, code)` from Appendices D/E/F/G/K/N/P/T, `pfs.cpt_modifier(modifier, title, text)` from Appendix A. Delete-then-insert per `edition_year`.
|
||||
|
||||
`cpt_load.ingest(store, con, *, years: Sequence[int] | None) -> dict`: finds bib items tagged `source:ama` whose title matches `CPT (Professional )?(\d{4})` with an EPUB attachment, parses each, writes, returns rows per table per year. CLI `stack pfs cpt-ingest [--edition 2024]… [--all] [--dry-run]`; `--dry-run` parses and prints counts only.
|
||||
|
||||
- [ ] Failing tests: DDL idempotent; write/read round trip for a two-code synthetic edition; `ingest` with a fake store/attachment path pointing at a tiny EPUB built in the test (zip the fixture xhtml + a minimal container) writes rows; CLI test with `_batch/_read/_store/_publish` monkeypatched as in slice 1.
|
||||
- [ ] Implement; live: `stack pfs cpt-ingest --all` (2019, 2021, 2022, 2024 EPUBs; 2023 Changes is a different book — skip it here, note for later) → report rows per table per year, and one sanity query per year: the section holding 99490 and the count of add-on codes.
|
||||
- [ ] Commit: `feat(pfs): pfs.cpt_section/cpt_code/cpt_instruction/cpt_reference + stack pfs cpt-ingest (refs #687)`.
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Families from the CPT hierarchy
|
||||
|
||||
**Files:** Modify `src/pfs/families.py`, `src/cli/pfs.py` (`families` command feeds CPT data), Test `tests/pfs/test_families.py`, `tests/cli/test_pfs_cli.py`
|
||||
|
||||
**Ruling (binding):** for CPT codes the family is the **lowest CPT heading that groups ≥ 2 codes** in the newest ingested edition (e.g. "Chronic Care Management Services" for 99490/99439/99491/99437; "Complex Chronic Care Management Services" for 99487/99489). Its key is a slug of the heading title (`CHRONIC-CARE-MANAGEMENT-SERVICES`), its name the heading title, and `since` = the earliest edition year the code appears in `pfs.cpt_code`. Slice-1 edges still apply on top (so G2058 still joins via its single-target replacement, and `use-with` instructions from `pfs.cpt_instruction` count as add-on edges). Hand families keep their keys and names: when a CPT heading's group intersects a hand family, the hand key/name wins for that component (Ruling 17), and the CPT heading title is stored as the family's `note`. Codes not in any CPT edition (HCPCS G-codes) keep the slice-1 derivation.
|
||||
|
||||
**Interfaces:**
|
||||
```python
|
||||
def cpt_groups(cpt_codes: Sequence[CptCodeRow], cpt_sections: Sequence[CptSectionRow]) -> dict[str, tuple[str, str, tuple[str, ...]]] # sec_id -> (key, title, codes) for the lowest heading with ≥2 codes
|
||||
def derive_families(elements, events, descriptions, *, cpt_codes=(), cpt_sections=(), cpt_instructions=()) -> list[FamilyRow] # new keyword inputs; existing calls unchanged
|
||||
```
|
||||
`FamilyRow` gains `note: str = ""` (append to the DDL and dataclass — last field, default, so slice-1 readers keep working; `read_families` reads it).
|
||||
|
||||
- [ ] Failing tests: a synthetic edition with "Care Management Services" (h1) → "Chronic Care Management Services" (h2, 99490 99439 99491 99437) and "Complex …" (h2, 99487 99489): `cpt_groups` returns the two h2 groups, not the h1; `derive_families` with those inputs and the slice-1 CCM fixture yields the hand key `CCM` for the merged component with `note == "Chronic Care Management Services"` (99487/99489 join via slice-1 stem+activity as before); a synthetic "Remote Physiologic Monitoring Treatment Management Services" heading with 99457/99458 and no elements yields key `REMOTE-PHYSIOLOGIC-MONITORING-TREATMENT-MANAGEMENT-SERVICES` with `since` = edition year.
|
||||
- [ ] Implement; live: `stack pfs families --write` → report: total families, families with ≥ 2 codes (expect thousands now), the five hand families' membership and notes, and five sample CPT-named families.
|
||||
- [ ] Commit: `feat(pfs): families from the CPT manual hierarchy — lowest multi-code heading names the family; use-with edges (refs #687)`.
|
||||
|
||||
---
|
||||
|
||||
### Task 5: CPT Changes years as lineage events and CPT elements in extraction
|
||||
|
||||
**Files:** Modify `src/pfs/lineage.py` (`cpt_events(con, code)` from `pfs.cpt_reference` kind `cpt-changes` → `EventRow(kind="cpt_changed", year=Y, source="cpt", item_key=<edition bib key>, anchored=True)`; `lineage()` merges them and an RVU event is also anchored by a `cpt_changed` within ±1 year), `src/pfs/extract.py` (`extract_code` adds `ElementRow`s from the newest `pfs.cpt_code` row: `parse_descriptor(stem + tail)` plus each bulleted element parsed and, when the regex finds nothing, a `ReviewRow` with `source="cpt"`; FR rows still win on duplicates), `src/cli/pfs.py` (lineage prints the new kind).
|
||||
- [ ] Failing tests for both; implement; live: `stack pfs lineage --code 99490 --write` shows `cpt_changed` 2015/2021/2022 anchored to the 2024 edition item; `stack pfs elements --family CCM --no-llm` review queue does not grow.
|
||||
- [ ] Commit: `feat(pfs): CPT Changes years as lineage events; CPT required-elements lists as an extraction source (refs #685 #686)`.
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Notebook — the CPT manual's own organizing principles
|
||||
|
||||
**Files:** Modify `notebooks/code_families.py` (section 1 gains "Where the manual files this code": the CPT heading path for the picked code from `pfs.cpt_section`/`pfs.cpt_code` and its instructions; section 4 shows the family's CPT heading `note` and the sibling headings under the same parent; a new short callout in section 0 that CPT's hierarchy is the first organizing principle and the FR is the second), `tests/notebooks/test_code_families_nb.py` (banner count unchanged; cells still degrade).
|
||||
- [ ] Implement; headless export clean; commit `feat(notebooks): code_families — CPT hierarchy, instructions and family headings (refs #687)`.
|
||||
|
||||
---
|
||||
|
||||
## Self-review
|
||||
|
||||
**Coverage.** User request "bring in the CPT manuals from Zotero so we have a better sense of organizing principles" → T1 (into bib), T2–T3 (structure into tables), T4 (organizing principle drives families), T5 (lineage/elements benefit), T6 (explainer). Copyright handling is a global constraint with a concrete mechanism (`llm:skip`, no text in docs/tracker/tests).
|
||||
|
||||
**Placeholders.** T2's parsing rules are exact selectors observed on the 2024/2022/2021 files; T1/T3–T6 give signatures, rules and test intents (mid-tier implementers).
|
||||
|
||||
**Types.** `FamilyRow.note` is appended last with a default so slice-1 readers/writers keep working; `EventRow.kind="cpt_changed"` and `source="cpt"` reuse the existing dataclass; `ElementRow.source="cpt"` reuses the existing column.
|
||||
@@ -34,7 +34,7 @@ The goal of P49 is that the chat, and the notebooks, can build a **longitudinal
|
||||
## Decisions
|
||||
|
||||
1. **Elements are a closed, typed vocabulary**, not free text. Element *types* are fixed in code; element *values* are a curated list that grows by review (P35 pattern: local model proposes from a closed list, humans accept). Types: `actor` (clinical staff directed by / personally by physician or QHP; RHC/FQHC), `time` (threshold minutes, "each additional"), `period` (calendar month, 30 days, per visit, 14/7 days post-discharge), `population` (≥2 chronic conditions ≥12 months; QMB; high risk), `activity` (consent; care plan; 24/7 access; transitions; medication reconciliation; …), `modality` (face-to-face; interactive telecommunications; audio-only; asynchronous), `relation` (add-on to; do-not-report-with; replaces; defined-by-reference-to; crosswalk-valued-to), `setting` (facility/non-facility; place of service), `billing` (one practitioner per month; initiating visit).
|
||||
2. **Lineage is a table of dated events with anchors**, derived from two independent sources that must agree or be flagged: (a) `pfs.rvu`/`rvu_proposed` (first year, last year, status transitions, descriptor changes, RVU changes beyond a threshold) and (b) FR paragraphs whose text carries the code and an event verb (create/adopt/replace/delete/crosswalk/bundle/telehealth-list). Every event row stores `item_key, p_id, page` so the chat cites a paragraph, never a table.
|
||||
2. **Lineage is a table of dated events with anchors**, derived from two independent sources that must agree or be flagged: (a) `pfs.rvu`/`rvu_proposed` (first year, last year, status transitions, descriptor changes, RVU changes beyond a threshold) and (b) FR paragraphs whose text carries the code and an event verb (create/adopt/replace/delete/crosswalk/bundle/telehealth-list). Every event row stores `item_key, p_id, page` so the chat cites a paragraph, never a table. Lineage now has a third source, `cpt` — the AMA CPT Changes book's own "Effective January 1, YYYY" statement per code, ingested as `pfs.cpt_reference`/`pfs.cpt_code` change years, which are effective years rather than paragraph-anchored FR events; a `cpt_changed` year anchors an RVU event (an appearance/status change with no FR paragraph nearby) the same way an FR event does.
|
||||
3. **Families are derived, not hand-listed.** A family = codes sharing a stem and differing in `time`/`actor`/`population` elements, plus CMS's own "code family" language in the FR ("The Chronic Care Management code family will be resurveyed", `YBM4IZUS` ¶1578) and `relation` elements (add-on/replaces). The five hand families become the regression fixture.
|
||||
4. **Anchors live in two places, one per consumer.** Items get `code:<hcpcs>` and `family:<key>` tags in bib/Zotero (browsable, syncable); chunks get `codes`, `families`, and `elements` metadata in pgvector (retrievable). `comments` and `corpus` are re-indexed once with `--force` to stamp them, as P48 did for `rules`.
|
||||
5. **Longitudinal retrieval is era-balanced, not recency-blended.** A "history of X" question retrieves per era (rule year) with a cap per era, so CY2015 and CY2027 both appear; the P34 recency blend stays for ordinary questions.
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
# The canonical coding books as schema — CPT, CPT Changes, HCPCS, ICD-10-CM
|
||||
|
||||
Source review 2026-09-09 of the AMA books in the Zotero collection "AMA Coding Publications" (files on disk: CPT 2018 PDF; CPT 2019 PDF+EPUB; CPT 2021, 2022, 2024 Professional EPUB (+2024 PDF); CPT Changes 2023 EPUB; E/M Office Visit Compendium 2021 EPUB; HCPCS 2019 EPUB+PDF; ICD-10-CM 2016–2025 and ICD-10-PCS 2019/2022 EPUB; Coding with Modifiers 4th ed.; Netter's Atlas for CPT Coding 2015). The 2021/2022/2024 CPT EPUBs share one publisher template; 2019 uses the same table classes without `code_` ids.
|
||||
|
||||
Purpose: read the books' own organizing structure and turn it into the schema that P49's element model, lineage and family derivation should be built on. The Federal Register tells us what Medicare *pays for*; these books tell us what a code *is* and how the code set is *organized* — and that organization is the first principle for "code families".
|
||||
|
||||
## 1. What the CPT codebook says about its own structure
|
||||
|
||||
**Sections and number ranges** (Introduction › "Section Numbers and Their Sequences"): Evaluation and Management 99202–99499; Anesthesiology 00100–01999, 99100–99140; Surgery 10004–69990; Radiology 70010–79999; Pathology and Laboratory 80047–89398, 0001U–0419U; Medicine 90281–99199, 99500–99607, 0001A–0174A; then Category II (performance measurement, `NNNNF`) and Category III (emerging technology, `NNNNT`). Each section opens with **Guidelines** (definitions and reporting rules that apply to every code in the section), and the book states that placement in a section "may reflect historical or other considerations" — i.e. the hierarchy is a curated taxonomy, not a strict classification.
|
||||
|
||||
**Hierarchy below the section** (the book's own words in the E/M guidelines): a section is "divided into broad **categories** … most of the categories are further divided into two or more **subcategories** … further classified into **levels** identified by specific codes." In the EPUB this is `div.h1` (subsection, e.g. *Care Management Services*), `div.h2` (category, e.g. *Chronic Care Management Services*), `div.h3`/`h4` (subcategory), each with a `sec_N` id, and the chapter TOC prints every heading with its code span, e.g. `Care Management Services* (99490-99427)` → `Chronic Care Management Services* (99490-99437)`, `Complex Chronic Care Management Services* (99487-99489)`, `Principal Care Management Services* (99424-99427)`. The asterisk marks headings that carry their own guideline text. Spans are print-order, not numeric (resequenced codes).
|
||||
|
||||
**Code entry format** ("Format of the Terminology"): a code is a stand-alone description *unless* it is indented, in which case it inherits the parent's text before the semicolon — the book's example `25100 Arthrotomy, wrist joint; with biopsy` / `25105 with synovectomy`. Care-management codes use the "with the following required elements:" form: a stem sentence, a bulleted list of elements, and a closing time clause. Add-on entries ("each additional 20 minutes …") inherit the primary's stem and elements. In the EPUB: `td.td-w1[id=code_NNNNN]` → `div.table-para1` (stem), `div.table-slist*` (elements), `div.table-para2` (tail), `div.table-para1-sub` (indented child).
|
||||
|
||||
**Symbols** (Legend): `●` new code, `▲` revised code, `✚` add-on, `⦸` modifier-51 exempt, `▶ ◀` new or revised text, `★` telemedicine (audio-video), audio-only glyph, `⚡` FDA approval pending, `#` resequenced, duplicate-PLA glyph, `⇅` Category I PLA, `➲` citations (CPT Changes / CPT Assistant). In the EPUB every glyph is a `span.ama-en`.
|
||||
|
||||
**Parenthetical instructions** (Introduction › "Instructions"): notes with selected codes that "indicate that a code should not be reported with another code or codes"; explicitly "not all inclusive". Observed kinds, each a `div.table-para2` beginning with `(`:
|
||||
- `(Use X in conjunction with Y)` — add-on binding;
|
||||
- `(Do not report X, Y in the same calendar month with …)` — exclusivity within a period;
|
||||
- `(Do not report X for service time reported with …)` — time double-counting exclusion;
|
||||
- `(Do not report X more than twice per calendar month)` / `(… of less than 20 minutes … are not reported separately)` — frequency/threshold;
|
||||
- `(… may be reported using 99487, 99489)` / `(For …, see …)` / `(… use NNNNN)` — cross-references;
|
||||
- `(NNNNN has been deleted. To report, see …)` — deletion pointers.
|
||||
ICD-10-CM has the same construct with different names (Excludes1 = mutually exclusive, Excludes2 = not included here, Code first / Use additional code = sequencing) — a useful cross-check that "instruction" is a first-class object in every canonical book.
|
||||
|
||||
**Per-code references**: `➲ CPT Changes: An Insider's View 2015, 2021, 2022` — the years the code was added or revised — and `➲ CPT Assistant Oct 14:3, Feb 15:3 …` — newsletter guidance citations. Both are `div.table-RT*` lines.
|
||||
|
||||
**Appendices that are authoritative lists** (2024 edition): A modifiers; **B Summary of Additions, Deletions, and Revisions** (the year's changes, with revised descriptors shown as strikethrough/underline diffs — descriptor-level change tracking); C clinical examples; **D add-on codes**; E modifier-51 exempt; F modifier-63 exempt; G moderate sedation included; H alphabetical clinical topics; I genetic testing modifiers; J electrodiagnostic nerves; K FDA-pending; L vascular families (a literal "family" taxonomy); **M renumbered codes citations crosswalk** (current ↔ former code, year deleted, 2007–2009); **N resequenced codes**; O MAAA/PLA; **P audio-video telemedicine codes**; Q COVID vaccines; R digital medicine services taxonomy; S AI taxonomy; **T audio-only telemedicine codes**. Appendices P/T carry the CPT Editorial Panel's own telemedicine criteria — the counterpart of the Medicare telehealth Steps in the FR.
|
||||
|
||||
**Annual cycle**: one edition per calendar year; codes effective January 1. *CPT Changes: An Insider's View* is the companion for each year: per section a tabular **Summary of Additions, Deletions, and Revisions**, then the new/revised entries reprinted with a shaded **Rationale** box and **Clinical Examples** (typical patient + description of procedure). This is the AMA's own "why" for each change — the CPT-side twin of the FR's Comment/Response.
|
||||
|
||||
## 2. What the other books add
|
||||
|
||||
- **HCPCS Level II** (CMS, annual + quarterly): letter-prefixed sections (A–V) by supply/service type; each entry = code, descriptor, coverage/payment indicators, cross-references to Pub 100 (IOM) and NCCI policy (the 2019 book reprints NCCI chapter 1 as an appendix); modifiers; table of drugs. G-codes are CMS's parallel to CPT for Medicare programmatic needs — the FR repeatedly says "we prefer CPT unless Medicare has a programmatic need" (CY2015 final ¶1249), and lineage often runs G-code → CPT (G2058 → 99439, G2064/5 → 99424/6).
|
||||
- **E/M Office Visit Compendium 2021**: the 2021 E/M redesign explained — MDM vs time selection, prolonged services, and a tabular review of guideline changes; the elements of E/M levels are a distinct element vocabulary (history, exam, MDM components, time).
|
||||
- **ICD-10-CM**: chapter → block → category → subcategory → code (7th-character extensions), official guidelines, instruction notes as above. Not a PFS object but the same schema shape; relevant later for diagnosis-conditioned coverage.
|
||||
- **Coding with Modifiers**, **Netter's Atlas**: modifier semantics (Appendix A's narrative) and anatomy → code mapping; reference only.
|
||||
|
||||
## 3. Schema derived from the books
|
||||
|
||||
All tables in DuckDB schema `pfs`, keyed by `edition_year` with `item_key` = the edition's bib item (provenance). Text columns hold the book's text verbatim for the user's own analysis; the CPT items are indexed into the chat corpus like every other item in the library.
|
||||
|
||||
| table | grain | columns (beyond `edition_year, item_key`) | source |
|
||||
|---|---|---|---|
|
||||
| `cpt_section` | one heading | `sec_id, level (1–4), title, path (ancestor titles), parent_sec_id, code_lo, code_hi, has_guidelines, guideline_text` | body `div.hN`, chapter TOC spans |
|
||||
| `cpt_code` | one code entry | `code, sec_id, category (I/II/III), stem, elements VARCHAR[], tail, descriptor_full (stem + elements + tail, child text expanded per the semicolon rule), parent_code (for indented/add-on rows), addon, resequenced, new, revised, mod51_exempt, telemedicine_av, audio_only, fda_pending, pla` | code rows + Legend glyphs |
|
||||
| `cpt_instruction` | one parenthetical | `owner (code or sec_id), kind (use-with / not-with-period / not-with-time / frequency / cross-ref / deleted-pointer / other), text, targets VARCHAR[] (ranges expanded)` | `div.table-para2` beginning `(` and guideline-level notes |
|
||||
| `cpt_reference` | one citation line | `code, kind (cpt-changes / cpt-assistant), years INTEGER[], text` | `div.table-RT*` |
|
||||
| `cpt_change` | one change in one edition | `code, kind (added / revised / deleted), old_text, new_text, section` | Appendix B (strike/underline diff), cross-checked with CPT Changes summary tables |
|
||||
| `cpt_rationale` | one rationale or clinical example | `code (or sec_id), kind (rationale / clinical-example), text` | *CPT Changes* rationale boxes and clinical examples |
|
||||
| `cpt_crosswalk` | one renumbering | `current_code, former_code, year_deleted, citations` | Appendix M |
|
||||
| `cpt_list` | one code in one appendix list | `appendix (D/E/F/G/K/N/P/T…), code` | Appendices D–T |
|
||||
| `cpt_modifier` | one modifier | `modifier, title, text` | Appendix A |
|
||||
|
||||
HCPCS Level II gets the same shape later (`hcpcs_section`, `hcpcs_code`, `hcpcs_note`) from the CMS quarterly files already in the pipeline plus the book's coverage notes.
|
||||
|
||||
## 4. How the schema feeds P49
|
||||
|
||||
- **Elements**: `cpt_code.elements` *are* the required-elements list; `stem`/`tail` carry actor, time and period phrases. `pfs.extract` reads them as a source (`source="cpt"`) before the FR paragraph runs, so element extraction covers every CPT code, not only the ones the FR happens to reprint. The element vocabulary grows from the book's phrasing (e.g. E/M levels, "medical decision making", "total time on the date of the encounter").
|
||||
- **Families**: the organizing principle is the CPT hierarchy. Family = the lowest heading that groups ≥ 2 codes (category or subcategory), named by the heading. Hand families keep their keys. Additional intra-family edges: `✚` + `use-with` (add-on → primary), the semicolon-inheritance parent (`parent_code`), and Appendix L vascular families. Cross-family edges are *not* created from `not-with` instructions (they express exclusivity across families, e.g. CCM vs PCM).
|
||||
- **Lineage**: `cpt_change` gives added/revised/deleted per edition year with the descriptor diff; `cpt_reference` gives the CPT Changes years per code (the AMA's own revision history); `cpt_crosswalk` gives renumberings; `cpt_rationale` gives the AMA's reason. These are `source="cpt"` events beside the FR (`fr`) and RVU-file (`rvu`) events, and they anchor RVU events the same way FR events do.
|
||||
- **Telehealth**: Appendix P/T membership per edition + the CPT criteria text sit beside the Medicare telehealth list and the FR Steps 1–3; the chat can show both bodies' reasoning for one code.
|
||||
- **Chat**: the CPT items are corpus documents (chunked, embedded, code-anchored) so a question about a code retrieves the manual's guidelines and instructions, the FR's policy, the RVU valuation, and the comments' reaction together.
|
||||
|
||||
## 5. Gaps and decisions
|
||||
|
||||
- Editions on disk cover 2019, 2021, 2022, 2024 (EPUB) and 2018 (PDF only). 2020, 2023, 2025, 2026 codebooks are in Zotero as records without files; `since` years will have gaps until they land. Appendix B of each present edition still gives the previous year's changes (2024's Appendix B = changes effective 2024).
|
||||
- The 2019 EPUB lacks `code_` ids; the parser keys on the `<b>NNNNN</b>` in `div.table-para` for that edition.
|
||||
- PDFs are not parsed in this slice.
|
||||
- `llm:skip` stays as a generic capability (an item the corpus should not embed) but is **not** applied to the AMA items; the seven tags added by the first import are removed.
|
||||
703
notebooks/code_families.py
Normal file
703
notebooks/code_families.py
Normal file
@@ -0,0 +1,703 @@
|
||||
import marimo
|
||||
|
||||
__generated_with = "0.23.13"
|
||||
app = marimo.App(width="medium")
|
||||
|
||||
|
||||
@app.cell(hide_code=True)
|
||||
def _():
|
||||
import marimo as mo
|
||||
|
||||
return (mo,)
|
||||
|
||||
|
||||
@app.cell(hide_code=True)
|
||||
def _(mo):
|
||||
mo.md(
|
||||
"""
|
||||
# Code families as first-class objects
|
||||
|
||||
A physician fee schedule code is not a number — it is a **bundle of logical elements**:
|
||||
who furnishes the service, for how long, per what period, to which patients, doing which
|
||||
activities, by which modality. This notebook walks through how the stack turns that idea
|
||||
into tables you can query and cite: elements → extraction → lineage → families → anchors
|
||||
→ guidance → public reaction. Every number on this page is read live from the replica and
|
||||
the bibliography; every claim links to the Federal Register paragraph it came from.
|
||||
"""
|
||||
)
|
||||
return
|
||||
|
||||
|
||||
@app.cell(hide_code=True)
|
||||
def _():
|
||||
# ── Setup ──
|
||||
import altair as alt
|
||||
import polars as pl
|
||||
|
||||
from conf import connect, path
|
||||
from conf.display import plain_years
|
||||
|
||||
connect.theme()
|
||||
NOTES = {}
|
||||
|
||||
# data/replica/<name>.ro.duckdb — same layout connect._replica_path resolves,
|
||||
# computed here (not imported) because it's a private helper.
|
||||
_primary = path("db.aco")
|
||||
REPLICA_PATH = _primary.parent / "replica" / f"{_primary.stem}.ro.duckdb"
|
||||
|
||||
def _open_replica():
|
||||
try:
|
||||
return connect.duckdb("aco", read_only=True)
|
||||
except Exception as e: # noqa: BLE001 — degrade, never crash the page
|
||||
NOTES["replica"] = f"replica unavailable: {e}"
|
||||
return None
|
||||
|
||||
def _open_bib():
|
||||
try:
|
||||
return connect.bib()
|
||||
except Exception as e: # noqa: BLE001
|
||||
NOTES["bib"] = f"bibliography unavailable: {e}"
|
||||
return None
|
||||
|
||||
con = _open_replica()
|
||||
store = _open_bib()
|
||||
|
||||
def q(sql, params=()):
|
||||
if con is None:
|
||||
return pl.DataFrame()
|
||||
try:
|
||||
return con.execute(sql, list(params)).pl()
|
||||
except Exception as e: # noqa: BLE001 — a missing table is a "not built yet"
|
||||
NOTES[sql[:40]] = str(e)
|
||||
return pl.DataFrame()
|
||||
|
||||
def fr_md(item_key, p_id):
|
||||
if store is None:
|
||||
return f"{item_key} ¶{p_id}"
|
||||
try:
|
||||
from bib.frlink import md_link
|
||||
|
||||
return md_link(
|
||||
f"p-{p_id}", store=store, item_key=item_key, text=f"{item_key} ¶{p_id}"
|
||||
)
|
||||
except Exception: # noqa: BLE001
|
||||
return f"{item_key} ¶{p_id}"
|
||||
|
||||
def not_built(cmd):
|
||||
return f"_Not built yet — run `{cmd}` and republish the replica._"
|
||||
|
||||
return NOTES, REPLICA_PATH, alt, con, fr_md, not_built, pl, plain_years, q, store
|
||||
|
||||
|
||||
@app.cell(hide_code=True)
|
||||
def _(fr_md, mo, store):
|
||||
# ── 0. Why a code is a bundle of elements ──
|
||||
_steps = []
|
||||
if store is not None:
|
||||
try:
|
||||
_con = store._con() # noqa: SLF001
|
||||
for _p in (394, 396, 398):
|
||||
_row = _con.execute(
|
||||
"SELECT text FROM fr_anchors WHERE item_key = ? AND p_id = ?",
|
||||
("2KVJ2HKX", _p),
|
||||
).fetchone()
|
||||
if _row:
|
||||
_steps.append(f"> {_row[0][:400]}… — {fr_md('2KVJ2HKX', _p)}")
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
mo.md(
|
||||
"## 0. Why a code is a bundle of elements\n\n"
|
||||
"CMS says so itself. When it decides whether a service can be furnished by telehealth it "
|
||||
"walks three steps, and the third is literally *review the elements of the service as "
|
||||
"described by the HCPCS code* (CY2026 proposed rule, 90 FR 32389):\n\n"
|
||||
+ (
|
||||
"\n\n".join(_steps)
|
||||
if _steps
|
||||
else "_(bibliography unavailable — quotes omitted)_"
|
||||
)
|
||||
+ "\n\nThat one sentence is the whole design: a code is not an opaque five-character "
|
||||
"string CMS prices as a unit. It is a bundle of *who* furnishes it, *how long* it takes, "
|
||||
"*how often* it can be billed, *which patients* qualify, *which activities* it covers, "
|
||||
"and *by what modality* — and CMS itself reasons about codes at that level of detail, one "
|
||||
"element at a time. Everything below builds machine-readable tables out of that same "
|
||||
"bundle: parse the descriptor into typed elements, extract them with their FR anchor, "
|
||||
"trace how a code's identity changes over the years (lineage), and group codes that share "
|
||||
"one clinical program into a family."
|
||||
)
|
||||
return
|
||||
|
||||
|
||||
@app.cell(hide_code=True)
|
||||
def _(mo, not_built, q):
|
||||
# Two organizing principles behind every table on this page
|
||||
_editions = q(
|
||||
"SELECT DISTINCT edition_year FROM pfs.cpt_code ORDER BY edition_year"
|
||||
)
|
||||
if _editions.is_empty():
|
||||
_view = mo.md(not_built("stack pfs cpt-ingest --all"))
|
||||
else:
|
||||
_years = ", ".join(str(y) for y in _editions["edition_year"].to_list())
|
||||
_view = mo.md(
|
||||
"**Two organizing principles, in that order.** The CPT codebook is the **first**: "
|
||||
"the American Medical Association's own hierarchy — Section → subsection → category "
|
||||
"→ subcategory → codes, with guideline text at every section, symbols marking new, "
|
||||
"revised, add-on and telemedicine codes, and parenthetical instructions "
|
||||
"cross-referencing related codes. That hierarchy is what groups codes into clinical "
|
||||
"families below, and it is what a descriptor's elements are drawn from. The Federal "
|
||||
"Register is the **second**: it is where CMS decides whether Medicare pays for a code "
|
||||
"the AMA has already defined, and how much — the FR reprices codes, it does not "
|
||||
f"reorganize the code set. CPT editions on this replica: {_years}."
|
||||
)
|
||||
_view
|
||||
return
|
||||
|
||||
|
||||
@app.cell(hide_code=True)
|
||||
def _(mo):
|
||||
# ── 1. Reading a descriptor ──
|
||||
from pfs.families import HAND_FAMILIES
|
||||
|
||||
_codes = sorted({c for f in HAND_FAMILIES.values() for c in f.codes})
|
||||
code_picker = mo.ui.dropdown(options=_codes, value="99490", label="Code")
|
||||
mo.vstack(
|
||||
[
|
||||
mo.md(
|
||||
"## 1. Reading a descriptor\n\n"
|
||||
"The Federal Register prints a code's descriptor as a *stem* paragraph — the "
|
||||
"sentence that opens with the code number and a parenthesis — followed by one "
|
||||
"paragraph per required element, each ending in a semicolon or closing "
|
||||
"parenthesis. `descriptor_runs` finds every place a rule prints that pattern for "
|
||||
"a code and pairs the stem with the element paragraphs that immediately follow "
|
||||
"it; `parse_descriptor` then reads whatever a regex can read out of that text — "
|
||||
"minutes, billing periods, populations, activities, modalities — against a "
|
||||
"**closed vocabulary** that only grows by human review. Pick a code from any of "
|
||||
"the five hand-registered families below."
|
||||
),
|
||||
code_picker,
|
||||
]
|
||||
)
|
||||
return (code_picker,)
|
||||
|
||||
|
||||
@app.cell(hide_code=True)
|
||||
def _(code_picker, fr_md, mo, pl, store):
|
||||
from pfs.descriptors import descriptor_runs
|
||||
from pfs.elements import VOCAB, parse_descriptor
|
||||
|
||||
code = code_picker.value
|
||||
runs = descriptor_runs(store, code) if store is not None else []
|
||||
if not runs:
|
||||
_view = mo.md(
|
||||
f"_No Federal Register descriptor run found for {code} (bibliography unavailable "
|
||||
"or code never printed as a stem)._"
|
||||
)
|
||||
elements = ()
|
||||
else:
|
||||
# The original codification, preferring the earliest run that carries its own
|
||||
# element paragraphs (a later rule often just cites the code inline mid-sentence,
|
||||
# with no paragraph break) — for 99490 this is the CY2015 final rule, stem ¶1244.
|
||||
run = next((r for r in runs if r.elements), runs[0])
|
||||
_paras = pl.DataFrame(
|
||||
{
|
||||
"p_id": [run.stem.p_id, *[p.p_id for p in run.elements]],
|
||||
"role": ["stem", *["element"] * len(run.elements)],
|
||||
"text": [run.stem.text[:300], *[p.text[:300] for p in run.elements]],
|
||||
}
|
||||
)
|
||||
elements = parse_descriptor(run.text)
|
||||
_els = pl.DataFrame(
|
||||
{
|
||||
"type": [e.type.value for e in elements],
|
||||
"value": [e.value for e in elements],
|
||||
"detail": [e.detail for e in elements],
|
||||
}
|
||||
)
|
||||
_vocab = pl.DataFrame(
|
||||
{
|
||||
"type": [t.value for t in VOCAB for _ in VOCAB[t]],
|
||||
"value": [v for t in VOCAB for v in VOCAB[t]],
|
||||
}
|
||||
)
|
||||
_view = mo.vstack(
|
||||
[
|
||||
mo.md(
|
||||
f"**{code}** as printed in {run.item_key} (CY{run.rule_year}), stem "
|
||||
f"{fr_md(run.item_key, run.stem.p_id)}: the stem paragraph opens the "
|
||||
"descriptor and each following paragraph is one element."
|
||||
),
|
||||
mo.ui.table(_paras, label="Descriptor paragraphs"),
|
||||
mo.md(
|
||||
"The deterministic parser reads what a regex can read — minutes, periods, "
|
||||
"code references, and the recurring phrases:"
|
||||
),
|
||||
mo.ui.table(_els, label="Typed elements"),
|
||||
mo.accordion(
|
||||
{
|
||||
"The closed vocabulary (values grow only by review)": mo.ui.table(
|
||||
_vocab
|
||||
)
|
||||
}
|
||||
),
|
||||
]
|
||||
)
|
||||
_view
|
||||
return code, elements
|
||||
|
||||
|
||||
@app.cell(hide_code=True)
|
||||
def _(code, mo, not_built, pl, q):
|
||||
# Where the manual files this code
|
||||
_ed = q("SELECT max(edition_year) y FROM pfs.cpt_code")
|
||||
if _ed.is_empty():
|
||||
_view = mo.md(
|
||||
"**Where the manual files this code.**\n\n"
|
||||
+ not_built("stack pfs cpt-ingest --all")
|
||||
)
|
||||
else:
|
||||
_year = _ed.item(0, "y")
|
||||
_cpt = q(
|
||||
"SELECT c.stem, c.elements, c.tail, c.addon, c.resequenced, c.new, c.revised, "
|
||||
"c.telemedicine, c.mod51_exempt, c.audio_only, c.fda_pending, c.pla, s.path_key "
|
||||
"FROM pfs.cpt_code c JOIN pfs.cpt_section s USING (edition_year, item_key, sec_id) "
|
||||
"WHERE c.edition_year = ? AND c.code = ?",
|
||||
(_year, code),
|
||||
)
|
||||
if _cpt.is_empty():
|
||||
_view = mo.md(
|
||||
f"**Where the manual files this code.** `{code}` is a HCPCS Level II code — "
|
||||
"CMS's own coding system, used when Medicare has a programmatic need CPT does "
|
||||
f"not cover — and is **not in the CPT {_year} book**; the manual's hierarchy "
|
||||
"below does not apply to it."
|
||||
)
|
||||
else:
|
||||
_row = _cpt.row(0, named=True)
|
||||
_crumb = " → ".join(_row["path_key"].split(" > "))
|
||||
_flags = [
|
||||
label
|
||||
for label, on in (
|
||||
("add-on ✚", _row["addon"]),
|
||||
("resequenced #", _row["resequenced"]),
|
||||
("new ●", _row["new"]),
|
||||
("revised ▲", _row["revised"]),
|
||||
("telemedicine ★", _row["telemedicine"]),
|
||||
("modifier-51 exempt ⦸", _row["mod51_exempt"]),
|
||||
("audio-only", _row["audio_only"]),
|
||||
("FDA-pending ⚡", _row["fda_pending"]),
|
||||
("PLA", _row["pla"]),
|
||||
)
|
||||
if on
|
||||
]
|
||||
_instr = q(
|
||||
"SELECT kind, text, targets FROM pfs.cpt_instruction WHERE edition_year = ? "
|
||||
"AND code = ? ORDER BY kind, text",
|
||||
(_year, code),
|
||||
)
|
||||
_view = mo.vstack(
|
||||
[
|
||||
mo.md(
|
||||
"**Where the manual files this code.** The CPT hierarchy is the first "
|
||||
f"organizing principle: in the CY{_year} book `{code}` sits under "
|
||||
f"**{_crumb}**."
|
||||
+ (
|
||||
f" Symbols: {', '.join(_flags)}."
|
||||
if _flags
|
||||
else " No symbols set."
|
||||
)
|
||||
),
|
||||
mo.ui.table(
|
||||
pl.DataFrame(
|
||||
{
|
||||
"part": [
|
||||
"stem",
|
||||
*["element"] * len(_row["elements"]),
|
||||
"tail",
|
||||
],
|
||||
"text": [_row["stem"], *_row["elements"], _row["tail"]],
|
||||
}
|
||||
),
|
||||
label="The manual's own stem / elements / tail",
|
||||
),
|
||||
(
|
||||
mo.ui.table(_instr, label="Parenthetical instructions")
|
||||
if not _instr.is_empty()
|
||||
else mo.md(
|
||||
"_No parenthetical instructions for this code in this edition._"
|
||||
)
|
||||
),
|
||||
]
|
||||
)
|
||||
_view
|
||||
return
|
||||
|
||||
|
||||
@app.cell(hide_code=True)
|
||||
def _(code, mo, not_built, q):
|
||||
# ── 2. What the extractor wrote ──
|
||||
_els = q(
|
||||
"SELECT type, value, detail, source, item_key, p_id, page FROM pfs.code_element "
|
||||
"WHERE code = ? ORDER BY type, value",
|
||||
(code,),
|
||||
)
|
||||
_rev = q(
|
||||
"SELECT text, proposed_value, item_key, p_id FROM pfs.code_element_review "
|
||||
"WHERE code = ? ORDER BY p_id",
|
||||
(code,),
|
||||
)
|
||||
if _els.is_empty():
|
||||
_view = mo.md(
|
||||
"## 2. What the extractor wrote\n\n"
|
||||
+ not_built(f"stack pfs elements --code {code}")
|
||||
)
|
||||
else:
|
||||
_view = mo.vstack(
|
||||
[
|
||||
mo.md(
|
||||
"## 2. What the extractor wrote\n\n"
|
||||
"Three passes, in order: the regex parser above finds what it can; a local "
|
||||
"model then reads every remaining candidate line and chooses **one slug from "
|
||||
"the closed list, or `none`**; whatever neither pass can place is queued for "
|
||||
"human review rather than guessed. Nothing enters `pfs.code_element` unless "
|
||||
"it is a member of the closed vocabulary, and every row keeps the exact "
|
||||
"Federal Register paragraph it came from."
|
||||
),
|
||||
mo.ui.table(
|
||||
_els, label=f"pfs.code_element — {code} ({_els.height} rows)"
|
||||
),
|
||||
(
|
||||
mo.ui.table(
|
||||
_rev,
|
||||
label=f"pfs.code_element_review — {code} ({_rev.height} lines)",
|
||||
)
|
||||
if not _rev.is_empty()
|
||||
else mo.md("_Review queue empty for this code._")
|
||||
),
|
||||
]
|
||||
)
|
||||
_view
|
||||
return
|
||||
|
||||
|
||||
@app.cell(hide_code=True)
|
||||
def _(alt, code, fr_md, mo, not_built, pl, plain_years, q):
|
||||
# ── 3. Lineage ──
|
||||
from pfs.families import family_of as _family_of
|
||||
|
||||
_fam = _family_of(code)
|
||||
_codes = list(_fam.codes) if _fam else [code]
|
||||
_ev = q(
|
||||
"SELECT code, year, kind, from_codes, to_codes, source, anchored, item_key, p_id, note "
|
||||
"FROM pfs.code_event WHERE code IN (" + ",".join("?" * len(_codes)) + ") "
|
||||
"ORDER BY year, code, kind",
|
||||
_codes,
|
||||
)
|
||||
if _ev.is_empty():
|
||||
_view = mo.md(
|
||||
"## 3. Lineage\n\n" + not_built(f"stack pfs lineage --code {code} --write")
|
||||
)
|
||||
else:
|
||||
_chart = (
|
||||
alt.Chart(_ev.to_pandas())
|
||||
.mark_circle(size=90)
|
||||
.encode(
|
||||
x=alt.X("year:O", title="Rule year"),
|
||||
y=alt.Y("kind:N", title=None),
|
||||
color=alt.Color("source:N", title="Source"),
|
||||
shape=alt.Shape("anchored:N", title="Anchored"),
|
||||
tooltip=[
|
||||
"code",
|
||||
"year",
|
||||
"kind",
|
||||
"from_codes",
|
||||
"to_codes",
|
||||
"item_key",
|
||||
"p_id",
|
||||
"note",
|
||||
],
|
||||
)
|
||||
.properties(height=260, width=640)
|
||||
)
|
||||
# Ruling A2: build the anchor column with a plain list comprehension, not
|
||||
# DataFrame.map_rows.
|
||||
_anchor_col = [
|
||||
fr_md(item_key, p_id) if item_key else "rvu"
|
||||
for item_key, p_id in zip(_ev["item_key"].to_list(), _ev["p_id"].to_list())
|
||||
]
|
||||
_links = _ev.with_columns(pl.Series("anchor", _anchor_col))
|
||||
_view = mo.vstack(
|
||||
[
|
||||
mo.md(
|
||||
"## 3. Lineage\n\n"
|
||||
f"Every dated event for the **{_fam.name if _fam else code}** codes. "
|
||||
"RVU-file events (`source=rvu`) are dated by the fee-schedule year; Federal "
|
||||
"Register events are dated by the **rule that mentions them** — a later rule "
|
||||
"recounting a code's creation adds a later `created` row, which is why the "
|
||||
"earliest anchored event is the origin, not the latest one. An RVU event is "
|
||||
"`anchored` when a Federal Register event for the same code lies within one "
|
||||
"rule year of it; an unanchored RVU event is evidence CMS never wrote a "
|
||||
"sentence about, and is weaker to cite. CPT-source events (`source=cpt`) are "
|
||||
"dated a third way: `cpt_changed` rows use the AMA's own CPT Changes edition "
|
||||
"year — the year the AMA revised the code, independent of whether or when any "
|
||||
"FR rule mentions it — and both an FR event and a CPT event for the same code "
|
||||
"can anchor the same RVU change from two independent directions."
|
||||
),
|
||||
mo.ui.altair_chart(_chart),
|
||||
mo.ui.table(
|
||||
plain_years(_links.drop("item_key", "p_id")), label="pfs.code_event"
|
||||
),
|
||||
]
|
||||
)
|
||||
_view
|
||||
return
|
||||
|
||||
|
||||
@app.cell(hide_code=True)
|
||||
def _(code, mo, not_built, pl, plain_years, q):
|
||||
# ── 4. Families ──
|
||||
from pfs.families import family_of as _family_of
|
||||
|
||||
_hand = _family_of(code)
|
||||
_key = _hand.key if _hand else ""
|
||||
_rows = (
|
||||
q(
|
||||
"SELECT key, name, code, role, since, until, item_key, p_id, note "
|
||||
"FROM pfs.code_family WHERE key = ? ORDER BY code",
|
||||
(_key,),
|
||||
)
|
||||
if _key
|
||||
else None
|
||||
)
|
||||
if _rows is None or _rows.is_empty():
|
||||
_view = mo.md("## 4. Families\n\n" + not_built("stack pfs families --write"))
|
||||
else:
|
||||
# The family's CPT heading — the organizing principle that classified its members
|
||||
# (Task 6 context: "note" is per-code, so different members can carry different
|
||||
# headings when the hand list or a derived edge pulls in a neighboring heading).
|
||||
_notes = [n for n in _rows["note"].to_list() if n]
|
||||
_note = _notes[0] if _notes else ""
|
||||
_siblings = pl.DataFrame()
|
||||
if _note:
|
||||
_parts = _note.split(" > ")
|
||||
_parent_prefix = " > ".join(_parts[:-1])
|
||||
_yr = q("SELECT max(edition_year) y FROM pfs.cpt_section")
|
||||
_year = _yr.item(0, "y") if not _yr.is_empty() else None
|
||||
if _year is not None:
|
||||
_lvl_row = q(
|
||||
"SELECT min(level) lvl FROM pfs.cpt_section WHERE edition_year = ? "
|
||||
"AND path_key = ?",
|
||||
(_year, _note),
|
||||
)
|
||||
_lvl = (
|
||||
_lvl_row.item(0, "lvl")
|
||||
if not _lvl_row.is_empty() and _lvl_row.item(0, "lvl") is not None
|
||||
else None
|
||||
)
|
||||
if _lvl is not None:
|
||||
_siblings = q(
|
||||
"SELECT DISTINCT title FROM pfs.cpt_section WHERE edition_year = ? "
|
||||
"AND path_key LIKE ? AND level = ? ORDER BY title",
|
||||
(_year, _parent_prefix + " > %", _lvl),
|
||||
)
|
||||
_summary = q(
|
||||
"WITH fam AS (SELECT key, count(*) n, bool_or(note <> '') has_note "
|
||||
"FROM pfs.code_family GROUP BY key) "
|
||||
"SELECT count(*) total, count(*) FILTER (WHERE n > 1) multi, "
|
||||
"count(*) FILTER (WHERE has_note) cpt_named FROM fam"
|
||||
)
|
||||
_by_chapter = q(
|
||||
"SELECT split_part(note, ' > ', 1) AS chapter, count(DISTINCT key) AS families "
|
||||
"FROM pfs.code_family WHERE note <> '' GROUP BY 1 ORDER BY families DESC"
|
||||
)
|
||||
_view = mo.vstack(
|
||||
[
|
||||
mo.md(
|
||||
"## 4. Families\n\n"
|
||||
"A family is a connected component over four kinds of edge: an **add-on** "
|
||||
"relation element (`in conjunction with 99490`), a **defined-by-reference** "
|
||||
"relation (`with the elements included in 99490`), a **single-target "
|
||||
"replacement** lineage event (one code's `replaced_by` names exactly one "
|
||||
"successor), and **stem similarity with an identical activity set** (two "
|
||||
"descriptors share half their service-naming words and every activity "
|
||||
"element). The hand-written registry is a floor, never a ceiling: "
|
||||
f"**{_key}** lists {len(_hand.codes)} hand codes; the derived table below "
|
||||
f"shows {_rows.height}, because the connected-component search also reaches "
|
||||
"codes the hand list never named."
|
||||
),
|
||||
mo.md(
|
||||
f"**Where this family sits in the manual.** {_note}"
|
||||
if _note
|
||||
else "_No member of this family carries a CPT heading (HCPCS-only family)._"
|
||||
),
|
||||
mo.ui.table(plain_years(_rows), label=f"pfs.code_family — {_key}"),
|
||||
(
|
||||
mo.ui.table(
|
||||
_siblings, label="Sibling headings under the same parent"
|
||||
)
|
||||
if not _siblings.is_empty()
|
||||
else mo.md("_No sibling headings found._")
|
||||
),
|
||||
mo.md(
|
||||
"**Example (single-target replacement):** HCPCS G2058, billable only in "
|
||||
"2020, was replaced the following year by CPT 99439 — an identical "
|
||||
"descriptor crosswalked at the same value. That `replaced_by` event names "
|
||||
"exactly one target code, so it is unambiguous evidence, and G2058 joins CCM "
|
||||
"as a *predecessor* rather than becoming a one-code family of its own."
|
||||
),
|
||||
(
|
||||
mo.md(
|
||||
f"**Across all families:** {_summary.item(0, 'total')} total, "
|
||||
f"{_summary.item(0, 'multi')} multi-code, "
|
||||
f"{_summary.item(0, 'cpt_named')} carry a CPT heading (`note <> ''`)."
|
||||
)
|
||||
if not _summary.is_empty()
|
||||
else mo.md("")
|
||||
),
|
||||
(
|
||||
mo.ui.table(
|
||||
_by_chapter, label="CPT-named families by top-level chapter"
|
||||
)
|
||||
if not _by_chapter.is_empty()
|
||||
else mo.md("")
|
||||
),
|
||||
]
|
||||
)
|
||||
_view
|
||||
return
|
||||
|
||||
|
||||
@app.cell(hide_code=True)
|
||||
def _(mo, not_built):
|
||||
# ── 5. Anchors in the corpus ──
|
||||
mo.md(
|
||||
"## 5. Anchors in the corpus\n\n"
|
||||
"How many chunks in the RAG index — comments, guidance, Federal Register text — carry "
|
||||
"this family's codes as metadata, and how many bibliography items carry a `family:` or "
|
||||
"`code:` tag. That count lives in pgvector, not the DuckDB replica this notebook reads.\n\n"
|
||||
+ not_built("stack llm restamp")
|
||||
)
|
||||
return
|
||||
|
||||
|
||||
@app.cell(hide_code=True)
|
||||
def _(mo, not_built):
|
||||
# ── 6. Guidance ──
|
||||
mo.md(
|
||||
"## 6. Guidance\n\n"
|
||||
"Sub-regulatory guidance — Medicare Learning Network articles, Internet-Only Manual "
|
||||
"sections, MACs' local coverage determinations — that cites a family's codes, with CFR "
|
||||
"cross-references where the guidance implements a rule.\n\n"
|
||||
+ not_built("stack pfs guidance --family CCM --write")
|
||||
)
|
||||
return
|
||||
|
||||
|
||||
@app.cell(hide_code=True)
|
||||
def _(mo, not_built):
|
||||
# ── 7. Reaction ──
|
||||
mo.md(
|
||||
"## 7. Reaction\n\n"
|
||||
"Public comment volume on a family's codes over time — how many comment letters mention "
|
||||
"them, in which rule years, and (on a sample) whether commenters supported or opposed the "
|
||||
"proposal.\n\n" + not_built("stack pfs reaction --family CCM --write")
|
||||
)
|
||||
return
|
||||
|
||||
|
||||
@app.cell(hide_code=True)
|
||||
def _(NOTES, REPLICA_PATH, con, mo, pl, q, store):
|
||||
# ── 8. Provenance ──
|
||||
import datetime as _dt
|
||||
|
||||
if REPLICA_PATH.exists():
|
||||
_mtime = _dt.datetime.fromtimestamp(REPLICA_PATH.stat().st_mtime).isoformat(
|
||||
timespec="seconds"
|
||||
)
|
||||
_replica_line = f"`{REPLICA_PATH}` — last published {_mtime}"
|
||||
else:
|
||||
_replica_line = f"`{REPLICA_PATH}` — not found"
|
||||
_counts = q(
|
||||
"SELECT 'code_element' t, count(*) n FROM pfs.code_element "
|
||||
"UNION ALL SELECT 'code_element_review', count(*) FROM pfs.code_element_review "
|
||||
"UNION ALL SELECT 'code_event', count(*) FROM pfs.code_event "
|
||||
"UNION ALL SELECT 'code_family', count(*) FROM pfs.code_family"
|
||||
)
|
||||
_log = q(
|
||||
"SELECT run_id, ingested_at, module, table_name, rule_id, source_file, sha256, rows, "
|
||||
"fr_citation, pincite_key FROM cms.ingest_log WHERE table_name LIKE 'pfs.%' "
|
||||
"ORDER BY ingested_at DESC LIMIT 20"
|
||||
)
|
||||
# pfs.cpt_* comes from a separate ingest path (stack pfs cpt-ingest) with its own
|
||||
# per-edition grain, so it gets its own counts table and its own bib provenance —
|
||||
# one row per CPT edition on the replica, with the edition's bib title.
|
||||
_cpt_counts = q(
|
||||
"SELECT 'cpt_section' t, edition_year, count(*) n FROM pfs.cpt_section GROUP BY 1, 2 "
|
||||
"UNION ALL SELECT 'cpt_code', edition_year, count(*) FROM pfs.cpt_code GROUP BY 1, 2 "
|
||||
"UNION ALL SELECT 'cpt_instruction', edition_year, count(*) "
|
||||
"FROM pfs.cpt_instruction GROUP BY 1, 2 "
|
||||
"UNION ALL SELECT 'cpt_reference', edition_year, count(*) "
|
||||
"FROM pfs.cpt_reference GROUP BY 1, 2 "
|
||||
"UNION ALL SELECT 'cpt_crosswalk', edition_year, count(*) "
|
||||
"FROM pfs.cpt_crosswalk GROUP BY 1, 2 "
|
||||
"UNION ALL SELECT 'cpt_list', edition_year, count(*) FROM pfs.cpt_list GROUP BY 1, 2 "
|
||||
"UNION ALL SELECT 'cpt_code_alt', edition_year, count(*) "
|
||||
"FROM pfs.cpt_code_alt GROUP BY 1, 2 "
|
||||
"ORDER BY edition_year, t"
|
||||
)
|
||||
_cpt_editions = q(
|
||||
"SELECT DISTINCT edition_year, item_key FROM pfs.cpt_code ORDER BY edition_year"
|
||||
)
|
||||
_bib_rows = []
|
||||
for _yr, _key in zip(
|
||||
[] if _cpt_editions.is_empty() else _cpt_editions["edition_year"].to_list(),
|
||||
[] if _cpt_editions.is_empty() else _cpt_editions["item_key"].to_list(),
|
||||
):
|
||||
if store is None:
|
||||
_title = "(bibliography unavailable)"
|
||||
else:
|
||||
try:
|
||||
_title = store.get(_key).title
|
||||
except KeyError:
|
||||
_title = "(item not found)"
|
||||
_bib_rows.append({"edition_year": _yr, "item_key": _key, "title": _title})
|
||||
_cpt_bib = pl.DataFrame(_bib_rows) if _bib_rows else pl.DataFrame()
|
||||
mo.vstack(
|
||||
[
|
||||
mo.md(
|
||||
"## 8. Provenance\n\n"
|
||||
f"Replica: {_replica_line}"
|
||||
+ (
|
||||
" (connection open)"
|
||||
if con is not None
|
||||
else " (connection unavailable)"
|
||||
)
|
||||
+ ". Row counts and the most recent ingest-log entries for every `pfs.code_*` "
|
||||
"table this notebook reads follow, so a reader can tell how fresh the family "
|
||||
"data is without leaving the page. The CPT codebook tables (`pfs.cpt_*`) are "
|
||||
"ingested separately, one edition at a time, so their counts and bib "
|
||||
"provenance are broken out by edition below."
|
||||
),
|
||||
mo.ui.table(_counts, label="pfs.code_* row counts")
|
||||
if not _counts.is_empty()
|
||||
else mo.md("_replica unavailable_"),
|
||||
mo.ui.table(_log, label="Ingest log — pfs.*")
|
||||
if not _log.is_empty()
|
||||
else mo.md("_no ingest log rows_"),
|
||||
mo.ui.table(_cpt_counts, label="pfs.cpt_* row counts by edition")
|
||||
if not _cpt_counts.is_empty()
|
||||
else mo.md(
|
||||
"_no CPT codebook tables built yet — run `stack pfs cpt-ingest --all`_"
|
||||
),
|
||||
mo.ui.table(_cpt_bib, label="CPT edition bib items")
|
||||
if not _cpt_bib.is_empty()
|
||||
else mo.md("_no CPT editions ingested_"),
|
||||
mo.md(
|
||||
"\n".join(f"- {k}: {v}" for k, v in NOTES.items())
|
||||
if NOTES
|
||||
else "_All sources available._"
|
||||
),
|
||||
]
|
||||
)
|
||||
return
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
app.run()
|
||||
261
src/bib/zotero_import.py
Normal file
261
src/bib/zotero_import.py
Normal file
@@ -0,0 +1,261 @@
|
||||
"""Pull items (with files) from a Zotero collection into bib.
|
||||
|
||||
``bib.sync`` only pushes bib → Zotero. This is the reverse: read-only
|
||||
access to the Zotero SQLite database (``conf.connect.zotero()``), so
|
||||
material the user already curated in Zotero — the AMA CPT manuals, for
|
||||
instance — can be cited by ``item_key`` like everything else in bib,
|
||||
files included, and indexed into the chat corpus like any other
|
||||
document (see ``llm.source.iter_corpus_refs``).
|
||||
|
||||
Usage::
|
||||
|
||||
from conf.connect import zotero, bib
|
||||
|
||||
zcon = zotero()
|
||||
books = list_collection(zcon, "AMA Coding Publications", with_files=True)
|
||||
stats = import_books(bib(), books, tags=["source:ama", "module:coding"])
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import sqlite3
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Sequence
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from bib.store import Store
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
_FIELD_NAMES = ("title", "url", "date", "publisher")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ZoteroBook:
|
||||
"""One Zotero book item: metadata + resolved attachment paths."""
|
||||
|
||||
key: str
|
||||
title: str
|
||||
url: str
|
||||
year: str
|
||||
publisher: str
|
||||
attachments: tuple[Path, ...]
|
||||
|
||||
|
||||
def list_collection(
|
||||
zcon: sqlite3.Connection,
|
||||
name: str,
|
||||
*,
|
||||
with_files: bool,
|
||||
storage_dir: Path | None = None,
|
||||
) -> list[ZoteroBook]:
|
||||
"""Book items in the named Zotero collection, ordered by item id.
|
||||
|
||||
Excludes anything in ``deletedItems`` (items or attachments). A
|
||||
``storage:<file>`` attachment path resolves against *storage_dir*
|
||||
(default ``conf.path("storage.zotero")``) at
|
||||
``<storage_dir>/<attachment item key>/<file>`` — the layout the
|
||||
Zotero desktop app uses on disk.
|
||||
|
||||
``with_files=True`` returns only items with at least one resolved
|
||||
attachment — for a citation-plus-file import like the CPT editions,
|
||||
a catalog stub with no book behind it isn't worth pulling in.
|
||||
"""
|
||||
if storage_dir is None:
|
||||
from conf import path
|
||||
|
||||
storage_dir = path("storage.zotero")
|
||||
storage_dir = Path(storage_dir)
|
||||
|
||||
rows = zcon.execute(
|
||||
"SELECT i.itemID, i.key FROM items i "
|
||||
"JOIN collectionItems ci ON ci.itemID = i.itemID "
|
||||
"JOIN collections c ON c.collectionID = ci.collectionID "
|
||||
"JOIN itemTypes it ON it.itemTypeID = i.itemTypeID AND it.typeName = 'book' "
|
||||
"WHERE c.collectionName = ? "
|
||||
"AND i.itemID NOT IN (SELECT itemID FROM deletedItems) "
|
||||
"ORDER BY i.itemID",
|
||||
(name,),
|
||||
).fetchall()
|
||||
|
||||
books: list[ZoteroBook] = []
|
||||
for item_id, key in rows:
|
||||
meta = dict(
|
||||
zcon.execute(
|
||||
"SELECT f.fieldName, idv.value FROM itemData id "
|
||||
"JOIN fields f ON f.fieldID = id.fieldID "
|
||||
"JOIN itemDataValues idv ON idv.valueID = id.valueID "
|
||||
"WHERE id.itemID = ? AND f.fieldName IN (?, ?, ?, ?)",
|
||||
(item_id, *_FIELD_NAMES),
|
||||
).fetchall()
|
||||
)
|
||||
att_rows = zcon.execute(
|
||||
"SELECT a.key, ia.path FROM itemAttachments ia "
|
||||
"JOIN items a ON a.itemID = ia.itemID "
|
||||
"WHERE ia.parentItemID = ? "
|
||||
"AND a.itemID NOT IN (SELECT itemID FROM deletedItems)",
|
||||
(item_id,),
|
||||
).fetchall()
|
||||
attachments = tuple(
|
||||
storage_dir / att_key / att_path[len("storage:") :]
|
||||
for att_key, att_path in att_rows
|
||||
if att_path and att_path.startswith("storage:")
|
||||
)
|
||||
if with_files and not attachments:
|
||||
continue
|
||||
|
||||
date = meta.get("date", "") or ""
|
||||
year = date[:4] if date[:4].isdigit() else ""
|
||||
books.append(
|
||||
ZoteroBook(
|
||||
key=key,
|
||||
title=meta.get("title", "") or "",
|
||||
url=meta.get("url", "") or "",
|
||||
year=year,
|
||||
publisher=meta.get("publisher", "") or "",
|
||||
attachments=attachments,
|
||||
)
|
||||
)
|
||||
return books
|
||||
|
||||
|
||||
def import_books(
|
||||
store: "Store",
|
||||
books: Sequence[ZoteroBook],
|
||||
*,
|
||||
tags: Sequence[str] = (),
|
||||
copy_files: bool = True,
|
||||
dry_run: bool = False,
|
||||
) -> dict:
|
||||
"""Upsert each :class:`ZoteroBook` into *store* as a ``Source`` item.
|
||||
|
||||
Dedup is by URL — Zotero items without one fall back to a stable
|
||||
``zotero://select/library/items/<key>`` URL, so re-running the
|
||||
import always resolves to the same bib item. Every item gets
|
||||
*tags* plus ``year:<YYYY>`` when the book has a year; every
|
||||
attachment is copied in via :meth:`Store.attach_file`, which is
|
||||
idempotent by filename — re-running never duplicates a file.
|
||||
|
||||
``dry_run=True`` makes no writes: it reports what a real run would
|
||||
do (bib keys are unknown for not-yet-created items, so ``items``
|
||||
entries carry an empty ``key`` in that case).
|
||||
|
||||
Returns a summary dict: ``created``/``updated``/``unchanged``
|
||||
counts items by :meth:`Store.upsert_status` outcome, ``attached``
|
||||
counts newly-copied files (a re-run's dedup'd attachments don't
|
||||
count), ``missing`` counts attachment paths that don't exist on
|
||||
disk (logged and skipped, never silently dropped), and ``items``
|
||||
lists one record per book for reporting — each with its own
|
||||
``attached``, ``missing``, and ``missing_paths`` (the paths that
|
||||
were missing, for a caller to surface them).
|
||||
"""
|
||||
from bib.item import Source
|
||||
|
||||
stats = {
|
||||
"created": 0,
|
||||
"updated": 0,
|
||||
"unchanged": 0,
|
||||
"attached": 0,
|
||||
"missing": 0,
|
||||
"items": [],
|
||||
}
|
||||
|
||||
for book in books:
|
||||
url = book.url or f"zotero://select/library/items/{book.key}"
|
||||
|
||||
if dry_run:
|
||||
existing = (
|
||||
store._con()
|
||||
.execute( # noqa: SLF001
|
||||
"SELECT key FROM items WHERE url = ?", (url,)
|
||||
)
|
||||
.fetchone()
|
||||
)
|
||||
status = "unchanged" if existing else "created"
|
||||
stats[status] += 1
|
||||
n_new, n_missing, missing_paths = 0, 0, []
|
||||
if copy_files:
|
||||
have = (
|
||||
{
|
||||
r[0]
|
||||
for r in store._con()
|
||||
.execute( # noqa: SLF001
|
||||
"SELECT a.filename FROM attachments a "
|
||||
"JOIN items i ON i.id = a.item_id WHERE i.key = ?",
|
||||
(existing[0],),
|
||||
)
|
||||
.fetchall()
|
||||
}
|
||||
if existing
|
||||
else set()
|
||||
)
|
||||
for p in book.attachments:
|
||||
if not p.exists():
|
||||
log.warning("missing on disk: %s", p)
|
||||
missing_paths.append(str(p))
|
||||
n_missing += 1
|
||||
elif p.name not in have:
|
||||
n_new += 1
|
||||
stats["attached"] += n_new
|
||||
stats["missing"] += n_missing
|
||||
stats["items"].append(
|
||||
{
|
||||
"key": existing[0] if existing else "",
|
||||
"zotero_key": book.key,
|
||||
"title": book.title,
|
||||
"status": status,
|
||||
"attached": n_new,
|
||||
"missing": n_missing,
|
||||
"missing_paths": missing_paths,
|
||||
}
|
||||
)
|
||||
continue
|
||||
|
||||
item = Source(
|
||||
title=book.title,
|
||||
url=url,
|
||||
date_published=book.year,
|
||||
institution=book.publisher,
|
||||
)
|
||||
item_tags = [*tags, *([f"year:{book.year}"] if book.year else [])]
|
||||
key, status = store.upsert_status(item, tags=item_tags)
|
||||
stats[status] += 1
|
||||
|
||||
n_new, n_missing, missing_paths = 0, 0, []
|
||||
if copy_files:
|
||||
for att_path in book.attachments:
|
||||
if not att_path.exists():
|
||||
log.warning("missing on disk: %s", att_path)
|
||||
missing_paths.append(str(att_path))
|
||||
n_missing += 1
|
||||
continue
|
||||
dup = (
|
||||
store._con()
|
||||
.execute( # noqa: SLF001
|
||||
"SELECT 1 FROM attachments a "
|
||||
"JOIN items i ON i.id = a.item_id "
|
||||
"WHERE i.key = ? AND a.filename = ?",
|
||||
(key, att_path.name),
|
||||
)
|
||||
.fetchone()
|
||||
)
|
||||
if dup is None:
|
||||
store.attach_file(key, att_path)
|
||||
n_new += 1
|
||||
stats["attached"] += n_new
|
||||
stats["missing"] += n_missing
|
||||
stats["items"].append(
|
||||
{
|
||||
"key": key,
|
||||
"zotero_key": book.key,
|
||||
"title": book.title,
|
||||
"status": status,
|
||||
"attached": n_new,
|
||||
"missing": n_missing,
|
||||
"missing_paths": missing_paths,
|
||||
}
|
||||
)
|
||||
|
||||
return stats
|
||||
@@ -521,6 +521,76 @@ def watch_iom() -> None:
|
||||
typer.echo(f"futurepdf.pdf: {status} (sha256={digest[:12]})")
|
||||
|
||||
|
||||
@app.command(name="import-zotero")
|
||||
def import_zotero(
|
||||
collection: str = typer.Option(
|
||||
..., "--collection", help="Zotero collection name to pull from."
|
||||
),
|
||||
with_files: bool = typer.Option(
|
||||
False,
|
||||
"--with-files/--no-with-files",
|
||||
help="Only import items that have at least one attachment file.",
|
||||
),
|
||||
only: str = typer.Option(
|
||||
"", "--only", help="Title substring filter (case-insensitive)."
|
||||
),
|
||||
tag: list[str] = typer.Option(
|
||||
None,
|
||||
"--tag",
|
||||
help="Tag every imported item with this. Repeat for multiple.",
|
||||
),
|
||||
dry_run: bool = typer.Option(
|
||||
False, "--dry-run", help="Report what would happen without writing."
|
||||
),
|
||||
) -> None:
|
||||
"""Pull a Zotero collection's book items (with files) into bib.
|
||||
|
||||
The reverse of ``sync-zotero``: reads the Zotero SQLite database
|
||||
read-only and upserts each book as a bib ``Source`` item (dedup by
|
||||
URL), copying its attachment file(s) into bib storage — an ordinary
|
||||
corpus document like any other, indexed like everything else in
|
||||
bib. ``llm:skip`` is available as a generic opt-out for material
|
||||
that should not be embedded (see ``llm.source.iter_corpus_refs``)
|
||||
— e.g. a scratch export or a duplicate scan — but it's never added
|
||||
automatically here; pass ``--tag llm:skip`` explicitly if it
|
||||
applies.
|
||||
"""
|
||||
from bib import connect
|
||||
from bib.zotero_import import import_books, list_collection
|
||||
from conf.connect import zotero
|
||||
|
||||
zcon = zotero()
|
||||
try:
|
||||
books = list_collection(zcon, collection, with_files=with_files)
|
||||
finally:
|
||||
zcon.close()
|
||||
|
||||
if only:
|
||||
needle = only.lower()
|
||||
books = [b for b in books if needle in b.title.lower()]
|
||||
|
||||
typer.echo(f"{len(books)} book(s) matched in '{collection}'")
|
||||
|
||||
store = connect()
|
||||
stats = import_books(store, books, tags=tag or [], dry_run=dry_run)
|
||||
|
||||
prefix = "would-" if dry_run else ""
|
||||
for rec in stats["items"]:
|
||||
typer.echo(
|
||||
f" {rec['zotero_key']} -> {rec['key'] or '(new)'}: "
|
||||
f'{prefix}{rec["status"]} "{rec["title"][:60]}" '
|
||||
f"attachments={rec['attached']}"
|
||||
)
|
||||
for missing_path in rec.get("missing_paths", []):
|
||||
typer.echo(f"! missing on disk: {missing_path}", err=True)
|
||||
|
||||
typer.echo(
|
||||
f"{prefix}created={stats['created']} {prefix}updated={stats['updated']} "
|
||||
f"{prefix}unchanged={stats['unchanged']} {prefix}attached={stats['attached']} "
|
||||
f"missing={stats['missing']}"
|
||||
)
|
||||
|
||||
|
||||
@app.command(name="sync-zotero")
|
||||
def sync_zotero(
|
||||
tag: str = typer.Option("", "--tag", help="Only sync items matching a tag."),
|
||||
|
||||
@@ -26,7 +26,7 @@ def _refs_for(
|
||||
zotero = ZoteroPdfIndex.lazy(
|
||||
path("db.zotero"), path("storage.zotero"), ROOT / ".state" / "llm"
|
||||
)
|
||||
return iter_corpus_refs(store, zotero=zotero)
|
||||
return iter_corpus_refs(store, keys=keys, zotero=zotero)
|
||||
|
||||
|
||||
@app.command()
|
||||
@@ -35,7 +35,9 @@ def index(
|
||||
"comments", help="Which collection: comments | rules | corpus | all."
|
||||
),
|
||||
docket: str = typer.Option("", help="Limit comments to one docket id."),
|
||||
key: list[str] = typer.Option([], "--key", help="Limit rules to these item keys."),
|
||||
key: list[str] = typer.Option(
|
||||
[], "--key", help="Limit rules/corpus to these item keys."
|
||||
),
|
||||
force: bool = typer.Option(False, help="Re-embed even when unchanged."),
|
||||
limit: int = typer.Option(0, help="Stop after N docs per collection (0 = all)."),
|
||||
) -> None:
|
||||
@@ -98,6 +100,68 @@ def index(
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
def restamp(
|
||||
collection: str = typer.Option(
|
||||
"comments", help="Which collection: comments | rules | corpus | all."
|
||||
),
|
||||
batch: int = typer.Option(2000, help="Rows per transaction."),
|
||||
dry_run: bool = typer.Option(
|
||||
False, "--dry-run", help="Scan and report without writing."
|
||||
),
|
||||
) -> None:
|
||||
"""Backfill codes/families/elements onto already-indexed chunks.
|
||||
|
||||
Metadata-only: computes `pfs.anchors.anchor_metadata` from each row's
|
||||
stored document text and merges it into `cmetadata` with a jsonb `||`,
|
||||
never touching the embedding — no GPU time, no re-chunking. Safe to
|
||||
re-run: rows already stamped with the current values are skipped.
|
||||
|
||||
F6: never rebuilds the HNSW index (`ensure_hnsw` ALTERs the vector
|
||||
column and takes an ACCESS EXCLUSIVE lock for the life of the
|
||||
rebuild) — only the cheap metadata GIN indexes, and not even those
|
||||
under `--dry-run`.
|
||||
"""
|
||||
import os
|
||||
|
||||
from llm import config as llm_config
|
||||
from llm.index import _engine
|
||||
from llm.migrate import ensure_metadata_indexes, migrate
|
||||
from llm.restamp import restamp as restamp_collection
|
||||
from pfs.anchors import code_family_index
|
||||
from pfs.families import FAMILIES, refresh_from
|
||||
|
||||
targets = _COLLECTIONS if collection == "all" else (collection,)
|
||||
if any(t not in _COLLECTIONS for t in targets):
|
||||
raise typer.BadParameter("collection must be comments, rules, corpus or all")
|
||||
cfg = llm_config.load()
|
||||
engine = _engine(cfg)
|
||||
migrate(engine)
|
||||
if not dry_run:
|
||||
ensure_metadata_indexes(engine)
|
||||
if os.path.exists(cfg.duckdb_replica):
|
||||
import duckdb
|
||||
|
||||
con = duckdb.connect(cfg.duckdb_replica, read_only=True)
|
||||
try:
|
||||
refresh_from(con)
|
||||
finally:
|
||||
con.close()
|
||||
code_index = code_family_index(FAMILIES)
|
||||
for target in targets:
|
||||
stats = restamp_collection(
|
||||
engine,
|
||||
collection=target,
|
||||
batch=batch,
|
||||
dry_run=dry_run,
|
||||
code_index=code_index,
|
||||
)
|
||||
typer.echo(
|
||||
f"{target}: scanned={stats['scanned']} updated={stats['updated']} "
|
||||
f"seconds={stats['seconds']}"
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
def hosts() -> None:
|
||||
"""Show the Ollama fleet: declared VRAM, liveness, models, and which
|
||||
|
||||
149
src/cli/pfs.py
149
src/cli/pfs.py
@@ -6,6 +6,8 @@
|
||||
uv run stack pfs lineage --code G2058 [--write]
|
||||
uv run stack pfs families [--write]
|
||||
uv run stack pfs review [--code G0556]
|
||||
uv run stack pfs cpt-ingest --edition 2024 [--edition 2022 …] [--dry-run]
|
||||
uv run stack pfs cpt-ingest --all
|
||||
|
||||
Writes go through ``duckdb_batch`` (single-writer rule) and republish
|
||||
the read-only replica so the chat and notebooks see them.
|
||||
@@ -19,13 +21,19 @@ from typing import Any
|
||||
import typer
|
||||
|
||||
from pfs.codetables import (
|
||||
cpt_years,
|
||||
ensure_tables,
|
||||
is_missing_table_error,
|
||||
read_all_elements,
|
||||
read_all_events,
|
||||
read_cpt_codes,
|
||||
read_cpt_instructions,
|
||||
read_cpt_sections,
|
||||
write_elements,
|
||||
write_events,
|
||||
write_families,
|
||||
)
|
||||
from pfs.cpt_load import ingest as cpt_ingest
|
||||
from pfs.extract import extract_code
|
||||
from pfs.families import FAMILIES, HAND_FAMILIES, derive_families, refresh_from
|
||||
from pfs.lineage import lineage
|
||||
@@ -38,10 +46,12 @@ app = typer.Typer(no_args_is_help=True)
|
||||
#: A replica that hasn't had `elements`/`lineage --write` (or an initial
|
||||
#: `families --write`) run yet is missing `pfs.code_element`/
|
||||
#: `pfs.code_event`/`pfs.code_family` — not a bug, just nothing derived
|
||||
#: yet (I4). DuckDB's "table does not exist" errors are Catalog errors.
|
||||
#: yet (I4). DuckDB's "table does not exist" errors are Catalog errors —
|
||||
#: the predicate itself lives in `pfs.codetables` so `pfs.lineage`/
|
||||
#: `pfs.extract` can reuse it for the same replica shape (a CPT-table-
|
||||
#: free replica is not a bug either).
|
||||
def _missing_table(exc: Exception) -> bool:
|
||||
msg = str(exc)
|
||||
return "Catalog" in msg or "does not exist" in msg
|
||||
return is_missing_table_error(exc)
|
||||
|
||||
|
||||
# ── indirections the tests monkeypatch ───────────────────────────────
|
||||
@@ -167,7 +177,12 @@ def elements(
|
||||
def _print_events(events: list[Any]) -> None:
|
||||
for e in events:
|
||||
arrow = f"{e.from_codes or '·'}→{e.to_codes or '·'}"
|
||||
anchor = f"{e.item_key} ¶{e.p_id}" if e.item_key else f"rvu {e.note}".strip()
|
||||
if e.source == "cpt":
|
||||
anchor = f"cpt {e.year} {e.item_key}"
|
||||
elif e.item_key:
|
||||
anchor = f"{e.item_key} ¶{e.p_id}"
|
||||
else:
|
||||
anchor = f"rvu {e.note}".strip()
|
||||
flag = "" if e.anchored else " UNANCHORED"
|
||||
typer.echo(f"{e.year} {e.kind:<18}{arrow:<16}{anchor}{flag}")
|
||||
|
||||
@@ -205,6 +220,40 @@ def lineage_cmd(
|
||||
con.close()
|
||||
|
||||
|
||||
def _cpt_inputs(
|
||||
con: Any,
|
||||
) -> tuple[list[Any], list[Any], list[Any], dict[str, tuple[int, ...]]]:
|
||||
"""The newest CPT edition's rows for ``derive_families`` (grouping/
|
||||
naming stays newest-edition-scoped, Ruling C12), plus ``cpt_presence``
|
||||
— every ingested edition's ``code -> years present`` map, one extra
|
||||
full-table pass, used only for ``since``/``until``: a code an earlier
|
||||
edition ingested but the newest one dropped still gets its true
|
||||
``since`` (not clamped to the newest edition) and a real ``until``.
|
||||
All empty on any replica shape that doesn't have them yet (I4: a
|
||||
plain read of an old replica, pre pfs.cpt_*, is not a bug)."""
|
||||
try:
|
||||
years = cpt_years(con)
|
||||
except Exception as exc:
|
||||
if _missing_table(exc):
|
||||
return [], [], [], {}
|
||||
raise
|
||||
if not years:
|
||||
return [], [], [], {}
|
||||
newest = max(years)
|
||||
presence: dict[str, list[int]] = {}
|
||||
for code, year in con.execute(
|
||||
"SELECT code, edition_year FROM pfs.cpt_code"
|
||||
).fetchall():
|
||||
presence.setdefault(code, []).append(year)
|
||||
cpt_presence = {code: tuple(sorted(set(yrs))) for code, yrs in presence.items()}
|
||||
return (
|
||||
read_cpt_codes(con, newest),
|
||||
read_cpt_sections(con, newest),
|
||||
read_cpt_instructions(con, newest),
|
||||
cpt_presence,
|
||||
)
|
||||
|
||||
|
||||
def _derive_rows(con: Any) -> list[Any]:
|
||||
codes = [
|
||||
r[0]
|
||||
@@ -225,28 +274,59 @@ def _derive_rows(con: Any) -> list[Any]:
|
||||
"SELECT hcpcs, arg_max(description, year) FROM pfs.rvu WHERE mod IS NULL OR mod = '' GROUP BY hcpcs"
|
||||
).fetchall()
|
||||
}
|
||||
return derive_families(elements_by_code, events_by_code, descriptions)
|
||||
cpt_codes, cpt_sections, cpt_instructions, cpt_presence = _cpt_inputs(con)
|
||||
return derive_families(
|
||||
elements_by_code,
|
||||
events_by_code,
|
||||
descriptions,
|
||||
cpt_codes=cpt_codes,
|
||||
cpt_sections=cpt_sections,
|
||||
cpt_instructions=cpt_instructions,
|
||||
cpt_presence=cpt_presence,
|
||||
)
|
||||
|
||||
|
||||
def _print_families(rows: list[Any]) -> None:
|
||||
by_key: dict[str, list[str]] = {}
|
||||
def _print_families(rows: list[Any], *, verbose: bool = False) -> None:
|
||||
by_key: dict[str, list[Any]] = {}
|
||||
for r in rows:
|
||||
by_key.setdefault(r.key, []).append(f"{r.code}({r.role})")
|
||||
for key, members in sorted(by_key.items()):
|
||||
typer.echo(f"{key}: {' '.join(members)}")
|
||||
by_key.setdefault(r.key, []).append(r)
|
||||
if verbose:
|
||||
for key, members in sorted(by_key.items()):
|
||||
typer.echo(f"{key}: {' '.join(f'{r.code}({r.role})' for r in members)}")
|
||||
total = len(by_key)
|
||||
multi = sum(1 for members in by_key.values() if len(members) >= 2)
|
||||
hand = sum(1 for key in by_key if key in HAND_FAMILIES)
|
||||
cpt_named = sum(
|
||||
1
|
||||
for key, members in by_key.items()
|
||||
if key not in HAND_FAMILIES and any(r.note for r in members)
|
||||
)
|
||||
other = total - hand - cpt_named
|
||||
typer.echo(
|
||||
f"families: total {total}, multi-code {multi}, cpt-named {cpt_named}, "
|
||||
f"hand {hand}, other {other}"
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
def families(write: bool = typer.Option(False, "--write")) -> None:
|
||||
def families(
|
||||
write: bool = typer.Option(False, "--write"),
|
||||
verbose: bool = typer.Option(
|
||||
False, "--verbose", help="Print every family's members, not just the summary."
|
||||
),
|
||||
) -> None:
|
||||
"""Derive families from pfs.code_element / pfs.code_event / pfs.rvu."""
|
||||
if write:
|
||||
with _batch() as con:
|
||||
ensure_tables(con)
|
||||
rows = _derive_rows(con)
|
||||
_print_families(rows)
|
||||
write_families(con, rows)
|
||||
refresh_from(con)
|
||||
_publish()
|
||||
# refresh_from and printing don't need the write lock open — do
|
||||
# them after it's released, from the freshly published replica
|
||||
# (the same pattern `stack llm restamp` uses).
|
||||
refresh_from(_read())
|
||||
_print_families(rows, verbose=verbose)
|
||||
return
|
||||
# No --write: derive-and-print never needs the write lock (I4).
|
||||
con = _read()
|
||||
@@ -258,7 +338,7 @@ def families(write: bool = typer.Option(False, "--write")) -> None:
|
||||
typer.echo("no derived tables yet — run with --write")
|
||||
return
|
||||
raise
|
||||
_print_families(rows)
|
||||
_print_families(rows, verbose=verbose)
|
||||
finally:
|
||||
con.close()
|
||||
|
||||
@@ -278,3 +358,44 @@ def review(code: str = typer.Option("", "--code")) -> None:
|
||||
sql + " ORDER BY code, p_id", params
|
||||
).fetchall():
|
||||
typer.echo(f"{c} {key} ¶{p_id} [{proposed or '?'}] {text[:120]}")
|
||||
|
||||
|
||||
def _print_cpt_counts(counts: dict[int, dict[str, int]]) -> None:
|
||||
for year in sorted(counts):
|
||||
c = counts[year]
|
||||
detail = ", ".join(f"{k}={v}" for k, v in c.items())
|
||||
typer.echo(f"{year}: {detail}")
|
||||
|
||||
|
||||
@app.command("cpt-ingest")
|
||||
def cpt_ingest_cmd(
|
||||
edition: list[int] = typer.Option(
|
||||
[], "--edition", help="CPT edition year to ingest (repeatable)."
|
||||
),
|
||||
all_editions: bool = typer.Option(
|
||||
False, "--all", help="Every CPT EPUB edition found in bib (source:ama)."
|
||||
),
|
||||
dry_run: bool = typer.Option(
|
||||
False, "--dry-run", help="Parse and report counts; write nothing."
|
||||
),
|
||||
) -> None:
|
||||
"""Parse CPT EPUB editions into pfs.cpt_section/cpt_code/cpt_instruction/
|
||||
cpt_reference/cpt_crosswalk/cpt_list (docs/superpowers/specs/
|
||||
2026-09-09-cpt-canonical-schema-design.md §3)."""
|
||||
if not edition and not all_editions:
|
||||
raise typer.BadParameter("pass --edition YEAR (repeatable) or --all")
|
||||
years = None if all_editions else sorted(set(edition))
|
||||
store = _store()
|
||||
if dry_run:
|
||||
# A preview never needs the DuckDB single-writer lock a notebook
|
||||
# may be holding (#508-#514) — parsing alone answers a dry run,
|
||||
# so no connection is opened at all (I4).
|
||||
counts = cpt_ingest(store, None, years=years, dry_run=True)
|
||||
_print_cpt_counts(counts)
|
||||
return
|
||||
with _batch() as con:
|
||||
# write_cpt_edition (via cpt_ingest) already calls ensure_tables
|
||||
# itself — no need to duplicate that here.
|
||||
counts = cpt_ingest(store, con, years=years, dry_run=False)
|
||||
_print_cpt_counts(counts)
|
||||
_publish()
|
||||
|
||||
@@ -13,8 +13,9 @@ from __future__ import annotations
|
||||
import hashlib
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from typing import Mapping
|
||||
|
||||
from pfs.families import find_codes
|
||||
from pfs.anchors import anchor_metadata
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -111,7 +112,11 @@ def _pack(section: str, target: int, overlap: int) -> list[str]:
|
||||
|
||||
|
||||
def chunk_doc(
|
||||
doc: Doc, *, target_chars: int = 2000, overlap_chars: int = 200
|
||||
doc: Doc,
|
||||
*,
|
||||
target_chars: int = 2000,
|
||||
overlap_chars: int = 200,
|
||||
code_index: Mapping[str, tuple[str, ...]] | None = None,
|
||||
) -> list[Chunk]:
|
||||
"""Chunk *doc* into <= target_chars windows with overlap between them.
|
||||
|
||||
@@ -120,13 +125,21 @@ def chunk_doc(
|
||||
Every chunk carries ``section`` — the markdown heading it sits under
|
||||
(``""`` when none), which is how comment chunks know their attachment.
|
||||
|
||||
*code_index* is ``pfs.anchors.code_family_index(families)`` — pass
|
||||
the same one the restamp backfill builds (hand families plus
|
||||
``pfs.families.refresh_from`` against the DuckDB replica) so a chunk
|
||||
stamped at index time and the same chunk restamped later carry
|
||||
identical ``families`` metadata (Ruling F1). ``None`` (the default)
|
||||
falls back to a live scan of the import-time ``pfs.families.FAMILIES``
|
||||
registry — only hand families, no CPT-derived ones.
|
||||
|
||||
Raises ``ValueError`` when ``overlap_chars >= target_chars`` (an
|
||||
overlap that large or larger would never let the window advance).
|
||||
"""
|
||||
if overlap_chars >= target_chars:
|
||||
raise ValueError("overlap_chars must be smaller than target_chars")
|
||||
if doc.paragraphs:
|
||||
return _chunk_paragraphs(doc, target_chars, overlap_chars)
|
||||
return _chunk_paragraphs(doc, target_chars, overlap_chars, code_index)
|
||||
body = _CONTROL.sub("", _FRONTMATTER.sub("", doc.text)).strip()
|
||||
if not body:
|
||||
return []
|
||||
@@ -148,14 +161,19 @@ def chunk_doc(
|
||||
# The heading is scanned with the piece: a section titled
|
||||
# "## G0556 — APCM" is about G0556 all the way down, but
|
||||
# only its first chunk repeats the code.
|
||||
"codes": " ".join(find_codes(f"{heading} {piece}")),
|
||||
**anchor_metadata(f"{heading} {piece}", code_index=code_index),
|
||||
},
|
||||
)
|
||||
for seq, (heading, piece) in enumerate(pieces)
|
||||
]
|
||||
|
||||
|
||||
def _chunk_paragraphs(doc: Doc, target: int, overlap: int) -> list[Chunk]:
|
||||
def _chunk_paragraphs(
|
||||
doc: Doc,
|
||||
target: int,
|
||||
overlap: int,
|
||||
code_index: Mapping[str, tuple[str, ...]] | None = None,
|
||||
) -> list[Chunk]:
|
||||
"""Greedy pack of whole paragraphs; an oversized paragraph is
|
||||
hard-wrapped with overlap, every piece keeping its own anchor."""
|
||||
packed: list[tuple[str, Paragraph, Paragraph]] = [] # text, first, last
|
||||
@@ -190,7 +208,7 @@ def _chunk_paragraphs(doc: Doc, target: int, overlap: int) -> list[Chunk]:
|
||||
"item_key": doc.key,
|
||||
"seq": str(seq),
|
||||
"section": "",
|
||||
"codes": " ".join(find_codes(text)),
|
||||
**anchor_metadata(text, code_index=code_index),
|
||||
"p_id": str(first.p_id),
|
||||
"p_id_last": str(last.p_id),
|
||||
"page": str(first.page),
|
||||
|
||||
@@ -20,6 +20,7 @@ table creation to first ``add_embeddings()``.
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
from typing import Iterable
|
||||
|
||||
from sqlalchemy import create_engine, text
|
||||
@@ -36,6 +37,27 @@ from llm.source import DocRef
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _code_index(cfg: LlmConfig) -> dict[str, tuple[str, ...]]:
|
||||
"""Build ``pfs.anchors.code_family_index`` the same way ``stack llm
|
||||
restamp`` does — hand families, plus ``pfs.families.refresh_from``
|
||||
against the DuckDB replica when it exists — so a chunk stamped here
|
||||
at index time carries the same ``families`` metadata a later restamp
|
||||
would compute for it (Ruling F1: the two paths must agree by
|
||||
construction, not by luck)."""
|
||||
from pfs.anchors import code_family_index
|
||||
from pfs.families import FAMILIES, refresh_from
|
||||
|
||||
if os.path.exists(cfg.duckdb_replica):
|
||||
import duckdb
|
||||
|
||||
con = duckdb.connect(cfg.duckdb_replica, read_only=True)
|
||||
try:
|
||||
refresh_from(con)
|
||||
finally:
|
||||
con.close()
|
||||
return code_family_index(FAMILIES)
|
||||
|
||||
|
||||
def _engine(cfg: LlmConfig) -> Engine: # pragma: no cover — needs a live DB
|
||||
return create_engine(pg_url(cfg))
|
||||
|
||||
@@ -162,6 +184,7 @@ def index_refs(
|
||||
pool.check(cfg.embed_model)
|
||||
store = vectorstore(collection, cfg, pool)
|
||||
seen = _state(engine, collection)
|
||||
code_index = _code_index(cfg)
|
||||
stats = {
|
||||
"indexed": 0,
|
||||
"skipped": 0,
|
||||
@@ -202,7 +225,7 @@ def index_refs(
|
||||
)
|
||||
stats["hash_skipped"] += 1
|
||||
continue
|
||||
chunks = enrich_pdf_pages(doc, chunk_doc(doc))
|
||||
chunks = enrich_pdf_pages(doc, chunk_doc(doc, code_index=code_index))
|
||||
if not chunks:
|
||||
_unindexable(ref)
|
||||
continue
|
||||
|
||||
@@ -44,18 +44,35 @@ CREATE TABLE IF NOT EXISTS index_docket_state (
|
||||
|
||||
# langchain-postgres creates langchain_pg_embedding with an untyped vector
|
||||
# column; HNSW needs a typed one. ALTER is a no-op when already typed.
|
||||
# The ALTER and the HNSW build both take an ACCESS EXCLUSIVE lock on the
|
||||
# whole table — fine right after a fresh index run's `add_embeddings`,
|
||||
# but never run this from `stack llm restamp` (F6): a metadata-only
|
||||
# backfill has no reason to lock out every reader for the minutes an
|
||||
# HNSW rebuild over 1M+ rows takes.
|
||||
_HNSW_DDL = [
|
||||
"ALTER TABLE langchain_pg_embedding ALTER COLUMN embedding TYPE vector({dim})",
|
||||
"CREATE INDEX IF NOT EXISTS ix_embedding_hnsw "
|
||||
"ON langchain_pg_embedding USING hnsw (embedding vector_cosine_ops)",
|
||||
# Code-cited sources (llm.evidence) match a code against the space-
|
||||
# separated cmetadata->>'codes' inside one collection; without these
|
||||
# two the query is a sequential scan of every embedding.
|
||||
"CREATE INDEX IF NOT EXISTS ix_embedding_collection "
|
||||
"ON langchain_pg_embedding (collection_id)",
|
||||
]
|
||||
|
||||
# Code-cited sources (llm.evidence) match a code against the space-
|
||||
# separated cmetadata->>'codes' inside one collection; without these the
|
||||
# query is a sequential scan of every embedding. Plain CREATE INDEX (no
|
||||
# ACCESS EXCLUSIVE, unlike ensure_hnsw's vector-column retype) — safe to
|
||||
# run against a live table, so `stack llm restamp` calls only this, never
|
||||
# ensure_hnsw.
|
||||
_METADATA_INDEX_DDL = [
|
||||
"CREATE INDEX IF NOT EXISTS ix_embedding_codes "
|
||||
"ON langchain_pg_embedding USING gin "
|
||||
"(string_to_array(coalesce(cmetadata->>'codes',''), ' '))",
|
||||
"CREATE INDEX IF NOT EXISTS ix_embedding_collection "
|
||||
"ON langchain_pg_embedding (collection_id)",
|
||||
"CREATE INDEX IF NOT EXISTS ix_embedding_families "
|
||||
"ON langchain_pg_embedding USING gin "
|
||||
"(string_to_array(coalesce(cmetadata->>'families',''), ' '))",
|
||||
"CREATE INDEX IF NOT EXISTS ix_embedding_elements "
|
||||
"ON langchain_pg_embedding USING gin "
|
||||
"(string_to_array(coalesce(cmetadata->>'elements',''), ' '))",
|
||||
]
|
||||
|
||||
|
||||
@@ -71,8 +88,23 @@ def ensure_hnsw(engine: Engine, dim: int) -> None:
|
||||
"""Type the embedding column and build the HNSW index.
|
||||
|
||||
Call after the PGVector store has created its tables (first
|
||||
``add_embeddings``), not before.
|
||||
``add_embeddings``), not before. ACCESS EXCLUSIVE for the life of
|
||||
the rebuild — never call this from a restamp run (F6); see
|
||||
``ensure_metadata_indexes``.
|
||||
"""
|
||||
with engine.begin() as conn:
|
||||
for ddl in _HNSW_DDL:
|
||||
conn.execute(text(ddl.format(dim=dim)))
|
||||
|
||||
|
||||
def ensure_metadata_indexes(engine: Engine) -> None:
|
||||
"""GIN indexes on the codes/families/elements metadata only.
|
||||
|
||||
No ACCESS EXCLUSIVE lock and no vector-column touch — safe to run
|
||||
against a live, queryable table. ``stack llm restamp`` calls this
|
||||
instead of ``ensure_hnsw`` (F6): a metadata-only backfill has no
|
||||
reason to rebuild the HNSW index.
|
||||
"""
|
||||
with engine.begin() as conn:
|
||||
for ddl in _METADATA_INDEX_DDL:
|
||||
conn.execute(text(ddl))
|
||||
|
||||
98
src/llm/restamp.py
Normal file
98
src/llm/restamp.py
Normal file
@@ -0,0 +1,98 @@
|
||||
"""Backfill anchor metadata on already-indexed chunks without re-embedding.
|
||||
|
||||
Re-indexing 1.15 M chunks costs days of GPU time; the anchors are a pure
|
||||
function of the chunk text, so they are recomputed in Python and merged
|
||||
into ``cmetadata`` with a jsonb concatenation. Keyset pagination on the
|
||||
primary key keeps memory flat; one transaction per batch keeps the lock
|
||||
short. Idempotent: rows whose three values already match are skipped.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
from typing import Any, Iterable, Mapping
|
||||
|
||||
from sqlalchemy import text
|
||||
|
||||
from pfs.anchors import anchor_metadata
|
||||
from pfs.families import Family
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
KEYS = ("codes", "families", "elements")
|
||||
|
||||
SELECT_SQL = (
|
||||
"SELECT e.id, e.document, e.cmetadata FROM langchain_pg_embedding e "
|
||||
"WHERE e.collection_id = (SELECT uuid FROM langchain_pg_collection WHERE name = :collection) "
|
||||
"AND e.id > :after ORDER BY e.id LIMIT :batch"
|
||||
)
|
||||
UPDATE_SQL = "UPDATE langchain_pg_embedding SET cmetadata = cmetadata || CAST(:patch AS jsonb) WHERE id = :id"
|
||||
|
||||
|
||||
def plan_patches(
|
||||
rows: Iterable[tuple[str, str, Mapping[str, Any] | None]],
|
||||
*,
|
||||
families: Mapping[str, Family] | None = None,
|
||||
code_index: Mapping[str, tuple[str, ...]] | None = None,
|
||||
) -> list[tuple[str, str]]:
|
||||
out: list[tuple[str, str]] = []
|
||||
for id_, document, md in rows:
|
||||
md = md or {}
|
||||
want = anchor_metadata(document or "", families=families, code_index=code_index)
|
||||
# A key that is entirely absent (the common case for a
|
||||
# pre-backfill row) must still be patched even when its computed
|
||||
# value happens to be "" — the acceptance check is key presence
|
||||
# (`cmetadata ? 'families'`), not value equality, so falling back
|
||||
# to a "" default for a missing key would make it look unchanged
|
||||
# and the key would never get written.
|
||||
if all(k in md and str(md[k]) == want[k] for k in KEYS):
|
||||
continue
|
||||
out.append((id_, json.dumps(want)))
|
||||
return out
|
||||
|
||||
|
||||
def restamp(
|
||||
engine: Any,
|
||||
*,
|
||||
collection: str,
|
||||
batch: int = 2000,
|
||||
dry_run: bool = False,
|
||||
families: Mapping[str, Family] | None = None,
|
||||
code_index: Mapping[str, tuple[str, ...]] | None = None,
|
||||
log_every: int = 50_000,
|
||||
) -> dict:
|
||||
scanned = updated = 0
|
||||
after = ""
|
||||
t0 = time.perf_counter()
|
||||
while True:
|
||||
with engine.begin() as conn:
|
||||
rows = conn.execute(
|
||||
text(SELECT_SQL),
|
||||
{"collection": collection, "after": after, "batch": batch},
|
||||
).fetchall()
|
||||
if not rows:
|
||||
break
|
||||
patches = plan_patches(rows, families=families, code_index=code_index)
|
||||
if patches and not dry_run:
|
||||
conn.execute(
|
||||
text(UPDATE_SQL), [{"id": i, "patch": p} for i, p in patches]
|
||||
)
|
||||
scanned += len(rows)
|
||||
updated += len(patches)
|
||||
after = rows[-1][0]
|
||||
if scanned % log_every < batch:
|
||||
log.info(
|
||||
"restamp %s: scanned %d updated %d (%.0fs)",
|
||||
collection,
|
||||
scanned,
|
||||
updated,
|
||||
time.perf_counter() - t0,
|
||||
)
|
||||
return {
|
||||
"collection": collection,
|
||||
"scanned": scanned,
|
||||
"updated": updated,
|
||||
"seconds": round(time.perf_counter() - t0, 1),
|
||||
}
|
||||
@@ -493,15 +493,29 @@ def _attachment_sections(
|
||||
|
||||
|
||||
def iter_corpus_refs(
|
||||
store: Store, *, tag: str = "", zotero: "ZoteroPdfIndex | None" = None
|
||||
store: Store,
|
||||
*,
|
||||
tag: str = "",
|
||||
keys: tuple[str, ...] = (),
|
||||
zotero: "ZoteroPdfIndex | None" = None,
|
||||
) -> Iterator[DocRef]:
|
||||
"""One DocRef per non-comment item. Fingerprint = updated_at + the
|
||||
bib attachment file stats; the Zotero fallback is only consulted by
|
||||
``load()``."""
|
||||
"""One DocRef per non-comment, non-skipped item. Fingerprint =
|
||||
updated_at + the bib attachment file stats; the Zotero fallback is
|
||||
only consulted by ``load()``.
|
||||
|
||||
Items tagged ``llm:skip`` are excluded — a generic opt-out for
|
||||
material that should not be embedded (e.g. a scratch export, or a
|
||||
duplicate scan of something already in the corpus). It's never
|
||||
applied automatically; a caller opts an item in by tagging it.
|
||||
|
||||
*keys*, when given, limits the yield to those item keys (same
|
||||
Python-side post-filter ``iter_rule_refs`` uses for its own
|
||||
``keys`` — the corpus scan is cheap enough that this doesn't need
|
||||
to be pushed into the SQL)."""
|
||||
sql = (
|
||||
"SELECT i.key, COALESCE(i.updated_at,'') AS updated_at FROM items i "
|
||||
"WHERE i.id NOT IN (SELECT item_id FROM item_tags WHERE tag_id IN "
|
||||
"(SELECT id FROM tags WHERE name = 'doctype:comment'))"
|
||||
"(SELECT id FROM tags WHERE name IN ('doctype:comment', 'llm:skip')))"
|
||||
+ (
|
||||
" AND i.id IN (SELECT item_id FROM item_tags WHERE tag_id IN "
|
||||
"(SELECT id FROM tags WHERE name = ?))"
|
||||
@@ -512,6 +526,8 @@ def iter_corpus_refs(
|
||||
)
|
||||
for row in store._con().execute(sql, (tag,) if tag else ()).fetchall():
|
||||
key = row["key"]
|
||||
if keys and key not in keys:
|
||||
continue
|
||||
fp = row["updated_at"] + "|" + fingerprint_files(_attachment_paths(store, key))
|
||||
|
||||
def _load(key=key) -> Doc | None:
|
||||
|
||||
92
src/pfs/anchors.py
Normal file
92
src/pfs/anchors.py
Normal file
@@ -0,0 +1,92 @@
|
||||
"""Anchor metadata for a piece of text: the codes it names, the families
|
||||
those codes belong to, and the element slugs its phrases match. One
|
||||
function, used by the indexer at chunk time (``llm.chunk.chunk_doc``,
|
||||
called from ``llm.index.index_refs``) and by the restamp backfill
|
||||
(``llm.restamp.restamp``). Both callers build one ``code_family_index``
|
||||
per run the same way — hand families, plus ``pfs.families.refresh_from``
|
||||
against the DuckDB replica when it exists — and pass it through as
|
||||
``code_index``, so the two paths compute identical values from the same
|
||||
family snapshot by construction, not by coincidence (Ruling F1). Pure."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Mapping
|
||||
|
||||
from pfs.elements import parse_descriptor
|
||||
from pfs.families import FAMILIES, Family, find_codes
|
||||
|
||||
|
||||
def code_family_index(families: Mapping[str, Family]) -> dict[str, tuple[str, ...]]:
|
||||
"""Code -> sorted tuple of every qualifying family key, built once so
|
||||
``anchor_metadata`` doesn't scan every family for every chunk (a
|
||||
restamp run calls it ~1.15M times).
|
||||
|
||||
Single-code families are excluded (Ruling A5): the derived registry
|
||||
is ~17,187 keys, all but a handful stem families with exactly one
|
||||
code, and a 'families' anchor on one of those would just echo the
|
||||
code already in 'codes'. Only families with two or more codes (today:
|
||||
ACP, CCM, PCM, TCM, APCM) are real groupings worth a separate anchor.
|
||||
|
||||
Ruling A6: a code that belongs to two qualifying families keeps
|
||||
*both* — a single-winner index made the indexed path (``restamp``)
|
||||
disagree with the un-indexed scan in ``anchor_metadata`` (which
|
||||
unions every match) the moment two families shared a code, which the
|
||||
next slice (CPT headings intersecting a hand family) will create.
|
||||
The two paths must write identical 'families' values for the same
|
||||
text by construction."""
|
||||
out: dict[str, list[str]] = {}
|
||||
for key, fam in families.items():
|
||||
if len(fam.codes) < 2:
|
||||
continue
|
||||
for code in fam.codes:
|
||||
out.setdefault(code, []).append(key)
|
||||
return {code: tuple(sorted(keys)) for code, keys in out.items()}
|
||||
|
||||
|
||||
def anchor_metadata(
|
||||
text: str,
|
||||
*,
|
||||
families: Mapping[str, Family] | None = None,
|
||||
code_index: Mapping[str, tuple[str, ...]] | None = None,
|
||||
) -> dict[str, str]:
|
||||
"""Extract codes, families, and elements from text.
|
||||
|
||||
Returns a dict with keys:
|
||||
- 'codes': space-joined sorted unique codes (from find_codes)
|
||||
- 'families': space-joined sorted unique family keys — every
|
||||
qualifying family a code belongs to, unioned across all codes
|
||||
(families with only one code are excluded — see
|
||||
``code_family_index``)
|
||||
- 'elements': space-joined sorted unique element slugs (type=value, no detail)
|
||||
|
||||
Args:
|
||||
text: The text to analyze
|
||||
families: Optional mapping of family keys to Family objects. Defaults to pfs.families.FAMILIES
|
||||
code_index: Optional precomputed ``code_family_index(families)``. When
|
||||
given, family lookup goes through it instead of scanning every
|
||||
family for every code — same result, built once per run by a
|
||||
caller like the restamp backfill.
|
||||
|
||||
Returns:
|
||||
dict with 'codes', 'families', and 'elements' keys
|
||||
"""
|
||||
codes = find_codes(text)
|
||||
if code_index is not None:
|
||||
keys = sorted({k for c in codes if c in code_index for k in code_index[c]})
|
||||
else:
|
||||
fams = FAMILIES if families is None else families
|
||||
keys = sorted(
|
||||
{
|
||||
k
|
||||
for k, f in fams.items()
|
||||
if len(f.codes) >= 2
|
||||
for c in codes
|
||||
if c in f.codes
|
||||
}
|
||||
)
|
||||
elements = sorted({f"{e.type.value}={e.value}" for e in parse_descriptor(text)})
|
||||
return {
|
||||
"codes": " ".join(codes),
|
||||
"families": " ".join(keys),
|
||||
"elements": " ".join(elements),
|
||||
}
|
||||
@@ -5,16 +5,34 @@
|
||||
``pfs.code_event`` dated lineage events (created, replaced, …)
|
||||
``pfs.code_family`` derived family membership with roles
|
||||
|
||||
Writers are delete-then-insert per code (events, elements) or full
|
||||
replace (families) so a re-run is idempotent. Callers own the
|
||||
connection (``conf.connect.duckdb_batch`` for writes) and republish the
|
||||
replica afterwards; nothing here opens a database.
|
||||
``pfs.cpt_*`` (docs/superpowers/specs/2026-09-09-cpt-canonical-schema-design.md
|
||||
§3) hold one CPT codebook edition's own organizing structure —
|
||||
``cpt_section``/``cpt_code``/``cpt_instruction``/``cpt_reference``/
|
||||
``cpt_crosswalk``/``cpt_list``/``cpt_code_alt`` — written by
|
||||
``pfs.cpt_load.ingest`` from ``pfs.cpt_epub.parse_epub``.
|
||||
``cpt_section.path_key`` is ``" > ".join(path)`` (C7): the same heading
|
||||
title can occur as several ``sec_id``s (2019's running-header repeats),
|
||||
so ``cpt_section`` keeps every row and consumers group by ``path_key``
|
||||
to treat those repeats as one heading. ``cpt_code_alt`` holds the
|
||||
losing entries when a code prints more than once with a real descriptor
|
||||
both times (C9 — the parser picks one canonical ``pfs.cpt_code`` row by
|
||||
structural score and keeps the rest here with a ``reason``, rather than
|
||||
either dropping or duplicating them).
|
||||
|
||||
Writers are delete-then-insert per code (events, elements) or per
|
||||
``edition_year`` (cpt_*), or full replace (families), so a re-run is
|
||||
idempotent. Callers own the connection (``conf.connect.duckdb_batch``
|
||||
for writes) and republish the replica afterwards; nothing here opens a
|
||||
database.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import astuple, dataclass
|
||||
from typing import Any, Sequence
|
||||
from typing import TYPE_CHECKING, Any, Sequence
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pfs.cpt_model import CptEdition
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -28,7 +46,7 @@ class ElementRow:
|
||||
item_key: str
|
||||
p_id: int
|
||||
page: int
|
||||
source: str # "fr" | "hcpcs" | "rvu"
|
||||
source: str # "fr" | "cpt" | "hcpcs" | "rvu"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -51,7 +69,7 @@ class EventRow:
|
||||
item_key: str
|
||||
p_id: int
|
||||
page: int
|
||||
source: str # "fr" | "rvu"
|
||||
source: str # "fr" | "rvu" | "cpt"
|
||||
anchored: bool
|
||||
note: str
|
||||
|
||||
@@ -66,6 +84,98 @@ class FamilyRow:
|
||||
until: int | None
|
||||
item_key: str
|
||||
p_id: int
|
||||
#: The code's own CPT heading path (``" > ".join(path)``, C7) — the
|
||||
#: heading that actually classified this code, which may differ from
|
||||
#: whichever heading named the family (a hand family or a slice-1
|
||||
#: edge can pull codes from more than one heading into one family).
|
||||
#: ``""`` for a code with no CPT heading (HCPCS, or a rare code alone
|
||||
#: under every ancestor heading). Last field, defaulted, so callers
|
||||
#: that build a ``FamilyRow`` without it keep working (Ruling C1).
|
||||
note: str = ""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CptSectionRow:
|
||||
edition_year: int
|
||||
item_key: str
|
||||
sec_id: str
|
||||
level: int
|
||||
title: str
|
||||
path: list[str]
|
||||
path_key: str # " > ".join(path) — see module docstring (C7)
|
||||
code_lo: str
|
||||
code_hi: str
|
||||
guideline: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CptCodeRow:
|
||||
edition_year: int
|
||||
item_key: str
|
||||
code: str
|
||||
sec_id: str
|
||||
category: str # "I" | "II" | "III"
|
||||
descriptor: str
|
||||
stem: str
|
||||
elements: list[str]
|
||||
tail: str
|
||||
parent: str
|
||||
addon: bool
|
||||
resequenced: bool
|
||||
new: bool
|
||||
revised: bool
|
||||
telemedicine: bool
|
||||
mod51_exempt: bool
|
||||
audio_only: bool
|
||||
fda_pending: bool
|
||||
pla: bool
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CptInstructionRow:
|
||||
edition_year: int
|
||||
item_key: str
|
||||
code: str
|
||||
kind: str
|
||||
text: str
|
||||
targets: list[str]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CptReferenceRow:
|
||||
edition_year: int
|
||||
item_key: str
|
||||
code: str
|
||||
kind: str # "cpt-changes" | "cpt-assistant"
|
||||
years: list[int]
|
||||
text: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CptCrosswalkRow:
|
||||
edition_year: int
|
||||
item_key: str
|
||||
current_code: str
|
||||
former_code: str
|
||||
year_deleted: str
|
||||
citations: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CptListRow:
|
||||
edition_year: int
|
||||
item_key: str
|
||||
appendix: str
|
||||
code: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CptCodeAltRow:
|
||||
edition_year: int
|
||||
item_key: str
|
||||
code: str
|
||||
sec_id: str
|
||||
reason: str # "guidelines-reprint" | "lower-score" (C9)
|
||||
|
||||
|
||||
_DDL = """
|
||||
@@ -81,14 +191,52 @@ CREATE TABLE IF NOT EXISTS pfs.code_event (
|
||||
item_key VARCHAR, p_id INTEGER, page INTEGER, source VARCHAR, anchored BOOLEAN, note VARCHAR);
|
||||
CREATE TABLE IF NOT EXISTS pfs.code_family (
|
||||
key VARCHAR, name VARCHAR, code VARCHAR, role VARCHAR, since INTEGER, until INTEGER,
|
||||
item_key VARCHAR, p_id INTEGER);
|
||||
item_key VARCHAR, p_id INTEGER, note VARCHAR);
|
||||
CREATE TABLE IF NOT EXISTS pfs.cpt_section (
|
||||
edition_year INTEGER, item_key VARCHAR, sec_id VARCHAR, level INTEGER, title VARCHAR,
|
||||
path VARCHAR[], path_key VARCHAR, code_lo VARCHAR, code_hi VARCHAR, guideline VARCHAR);
|
||||
CREATE TABLE IF NOT EXISTS pfs.cpt_code (
|
||||
edition_year INTEGER, item_key VARCHAR, code VARCHAR, sec_id VARCHAR, category VARCHAR,
|
||||
descriptor VARCHAR, stem VARCHAR, elements VARCHAR[], tail VARCHAR, parent VARCHAR,
|
||||
addon BOOLEAN, resequenced BOOLEAN, new BOOLEAN, revised BOOLEAN, telemedicine BOOLEAN,
|
||||
mod51_exempt BOOLEAN, audio_only BOOLEAN, fda_pending BOOLEAN, pla BOOLEAN);
|
||||
CREATE TABLE IF NOT EXISTS pfs.cpt_instruction (
|
||||
edition_year INTEGER, item_key VARCHAR, code VARCHAR, kind VARCHAR, text VARCHAR,
|
||||
targets VARCHAR[]);
|
||||
CREATE TABLE IF NOT EXISTS pfs.cpt_reference (
|
||||
edition_year INTEGER, item_key VARCHAR, code VARCHAR, kind VARCHAR, years INTEGER[],
|
||||
text VARCHAR);
|
||||
CREATE TABLE IF NOT EXISTS pfs.cpt_crosswalk (
|
||||
edition_year INTEGER, item_key VARCHAR, current_code VARCHAR, former_code VARCHAR,
|
||||
year_deleted VARCHAR, citations VARCHAR);
|
||||
CREATE TABLE IF NOT EXISTS pfs.cpt_list (
|
||||
edition_year INTEGER, item_key VARCHAR, appendix VARCHAR, code VARCHAR);
|
||||
CREATE TABLE IF NOT EXISTS pfs.cpt_code_alt (
|
||||
edition_year INTEGER, item_key VARCHAR, code VARCHAR, sec_id VARCHAR, reason VARCHAR);
|
||||
"""
|
||||
|
||||
|
||||
def is_missing_table_error(exc: Exception) -> bool:
|
||||
"""True for a DuckDB "table/schema does not exist" Catalog error — a
|
||||
replica that hasn't had ``cpt-ingest``/``elements``/``lineage
|
||||
--write``/``families --write`` run yet is missing
|
||||
``pfs.cpt_*``/``pfs.code_element``/``pfs.code_event``/
|
||||
``pfs.code_family``, which is not a bug, just nothing derived or
|
||||
ingested yet (I4). Shared by ``src/cli/pfs.py``, ``pfs.lineage`` and
|
||||
``pfs.extract`` so every read-only CPT-table lookup treats a missing
|
||||
table the same way."""
|
||||
msg = str(exc)
|
||||
return "Catalog" in msg and "does not exist" in msg
|
||||
|
||||
|
||||
def ensure_tables(con: Any) -> None:
|
||||
for stmt in _DDL.strip().split(";"):
|
||||
if stmt.strip():
|
||||
con.execute(stmt)
|
||||
# Ruling C1: a pre-existing pfs.code_family table (created before
|
||||
# `note` existed) needs the column added — CREATE TABLE IF NOT EXISTS
|
||||
# above is a no-op once the table already exists.
|
||||
con.execute("ALTER TABLE pfs.code_family ADD COLUMN IF NOT EXISTS note VARCHAR")
|
||||
|
||||
|
||||
def _insert(con: Any, table: str, rows: Sequence[Any]) -> int:
|
||||
@@ -171,3 +319,190 @@ def read_families(con: Any) -> list[FamilyRow]:
|
||||
"SELECT * FROM pfs.code_family ORDER BY key, role, code"
|
||||
).fetchall()
|
||||
return [FamilyRow(*r) for r in rows]
|
||||
|
||||
|
||||
_CPT_TABLES = (
|
||||
"pfs.cpt_section",
|
||||
"pfs.cpt_code",
|
||||
"pfs.cpt_instruction",
|
||||
"pfs.cpt_reference",
|
||||
"pfs.cpt_crosswalk",
|
||||
"pfs.cpt_list",
|
||||
"pfs.cpt_code_alt",
|
||||
)
|
||||
|
||||
|
||||
def write_cpt_edition(
|
||||
con: Any, edition: "CptEdition", item_key: str, *, force: bool = False
|
||||
) -> dict[str, int]:
|
||||
"""Delete-then-insert one CPT codebook edition's parsed rows into
|
||||
``pfs.cpt_*``, keyed by ``edition.year``. Returns rows written per
|
||||
table. C8: asserts one ``pfs.cpt_code`` row per (edition_year, code)
|
||||
before inserting — a duplicate means a resequenced placeholder row
|
||||
slipped past the parser's own filter (``pfs.cpt_epub``). C9's
|
||||
guideline-reprint/lower-score losers are resolved by the parser
|
||||
before this ever runs (``edition.alternates``, written to
|
||||
``pfs.cpt_code_alt``), so that assertion should now always pass.
|
||||
|
||||
F4: refuses (raises ``ValueError``) an edition with zero codes
|
||||
unless ``force=True`` — this is delete-then-insert, so writing an
|
||||
empty parse would silently wipe whatever real edition data is
|
||||
already on file for that year. ``pfs.cpt_epub.parse_epub`` already
|
||||
raises before returning an edition this empty in the ordinary case
|
||||
(no chapter content found at all); this is the second guard, for an
|
||||
edition that parsed some structure but named not one actual code."""
|
||||
if not edition.codes and not force:
|
||||
raise ValueError(
|
||||
f"pfs.cpt_code: edition_year={edition.year} parsed zero codes — "
|
||||
"refusing to overwrite any existing edition data; pass force=True "
|
||||
"to write it anyway"
|
||||
)
|
||||
ensure_tables(con)
|
||||
year = edition.year
|
||||
|
||||
seen: dict[str, int] = {}
|
||||
for c in edition.codes:
|
||||
seen[c.code] = seen.get(c.code, 0) + 1
|
||||
dupes = sorted(code for code, n in seen.items() if n > 1)
|
||||
if dupes:
|
||||
raise ValueError(
|
||||
f"pfs.cpt_code: duplicate rows for edition_year={year}, "
|
||||
f"codes: {', '.join(dupes)}"
|
||||
)
|
||||
|
||||
section_rows = [
|
||||
CptSectionRow(
|
||||
edition_year=year,
|
||||
item_key=item_key,
|
||||
sec_id=s.sec_id,
|
||||
level=s.level,
|
||||
title=s.title,
|
||||
path=list(s.path),
|
||||
path_key=" > ".join(s.path),
|
||||
code_lo=s.code_lo,
|
||||
code_hi=s.code_hi,
|
||||
guideline=s.guideline,
|
||||
)
|
||||
for s in edition.sections
|
||||
]
|
||||
code_rows = [
|
||||
CptCodeRow(
|
||||
edition_year=year,
|
||||
item_key=item_key,
|
||||
code=c.code,
|
||||
sec_id=c.sec_id,
|
||||
category=c.category,
|
||||
descriptor=c.descriptor,
|
||||
stem=c.stem,
|
||||
elements=list(c.elements),
|
||||
tail=c.tail,
|
||||
parent=c.parent,
|
||||
addon=c.addon,
|
||||
resequenced=c.resequenced,
|
||||
new=c.new,
|
||||
revised=c.revised,
|
||||
telemedicine=c.telemedicine,
|
||||
mod51_exempt=c.mod51_exempt,
|
||||
audio_only=c.audio_only,
|
||||
fda_pending=c.fda_pending,
|
||||
pla=c.pla,
|
||||
)
|
||||
for c in edition.codes
|
||||
]
|
||||
instruction_rows = [
|
||||
CptInstructionRow(
|
||||
edition_year=year,
|
||||
item_key=item_key,
|
||||
code=i.code,
|
||||
kind=i.kind,
|
||||
text=i.text,
|
||||
targets=list(i.targets),
|
||||
)
|
||||
for i in edition.instructions
|
||||
]
|
||||
reference_rows = [
|
||||
CptReferenceRow(
|
||||
edition_year=year,
|
||||
item_key=item_key,
|
||||
code=r.code,
|
||||
kind=r.kind,
|
||||
years=list(r.years),
|
||||
text=r.text,
|
||||
)
|
||||
for r in edition.references
|
||||
]
|
||||
crosswalk_rows = [
|
||||
CptCrosswalkRow(
|
||||
edition_year=year,
|
||||
item_key=item_key,
|
||||
current_code=x.current_code,
|
||||
former_code=x.former_code,
|
||||
year_deleted=x.year_deleted,
|
||||
citations=x.citations,
|
||||
)
|
||||
for x in edition.crosswalks
|
||||
]
|
||||
list_rows = [
|
||||
CptListRow(
|
||||
edition_year=year, item_key=item_key, appendix=e.appendix, code=e.code
|
||||
)
|
||||
for e in edition.lists
|
||||
]
|
||||
alt_rows = [
|
||||
CptCodeAltRow(
|
||||
edition_year=year,
|
||||
item_key=item_key,
|
||||
code=a.code,
|
||||
sec_id=a.sec_id,
|
||||
reason=a.reason,
|
||||
)
|
||||
for a in edition.alternates
|
||||
]
|
||||
|
||||
for table in _CPT_TABLES:
|
||||
con.execute(f"DELETE FROM {table} WHERE edition_year = ?", [year])
|
||||
|
||||
return {
|
||||
"sections": _insert(con, "pfs.cpt_section", section_rows),
|
||||
"codes": _insert(con, "pfs.cpt_code", code_rows),
|
||||
"instructions": _insert(con, "pfs.cpt_instruction", instruction_rows),
|
||||
"references": _insert(con, "pfs.cpt_reference", reference_rows),
|
||||
"crosswalks": _insert(con, "pfs.cpt_crosswalk", crosswalk_rows),
|
||||
"lists": _insert(con, "pfs.cpt_list", list_rows),
|
||||
"alternates": _insert(con, "pfs.cpt_code_alt", alt_rows),
|
||||
}
|
||||
|
||||
|
||||
def read_cpt_sections(con: Any, year: int) -> list[CptSectionRow]:
|
||||
rows = con.execute(
|
||||
"SELECT * FROM pfs.cpt_section WHERE edition_year = ? ORDER BY sec_id", [year]
|
||||
).fetchall()
|
||||
return [CptSectionRow(*r) for r in rows]
|
||||
|
||||
|
||||
def read_cpt_codes(con: Any, year: int) -> list[CptCodeRow]:
|
||||
rows = con.execute(
|
||||
"SELECT * FROM pfs.cpt_code WHERE edition_year = ? ORDER BY code", [year]
|
||||
).fetchall()
|
||||
return [CptCodeRow(*r) for r in rows]
|
||||
|
||||
|
||||
def read_cpt_instructions(con: Any, year: int) -> list[CptInstructionRow]:
|
||||
rows = con.execute(
|
||||
"SELECT * FROM pfs.cpt_instruction WHERE edition_year = ? ORDER BY code", [year]
|
||||
).fetchall()
|
||||
return [CptInstructionRow(*r) for r in rows]
|
||||
|
||||
|
||||
def read_cpt_references(con: Any, year: int) -> list[CptReferenceRow]:
|
||||
rows = con.execute(
|
||||
"SELECT * FROM pfs.cpt_reference WHERE edition_year = ? ORDER BY code", [year]
|
||||
).fetchall()
|
||||
return [CptReferenceRow(*r) for r in rows]
|
||||
|
||||
|
||||
def cpt_years(con: Any) -> list[int]:
|
||||
rows = con.execute(
|
||||
"SELECT DISTINCT edition_year FROM pfs.cpt_section ORDER BY edition_year"
|
||||
).fetchall()
|
||||
return [r[0] for r in rows]
|
||||
|
||||
1001
src/pfs/cpt_epub.py
Normal file
1001
src/pfs/cpt_epub.py
Normal file
File diff suppressed because it is too large
Load Diff
137
src/pfs/cpt_load.py
Normal file
137
src/pfs/cpt_load.py
Normal file
@@ -0,0 +1,137 @@
|
||||
"""CPT EPUB editions -> ``pfs.cpt_*`` (task 3, refs #687).
|
||||
|
||||
``find_editions`` locates the bib items tagged ``source:ama`` whose
|
||||
title names a CPT Professional edition year and that carry an EPUB
|
||||
attachment (docs/superpowers/specs/2026-09-09-cpt-canonical-schema-design.md
|
||||
§5, C2): 2019, 2021, 2022 and 2024 are on disk today; 2018 is PDF-only,
|
||||
the "CPT Changes" book and Netter's Atlas are different books entirely
|
||||
— none of the three match the title pattern or (for 2018) carry an
|
||||
EPUB, so they're skipped without a special case. The edition year comes
|
||||
from the item's own title, never the bib ``year:`` tag (that's Zotero's
|
||||
publication date, which runs a year behind the edition).
|
||||
|
||||
``ingest`` parses each matched edition (``pfs.cpt_epub.parse_epub``) and
|
||||
writes it (``pfs.codetables.write_cpt_edition``), delete-then-insert per
|
||||
``edition_year`` so a re-run is idempotent. ``dry_run=True`` only
|
||||
parses and counts — ``con`` is never touched, so the CLI never has to
|
||||
open the DuckDB single-writer lock for a preview (#508-#514).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Any, Sequence
|
||||
|
||||
from pfs.codetables import write_cpt_edition
|
||||
from pfs.cpt_epub import parse_epub
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
# "CPT Professional 2024", "CPT 2021 Professional Edition", "CPT 2019".
|
||||
# Deliberately does *not* match "CPT Changes 2023" (no "Professional"/
|
||||
# bare-year token follows "CPT " there) or "Netter's Atlas for CPT
|
||||
# Coding 2015" (doesn't start with "CPT").
|
||||
_TITLE_RE = re.compile(
|
||||
r"^CPT\s+(?:Professional\s+(\d{4})|(\d{4})\s+Professional(?:\s+Edition)?\b|(\d{4})\b)"
|
||||
)
|
||||
|
||||
|
||||
def find_editions(store: Any) -> list[tuple[int, str, Path]]:
|
||||
"""(year, item_key, epub path) for every ``source:ama``-tagged bib
|
||||
item whose title names a CPT Professional edition and carries an
|
||||
``.epub`` attachment, sorted by year. One entry per item (the first
|
||||
matching ``.epub`` attachment wins if an item somehow has more than
|
||||
one)."""
|
||||
con = store._con()
|
||||
rows = con.execute(
|
||||
"SELECT i.key AS item_key, i.title AS title, a.filename AS filename, "
|
||||
" a.storage_path AS storage_path "
|
||||
"FROM items i "
|
||||
"JOIN item_tags it ON it.item_id = i.id "
|
||||
"JOIN tags t ON t.id = it.tag_id "
|
||||
"JOIN attachments a ON a.item_id = i.id "
|
||||
"WHERE t.name = 'source:ama' "
|
||||
"ORDER BY i.key, a.id"
|
||||
).fetchall()
|
||||
|
||||
out: dict[str, tuple[int, str, Path]] = {}
|
||||
for row in rows:
|
||||
item_key = row["item_key"]
|
||||
if item_key in out:
|
||||
continue
|
||||
filename = row["filename"] or ""
|
||||
if not filename.lower().endswith(".epub"):
|
||||
continue
|
||||
title = row["title"] or ""
|
||||
m = _TITLE_RE.match(title)
|
||||
if not m:
|
||||
log.warning(
|
||||
"source:ama item %s carries an .epub but its title %r "
|
||||
"doesn't match a CPT Professional edition — skipping",
|
||||
item_key,
|
||||
title,
|
||||
)
|
||||
continue
|
||||
year = int(next(g for g in m.groups() if g))
|
||||
out[item_key] = (year, item_key, Path(row["storage_path"]))
|
||||
return sorted(out.values())
|
||||
|
||||
|
||||
def ingest(
|
||||
store: Any,
|
||||
con: Any,
|
||||
*,
|
||||
years: Sequence[int] | None = None,
|
||||
dry_run: bool = False,
|
||||
) -> dict[int, dict[str, Any]]:
|
||||
"""Parse and (unless ``dry_run``) write every matched CPT edition.
|
||||
|
||||
Returns ``{year: {table_short_name: row_count}}`` — parsed counts
|
||||
(sections/codes/instructions/references/crosswalks/lists) when
|
||||
``dry_run``, written counts (``write_cpt_edition``'s return value)
|
||||
otherwise. Raises ``ValueError`` if ``years`` names a year with no
|
||||
matching bib item/attachment.
|
||||
|
||||
F4: an edition that fails to parse (``parse_epub`` found no chapter
|
||||
content — a bad/empty EPUB) or fails to write (``write_cpt_edition``
|
||||
refusing a zero-code edition) is a per-edition failure, not a whole-
|
||||
run one — its entry is ``{"error": <message>}`` and every other
|
||||
requested edition is still parsed/written.
|
||||
"""
|
||||
editions = find_editions(store)
|
||||
by_year = {year: (item_key, path) for year, item_key, path in editions}
|
||||
|
||||
if years is None:
|
||||
wanted = sorted(by_year)
|
||||
else:
|
||||
wanted = sorted(set(years))
|
||||
missing = [y for y in wanted if y not in by_year]
|
||||
if missing:
|
||||
known = ", ".join(str(y) for y in sorted(by_year)) or "none"
|
||||
raise ValueError(
|
||||
f"no CPT EPUB found for edition(s) {missing} (known editions: {known})"
|
||||
)
|
||||
|
||||
out: dict[int, dict[str, Any]] = {}
|
||||
for year in wanted:
|
||||
item_key, path = by_year[year]
|
||||
try:
|
||||
edition = parse_epub(path, year=year)
|
||||
if dry_run:
|
||||
out[year] = {
|
||||
"sections": len(edition.sections),
|
||||
"codes": len(edition.codes),
|
||||
"instructions": len(edition.instructions),
|
||||
"references": len(edition.references),
|
||||
"crosswalks": len(edition.crosswalks),
|
||||
"lists": len(edition.lists),
|
||||
"alternates": len(edition.alternates),
|
||||
}
|
||||
else:
|
||||
out[year] = write_cpt_edition(con, edition, item_key)
|
||||
except ValueError as exc:
|
||||
log.error("CPT edition %s (%s) failed: %s", year, item_key, exc)
|
||||
out[year] = {"error": str(exc)}
|
||||
return out
|
||||
136
src/pfs/cpt_model.py
Normal file
136
src/pfs/cpt_model.py
Normal file
@@ -0,0 +1,136 @@
|
||||
"""Dataclasses for the CPT organizing-principles schema (docs/superpowers/
|
||||
specs/2026-09-09-cpt-canonical-schema-design.md §3). Pure — no DuckDB, no
|
||||
bib, no I/O. ``cpt_epub.parse_epub``/``parse_xhtml`` build these from the
|
||||
AMA EPUB markup; nothing here reads a file.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CptSection:
|
||||
"""One heading (``div.hN``) in the codebook's own hierarchy.
|
||||
|
||||
``path`` is the ancestor titles plus this heading's own title, root
|
||||
first — e.g. ``("Care Management Services", "Chronic Care Management
|
||||
Services")``. ``code_lo``/``code_hi`` come from the chapter TOC's
|
||||
print-order code span (``"Title* (LO-HI)"``); both are ``""`` when
|
||||
the TOC didn't carry a range for this heading (headings without a
|
||||
trailing ``*`` carry no guideline and often no range either).
|
||||
"""
|
||||
|
||||
sec_id: str
|
||||
level: int
|
||||
title: str
|
||||
path: tuple[str, ...]
|
||||
code_lo: str
|
||||
code_hi: str
|
||||
guideline: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CptCode:
|
||||
"""One code entry (a table row keyed by its ``table-para`` glyph
|
||||
cell). ``stem``/``elements`` are inherited from the nearest preceding
|
||||
non-child code when this entry is an indented child or an add-on
|
||||
(``parent`` non-empty) — CPT's own semicolon-continuation rule, which
|
||||
add-on rows use too even though their own sentence reads standalone.
|
||||
``tail`` is always this entry's own trailing prose (the non-
|
||||
parenthetical text after the stem/elements, e.g. a time clause), and
|
||||
``descriptor`` is the assembled reading: stem + elements + tail.
|
||||
"""
|
||||
|
||||
code: str
|
||||
sec_id: str
|
||||
descriptor: str
|
||||
stem: str
|
||||
elements: tuple[str, ...]
|
||||
tail: str
|
||||
addon: bool
|
||||
resequenced: bool
|
||||
new: bool
|
||||
revised: bool
|
||||
telemedicine: bool
|
||||
parent: str
|
||||
mod51_exempt: bool
|
||||
audio_only: bool
|
||||
fda_pending: bool
|
||||
pla: bool
|
||||
category: str # "I" | "II" | "III" — from the code's own shape
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CptInstruction:
|
||||
"""One parenthetical note (``div.table-para2`` — or, in editions that
|
||||
don't use that class, any trailing prose block — beginning with
|
||||
``"("``). ``targets`` are the codes it names, ranges expanded (capped
|
||||
at 20 codes per range so a malformed range can't blow up)."""
|
||||
|
||||
code: str
|
||||
kind: str # "use-with" | "not-with" | "not-with-time" | "see" | "other"
|
||||
text: str
|
||||
targets: tuple[str, ...]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CptReference:
|
||||
"""One ``div.table-RT*`` citation line under a code."""
|
||||
|
||||
code: str
|
||||
kind: str # "cpt-changes" | "cpt-assistant"
|
||||
years: tuple[int, ...]
|
||||
text: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CptCrosswalk:
|
||||
"""One row of Appendix M — a deleted/renumbered code's citations
|
||||
crosswalked to its current code."""
|
||||
|
||||
current_code: str
|
||||
former_code: str
|
||||
year_deleted: str
|
||||
citations: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CptListEntry:
|
||||
"""One code in one authoritative appendix list (D add-on codes, E
|
||||
modifier-51 exempt, F modifier-63 exempt, G moderate sedation
|
||||
included, K FDA-pending, N resequenced, P audio-video telemedicine,
|
||||
T audio-only telemedicine)."""
|
||||
|
||||
appendix: str
|
||||
code: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CptAlternate:
|
||||
"""A losing duplicate ``CptCode`` entry for a code the book prints
|
||||
more than once (C9) — a guideline "Unlisted Service or Procedure"
|
||||
summary table's copy, a "Qualifying Circumstances for Anesthesia"
|
||||
cross-reference reprint, etc. — structurally distinct from C8's
|
||||
resequenced-code placeholder (which the parser drops entirely, never
|
||||
reaching this stage): here *both* entries carry a real descriptor,
|
||||
so the loser is kept, not discarded, as a record of what didn't
|
||||
become the canonical ``CptCode`` row and why."""
|
||||
|
||||
code: str
|
||||
sec_id: str
|
||||
reason: str # "guidelines-reprint" | "lower-score"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CptEdition:
|
||||
"""Everything parsed from one CPT codebook edition."""
|
||||
|
||||
year: int
|
||||
sections: tuple[CptSection, ...]
|
||||
codes: tuple[CptCode, ...]
|
||||
instructions: tuple[CptInstruction, ...]
|
||||
references: tuple[CptReference, ...]
|
||||
crosswalks: tuple[CptCrosswalk, ...]
|
||||
lists: tuple[CptListEntry, ...]
|
||||
alternates: tuple[CptAlternate, ...] = ()
|
||||
@@ -4,7 +4,11 @@ model second (closed vocabulary only), a review queue for the rest.
|
||||
``extract_run`` works on an FR descriptor run (stem + element
|
||||
paragraphs); ``extract_text`` on a flat descriptor (HCPCS long
|
||||
description, RVU short description); ``extract_code`` gathers every
|
||||
source for a code. No I/O here beyond what the caller hands in.
|
||||
source for a code — FR, then the newest CPT codebook edition's own
|
||||
``pfs.cpt_code`` row (stem + tail, then each required-elements list
|
||||
item), then HCPCS, then RVU — with FR rows winning over CPT and CPT
|
||||
winning over HCPCS/RVU on a duplicate ``(type, value, detail)``. No I/O
|
||||
here beyond what the caller hands in.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -12,7 +16,7 @@ from __future__ import annotations
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Callable, Sequence
|
||||
|
||||
from pfs.codetables import ElementRow, ReviewRow
|
||||
from pfs.codetables import ElementRow, ReviewRow, is_missing_table_error
|
||||
from pfs.descriptors import (
|
||||
DescriptorRun,
|
||||
Para,
|
||||
@@ -88,11 +92,73 @@ def extract_text(code: str, text: str, *, year: int, source: str) -> Extraction:
|
||||
return Extraction(code, tuple(rows), ())
|
||||
|
||||
|
||||
def _cpt_row(
|
||||
code: str, edition_year: int, item_key: str, e: Element, text: str
|
||||
) -> ElementRow:
|
||||
return ElementRow(
|
||||
code, edition_year, e.type.value, e.value, e.detail, text, item_key, 0, 0, "cpt"
|
||||
)
|
||||
|
||||
|
||||
def _cpt_elements(
|
||||
con: Any, code: str, *, classify: Classifier | None = None
|
||||
) -> tuple[list[ElementRow], list[ReviewRow]]:
|
||||
"""The newest CPT codebook edition's own row for *code*: one pass of
|
||||
``parse_descriptor`` over ``stem + " " + tail`` (anchored to the
|
||||
stem), then one pass per required-elements list item (each item's
|
||||
trailing ``,``/``;`` stripped first) — an item nothing can place goes
|
||||
to *classify* if given, else to the review queue. The ``parent``
|
||||
code's stem/elements already apply to add-on rows (the CPT parser
|
||||
inherits them onto the child's own ``pfs.cpt_code`` row), so nothing
|
||||
extra is done here for add-ons. A replica with no ``pfs.cpt_code``
|
||||
table yet (CPT ingestion hasn't run) is not a bug — it just yields no
|
||||
CPT rows/reviews (I4, shared with the CLI's read-only paths via
|
||||
``is_missing_table_error``)."""
|
||||
try:
|
||||
row = con.execute(
|
||||
"SELECT edition_year, item_key, stem, elements, tail FROM pfs.cpt_code "
|
||||
"WHERE code = ? ORDER BY edition_year DESC LIMIT 1",
|
||||
[code],
|
||||
).fetchone()
|
||||
except Exception as exc:
|
||||
if is_missing_table_error(exc):
|
||||
return [], []
|
||||
raise
|
||||
if row is None:
|
||||
return [], []
|
||||
edition_year, item_key, stem, elements, tail = row
|
||||
stem_text = (stem or "").strip()
|
||||
tail_text = (tail or "").strip()
|
||||
rows: list[ElementRow] = []
|
||||
for e in parse_descriptor(f"{stem_text} {tail_text}".strip()):
|
||||
rows.append(_cpt_row(code, edition_year, item_key, e, stem_text))
|
||||
reviews: list[ReviewRow] = []
|
||||
for raw in elements or ():
|
||||
text = raw.strip()
|
||||
if text.endswith((",", ";")):
|
||||
text = text[:-1].strip()
|
||||
if not text:
|
||||
continue
|
||||
found = parse_descriptor(text)
|
||||
if found:
|
||||
for e in found:
|
||||
rows.append(_cpt_row(code, edition_year, item_key, e, text))
|
||||
continue
|
||||
choice = classify(text, _CHOICES) if classify else None
|
||||
if choice is not None and choice in _TYPE_OF:
|
||||
e = Element(_TYPE_OF[choice], choice)
|
||||
rows.append(_cpt_row(code, edition_year, item_key, e, text))
|
||||
else:
|
||||
reviews.append(ReviewRow(code, text, "", "", item_key, 0))
|
||||
return rows, reviews
|
||||
|
||||
|
||||
def extract_code(
|
||||
store: Any, con: Any, code: str, *, classify: Classifier | None = None
|
||||
) -> Extraction:
|
||||
"""All sources for *code*; FR rows win over HCPCS/RVU rows for the same
|
||||
element, and each source contributes its own year."""
|
||||
"""All sources for *code*; FR rows win over CPT, and CPT wins over
|
||||
HCPCS/RVU rows for the same element — each source contributes its own
|
||||
year."""
|
||||
code = code.upper()
|
||||
merged: dict[tuple[str, str, str], ElementRow] = {}
|
||||
reviews: list[ReviewRow] = []
|
||||
@@ -101,6 +167,10 @@ def extract_code(
|
||||
for r in x.rows:
|
||||
merged.setdefault((r.type, r.value, r.detail), r)
|
||||
reviews.extend(x.reviews)
|
||||
cpt_rows, cpt_reviews = _cpt_elements(con, code, classify=classify)
|
||||
for r in cpt_rows:
|
||||
merged.setdefault((r.type, r.value, r.detail), r)
|
||||
reviews.extend(cpt_reviews)
|
||||
long_desc = hcpcs_long_description(con, code)
|
||||
years = rvu_descriptions(con, code)
|
||||
if long_desc:
|
||||
|
||||
@@ -188,7 +188,9 @@ def stem_tokens(description: str | None) -> frozenset[str]:
|
||||
# ``crosswalk`` events are dropped entirely — a rule's "codes ... are
|
||||
# crosswalked to X for valuation" sentence routinely lists a whole
|
||||
# family's worth of codes together and is evidence, not a merge signal.
|
||||
_LINK_RELATIONS = {"addon-of", "defined-by-reference-to"}
|
||||
# ``addon-of`` moved out to ``_cpt_edges`` (Ruling C11: it needs the same
|
||||
# cross-heading guard as ``use-with``/``parent``).
|
||||
_LINK_RELATIONS = {"defined-by-reference-to"}
|
||||
#: ``replaces``/``replaced_by`` events still count, but only when the
|
||||
#: relevant field names exactly one code — a multi-code list (a blanket
|
||||
#: "these codes are replaced by ..." sentence spanning several families,
|
||||
@@ -234,14 +236,128 @@ def _unique_key(key: str, rep: str, used_keys: set[str]) -> str:
|
||||
return key
|
||||
|
||||
|
||||
_SLUG_RE = re.compile(r"[^A-Za-z0-9]+")
|
||||
|
||||
|
||||
def _slug(text: str) -> str:
|
||||
"""UPPER-KEBAB slug of a CPT heading title (``Chronic Care Management
|
||||
Services`` -> ``CHRONIC-CARE-MANAGEMENT-SERVICES``)."""
|
||||
return _SLUG_RE.sub("-", text.strip()).strip("-").upper()
|
||||
|
||||
|
||||
def cpt_groups(
|
||||
cpt_codes: Sequence[Any], cpt_sections: Sequence[Any]
|
||||
) -> dict[str, tuple[str, str, str, tuple[str, ...]]]:
|
||||
"""For every code in *cpt_codes*, the lowest CPT heading (by path,
|
||||
walking up from the code's own section) that groups >= 2 codes.
|
||||
|
||||
Ruling C7: several ``sec_id``s can share a ``path_key`` (2019's
|
||||
running-header repeats), so grouping is by the heading's own path —
|
||||
not ``sec_id`` — from the start: codes are bucketed by their
|
||||
section's ``path`` tuple, and two sections with the same path pool
|
||||
their codes into one heading automatically.
|
||||
|
||||
Returns ``code -> (key, title, path_key, group_codes)``. A code that
|
||||
is alone under its own heading *and* every ancestor heading (rare) is
|
||||
omitted — it keeps its own stem-derived/singleton family instead.
|
||||
``key`` is the UPPER-KEBAB slug of the heading title, disambiguated
|
||||
on a collision between distinct headings sharing a title by
|
||||
appending ``@`` + the parent heading's title slug (deterministic:
|
||||
headings are processed in sorted path order; the first occupant of a
|
||||
slug keeps the bare key).
|
||||
|
||||
Built as one pass over every code's path prefixes (a trie-style
|
||||
rollup), not a per-code rescan of all codes — 12k codes x <= 5 path
|
||||
levels is a single ~60k-entry pass.
|
||||
"""
|
||||
path_by_sec: dict[str, tuple[str, ...]] = {
|
||||
s.sec_id: tuple(s.path) for s in cpt_sections
|
||||
}
|
||||
code_path: dict[str, tuple[str, ...]] = {}
|
||||
for c in cpt_codes:
|
||||
path = path_by_sec.get(c.sec_id)
|
||||
if path:
|
||||
code_path[c.code] = path
|
||||
|
||||
prefix_codes: dict[tuple[str, ...], list[str]] = {}
|
||||
for code, path in code_path.items():
|
||||
for i in range(len(path), 0, -1):
|
||||
prefix_codes.setdefault(path[:i], []).append(code)
|
||||
|
||||
# The *qualifying test* at each level is the full rollup (does this
|
||||
# ancestor have >= 2 codes anywhere beneath it — that's what "walk up
|
||||
# to the parent" means); but the *membership* of the winning group
|
||||
# must be only the codes that actually settle there — codes that
|
||||
# resolve to a deeper, more specific heading (e.g. "Fine Needle
|
||||
# Aspiration (FNA) Biopsy" under "Surgery") are not members of every
|
||||
# ancestor they happen to roll up through, or one code walking up to
|
||||
# a broad top-level chapter (because its own leaf was a singleton)
|
||||
# would drag the chapter's thousands of already-classified codes
|
||||
# along with it.
|
||||
win_path_for: dict[str, tuple[str, ...]] = {}
|
||||
for code, path in code_path.items():
|
||||
for i in range(len(path), 0, -1):
|
||||
prefix = path[:i]
|
||||
if len(prefix_codes[prefix]) >= 2:
|
||||
win_path_for[code] = prefix
|
||||
break
|
||||
|
||||
settled: dict[tuple[str, ...], list[str]] = {}
|
||||
for code, path in win_path_for.items():
|
||||
settled.setdefault(path, []).append(code)
|
||||
|
||||
used_keys: set[str] = set()
|
||||
key_for_path: dict[tuple[str, ...], str] = {}
|
||||
for path in sorted(settled):
|
||||
title = path[-1]
|
||||
base = _slug(title)
|
||||
if base not in used_keys:
|
||||
key = base
|
||||
else:
|
||||
parent = path[-2] if len(path) >= 2 else ""
|
||||
key = f"{base}@{_slug(parent)}" if parent else base
|
||||
n = 2
|
||||
while key in used_keys:
|
||||
key = f"{base}@{_slug(parent)}-{n}" if parent else f"{base}-{n}"
|
||||
n += 1
|
||||
used_keys.add(key)
|
||||
key_for_path[path] = key
|
||||
|
||||
out: dict[str, tuple[str, str, str, tuple[str, ...]]] = {}
|
||||
for path, members in settled.items():
|
||||
codes = tuple(sorted(members))
|
||||
key, title = key_for_path[path], path[-1]
|
||||
note = " > ".join(path)
|
||||
for code in members:
|
||||
out[code] = (key, title, note, codes)
|
||||
return out
|
||||
|
||||
|
||||
def _group_rows(
|
||||
key: str,
|
||||
name: str,
|
||||
members: Sequence[str],
|
||||
elements: Mapping[str, Sequence[Any]],
|
||||
events: Mapping[str, Sequence[Any]],
|
||||
groups_by_code: Mapping[str, tuple[str, str, str, tuple[str, ...]]],
|
||||
cpt_code_rows: Mapping[str, Any],
|
||||
cpt_since: Mapping[str, int],
|
||||
cpt_last_year: Mapping[str, int],
|
||||
newest_cpt_year: int | None,
|
||||
) -> list[Any]:
|
||||
"""``FamilyRow`` per code in *members*, labeled ``key``/``name``."""
|
||||
"""``FamilyRow`` per code in *members*, labeled ``key``/``name``.
|
||||
|
||||
``note`` is the code's own CPT heading path (``groups_by_code``), not
|
||||
the family's — a hand family or a slice-1 edge can pull codes from
|
||||
more than one CPT heading into one component, and the heading that
|
||||
actually classified *this* code is more informative than whichever
|
||||
heading happened to win the family's name.
|
||||
|
||||
``since``/``until`` come from ``pfs.cpt_code`` (the book's own record
|
||||
of a code's active years) when the code is a CPT code; otherwise they
|
||||
fall back to the FR/RVU-derived lineage events, unchanged from
|
||||
slice-1.
|
||||
"""
|
||||
from pfs.codetables import FamilyRow
|
||||
|
||||
out = []
|
||||
@@ -249,21 +365,87 @@ def _group_rows(
|
||||
els = elements.get(c, ())
|
||||
evs = events.get(c, ())
|
||||
kinds = {ev.kind for ev in evs}
|
||||
cpt_row = cpt_code_rows.get(c)
|
||||
if "replaced_by" in kinds:
|
||||
role = "predecessor"
|
||||
elif any(e.type == "relation" and e.value == "addon-of" for e in els):
|
||||
elif (cpt_row is not None and cpt_row.addon) or any(
|
||||
e.type == "relation" and e.value == "addon-of" for e in els
|
||||
):
|
||||
role = "add-on"
|
||||
elif "replaces" in kinds:
|
||||
role = "successor"
|
||||
else:
|
||||
role = "base"
|
||||
since = next((ev.year for ev in evs if ev.kind == "appeared"), None)
|
||||
until = next((ev.year for ev in evs if ev.kind == "disappeared"), None)
|
||||
if c in cpt_since:
|
||||
appeared = next((ev.year for ev in evs if ev.kind == "appeared"), None)
|
||||
# C13: cpt_since is only the earliest edition *on hand* — it
|
||||
# is not evidence the code didn't exist earlier. When an
|
||||
# RVU/FR "appeared" event is older, that's the real origin
|
||||
# (e.g. 99490 created CY2015, first ingested CPT edition
|
||||
# 2019); when no such event exists, cpt_since is the only
|
||||
# evidence there is.
|
||||
since = (
|
||||
min(cpt_since[c], appeared) if appeared is not None else cpt_since[c]
|
||||
)
|
||||
until = (
|
||||
None
|
||||
if newest_cpt_year is not None and cpt_last_year[c] >= newest_cpt_year
|
||||
else cpt_last_year[c] + 1
|
||||
)
|
||||
else:
|
||||
since = next((ev.year for ev in evs if ev.kind == "appeared"), None)
|
||||
until = next((ev.year for ev in evs if ev.kind == "disappeared"), None)
|
||||
anchor = next(((ev.item_key, ev.p_id) for ev in evs if ev.item_key), ("", 0))
|
||||
out.append(FamilyRow(key, name, c, role, since, until, anchor[0], anchor[1]))
|
||||
note = groups_by_code[c][2] if c in groups_by_code else ""
|
||||
out.append(
|
||||
FamilyRow(key, name, c, role, since, until, anchor[0], anchor[1], note)
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def _cpt_edges(
|
||||
union: Any,
|
||||
groups_by_code: Mapping[str, tuple[str, str, str, tuple[str, ...]]],
|
||||
cpt_instructions: Sequence[Any],
|
||||
cpt_code_rows: Mapping[str, Any],
|
||||
elements: Mapping[str, Sequence[Any]],
|
||||
) -> None:
|
||||
"""Wire the three CPT-sourced intra-family edges — a single-target
|
||||
``use-with`` instruction, ``pfs.cpt_code.parent`` (the semicolon
|
||||
rule), and an ``addon-of`` relation element — each subject to
|
||||
Ruling C11: when *both* endpoints already resolve to a CPT heading
|
||||
(``cpt_groups``) and those headings differ, the edge crosses
|
||||
families by design and must not be unioned (#687 live-corpus finds:
|
||||
``0133U``'s ``use-with`` note reaching into Tier 1 Molecular
|
||||
Pathology blended it with Proprietary Laboratory Analyses; a
|
||||
``33224 -> 0695T`` addon-of edge blended a pacemaker heading with a
|
||||
Category III one). An edge where at least one endpoint has no
|
||||
heading at all (a HCPCS code, or a CPT code alone at every ancestor)
|
||||
still joins as before — only a *headed-to-differently-headed* edge
|
||||
is skipped.
|
||||
"""
|
||||
|
||||
def bridges_distinct_headings(a: str, b: str) -> bool:
|
||||
ia = groups_by_code.get(a)
|
||||
ib = groups_by_code.get(b)
|
||||
return ia is not None and ib is not None and ia[2] != ib[2]
|
||||
|
||||
def guarded_union(a: str, b: str) -> None:
|
||||
if not bridges_distinct_headings(a, b):
|
||||
union(a, b)
|
||||
|
||||
for instr in cpt_instructions:
|
||||
if instr.kind == "use-with" and len(instr.targets) == 1:
|
||||
guarded_union(instr.code, instr.targets[0].upper())
|
||||
for code, row in cpt_code_rows.items():
|
||||
if row.parent:
|
||||
guarded_union(code, row.parent.upper())
|
||||
for code, els in elements.items():
|
||||
for e in els:
|
||||
if e.type == "relation" and e.value == "addon-of" and e.detail:
|
||||
guarded_union(code, e.detail.upper())
|
||||
|
||||
|
||||
def _connected(nodes: Sequence[str], adj: Mapping[str, set[str]]) -> list[list[str]]:
|
||||
"""Connected components of *nodes* using only edges between members of
|
||||
*nodes* itself (an edge to a node outside the set doesn't count)."""
|
||||
@@ -291,14 +473,22 @@ def derive_families(
|
||||
elements: Mapping[str, Sequence[Any]],
|
||||
events: Mapping[str, Sequence[Any]],
|
||||
descriptions: Mapping[str, str],
|
||||
*,
|
||||
cpt_codes: Sequence[Any] = (),
|
||||
cpt_sections: Sequence[Any] = (),
|
||||
cpt_instructions: Sequence[Any] = (),
|
||||
cpt_presence: Mapping[str, tuple[int, ...]] | None = None,
|
||||
) -> list[Any]:
|
||||
"""Connected components over codes linked by relation elements,
|
||||
"""Connected components over codes linked by the CPT manual's own
|
||||
hierarchy (``cpt_codes``/``cpt_sections`` — the primary organizing
|
||||
principle, see ``cpt_groups``), ``use-with`` instructions and
|
||||
``parent`` (the semicolon-inheritance rule), relation elements,
|
||||
lineage events, or a shared service stem + activity set.
|
||||
|
||||
Role precedence when a code qualifies for more than one:
|
||||
predecessor (has a ``replaced_by`` event) > add-on (has an
|
||||
``addon-of`` relation element) > successor (has a ``replaces``
|
||||
event) > base.
|
||||
``addon-of`` relation element or ``pfs.cpt_code.addon``) > successor
|
||||
(has a ``replaces`` event) > base.
|
||||
|
||||
A component that reaches into two or more hand families (#687: the
|
||||
real corpus's blanket multi-code lineage sentences used to merge ACP,
|
||||
@@ -308,14 +498,62 @@ def derive_families(
|
||||
order, and every code the BFS reaches joins that hand family. Codes no
|
||||
BFS reaches keep their own stem-keyed family, same as an ungrouped
|
||||
component.
|
||||
|
||||
HCPCS codes (not in ``cpt_codes``) keep the slice-1 derivation
|
||||
unchanged — they only join a CPT-headed family through one of the
|
||||
slice-1 edges (e.g. a ``replaced_by`` event into a CPT code).
|
||||
|
||||
``cpt_codes``/``cpt_sections``/``cpt_instructions`` stay scoped to
|
||||
one (the newest) edition — grouping and naming are newest-edition
|
||||
concepts. ``cpt_presence`` (Ruling C12) is separate: a ``code ->
|
||||
sorted edition years present`` map spanning *every* ingested
|
||||
edition, used only for ``since``/``until`` — a code the newest
|
||||
edition dropped still gets its true earliest year and a real
|
||||
``until``, rather than being clamped to the one edition
|
||||
``cpt_codes`` carries. When omitted, ``since``/``until`` fall back
|
||||
to ``cpt_codes`` alone (single-edition — the pre-C12 behavior).
|
||||
"""
|
||||
cpt_code_rows: dict[str, Any] = {c.code: c for c in cpt_codes}
|
||||
groups_by_code = cpt_groups(cpt_codes, cpt_sections)
|
||||
|
||||
cpt_since: dict[str, int] = {}
|
||||
cpt_last_year: dict[str, int] = {}
|
||||
if cpt_presence:
|
||||
for code, years in cpt_presence.items():
|
||||
if not years:
|
||||
continue
|
||||
cpt_since[code] = min(years)
|
||||
cpt_last_year[code] = max(years)
|
||||
else:
|
||||
for c in cpt_codes:
|
||||
cpt_since[c.code] = min(
|
||||
cpt_since.get(c.code, c.edition_year), c.edition_year
|
||||
)
|
||||
cpt_last_year[c.code] = max(
|
||||
cpt_last_year.get(c.code, c.edition_year), c.edition_year
|
||||
)
|
||||
newest_cpt_year = max(cpt_last_year.values(), default=None)
|
||||
|
||||
# Reject members that aren't code-shaped before deriving anything —
|
||||
# a corpus parsing artifact (e.g. a bare '\x1a') must never become a
|
||||
# family of its own.
|
||||
# family of its own. `cpt_code_rows`/`cpt_presence` are exempt from
|
||||
# the CODE_RE shape check: those codes come from the structured
|
||||
# `pfs.cpt_code` table, not a free-text scan, and CODE_RE (5 digits
|
||||
# or a letter + 4 digits) doesn't cover Category II/III/PLA's
|
||||
# 4-digits-then-letter shape (0001U, 0001T, 0001F) — excluding them
|
||||
# would silently drop ~1,400 real 2024-edition codes from every
|
||||
# family (#687-style regression found live: those categories
|
||||
# vanished from the derivation entirely). `cpt_presence` also brings
|
||||
# in a code the newest edition dropped (Ruling C12) so it still gets
|
||||
# a family row with its true since/until instead of vanishing.
|
||||
codes = sorted(
|
||||
c
|
||||
for c in (set(elements) | set(events) | set(descriptions))
|
||||
if CODE_RE.fullmatch(c)
|
||||
set(cpt_code_rows)
|
||||
| set(cpt_presence or {})
|
||||
| {
|
||||
c
|
||||
for c in (set(elements) | set(events) | set(descriptions))
|
||||
if CODE_RE.fullmatch(c)
|
||||
}
|
||||
)
|
||||
parent = {c: c for c in codes}
|
||||
adj: dict[str, set[str]] = {c: set() for c in codes}
|
||||
@@ -332,6 +570,35 @@ def derive_families(
|
||||
adj[b].add(a)
|
||||
parent[find(a)] = find(b)
|
||||
|
||||
# CPT heading co-membership: every code the lowest qualifying heading
|
||||
# groups is a family-membership edge by itself — this is what lets a
|
||||
# heading with no extracted elements at all (no FR paragraph reprints
|
||||
# it) still become a family. One heading (by path) chained per pass,
|
||||
# not re-walked per member.
|
||||
seen_paths: set[str] = set()
|
||||
for info in groups_by_code.values():
|
||||
path_key = info[2]
|
||||
if path_key in seen_paths:
|
||||
continue
|
||||
seen_paths.add(path_key)
|
||||
members = [c for c in info[3] if c in parent]
|
||||
for a, b in zip(members, members[1:]):
|
||||
union(a, b)
|
||||
|
||||
# Same #687 discipline as _LINK_EVENTS below: a `use-with` note naming
|
||||
# one code is an unambiguous add-on -> primary edge; a note naming a
|
||||
# whole range ("Use 0690T in conjunction with 76536, 76604, ...
|
||||
# [21 codes]") is the same kind of blanket multi-code sentence that
|
||||
# bridges unrelated families and is evidence, not a merge signal
|
||||
# (live corpus: 637 use-with instructions, 325 name >1 target, one
|
||||
# names 228 — narrowing this is what keeps the CPT heading grouping
|
||||
# from cascading into a single book-spanning component). Ruling C11
|
||||
# adds a second guard on top, in _cpt_edges: an edge (use-with,
|
||||
# parent, or addon-of) between two codes that each already resolve
|
||||
# to a *different* CPT heading crosses families by design and is
|
||||
# skipped, even when single-target/unambiguous.
|
||||
_cpt_edges(union, groups_by_code, cpt_instructions, cpt_code_rows, elements)
|
||||
|
||||
for code, els in elements.items():
|
||||
for e in els:
|
||||
if e.type == "relation" and e.value in _LINK_RELATIONS and e.detail:
|
||||
@@ -383,8 +650,49 @@ def derive_families(
|
||||
)
|
||||
return until - since
|
||||
|
||||
def _rows(key: str, name: str, members: Sequence[str]) -> list[Any]:
|
||||
return _group_rows(
|
||||
key,
|
||||
name,
|
||||
members,
|
||||
elements,
|
||||
events,
|
||||
groups_by_code,
|
||||
cpt_code_rows,
|
||||
cpt_since,
|
||||
cpt_last_year,
|
||||
newest_cpt_year,
|
||||
)
|
||||
|
||||
def _cpt_name(members: Sequence[str]) -> tuple[str, str] | None:
|
||||
"""The CPT-derived ``(key, name)`` for *members*, by majority
|
||||
heading — a component can span more than one CPT heading once
|
||||
slice-1/``use-with``/``parent`` edges are layered on (the same
|
||||
bridging *derive_families* already guards for hand families), so
|
||||
the heading that classifies the most of this component's
|
||||
CPT-covered codes names the whole group; ties break on the
|
||||
heading's path (deterministic). ``None`` when no member has a
|
||||
CPT heading at all (an HCPCS-only or pre-CPT-book component)."""
|
||||
counts: dict[str, int] = {}
|
||||
info_by_path: dict[str, tuple[str, str, str, tuple[str, ...]]] = {}
|
||||
for c in members:
|
||||
info = groups_by_code.get(c)
|
||||
if info is None:
|
||||
continue
|
||||
path_key = info[2]
|
||||
counts[path_key] = counts.get(path_key, 0) + 1
|
||||
info_by_path[path_key] = info
|
||||
if not counts:
|
||||
return None
|
||||
win = min(counts, key=lambda pk: (-counts[pk], pk))
|
||||
key, title, _, _ = info_by_path[win]
|
||||
return key, title
|
||||
|
||||
rows: list[Any] = []
|
||||
used_keys: set[str] = set()
|
||||
# Seed with every CPT-derived key so a later stem-derived key
|
||||
# (_name_group/_unique_key, for a component with no CPT heading) can
|
||||
# never collide with one cpt_groups already handed out.
|
||||
used_keys: set[str] = {info[0] for info in groups_by_code.values()}
|
||||
for members in groups.values():
|
||||
member_set = set(members)
|
||||
hands = [k for k in HAND_FAMILIES if set(HAND_FAMILIES[k].codes) & member_set]
|
||||
@@ -413,22 +721,28 @@ def derive_families(
|
||||
for hk in hands:
|
||||
grp = [c for c, v in assigned.items() if v == hk]
|
||||
if grp:
|
||||
rows.extend(
|
||||
_group_rows(hk, HAND_FAMILIES[hk].name, grp, elements, events)
|
||||
)
|
||||
rows.extend(_rows(hk, HAND_FAMILIES[hk].name, grp))
|
||||
unreached = [c for c in members if c not in assigned]
|
||||
for sub in _connected(unreached, adj):
|
||||
key, name, rep = _name_group(sub, stems, descriptions, tenure)
|
||||
key = _unique_key(key, rep, used_keys)
|
||||
rows.extend(_group_rows(key, name, sub, elements, events))
|
||||
cpt_name = _cpt_name(sub)
|
||||
if cpt_name:
|
||||
key, name = cpt_name
|
||||
else:
|
||||
key, name, rep = _name_group(sub, stems, descriptions, tenure)
|
||||
key = _unique_key(key, rep, used_keys)
|
||||
rows.extend(_rows(key, name, sub))
|
||||
continue
|
||||
hand = hands[0] if hands else None
|
||||
if hand:
|
||||
key, name = hand, HAND_FAMILIES[hand].name
|
||||
else:
|
||||
key, name, rep = _name_group(members, stems, descriptions, tenure)
|
||||
key = _unique_key(key, rep, used_keys)
|
||||
rows.extend(_group_rows(key, name, members, elements, events))
|
||||
cpt_name = _cpt_name(members)
|
||||
if cpt_name:
|
||||
key, name = cpt_name
|
||||
else:
|
||||
key, name, rep = _name_group(members, stems, descriptions, tenure)
|
||||
key = _unique_key(key, rep, used_keys)
|
||||
rows.extend(_rows(key, name, members))
|
||||
return sorted(rows, key=lambda r: (r.key, r.code))
|
||||
|
||||
|
||||
@@ -447,11 +761,14 @@ def load_families(con: Any) -> dict[str, Family]:
|
||||
for r in rows:
|
||||
fam = out.get(r.key)
|
||||
codes = (*(fam.codes if fam else ()), r.code)
|
||||
syn = (
|
||||
HAND_FAMILIES[r.key].synonyms
|
||||
if r.key in HAND_FAMILIES
|
||||
else (r.key.lower(), r.name.lower())
|
||||
)
|
||||
# Ruling F3: a derived (non-hand) family gets no synonyms at all
|
||||
# — one/two-word CPT heading names ("Repair", "Excision") would
|
||||
# otherwise become chat triggers across ~1,334 derived families,
|
||||
# and scanning every family's synonyms against every chunk costs
|
||||
# 0.50s/call at ~8.5k families. Matching a derived family by name
|
||||
# is #699's job (a precompiled alternation), not this one. Hand
|
||||
# families keep their curated synonyms.
|
||||
syn = HAND_FAMILIES[r.key].synonyms if r.key in HAND_FAMILIES else ()
|
||||
out[r.key] = Family(r.key, r.name, tuple(sorted(set(codes))), syn)
|
||||
return out
|
||||
|
||||
|
||||
@@ -1,14 +1,22 @@
|
||||
"""Lineage events for a code: what happened, when, and where the record
|
||||
says so.
|
||||
|
||||
Two independent sources: the RVU files (``pfs.rvu`` base rows year over
|
||||
Three independent sources: the RVU files (``pfs.rvu`` base rows year over
|
||||
year — appeared / disappeared / status_change / descriptor_change /
|
||||
revalued) and the Federal Register (``fr_anchors`` paragraphs that name
|
||||
the code next to an event verb — created / adopted_cpt / replaces /
|
||||
replaced_by / deleted / crosswalk / bundled / telehealth_list).
|
||||
``lineage`` merges them and marks each RVU event ``anchored`` when an FR
|
||||
event for the same code lies within ±1 rule year; the rest are
|
||||
``unanchored`` for review, never silently accepted.
|
||||
revalued), the Federal Register (``fr_anchors`` paragraphs that name the
|
||||
code next to an event verb — created / adopted_cpt / replaces /
|
||||
replaced_by / deleted / crosswalk / bundled / telehealth_list), and the
|
||||
CPT codebook's own "CPT Changes" citations (``pfs.cpt_reference`` kind
|
||||
``cpt-changes`` on the newest ingested edition's row for the code — one
|
||||
``cpt_changed`` event per year the AMA itself revised the code, plus a
|
||||
``cpt_deleted`` event when a code an older edition carried is absent from
|
||||
the newest one). CPT Changes years are *effective* years — the AMA's own
|
||||
change year — while FR events are dated by the *mentioning* rule's
|
||||
payment year; the two can and do disagree, and surfacing that
|
||||
disagreement is exactly the cross-check this module exists for.
|
||||
``lineage`` merges all three and marks each RVU event ``anchored`` when
|
||||
an FR **or** CPT event for the same code lies within ±1 rule year; the
|
||||
rest are ``unanchored`` for review, never silently accepted.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -17,7 +25,7 @@ import dataclasses
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
from pfs.codetables import EventRow
|
||||
from pfs.codetables import EventRow, cpt_years, is_missing_table_error
|
||||
from pfs.descriptors import rule_year_of
|
||||
from pfs.families import FR_CITE_RE, find_codes
|
||||
|
||||
@@ -257,11 +265,67 @@ def fr_events(store: Any, code: str) -> list[EventRow]:
|
||||
return sorted(out, key=lambda e: (e.year, FR_KINDS.index(e.kind), e.p_id))
|
||||
|
||||
|
||||
def _cpt_ev(code: str, year: int, kind: str, item_key: str, note: str) -> EventRow:
|
||||
return EventRow(code, year, kind, "", "", item_key, 0, 0, "cpt", True, note)
|
||||
|
||||
|
||||
def cpt_events(con: Any, code: str) -> list[EventRow]:
|
||||
"""CPT codebook lineage for *code*: one ``cpt_changed`` event per year
|
||||
the newest ingested edition's ``cpt-changes`` reference row names
|
||||
(``cpt_present`` is deliberately not an event — too noisy), or a
|
||||
single ``cpt_deleted`` event when *code* is present in an older
|
||||
edition's ``pfs.cpt_code`` but absent from the newest one. A replica
|
||||
with no ``pfs.cpt_*`` tables yet (CPT ingestion hasn't run) is not a
|
||||
bug — it just yields no CPT events (I4, shared with the CLI's
|
||||
read-only paths via ``is_missing_table_error``)."""
|
||||
code = code.upper()
|
||||
try:
|
||||
years = cpt_years(con)
|
||||
if not years:
|
||||
return []
|
||||
newest = max(years)
|
||||
ref = con.execute(
|
||||
"SELECT item_key, years FROM pfs.cpt_reference "
|
||||
"WHERE edition_year = ? AND code = ? AND kind = 'cpt-changes'",
|
||||
[newest, code],
|
||||
).fetchone()
|
||||
if ref is not None:
|
||||
item_key, yrs = ref
|
||||
return [
|
||||
_cpt_ev(code, int(y), "cpt_changed", item_key, f"CPT Changes {int(y)}")
|
||||
for y in sorted({int(y) for y in (yrs or [])})
|
||||
]
|
||||
present = con.execute(
|
||||
"SELECT edition_year, item_key FROM pfs.cpt_code WHERE code = ? "
|
||||
"ORDER BY edition_year",
|
||||
[code],
|
||||
).fetchall()
|
||||
except Exception as exc:
|
||||
if is_missing_table_error(exc):
|
||||
return []
|
||||
raise
|
||||
if present and int(present[-1][0]) < newest:
|
||||
last_year, last_key = present[-1]
|
||||
return [
|
||||
_cpt_ev(
|
||||
code,
|
||||
int(last_year) + 1,
|
||||
"cpt_deleted",
|
||||
last_key,
|
||||
f"absent from CPT {newest}",
|
||||
)
|
||||
]
|
||||
return []
|
||||
|
||||
|
||||
def lineage(con: Any, store: Any, code: str) -> list[EventRow]:
|
||||
fr = fr_events(store, code)
|
||||
fr_years = {e.year for e in fr}
|
||||
cpt = cpt_events(con, code)
|
||||
anchor_years = {e.year for e in fr} | {e.year for e in cpt}
|
||||
rvu = [
|
||||
dataclasses.replace(ev, anchored=any(abs(ev.year - y) <= 1 for y in fr_years))
|
||||
dataclasses.replace(
|
||||
ev, anchored=any(abs(ev.year - y) <= 1 for y in anchor_years)
|
||||
)
|
||||
for ev in rvu_events(con, code)
|
||||
]
|
||||
return sorted([*rvu, *fr], key=lambda e: (e.year, e.source != "fr", e.kind))
|
||||
return sorted([*rvu, *fr, *cpt], key=lambda e: (e.year, e.source != "fr", e.kind))
|
||||
|
||||
130
src/rex/comments/epub_text.py
Normal file
130
src/rex/comments/epub_text.py
Normal file
@@ -0,0 +1,130 @@
|
||||
"""Shared EPUB container/spine resolution: reading order and plain text.
|
||||
|
||||
Every EPUB carries ``META-INF/container.xml``, which points at the
|
||||
package document (the "OPF") — the document that lists every content
|
||||
file (the manifest) and the order a reader walks them in (the spine).
|
||||
That's the only reliable way to know a book's own content-file prefix
|
||||
(``OPS/`` for the AMA's CPT books, something else for another
|
||||
publisher) and reading order — the zip's own member order is not
|
||||
necessarily book order (``pfs.cpt_epub`` hit this: 2021/2022/2024's
|
||||
namelist() interleaves chapters arbitrarily).
|
||||
|
||||
Two consumers share this: ``rex.comments.extract`` (F2 — the corpus
|
||||
indexer needs whole-book plain text out of a CPT EPUB attachment) and
|
||||
``pfs.cpt_epub`` (which used to hard-code the ``OPS/`` prefix three
|
||||
times; it now resolves the same way, so a differently-templated EPUB
|
||||
would still locate its content instead of silently finding nothing).
|
||||
|
||||
Stdlib only (``zipfile``, ``xml.etree``, ``re``, ``html``) — no bib, no
|
||||
DuckDB, no network.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import html
|
||||
import re
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
from xml.etree import ElementTree as ET
|
||||
|
||||
_CONTAINER_PATH = "META-INF/container.xml"
|
||||
_CONTAINER_NS = {"c": "urn:oasis:names:tc:opendocument:xmlns:container"}
|
||||
_OPF_NS = {"o": "http://www.idpf.org/2007/opf"}
|
||||
|
||||
_TAG_RE = re.compile(r"<[^>]+>")
|
||||
_BLOCK_RE = re.compile(r"<(?:p|div|br|li|h[1-6]|tr)[^>]*>", re.IGNORECASE)
|
||||
_SCRIPT_STYLE_RE = re.compile(
|
||||
r"<(script|style)[^>]*>.*?</\1>", re.IGNORECASE | re.DOTALL
|
||||
)
|
||||
_BODY_RE = re.compile(r"<body[^>]*>(.*)</body>", re.IGNORECASE | re.DOTALL)
|
||||
|
||||
|
||||
def _opf_info(zf: zipfile.ZipFile) -> tuple[str, str] | None:
|
||||
"""(content-root dir, OPF member path), resolved via
|
||||
``META-INF/container.xml`` -> the first ``<rootfile>``'s
|
||||
``full-path``. ``None`` when container.xml is absent or malformed."""
|
||||
try:
|
||||
data = zf.read(_CONTAINER_PATH)
|
||||
except KeyError:
|
||||
return None
|
||||
try:
|
||||
root = ET.fromstring(data)
|
||||
except ET.ParseError:
|
||||
return None
|
||||
rootfile = root.find(".//c:rootfile", _CONTAINER_NS)
|
||||
opf_path = rootfile.get("full-path") if rootfile is not None else None
|
||||
if not opf_path:
|
||||
return None
|
||||
opf_dir = opf_path.rsplit("/", 1)[0] + "/" if "/" in opf_path else ""
|
||||
return opf_dir, opf_path
|
||||
|
||||
|
||||
def content_root(zf: zipfile.ZipFile) -> str:
|
||||
"""The directory prefix (``"OPS/"``, ``""``, ...) holding this
|
||||
book's content files. Falls back to ``"OPS/"`` — the AMA books' own
|
||||
layout, and what every caller hard-coded before this existed — when
|
||||
container.xml can't be resolved."""
|
||||
info = _opf_info(zf)
|
||||
return info[0] if info is not None else "OPS/"
|
||||
|
||||
|
||||
def spine_paths(zf: zipfile.ZipFile) -> list[str] | None:
|
||||
"""Full zip-member paths in the book's spine reading order,
|
||||
resolved via container.xml -> the OPF's manifest + spine. ``None``
|
||||
when the OPF can't be located or parsed — callers fall back to
|
||||
sorted content-file names."""
|
||||
info = _opf_info(zf)
|
||||
if info is None:
|
||||
return None
|
||||
opf_dir, opf_path = info
|
||||
try:
|
||||
opf = ET.fromstring(zf.read(opf_path))
|
||||
except (KeyError, ET.ParseError):
|
||||
return None
|
||||
manifest = {
|
||||
item.get("id"): item.get("href")
|
||||
for item in opf.findall(".//o:manifest/o:item", _OPF_NS)
|
||||
}
|
||||
paths = []
|
||||
for itemref in opf.findall(".//o:spine/o:itemref", _OPF_NS):
|
||||
href = manifest.get(itemref.get("idref"))
|
||||
if href:
|
||||
paths.append(opf_dir + href)
|
||||
return paths or None
|
||||
|
||||
|
||||
def _to_text(raw: str) -> str:
|
||||
"""One content file's markup -> plain text: body only, block tags
|
||||
become line breaks, entities unescaped, whitespace collapsed."""
|
||||
m = _BODY_RE.search(raw)
|
||||
body = m.group(1) if m else raw
|
||||
body = _SCRIPT_STYLE_RE.sub("", body)
|
||||
body = _BLOCK_RE.sub("\n", body)
|
||||
text = html.unescape(_TAG_RE.sub("", body))
|
||||
text = re.sub(r"[ \t]+", " ", text)
|
||||
text = re.sub(r"\n\s*\n+", "\n\n", text)
|
||||
return text.strip()
|
||||
|
||||
|
||||
def extract_text(path: str | Path) -> str:
|
||||
"""Whole-book plain text, in reading order: spine order when the
|
||||
OPF resolves, else every ``.xhtml``/``.html`` member sorted by
|
||||
name. Sections are joined with a blank line; empty sections
|
||||
(nav/cover files with no text) are dropped."""
|
||||
with zipfile.ZipFile(path) as zf:
|
||||
names = spine_paths(zf)
|
||||
if not names:
|
||||
names = sorted(
|
||||
n
|
||||
for n in zf.namelist()
|
||||
if n.lower().endswith((".xhtml", ".html", ".htm"))
|
||||
and not n.startswith("META-INF/")
|
||||
)
|
||||
parts = []
|
||||
for name in names:
|
||||
try:
|
||||
raw = zf.read(name).decode("utf-8", errors="replace")
|
||||
except KeyError:
|
||||
continue
|
||||
parts.append(_to_text(raw))
|
||||
return "\n\n".join(p for p in parts if p.strip())
|
||||
@@ -45,6 +45,8 @@ def extract_attachment(path: Path, *, ocr_engine=None) -> ExtractResult:
|
||||
return _extract_docx(path)
|
||||
if suffix in (".txt", ".html", ".htm"):
|
||||
return _extract_text_like(path, suffix)
|
||||
if suffix == ".epub":
|
||||
return _extract_epub(path)
|
||||
if suffix in _IMAGE_SUFFIXES:
|
||||
# Real CMS comments include scanned-letter images; route to
|
||||
# phase-2 OCR queue, don't drop them as "unsupported".
|
||||
@@ -110,6 +112,24 @@ def _extract_text_like(path: Path, suffix: str) -> ExtractResult:
|
||||
return ExtractResult(text=text, status=status, chars=chars)
|
||||
|
||||
|
||||
def _extract_epub(path: Path) -> ExtractResult:
|
||||
"""F2: an EPUB attachment (the AMA's CPT codebooks — 2021/2022 and
|
||||
the CPT Changes annuals ship EPUB-only) is real book text, not an
|
||||
OCR candidate; ``rex.comments.epub_text`` resolves reading order via
|
||||
the container/spine, the same mechanism ``pfs.cpt_epub`` uses for
|
||||
its own content root."""
|
||||
from rex.comments.epub_text import extract_text
|
||||
|
||||
try:
|
||||
text = extract_text(path)
|
||||
except Exception as e: # noqa: BLE001 — a malformed zip/xml is not fatal
|
||||
log.warning("epub extract failed for %s: %s", path, e)
|
||||
return ExtractResult(text="", status="failed", chars=0)
|
||||
chars = len(text)
|
||||
status = "ok" if chars > _OCR_THRESHOLD else "ocr_needed"
|
||||
return ExtractResult(text=text, status=status, chars=chars)
|
||||
|
||||
|
||||
def _strip_html(s: str) -> str:
|
||||
"""Cheap HTML→text — break on block tags, drop the rest."""
|
||||
s = re.sub(r"<(script|style)[^>]*>.*?</\1>", "", s, flags=re.S | re.I)
|
||||
|
||||
275
tests/bib/test_zotero_import.py
Normal file
275
tests/bib/test_zotero_import.py
Normal file
@@ -0,0 +1,275 @@
|
||||
"""bib.zotero_import — pull a Zotero collection's book items (with
|
||||
files) into bib."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlite3
|
||||
|
||||
import pytest
|
||||
|
||||
from bib.store import Store
|
||||
from bib.zotero_import import ZoteroBook, import_books, list_collection
|
||||
|
||||
|
||||
def _zotero_db(tmp_path):
|
||||
"""A tiny sqlite db with the Zotero schema subset list_collection needs:
|
||||
two book items in "AMA Coding Publications" (one with a storage:
|
||||
attachment, one without), plus a distractor item in another
|
||||
collection and a deleted item that must never surface."""
|
||||
db = tmp_path / "zotero.sqlite"
|
||||
con = sqlite3.connect(db)
|
||||
con.executescript(
|
||||
"""
|
||||
CREATE TABLE items (itemID INTEGER PRIMARY KEY, itemTypeID INTEGER, key TEXT);
|
||||
CREATE TABLE itemTypes (itemTypeID INTEGER PRIMARY KEY, typeName TEXT);
|
||||
CREATE TABLE fields (fieldID INTEGER PRIMARY KEY, fieldName TEXT);
|
||||
CREATE TABLE itemDataValues (valueID INTEGER PRIMARY KEY, value TEXT);
|
||||
CREATE TABLE itemData (itemID INTEGER, fieldID INTEGER, valueID INTEGER);
|
||||
CREATE TABLE collections (collectionID INTEGER PRIMARY KEY, collectionName TEXT);
|
||||
CREATE TABLE collectionItems (collectionID INTEGER, itemID INTEGER);
|
||||
CREATE TABLE itemAttachments (itemID INTEGER, parentItemID INTEGER, path TEXT, contentType TEXT);
|
||||
CREATE TABLE deletedItems (itemID INTEGER);
|
||||
|
||||
INSERT INTO itemTypes VALUES (1, 'book'), (2, 'note');
|
||||
|
||||
-- item 1: CPT 2018, has a pdf attachment (item 10)
|
||||
INSERT INTO items VALUES (1, 1, 'CPT2018K');
|
||||
-- item 2: CPT Changes 2020, no attachment
|
||||
INSERT INTO items VALUES (2, 1, 'CPTCHG20');
|
||||
-- item 3: a note item in the same collection — never a "book"
|
||||
INSERT INTO items VALUES (3, 2, 'NOTEKEY1');
|
||||
-- item 4: deleted book — must never surface
|
||||
INSERT INTO items VALUES (4, 1, 'DELETED1');
|
||||
-- item 10: the attachment for item 1
|
||||
INSERT INTO items VALUES (10, 3, 'ATTKEY01');
|
||||
|
||||
INSERT INTO fields VALUES (1, 'title'), (2, 'url'), (3, 'date'), (4, 'publisher');
|
||||
|
||||
INSERT INTO itemDataValues VALUES
|
||||
(100, 'CPT 2018'), (101, 'https://ama.example/cpt2018'),
|
||||
(102, '2017-11-01'), (103, 'American Medical Association'),
|
||||
(200, 'CPT Changes 2020');
|
||||
|
||||
INSERT INTO itemData VALUES
|
||||
(1, 1, 100), (1, 2, 101), (1, 3, 102), (1, 4, 103),
|
||||
(2, 1, 200);
|
||||
|
||||
INSERT INTO collections VALUES (1, 'AMA Coding Publications'), (2, 'Other');
|
||||
INSERT INTO collectionItems VALUES (1, 1), (1, 2), (1, 3), (1, 4), (2, 1);
|
||||
|
||||
INSERT INTO itemAttachments VALUES (10, 1, 'storage:CPT 2018.pdf', 'application/pdf');
|
||||
|
||||
INSERT INTO deletedItems VALUES (4);
|
||||
"""
|
||||
)
|
||||
con.commit()
|
||||
con.close()
|
||||
return db
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def zcon(tmp_path):
|
||||
con = sqlite3.connect(_zotero_db(tmp_path))
|
||||
yield con
|
||||
con.close()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def storage_dir(tmp_path):
|
||||
d = tmp_path / "storage" / "ATTKEY01"
|
||||
d.mkdir(parents=True)
|
||||
(d / "CPT 2018.pdf").write_bytes(b"%PDF-1.4 fake cpt 2018")
|
||||
return tmp_path / "storage"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def store(tmp_path):
|
||||
s = Store(":memory:", storage_dir=tmp_path / "bib-storage")
|
||||
yield s
|
||||
s.close()
|
||||
|
||||
|
||||
class TestListCollection:
|
||||
def test_two_books_one_with_attachment(self, zcon, storage_dir):
|
||||
books = list_collection(
|
||||
zcon, "AMA Coding Publications", with_files=False, storage_dir=storage_dir
|
||||
)
|
||||
assert {b.key for b in books} == {"CPT2018K", "CPTCHG20"}
|
||||
cpt = next(b for b in books if b.key == "CPT2018K")
|
||||
assert cpt.title == "CPT 2018"
|
||||
assert cpt.url == "https://ama.example/cpt2018"
|
||||
assert cpt.year == "2017"
|
||||
assert cpt.publisher == "American Medical Association"
|
||||
assert cpt.attachments == (storage_dir / "ATTKEY01" / "CPT 2018.pdf",)
|
||||
|
||||
changes = next(b for b in books if b.key == "CPTCHG20")
|
||||
assert changes.attachments == ()
|
||||
assert changes.url == ""
|
||||
|
||||
def test_excludes_notes_and_deleted_items(self, zcon, storage_dir):
|
||||
books = list_collection(
|
||||
zcon, "AMA Coding Publications", with_files=False, storage_dir=storage_dir
|
||||
)
|
||||
keys = {b.key for b in books}
|
||||
assert "NOTEKEY1" not in keys
|
||||
assert "DELETED1" not in keys
|
||||
|
||||
def test_with_files_filters_to_items_with_attachments(self, zcon, storage_dir):
|
||||
books = list_collection(
|
||||
zcon, "AMA Coding Publications", with_files=True, storage_dir=storage_dir
|
||||
)
|
||||
assert [b.key for b in books] == ["CPT2018K"]
|
||||
|
||||
def test_unknown_collection_returns_empty(self, zcon, storage_dir):
|
||||
assert (
|
||||
list_collection(zcon, "Nope", with_files=False, storage_dir=storage_dir)
|
||||
== []
|
||||
)
|
||||
|
||||
|
||||
class TestImportBooks:
|
||||
def _cpt_book(self, storage_dir):
|
||||
return ZoteroBook(
|
||||
key="CPT2018K",
|
||||
title="CPT 2018",
|
||||
url="https://ama.example/cpt2018",
|
||||
year="2017",
|
||||
publisher="American Medical Association",
|
||||
attachments=(storage_dir / "ATTKEY01" / "CPT 2018.pdf",),
|
||||
)
|
||||
|
||||
def test_creates_source_item_with_tags_year_and_attachment(
|
||||
self, store, storage_dir
|
||||
):
|
||||
book = self._cpt_book(storage_dir)
|
||||
stats = import_books(
|
||||
store, [book], tags=["source:ama", "module:coding", "llm:skip"]
|
||||
)
|
||||
assert stats["created"] == 1
|
||||
assert stats["updated"] == 0
|
||||
assert stats["attached"] == 1
|
||||
|
||||
(rec,) = stats["items"]
|
||||
assert rec["zotero_key"] == "CPT2018K"
|
||||
item = store.get(rec["key"])
|
||||
assert item.item_type == "source"
|
||||
assert item.title == "CPT 2018"
|
||||
assert item.url == "https://ama.example/cpt2018"
|
||||
assert set(item.tags) == {
|
||||
"source:ama",
|
||||
"module:coding",
|
||||
"llm:skip",
|
||||
"year:2017",
|
||||
}
|
||||
|
||||
attachments = (
|
||||
store._con()
|
||||
.execute(
|
||||
"SELECT filename FROM attachments a "
|
||||
"JOIN items i ON i.id = a.item_id WHERE i.key = ?",
|
||||
(rec["key"],),
|
||||
)
|
||||
.fetchall()
|
||||
)
|
||||
assert [a[0] for a in attachments] == ["CPT 2018.pdf"]
|
||||
|
||||
def test_fallback_url_used_when_book_has_none(self, store):
|
||||
book = ZoteroBook(
|
||||
key="NOURLKEY",
|
||||
title="No URL Book",
|
||||
url="",
|
||||
year="2020",
|
||||
publisher="",
|
||||
attachments=(),
|
||||
)
|
||||
import_books(store, [book], tags=["source:ama"])
|
||||
item = store.get(store.list_items(query="No URL Book")[0].key)
|
||||
assert item.url == "zotero://select/library/items/NOURLKEY"
|
||||
|
||||
def test_rerun_is_unchanged_with_no_duplicate_attachment(self, store, storage_dir):
|
||||
book = self._cpt_book(storage_dir)
|
||||
tags = ["source:ama", "module:coding", "llm:skip"]
|
||||
first = import_books(store, [book], tags=tags)
|
||||
second = import_books(store, [book], tags=tags)
|
||||
|
||||
assert second["created"] == 0
|
||||
assert second["unchanged"] == 1
|
||||
assert second["attached"] == 0
|
||||
|
||||
bib_key = first["items"][0]["key"]
|
||||
rows = (
|
||||
store._con()
|
||||
.execute(
|
||||
"SELECT filename FROM attachments a "
|
||||
"JOIN items i ON i.id = a.item_id WHERE i.key = ?",
|
||||
(bib_key,),
|
||||
)
|
||||
.fetchall()
|
||||
)
|
||||
assert len(rows) == 1 # no duplicate attachment
|
||||
|
||||
def test_rerun_never_calls_attach_file_for_a_duplicate(self, store, storage_dir):
|
||||
"""Minor: attach_file is idempotent-by-filename, but a re-run
|
||||
used to call it anyway (a wasted copy) — it must now be skipped
|
||||
entirely once the dup check finds the attachment already there."""
|
||||
book = self._cpt_book(storage_dir)
|
||||
tags = ["source:ama"]
|
||||
import_books(store, [book], tags=tags)
|
||||
|
||||
calls = []
|
||||
orig_attach_file = store.attach_file
|
||||
store.attach_file = lambda *a, **k: (
|
||||
calls.append((a, k)) or orig_attach_file(*a, **k)
|
||||
)
|
||||
try:
|
||||
second = import_books(store, [book], tags=tags)
|
||||
finally:
|
||||
store.attach_file = orig_attach_file
|
||||
|
||||
assert calls == []
|
||||
assert second["attached"] == 0
|
||||
|
||||
def test_dry_run_makes_no_writes(self, store, storage_dir):
|
||||
book = self._cpt_book(storage_dir)
|
||||
stats = import_books(store, [book], tags=["source:ama"], dry_run=True)
|
||||
assert stats["created"] == 1
|
||||
assert stats["attached"] == 1
|
||||
assert store.list_items() == [] # nothing written
|
||||
|
||||
def test_dry_run_after_real_import_reports_unchanged(self, store, storage_dir):
|
||||
book = self._cpt_book(storage_dir)
|
||||
tags = ["source:ama"]
|
||||
import_books(store, [book], tags=tags)
|
||||
stats = import_books(store, [book], tags=tags, dry_run=True)
|
||||
assert stats["unchanged"] == 1
|
||||
assert stats["attached"] == 0
|
||||
assert stats["items"][0]["key"] # bib key resolved from the existing url
|
||||
|
||||
def test_missing_attachment_file_counts_and_skips(self, store, storage_dir):
|
||||
ghost_path = storage_dir / "GHOSTKEY" / "missing.pdf"
|
||||
book = ZoteroBook(
|
||||
key="GHOSTBK1",
|
||||
title="Ghost Book",
|
||||
url="https://ama.example/ghost",
|
||||
year="2020",
|
||||
publisher="American Medical Association",
|
||||
attachments=(ghost_path,),
|
||||
)
|
||||
stats = import_books(store, [book], tags=["source:ama"])
|
||||
|
||||
assert stats["missing"] == 1
|
||||
assert stats["attached"] == 0
|
||||
(rec,) = stats["items"]
|
||||
assert rec["missing"] == 1
|
||||
assert rec["missing_paths"] == [str(ghost_path)]
|
||||
|
||||
rows = (
|
||||
store._con()
|
||||
.execute(
|
||||
"SELECT filename FROM attachments a "
|
||||
"JOIN items i ON i.id = a.item_id WHERE i.key = ?",
|
||||
(rec["key"],),
|
||||
)
|
||||
.fetchall()
|
||||
)
|
||||
assert rows == [] # no attachment row for the missing file
|
||||
@@ -55,6 +55,176 @@ class TestDiscoverPfsRules:
|
||||
assert result.exit_code == 0
|
||||
|
||||
|
||||
class TestImportZotero:
|
||||
@patch("bib.zotero_import.import_books")
|
||||
@patch("bib.zotero_import.list_collection")
|
||||
@patch("conf.connect.zotero", return_value=MagicMock())
|
||||
@patch("bib.connect", return_value=MagicMock())
|
||||
def test_dry_run_prints_would_create_and_writes_nothing(
|
||||
self, mc_connect, mc_zotero, mc_list, mc_import
|
||||
):
|
||||
from bib.zotero_import import ZoteroBook
|
||||
|
||||
book = ZoteroBook(
|
||||
key="ZKEY1",
|
||||
title="CPT 2018",
|
||||
url="https://x",
|
||||
year="2017",
|
||||
publisher="AMA",
|
||||
attachments=(),
|
||||
)
|
||||
mc_list.return_value = [book]
|
||||
mc_import.return_value = {
|
||||
"created": 1,
|
||||
"updated": 0,
|
||||
"unchanged": 0,
|
||||
"attached": 0,
|
||||
"missing": 0,
|
||||
"items": [
|
||||
{
|
||||
"key": "",
|
||||
"zotero_key": "ZKEY1",
|
||||
"title": "CPT 2018",
|
||||
"status": "created",
|
||||
"attached": 0,
|
||||
"missing": 0,
|
||||
"missing_paths": [],
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"import-zotero",
|
||||
"--collection",
|
||||
"AMA Coding Publications",
|
||||
"--dry-run",
|
||||
],
|
||||
)
|
||||
|
||||
assert result.exit_code == 0
|
||||
assert "would-created" in result.stdout
|
||||
assert "CPT 2018" in result.stdout
|
||||
# dry_run=True is forwarded to import_books, which is the sole
|
||||
# place writes happen — the CLI itself performs none.
|
||||
_, kwargs = mc_import.call_args
|
||||
assert kwargs["dry_run"] is True
|
||||
assert kwargs["tags"] == []
|
||||
|
||||
@patch("bib.zotero_import.import_books")
|
||||
@patch("bib.zotero_import.list_collection")
|
||||
@patch("conf.connect.zotero", return_value=MagicMock())
|
||||
@patch("bib.connect", return_value=MagicMock())
|
||||
def test_only_filters_by_title_substring(
|
||||
self, mc_connect, mc_zotero, mc_list, mc_import
|
||||
):
|
||||
from bib.zotero_import import ZoteroBook
|
||||
|
||||
cpt = ZoteroBook(
|
||||
key="A", title="CPT 2018", url="", year="2017", publisher="", attachments=()
|
||||
)
|
||||
icd = ZoteroBook(
|
||||
key="B",
|
||||
title="ICD-10-CM 2018",
|
||||
url="",
|
||||
year="2017",
|
||||
publisher="",
|
||||
attachments=(),
|
||||
)
|
||||
mc_list.return_value = [cpt, icd]
|
||||
mc_import.return_value = {
|
||||
"created": 0,
|
||||
"updated": 0,
|
||||
"unchanged": 0,
|
||||
"attached": 0,
|
||||
"missing": 0,
|
||||
"items": [],
|
||||
}
|
||||
|
||||
runner.invoke(
|
||||
app,
|
||||
[
|
||||
"import-zotero",
|
||||
"--collection",
|
||||
"X",
|
||||
"--only",
|
||||
"cpt",
|
||||
"--dry-run",
|
||||
],
|
||||
)
|
||||
|
||||
passed_books = mc_import.call_args[0][1]
|
||||
assert [b.key for b in passed_books] == ["A"]
|
||||
|
||||
@patch("bib.zotero_import.import_books")
|
||||
@patch("bib.zotero_import.list_collection", return_value=[])
|
||||
@patch("conf.connect.zotero", return_value=MagicMock())
|
||||
@patch("bib.connect", return_value=MagicMock())
|
||||
def test_tags_passed_through(self, mc_connect, mc_zotero, mc_list, mc_import):
|
||||
mc_import.return_value = {
|
||||
"created": 0,
|
||||
"updated": 0,
|
||||
"unchanged": 0,
|
||||
"attached": 0,
|
||||
"missing": 0,
|
||||
"items": [],
|
||||
}
|
||||
|
||||
runner.invoke(
|
||||
app,
|
||||
[
|
||||
"import-zotero",
|
||||
"--collection",
|
||||
"X",
|
||||
"--tag",
|
||||
"source:ama",
|
||||
"--tag",
|
||||
"module:coding",
|
||||
],
|
||||
)
|
||||
|
||||
_, kwargs = mc_import.call_args
|
||||
assert kwargs["tags"] == ["source:ama", "module:coding"]
|
||||
|
||||
@patch("bib.zotero_import.import_books")
|
||||
@patch("bib.zotero_import.list_collection")
|
||||
@patch("conf.connect.zotero", return_value=MagicMock())
|
||||
@patch("bib.connect", return_value=MagicMock())
|
||||
def test_missing_file_printed_to_stderr(
|
||||
self, mc_connect, mc_zotero, mc_list, mc_import
|
||||
):
|
||||
from bib.zotero_import import ZoteroBook
|
||||
|
||||
book = ZoteroBook(
|
||||
key="G1", title="Ghost", url="", year="2020", publisher="", attachments=()
|
||||
)
|
||||
mc_list.return_value = [book]
|
||||
mc_import.return_value = {
|
||||
"created": 1,
|
||||
"updated": 0,
|
||||
"unchanged": 0,
|
||||
"attached": 0,
|
||||
"missing": 1,
|
||||
"items": [
|
||||
{
|
||||
"key": "GKEY",
|
||||
"zotero_key": "G1",
|
||||
"title": "Ghost",
|
||||
"status": "created",
|
||||
"attached": 0,
|
||||
"missing": 1,
|
||||
"missing_paths": ["/data/storage/XX/gone.pdf"],
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
result = runner.invoke(app, ["import-zotero", "--collection", "X"])
|
||||
|
||||
assert "! missing on disk: /data/storage/XX/gone.pdf" in result.output
|
||||
assert "missing=1" in result.output
|
||||
|
||||
|
||||
class TestAllHelp:
|
||||
"""Exercise every registered command's --help (exercises Typer paths)."""
|
||||
|
||||
@@ -79,6 +249,9 @@ class TestAllHelp:
|
||||
def test_sync_zotero(self):
|
||||
assert runner.invoke(app, ["sync-zotero", "--help"]).exit_code == 0
|
||||
|
||||
def test_import_zotero(self):
|
||||
assert runner.invoke(app, ["import-zotero", "--help"]).exit_code == 0
|
||||
|
||||
def test_backfill(self):
|
||||
assert runner.invoke(app, ["backfill-comments", "--help"]).exit_code == 0
|
||||
|
||||
|
||||
@@ -96,11 +96,58 @@ class TestIndexCorpus:
|
||||
result = runner.invoke(app, ["index", "--collection", "corpus"])
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
mock_iter.assert_called_once_with(store, zotero=zot)
|
||||
mock_iter.assert_called_once_with(store, keys=(), zotero=zot)
|
||||
kwargs = mock_index_refs.call_args.kwargs
|
||||
assert kwargs["collection"] == "corpus"
|
||||
assert "corpus: indexed=3 skipped=1 chunks=7" in result.output
|
||||
|
||||
@patch("llm.index.docket_complete", return_value={})
|
||||
@patch("llm.index._engine")
|
||||
@patch("llm.source.ZoteroPdfIndex.lazy")
|
||||
@patch("llm.source.iter_corpus_refs")
|
||||
@patch("llm.pool.HostPool.from_config")
|
||||
@patch("llm.index.index_refs")
|
||||
@patch("conf.connect.bib")
|
||||
@patch("llm.config.load")
|
||||
def test_key_option_limits_corpus_to_those_items(
|
||||
self,
|
||||
mock_load,
|
||||
mock_bib,
|
||||
mock_index_refs,
|
||||
mock_from_config,
|
||||
mock_iter,
|
||||
mock_lazy,
|
||||
_engine,
|
||||
_complete,
|
||||
):
|
||||
mock_load.return_value = MagicMock()
|
||||
store = MagicMock()
|
||||
store.sealed_dockets.return_value = {}
|
||||
mock_bib.return_value = store
|
||||
mock_iter.return_value = iter(["doc1"])
|
||||
mock_from_config.return_value = MagicMock()
|
||||
mock_index_refs.return_value = _STATS
|
||||
zot = MagicMock()
|
||||
mock_lazy.return_value = zot
|
||||
|
||||
result = runner.invoke(
|
||||
app,
|
||||
[
|
||||
"index",
|
||||
"--collection",
|
||||
"corpus",
|
||||
"--key",
|
||||
"GQGTPGYV",
|
||||
"--key",
|
||||
"T6H4ZPEQ",
|
||||
],
|
||||
)
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
mock_iter.assert_called_once_with(
|
||||
store, keys=("GQGTPGYV", "T6H4ZPEQ"), zotero=zot
|
||||
)
|
||||
|
||||
|
||||
class TestIndexAll:
|
||||
@patch("llm.index.docket_complete", return_value={})
|
||||
@@ -232,6 +279,72 @@ class TestIndexLimit:
|
||||
assert mock_index_refs.call_args.kwargs["mark_complete"] is False
|
||||
|
||||
|
||||
class TestRestamp:
|
||||
"""F6: `stack llm restamp` must never rebuild the HNSW index — only
|
||||
the cheap metadata GIN indexes, and not even those under --dry-run."""
|
||||
|
||||
@patch("os.path.exists", return_value=False)
|
||||
@patch("pfs.anchors.code_family_index", return_value={})
|
||||
@patch("llm.restamp.restamp")
|
||||
@patch("llm.migrate.ensure_metadata_indexes")
|
||||
@patch("llm.migrate.migrate")
|
||||
@patch("llm.index._engine")
|
||||
@patch("llm.config.load")
|
||||
def test_dry_run_issues_no_ddl(
|
||||
self,
|
||||
mock_load,
|
||||
mock_engine,
|
||||
mock_migrate,
|
||||
mock_ensure_metadata,
|
||||
mock_restamp,
|
||||
mock_code_index,
|
||||
mock_exists,
|
||||
):
|
||||
mock_load.return_value = MagicMock(duckdb_replica="nope.duckdb")
|
||||
mock_restamp.return_value = {
|
||||
"collection": "rules",
|
||||
"scanned": 0,
|
||||
"updated": 0,
|
||||
"seconds": 0.0,
|
||||
}
|
||||
|
||||
result = runner.invoke(app, ["restamp", "--collection", "rules", "--dry-run"])
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
mock_ensure_metadata.assert_not_called()
|
||||
assert mock_restamp.call_args.kwargs["dry_run"] is True
|
||||
|
||||
@patch("os.path.exists", return_value=False)
|
||||
@patch("pfs.anchors.code_family_index", return_value={})
|
||||
@patch("llm.restamp.restamp")
|
||||
@patch("llm.migrate.ensure_metadata_indexes")
|
||||
@patch("llm.migrate.migrate")
|
||||
@patch("llm.index._engine")
|
||||
@patch("llm.config.load")
|
||||
def test_real_run_calls_ensure_metadata_indexes_not_hnsw(
|
||||
self,
|
||||
mock_load,
|
||||
mock_engine,
|
||||
mock_migrate,
|
||||
mock_ensure_metadata,
|
||||
mock_restamp,
|
||||
mock_code_index,
|
||||
mock_exists,
|
||||
):
|
||||
mock_load.return_value = MagicMock(duckdb_replica="nope.duckdb")
|
||||
mock_restamp.return_value = {
|
||||
"collection": "rules",
|
||||
"scanned": 0,
|
||||
"updated": 0,
|
||||
"seconds": 0.0,
|
||||
}
|
||||
|
||||
result = runner.invoke(app, ["restamp", "--collection", "rules"])
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
mock_ensure_metadata.assert_called_once()
|
||||
|
||||
|
||||
class TestServe:
|
||||
@patch("uvicorn.run")
|
||||
def test_serve_starts_uvicorn(self, mock_run):
|
||||
|
||||
@@ -15,9 +15,11 @@ from pfs.codetables import (
|
||||
read_elements,
|
||||
read_events,
|
||||
read_families,
|
||||
write_cpt_edition,
|
||||
write_elements,
|
||||
write_events,
|
||||
)
|
||||
from pfs.cpt_model import CptCode, CptEdition, CptSection
|
||||
from pfs.extract import Extraction
|
||||
|
||||
runner = CliRunner()
|
||||
@@ -211,6 +213,26 @@ class TestLineage:
|
||||
assert read_events(con, "G2058")[0].to_codes == "99439"
|
||||
assert con.published == [True]
|
||||
|
||||
def test_prints_cpt_event_with_cpt_anchor_text(self, con, monkeypatch):
|
||||
ev = EventRow(
|
||||
"99490",
|
||||
2015,
|
||||
"cpt_changed",
|
||||
"",
|
||||
"",
|
||||
"GQGTPGYV",
|
||||
0,
|
||||
0,
|
||||
"cpt",
|
||||
True,
|
||||
"CPT Changes 2015",
|
||||
)
|
||||
monkeypatch.setattr(pfs_cli, "lineage", lambda c, s, code: [ev])
|
||||
res = runner.invoke(app, ["pfs", "lineage", "--code", "99490", "--write"])
|
||||
assert res.exit_code == 0, res.output
|
||||
assert "2015 cpt_changed" in res.output
|
||||
assert "cpt 2015 GQGTPGYV" in res.output
|
||||
|
||||
def test_without_write_reads_and_never_takes_the_write_lock(self, con, monkeypatch):
|
||||
# I4: `stack pfs lineage` without --write must read the replica,
|
||||
# not open a RW duckdb_batch connection (the DuckDB single-writer
|
||||
@@ -263,7 +285,123 @@ class TestFamilies:
|
||||
assert res.exit_code == 0, res.output
|
||||
fams = read_families(con)
|
||||
assert {r.code for r in fams if r.key == "CCM"} >= {"99490", "99439", "G2058"}
|
||||
# Minor: --write prints the summary line by default; the
|
||||
# per-family dump ("CCM" among them) is --verbose only.
|
||||
assert "CCM" not in res.output
|
||||
# No pfs.cpt_* rows seeded in this test — the summary line must
|
||||
# still print, with cpt-named at 0 (I4: read-only-tolerant path
|
||||
# also applies to the write path when the CPT tables are empty).
|
||||
assert "families: total 2, multi-code 1, cpt-named 0, hand 1, other 1" in (
|
||||
res.output
|
||||
)
|
||||
|
||||
def test_verbose_prints_the_per_family_dump(self, con):
|
||||
write_elements(
|
||||
con,
|
||||
"99439",
|
||||
[
|
||||
ElementRow(
|
||||
"99439", 2021, "relation", "addon-of", "99490", "", "K", 1, 1, "fr"
|
||||
)
|
||||
],
|
||||
[],
|
||||
)
|
||||
res = runner.invoke(app, ["pfs", "families", "--write", "--verbose"])
|
||||
assert res.exit_code == 0, res.output
|
||||
assert "CCM" in res.output
|
||||
assert "families: total" in res.output
|
||||
|
||||
def _ccm_cpt_edition(self, year=2024):
|
||||
section = CptSection(
|
||||
sec_id="sec_ccm",
|
||||
level=3,
|
||||
title="Chronic Care Management Services",
|
||||
path=(
|
||||
"Evaluation and Management",
|
||||
"Care Management Services",
|
||||
"Chronic Care Management Services",
|
||||
),
|
||||
code_lo="99490",
|
||||
code_hi="99439",
|
||||
guideline="",
|
||||
)
|
||||
codes = (
|
||||
CptCode(
|
||||
code="99490",
|
||||
sec_id="sec_ccm",
|
||||
descriptor="Chronic care management services, first 20 minutes.",
|
||||
stem="Chronic care management services",
|
||||
elements=(),
|
||||
tail="first 20 minutes.",
|
||||
addon=False,
|
||||
resequenced=False,
|
||||
new=False,
|
||||
revised=False,
|
||||
telemedicine=False,
|
||||
parent="",
|
||||
mod51_exempt=False,
|
||||
audio_only=False,
|
||||
fda_pending=False,
|
||||
pla=False,
|
||||
category="I",
|
||||
),
|
||||
CptCode(
|
||||
code="99439",
|
||||
sec_id="sec_ccm",
|
||||
descriptor="each additional 20 minutes.",
|
||||
stem="Chronic care management services",
|
||||
elements=(),
|
||||
tail="each additional 20 minutes.",
|
||||
addon=True,
|
||||
resequenced=False,
|
||||
new=False,
|
||||
revised=False,
|
||||
telemedicine=False,
|
||||
parent="99490",
|
||||
mod51_exempt=False,
|
||||
audio_only=False,
|
||||
fda_pending=False,
|
||||
pla=False,
|
||||
category="I",
|
||||
),
|
||||
)
|
||||
return CptEdition(
|
||||
year=year,
|
||||
sections=(section,),
|
||||
codes=codes,
|
||||
instructions=(),
|
||||
references=(),
|
||||
crosswalks=(),
|
||||
lists=(),
|
||||
)
|
||||
|
||||
def test_feeds_cpt_inputs_when_present_and_prints_summary_line(self, con):
|
||||
write_cpt_edition(con, self._ccm_cpt_edition(), "ITEM0001")
|
||||
res = runner.invoke(app, ["pfs", "families", "--write"])
|
||||
assert res.exit_code == 0, res.output
|
||||
fams = read_families(con)
|
||||
ccm = {r.code: r for r in fams if r.key == "CCM"}
|
||||
assert {"99490", "99439"} <= set(ccm)
|
||||
assert ccm["99490"].note == (
|
||||
"Evaluation and Management > Care Management Services > "
|
||||
"Chronic Care Management Services"
|
||||
)
|
||||
assert "families: total" in res.output
|
||||
assert "cpt-named" in res.output
|
||||
|
||||
def test_since_spans_all_ingested_editions_not_just_the_newest(self, con):
|
||||
# Ruling C12: _cpt_inputs' cpt_presence comes from a full
|
||||
# SELECT code, edition_year FROM pfs.cpt_code (every edition
|
||||
# write_cpt_edition has ever written), so `since` is the
|
||||
# earliest edition, not clamped to whichever one is newest.
|
||||
write_cpt_edition(con, self._ccm_cpt_edition(year=2019), "ITEM_2019")
|
||||
write_cpt_edition(con, self._ccm_cpt_edition(year=2024), "ITEM_2024")
|
||||
res = runner.invoke(app, ["pfs", "families", "--write"])
|
||||
assert res.exit_code == 0, res.output
|
||||
fams = read_families(con)
|
||||
ccm = {r.code: r for r in fams if r.key == "CCM"}
|
||||
assert ccm["99490"].since == 2019
|
||||
assert ccm["99490"].until is None
|
||||
|
||||
def test_write_tolerates_null_rvu_description(self, con):
|
||||
# #687 regression: a real pfs.rvu row can carry a NULL description
|
||||
@@ -332,6 +470,70 @@ class TestFamilies:
|
||||
assert key in res.output
|
||||
|
||||
|
||||
class TestCptIngest:
|
||||
def test_dry_run_never_opens_batch(self, con, monkeypatch):
|
||||
seen = {}
|
||||
|
||||
def fake_ingest(store, con_, *, years=None, dry_run=False):
|
||||
seen["store"] = store
|
||||
seen["con"] = con_
|
||||
seen["years"] = years
|
||||
seen["dry_run"] = dry_run
|
||||
return {2024: {"sections": 3, "codes": 5}}
|
||||
|
||||
monkeypatch.setattr(pfs_cli, "cpt_ingest", fake_ingest)
|
||||
|
||||
def fail_batch():
|
||||
raise AssertionError("--dry-run must not open a RW duckdb_batch connection")
|
||||
|
||||
monkeypatch.setattr(pfs_cli, "_batch", fail_batch)
|
||||
|
||||
res = runner.invoke(
|
||||
app, ["pfs", "cpt-ingest", "--edition", "2024", "--dry-run"]
|
||||
)
|
||||
assert res.exit_code == 0, res.output
|
||||
assert seen["dry_run"] is True
|
||||
assert seen["con"] is None
|
||||
assert seen["years"] == [2024]
|
||||
assert "2024: sections=3, codes=5" in res.output
|
||||
assert con.published == []
|
||||
|
||||
def test_all_flag_passes_years_none(self, con, monkeypatch):
|
||||
seen = {}
|
||||
monkeypatch.setattr(
|
||||
pfs_cli,
|
||||
"cpt_ingest",
|
||||
lambda store, con_, *, years=None, dry_run=False: (
|
||||
seen.update(years=years) or {}
|
||||
),
|
||||
)
|
||||
res = runner.invoke(app, ["pfs", "cpt-ingest", "--all", "--dry-run"])
|
||||
assert res.exit_code == 0, res.output
|
||||
assert seen["years"] is None
|
||||
|
||||
def test_requires_edition_or_all(self, con):
|
||||
res = runner.invoke(app, ["pfs", "cpt-ingest"])
|
||||
assert res.exit_code != 0
|
||||
|
||||
def test_write_path_opens_batch_ensures_tables_and_publishes(
|
||||
self, con, monkeypatch
|
||||
):
|
||||
seen = {}
|
||||
|
||||
def fake_ingest(store, con_, *, years=None, dry_run=False):
|
||||
seen["con"] = con_
|
||||
seen["dry_run"] = dry_run
|
||||
return {2024: {"sections": 1, "codes": 2}}
|
||||
|
||||
monkeypatch.setattr(pfs_cli, "cpt_ingest", fake_ingest)
|
||||
res = runner.invoke(app, ["pfs", "cpt-ingest", "--edition", "2024"])
|
||||
assert res.exit_code == 0, res.output
|
||||
assert seen["dry_run"] is False
|
||||
assert seen["con"] is not None
|
||||
assert "2024: sections=1, codes=2" in res.output
|
||||
assert con.published == [True]
|
||||
|
||||
|
||||
class TestReview:
|
||||
def test_lists_queue(self, con):
|
||||
from pfs.codetables import ReviewRow
|
||||
|
||||
@@ -184,3 +184,70 @@ class TestCodesMetadata:
|
||||
)
|
||||
(chunk,) = chunk_doc(doc)
|
||||
assert chunk.metadata["codes"] == "G0556"
|
||||
|
||||
|
||||
class TestAnchorMetadata:
|
||||
def test_plain_doc_chunks_have_codes_families_elements_keys(self):
|
||||
text = "Use 99439; consent; per calendar month."
|
||||
chunks = chunk_doc(_doc(text))
|
||||
assert len(chunks) == 1
|
||||
chunk = chunks[0]
|
||||
assert "codes" in chunk.metadata
|
||||
assert "families" in chunk.metadata
|
||||
assert "elements" in chunk.metadata
|
||||
assert chunk.metadata["codes"] == "99439"
|
||||
assert chunk.metadata["families"] == "CCM"
|
||||
|
||||
def test_paragraph_chunks_have_codes_families_elements_keys(self):
|
||||
from llm.chunk import Paragraph
|
||||
|
||||
paras = (Paragraph(1, 100, 1, "G0556 APCM code with consent."),)
|
||||
doc = Doc(
|
||||
key="R", text="G0556 APCM code with consent.", metadata={}, paragraphs=paras
|
||||
)
|
||||
chunks = chunk_doc(doc)
|
||||
assert len(chunks) == 1
|
||||
chunk = chunks[0]
|
||||
assert "codes" in chunk.metadata
|
||||
assert "families" in chunk.metadata
|
||||
assert "elements" in chunk.metadata
|
||||
assert chunk.metadata["codes"] == "G0556"
|
||||
assert chunk.metadata["families"] == "APCM"
|
||||
|
||||
|
||||
class TestCodeIndexParam:
|
||||
"""F1: `chunk_doc(..., code_index=...)` must thread the caller's
|
||||
prebuilt index into every `anchor_metadata` call — the same index
|
||||
`stack llm restamp` builds — instead of scanning the import-time
|
||||
`pfs.families.FAMILIES` (hand families only)."""
|
||||
|
||||
def test_code_index_stamps_a_family_absent_from_hand_families(self):
|
||||
# "90001" belongs to no hand family — without an index, the
|
||||
# chunk's families metadata is empty.
|
||||
text = "Sample text about 90001."
|
||||
without = chunk_doc(_doc(text))
|
||||
assert without[0].metadata["families"] == ""
|
||||
|
||||
idx = {"90001": ("CPT-HEADING-X",)}
|
||||
with_idx = chunk_doc(_doc(text), code_index=idx)
|
||||
assert with_idx[0].metadata["families"] == "CPT-HEADING-X"
|
||||
|
||||
def test_default_none_behaviour_is_unchanged(self):
|
||||
text = "Use 99439; consent; per calendar month."
|
||||
chunks = chunk_doc(_doc(text))
|
||||
assert chunks[0].metadata["codes"] == "99439"
|
||||
assert chunks[0].metadata["families"] == "CCM"
|
||||
|
||||
def test_code_index_threads_through_paragraph_chunks(self):
|
||||
from llm.chunk import Paragraph
|
||||
|
||||
paras = (Paragraph(1, 100, 1, "90001 needs a heading family."),)
|
||||
doc = Doc(
|
||||
key="R", text="90001 needs a heading family.", metadata={}, paragraphs=paras
|
||||
)
|
||||
without = chunk_doc(doc)
|
||||
assert without[0].metadata["families"] == ""
|
||||
|
||||
idx = {"90001": ("CPT-HEADING-X",)}
|
||||
with_idx = chunk_doc(doc, code_index=idx)
|
||||
assert with_idx[0].metadata["families"] == "CPT-HEADING-X"
|
||||
|
||||
@@ -35,6 +35,9 @@ def _run(docs, state_rows, force=False, cfg=CFG):
|
||||
patch("llm.index.embed_texts", return_value=[[0.0] * 3]),
|
||||
patch("llm.index.ensure_hnsw") as mock_ensure_hnsw,
|
||||
patch("llm.index.HostPool") as MockPool,
|
||||
# Real `_code_index` opens the live DuckDB replica — never in a
|
||||
# unit test.
|
||||
patch("llm.index._code_index", return_value={}),
|
||||
):
|
||||
MockPool.return_value.check.return_value = ["http://h1:11434"]
|
||||
stats = index_docs(
|
||||
@@ -138,6 +141,7 @@ class TestDeleteOldChunksFallback:
|
||||
patch("llm.index.embed_texts", return_value=[[0.0] * 3]),
|
||||
patch("llm.index.ensure_hnsw"),
|
||||
patch("llm.index.HostPool") as MockPool,
|
||||
patch("llm.index._code_index", return_value={}),
|
||||
):
|
||||
MockPool.return_value.check.return_value = ["http://h1:11434"]
|
||||
stats = index_docs(
|
||||
|
||||
@@ -48,6 +48,7 @@ def _run(
|
||||
mark_complete=True,
|
||||
complete_rows=(),
|
||||
chunk_doc=None,
|
||||
code_index=None,
|
||||
):
|
||||
store = MagicMock()
|
||||
engine = MagicMock()
|
||||
@@ -72,6 +73,10 @@ def _run(
|
||||
patch("llm.index.ensure_hnsw"),
|
||||
patch("llm.index.enrich_pdf_pages", side_effect=lambda d, c: c) as enrich,
|
||||
patch("llm.index.HostPool") as MockPool,
|
||||
# Real `_code_index` opens the live DuckDB replica — never in a
|
||||
# unit test; every test here gets a fixed (default empty) index
|
||||
# unless it asks for one.
|
||||
patch("llm.index._code_index", return_value=code_index or {}),
|
||||
_maybe_chunk_doc(chunk_doc),
|
||||
):
|
||||
MockPool.return_value.check.return_value = ["http://h1:11434"]
|
||||
@@ -211,8 +216,81 @@ def test_sealed_docket_not_marked_when_a_ref_yields_no_chunks():
|
||||
"""Text that chunks to nothing is a real failure — it blocks the seal
|
||||
and is not stamped."""
|
||||
stats, _, conn, _ = _run(
|
||||
[_ref("fp1")], [], sealed={"D": "s"}, chunk_doc=lambda d: []
|
||||
[_ref("fp1")], [], sealed={"D": "s"}, chunk_doc=lambda d, **k: []
|
||||
)
|
||||
assert stats["skipped"] == 1 and stats["docket_complete"] == 0
|
||||
assert _state_inserts(conn) == []
|
||||
assert _docket_inserts(conn) == []
|
||||
|
||||
|
||||
# ── F1: indexer and restamp must write the same `families` ──────────
|
||||
|
||||
|
||||
def test_index_refs_threads_code_index_into_chunk_doc():
|
||||
"""`index_refs` passes the same `code_index` it built to every
|
||||
`chunk_doc` call — a code that maps to a CPT-derived family in the
|
||||
index built for this run must show up in the chunk's metadata."""
|
||||
captured = []
|
||||
|
||||
def fake_chunk_doc(doc, **kwargs):
|
||||
from llm.chunk import Chunk
|
||||
|
||||
captured.append(kwargs.get("code_index"))
|
||||
return [Chunk(id="c1", text=doc.text, metadata={})]
|
||||
|
||||
fixed_index = {"99490": ("CCM",)}
|
||||
stats, store, conn, enrich = _run(
|
||||
[_ref("fp2")],
|
||||
[("K1", "stale", "fp1")],
|
||||
chunk_doc=fake_chunk_doc,
|
||||
code_index=fixed_index,
|
||||
)
|
||||
assert stats["indexed"] == 1
|
||||
assert captured == [fixed_index]
|
||||
|
||||
|
||||
def test_code_index_builds_once_per_run_via_refresh_from_replica():
|
||||
"""`llm.index._code_index` refreshes from the DuckDB replica when it
|
||||
exists — the same mechanism `stack llm restamp` uses — before
|
||||
building `code_family_index`, so both paths compute `families` from
|
||||
the same snapshot (Ruling F1)."""
|
||||
import llm.index as index_mod
|
||||
|
||||
calls = []
|
||||
|
||||
class FakeCon:
|
||||
def close(self):
|
||||
calls.append("closed")
|
||||
|
||||
def fake_connect(path, read_only):
|
||||
calls.append(("connect", path, read_only))
|
||||
return FakeCon()
|
||||
|
||||
def fake_refresh_from(con):
|
||||
calls.append("refreshed")
|
||||
return 0
|
||||
|
||||
with (
|
||||
patch("llm.index.os.path.exists", return_value=True),
|
||||
patch("duckdb.connect", side_effect=fake_connect),
|
||||
patch("pfs.families.refresh_from", side_effect=fake_refresh_from),
|
||||
):
|
||||
idx = index_mod._code_index(CFG)
|
||||
|
||||
assert calls == [("connect", CFG.duckdb_replica, True), "refreshed", "closed"]
|
||||
assert isinstance(idx, dict)
|
||||
|
||||
|
||||
def test_code_index_skips_refresh_when_no_replica_file():
|
||||
"""No replica on disk (a fresh checkout, or replica=False) — the
|
||||
index falls back to hand families only, no DuckDB touched."""
|
||||
import llm.index as index_mod
|
||||
|
||||
with (
|
||||
patch("llm.index.os.path.exists", return_value=False),
|
||||
patch("pfs.families.refresh_from") as refresh,
|
||||
):
|
||||
idx = index_mod._code_index(CFG)
|
||||
|
||||
refresh.assert_not_called()
|
||||
assert isinstance(idx, dict)
|
||||
|
||||
@@ -24,15 +24,38 @@ class TestDdl:
|
||||
executed = " ".join(str(call.args[0]) for call in conn.execute.call_args_list)
|
||||
assert "vector(768)" in executed
|
||||
assert "USING hnsw" in executed
|
||||
assert (
|
||||
"ix_embedding_collection" in executed
|
||||
and "langchain_pg_embedding (collection_id)" in executed
|
||||
)
|
||||
# F6: the metadata GIN indexes are ensure_metadata_indexes' job now,
|
||||
# not ensure_hnsw's — this must never take the ACCESS EXCLUSIVE lock.
|
||||
assert "ix_embedding_codes" not in executed
|
||||
assert "ix_embedding_families" not in executed
|
||||
assert "ix_embedding_elements" not in executed
|
||||
|
||||
def test_ensure_metadata_indexes_issues_exactly_the_three_gin_statements(self):
|
||||
engine = MagicMock()
|
||||
conn = engine.begin.return_value.__enter__.return_value
|
||||
migrate.ensure_metadata_indexes(engine)
|
||||
calls = [str(call.args[0]) for call in conn.execute.call_args_list]
|
||||
assert len(calls) == 3
|
||||
executed = " ".join(calls)
|
||||
# code-cited lookups scan cmetadata->>'codes' inside one collection
|
||||
assert (
|
||||
"ix_embedding_codes" in executed
|
||||
and "USING gin (string_to_array" in executed
|
||||
)
|
||||
assert (
|
||||
"ix_embedding_collection" in executed
|
||||
and "langchain_pg_embedding (collection_id)" in executed
|
||||
"ix_embedding_families" in executed and "cmetadata->>'families'" in executed
|
||||
)
|
||||
assert (
|
||||
"ix_embedding_elements" in executed and "cmetadata->>'elements'" in executed
|
||||
)
|
||||
# No vector-column ALTER, no HNSW build — this must be safe on a
|
||||
# live table.
|
||||
assert "ALTER TABLE" not in executed
|
||||
assert "hnsw" not in executed.lower()
|
||||
|
||||
def test_fingerprint_column_and_docket_state(self):
|
||||
assert "fingerprint" in migrate.INDEX_STATE_DDL
|
||||
|
||||
154
tests/llm/test_restamp.py
Normal file
154
tests/llm/test_restamp.py
Normal file
@@ -0,0 +1,154 @@
|
||||
"""llm.restamp — metadata-only anchor backfill on already-indexed chunks."""
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
import pytest
|
||||
|
||||
from llm.restamp import plan_patches, restamp
|
||||
|
||||
|
||||
def test_plan_skips_unchanged_and_patches_changed():
|
||||
rows = [
|
||||
(
|
||||
"a",
|
||||
"Use 99439 in conjunction with 99490; consent",
|
||||
{
|
||||
"codes": "99439 99490",
|
||||
"families": "CCM",
|
||||
"elements": "activity=consent relation=addon-of",
|
||||
},
|
||||
),
|
||||
(
|
||||
"b",
|
||||
"Use 99439 in conjunction with 99490; consent",
|
||||
{"codes": "99439 99490"},
|
||||
),
|
||||
("c", "nothing", {}),
|
||||
]
|
||||
patches = plan_patches(rows)
|
||||
ids = [p[0] for p in patches]
|
||||
assert ids == ["b", "c"]
|
||||
assert json.loads(patches[0][1])["families"] == "CCM"
|
||||
assert json.loads(patches[1][1]) == {"codes": "", "families": "", "elements": ""}
|
||||
|
||||
|
||||
# ── F8: fake-engine pagination-loop mechanics ─────────────────────────
|
||||
|
||||
|
||||
class _FakeResult:
|
||||
def __init__(self, rows):
|
||||
self._rows = rows
|
||||
|
||||
def fetchall(self):
|
||||
return self._rows
|
||||
|
||||
|
||||
class _FakeConn:
|
||||
"""Records every ``execute`` call (SQL text + params) and serves
|
||||
*batches* — one list of rows per SELECT, in order — then empty lists
|
||||
forever once exhausted."""
|
||||
|
||||
def __init__(self, calls, batches):
|
||||
self._calls = calls
|
||||
self._batches = list(batches)
|
||||
|
||||
def execute(self, stmt, params=None):
|
||||
sql = str(stmt)
|
||||
self._calls.append((sql, params))
|
||||
if sql.strip().startswith("SELECT"):
|
||||
rows = self._batches.pop(0) if self._batches else []
|
||||
return _FakeResult(rows)
|
||||
return _FakeResult([])
|
||||
|
||||
|
||||
class _FakeBegin:
|
||||
def __init__(self, conn):
|
||||
self._conn = conn
|
||||
|
||||
def __enter__(self):
|
||||
return self._conn
|
||||
|
||||
def __exit__(self, *exc):
|
||||
return False
|
||||
|
||||
|
||||
class _FakeEngine:
|
||||
"""``restamp`` opens a fresh ``with engine.begin() as conn`` per
|
||||
batch; every call shares one connection so ``calls`` sees the whole
|
||||
run in order."""
|
||||
|
||||
def __init__(self, batches):
|
||||
self.calls: list[tuple[str, object]] = []
|
||||
self._conn = _FakeConn(self.calls, batches)
|
||||
|
||||
def begin(self):
|
||||
return _FakeBegin(self._conn)
|
||||
|
||||
|
||||
def _selects(engine):
|
||||
return [c for c in engine.calls if c[0].strip().startswith("SELECT")]
|
||||
|
||||
|
||||
def _updates(engine):
|
||||
return [c for c in engine.calls if c[0].strip().startswith("UPDATE")]
|
||||
|
||||
|
||||
def test_loop_advances_after_to_the_last_id_of_each_batch():
|
||||
batches = [
|
||||
[("id1", "text about 99490", {})],
|
||||
[("id2", "text about 99439", {})],
|
||||
[("id3", "text about G2058", {})],
|
||||
]
|
||||
engine = _FakeEngine(batches)
|
||||
stats = restamp(engine, collection="rules")
|
||||
# One SELECT per batch plus the final empty one that ends the loop.
|
||||
afters = [c[1]["after"] for c in _selects(engine)]
|
||||
assert afters == ["", "id1", "id2", "id3"]
|
||||
assert stats["scanned"] == 3
|
||||
|
||||
|
||||
def test_update_executed_per_batch_with_the_planned_patches():
|
||||
batches = [
|
||||
[("id1", "text about 99490", {})],
|
||||
[("id2", "text about 99439", {})],
|
||||
]
|
||||
engine = _FakeEngine(batches)
|
||||
stats = restamp(engine, collection="rules")
|
||||
updates = _updates(engine)
|
||||
assert len(updates) == 2 # one UPDATE per batch, not per row
|
||||
assert stats["updated"] == 2
|
||||
first_patch_ids = [p["id"] for p in updates[0][1]]
|
||||
assert first_patch_ids == ["id1"]
|
||||
|
||||
|
||||
def test_dry_run_executes_no_update_but_still_counts():
|
||||
batches = [[("id1", "text about 99490", {})]]
|
||||
engine = _FakeEngine(batches)
|
||||
stats = restamp(engine, collection="rules", dry_run=True)
|
||||
assert stats["scanned"] == 1
|
||||
assert stats["updated"] == 1
|
||||
assert _updates(engine) == []
|
||||
|
||||
|
||||
def test_loop_terminates_on_the_first_empty_batch():
|
||||
engine = _FakeEngine([])
|
||||
stats = restamp(engine, collection="rules")
|
||||
assert stats == {
|
||||
"collection": "rules",
|
||||
"scanned": 0,
|
||||
"updated": 0,
|
||||
"seconds": stats["seconds"],
|
||||
}
|
||||
assert len(_selects(engine)) == 1
|
||||
|
||||
|
||||
@pytest.mark.skipif(not os.environ.get("LLM_DB_PASSWORD"), reason="needs pgvector")
|
||||
def test_restamp_dry_run_scans_live_rules():
|
||||
from llm import config as llm_config
|
||||
from llm.index import _engine
|
||||
|
||||
cfg = llm_config.load()
|
||||
engine = _engine(cfg)
|
||||
stats = restamp(engine, collection="rules", batch=50, dry_run=True)
|
||||
assert stats["scanned"] > 0
|
||||
@@ -133,6 +133,13 @@ class TestCorpusDocs:
|
||||
def test_comments_excluded_from_corpus(self, store):
|
||||
assert all(d.metadata["doctype"] != "comment" for d in iter_corpus_docs(store))
|
||||
|
||||
def test_llm_skip_tag_excludes_item_from_corpus(self, store):
|
||||
key = store.create(
|
||||
Item(item_type="source", title="Scratch export", abstract="Draft notes.")
|
||||
)
|
||||
store.add_tag(key, "llm:skip")
|
||||
assert key not in {d.key for d in iter_corpus_docs(store)}
|
||||
|
||||
def test_attachment_text_sectioned_and_listed_in_files(self, store, tmp_path):
|
||||
key = store.create(Item(item_type="rule", title="Rule with attachment"))
|
||||
store.add_tag(key, "year:2021")
|
||||
@@ -143,6 +150,27 @@ class TestCorpusDocs:
|
||||
assert docs[key].text.startswith("## letter.txt\n\nAttachment body text")
|
||||
assert [name for name, _ in docs[key].files] == ["letter.txt"]
|
||||
|
||||
def test_epub_attachment_text_included(self, store, tmp_path):
|
||||
"""F2: an EPUB attachment (the AMA's CPT codebooks) is real book
|
||||
text for the corpus indexer, same as a PDF/DOCX/txt attachment."""
|
||||
import zipfile
|
||||
|
||||
key = store.create(Item(item_type="source", title="CPT Professional 2024"))
|
||||
store.add_tag(key, "year:2024")
|
||||
epub = tmp_path / "CPT Professional 2024.epub"
|
||||
with zipfile.ZipFile(epub, "w") as zf:
|
||||
zf.writestr("mimetype", "application/epub+zip")
|
||||
zf.writestr(
|
||||
"OPS/Chapter01.xhtml",
|
||||
"<html><body><p>"
|
||||
+ "Chronic care management services, first 20 minutes. " * 4
|
||||
+ "</p></body></html>",
|
||||
)
|
||||
store.attach_file(key, epub)
|
||||
docs = {d.key: d for d in iter_corpus_docs(store)}
|
||||
assert "Chronic care management services" in docs[key].text
|
||||
assert [name for name, _ in docs[key].files] == [epub.name]
|
||||
|
||||
def test_zotero_pdf_fallback_used_when_bib_has_no_attachments(
|
||||
self, store, tmp_path
|
||||
):
|
||||
|
||||
@@ -188,6 +188,17 @@ class TestCorpusRefs:
|
||||
store.create(Item(item_type="report", title="U", abstract="Body."))
|
||||
assert [r.key for r in iter_corpus_refs(store, tag="project:pfs")] == [tagged]
|
||||
|
||||
def test_keys_filter_selects_only_named_items(self, store):
|
||||
wanted = store.create(Item(item_type="source", title="CPT 2024", abstract="."))
|
||||
store.create(Item(item_type="source", title="Other book", abstract="."))
|
||||
refs = list(iter_corpus_refs(store, keys=(wanted,)))
|
||||
assert [r.key for r in refs] == [wanted]
|
||||
|
||||
def test_keys_empty_tuple_yields_every_item(self, store):
|
||||
a = store.create(Item(item_type="source", title="A", abstract="."))
|
||||
b = store.create(Item(item_type="source", title="B", abstract="."))
|
||||
assert {r.key for r in iter_corpus_refs(store, keys=())} == {a, b}
|
||||
|
||||
|
||||
class TestLazyZotero:
|
||||
def test_no_copy_until_first_lookup(self, tmp_path):
|
||||
|
||||
0
tests/notebooks/__init__.py
Normal file
0
tests/notebooks/__init__.py
Normal file
49
tests/notebooks/test_code_families_nb.py
Normal file
49
tests/notebooks/test_code_families_nb.py
Normal file
@@ -0,0 +1,49 @@
|
||||
"""notebooks/code_families.py — structure and headless degradation."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
import importlib.util
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
NB = Path(__file__).resolve().parents[2] / "notebooks" / "code_families.py"
|
||||
|
||||
|
||||
def _load():
|
||||
spec = importlib.util.spec_from_file_location("code_families_nb", NB)
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(mod)
|
||||
return mod
|
||||
|
||||
|
||||
def test_notebook_is_a_marimo_app():
|
||||
mod = _load()
|
||||
assert mod.app.__class__.__name__ == "App"
|
||||
|
||||
|
||||
def test_cells_are_anonymous_and_banners_present():
|
||||
src = NB.read_text()
|
||||
tree = ast.parse(src)
|
||||
names = [n.name for n in tree.body if isinstance(n, ast.FunctionDef)]
|
||||
assert names and set(names) == {"_"}
|
||||
banners = re.findall(r"# ── (\d)\. ", src)
|
||||
assert [int(b) for b in banners] == list(range(0, 9))
|
||||
|
||||
|
||||
def test_headless_run_degrades_without_data(monkeypatch, tmp_path):
|
||||
"""With no replica and no bib the guards render notes instead of raising."""
|
||||
monkeypatch.setenv("STACK_DUCKDB_REPLICA", "1")
|
||||
mod = _load()
|
||||
import conf.connect as cc
|
||||
|
||||
monkeypatch.setattr(
|
||||
cc,
|
||||
"duckdb",
|
||||
lambda *a, **k: (_ for _ in ()).throw(FileNotFoundError("no replica")),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
cc, "bib", lambda *a, **k: (_ for _ in ()).throw(FileNotFoundError("no bib"))
|
||||
)
|
||||
result = mod.app.run() # marimo runs all cells; mo.stop cascades are fine
|
||||
assert result is not None
|
||||
72
tests/pfs/fixtures/cpt_sample.xhtml
Normal file
72
tests/pfs/fixtures/cpt_sample.xhtml
Normal file
@@ -0,0 +1,72 @@
|
||||
<html xmlns="http://www.w3.org/1999/xhtml">
|
||||
<head>
|
||||
<title>Synthetic CPT sample</title>
|
||||
<link href="stylesheet.css" rel="stylesheet" type="text/css"/>
|
||||
</head>
|
||||
<body>
|
||||
<div class="h1-toc"><a href="body.xhtml#sec_1"><span class="green">Care Coordination Services* (55000-55010)</span></a></div>
|
||||
<div class="h2-toc"><a href="body.xhtml#sec_2"><span class="red">Sample Chronic Wellness Services* (55000-55003)</span></a></div>
|
||||
<div class="h1-toc"><a href="body.xhtml#sec_3"><span class="green">Other Sample Services* (55100-55101)</span></a></div>
|
||||
<div class="h2-toc"><a href="body.xhtml#sec_4"><span class="red">Empty Category Services*</span></a></div>
|
||||
|
||||
<div class="h1" id="sec_1">Care Coordination Services</div>
|
||||
<div class="noindent">Sample guideline text for the invented care-coordination family of services, used only to test parsing & not copied from any real codebook.</div>
|
||||
<div class="h2" id="sec_2">Sample Chronic Wellness Services</div>
|
||||
<div class="noindent">Sample per-category guideline text describing invented wellness-coordination activities.</div>
|
||||
<table class="table1">
|
||||
<tbody>
|
||||
<tr>
|
||||
<td class="td-w1" id="code_55000"><div class="table-para"><span class="ama-en">#</span> <b>55000</b></div></td>
|
||||
<td class="td"><div class="table-para1">Sample wellness coordination services with the following required elements:</div>
|
||||
<div class="table-slist">■<span class="ssp"/>first invented element of the service,</div>
|
||||
<div class="table-slist1">■<span class="ssp"/>second invented element of the service,</div>
|
||||
<div class="table-slist1">■<span class="ssp"/>third invented element of the service;</div>
|
||||
<div class="table-para2">first 20 minutes of invented staff time, per calendar month.</div>
|
||||
<div class="table-RT"><span class="blue"><span class="ama-en">➲</span></span><span class="sp1"/><i>CPT Changes: An Insider's View</i> 2015, 2021, 2022</div>
|
||||
<div class="table-RT1"><span class="green"><span class="ama-en">➲</span></span><span class="sp1"/><i>CPT Assistant</i> Jan 21:5</div></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="td-w1" id="code_55001"><div class="table-para"><span class="ama-en">#</span><span class="ama-en">✚</span> <b>55001</b></div></td>
|
||||
<td class="td"><div class="table-para1-sub">each additional 20 minutes of invented staff time, per calendar month (List separately in addition to code for primary procedure)</div>
|
||||
<div class="table-RT-sub"><span class="blue"><span class="ama-en">➲</span></span><span class="sp1"/><i>CPT Changes: An Insider's View</i> 2022</div>
|
||||
<div class="table-para2">(Use 55001 in conjunction with 55000)</div>
|
||||
<div class="table-para2">(Do not report 55001, 55000 in the same calendar month with 90951-90970, 55555)</div>
|
||||
<div class="table-para2">(Sample wellness services of less than 20 minutes duration are not reported separately)</div>
|
||||
<div class="table-para2">(Do not report 55001 more than twice per calendar month)</div>
|
||||
<div class="table-para2">(Do not report 55001 in addition to 55002 more than once per calendar month)</div></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="td-w1" id="code_56000"><div class="table-para"><b>56000</b></div></td>
|
||||
<td class="td"><div class="table-para1">Sample incision, deep fascial plane; with exploratory biopsy</div></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="td-w1" id="code_56005"><div class="table-para"><b>56005</b></div></td>
|
||||
<td class="td"><div class="table-para1-sub">with synovial debridement</div></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<div class="h1" id="sec_3">Other Sample Services</div>
|
||||
<table class="table1">
|
||||
<tbody>
|
||||
<tr>
|
||||
<td class="td-w1"><div class="table-para"><span class="red"><b>55000</b></span></div></td>
|
||||
<td class="td"><div class="table-para1">Code is out of numerical sequence. See 55010-55019</div></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<div class="h2" id="sec_4">Empty Category Services</div>
|
||||
<div class="noindent">Guideline text with no code rows, to test that a section with zero codes still gets a path.</div>
|
||||
|
||||
<div class="h1">No Id Category Services</div>
|
||||
<div class="noindent">Sample guideline text for a section whose heading carries no id attribute at all, used to test the synthetic sec_id fallback.</div>
|
||||
<table class="table1">
|
||||
<tbody>
|
||||
<tr>
|
||||
<td class="td-w1" id="code_57000"><div class="table-para"><b>57000</b></div></td>
|
||||
<td class="td"><div class="table-para1">Sample invented service with no indentation family.</div></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</body>
|
||||
</html>
|
||||
75
tests/pfs/test_anchors.py
Normal file
75
tests/pfs/test_anchors.py
Normal file
@@ -0,0 +1,75 @@
|
||||
"""pfs.anchors — metadata for chunks: codes, families, and element slugs."""
|
||||
|
||||
from pfs.anchors import anchor_metadata, code_family_index
|
||||
from pfs.families import FAMILIES, Family
|
||||
|
||||
|
||||
def test_codes_families_elements():
|
||||
md = anchor_metadata(
|
||||
"Use 99439 in conjunction with 99490; consent; per calendar month"
|
||||
)
|
||||
assert md["codes"] == "99439 99490"
|
||||
assert md["families"] == "CCM"
|
||||
assert (
|
||||
"activity=consent" in md["elements"].split()
|
||||
and "period=calendar-month" in md["elements"].split()
|
||||
)
|
||||
assert "relation=addon-of" in md["elements"].split() # detail dropped
|
||||
|
||||
|
||||
def test_empty_and_unknown():
|
||||
assert anchor_metadata("nothing here") == {
|
||||
"codes": "",
|
||||
"families": "",
|
||||
"elements": "",
|
||||
}
|
||||
fams = {"X": Family("X", "X fam", ("12345", "23456"), ("x fam",))}
|
||||
assert anchor_metadata("code 12345", families=fams)["families"] == "X"
|
||||
|
||||
|
||||
def test_code_index_matches_unindexed_scan():
|
||||
text = "Use 99439 in conjunction with 99490; consent; per calendar month"
|
||||
index = code_family_index(FAMILIES)
|
||||
assert anchor_metadata(text, code_index=index) == anchor_metadata(text)
|
||||
|
||||
|
||||
def test_code_index_keeps_every_family_on_collision():
|
||||
"""Ruling A6: a code shared by two qualifying families must keep
|
||||
both in the index — a single-winner index made the indexed path
|
||||
(restamp) disagree with the un-indexed scan the moment two families
|
||||
shared a code, which the next slice (CPT headings intersecting a
|
||||
hand family) will create."""
|
||||
fams = {
|
||||
"A": Family("A", "A fam", ("12345", "23456"), ("a fam",)),
|
||||
"B": Family("B", "B fam", ("12345", "34567"), ("b fam",)),
|
||||
}
|
||||
index = code_family_index(fams)
|
||||
assert index["12345"] == ("A", "B")
|
||||
|
||||
|
||||
def test_code_index_matches_unindexed_scan_on_collision():
|
||||
"""The indexed and un-indexed paths must write identical 'families'
|
||||
values for the same text once a code belongs to two families."""
|
||||
fams = {
|
||||
"A": Family("A", "A fam", ("12345", "23456"), ("a fam",)),
|
||||
"B": Family("B", "B fam", ("12345", "34567"), ("b fam",)),
|
||||
}
|
||||
text = "code 12345"
|
||||
index = code_family_index(fams)
|
||||
assert anchor_metadata(text, families=fams, code_index=index) == anchor_metadata(
|
||||
text, families=fams
|
||||
)
|
||||
assert anchor_metadata(text, families=fams)["families"] == "A B"
|
||||
|
||||
|
||||
def test_single_code_family_never_stamped():
|
||||
"""A stem-derived family of exactly one code (~17,182 of 17,187
|
||||
derived keys) is not a grouping worth its own anchor — it would just
|
||||
echo the code already in 'codes' (Ruling A5)."""
|
||||
fams = {"SOLO": Family("SOLO", "Solo", ("54321",), ("solo",))}
|
||||
assert "54321" not in code_family_index(fams)
|
||||
assert anchor_metadata("code 54321", families=fams)["families"] == ""
|
||||
index = code_family_index(fams)
|
||||
assert (
|
||||
anchor_metadata("code 54321", families=fams, code_index=index)["families"] == ""
|
||||
)
|
||||
@@ -10,16 +10,32 @@ from pfs.codetables import (
|
||||
EventRow,
|
||||
FamilyRow,
|
||||
ReviewRow,
|
||||
cpt_years,
|
||||
ensure_tables,
|
||||
is_missing_table_error,
|
||||
read_all_elements,
|
||||
read_all_events,
|
||||
read_cpt_codes,
|
||||
read_cpt_instructions,
|
||||
read_cpt_sections,
|
||||
read_elements,
|
||||
read_events,
|
||||
read_families,
|
||||
write_cpt_edition,
|
||||
write_elements,
|
||||
write_events,
|
||||
write_families,
|
||||
)
|
||||
from pfs.cpt_model import (
|
||||
CptAlternate,
|
||||
CptCode,
|
||||
CptCrosswalk,
|
||||
CptEdition,
|
||||
CptInstruction,
|
||||
CptListEntry,
|
||||
CptReference,
|
||||
CptSection,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
@@ -36,6 +52,27 @@ def _el(code="99490", value="consent", year=2015):
|
||||
)
|
||||
|
||||
|
||||
class TestIsMissingTableError:
|
||||
def test_true_for_a_real_duckdb_catalog_error(self, con):
|
||||
with pytest.raises(Exception) as exc_info:
|
||||
con.execute("SELECT * FROM pfs.nope")
|
||||
assert is_missing_table_error(exc_info.value)
|
||||
|
||||
def test_false_when_only_catalog_present(self):
|
||||
assert not is_missing_table_error(Exception("Catalog Error: something else"))
|
||||
|
||||
def test_false_when_only_does_not_exist_present(self):
|
||||
# A message that happens to say "does not exist" for an unrelated
|
||||
# reason (not a Catalog error) must not be treated as "missing
|
||||
# table, just nothing derived yet" — that would swallow a real bug.
|
||||
assert not is_missing_table_error(Exception("file does not exist"))
|
||||
|
||||
def test_true_when_both_substrings_present(self):
|
||||
assert is_missing_table_error(
|
||||
Exception('Catalog Error: Table "pfs.code_family" does not exist')
|
||||
)
|
||||
|
||||
|
||||
class TestDDL:
|
||||
def test_idempotent(self, con):
|
||||
ensure_tables(con)
|
||||
@@ -50,6 +87,13 @@ class TestDDL:
|
||||
"code_element_review",
|
||||
"code_event",
|
||||
"code_family",
|
||||
"cpt_section",
|
||||
"cpt_code",
|
||||
"cpt_instruction",
|
||||
"cpt_reference",
|
||||
"cpt_crosswalk",
|
||||
"cpt_list",
|
||||
"cpt_code_alt",
|
||||
} <= names
|
||||
|
||||
|
||||
@@ -179,3 +223,317 @@ class TestFamilies:
|
||||
)
|
||||
fams = read_families(con)
|
||||
assert len(fams) == 1 and fams[0].code == "99439" and fams[0].role == "add-on"
|
||||
|
||||
def test_note_round_trips(self, con):
|
||||
# Ruling C1: `note` is the last field, defaulted, so a caller that
|
||||
# omits it (as `test_full_replace` above does) still works — and
|
||||
# a caller that sets it gets it back unchanged.
|
||||
path_key = (
|
||||
"Evaluation and Management > Care Management Services > "
|
||||
"Chronic Care Management Services"
|
||||
)
|
||||
write_families(
|
||||
con,
|
||||
[
|
||||
FamilyRow(
|
||||
"CCM",
|
||||
"Chronic Care Management",
|
||||
"99490",
|
||||
"base",
|
||||
2015,
|
||||
None,
|
||||
"",
|
||||
0,
|
||||
path_key,
|
||||
)
|
||||
],
|
||||
)
|
||||
fams = read_families(con)
|
||||
assert len(fams) == 1
|
||||
assert fams[0].note == path_key
|
||||
|
||||
def test_note_defaults_to_empty_string(self, con):
|
||||
write_families(
|
||||
con,
|
||||
[
|
||||
FamilyRow(
|
||||
"CCM", "Chronic Care Management", "99490", "base", 2015, None, "", 0
|
||||
)
|
||||
],
|
||||
)
|
||||
assert read_families(con)[0].note == ""
|
||||
|
||||
def test_ensure_tables_upgrades_a_pre_existing_table_without_note(self):
|
||||
# A replica created before `note` existed has an 8-column
|
||||
# pfs.code_family — ensure_tables must add the column in place,
|
||||
# not require a drop/recreate (Ruling C1).
|
||||
con = duckdb.connect(":memory:")
|
||||
try:
|
||||
con.execute("CREATE SCHEMA IF NOT EXISTS pfs;")
|
||||
con.execute(
|
||||
"CREATE TABLE pfs.code_family ("
|
||||
"key VARCHAR, name VARCHAR, code VARCHAR, role VARCHAR, "
|
||||
"since INTEGER, until INTEGER, item_key VARCHAR, p_id INTEGER)"
|
||||
)
|
||||
con.execute(
|
||||
"INSERT INTO pfs.code_family VALUES "
|
||||
"('CCM', 'Chronic Care Management', '99490', 'base', 2015, NULL, '', 0)"
|
||||
)
|
||||
ensure_tables(con)
|
||||
cols = {
|
||||
r[0]
|
||||
for r in con.execute(
|
||||
"SELECT column_name FROM information_schema.columns "
|
||||
"WHERE table_schema='pfs' AND table_name='code_family'"
|
||||
).fetchall()
|
||||
}
|
||||
assert "note" in cols
|
||||
# The pre-existing row survives the upgrade with note = NULL,
|
||||
# and a fresh write still round-trips.
|
||||
row = read_families(con)[0]
|
||||
assert row.code == "99490" and row.note is None
|
||||
write_families(
|
||||
con,
|
||||
[
|
||||
FamilyRow(
|
||||
"CCM",
|
||||
"Chronic Care Management",
|
||||
"99491",
|
||||
"base",
|
||||
2015,
|
||||
None,
|
||||
"",
|
||||
0,
|
||||
"some path key",
|
||||
)
|
||||
],
|
||||
)
|
||||
assert read_families(con)[0].note == "some path key"
|
||||
finally:
|
||||
con.close()
|
||||
|
||||
|
||||
def _tiny_edition(year=2024, codes=None, alternates=()):
|
||||
sections = (
|
||||
CptSection(
|
||||
sec_id="sec_1",
|
||||
level=2,
|
||||
title="Chronic Care Management Services",
|
||||
path=(
|
||||
"Evaluation and Management",
|
||||
"Care Management Services",
|
||||
"Chronic Care Management Services",
|
||||
),
|
||||
code_lo="99490",
|
||||
code_hi="99491",
|
||||
guideline="Sample guideline text.",
|
||||
),
|
||||
)
|
||||
if codes is None:
|
||||
codes = (
|
||||
CptCode(
|
||||
code="99490",
|
||||
sec_id="sec_1",
|
||||
descriptor="Chronic care management services first 20 minutes.",
|
||||
stem="Chronic care management services",
|
||||
elements=("first element;",),
|
||||
tail="first 20 minutes.",
|
||||
addon=False,
|
||||
resequenced=False,
|
||||
new=False,
|
||||
revised=False,
|
||||
telemedicine=False,
|
||||
parent="",
|
||||
mod51_exempt=False,
|
||||
audio_only=False,
|
||||
fda_pending=False,
|
||||
pla=False,
|
||||
category="I",
|
||||
),
|
||||
CptCode(
|
||||
code="99439",
|
||||
sec_id="sec_1",
|
||||
descriptor="each additional 20 minutes.",
|
||||
stem="Chronic care management services",
|
||||
elements=("first element;",),
|
||||
tail="each additional 20 minutes.",
|
||||
addon=True,
|
||||
resequenced=False,
|
||||
new=False,
|
||||
revised=False,
|
||||
telemedicine=False,
|
||||
parent="99490",
|
||||
mod51_exempt=False,
|
||||
audio_only=False,
|
||||
fda_pending=False,
|
||||
pla=False,
|
||||
category="I",
|
||||
),
|
||||
)
|
||||
instructions = (
|
||||
CptInstruction(
|
||||
code="99439",
|
||||
kind="use-with",
|
||||
text="(Use 99439 in conjunction with 99490)",
|
||||
targets=("99490",),
|
||||
),
|
||||
)
|
||||
references = (
|
||||
CptReference(
|
||||
code="99490",
|
||||
kind="cpt-changes",
|
||||
years=(2015, 2022),
|
||||
text="CPT Changes: An Insider's View 2015, 2022",
|
||||
),
|
||||
)
|
||||
crosswalks = (
|
||||
CptCrosswalk(
|
||||
current_code="99490",
|
||||
former_code="99091T",
|
||||
year_deleted="2022",
|
||||
citations="CPT Changes 2022",
|
||||
),
|
||||
)
|
||||
lists = (CptListEntry(appendix="D", code="99439"),)
|
||||
return CptEdition(
|
||||
year=year,
|
||||
sections=sections,
|
||||
codes=codes,
|
||||
instructions=instructions,
|
||||
references=references,
|
||||
crosswalks=crosswalks,
|
||||
lists=lists,
|
||||
alternates=alternates,
|
||||
)
|
||||
|
||||
|
||||
class TestCptEditionEmptyGuardF4:
|
||||
def test_zero_codes_refused_without_force(self, con):
|
||||
edition = _tiny_edition(codes=())
|
||||
with pytest.raises(ValueError, match="zero codes"):
|
||||
write_cpt_edition(con, edition, "GQGTPGYV")
|
||||
# And it must never have wiped whatever was already on file —
|
||||
# write a real edition first, then the refused zero-code write
|
||||
# for the same year must leave it untouched.
|
||||
write_cpt_edition(con, _tiny_edition(), "GQGTPGYV")
|
||||
with pytest.raises(ValueError, match="zero codes"):
|
||||
write_cpt_edition(con, edition, "GQGTPGYV")
|
||||
assert len(read_cpt_codes(con, 2024)) == 2
|
||||
|
||||
def test_zero_codes_written_with_force(self, con):
|
||||
edition = _tiny_edition(codes=())
|
||||
counts = write_cpt_edition(con, edition, "GQGTPGYV", force=True)
|
||||
assert counts["codes"] == 0
|
||||
|
||||
|
||||
class TestCptEdition:
|
||||
def test_write_returns_counts_per_table(self, con):
|
||||
counts = write_cpt_edition(con, _tiny_edition(), "GQGTPGYV")
|
||||
assert counts == {
|
||||
"sections": 1,
|
||||
"codes": 2,
|
||||
"instructions": 1,
|
||||
"references": 1,
|
||||
"crosswalks": 1,
|
||||
"lists": 1,
|
||||
"alternates": 0,
|
||||
}
|
||||
|
||||
def test_alternates_round_trip(self, con):
|
||||
alt = CptAlternate(
|
||||
code="99490", sec_id="sec_guideline", reason="guidelines-reprint"
|
||||
)
|
||||
counts = write_cpt_edition(con, _tiny_edition(alternates=(alt,)), "GQGTPGYV")
|
||||
assert counts["alternates"] == 1
|
||||
row = con.execute(
|
||||
"SELECT edition_year, item_key, code, sec_id, reason FROM pfs.cpt_code_alt"
|
||||
).fetchone()
|
||||
assert row == (2024, "GQGTPGYV", "99490", "sec_guideline", "guidelines-reprint")
|
||||
|
||||
def test_round_trip_reads(self, con):
|
||||
write_cpt_edition(con, _tiny_edition(), "GQGTPGYV")
|
||||
|
||||
secs = read_cpt_sections(con, 2024)
|
||||
assert len(secs) == 1
|
||||
assert secs[0].item_key == "GQGTPGYV"
|
||||
assert secs[0].path == [
|
||||
"Evaluation and Management",
|
||||
"Care Management Services",
|
||||
"Chronic Care Management Services",
|
||||
]
|
||||
assert (
|
||||
secs[0].path_key
|
||||
== "Evaluation and Management > Care Management Services > Chronic Care Management Services"
|
||||
)
|
||||
|
||||
codes = read_cpt_codes(con, 2024)
|
||||
assert {c.code for c in codes} == {"99490", "99439"}
|
||||
addon = next(c for c in codes if c.code == "99439")
|
||||
assert addon.addon is True
|
||||
assert addon.parent == "99490"
|
||||
assert addon.elements == ["first element;"]
|
||||
|
||||
instr = read_cpt_instructions(con, 2024)
|
||||
assert len(instr) == 1
|
||||
assert instr[0].targets == ["99490"]
|
||||
|
||||
assert cpt_years(con) == [2024]
|
||||
|
||||
def test_delete_then_insert_per_edition_year(self, con):
|
||||
write_cpt_edition(con, _tiny_edition(year=2021), "AJM4KF4G")
|
||||
write_cpt_edition(con, _tiny_edition(year=2024), "GQGTPGYV")
|
||||
assert cpt_years(con) == [2021, 2024]
|
||||
|
||||
write_cpt_edition(
|
||||
con, _tiny_edition(year=2024), "GQGTPGYV"
|
||||
) # idempotent re-run
|
||||
assert cpt_years(con) == [2021, 2024]
|
||||
assert len(read_cpt_codes(con, 2021)) == 2 # untouched by the 2024 rewrite
|
||||
assert len(read_cpt_codes(con, 2024)) == 2
|
||||
|
||||
def test_duplicate_code_raises_and_writes_nothing(self, con):
|
||||
# C8: the loader's own last-ditch guard against a resequenced
|
||||
# placeholder row slipping past the parser's filter.
|
||||
dup_codes = (
|
||||
CptCode(
|
||||
code="99490",
|
||||
sec_id="sec_1",
|
||||
descriptor="a",
|
||||
stem="a",
|
||||
elements=(),
|
||||
tail="",
|
||||
addon=False,
|
||||
resequenced=False,
|
||||
new=False,
|
||||
revised=False,
|
||||
telemedicine=False,
|
||||
parent="",
|
||||
mod51_exempt=False,
|
||||
audio_only=False,
|
||||
fda_pending=False,
|
||||
pla=False,
|
||||
category="I",
|
||||
),
|
||||
CptCode(
|
||||
code="99490",
|
||||
sec_id="sec_1",
|
||||
descriptor="b",
|
||||
stem="b",
|
||||
elements=(),
|
||||
tail="",
|
||||
addon=False,
|
||||
resequenced=False,
|
||||
new=False,
|
||||
revised=False,
|
||||
telemedicine=False,
|
||||
parent="",
|
||||
mod51_exempt=False,
|
||||
audio_only=False,
|
||||
fda_pending=False,
|
||||
pla=False,
|
||||
category="I",
|
||||
),
|
||||
)
|
||||
with pytest.raises(ValueError, match="99490"):
|
||||
write_cpt_edition(con, _tiny_edition(codes=dup_codes), "GQGTPGYV")
|
||||
assert read_cpt_codes(con, 2024) == []
|
||||
|
||||
740
tests/pfs/test_cpt_epub.py
Normal file
740
tests/pfs/test_cpt_epub.py
Normal file
@@ -0,0 +1,740 @@
|
||||
"""Tests for pfs.cpt_epub — the pure CPT EPUB parser.
|
||||
|
||||
The synthetic fixture (tests/pfs/fixtures/cpt_sample.xhtml) is written
|
||||
in the 2024 template's markup style with invented text; it is never
|
||||
copied from the real AMA codebook. The integration tests at the bottom
|
||||
run the real parser against the actual EPUB files on disk (read-only,
|
||||
in data/zotero/data/storage/) and are skipped when a file is absent —
|
||||
they assert structure and counts only, never descriptor text.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from pfs.cpt_epub import parse_epub, parse_xhtml
|
||||
|
||||
FIXTURE = Path(__file__).parent / "fixtures" / "cpt_sample.xhtml"
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
EPUB_2024 = (
|
||||
REPO_ROOT
|
||||
/ "data/zotero/data/storage/EIGIRKRK/CPT Professional 2024 - American Medical Association.epub"
|
||||
)
|
||||
EPUB_2022 = REPO_ROOT / "data/zotero/data/storage/UG4R55FW/CPT Professional 2022.epub"
|
||||
EPUB_2021 = (
|
||||
REPO_ROOT / "data/zotero/data/storage/NM3NJZV5/CPT 2021 Professional Edition.epub"
|
||||
)
|
||||
EPUB_2019 = REPO_ROOT / "data/zotero/data/storage/VA34EEIM/CPT 2019.epub"
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def parsed():
|
||||
text = FIXTURE.read_text(encoding="utf-8")
|
||||
sections, codes, instructions, references = parse_xhtml(
|
||||
text, year=2024, source="cpt_sample.xhtml"
|
||||
)
|
||||
return {
|
||||
"sections": {s.sec_id: s for s in sections},
|
||||
"sections_list": sections,
|
||||
"codes": {c.code: c for c in codes},
|
||||
"codes_list": codes,
|
||||
"instructions": instructions,
|
||||
"references": references,
|
||||
}
|
||||
|
||||
|
||||
class TestSections:
|
||||
def test_two_top_level_and_two_nested(self, parsed):
|
||||
levels = sorted(
|
||||
(s.sec_id, s.level)
|
||||
for s in parsed["sections_list"]
|
||||
if s.sec_id in {"sec_1", "sec_2", "sec_3", "sec_4"}
|
||||
)
|
||||
assert levels == [
|
||||
("sec_1", 1),
|
||||
("sec_2", 2),
|
||||
("sec_3", 1),
|
||||
("sec_4", 2),
|
||||
]
|
||||
# plus the id-less "No Id Category Services" h1 (TestSecIdFallback)
|
||||
assert len(parsed["sections_list"]) == 5
|
||||
|
||||
def test_paths_include_ancestors(self, parsed):
|
||||
sec2 = parsed["sections"]["sec_2"]
|
||||
assert sec2.title == "Sample Chronic Wellness Services"
|
||||
assert sec2.path == (
|
||||
"Care Coordination Services",
|
||||
"Sample Chronic Wellness Services",
|
||||
)
|
||||
|
||||
sec4 = parsed["sections"]["sec_4"]
|
||||
assert sec4.path == ("Other Sample Services", "Empty Category Services")
|
||||
|
||||
def test_lo_hi_from_toc_snippet(self, parsed):
|
||||
sec1 = parsed["sections"]["sec_1"]
|
||||
assert (sec1.code_lo, sec1.code_hi) == ("55000", "55010")
|
||||
sec2 = parsed["sections"]["sec_2"]
|
||||
assert (sec2.code_lo, sec2.code_hi) == ("55000", "55003")
|
||||
sec3 = parsed["sections"]["sec_3"]
|
||||
assert (sec3.code_lo, sec3.code_hi) == ("55100", "55101")
|
||||
|
||||
def test_heading_without_range_is_empty(self, parsed):
|
||||
sec4 = parsed["sections"]["sec_4"]
|
||||
assert (sec4.code_lo, sec4.code_hi) == ("", "")
|
||||
|
||||
def test_section_with_no_codes_still_has_a_path(self, parsed):
|
||||
sec4 = parsed["sections"]["sec_4"]
|
||||
assert sec4.path
|
||||
assert sec4.guideline
|
||||
|
||||
def test_entities_and_whitespace_cleaned(self, parsed):
|
||||
sec1 = parsed["sections"]["sec_1"]
|
||||
assert "&" not in sec1.guideline
|
||||
assert "&" in sec1.guideline # entity decoded, not dropped
|
||||
assert " " not in sec1.guideline
|
||||
|
||||
|
||||
class TestSecIdFallback:
|
||||
""" "No Id Category Services" (the fixture's last h1) has no ``id``
|
||||
attribute and no nested anchor carrying one — same as most 2019
|
||||
headings. It must still get a unique, non-empty ``sec_id`` (never
|
||||
collapse onto ``""`` alongside every other id-less heading), and
|
||||
the code under it must carry that same id."""
|
||||
|
||||
def test_no_id_heading_gets_a_synthetic_sec_id(self, parsed):
|
||||
no_id_section = next(
|
||||
s for s in parsed["sections_list"] if s.title == "No Id Category Services"
|
||||
)
|
||||
assert no_id_section.sec_id != ""
|
||||
assert no_id_section.sec_id.startswith("cpt_sample.xhtml:h")
|
||||
|
||||
def test_synthetic_sec_ids_are_unique(self, parsed):
|
||||
synthetic = [
|
||||
s.sec_id
|
||||
for s in parsed["sections_list"]
|
||||
if s.sec_id.startswith("cpt_sample.xhtml:h")
|
||||
]
|
||||
assert len(synthetic) == len(set(synthetic))
|
||||
assert synthetic # at least the one id-less heading produced one
|
||||
|
||||
def test_code_under_id_less_heading_gets_the_synthetic_sec_id(self, parsed):
|
||||
no_id_section = next(
|
||||
s for s in parsed["sections_list"] if s.title == "No Id Category Services"
|
||||
)
|
||||
code = parsed["codes"]["57000"]
|
||||
assert code.sec_id == no_id_section.sec_id
|
||||
assert code.sec_id != ""
|
||||
|
||||
|
||||
class TestPrimaryCode:
|
||||
def test_three_elements_and_tail(self, parsed):
|
||||
code = parsed["codes"]["55000"]
|
||||
assert code.sec_id == "sec_2"
|
||||
assert len(code.elements) == 3
|
||||
assert code.elements[0] == "first invented element of the service,"
|
||||
assert (
|
||||
code.tail == "first 20 minutes of invented staff time, per calendar month."
|
||||
)
|
||||
assert (
|
||||
code.stem
|
||||
== "Sample wellness coordination services with the following required elements:"
|
||||
)
|
||||
|
||||
def test_symbols(self, parsed):
|
||||
code = parsed["codes"]["55000"]
|
||||
assert code.resequenced is True
|
||||
assert code.addon is False
|
||||
assert code.new is False
|
||||
assert code.revised is False
|
||||
assert code.telemedicine is False
|
||||
assert code.parent == ""
|
||||
|
||||
def test_category_i(self, parsed):
|
||||
assert parsed["codes"]["55000"].category == "I"
|
||||
|
||||
def test_descriptor_assembled(self, parsed):
|
||||
code = parsed["codes"]["55000"]
|
||||
assert code.stem in code.descriptor
|
||||
assert code.tail in code.descriptor
|
||||
assert code.elements[0].rstrip(",;") in code.descriptor
|
||||
|
||||
def test_descriptor_element_join_has_no_double_punctuation(self, parsed):
|
||||
code = parsed["codes"]["55000"]
|
||||
assert ",;" not in code.descriptor
|
||||
assert ";;" not in code.descriptor
|
||||
|
||||
|
||||
class TestSemicolonRule:
|
||||
"""25100 "Arthrotomy, wrist joint; with biopsy" / 25105 "with
|
||||
synovectomy" is the book's own worked example of the indentation
|
||||
convention (Introduction, "Format of the Terminology"): an
|
||||
indented child's descriptor is the parent's pre-semicolon stem
|
||||
plus the child's own fragment — never the parent's *whole* stem
|
||||
(which would duplicate the parent's own post-semicolon text)."""
|
||||
|
||||
def test_parent_stem_excludes_own_continuation(self, parsed):
|
||||
parent = parsed["codes"]["56000"]
|
||||
assert parent.stem == "Sample incision, deep fascial plane"
|
||||
assert "with exploratory biopsy" not in parent.stem
|
||||
|
||||
def test_parent_descriptor_has_the_continuation(self, parsed):
|
||||
parent = parsed["codes"]["56000"]
|
||||
assert (
|
||||
parent.descriptor
|
||||
== "Sample incision, deep fascial plane; with exploratory biopsy"
|
||||
)
|
||||
|
||||
def test_child_inherits_only_the_shared_stem(self, parsed):
|
||||
child = parsed["codes"]["56005"]
|
||||
assert child.stem == "Sample incision, deep fascial plane"
|
||||
assert child.parent == "56000"
|
||||
|
||||
def test_child_descriptor_substitutes_its_own_fragment(self, parsed):
|
||||
child = parsed["codes"]["56005"]
|
||||
assert (
|
||||
child.descriptor
|
||||
== "Sample incision, deep fascial plane; with synovial debridement"
|
||||
)
|
||||
# the parent's own fragment must not leak into the child's descriptor
|
||||
assert "exploratory biopsy" not in child.descriptor
|
||||
|
||||
|
||||
class TestAddonCode:
|
||||
def test_addon_flag_and_parent(self, parsed):
|
||||
code = parsed["codes"]["55001"]
|
||||
assert code.addon is True
|
||||
assert code.resequenced is True
|
||||
assert code.parent == "55000"
|
||||
|
||||
def test_inherits_stem_and_elements(self, parsed):
|
||||
addon = parsed["codes"]["55001"]
|
||||
primary = parsed["codes"]["55000"]
|
||||
assert addon.stem == primary.stem
|
||||
assert addon.elements == primary.elements
|
||||
|
||||
def test_own_tail(self, parsed):
|
||||
addon = parsed["codes"]["55001"]
|
||||
assert "each additional 20 minutes" in addon.tail
|
||||
assert "primary procedure" in addon.tail
|
||||
|
||||
|
||||
class TestInstructions:
|
||||
def test_use_with_targets_primary(self, parsed):
|
||||
use_with = [
|
||||
i
|
||||
for i in parsed["instructions"]
|
||||
if i.code == "55001" and i.kind == "use-with"
|
||||
]
|
||||
assert len(use_with) == 1
|
||||
assert use_with[0].targets == ("55000",)
|
||||
|
||||
def test_not_with_targets_expanded_range_plus_listed(self, parsed):
|
||||
not_with = [
|
||||
i
|
||||
for i in parsed["instructions"]
|
||||
if i.code == "55001" and i.kind == "not-with"
|
||||
]
|
||||
assert len(not_with) == 1
|
||||
targets = not_with[0].targets
|
||||
expanded_range = tuple(f"909{n}" for n in range(51, 71))
|
||||
assert len(expanded_range) == 20
|
||||
for t in expanded_range:
|
||||
assert t in targets
|
||||
assert "55555" in targets
|
||||
|
||||
def test_plain_parenthetical_is_other(self, parsed):
|
||||
others = [
|
||||
i for i in parsed["instructions"] if i.code == "55001" and i.kind == "other"
|
||||
]
|
||||
assert len(others) == 3
|
||||
assert others[0].text.startswith("(Sample wellness services")
|
||||
|
||||
def test_five_instructions_on_addon(self, parsed):
|
||||
addon_instructions = [i for i in parsed["instructions"] if i.code == "55001"]
|
||||
assert len(addon_instructions) == 5
|
||||
|
||||
def test_owner_code_excluded_from_targets(self, parsed):
|
||||
# "(Do not report 55001 more than twice per calendar month)" —
|
||||
# 55001 is the instruction's own owning code (a subject), not
|
||||
# a target of itself.
|
||||
matches = [
|
||||
i
|
||||
for i in parsed["instructions"]
|
||||
if i.code == "55001" and "more than twice" in i.text
|
||||
]
|
||||
assert len(matches) == 1
|
||||
assert matches[0].kind == "other"
|
||||
assert matches[0].targets == ()
|
||||
|
||||
def test_sibling_code_kept_as_target(self, parsed):
|
||||
# "(Do not report 55001 in addition to 55002 more than once
|
||||
# per calendar month)" — 55001 is the owner (excluded), 55002
|
||||
# is a genuine sibling code and must survive the owner filter.
|
||||
matches = [
|
||||
i
|
||||
for i in parsed["instructions"]
|
||||
if i.code == "55001" and "in addition to 55002" in i.text
|
||||
]
|
||||
assert len(matches) == 1
|
||||
assert matches[0].targets == ("55002",)
|
||||
|
||||
def test_not_with_never_includes_the_owner(self, parsed):
|
||||
not_with = [
|
||||
i
|
||||
for i in parsed["instructions"]
|
||||
if i.code == "55001" and i.kind == "not-with"
|
||||
]
|
||||
assert "55001" not in not_with[0].targets
|
||||
assert "55555" in not_with[0].targets
|
||||
|
||||
|
||||
class TestResequencedPlaceholder:
|
||||
"""C8 (task 3): a resequenced code prints twice — a bare pointer row
|
||||
at its old numeric position ("Code is out of numerical sequence.
|
||||
See 55010-55019") plus its real entry (with elements/tail/refs)
|
||||
under its real section. Only the real entry may survive."""
|
||||
|
||||
def test_placeholder_row_is_not_emitted(self, parsed):
|
||||
matches = [c for c in parsed["codes_list"] if c.code == "55000"]
|
||||
assert len(matches) == 1
|
||||
|
||||
def test_surviving_row_is_the_real_entry_under_its_real_section(self, parsed):
|
||||
code = parsed["codes"]["55000"]
|
||||
assert code.sec_id == "sec_2"
|
||||
assert code.elements # only the real entry has elements/tail
|
||||
assert "out of numerical sequence" not in code.descriptor.lower()
|
||||
|
||||
|
||||
class TestReferences:
|
||||
def test_cpt_changes_years(self, parsed):
|
||||
refs = [
|
||||
r
|
||||
for r in parsed["references"]
|
||||
if r.code == "55000" and r.kind == "cpt-changes"
|
||||
]
|
||||
assert len(refs) == 1
|
||||
assert refs[0].years == (2015, 2021, 2022)
|
||||
|
||||
def test_cpt_assistant_kept_as_text(self, parsed):
|
||||
refs = [
|
||||
r
|
||||
for r in parsed["references"]
|
||||
if r.code == "55000" and r.kind == "cpt-assistant"
|
||||
]
|
||||
assert len(refs) == 1
|
||||
assert "Jan 21:5" in refs[0].text
|
||||
|
||||
def test_addon_reference_year(self, parsed):
|
||||
refs = [
|
||||
r
|
||||
for r in parsed["references"]
|
||||
if r.code == "55001" and r.kind == "cpt-changes"
|
||||
]
|
||||
assert refs[0].years == (2022,)
|
||||
|
||||
|
||||
# --- C9/C10: guideline-reprint / cross-reference duplicates --------------
|
||||
#
|
||||
# Unlike C8's resequenced placeholder (a bare pointer row with no real
|
||||
# descriptor, dropped entirely by the parser), C9 handles a code that
|
||||
# prints *twice with a real descriptor both times* — once inside a
|
||||
# guideline "convenience" table (no TOC code range of its own) and once
|
||||
# under its real, ranged home section. This can only be exercised at
|
||||
# the ``parse_epub`` (edition) level, since the resolution runs after
|
||||
# every chapter has been parsed — so these tests build a tiny synthetic
|
||||
# EPUB rather than using ``cpt_sample.xhtml``/``parse_xhtml``.
|
||||
#
|
||||
# C10 corrected a structural assumption: the fixture below now uses the
|
||||
# book's own ``div.ch-title`` chapter banners (one per synthetic chapter
|
||||
# file) rather than hand-nested ``h1``/``h2``, so it exercises the real
|
||||
# mechanism — a chapter banner parents every heading in its own chapter
|
||||
# and every later chapter that carries no banner of its own — the same
|
||||
# mechanism the real book's "Surgery Guidelines" (Chapter05) / "Surgery"
|
||||
# (Chapter06) / "Other Procedures" (Chapter07, no ch-title of its own)
|
||||
# split relies on.
|
||||
|
||||
|
||||
def _build_epub(path: Path, chapters: dict[str, str]) -> None:
|
||||
"""``chapters``: ``{"Chapter01.xhtml": body, "Chapter02.xhtml": body,
|
||||
…}`` — each becomes its own ``_chapter_groups`` group (a distinct
|
||||
``parse_epub`` chapter-group call), so ``chapter_title`` threading
|
||||
across groups can be exercised."""
|
||||
with zipfile.ZipFile(path, "w") as zf:
|
||||
zf.writestr("mimetype", "application/epub+zip")
|
||||
for name, body in chapters.items():
|
||||
zf.writestr(f"OPS/{name}", f"<html><body>{body}</body></html>")
|
||||
|
||||
|
||||
# Chapter01: the "Surgery Guidelines" banner — a guideline table
|
||||
# reprinting codes with no TOC range of their own.
|
||||
_C9_CH1 = """
|
||||
<div class="ch-title">Surgery Guidelines</div>
|
||||
<div class="h1" id="sec_g2">Unlisted Service or Procedure</div>
|
||||
<div class="noindent">The “Unlisted Procedures” and accompanying codes are as follows:</div>
|
||||
<table class="table1"><tbody>
|
||||
<tr>
|
||||
<td class="td-w1"><div class="table-para"><b>54321</b></div></td>
|
||||
<td class="td"><div class="table-para1">Unlisted procedure, sample</div></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="td-w1"><div class="table-para"><b>54322</b></div></td>
|
||||
<td class="td"><div class="table-para1">Unlisted procedure, only ever printed in the guidelines</div></td>
|
||||
</tr>
|
||||
</tbody></table>
|
||||
"""
|
||||
|
||||
# Chapter02: a *different* banner, "Surgery" — the real, ranged home.
|
||||
_C9_CH2 = """
|
||||
<div class="h1-toc"><a href="body.xhtml#sec_r"><span class="green">Sample Procedures* (54321-54329)</span></a></div>
|
||||
<div class="ch-title">Surgery</div>
|
||||
<div class="h1" id="sec_r">Sample Procedures</div>
|
||||
<div class="noindent">Guideline text for the real, ranged section.</div>
|
||||
<table class="table1"><tbody>
|
||||
<tr>
|
||||
<td class="td-w1" id="code_54321"><div class="table-para"><b>54321</b></div></td>
|
||||
<td class="td"><div class="table-para1">Unlisted procedure, sample with the following required elements:</div>
|
||||
<div class="table-slist">first element;</div>
|
||||
<div class="table-RT"><span class="blue"><span class="ama-en">➲</span></span><i>CPT Changes: An Insider's View</i> 2020</div></td>
|
||||
</tr>
|
||||
</tbody></table>
|
||||
"""
|
||||
|
||||
# Chapter03: carries *no* ch-title of its own at all — proves the
|
||||
# "Surgery" banner from Chapter02 keeps parenting headings until a new
|
||||
# banner replaces it (exactly 2019's Surgery body, Chapter07-16, which
|
||||
# has no ch-title of its own and inherits Chapter06's "Surgery").
|
||||
_C9_CH3 = """
|
||||
<div class="h1" id="sec_o">Other Procedures</div>
|
||||
<table class="table1"><tbody>
|
||||
<tr>
|
||||
<td class="td-w1" id="code_54323"><div class="table-para"><b>54323</b></div></td>
|
||||
<td class="td"><div class="table-para1">Unlisted procedure, no ch-title of its own</div></td>
|
||||
</tr>
|
||||
</tbody></table>
|
||||
"""
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def c9_edition(tmp_path_factory):
|
||||
path = tmp_path_factory.mktemp("c9") / "sample.epub"
|
||||
_build_epub(
|
||||
path,
|
||||
{
|
||||
"Chapter01.xhtml": _C9_CH1,
|
||||
"Chapter02.xhtml": _C9_CH2,
|
||||
"Chapter03.xhtml": _C9_CH3,
|
||||
},
|
||||
)
|
||||
return parse_epub(path, year=2024)
|
||||
|
||||
|
||||
class TestChapterTitleParenting:
|
||||
"""C10: div.ch-title is the level-0 parent of every heading in its
|
||||
own chapter-group and every later one that carries no banner of its
|
||||
own, replaced only by the next ch-title."""
|
||||
|
||||
def test_heading_nests_under_its_own_chapter_banner(self, c9_edition):
|
||||
sec = next(s for s in c9_edition.sections if s.sec_id == "sec_g2")
|
||||
assert sec.path == ("Surgery Guidelines", "Unlisted Service or Procedure")
|
||||
|
||||
def test_a_new_banner_replaces_the_previous_one(self, c9_edition):
|
||||
sec = next(s for s in c9_edition.sections if s.sec_id == "sec_r")
|
||||
assert sec.path == ("Surgery", "Sample Procedures")
|
||||
|
||||
def test_chapter_with_no_banner_of_its_own_inherits_the_previous_one(
|
||||
self, c9_edition
|
||||
):
|
||||
# Chapter03 has no div.ch-title at all — "Surgery" (Chapter02's
|
||||
# banner) must still parent its heading.
|
||||
sec = next(s for s in c9_edition.sections if s.sec_id == "sec_o")
|
||||
assert sec.path == ("Surgery", "Other Procedures")
|
||||
|
||||
|
||||
class TestEmptyParseRaisesF4:
|
||||
"""F4: an empty parse must never reach write_cpt_edition's
|
||||
delete-then-insert silently — parse_epub raises instead."""
|
||||
|
||||
def test_no_chapter_files_at_all_raises(self, tmp_path):
|
||||
path = tmp_path / "empty.epub"
|
||||
_build_epub(path, {}) # mimetype only, no OPS/ChapterNN.xhtml
|
||||
with pytest.raises(ValueError, match="no chapter content found"):
|
||||
parse_epub(path, year=2024)
|
||||
|
||||
def test_files_present_but_none_named_chapter_raises(self, tmp_path):
|
||||
path = tmp_path / "no_chapters.epub"
|
||||
_build_epub(path, {"Frontmatter.xhtml": "<p>Not a chapter file.</p>"})
|
||||
with pytest.raises(ValueError, match="no chapter content found"):
|
||||
parse_epub(path, year=2024)
|
||||
|
||||
|
||||
class TestContentRootResolution:
|
||||
"""F2/F4: cpt_epub resolves the OPS/ content-file prefix via
|
||||
container.xml -> the OPF's own path, not a hard-coded string —
|
||||
falling back to "OPS/" (every edition checked so far, and what
|
||||
``_build_epub``'s fixtures use with no container.xml at all)."""
|
||||
|
||||
def test_falls_back_to_ops_without_a_container_xml(self, tmp_path):
|
||||
path = tmp_path / "sample.epub"
|
||||
_build_epub(path, {"Chapter01.xhtml": _C9_CH2})
|
||||
edition = parse_epub(path, year=2024)
|
||||
assert any(c.code == "54321" for c in edition.codes)
|
||||
|
||||
def test_resolves_a_non_ops_content_root_via_container_xml(self, tmp_path):
|
||||
# A differently-templated EPUB whose content root isn't "OPS/" —
|
||||
# before F2, this silently parsed to nothing.
|
||||
path = tmp_path / "other_root.epub"
|
||||
with zipfile.ZipFile(path, "w") as zf:
|
||||
zf.writestr("mimetype", "application/epub+zip")
|
||||
zf.writestr(
|
||||
"META-INF/container.xml",
|
||||
'<?xml version="1.0"?>\n'
|
||||
'<container version="1.0" '
|
||||
'xmlns="urn:oasis:names:tc:opendocument:xmlns:container">\n'
|
||||
"<rootfiles>\n"
|
||||
'<rootfile full-path="EPUB/content.opf" '
|
||||
'media-type="application/oebps-package+xml"/>\n'
|
||||
"</rootfiles>\n</container>",
|
||||
)
|
||||
zf.writestr(
|
||||
"EPUB/content.opf",
|
||||
'<?xml version="1.0"?>\n'
|
||||
'<package xmlns="http://www.idpf.org/2007/opf" version="3.0">\n'
|
||||
"<manifest/><spine/>\n</package>",
|
||||
)
|
||||
zf.writestr(
|
||||
"EPUB/Chapter01.xhtml",
|
||||
f"<html><body>{_C9_CH2}</body></html>",
|
||||
)
|
||||
edition = parse_epub(path, year=2024)
|
||||
assert any(c.code == "54321" for c in edition.codes)
|
||||
|
||||
|
||||
class TestAlternates:
|
||||
def test_ranged_entry_wins_as_canonical(self, c9_edition):
|
||||
matches = [c for c in c9_edition.codes if c.code == "54321"]
|
||||
assert len(matches) == 1
|
||||
assert matches[0].sec_id == "sec_r"
|
||||
assert matches[0].elements # the real entry, not the bare guideline row
|
||||
|
||||
def test_guideline_entry_becomes_an_alternate(self, c9_edition):
|
||||
alts = [a for a in c9_edition.alternates if a.code == "54321"]
|
||||
assert len(alts) == 1
|
||||
assert alts[0].sec_id == "sec_g2"
|
||||
assert alts[0].reason == "guidelines-reprint"
|
||||
|
||||
def test_code_only_in_guidelines_is_kept_as_canonical(self, c9_edition):
|
||||
matches = [c for c in c9_edition.codes if c.code == "54322"]
|
||||
assert len(matches) == 1
|
||||
assert matches[0].sec_id == "sec_g2"
|
||||
alts = [a for a in c9_edition.alternates if a.code == "54322"]
|
||||
assert alts == []
|
||||
|
||||
def test_c8_assertion_would_now_pass(self, c9_edition):
|
||||
# write_cpt_edition's own "one row per (edition_year, code)"
|
||||
# check (C8) — the whole point of C9 is that this always holds.
|
||||
from collections import Counter
|
||||
|
||||
counts = Counter(c.code for c in c9_edition.codes)
|
||||
assert all(n == 1 for n in counts.values())
|
||||
|
||||
|
||||
# --- Real-file integration tests (skipped when the file is absent) ---
|
||||
|
||||
|
||||
@pytest.mark.skipif(not EPUB_2024.exists(), reason="real 2024 CPT EPUB not on disk")
|
||||
class TestReal2024:
|
||||
@pytest.fixture(scope="class")
|
||||
def edition(self):
|
||||
return parse_epub(EPUB_2024, year=2024)
|
||||
|
||||
def test_at_least_8000_codes(self, edition):
|
||||
assert len(edition.codes) >= 8000
|
||||
|
||||
def test_every_section_has_a_path(self, edition):
|
||||
assert edition.sections
|
||||
for sec in edition.sections:
|
||||
assert sec.path
|
||||
assert sec.path[-1] == sec.title
|
||||
|
||||
def test_chronic_care_management_section_has_expected_codes(self, edition):
|
||||
by_code = {c.code: c for c in edition.codes}
|
||||
sec = next(
|
||||
s for s in edition.sections if s.title == "Chronic Care Management Services"
|
||||
)
|
||||
codes_in_sec = {c.code for c in edition.codes if c.sec_id == sec.sec_id}
|
||||
for code in ("99490", "99439", "99491", "99437"):
|
||||
assert code in codes_in_sec, code
|
||||
assert code in by_code
|
||||
|
||||
def test_99439_is_addon_using_99490(self, edition):
|
||||
by_code = {c.code: c for c in edition.codes}
|
||||
code_99439 = by_code["99439"]
|
||||
assert code_99439.addon is True
|
||||
use_with = [
|
||||
i
|
||||
for i in edition.instructions
|
||||
if i.code == "99439" and i.kind == "use-with"
|
||||
]
|
||||
assert use_with
|
||||
assert "99490" in use_with[0].targets
|
||||
|
||||
def test_appendix_d_addon_codes(self, edition):
|
||||
d_entries = [e for e in edition.lists if e.appendix == "D"]
|
||||
assert len(d_entries) >= 500
|
||||
|
||||
def test_appendix_m_crosswalk_former_codes_end_in_t(self, edition):
|
||||
assert edition.crosswalks
|
||||
assert any(row.former_code.endswith("T") for row in edition.crosswalks)
|
||||
|
||||
def test_25100_25105_semicolon_rule(self, edition):
|
||||
# The book's own worked example (Introduction, "Format of the
|
||||
# Terminology"): 25100 "Arthrotomy, wrist joint; with biopsy"
|
||||
# / 25105 "with synovectomy". Structure only — never asserts
|
||||
# or prints the actual descriptor text.
|
||||
by_code = {c.code: c for c in edition.codes}
|
||||
parent = by_code["25100"]
|
||||
child = by_code["25105"]
|
||||
assert child.parent == "25100"
|
||||
assert child.stem == parent.stem
|
||||
assert child.descriptor.startswith(parent.stem)
|
||||
parent_own_fragment = parent.descriptor[len(parent.stem) :].lstrip("; ").strip()
|
||||
assert parent_own_fragment # the parent really did have a post-semicolon part
|
||||
assert parent_own_fragment not in child.descriptor
|
||||
|
||||
def test_no_code_has_an_empty_sec_id(self, edition):
|
||||
assert all(c.sec_id != "" for c in edition.codes)
|
||||
|
||||
def test_c9_zero_codes_with_more_than_one_canonical_row(self, edition):
|
||||
# C9: every code the real book prints more than once (guideline
|
||||
# "Unlisted Service or Procedure" summary tables, "Qualifying
|
||||
# Circumstances for Anesthesia" cross-reference reprints, and
|
||||
# section-specific variants of the same pattern) must resolve
|
||||
# to exactly one canonical pfs.cpt_code row — write_cpt_edition's
|
||||
# own C8 assertion depends on this.
|
||||
from collections import Counter
|
||||
|
||||
counts = Counter(c.code for c in edition.codes)
|
||||
dupes = sorted(code for code, n in counts.items() if n > 1)
|
||||
assert dupes == [], f"duplicate canonical codes: {dupes}"
|
||||
# Not a hard-coded expectation of the real book's content (that
|
||||
# would be fragile) — just proof the mechanism actually engaged
|
||||
# on real data, and a number for the report.
|
||||
assert len(edition.alternates) > 100
|
||||
print(f"2024 alternates: {len(edition.alternates)}")
|
||||
|
||||
def test_c10_21089_resolves_under_surgery(self, edition):
|
||||
# C10 regression (structure only, no descriptor text): 21089 is
|
||||
# printed twice — a bare "Unlisted procedure" row inside the
|
||||
# Surgery Guidelines chapter's "Unlisted Service or Procedure"
|
||||
# convenience table, and its real entry under its actual
|
||||
# subsection. The chapter-title-parents-headings fix must land
|
||||
# the canonical row under "Surgery > …", and the guideline copy
|
||||
# in cpt_code_alt with reason "guidelines-reprint".
|
||||
by_code = {c.code: c for c in edition.codes}
|
||||
sec_by_id = {s.sec_id: s for s in edition.sections}
|
||||
canonical = by_code["21089"]
|
||||
sec = sec_by_id[canonical.sec_id]
|
||||
assert sec.path[0] == "Surgery"
|
||||
assert sec.path[-1] != "Unlisted Service or Procedure"
|
||||
|
||||
alts = [a for a in edition.alternates if a.code == "21089"]
|
||||
assert len(alts) == 1
|
||||
alt_sec = sec_by_id[alts[0].sec_id]
|
||||
assert alt_sec.path == ("Surgery Guidelines", "Unlisted Service or Procedure")
|
||||
assert alts[0].reason == "guidelines-reprint"
|
||||
|
||||
def test_c10_99100_resolves_to_the_ranged_medicine_listing(self, edition):
|
||||
# C10 regression / documented accepted outcome (see parse_epub's
|
||||
# docstring): 99100 (a Qualifying Circumstances for Anesthesia
|
||||
# add-on) prints under Anesthesia's own "Qualifying
|
||||
# Circumstances" guideline *and* under Medicine's "Qualifying
|
||||
# Circumstances for Anesthesia" heading, which carries the
|
||||
# book's own TOC-declared numeric range (99100-99140) — C9's
|
||||
# scoring picks the ranged entry as canonical regardless of
|
||||
# which chapter "feels" more natural, and that's intentional.
|
||||
by_code = {c.code: c for c in edition.codes}
|
||||
sec_by_id = {s.sec_id: s for s in edition.sections}
|
||||
canonical = by_code["99100"]
|
||||
sec = sec_by_id[canonical.sec_id]
|
||||
assert sec.path == ("Medicine", "Qualifying Circumstances for Anesthesia")
|
||||
|
||||
alts = [a for a in edition.alternates if a.code == "99100"]
|
||||
assert len(alts) == 1
|
||||
alt_sec = sec_by_id[alts[0].sec_id]
|
||||
assert alt_sec.path == ("Anesthesia Guidelines", "Qualifying Circumstances")
|
||||
assert alts[0].reason == "guidelines-reprint"
|
||||
|
||||
def test_c10_99490_path_key(self, edition):
|
||||
by_code = {c.code: c for c in edition.codes}
|
||||
sec_by_id = {s.sec_id: s for s in edition.sections}
|
||||
sec = sec_by_id[by_code["99490"].sec_id]
|
||||
assert sec.path == (
|
||||
"Evaluation and Management",
|
||||
"Care Management Services",
|
||||
"Chronic Care Management Services",
|
||||
)
|
||||
|
||||
def test_c10_alternate_reasons_include_guidelines_reprint(self, edition):
|
||||
from collections import Counter
|
||||
|
||||
reasons = Counter(a.reason for a in edition.alternates)
|
||||
assert reasons["guidelines-reprint"] > 0
|
||||
print(f"2024 cpt_code_alt reasons: {dict(reasons)}")
|
||||
|
||||
|
||||
@pytest.mark.skipif(not EPUB_2022.exists(), reason="real 2022 CPT EPUB not on disk")
|
||||
def test_real_2022_parses():
|
||||
edition = parse_epub(EPUB_2022, year=2022)
|
||||
assert len(edition.codes) >= 5000
|
||||
assert edition.sections
|
||||
|
||||
|
||||
@pytest.mark.skipif(not EPUB_2022.exists(), reason="real 2022 CPT EPUB not on disk")
|
||||
def test_real_2022_c9_zero_duplicate_canonical_codes():
|
||||
from collections import Counter
|
||||
|
||||
edition = parse_epub(EPUB_2022, year=2022)
|
||||
counts = Counter(c.code for c in edition.codes)
|
||||
dupes = sorted(code for code, n in counts.items() if n > 1)
|
||||
assert dupes == [], f"duplicate canonical codes: {dupes}"
|
||||
|
||||
|
||||
@pytest.mark.skipif(not EPUB_2021.exists(), reason="real 2021 CPT EPUB not on disk")
|
||||
def test_real_2021_parses():
|
||||
edition = parse_epub(EPUB_2021, year=2021)
|
||||
assert len(edition.codes) >= 5000
|
||||
assert edition.sections
|
||||
|
||||
|
||||
@pytest.mark.skipif(not EPUB_2021.exists(), reason="real 2021 CPT EPUB not on disk")
|
||||
def test_real_2021_c9_zero_duplicate_canonical_codes():
|
||||
from collections import Counter
|
||||
|
||||
edition = parse_epub(EPUB_2021, year=2021)
|
||||
counts = Counter(c.code for c in edition.codes)
|
||||
dupes = sorted(code for code, n in counts.items() if n > 1)
|
||||
assert dupes == [], f"duplicate canonical codes: {dupes}"
|
||||
|
||||
|
||||
@pytest.mark.skipif(not EPUB_2019.exists(), reason="real 2019 CPT EPUB not on disk")
|
||||
def test_real_2019_parses_without_code_ids():
|
||||
edition = parse_epub(EPUB_2019, year=2019)
|
||||
assert len(edition.codes) >= 3000
|
||||
assert edition.sections
|
||||
# 2019 headings mostly lack an id in the source markup — every
|
||||
# code must still land under a non-empty (real or synthetic) sec_id.
|
||||
assert all(c.sec_id != "" for c in edition.codes)
|
||||
|
||||
|
||||
@pytest.mark.skipif(not EPUB_2019.exists(), reason="real 2019 CPT EPUB not on disk")
|
||||
def test_real_2019_c9_zero_duplicate_canonical_codes():
|
||||
from collections import Counter
|
||||
|
||||
edition = parse_epub(EPUB_2019, year=2019)
|
||||
counts = Counter(c.code for c in edition.codes)
|
||||
dupes = sorted(code for code, n in counts.items() if n > 1)
|
||||
assert dupes == [], f"duplicate canonical codes: {dupes}"
|
||||
283
tests/pfs/test_cpt_load.py
Normal file
283
tests/pfs/test_cpt_load.py
Normal file
@@ -0,0 +1,283 @@
|
||||
"""Tests for pfs.cpt_load — CPT EPUB editions -> pfs.cpt_* (task 3, #687)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
import duckdb
|
||||
import pytest
|
||||
|
||||
from bib.item import Item
|
||||
from bib.store import Store
|
||||
from pfs.cpt_load import find_editions, ingest
|
||||
|
||||
_CHAPTER_2024 = """
|
||||
<div class="h1" id="sec_1">Care Management Services</div>
|
||||
<div class="h2" id="sec_2">Chronic Care Management Services</div>
|
||||
<div class="noindent">Sample guideline text.</div>
|
||||
<table class="table1"><tbody>
|
||||
<tr>
|
||||
<td class="td-w1" id="code_99490"><div class="table-para"><b>99490</b></div></td>
|
||||
<td class="td"><div class="table-para1">Chronic care management services with the following required elements:</div>
|
||||
<div class="table-slist">first element;</div>
|
||||
<div class="table-para2">first 20 minutes of clinical staff time.</div></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="td-w1" id="code_99439"><div class="table-para"><span class="ama-en">✚</span> <b>99439</b></div></td>
|
||||
<td class="td"><div class="table-para1-sub">each additional 20 minutes of clinical staff time (List separately in addition to code for primary procedure)</div>
|
||||
<div class="table-para2">(Use 99439 in conjunction with 99490)</div></td>
|
||||
</tr>
|
||||
</tbody></table>
|
||||
"""
|
||||
|
||||
_CHAPTER_2021 = """
|
||||
<div class="h1" id="sec_1">Care Management Services</div>
|
||||
<div class="h2" id="sec_2">Chronic Care Management Services</div>
|
||||
<div class="noindent">Sample guideline text.</div>
|
||||
<table class="table1"><tbody>
|
||||
<tr>
|
||||
<td class="td-w1" id="code_99490"><div class="table-para"><b>99490</b></div></td>
|
||||
<td class="td"><div class="table-para1">Chronic care management services with the following required elements:</div>
|
||||
<div class="table-slist">first element;</div>
|
||||
<div class="table-para2">first 20 minutes of clinical staff time.</div></td>
|
||||
</tr>
|
||||
</tbody></table>
|
||||
"""
|
||||
|
||||
|
||||
def _build_epub(path: Path, chapter_body: str) -> None:
|
||||
with zipfile.ZipFile(path, "w") as zf:
|
||||
zf.writestr("mimetype", "application/epub+zip")
|
||||
zf.writestr("OPS/Chapter01.xhtml", f"<html><body>{chapter_body}</body></html>")
|
||||
|
||||
|
||||
def _store(tmp_path) -> Store:
|
||||
return Store(tmp_path / "bib.sqlite3", storage_dir=tmp_path / "storage")
|
||||
|
||||
|
||||
def _add_edition(
|
||||
store: Store, tmp_path: Path, *, title: str, filename: str, chapter_body: str
|
||||
) -> str:
|
||||
item_key = store.create(Item(title=title, item_type="book"), tags=["source:ama"])
|
||||
src = tmp_path / filename
|
||||
_build_epub(src, chapter_body)
|
||||
store.attach_file(item_key, src, title=filename)
|
||||
return item_key
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def store_with_editions(tmp_path):
|
||||
s = _store(tmp_path)
|
||||
key_2024 = _add_edition(
|
||||
s,
|
||||
tmp_path,
|
||||
title="CPT Professional 2024 - American Medical Association",
|
||||
filename="CPT Professional 2024 - American Medical Association.epub",
|
||||
chapter_body=_CHAPTER_2024,
|
||||
)
|
||||
key_2021 = _add_edition(
|
||||
s,
|
||||
tmp_path,
|
||||
title="CPT 2021 Professional Edition",
|
||||
filename="CPT 2021 Professional Edition.epub",
|
||||
chapter_body=_CHAPTER_2021,
|
||||
)
|
||||
# A 2024 PDF sibling attachment — never matched (no .epub suffix).
|
||||
pdf = tmp_path / "CPT Professional 2024.pdf"
|
||||
pdf.write_bytes(b"%PDF-1.4 not a real pdf")
|
||||
s.attach_file(key_2024, pdf, title="CPT Professional 2024.pdf")
|
||||
# The 2023 "CPT Changes" book — tagged source:ama, has an .epub, but
|
||||
# its title never matches the edition pattern (C2: skip).
|
||||
key_changes = s.create(
|
||||
Item(title="CPT Changes 2023", item_type="book"), tags=["source:ama"]
|
||||
)
|
||||
changes_epub = tmp_path / "CPT Changes 2023.epub"
|
||||
_build_epub(changes_epub, _CHAPTER_2021)
|
||||
s.attach_file(key_changes, changes_epub, title="CPT Changes 2023.epub")
|
||||
# Netter's Atlas — tagged source:ama, has an .epub, title doesn't
|
||||
# start with "CPT" at all.
|
||||
key_netter = s.create(
|
||||
Item(title="Netter's Atlas for CPT Coding 2015", item_type="book"),
|
||||
tags=["source:ama"],
|
||||
)
|
||||
netter_epub = tmp_path / "Netter.epub"
|
||||
_build_epub(netter_epub, _CHAPTER_2021)
|
||||
s.attach_file(key_netter, netter_epub, title="Netter.epub")
|
||||
# 2018 PDF-only edition — tagged source:ama, title matches, but no
|
||||
# .epub attachment at all (C2: skip).
|
||||
key_2018 = s.create(Item(title="CPT 2018", item_type="book"), tags=["source:ama"])
|
||||
pdf_2018 = tmp_path / "CPT 2018.pdf"
|
||||
pdf_2018.write_bytes(b"%PDF-1.4 not a real pdf")
|
||||
s.attach_file(key_2018, pdf_2018, title="CPT 2018.pdf")
|
||||
# An item tagged something else entirely, with an .epub and a
|
||||
# matching-looking title — must never surface (wrong tag).
|
||||
key_other = s.create(
|
||||
Item(title="CPT Professional 2099", item_type="book"), tags=["module:coding"]
|
||||
)
|
||||
other_epub = tmp_path / "CPT Professional 2099.epub"
|
||||
_build_epub(other_epub, _CHAPTER_2021)
|
||||
s.attach_file(key_other, other_epub, title="CPT Professional 2099.epub")
|
||||
|
||||
yield s, key_2024, key_2021
|
||||
s.close()
|
||||
|
||||
|
||||
class TestTitleMismatchWarnsF4:
|
||||
def test_unrecognized_title_logs_a_warning_naming_the_item(self, tmp_path, caplog):
|
||||
s = _store(tmp_path)
|
||||
key = s.create(
|
||||
Item(title="Netter's Atlas for CPT Coding 2015", item_type="book"),
|
||||
tags=["source:ama"],
|
||||
)
|
||||
epub = tmp_path / "Netter.epub"
|
||||
_build_epub(epub, _CHAPTER_2021)
|
||||
s.attach_file(key, epub, title="Netter.epub")
|
||||
try:
|
||||
with caplog.at_level("WARNING", logger="pfs.cpt_load"):
|
||||
editions = find_editions(s)
|
||||
assert editions == []
|
||||
messages = [r.getMessage() for r in caplog.records]
|
||||
assert any(key in m for m in messages)
|
||||
assert any("Netter's Atlas" in m for m in messages)
|
||||
finally:
|
||||
s.close()
|
||||
|
||||
|
||||
class TestFindEditions:
|
||||
def test_finds_only_the_matching_epub_editions(self, store_with_editions):
|
||||
store, key_2024, key_2021 = store_with_editions
|
||||
editions = find_editions(store)
|
||||
assert [(year, key) for year, key, _ in editions] == [
|
||||
(2021, key_2021),
|
||||
(2024, key_2024),
|
||||
]
|
||||
|
||||
def test_paths_point_at_the_epub_not_the_pdf(self, store_with_editions):
|
||||
store, key_2024, _ = store_with_editions
|
||||
editions = find_editions(store)
|
||||
path = next(p for y, k, p in editions if k == key_2024)
|
||||
assert path.suffix == ".epub"
|
||||
|
||||
|
||||
class TestIngestDryRun:
|
||||
def test_dry_run_never_touches_con(self, store_with_editions):
|
||||
store, key_2024, key_2021 = store_with_editions
|
||||
out = ingest(store, None, dry_run=True)
|
||||
assert set(out) == {2021, 2024}
|
||||
assert out[2024] == {
|
||||
"sections": 2,
|
||||
"codes": 2,
|
||||
"instructions": 1,
|
||||
"references": 0,
|
||||
"crosswalks": 0,
|
||||
"lists": 0,
|
||||
"alternates": 0,
|
||||
}
|
||||
assert out[2021]["codes"] == 1
|
||||
|
||||
def test_dry_run_can_be_scoped_to_one_year(self, store_with_editions):
|
||||
store, _, _ = store_with_editions
|
||||
out = ingest(store, None, years=[2024], dry_run=True)
|
||||
assert set(out) == {2024}
|
||||
|
||||
def test_unknown_year_raises(self, store_with_editions):
|
||||
store, _, _ = store_with_editions
|
||||
with pytest.raises(ValueError, match="2099"):
|
||||
ingest(store, None, years=[2099], dry_run=True)
|
||||
|
||||
|
||||
class TestIngestWrite:
|
||||
def test_writes_rows_per_year(self, store_with_editions):
|
||||
store, key_2024, key_2021 = store_with_editions
|
||||
con = duckdb.connect(":memory:")
|
||||
try:
|
||||
out = ingest(store, con, dry_run=False)
|
||||
assert out[2024]["codes"] == 2
|
||||
assert out[2021]["codes"] == 1
|
||||
|
||||
rows = con.execute(
|
||||
"SELECT code, item_key FROM pfs.cpt_code WHERE edition_year = 2024 ORDER BY code"
|
||||
).fetchall()
|
||||
assert rows == [("99439", key_2024), ("99490", key_2024)]
|
||||
|
||||
rows_2021 = con.execute(
|
||||
"SELECT code, item_key FROM pfs.cpt_code WHERE edition_year = 2021"
|
||||
).fetchall()
|
||||
assert rows_2021 == [("99490", key_2021)]
|
||||
finally:
|
||||
con.close()
|
||||
|
||||
def test_scoped_to_one_year_leaves_others_absent(self, store_with_editions):
|
||||
store, key_2024, _ = store_with_editions
|
||||
con = duckdb.connect(":memory:")
|
||||
try:
|
||||
ingest(store, con, years=[2024], dry_run=False)
|
||||
years = con.execute(
|
||||
"SELECT DISTINCT edition_year FROM pfs.cpt_code"
|
||||
).fetchall()
|
||||
assert years == [(2024,)]
|
||||
finally:
|
||||
con.close()
|
||||
|
||||
|
||||
def _build_epub_with_no_chapters(path: Path) -> None:
|
||||
"""An EPUB with content files but none named ``ChapterNN`` — parses
|
||||
to zero chapter groups, so ``parse_epub`` raises (F4)."""
|
||||
with zipfile.ZipFile(path, "w") as zf:
|
||||
zf.writestr("mimetype", "application/epub+zip")
|
||||
zf.writestr(
|
||||
"OPS/Frontmatter.xhtml", "<html><body><p>Not a chapter.</p></body></html>"
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def store_with_one_good_one_bad_edition(tmp_path):
|
||||
s = _store(tmp_path)
|
||||
key_2024 = _add_edition(
|
||||
s,
|
||||
tmp_path,
|
||||
title="CPT Professional 2024 - American Medical Association",
|
||||
filename="CPT Professional 2024 - American Medical Association.epub",
|
||||
chapter_body=_CHAPTER_2024,
|
||||
)
|
||||
key_2021 = s.create(
|
||||
Item(title="CPT 2021 Professional Edition", item_type="book"),
|
||||
tags=["source:ama"],
|
||||
)
|
||||
bad_epub = tmp_path / "CPT 2021 Professional Edition.epub"
|
||||
_build_epub_with_no_chapters(bad_epub)
|
||||
s.attach_file(key_2021, bad_epub, title="CPT 2021 Professional Edition.epub")
|
||||
yield s, key_2024, key_2021
|
||||
s.close()
|
||||
|
||||
|
||||
class TestIngestPerEditionFailureF4:
|
||||
"""A CPT edition that fails to parse is a per-edition failure, not a
|
||||
whole-run one — ingest surfaces it and continues with the others."""
|
||||
|
||||
def test_write_surfaces_the_bad_edition_and_writes_the_good_one(
|
||||
self, store_with_one_good_one_bad_edition
|
||||
):
|
||||
store, key_2024, key_2021 = store_with_one_good_one_bad_edition
|
||||
con = duckdb.connect(":memory:")
|
||||
try:
|
||||
out = ingest(store, con, dry_run=False)
|
||||
assert out[2024]["codes"] == 2
|
||||
assert "error" in out[2021]
|
||||
assert "no chapter content found" in out[2021]["error"]
|
||||
|
||||
years = con.execute(
|
||||
"SELECT DISTINCT edition_year FROM pfs.cpt_code"
|
||||
).fetchall()
|
||||
assert years == [(2024,)]
|
||||
finally:
|
||||
con.close()
|
||||
|
||||
def test_dry_run_surfaces_the_bad_edition_too(
|
||||
self, store_with_one_good_one_bad_edition
|
||||
):
|
||||
store, key_2024, key_2021 = store_with_one_good_one_bad_edition
|
||||
out = ingest(store, None, dry_run=True)
|
||||
assert out[2024]["codes"] == 2
|
||||
assert "error" in out[2021]
|
||||
@@ -4,8 +4,11 @@ from __future__ import annotations
|
||||
|
||||
import sqlite3
|
||||
|
||||
import duckdb
|
||||
import pytest
|
||||
|
||||
from pfs.descriptors import DescriptorRun, Para, descriptor_runs
|
||||
from pfs.extract import Extraction, extract_run, extract_text
|
||||
from pfs.extract import Extraction, extract_code, extract_run, extract_text
|
||||
|
||||
STEM = Para(
|
||||
"JJ6AM5HJ",
|
||||
@@ -151,6 +154,207 @@ class TestAnchorIntegrity:
|
||||
con.close()
|
||||
|
||||
|
||||
def _sqlite_store_con():
|
||||
con = sqlite3.connect(":memory:")
|
||||
con.row_factory = sqlite3.Row
|
||||
con.executescript(
|
||||
"CREATE TABLE items (key TEXT PRIMARY KEY, title TEXT, date_published TEXT);"
|
||||
"CREATE TABLE fr_anchors (item_key TEXT, p_id INTEGER, page INTEGER, ordinal INTEGER, text TEXT);"
|
||||
)
|
||||
return con
|
||||
|
||||
|
||||
class _Store:
|
||||
def __init__(self, con):
|
||||
self._c = con
|
||||
|
||||
def _con(self):
|
||||
return self._c
|
||||
|
||||
|
||||
def _duckdb_con():
|
||||
c = duckdb.connect(":memory:")
|
||||
c.execute("CREATE SCHEMA pfs")
|
||||
c.execute(
|
||||
"CREATE TABLE pfs.rvu (hcpcs VARCHAR, mod VARCHAR, description VARCHAR, "
|
||||
"status_code VARCHAR, non_fac_total DOUBLE, year INTEGER)"
|
||||
)
|
||||
c.execute("CREATE SCHEMA terminology")
|
||||
c.execute(
|
||||
"CREATE TABLE terminology.hcpcs_level_2 (hcpcs VARCHAR, long_description VARCHAR, "
|
||||
"seqnum VARCHAR, recid VARCHAR)"
|
||||
)
|
||||
c.execute(
|
||||
"CREATE TABLE pfs.cpt_code (edition_year INTEGER, item_key VARCHAR, code VARCHAR, "
|
||||
"sec_id VARCHAR, category VARCHAR, descriptor VARCHAR, stem VARCHAR, "
|
||||
"elements VARCHAR[], tail VARCHAR, parent VARCHAR, addon BOOLEAN, resequenced BOOLEAN, "
|
||||
"new BOOLEAN, revised BOOLEAN, telemedicine BOOLEAN, mod51_exempt BOOLEAN, "
|
||||
"audio_only BOOLEAN, fda_pending BOOLEAN, pla BOOLEAN)"
|
||||
)
|
||||
return c
|
||||
|
||||
|
||||
def _duckdb_con_no_cpt():
|
||||
"""A replica shape that hasn't been through `cpt-ingest` yet: `pfs.rvu`
|
||||
and `terminology.hcpcs_level_2` exist, but no `pfs.cpt_*` table does —
|
||||
the read-only path deliberately never calls `ensure_tables` (I4)."""
|
||||
c = duckdb.connect(":memory:")
|
||||
c.execute("CREATE SCHEMA pfs")
|
||||
c.execute(
|
||||
"CREATE TABLE pfs.rvu (hcpcs VARCHAR, mod VARCHAR, description VARCHAR, "
|
||||
"status_code VARCHAR, non_fac_total DOUBLE, year INTEGER)"
|
||||
)
|
||||
c.execute("CREATE SCHEMA terminology")
|
||||
c.execute(
|
||||
"CREATE TABLE terminology.hcpcs_level_2 (hcpcs VARCHAR, long_description VARCHAR, "
|
||||
"seqnum VARCHAR, recid VARCHAR)"
|
||||
)
|
||||
return c
|
||||
|
||||
|
||||
def _insert_cpt_code(con, year, item_key, code, *, stem, elements, tail):
|
||||
con.execute(
|
||||
"INSERT INTO pfs.cpt_code VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
|
||||
[
|
||||
year,
|
||||
item_key,
|
||||
code,
|
||||
"sec1",
|
||||
"I",
|
||||
f"{stem}; {tail}",
|
||||
stem,
|
||||
list(elements),
|
||||
tail,
|
||||
"",
|
||||
False,
|
||||
False,
|
||||
False,
|
||||
False,
|
||||
False,
|
||||
False,
|
||||
False,
|
||||
False,
|
||||
False,
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
class TestExtractCodeCpt:
|
||||
@pytest.fixture
|
||||
def store(self):
|
||||
con = _sqlite_store_con()
|
||||
yield _Store(con)
|
||||
con.close()
|
||||
|
||||
@pytest.fixture
|
||||
def con(self):
|
||||
c = _duckdb_con()
|
||||
yield c
|
||||
c.close()
|
||||
|
||||
def test_cpt_row_yields_source_cpt_rows_and_a_review_row(self, store, con):
|
||||
_insert_cpt_code(
|
||||
con,
|
||||
2024,
|
||||
"GQGTPGYV",
|
||||
"99490",
|
||||
stem="Chronic care management services",
|
||||
elements=[
|
||||
"Consent;",
|
||||
"Something the vocabulary does not know about at all;",
|
||||
],
|
||||
tail="first 20 minutes, per calendar month.",
|
||||
)
|
||||
x = extract_code(store, con, "99490")
|
||||
cpt_rows = [r for r in x.rows if r.source == "cpt"]
|
||||
assert cpt_rows
|
||||
assert all(r.item_key == "GQGTPGYV" and r.year == 2024 for r in cpt_rows)
|
||||
assert any(r.value == "consent" for r in cpt_rows)
|
||||
stem_row = next(r for r in cpt_rows if r.value == "calendar-month")
|
||||
assert stem_row.text == "Chronic care management services"
|
||||
assert stem_row.p_id == 0 and stem_row.page == 0
|
||||
assert [(r.text, r.item_key, r.p_id) for r in x.reviews] == [
|
||||
(
|
||||
"Something the vocabulary does not know about at all",
|
||||
"GQGTPGYV",
|
||||
0,
|
||||
)
|
||||
]
|
||||
|
||||
def test_no_cpt_code_row_yields_no_cpt_rows(self, store, con):
|
||||
x = extract_code(store, con, "99490")
|
||||
assert not any(r.source == "cpt" for r in x.rows)
|
||||
assert x.reviews == ()
|
||||
|
||||
def test_replica_with_no_cpt_tables_at_all_does_not_raise(self, store):
|
||||
# #685/#686 review finding: pfs.cpt_code may not exist yet (an
|
||||
# older replica, or one that hasn't been through cpt-ingest) —
|
||||
# the read-only `elements --dry-run` path never calls
|
||||
# ensure_tables, so this must not raise CatalogException.
|
||||
con = _duckdb_con_no_cpt()
|
||||
try:
|
||||
x = extract_code(store, con, "99490")
|
||||
finally:
|
||||
con.close()
|
||||
assert not any(r.source == "cpt" for r in x.rows)
|
||||
assert x.reviews == ()
|
||||
|
||||
def test_fr_wins_over_cpt_on_duplicate(self, con):
|
||||
store_con = _sqlite_store_con()
|
||||
try:
|
||||
store_con.execute(
|
||||
"INSERT INTO items VALUES ('XFGGRBDH', "
|
||||
"'Medicare and Medicaid Programs; CY 2025 Payment Policies', '2024-07-31')"
|
||||
)
|
||||
rows = [
|
||||
(
|
||||
900,
|
||||
60000,
|
||||
"CPT code 99490 (Chronic care management services, with the "
|
||||
"following required elements:",
|
||||
),
|
||||
(901, 60000, "Consent;"),
|
||||
]
|
||||
store_con.executemany(
|
||||
"INSERT INTO fr_anchors VALUES (?,?,?,?,?)",
|
||||
[("XFGGRBDH", p, pg, p, t) for p, pg, t in rows],
|
||||
)
|
||||
_insert_cpt_code(
|
||||
con,
|
||||
2024,
|
||||
"GQGTPGYV",
|
||||
"99490",
|
||||
stem="Chronic care management services",
|
||||
elements=["Consent;"],
|
||||
tail="with the following required elements.",
|
||||
)
|
||||
x = extract_code(_Store(store_con), con, "99490")
|
||||
consent_rows = [r for r in x.rows if r.value == "consent"]
|
||||
assert len(consent_rows) == 1
|
||||
assert consent_rows[0].source == "fr" and consent_rows[0].p_id == 901
|
||||
finally:
|
||||
store_con.close()
|
||||
|
||||
def test_cpt_wins_over_hcpcs_on_duplicate(self, store, con):
|
||||
con.execute(
|
||||
"INSERT INTO terminology.hcpcs_level_2 VALUES (?,?,?,?)",
|
||||
["99490", "… per calendar month, consent …", "1", "1"],
|
||||
)
|
||||
_insert_cpt_code(
|
||||
con,
|
||||
2024,
|
||||
"GQGTPGYV",
|
||||
"99490",
|
||||
stem="Chronic care management services",
|
||||
elements=["Consent;"],
|
||||
tail="per calendar month.",
|
||||
)
|
||||
x = extract_code(store, con, "99490")
|
||||
consent_rows = [r for r in x.rows if r.value == "consent"]
|
||||
assert len(consent_rows) == 1
|
||||
assert consent_rows[0].source == "cpt"
|
||||
|
||||
|
||||
class TestText:
|
||||
def test_hcpcs_long_description(self):
|
||||
x = extract_text(
|
||||
|
||||
@@ -9,6 +9,9 @@ import duckdb
|
||||
import pytest
|
||||
|
||||
from pfs.codetables import (
|
||||
CptCodeRow,
|
||||
CptInstructionRow,
|
||||
CptSectionRow,
|
||||
ElementRow,
|
||||
EventRow,
|
||||
FamilyRow,
|
||||
@@ -19,6 +22,8 @@ from pfs.families import (
|
||||
FAMILIES,
|
||||
HAND_FAMILIES,
|
||||
Detection,
|
||||
_cpt_edges,
|
||||
cpt_groups,
|
||||
derive_families,
|
||||
detect_codes,
|
||||
family_of,
|
||||
@@ -396,6 +401,687 @@ class TestDerive:
|
||||
assert elapsed < 2.0
|
||||
|
||||
|
||||
def _cpt_section(sec_id, path, guideline=""):
|
||||
path = tuple(path)
|
||||
return CptSectionRow(
|
||||
edition_year=2024,
|
||||
item_key="ITEM0001",
|
||||
sec_id=sec_id,
|
||||
level=len(path),
|
||||
title=path[-1],
|
||||
path=list(path),
|
||||
path_key=" > ".join(path),
|
||||
code_lo="",
|
||||
code_hi="",
|
||||
guideline=guideline,
|
||||
)
|
||||
|
||||
|
||||
def _cpt_code(code, sec_id, *, year=2024, parent="", addon=False, descriptor=""):
|
||||
return CptCodeRow(
|
||||
edition_year=year,
|
||||
item_key="ITEM0001",
|
||||
code=code,
|
||||
sec_id=sec_id,
|
||||
category="I",
|
||||
descriptor=descriptor,
|
||||
stem="",
|
||||
elements=[],
|
||||
tail="",
|
||||
parent=parent,
|
||||
addon=addon,
|
||||
resequenced=False,
|
||||
new=False,
|
||||
revised=False,
|
||||
telemedicine=False,
|
||||
mod51_exempt=False,
|
||||
audio_only=False,
|
||||
fda_pending=False,
|
||||
pla=False,
|
||||
)
|
||||
|
||||
|
||||
class TestCptGroups:
|
||||
def test_lowest_heading_wins_not_the_parent(self):
|
||||
# Task 4 ruling: "Care Management Services" (h1) groups >= 2 codes
|
||||
# too (6, rolled up), but cpt_groups must stop at the h2 leaf
|
||||
# heading each code actually sits under, not walk up past it.
|
||||
sections = [
|
||||
_cpt_section(
|
||||
"sec_ccm",
|
||||
(
|
||||
"Evaluation and Management",
|
||||
"Care Management Services",
|
||||
"Chronic Care Management Services",
|
||||
),
|
||||
),
|
||||
_cpt_section(
|
||||
"sec_ccx",
|
||||
(
|
||||
"Evaluation and Management",
|
||||
"Care Management Services",
|
||||
"Complex Chronic Care Management Services",
|
||||
),
|
||||
),
|
||||
]
|
||||
codes = [
|
||||
_cpt_code("99490", "sec_ccm"),
|
||||
_cpt_code("99439", "sec_ccm"),
|
||||
_cpt_code("99491", "sec_ccm"),
|
||||
_cpt_code("99437", "sec_ccm"),
|
||||
_cpt_code("99487", "sec_ccx"),
|
||||
_cpt_code("99489", "sec_ccx"),
|
||||
]
|
||||
groups = cpt_groups(codes, sections)
|
||||
assert groups["99490"][:3] == (
|
||||
"CHRONIC-CARE-MANAGEMENT-SERVICES",
|
||||
"Chronic Care Management Services",
|
||||
"Evaluation and Management > Care Management Services > "
|
||||
"Chronic Care Management Services",
|
||||
)
|
||||
assert set(groups["99490"][3]) == {"99490", "99439", "99491", "99437"}
|
||||
assert groups["99487"][:3] == (
|
||||
"COMPLEX-CHRONIC-CARE-MANAGEMENT-SERVICES",
|
||||
"Complex Chronic Care Management Services",
|
||||
"Evaluation and Management > Care Management Services > "
|
||||
"Complex Chronic Care Management Services",
|
||||
)
|
||||
assert set(groups["99487"][3]) == {"99487", "99489"}
|
||||
# Never the h1 "Care Management Services" rollup:
|
||||
assert all("Care Management Services)" not in g[1] for g in groups.values())
|
||||
assert all(g[1] != "Care Management Services" for g in groups.values())
|
||||
|
||||
def test_singleton_leaf_walks_up_to_the_grouping_parent(self):
|
||||
sections = [
|
||||
_cpt_section("sec_parent", ("Chapter X", "Section A")),
|
||||
_cpt_section("sec_leaf1", ("Chapter X", "Section A", "Leaf One")),
|
||||
_cpt_section("sec_leaf2", ("Chapter X", "Section A", "Leaf Two")),
|
||||
]
|
||||
codes = [
|
||||
_cpt_code("10001", "sec_leaf1"),
|
||||
_cpt_code("10002", "sec_leaf2"),
|
||||
]
|
||||
groups = cpt_groups(codes, sections)
|
||||
assert groups["10001"][:3] == (
|
||||
"SECTION-A",
|
||||
"Section A",
|
||||
"Chapter X > Section A",
|
||||
)
|
||||
assert groups["10002"][:3] == groups["10001"][:3]
|
||||
assert set(groups["10001"][3]) == {"10001", "10002"}
|
||||
|
||||
def test_same_title_under_different_parents_disambiguated(self):
|
||||
sections = [
|
||||
_cpt_section(
|
||||
"sec_office",
|
||||
("Chapter A", "Office Visits", "New or Established Patient"),
|
||||
),
|
||||
_cpt_section(
|
||||
"sec_home",
|
||||
("Chapter B", "Home Visits", "New or Established Patient"),
|
||||
),
|
||||
]
|
||||
codes = [
|
||||
_cpt_code("20001", "sec_office"),
|
||||
_cpt_code("20002", "sec_office"),
|
||||
_cpt_code("20003", "sec_home"),
|
||||
_cpt_code("20004", "sec_home"),
|
||||
]
|
||||
groups = cpt_groups(codes, sections)
|
||||
# "Chapter A ..." sorts first, so it keeps the bare slug; the
|
||||
# colliding "Chapter B ..." heading is disambiguated by its
|
||||
# parent title (deterministic — sorted path order, first occupant
|
||||
# wins the bare key).
|
||||
assert groups["20001"][0] == "NEW-OR-ESTABLISHED-PATIENT"
|
||||
assert groups["20003"][0] == "NEW-OR-ESTABLISHED-PATIENT@HOME-VISITS"
|
||||
assert groups["20001"][0] != groups["20003"][0]
|
||||
|
||||
|
||||
class TestDeriveCpt:
|
||||
def test_ccm_hand_family_wins_merged_component_note_is_own_heading(self):
|
||||
# The CCM and Complex CCM CPT headings are two separate multi-code
|
||||
# groups; slice-1's stem+activity edge still joins 99487 into the
|
||||
# same component as before (unchanged fixture from
|
||||
# test_ccm_reproduced_with_predecessor_and_addon_roles), so the
|
||||
# merged component intersects the CCM hand family and the hand
|
||||
# key/name win for every code in it — but each code's `note`
|
||||
# stays its own CPT heading's path, not the family's.
|
||||
sections = [
|
||||
_cpt_section(
|
||||
"sec_ccm",
|
||||
(
|
||||
"Evaluation and Management",
|
||||
"Care Management Services",
|
||||
"Chronic Care Management Services",
|
||||
),
|
||||
),
|
||||
_cpt_section(
|
||||
"sec_ccx",
|
||||
(
|
||||
"Evaluation and Management",
|
||||
"Care Management Services",
|
||||
"Complex Chronic Care Management Services",
|
||||
),
|
||||
),
|
||||
]
|
||||
cpt_codes = [
|
||||
_cpt_code("99490", "sec_ccm"),
|
||||
_cpt_code("99439", "sec_ccm", parent="99490", addon=True),
|
||||
_cpt_code("99491", "sec_ccm"),
|
||||
_cpt_code("99437", "sec_ccm", parent="99491", addon=True),
|
||||
_cpt_code("99487", "sec_ccx"),
|
||||
_cpt_code("99489", "sec_ccx", parent="99487", addon=True),
|
||||
]
|
||||
elements = {
|
||||
"99490": [_el("99490", "activity", "comprehensive-care-plan")],
|
||||
"99487": [_el("99487", "activity", "comprehensive-care-plan")],
|
||||
"99491": [_el("99491", "activity", "comprehensive-care-plan")],
|
||||
}
|
||||
descriptions = {
|
||||
"99490": "Chrnc care mgmt staff 1st 20",
|
||||
"99439": "Chrnc care mgmt staf ea addl",
|
||||
"99487": "Cplx chrnc care 1st 60 min",
|
||||
"99489": "Cplx chrnc care ea addl 30",
|
||||
"99491": "Chrnc care mgmt phys 1st 30",
|
||||
"99437": "Chrnc care mgmt phys ea addl",
|
||||
}
|
||||
rows = derive_families(
|
||||
elements,
|
||||
{},
|
||||
descriptions,
|
||||
cpt_codes=cpt_codes,
|
||||
cpt_sections=sections,
|
||||
)
|
||||
by_code = {r.code: r for r in rows}
|
||||
assert {c for c in by_code} == {
|
||||
"99490",
|
||||
"99439",
|
||||
"99487",
|
||||
"99489",
|
||||
"99491",
|
||||
"99437",
|
||||
}
|
||||
assert all(r.key == "CCM" for r in by_code.values())
|
||||
assert all(r.name == HAND_FAMILIES["CCM"].name for r in by_code.values())
|
||||
ccm_path_key = (
|
||||
"Evaluation and Management > Care Management Services > "
|
||||
"Chronic Care Management Services"
|
||||
)
|
||||
ccx_path_key = (
|
||||
"Evaluation and Management > Care Management Services > "
|
||||
"Complex Chronic Care Management Services"
|
||||
)
|
||||
for c in ("99490", "99439", "99491", "99437"):
|
||||
assert by_code[c].note == ccm_path_key
|
||||
for c in ("99487", "99489"):
|
||||
assert by_code[c].note == ccx_path_key
|
||||
# cpt_code.addon and parent both feed the add-on role and its edge.
|
||||
assert by_code["99439"].role == "add-on"
|
||||
assert by_code["99489"].role == "add-on"
|
||||
# Book-derived since: earliest edition_year in cpt_codes.
|
||||
assert by_code["99490"].since == 2024
|
||||
|
||||
def test_heading_with_no_elements_still_becomes_a_family(self):
|
||||
sections = [
|
||||
_cpt_section(
|
||||
"sec_rpm",
|
||||
(
|
||||
"Medicine",
|
||||
"Remote Physiologic Monitoring Treatment Management Services",
|
||||
),
|
||||
)
|
||||
]
|
||||
cpt_codes = [
|
||||
_cpt_code("99457", "sec_rpm", year=2024),
|
||||
_cpt_code("99458", "sec_rpm", year=2024, parent="99457", addon=True),
|
||||
]
|
||||
rows = derive_families({}, {}, {}, cpt_codes=cpt_codes, cpt_sections=sections)
|
||||
by_code = {r.code: r for r in rows}
|
||||
assert set(by_code) == {"99457", "99458"}
|
||||
assert all(
|
||||
r.key == "REMOTE-PHYSIOLOGIC-MONITORING-TREATMENT-MANAGEMENT-SERVICES"
|
||||
for r in by_code.values()
|
||||
)
|
||||
assert all(r.since == 2024 for r in by_code.values())
|
||||
assert all(r.until is None for r in by_code.values())
|
||||
assert by_code["99458"].role == "add-on"
|
||||
|
||||
def test_singleton_leaf_grouped_at_parent_heading(self):
|
||||
sections = [
|
||||
_cpt_section("sec_parent", ("Chapter X", "Section A")),
|
||||
_cpt_section("sec_leaf1", ("Chapter X", "Section A", "Leaf One")),
|
||||
_cpt_section("sec_leaf2", ("Chapter X", "Section A", "Leaf Two")),
|
||||
]
|
||||
cpt_codes = [
|
||||
_cpt_code("30001", "sec_leaf1"),
|
||||
_cpt_code("30002", "sec_leaf2"),
|
||||
]
|
||||
rows = derive_families({}, {}, {}, cpt_codes=cpt_codes, cpt_sections=sections)
|
||||
keys = {r.code: r.key for r in rows}
|
||||
assert keys["30001"] == keys["30002"] == "SECTION-A"
|
||||
|
||||
def test_same_title_headings_get_distinct_keys(self):
|
||||
sections = [
|
||||
_cpt_section(
|
||||
"sec_office",
|
||||
("Chapter A", "Office Visits", "New or Established Patient"),
|
||||
),
|
||||
_cpt_section(
|
||||
"sec_home",
|
||||
("Chapter B", "Home Visits", "New or Established Patient"),
|
||||
),
|
||||
]
|
||||
cpt_codes = [
|
||||
_cpt_code("40001", "sec_office"),
|
||||
_cpt_code("40002", "sec_office"),
|
||||
_cpt_code("40003", "sec_home"),
|
||||
_cpt_code("40004", "sec_home"),
|
||||
]
|
||||
rows = derive_families({}, {}, {}, cpt_codes=cpt_codes, cpt_sections=sections)
|
||||
keys = {r.code: r.key for r in rows}
|
||||
assert keys["40001"] == keys["40002"]
|
||||
assert keys["40003"] == keys["40004"]
|
||||
assert keys["40001"] != keys["40003"]
|
||||
|
||||
def test_use_with_instruction_and_parent_join_codes_with_no_heading(self):
|
||||
# No cpt_sections at all — these codes have no CPT heading of
|
||||
# their own, so the only thing that can join them is the
|
||||
# use-with instruction (add-on -> primary) and the parent field
|
||||
# (semicolon-rule child -> parent).
|
||||
cpt_codes = [
|
||||
_cpt_code("50000", "sec_missing", descriptor="Alpha widget"),
|
||||
_cpt_code("50001", "sec_missing", descriptor="Beta gadget"),
|
||||
_cpt_code(
|
||||
"50002", "sec_missing", parent="50000", descriptor="Gamma sprocket"
|
||||
),
|
||||
]
|
||||
instructions = [
|
||||
CptInstructionRow(
|
||||
edition_year=2024,
|
||||
item_key="ITEM0001",
|
||||
code="50001",
|
||||
kind="use-with",
|
||||
text="(Use 50001 in conjunction with 50000)",
|
||||
targets=["50000"],
|
||||
)
|
||||
]
|
||||
rows = derive_families(
|
||||
{},
|
||||
{},
|
||||
{
|
||||
"50000": "Alpha widget",
|
||||
"50001": "Beta gadget",
|
||||
"50002": "Gamma sprocket",
|
||||
},
|
||||
cpt_codes=cpt_codes,
|
||||
cpt_sections=[],
|
||||
cpt_instructions=instructions,
|
||||
)
|
||||
keys = {r.code: r.key for r in rows}
|
||||
assert keys["50000"] == keys["50001"] == keys["50002"]
|
||||
assert {r.note for r in rows if r.code in ("50000", "50001", "50002")} == {""}
|
||||
|
||||
def test_not_with_instruction_never_joins(self):
|
||||
cpt_codes = [
|
||||
_cpt_code("60000", "sec_missing", descriptor="Zeta thing"),
|
||||
_cpt_code("60001", "sec_missing", descriptor="Eta gizmo"),
|
||||
]
|
||||
instructions = [
|
||||
CptInstructionRow(
|
||||
edition_year=2024,
|
||||
item_key="ITEM0001",
|
||||
code="60001",
|
||||
kind="not-with-time",
|
||||
text="(Do not report 60001 for service time reported with 60000)",
|
||||
targets=["60000"],
|
||||
)
|
||||
]
|
||||
rows = derive_families(
|
||||
{},
|
||||
{},
|
||||
{"60000": "Zeta thing", "60001": "Eta gizmo"},
|
||||
cpt_codes=cpt_codes,
|
||||
cpt_sections=[],
|
||||
cpt_instructions=instructions,
|
||||
)
|
||||
keys = {r.code: r.key for r in rows}
|
||||
assert keys["60000"] != keys["60001"]
|
||||
|
||||
def test_hcpcs_code_keeps_slice1_derivation(self):
|
||||
# A HCPCS G-code absent from pfs.cpt_code joins a CPT family only
|
||||
# through a slice-1 edge (here: a replaced_by event), never
|
||||
# through cpt_groups.
|
||||
sections = [
|
||||
_cpt_section(
|
||||
"sec_ccm",
|
||||
(
|
||||
"Evaluation and Management",
|
||||
"Care Management Services",
|
||||
"Chronic Care Management Services",
|
||||
),
|
||||
)
|
||||
]
|
||||
cpt_codes = [
|
||||
_cpt_code("99490", "sec_ccm"),
|
||||
_cpt_code("99439", "sec_ccm", parent="99490", addon=True),
|
||||
]
|
||||
events = {
|
||||
"G2058": [
|
||||
_ev("G2058", 2020, "appeared"),
|
||||
_ev("G2058", 2021, "disappeared"),
|
||||
_ev("G2058", 2021, "replaced_by", to="99439"),
|
||||
],
|
||||
"99439": [_ev("99439", 2021, "replaces", frm="G2058")],
|
||||
}
|
||||
rows = derive_families(
|
||||
{}, events, {}, cpt_codes=cpt_codes, cpt_sections=sections
|
||||
)
|
||||
by_code = {r.code: r for r in rows}
|
||||
assert by_code["G2058"].key == by_code["99490"].key == "CCM"
|
||||
assert by_code["G2058"].note == ""
|
||||
assert by_code["G2058"].role == "predecessor"
|
||||
assert by_code["G2058"].since == 2020 and by_code["G2058"].until == 2021
|
||||
|
||||
|
||||
class TestCptEdgesHelper:
|
||||
def test_skips_only_edges_between_distinct_headings(self):
|
||||
# #687 controller review (Ruling C11), tested against the helper
|
||||
# in isolation: an edge is skipped only when BOTH endpoints
|
||||
# already resolve to a heading and those headings differ.
|
||||
groups_by_code = {
|
||||
"70001": (
|
||||
"HEADING-A",
|
||||
"Heading A",
|
||||
"Chapter > Heading A",
|
||||
("70001", "70002"),
|
||||
),
|
||||
"70002": (
|
||||
"HEADING-A",
|
||||
"Heading A",
|
||||
"Chapter > Heading A",
|
||||
("70001", "70002"),
|
||||
),
|
||||
"70003": (
|
||||
"HEADING-B",
|
||||
"Heading B",
|
||||
"Chapter > Heading B",
|
||||
("70003", "70004"),
|
||||
),
|
||||
}
|
||||
unioned = []
|
||||
|
||||
def fake_union(a, b):
|
||||
unioned.append((a, b))
|
||||
|
||||
instructions = [
|
||||
CptInstructionRow(
|
||||
edition_year=2024,
|
||||
item_key="I",
|
||||
code="70001",
|
||||
kind="use-with",
|
||||
text="",
|
||||
targets=["70003"], # both headed, distinct -> skipped
|
||||
),
|
||||
CptInstructionRow(
|
||||
edition_year=2024,
|
||||
item_key="I",
|
||||
code="70001",
|
||||
kind="use-with",
|
||||
text="",
|
||||
targets=["70005"], # 70005 unheaded -> unioned
|
||||
),
|
||||
]
|
||||
_cpt_edges(fake_union, groups_by_code, instructions, {}, {})
|
||||
assert unioned == [("70001", "70005")]
|
||||
|
||||
|
||||
class TestCptEdgeGuardC11:
|
||||
def _two_headings(self):
|
||||
sections = [
|
||||
_cpt_section("sec_a", ("Chapter", "Heading A")),
|
||||
_cpt_section("sec_b", ("Chapter", "Heading B")),
|
||||
]
|
||||
cpt_codes = [
|
||||
_cpt_code("70001", "sec_a"),
|
||||
_cpt_code("70002", "sec_a"),
|
||||
_cpt_code("70003", "sec_b"),
|
||||
_cpt_code("70004", "sec_b"),
|
||||
]
|
||||
return sections, cpt_codes
|
||||
|
||||
def test_use_with_between_two_headed_codes_does_not_bridge(self):
|
||||
sections, cpt_codes = self._two_headings()
|
||||
instructions = [
|
||||
CptInstructionRow(
|
||||
edition_year=2024,
|
||||
item_key="ITEM0001",
|
||||
code="70001",
|
||||
kind="use-with",
|
||||
text="(Use 70001 in conjunction with 70003)",
|
||||
targets=["70003"],
|
||||
)
|
||||
]
|
||||
rows = derive_families(
|
||||
{},
|
||||
{},
|
||||
{},
|
||||
cpt_codes=cpt_codes,
|
||||
cpt_sections=sections,
|
||||
cpt_instructions=instructions,
|
||||
)
|
||||
keys = {r.code: r.key for r in rows}
|
||||
assert keys["70001"] == keys["70002"]
|
||||
assert keys["70003"] == keys["70004"]
|
||||
assert keys["70001"] != keys["70003"]
|
||||
|
||||
def test_use_with_from_unheaded_code_still_joins_the_headed_family(self):
|
||||
sections, cpt_codes = self._two_headings()
|
||||
# 70005 sits alone under its own chapter — no sibling anywhere it
|
||||
# rolls up to, so it never qualifies for a heading of its own.
|
||||
sections = sections + [_cpt_section("sec_x", ("Solo Chapter", "Heading X"))]
|
||||
cpt_codes = cpt_codes + [_cpt_code("70005", "sec_x")]
|
||||
instructions = [
|
||||
CptInstructionRow(
|
||||
edition_year=2024,
|
||||
item_key="ITEM0001",
|
||||
code="70005",
|
||||
kind="use-with",
|
||||
text="(Use 70005 in conjunction with 70001)",
|
||||
targets=["70001"],
|
||||
)
|
||||
]
|
||||
rows = derive_families(
|
||||
{},
|
||||
{},
|
||||
{},
|
||||
cpt_codes=cpt_codes,
|
||||
cpt_sections=sections,
|
||||
cpt_instructions=instructions,
|
||||
)
|
||||
keys = {r.code: r.key for r in rows}
|
||||
assert keys["70005"] == keys["70001"] == keys["70002"]
|
||||
|
||||
def test_parent_between_two_headed_codes_does_not_bridge(self):
|
||||
sections, cpt_codes = self._two_headings()
|
||||
# 70003's `parent` names a code in a DIFFERENT heading — the
|
||||
# guard must hold even though a real semicolon-rule parent is
|
||||
# always within the same heading in practice.
|
||||
cpt_codes = [
|
||||
c if c.code != "70003" else _cpt_code("70003", "sec_b", parent="70001")
|
||||
for c in cpt_codes
|
||||
]
|
||||
rows = derive_families({}, {}, {}, cpt_codes=cpt_codes, cpt_sections=sections)
|
||||
keys = {r.code: r.key for r in rows}
|
||||
assert keys["70003"] == keys["70004"]
|
||||
assert keys["70003"] != keys["70001"]
|
||||
|
||||
def test_addon_of_between_two_headed_codes_does_not_bridge(self):
|
||||
sections, cpt_codes = self._two_headings()
|
||||
elements = {"70003": [_el("70003", "relation", "addon-of", "70001")]}
|
||||
rows = derive_families(
|
||||
elements, {}, {}, cpt_codes=cpt_codes, cpt_sections=sections
|
||||
)
|
||||
keys = {r.code: r.key for r in rows}
|
||||
assert keys["70003"] == keys["70004"]
|
||||
assert keys["70003"] != keys["70001"]
|
||||
|
||||
def test_addon_of_from_unheaded_code_still_joins(self):
|
||||
sections, cpt_codes = self._two_headings()
|
||||
elements = {"77000": [_el("77000", "relation", "addon-of", "70001")]}
|
||||
rows = derive_families(
|
||||
elements, {}, {}, cpt_codes=cpt_codes, cpt_sections=sections
|
||||
)
|
||||
keys = {r.code: r.key for r in rows}
|
||||
assert keys["77000"] == keys["70001"]
|
||||
|
||||
def test_note_value_is_unaffected_by_the_guard(self):
|
||||
# The guard only skips a union edge; it must not touch a code's
|
||||
# own note (still its own heading's path_key).
|
||||
sections, cpt_codes = self._two_headings()
|
||||
instructions = [
|
||||
CptInstructionRow(
|
||||
edition_year=2024,
|
||||
item_key="ITEM0001",
|
||||
code="70001",
|
||||
kind="use-with",
|
||||
text="(Use 70001 in conjunction with 70003)",
|
||||
targets=["70003"],
|
||||
)
|
||||
]
|
||||
rows = derive_families(
|
||||
{},
|
||||
{},
|
||||
{},
|
||||
cpt_codes=cpt_codes,
|
||||
cpt_sections=sections,
|
||||
cpt_instructions=instructions,
|
||||
)
|
||||
by_code = {r.code: r for r in rows}
|
||||
assert by_code["70001"].note == "Chapter > Heading A"
|
||||
assert by_code["70003"].note == "Chapter > Heading B"
|
||||
|
||||
|
||||
class TestCptPresenceC12:
|
||||
def test_code_absent_from_newest_edition_gets_since_and_until_from_all_editions(
|
||||
self,
|
||||
):
|
||||
sections = [_cpt_section("sec_a", ("Chapter", "Heading A"))]
|
||||
cpt_codes = [
|
||||
_cpt_code("90001", "sec_a", year=2024),
|
||||
_cpt_code("90002", "sec_a", year=2024),
|
||||
]
|
||||
cpt_presence = {
|
||||
"90001": (2019, 2021, 2022, 2024),
|
||||
"90002": (2019, 2021, 2022, 2024),
|
||||
"80001": (2019, 2020, 2021, 2022), # gone by 2024, not in cpt_codes
|
||||
}
|
||||
rows = derive_families(
|
||||
{},
|
||||
{},
|
||||
{},
|
||||
cpt_codes=cpt_codes,
|
||||
cpt_sections=sections,
|
||||
cpt_presence=cpt_presence,
|
||||
)
|
||||
by_code = {r.code: r for r in rows}
|
||||
assert "80001" in by_code
|
||||
assert by_code["80001"].since == 2019
|
||||
assert by_code["80001"].until == 2023
|
||||
|
||||
def test_code_new_in_newest_edition_has_no_until(self):
|
||||
sections = [_cpt_section("sec_a", ("Chapter", "Heading A"))]
|
||||
cpt_codes = [
|
||||
_cpt_code("90001", "sec_a", year=2024),
|
||||
_cpt_code("90003", "sec_a", year=2024),
|
||||
]
|
||||
cpt_presence = {
|
||||
"90001": (2019, 2021, 2022, 2024),
|
||||
"90003": (2024,),
|
||||
}
|
||||
rows = derive_families(
|
||||
{},
|
||||
{},
|
||||
{},
|
||||
cpt_codes=cpt_codes,
|
||||
cpt_sections=sections,
|
||||
cpt_presence=cpt_presence,
|
||||
)
|
||||
by_code = {r.code: r for r in rows}
|
||||
assert by_code["90001"].since == 2019
|
||||
assert by_code["90003"].since == 2024
|
||||
assert by_code["90003"].until is None
|
||||
|
||||
def test_without_cpt_presence_falls_back_to_cpt_codes_alone(self):
|
||||
# Pre-C12 behavior preserved when the caller doesn't have a
|
||||
# multi-edition presence map to give.
|
||||
sections = [_cpt_section("sec_a", ("Chapter", "Heading A"))]
|
||||
cpt_codes = [
|
||||
_cpt_code("90001", "sec_a", year=2024),
|
||||
_cpt_code("90002", "sec_a", year=2024),
|
||||
]
|
||||
rows = derive_families({}, {}, {}, cpt_codes=cpt_codes, cpt_sections=sections)
|
||||
by_code = {r.code: r for r in rows}
|
||||
assert by_code["90001"].since == 2024
|
||||
assert by_code["90001"].until is None
|
||||
|
||||
|
||||
class TestSinceEarliestEvidenceC13:
|
||||
"""F5 / Ruling C13: ``since`` = min(cpt_since, appeared-event year)
|
||||
when an ``appeared`` event exists — cpt_since is only the earliest
|
||||
*ingested* edition, not evidence the code didn't exist earlier."""
|
||||
|
||||
def test_since_is_min_of_cpt_presence_and_appeared_event(self):
|
||||
# 99490 was created CY2015 but the oldest CPT edition on hand is
|
||||
# 2019 — since must read 2015, not 2019.
|
||||
sections = [_cpt_section("sec_a", ("Chapter", "Heading A"))]
|
||||
cpt_codes = [_cpt_code("99490", "sec_a", year=2024)]
|
||||
cpt_presence = {"99490": (2019, 2021, 2022, 2024)}
|
||||
events = {"99490": [_ev("99490", 2015, "appeared")]}
|
||||
rows = derive_families(
|
||||
{},
|
||||
events,
|
||||
{},
|
||||
cpt_codes=cpt_codes,
|
||||
cpt_sections=sections,
|
||||
cpt_presence=cpt_presence,
|
||||
)
|
||||
by_code = {r.code: r for r in rows}
|
||||
assert by_code["99490"].since == 2015
|
||||
|
||||
def test_since_falls_back_to_cpt_presence_when_no_appeared_event(self):
|
||||
sections = [_cpt_section("sec_a", ("Chapter", "Heading A"))]
|
||||
cpt_codes = [_cpt_code("90001", "sec_a", year=2024)]
|
||||
cpt_presence = {"90001": (2019, 2021, 2022, 2024)}
|
||||
rows = derive_families(
|
||||
{},
|
||||
{},
|
||||
{},
|
||||
cpt_codes=cpt_codes,
|
||||
cpt_sections=sections,
|
||||
cpt_presence=cpt_presence,
|
||||
)
|
||||
by_code = {r.code: r for r in rows}
|
||||
assert by_code["90001"].since == 2019
|
||||
|
||||
def test_since_keeps_cpt_presence_when_appeared_event_is_later(self):
|
||||
# An appeared event newer than the CPT evidence must not win —
|
||||
# min(), not "prefer the event".
|
||||
sections = [_cpt_section("sec_a", ("Chapter", "Heading A"))]
|
||||
cpt_codes = [_cpt_code("90001", "sec_a", year=2024)]
|
||||
cpt_presence = {"90001": (2019, 2021, 2022, 2024)}
|
||||
events = {"90001": [_ev("90001", 2021, "appeared")]}
|
||||
rows = derive_families(
|
||||
{},
|
||||
events,
|
||||
{},
|
||||
cpt_codes=cpt_codes,
|
||||
cpt_sections=sections,
|
||||
cpt_presence=cpt_presence,
|
||||
)
|
||||
by_code = {r.code: r for r in rows}
|
||||
assert by_code["90001"].since == 2019
|
||||
|
||||
|
||||
class TestLoadAndRefresh:
|
||||
def test_load_and_refresh_in_place(self, restore_families):
|
||||
mod = restore_families
|
||||
@@ -470,6 +1156,59 @@ class TestLoadAndRefresh:
|
||||
finally:
|
||||
con.close()
|
||||
|
||||
def test_derived_family_gets_empty_synonyms(self, restore_families):
|
||||
# F3: a derived (non-hand) family must not become a chat trigger
|
||||
# off its own one/two-word name — matching by name is #699's job.
|
||||
mod = restore_families
|
||||
con = duckdb.connect(":memory:")
|
||||
try:
|
||||
ensure_tables(con)
|
||||
write_families(
|
||||
con,
|
||||
[
|
||||
FamilyRow(
|
||||
"AORTA-REPAIR",
|
||||
"Repair",
|
||||
"99999",
|
||||
"base",
|
||||
None,
|
||||
None,
|
||||
"",
|
||||
0,
|
||||
)
|
||||
],
|
||||
)
|
||||
refresh_from(con)
|
||||
assert mod.FAMILIES["AORTA-REPAIR"].synonyms == ()
|
||||
assert detect_codes("repair of the aorta").families == ()
|
||||
finally:
|
||||
con.close()
|
||||
|
||||
def test_hand_family_keeps_its_synonyms_after_refresh(self, restore_families):
|
||||
mod = restore_families
|
||||
con = duckdb.connect(":memory:")
|
||||
try:
|
||||
ensure_tables(con)
|
||||
write_families(
|
||||
con,
|
||||
[
|
||||
FamilyRow(
|
||||
"CCM",
|
||||
"Chronic Care Management",
|
||||
"99490",
|
||||
"base",
|
||||
None,
|
||||
None,
|
||||
"",
|
||||
0,
|
||||
)
|
||||
],
|
||||
)
|
||||
refresh_from(con)
|
||||
assert mod.FAMILIES["CCM"].synonyms == HAND_FAMILIES["CCM"].synonyms
|
||||
finally:
|
||||
con.close()
|
||||
|
||||
def test_schema_present_table_absent_returns_empty(self):
|
||||
# pfs schema created (e.g. by an earlier ensure_tables call for a
|
||||
# sibling table) but pfs.code_family itself never materialized —
|
||||
|
||||
@@ -7,15 +7,64 @@ import sqlite3
|
||||
import duckdb
|
||||
import pytest
|
||||
|
||||
from pfs.lineage import _expand_ranges, fr_events, lineage, rvu_events, rvu_year_span
|
||||
from pfs.codetables import ensure_tables
|
||||
from pfs.lineage import (
|
||||
_expand_ranges,
|
||||
cpt_events,
|
||||
fr_events,
|
||||
lineage,
|
||||
rvu_events,
|
||||
rvu_year_span,
|
||||
)
|
||||
|
||||
RVU_COLS = "hcpcs VARCHAR, mod VARCHAR, description VARCHAR, status_code VARCHAR, non_fac_total DOUBLE, year INTEGER"
|
||||
|
||||
|
||||
def _cpt_section(con, year, item_key):
|
||||
con.execute(
|
||||
"INSERT INTO pfs.cpt_section VALUES (?,?,?,?,?,?,?,?,?,?)",
|
||||
[year, item_key, "sec1", 1, "T", [], "T", "", "", ""],
|
||||
)
|
||||
|
||||
|
||||
def _cpt_reference(con, year, item_key, code, years, text="text"):
|
||||
con.execute(
|
||||
"INSERT INTO pfs.cpt_reference VALUES (?,?,?,?,?,?)",
|
||||
[year, item_key, code, "cpt-changes", years, text],
|
||||
)
|
||||
|
||||
|
||||
def _cpt_code(con, year, item_key, code):
|
||||
con.execute(
|
||||
"INSERT INTO pfs.cpt_code VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
|
||||
[
|
||||
year,
|
||||
item_key,
|
||||
code,
|
||||
"sec1",
|
||||
"I",
|
||||
"d",
|
||||
"s",
|
||||
[],
|
||||
"t",
|
||||
"",
|
||||
False,
|
||||
False,
|
||||
False,
|
||||
False,
|
||||
False,
|
||||
False,
|
||||
False,
|
||||
False,
|
||||
False,
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def con():
|
||||
c = duckdb.connect(":memory:")
|
||||
c.execute("CREATE SCHEMA pfs")
|
||||
ensure_tables(c)
|
||||
c.execute(f"CREATE TABLE pfs.rvu ({RVU_COLS})")
|
||||
rows = [
|
||||
# table spans 2015..2026 (filler code keeps the span honest)
|
||||
@@ -219,7 +268,83 @@ class TestExpandRanges:
|
||||
assert _expand_ranges("G0008-G0010") == "G0008 G0009 G0010"
|
||||
|
||||
|
||||
class TestCptEvents:
|
||||
def test_cpt_changed_one_per_year_from_newest_edition(self, con):
|
||||
_cpt_section(con, 2019, "ITEM2019")
|
||||
_cpt_section(con, 2024, "GQGTPGYV")
|
||||
_cpt_reference(con, 2019, "ITEM2019", "99490", [2015])
|
||||
_cpt_reference(con, 2024, "GQGTPGYV", "99490", [2015, 2021, 2022])
|
||||
ev = cpt_events(con, "99490")
|
||||
assert [(e.year, e.kind, e.item_key, e.note) for e in ev] == [
|
||||
(2015, "cpt_changed", "GQGTPGYV", "CPT Changes 2015"),
|
||||
(2021, "cpt_changed", "GQGTPGYV", "CPT Changes 2021"),
|
||||
(2022, "cpt_changed", "GQGTPGYV", "CPT Changes 2022"),
|
||||
]
|
||||
assert all(e.source == "cpt" and e.anchored is True for e in ev)
|
||||
assert all(e.code == "99490" for e in ev)
|
||||
|
||||
def test_cpt_deleted_when_absent_from_newest_edition(self, con):
|
||||
_cpt_section(con, 2019, "ITEM2019")
|
||||
_cpt_section(con, 2024, "GQGTPGYV")
|
||||
_cpt_code(con, 2019, "ITEM2019", "99999")
|
||||
ev = cpt_events(con, "99999")
|
||||
assert [(e.year, e.kind, e.item_key, e.note) for e in ev] == [
|
||||
(2020, "cpt_deleted", "ITEM2019", "absent from CPT 2024")
|
||||
]
|
||||
|
||||
def test_no_events_for_a_code_absent_everywhere(self, con):
|
||||
_cpt_section(con, 2024, "GQGTPGYV")
|
||||
assert cpt_events(con, "00000") == []
|
||||
|
||||
def test_no_cpt_tables_yet_is_empty(self, con):
|
||||
assert cpt_events(con, "99490") == []
|
||||
|
||||
def test_no_cpt_tables_at_all_does_not_raise(self):
|
||||
# #685/#686 review finding: a replica that has never had
|
||||
# cpt-ingest run has no pfs.cpt_* schema at all (not merely empty
|
||||
# tables) — pfs.cpt_section itself doesn't exist, so cpt_years()
|
||||
# raises CatalogException. cpt_events must swallow that (I4), the
|
||||
# same way the CLI's read-only paths already tolerate a missing
|
||||
# pfs.code_element/pfs.code_event.
|
||||
c = duckdb.connect(":memory:")
|
||||
c.execute("CREATE SCHEMA pfs")
|
||||
c.execute(f"CREATE TABLE pfs.rvu ({RVU_COLS})")
|
||||
try:
|
||||
assert cpt_events(c, "99490") == []
|
||||
finally:
|
||||
c.close()
|
||||
|
||||
|
||||
class TestCrossCheck:
|
||||
def test_lineage_with_no_cpt_tables_at_all_does_not_raise(self, store):
|
||||
c = duckdb.connect(":memory:")
|
||||
c.execute("CREATE SCHEMA pfs")
|
||||
c.execute(f"CREATE TABLE pfs.rvu ({RVU_COLS})")
|
||||
c.executemany(
|
||||
"INSERT INTO pfs.rvu VALUES (?,?,?,?,?,?)",
|
||||
[
|
||||
("00000", None, "filler", "A", 1.0, 2019),
|
||||
("00000", None, "filler", "A", 1.0, 2021),
|
||||
("G2058", None, "Ccm add 20min", "A", 1.0, 2020),
|
||||
],
|
||||
)
|
||||
try:
|
||||
ev = lineage(c, store, "G2058")
|
||||
finally:
|
||||
c.close()
|
||||
assert not any(e.source == "cpt" for e in ev)
|
||||
assert any(e.kind == "appeared" for e in ev)
|
||||
|
||||
def test_rvu_event_anchored_by_cpt_changed_within_one_year(self, con, store):
|
||||
# 99487 has no FR paragraph in `store`, but a cpt_changed at 2016
|
||||
# (Y-1) must still anchor the 2017 status_change.
|
||||
_cpt_section(con, 2024, "GQGTPGYV")
|
||||
_cpt_reference(con, 2024, "GQGTPGYV", "99487", [2016])
|
||||
ev = lineage(con, store, "99487")
|
||||
by = {(e.kind, e.year): e for e in ev}
|
||||
assert by[("status_change", 2017)].anchored is True
|
||||
assert ("cpt_changed", 2016) in by
|
||||
|
||||
def test_rvu_events_anchored_by_nearby_fr_event(self, con, store):
|
||||
ev = lineage(con, store, "G2058")
|
||||
by_kind = {e.kind: e for e in ev}
|
||||
|
||||
106
tests/rex/comments/test_epub_text.py
Normal file
106
tests/rex/comments/test_epub_text.py
Normal file
@@ -0,0 +1,106 @@
|
||||
"""rex.comments.epub_text — shared EPUB container/spine resolution."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
from rex.comments.epub_text import content_root, extract_text, spine_paths
|
||||
|
||||
_CONTAINER_XML = (
|
||||
'<?xml version="1.0"?>\n'
|
||||
'<container version="1.0" '
|
||||
'xmlns="urn:oasis:names:tc:opendocument:xmlns:container">\n'
|
||||
"<rootfiles>\n"
|
||||
'<rootfile full-path="OEBPS/content.opf" '
|
||||
'media-type="application/oebps-package+xml"/>\n'
|
||||
"</rootfiles>\n</container>"
|
||||
)
|
||||
|
||||
|
||||
def _opf(spine_idrefs: list[str]) -> str:
|
||||
spine = "\n".join(f'<itemref idref="{i}"/>' for i in spine_idrefs)
|
||||
return (
|
||||
'<?xml version="1.0"?>\n'
|
||||
'<package xmlns="http://www.idpf.org/2007/opf" version="3.0">\n'
|
||||
"<manifest>\n"
|
||||
'<item id="a" href="a.xhtml" media-type="application/xhtml+xml"/>\n'
|
||||
'<item id="b" href="b.xhtml" media-type="application/xhtml+xml"/>\n'
|
||||
"</manifest>\n"
|
||||
f"<spine>\n{spine}\n</spine>\n"
|
||||
"</package>"
|
||||
)
|
||||
|
||||
|
||||
def _build_epub_with_container(path: Path, *, spine_order: list[str]) -> None:
|
||||
with zipfile.ZipFile(path, "w") as zf:
|
||||
zf.writestr("mimetype", "application/epub+zip")
|
||||
zf.writestr("META-INF/container.xml", _CONTAINER_XML)
|
||||
zf.writestr("OEBPS/content.opf", _opf(spine_order))
|
||||
zf.writestr("OEBPS/a.xhtml", "<html><body><p>Section A text.</p></body></html>")
|
||||
zf.writestr("OEBPS/b.xhtml", "<html><body><p>Section B text.</p></body></html>")
|
||||
|
||||
|
||||
def _build_epub_without_container(path: Path) -> None:
|
||||
with zipfile.ZipFile(path, "w") as zf:
|
||||
zf.writestr("mimetype", "application/epub+zip")
|
||||
# Deliberately named so name-sort order differs from any
|
||||
# plausible spine order — proves the fallback really is name-sort.
|
||||
zf.writestr("OPS/z_first.xhtml", "<html><body><p>Z content.</p></body></html>")
|
||||
zf.writestr("OPS/a_second.xhtml", "<html><body><p>A content.</p></body></html>")
|
||||
|
||||
|
||||
class TestContentRoot:
|
||||
def test_resolves_via_container_xml(self, tmp_path):
|
||||
path = tmp_path / "book.epub"
|
||||
_build_epub_with_container(path, spine_order=["a", "b"])
|
||||
with zipfile.ZipFile(path) as zf:
|
||||
assert content_root(zf) == "OEBPS/"
|
||||
|
||||
def test_falls_back_to_ops_without_container_xml(self, tmp_path):
|
||||
path = tmp_path / "book.epub"
|
||||
_build_epub_without_container(path)
|
||||
with zipfile.ZipFile(path) as zf:
|
||||
assert content_root(zf) == "OPS/"
|
||||
|
||||
|
||||
class TestSpinePaths:
|
||||
def test_resolves_spine_order_from_the_opf(self, tmp_path):
|
||||
path = tmp_path / "book.epub"
|
||||
_build_epub_with_container(path, spine_order=["b", "a"])
|
||||
with zipfile.ZipFile(path) as zf:
|
||||
assert spine_paths(zf) == ["OEBPS/b.xhtml", "OEBPS/a.xhtml"]
|
||||
|
||||
def test_none_without_container_xml(self, tmp_path):
|
||||
path = tmp_path / "book.epub"
|
||||
_build_epub_without_container(path)
|
||||
with zipfile.ZipFile(path) as zf:
|
||||
assert spine_paths(zf) is None
|
||||
|
||||
|
||||
class TestExtractText:
|
||||
def test_extracts_in_spine_order_not_alphabetical(self, tmp_path):
|
||||
path = tmp_path / "book.epub"
|
||||
# Spine deliberately reversed from alphabetical (a, b) name order.
|
||||
_build_epub_with_container(path, spine_order=["b", "a"])
|
||||
text = extract_text(path)
|
||||
assert text.index("Section B text") < text.index("Section A text")
|
||||
|
||||
def test_falls_back_to_sorted_names_without_container_xml(self, tmp_path):
|
||||
path = tmp_path / "book.epub"
|
||||
_build_epub_without_container(path)
|
||||
text = extract_text(path)
|
||||
# a_second.xhtml sorts before z_first.xhtml by name.
|
||||
assert text.index("A content") < text.index("Z content")
|
||||
|
||||
def test_strips_tags_and_unescapes_entities(self, tmp_path):
|
||||
path = tmp_path / "book.epub"
|
||||
with zipfile.ZipFile(path, "w") as zf:
|
||||
zf.writestr("mimetype", "application/epub+zip")
|
||||
zf.writestr(
|
||||
"OPS/a.xhtml",
|
||||
"<html><body><p>CPT® codes & values</p></body></html>",
|
||||
)
|
||||
text = extract_text(path)
|
||||
assert "<p>" not in text
|
||||
assert "CPT® codes & values" in text
|
||||
55
tests/rex/comments/test_extract_epub.py
Normal file
55
tests/rex/comments/test_extract_epub.py
Normal file
@@ -0,0 +1,55 @@
|
||||
"""F2: extract_attachment's .epub branch."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
from rex.comments import extract_attachment
|
||||
|
||||
|
||||
def _build_epub(path: Path, body: str) -> None:
|
||||
with zipfile.ZipFile(path, "w") as zf:
|
||||
zf.writestr("mimetype", "application/epub+zip")
|
||||
zf.writestr("OPS/Chapter01.xhtml", f"<html><body>{body}</body></html>")
|
||||
|
||||
|
||||
def test_epub_extracted_ok(tmp_path: Path):
|
||||
p = tmp_path / "book.epub"
|
||||
_build_epub(p, "<p>" + "A CPT codebook chapter with real text. " * 5 + "</p>")
|
||||
result = extract_attachment(p)
|
||||
assert result.status == "ok"
|
||||
assert "CPT codebook chapter" in result.text
|
||||
assert result.chars == len(result.text)
|
||||
|
||||
|
||||
def test_epub_below_threshold_marked_ocr_needed(tmp_path: Path):
|
||||
p = tmp_path / "book.epub"
|
||||
_build_epub(p, "<p>short</p>")
|
||||
result = extract_attachment(p)
|
||||
assert result.status == "ocr_needed"
|
||||
|
||||
|
||||
def test_epub_malformed_zip_marked_failed(tmp_path: Path):
|
||||
p = tmp_path / "book.epub"
|
||||
p.write_bytes(b"not a real zip file")
|
||||
result = extract_attachment(p)
|
||||
assert result.status == "failed"
|
||||
assert result.text == ""
|
||||
|
||||
|
||||
def test_epub_extracted_in_reading_order(tmp_path: Path):
|
||||
p = tmp_path / "book.epub"
|
||||
with zipfile.ZipFile(p, "w") as zf:
|
||||
zf.writestr("mimetype", "application/epub+zip")
|
||||
zf.writestr(
|
||||
"OPS/Chapter01.xhtml",
|
||||
"<html><body><p>First chapter content here, plenty of text.</p></body></html>",
|
||||
)
|
||||
zf.writestr(
|
||||
"OPS/Chapter02.xhtml",
|
||||
"<html><body><p>Second chapter content here, plenty of text.</p></body></html>",
|
||||
)
|
||||
result = extract_attachment(p)
|
||||
assert result.status == "ok"
|
||||
assert result.text.index("First chapter") < result.text.index("Second chapter")
|
||||
Reference in New Issue
Block a user