homelab/stack
1
0
Fork 0
Code Issues 59 Pull Requests 1 Actions Packages Projects 1 Releases Wiki Activity

merge: P49 anchors slice — code:/family: item tags, cross-collection cited sources, IOM/CFR guidance crosswalk, public-reaction series, notebook sections 5–7 (refs #688 #689 #690)
Some checks failed
CI / lint (push) Successful in 33s
Details
CI / notebooks-smoke (push) Successful in 1m26s
Details
Deploy / notebooks (push) Has been skipped
Details
Deploy / zotero (push) Has been skipped
Details
Deploy / docs (push) Has been skipped
Details
Deploy / api (push) Has been skipped
Details
Deploy / llm (push) Has been skipped
Details
Deploy / mc (push) Has been skipped
Details
Infra CI / notebooks (push) Successful in 1m22s
Details
Infra CI / zotero (push) Successful in 15s
Details
Infra CI / docs (push) Successful in 1m36s
Details
Infra CI / api (push) Successful in 1m1s
Details
Infra CI / llm (push) Successful in 50s
Details
Infra CI / mc (push) Successful in 20s
Details
Deploy / report (push) Successful in 12s
Details
CI / test (push) Failing after 13m53s
Details
Package Supply Chain / pkg-supply-chain (push) Successful in 1m1s
Details

Browse Source 
Branch worktree-p49-anchors-2 (Tasks 4–8 of docs/superpowers/plans/2026-09-09-code-family-anchors.md):

- stack bib code-tags: code:<HCPCS>/family:<KEY> tags on bib items from chunk anchors and FR paragraphs, merged across sources before the 200-code wide-item cap, Store.add_tags batched per item (67,198 items, 486,519 tags live).
- llm.evidence.code_cited_sources over rules/comments/corpus plus family-cited chunks (one chunk per item, newest first), round-robin capped by code_cited_max (12).
- pfs.guidance + stack pfs guidance: CFR/IOM/MLN references from FR paragraphs naming a family's codes, resolved to eCFR/IOM bib items with FR provenance (pfs.code_guidance; 260 rows for the five hand families, CFR lists captured in full).
- pfs.reaction + stack pfs reaction: per-docket commenter counts, self-hosted stance sampling, FR Comment/Response pairs as the pre-2017 proxy (pfs.code_reaction; G2211 3,663/22,021 in CMS-2023-0121).
- notebooks/code_families.py sections 5–7 on live data; CLI docs regenerated (sidebar collision fixed).
This commit is contained in:
kert
2026-09-10 01:29:33 -04:00
parent d016f707ac 1c542ae708
commit 4616d36ada
111 changed files with 5368 additions and 95 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/docs/cli/api-serve.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack api serve title: stack api serve
sidebar_position: 28 sidebar_position: 58
--- ---
# `stack api serve` # `stack api serve`

2
docs/docs/cli/api.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack api title: stack api
sidebar_position: 27 sidebar_position: 57
--- ---
# `stack api` # `stack api`

21
docs/docs/cli/bib-attach-iom.md Normal file
View File

@@ -0,0 +1,21 @@
---
title: stack bib attach-iom
sidebar_position: 21
---
# `stack bib attach-iom`
```
Usage: stack bib attach-iom [OPTIONS]
Download the PDF for every Manual and attach it to the record.
Idempotent by SHA-256 — only downloads files that have changed on
CMS since the last run. Safe to schedule on a timer.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --pub -p TEXT Limit to specific pub numbers. Repeat for multiple. │
│ --force Re-download even if local SHA-256 matches. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

17
docs/docs/cli/bib-attach-oig.md Normal file
View File

@@ -0,0 +1,17 @@
---
title: stack bib attach-oig
sidebar_position: 26
---
# `stack bib attach-oig`
```
Usage: stack bib attach-oig [OPTIONS]
Download and attach the PDF/HTML for every OIG guidance record.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --force Re-download even if unchanged. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

49
docs/docs/cli/bib-backfill-comments.md Normal file
View File

@@ -0,0 +1,49 @@
---
title: stack bib backfill-comments
sidebar_position: 19
---
# `stack bib backfill-comments`
```
Usage: stack bib backfill-comments [OPTIONS]
Enrich every reg-gov comment stub with body + attachments.
Walks items tagged ``source:regulations-gov`` whose ``abstract`` is
still empty and hits ``/v4/comments/{id}?include=attachments`` for
each one. Resumable: stop any time, start again, it picks up where
it left off by skipping already-enriched items. At 1.3s/req the API
calls run at ~2700/hr; attachment downloads are concurrent on the
CDN and don't count against the rate limit.
``--mirror`` swaps the API for the Mirrulations public S3 mirror
(#662): no API key, no cap, and mirror-only comments get created.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --docket TEXT Limit to one docket (e.g. CMS-2026-2377). Without │
│ it the walk covers every un-enriched stub in the │
│ store — a multi-day crawl at the reg.gov rate │
│ cap. │
│ --limit -n INTEGER Cap items processed this run. 0 = no cap │
│ (multi-day crawl). │
│ [default: 0] │
│ --sleep FLOAT Seconds between API calls. 3.7s ≈ 970/hr — just │
│ under the 1000/hr reg.gov hourly cap (the binding │
│ limit; the 50/min burst cap is looser). CDN │
│ downloads are concurrent and don't count. │
│ [default: 3.7] │
│ --log PATH Append-only progress log (survives restarts). │
│ [default: /tmp/bib-backfill-comments.log] │
│ --mirror Pull from the Mirrulations S3 mirror instead of │
│ the reg.gov API — bulk speed, no rate cap, also │
│ ingests comments the original farm never listed. │
│ Requires --docket. │
│ --force Run even if the docket is sealed. Bypasses the │
│ docket seal only: items already tagged │
│ enriched:ok (or enriched:gone) are still skipped, │
│ so this resumes an interrupted enrichment — it │
│ never re-enriches an item. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

25
docs/docs/cli/bib-cfr-jump.md Normal file
View File

@@ -0,0 +1,25 @@
---
title: stack bib cfr-jump
sidebar_position: 31
---
# `stack bib cfr-jump`
```
Usage: stack bib cfr-jump [OPTIONS] [REF]
Bidirectional CFR cite ↔ eCFR URL (#640); optionally place the link.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ ref [REF] CFR cite: "42 CFR 425.400(c)(1)(iv)" or "42 CFR Part 425". │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --url TEXT Reverse mode: print the cite for an eCFR URL. │
│ --md Print a markdown link. │
│ --place Attach the link to its bib regulation item (fr_links) │
│ so the next zotero sync carries it as a child link │
│ attachment. │
│ --label TEXT Label for the placed link. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

45
docs/docs/cli/bib-code-tags.md Normal file
View File

@@ -0,0 +1,45 @@
---
title: stack bib code-tags
sidebar_position: 32
---
# `stack bib code-tags`
```
Usage: stack bib code-tags [OPTIONS]
Stamp code:/family: item tags from chunk anchors and FR paragraphs.
Rolls up ``pfs.anchors.anchor_metadata`` — already stamped on
pgvector chunks by ``stack llm index``/``stack llm restamp`` — to
the bib item level: every code an item's chunks name becomes a
``code:<CODE>`` tag, every family a ``family:<KEY>`` tag. Rules also
get a pgvector-free source (``fr_anchors.text``, the FR paragraph
anchor map from ``stack bib fr-grab``); pass ``--anchors-only`` to
use only that and skip pgvector for rules (and skip comments/corpus
entirely — they have no other source). Additive and idempotent
(``bib.codetags.apply_code_tags``) — safe to rerun after every
``stack llm restamp``. Every selected source is merged into one
``item_key -> (codes, families)`` mapping before tagging (Ruling
A7), so an item that straddles two collections (e.g. rules and
corpus) is capped by ``--max-codes-per-item`` once, on its combined
code set — the printed per-source lines are informational only;
``tags_added``/``skipped_wide`` in the total come from that one
merged pass.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --collection TEXT Which collection: all | comments | │
│ rules | corpus. │
│ [default: all] │
│ --anchors-only Rules only, straight from fr_anchors │
│ text — skip pgvector entirely. │
│ --dry-run Compute the merged item→codes mapping │
│ and report what would be tagged; write │
│ nothing. │
│ --max-codes-per-item INTEGER Items with more distinct codes get │
│ family: tags only (+ hand-family │
│ codes). │
│ [default: 200] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

24
docs/docs/cli/bib-discover-pfs-rules.md Normal file
View File

@@ -0,0 +1,24 @@
---
title: stack bib discover-pfs-rules
sidebar_position: 15
---
# `stack bib discover-pfs-rules`
```
Usage: stack bib discover-pfs-rules [OPTIONS]
Walk the Federal Register API for every PFS rule since a date.
Upserts each as a bib ``Rule`` record tagged ``module:pfs``,
``source:federal-register``, ``rule:<cms-id>``. Cheap — the FR API
has no auth and no rate charge we can trigger at this scale
(~20 rules since 2017).
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --since TEXT [default: 2017-01-01] │
│ --until TEXT │
│ --dry-run │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

32
docs/docs/cli/bib-fetch-docket-comments.md Normal file
View File

@@ -0,0 +1,32 @@
---
title: stack bib fetch-docket-comments
sidebar_position: 16
---
# `stack bib fetch-docket-comments`
```
Usage: stack bib fetch-docket-comments [OPTIONS] DOCKET
Walk one docket's comments from its stored watermark; upsert as Source items.
The first run discovers the docket's commentable FR document (one
listing call) and stores it; later runs make no discovery calls.
Sealed dockets are skipped unless --force.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ * docket TEXT reg.gov docket id, e.g. CMS-2026-2377 [required] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --cms-id TEXT Associate comments with a specific CMS rule │
│ tag. │
│ --limit -n INTEGER Stop after N comments. 0 = all. [default: 0] │
│ --attachments Also download each comment's PDF/DOCX │
│ attachments. │
│ --sleep FLOAT Seconds between API calls (rate budget). │
│ [default: 1.3] │
│ --force Walk from page 1 even if sealed / │
│ watermarked. Never unseals. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

32
docs/docs/cli/bib-fetch-pfs-comments.md Normal file
View File

@@ -0,0 +1,32 @@
---
title: stack bib fetch-pfs-comments
sidebar_position: 17
---
# `stack bib fetch-pfs-comments`
```
Usage: stack bib fetch-pfs-comments [OPTIONS]
Discover every PFS proposed rule since *since* and pull new comments on each
docket from its stored watermark.
Sealed dockets (comment period closed + quiet period + an empty pull)
cost nothing: no rule-metadata fetch, no resolve call, no walk.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --since TEXT [default: 2017-01-01] │
│ --until TEXT │
│ --attachments │
│ --per-docket-limit INTEGER Cap comments fetched per docket. 0 = │
│ unlimited. │
│ [default: 0] │
│ --sleep FLOAT [default: 1.3] │
│ --docket TEXT Only pull this reg.gov docket (e.g. │
│ CMS-2026-2377); other dockets are skipped │
│ before any API call once known. │
│ --force Walk every docket from page 1, sealed or │
│ not. Never unseals. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

22
docs/docs/cli/bib-fr-grab.md Normal file
View File

@@ -0,0 +1,22 @@
---
title: stack bib fr-grab
sidebar_position: 29
---
# `stack bib fr-grab`
```
Usage: stack bib fr-grab [OPTIONS]
Grab federalregister.gov paragraph/page anchor maps into bib (#634).
One body-HTML fetch per rule; anchors persist in ``fr_anchors`` /
``fr_anchor_docs`` for offline jump-link resolution.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --key TEXT Grab one rule item by bib key. │
│ --all Backfill every rule item with an FR URL. │
│ --force Re-grab even if an anchor map already exists. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

26
docs/docs/cli/bib-fr-jump.md Normal file
View File

@@ -0,0 +1,26 @@
---
title: stack bib fr-jump
sidebar_position: 30
---
# `stack bib fr-jump`
```
Usage: stack bib fr-jump [OPTIONS] REF
Resolve an FR citation to its web jump link (#637); optionally place it.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ * ref TEXT FR reference: "91 FR 44218", "91 FR 44218 ¶3", "p-3600", │
│ or a quote. │
│ [required] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --key TEXT Bib item key to scope resolution. │
│ --md Print a markdown link. │
│ --place Record the link in fr_links so the next zotero sync │
│ carries it as a child link attachment. │
│ --label TEXT Label for the placed link. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

22
docs/docs/cli/bib-ingest-iom.md Normal file
View File

@@ -0,0 +1,22 @@
---
title: stack bib ingest-iom
sidebar_position: 20
---
# `stack bib ingest-iom`
```
Usage: stack bib ingest-iom [OPTIONS]
Crawl the CMS IOM index and upsert every chapter as a Manual.
Idempotent: re-running only adds missing chapters (dedup is by URL
in the store). Without --pub, ingests every manual the CMS index
advertises.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --pub -p TEXT Limit to specific pub numbers (e.g. 100-02). Repeat │
│ for multiple. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

28
docs/docs/cli/bib-ingest-mail.md Normal file
View File

@@ -0,0 +1,28 @@
---
title: stack bib ingest-mail
sidebar_position: 18
---
# `stack bib ingest-mail`
```
Usage: stack bib ingest-mail [OPTIONS]
Pull UNSEEN mail from a mailbox via IMAPS, upsert as bib Sources.
Idempotent: marks each ingested message ``\Seen`` on the server,
so re-runs only fetch new mail. Attachments ride along into bib
storage. After this, ``stack bib sync-zotero`` routes everything
tagged ``source:email`` into the appropriate Zotero collection.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --user TEXT Mailbox address to poll (must have a password in │
│ .state/mail/credentials.json). │
│ [default: cmsupdates@mail.fhirworx.io] │
│ --host TEXT [default: mail.fhirworx.io] │
│ --port INTEGER [default: 993] │
│ --folder TEXT [default: INBOX] │
│ --limit -n INTEGER [default: 0] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

18
docs/docs/cli/bib-ingest-oig.md Normal file
View File

@@ -0,0 +1,18 @@
---
title: stack bib ingest-oig
sidebar_position: 25
---
# `stack bib ingest-oig`
```
Usage: stack bib ingest-oig [OPTIONS]
Crawl OIG Compliance Program Guidance + fraud alerts indexes.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --kind -k TEXT Restrict to a subset: 'cpg' and/or 'alerts'. Repeat │
│ for multiple. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

30
docs/docs/cli/bib-refresh-iom.md Normal file
View File

@@ -0,0 +1,30 @@
---
title: stack bib refresh-iom
sidebar_position: 28
---
# `stack bib refresh-iom`
```
Usage: stack bib refresh-iom [OPTIONS]
One-shot: ingest-iom → watch-iom → attach-iom → sync-zotero.
Safe to put behind a cron. Idempotent at every step:
- ingest: upserts by URL, prunes stale records
- watch: re-attaches futurepdf.pdf only when its SHA-256 changes
- attach: skips PDFs whose checksum matches the stored copy
- sync: skips Zotero items whose URL already exists
Zotero is held (stopped) during the sync phase only.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --pub -p TEXT │
│ --file-issue --no-file-issue When watch-iom detects a │
│ futurepdf.pdf change, auto-file a │
│ Gitea issue tagged 'cms-update'. │
│ Disable for local/dev runs. │
│ [default: file-issue] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

16
docs/docs/cli/bib-refresh-oig.md Normal file
View File

@@ -0,0 +1,16 @@
---
title: stack bib refresh-oig
sidebar_position: 27
---
# `stack bib refresh-oig`
```
Usage: stack bib refresh-oig [OPTIONS]
One-shot: ingest-oig → attach-oig → sync-zotero (OIG tag).
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

51
docs/docs/cli/bib-sync-zotero.md Normal file
View File

@@ -0,0 +1,51 @@
---
title: stack bib sync-zotero
sidebar_position: 24
---
# `stack bib sync-zotero`
```
Usage: stack bib sync-zotero [OPTIONS]
Push bib items (and their attachments) to the Zotero database.
Zotero holds an exclusive write lock on its SQLite file while
running. By default we stop the 'zotero' compose service for the
duration of the sync and start it again afterward.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --tag TEXT Only sync items matching a │
│ tag. │
│ --type TEXT Only sync one item_type. │
│ --hold-zotero --no-hold Stop the 'zotero' container │
│ during sync so Zotero │
│ releases its exclusive │
│ SQLite lock; restart │
│ afterwards. │
│ [default: hold-zotero] │
│ --dedupe-collection TEXT DEPRECATED — use │
│ --merge-dupes instead, │
│ which merges rather than │
│ deletes. After the push, │
│ remove title+date │
│ duplicates from this Zotero │
│ collection (and from bib, │
│ so the next sync can't │
│ resurrect them). CMS │
│ listservs re-send identical │
│ mail with fresh │
│ Message-IDs, so duplicates │
│ otherwise accumulate — e.g. │
│ 'Cmsupdates'. │
│ --merge-dupes --no-merge-dupes After the push, merge │
│ same-URL duplicate items │
│ library-wide (union │
│ tags/collections/notes/att… │
│ keep-earliest). Supersedes │
│ --dedupe-collection, which │
│ deletes without merging. │
│ [default: no-merge-dupes] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

21
docs/docs/cli/bib-watch-iom.md Normal file
View File

@@ -0,0 +1,21 @@
---
title: stack bib watch-iom
sidebar_position: 22
---
# `stack bib watch-iom`
```
Usage: stack bib watch-iom [OPTIONS]
Poll the CMS futurepdf.pdf schedule; attach if changed.
CMS publishes upcoming IOM transmittals at
``/manuals/downloads/futurepdf.pdf``. When its checksum changes,
chapters in the main manuals are probably about to update — run
``ingest-iom`` + ``attach-iom`` afterward to pick them up.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

2
docs/docs/cli/bib.md
View File

@@ -52,5 +52,7 @@ Usage: stack bib [OPTIONS] COMMAND [ARGS]...
│ optionally place it. │ │ optionally place it. │
│ cfr-jump Bidirectional CFR cite ↔ eCFR URL (#640); optionally │ │ cfr-jump Bidirectional CFR cite ↔ eCFR URL (#640); optionally │
│ place the link. │ │ place the link. │
│ code-tags Stamp code:/family: item tags from chunk anchors and │
│ FR paragraphs. │
╰──────────────────────────────────────────────────────────────────────────────╯ ╰──────────────────────────────────────────────────────────────────────────────╯
``` ```

18
docs/docs/cli/comments-dockets.md Normal file
View File

@@ -0,0 +1,18 @@
---
title: stack comments dockets
sidebar_position: 37
---
# `stack comments dockets`
```
Usage: stack comments dockets [OPTIONS]
List dockets: id, rule, close date, watermark, counts, seal.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --discover Populate rows for every reg-docket: tag in bib that has │
│ no dockets row (one reg.gov listing call each). │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

27
docs/docs/cli/comments-extract-ocr.md Normal file
View File

@@ -0,0 +1,27 @@
---
title: stack comments extract-ocr
sidebar_position: 35
---
# `stack comments extract-ocr`
```
Usage: stack comments extract-ocr [OPTIONS]
OCR pass for ``ocr_needed`` attachments (#663).
Finds comment dirs whose combined.md frontmatter flags any
attachment ``ocr_needed`` and re-extracts them with the RapidOCR
engine (rewrites combined.md + sibling transcripts, keeps the flag
on anything OCR can't clear). CPU-only, pip-only:
uv run --with rapidocr-onnxruntime stack comments extract-ocr
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --root PATH Comments root dir. [default: .state/comments] │
│ --docket TEXT Limit to one docket (e.g. CMS-2023-0121). │
│ --limit INTEGER Cap comment dirs processed this run. 0 = no cap. │
│ [default: 0] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

37
docs/docs/cli/comments-extract.md Normal file
View File

@@ -0,0 +1,37 @@
---
title: stack comments extract
sidebar_position: 34
---
# `stack comments extract`
```
Usage: stack comments extract [OPTIONS]
Walk comment dirs and write combined.md (PDF/DOCX → text).
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --root PATH Comments root dir. │
│ [default: .state/comments] │
│ --docket TEXT Limit to one docket (e.g. │
│ CMS-2023-0121). │
│ --limit INTEGER Cap dirs processed this run. 0 = no │
│ cap. │
│ [default: 0] │
│ --workers INTEGER Concurrent extraction threads. │
│ [default: 8] │
│ --force Re-extract dirs that already have │
│ combined.md. │
│ --bib --no-bib Look up inline body from bib.sqlite │
│ items.abstract. │
│ [default: bib] │
│ --attach --no-attach Attach combined.md back to its bib │
│ item (so it flows to Zotero via `bib │
│ sync`). Implies --bib; no-op when │
│ --no-bib. │
│ [default: attach] │
│ --reattach Also attach combined.md notes for dirs │
│ that were already current (repair). │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

20
docs/docs/cli/comments-seal.md Normal file
View File

@@ -0,0 +1,20 @@
---
title: stack comments seal
sidebar_position: 38
---
# `stack comments seal`
```
Usage: stack comments seal [OPTIONS] DOCKET
Mark a docket complete: every stage skips it until unsealed or --force.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ * docket TEXT reg.gov docket id [required] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --reason TEXT [default: manual] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

17
docs/docs/cli/comments-stats.md Normal file
View File

@@ -0,0 +1,17 @@
---
title: stack comments stats
sidebar_position: 36
---
# `stack comments stats`
```
Usage: stack comments stats [OPTIONS]
Rebuild the index and print per-status counts.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --root PATH Comments root dir. [default: .state/comments] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

19
docs/docs/cli/comments-unseal.md Normal file
View File

@@ -0,0 +1,19 @@
---
title: stack comments unseal
sidebar_position: 39
---
# `stack comments unseal`
```
Usage: stack comments unseal [OPTIONS] DOCKET
Reopen a sealed docket.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ * docket TEXT reg.gov docket id [required] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

25
docs/docs/cli/comments.md Normal file
View File

@@ -0,0 +1,25 @@
---
title: stack comments
sidebar_position: 33
---
# `stack comments`
```
Usage: stack comments [OPTIONS] COMMAND [ARGS]...
CMS rulemaking comment text extraction & analysis.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ extract Walk comment dirs and write combined.md (PDF/DOCX → text). │
│ extract-ocr OCR pass for ``ocr_needed`` attachments (#663). │
│ stats Rebuild the index and print per-status counts. │
│ dockets List dockets: id, rule, close date, watermark, counts, seal. │
│ seal Mark a docket complete: every stage skips it until unsealed or │
│ --force. │
│ unseal Reopen a sealed docket. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

2
docs/docs/cli/db-comment.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack db comment title: stack db comment
sidebar_position: 21 sidebar_position: 51
--- ---
# `stack db comment` # `stack db comment`

2
docs/docs/cli/db-inspect.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack db inspect title: stack db inspect
sidebar_position: 22 sidebar_position: 52
--- ---
# `stack db inspect` # `stack db inspect`

2
docs/docs/cli/db.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack db title: stack db
sidebar_position: 20 sidebar_position: 50
--- ---
# `stack db` # `stack db`

2
docs/docs/cli/docs-build.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack docs build title: stack docs build
sidebar_position: 24 sidebar_position: 54
--- ---
# `stack docs build` # `stack docs build`

2
docs/docs/cli/docs-generate.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack docs generate title: stack docs generate
sidebar_position: 26 sidebar_position: 56
--- ---
# `stack docs generate` # `stack docs generate`

2
docs/docs/cli/docs-serve.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack docs serve title: stack docs serve
sidebar_position: 25 sidebar_position: 55
--- ---
# `stack docs serve` # `stack docs serve`

2
docs/docs/cli/docs.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack docs title: stack docs
sidebar_position: 23 sidebar_position: 53
--- ---
# `stack docs` # `stack docs`

2
docs/docs/cli/lake-deploy.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack lake deploy title: stack lake deploy
sidebar_position: 16 sidebar_position: 41
--- ---
# `stack lake deploy` # `stack lake deploy`

2
docs/docs/cli/lake-load.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack lake load title: stack lake load
sidebar_position: 17 sidebar_position: 42
--- ---
# `stack lake load` # `stack lake load`

2
docs/docs/cli/lake-parity.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack lake parity title: stack lake parity
sidebar_position: 19 sidebar_position: 44
--- ---
# `stack lake parity` # `stack lake parity`

2
docs/docs/cli/lake-validate.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack lake validate title: stack lake validate
sidebar_position: 18 sidebar_position: 43
--- ---
# `stack lake validate` # `stack lake validate`

2
docs/docs/cli/lake.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack lake title: stack lake
sidebar_position: 15 sidebar_position: 40
--- ---
# `stack lake` # `stack lake`

20
docs/docs/cli/llm-hosts.md Normal file
View File

@@ -0,0 +1,20 @@
---
title: stack llm hosts
sidebar_position: 48
---
# `stack llm hosts`
```
Usage: stack llm hosts [OPTIONS]
Show the Ollama fleet: declared VRAM, liveness, models, and which host + model
would answer a chat right now.
Source .env first (`set -a; . ./.env; set +a`) so LLM_OLLAMA_HOSTS
lists the whole fleet; the compose service gets it from its environment.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

31
docs/docs/cli/llm-index.md Normal file
View File

@@ -0,0 +1,31 @@
---
title: stack llm index
sidebar_position: 46
---
# `stack llm index`
```
Usage: stack llm index [OPTIONS]
Embed comments/rules/corpus into pgvector (incremental, resumable).
Comments are processed newest-posted first; rules come from their FR
paragraph anchor maps (run `stack bib fr-grab` first for exact links);
corpus = every non-comment bib item incl. Zotero-only storage PDFs.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --collection TEXT Which collection: comments | rules | │
│ corpus | all. │
│ [default: comments] │
│ --docket TEXT Limit comments to one docket id. │
│ --key TEXT Limit rules/corpus to these item │
│ keys. │
│ --force --no-force Re-embed even when unchanged. │
│ [default: no-force] │
│ --limit INTEGER Stop after N docs per collection (0 = │
│ all). │
│ [default: 0] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

31
docs/docs/cli/llm-restamp.md Normal file
View File

@@ -0,0 +1,31 @@
---
title: stack llm restamp
sidebar_position: 47
---
# `stack llm restamp`
```
Usage: stack llm restamp [OPTIONS]
Backfill codes/families/elements onto already-indexed chunks.
Metadata-only: computes `pfs.anchors.anchor_metadata` from each row's
stored document text and merges it into `cmetadata` with a jsonb `||`,
never touching the embedding — no GPU time, no re-chunking. Safe to
re-run: rows already stamped with the current values are skipped.
F6: never rebuilds the HNSW index (`ensure_hnsw` ALTERs the vector
column and takes an ACCESS EXCLUSIVE lock for the life of the
rebuild) — only the cheap metadata GIN indexes, and not even those
under `--dry-run`.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --collection TEXT Which collection: comments | rules | corpus | │
│ all. │
│ [default: comments] │
│ --batch INTEGER Rows per transaction. [default: 2000] │
│ --dry-run Scan and report without writing. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

18
docs/docs/cli/llm-serve.md Normal file
View File

@@ -0,0 +1,18 @@
---
title: stack llm serve
sidebar_position: 49
---
# `stack llm serve`
```
Usage: stack llm serve [OPTIONS]
Serve the SSO-guarded chat UI (llm.api:app).
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --host TEXT Bind address. [default: 127.0.0.1] │
│ --port INTEGER Port. [default: 8000] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

23
docs/docs/cli/llm.md Normal file
View File

@@ -0,0 +1,23 @@
---
title: stack llm
sidebar_position: 45
---
# `stack llm`
```
Usage: stack llm [OPTIONS] COMMAND [ARGS]...
Local RAG — index, search, tag.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ index Embed comments/rules/corpus into pgvector (incremental, resumable). │
│ restamp Backfill codes/families/elements onto already-indexed chunks. │
│ hosts Show the Ollama fleet: declared VRAM, liveness, models, and which │
│ host + model would answer a chat right now. │
│ serve Serve the SSO-guarded chat UI (llm.api:app). │
╰──────────────────────────────────────────────────────────────────────────────╯
```

16
docs/docs/cli/mail-attach-smarthost.md Normal file
View File

@@ -0,0 +1,16 @@
---
title: stack mail attach-smarthost
sidebar_position: 97
---
# `stack mail attach-smarthost`
```
Usage: stack mail attach-smarthost [OPTIONS]
Register at Resend, push DNS, install smarthost on droplet.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

16
docs/docs/cli/mail-dkim-export.md Normal file
View File

@@ -0,0 +1,16 @@
---
title: stack mail dkim-export
sidebar_position: 96
---
# `stack mail dkim-export`
```
Usage: stack mail dkim-export [OPTIONS]
Publish Maddy's DKIM public key to Cloudflare. Idempotent.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

19
docs/docs/cli/mail-dns.md Normal file
View File

@@ -0,0 +1,19 @@
---
title: stack mail dns
sidebar_position: 95
---
# `stack mail dns`
```
Usage: stack mail dns [OPTIONS]
Upsert mail DNS records in Cloudflare. Idempotent.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --claim-apex Also write apex MX/SPF/DMARC. Skip when another │
│ provider (Proton, Fastmail, Workspace) already owns │
│ the apex. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

17
docs/docs/cli/mail-down.md Normal file
View File

@@ -0,0 +1,17 @@
---
title: stack mail down
sidebar_position: 93
---
# `stack mail down`
```
Usage: stack mail down [OPTIONS]
Destroy the mail droplet. Idempotent no-op if none exists.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --yes Skip confirmation. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

18
docs/docs/cli/mail-provision.md Normal file
View File

@@ -0,0 +1,18 @@
---
title: stack mail provision
sidebar_position: 91
---
# `stack mail provision`
```
Usage: stack mail provision [OPTIONS]
End-to-end idempotent: up → wait → DNS → DKIM → smarthost → git env.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --region TEXT DO region slug (one your ISP can reach). │
│ [default: nyc3] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

20
docs/docs/cli/mail-rotate-creds.md Normal file
View File

@@ -0,0 +1,20 @@
---
title: stack mail rotate-creds
sidebar_position: 98
---
# `stack mail rotate-creds`
```
Usage: stack mail rotate-creds [OPTIONS] USER
Mint a new password for *user*@fhirworx.io and push to droplet.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ * user TEXT Mailbox to rotate (e.g. git, postmaster, cmsupdates). │
│ [required] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

20
docs/docs/cli/mail-seed-mailboxes.md Normal file
View File

@@ -0,0 +1,20 @@
---
title: stack mail seed-mailboxes
sidebar_position: 99
---
# `stack mail seed-mailboxes`
```
Usage: stack mail seed-mailboxes [OPTIONS]
Ensure the standard set of mailboxes exists (postmaster, git, cmsupdates).
Idempotent: only creates missing ones; existing mailboxes keep
their cached passwords. Useful after adding a new entry to
``DEFAULT_MAILBOXES`` in mail/droplet.py.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

16
docs/docs/cli/mail-status.md Normal file
View File

@@ -0,0 +1,16 @@
---
title: stack mail status
sidebar_position: 94
---
# `stack mail status`
```
Usage: stack mail status [OPTIONS]
Show droplet state from DO + PTR check.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

17
docs/docs/cli/mail-up.md Normal file
View File

@@ -0,0 +1,17 @@
---
title: stack mail up
sidebar_position: 92
---
# `stack mail up`
```
Usage: stack mail up [OPTIONS]
Adopt-or-create the mail droplet. Idempotent.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --region TEXT DO region slug. [default: nyc3] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

19
docs/docs/cli/mail-wire-git.md Normal file
View File

@@ -0,0 +1,19 @@
---
title: stack mail wire-git
sidebar_position: 100
---
# `stack mail wire-git`
```
Usage: stack mail wire-git [OPTIONS]
Write .state/git/mailer.env from the mail droplet's credentials.
Then run ``docker compose up -d git`` to apply (a plain ``docker
restart`` won't re-read the new env_file directive in compose.yml).
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

33
docs/docs/cli/mail.md Normal file
View File

@@ -0,0 +1,33 @@
---
title: stack mail
sidebar_position: 90
---
# `stack mail`
```
Usage: stack mail [OPTIONS] COMMAND [ARGS]...
Maddy mail server (DO + Resend smarthost).
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ provision End-to-end idempotent: up → wait → DNS → DKIM → smarthost │
│ → git env. │
│ up Adopt-or-create the mail droplet. Idempotent. │
│ down Destroy the mail droplet. Idempotent no-op if none exists. │
│ status Show droplet state from DO + PTR check. │
│ dns Upsert mail DNS records in Cloudflare. Idempotent. │
│ dkim-export Publish Maddy's DKIM public key to Cloudflare. Idempotent. │
│ attach-smarthost Register at Resend, push DNS, install smarthost on │
│ droplet. │
│ rotate-creds Mint a new password for *user*@fhirworx.io and push to │
│ droplet. │
│ seed-mailboxes Ensure the standard set of mailboxes exists (postmaster, │
│ git, cmsupdates). │
│ wire-git Write .state/git/mailer.env from the mail droplet's │
│ credentials. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

19
docs/docs/cli/perf-show.md Normal file
View File

@@ -0,0 +1,19 @@
---
title: stack perf show
sidebar_position: 60
---
# `stack perf show`
```
Usage: stack perf show [OPTIONS]
Pretty-print spans from the local JSONL trace file.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --path TEXT Path to JSONL span file. │
│ [default: traces/spans.jsonl] │
│ --last INTEGER Show last N spans. [default: 20] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

19
docs/docs/cli/perf.md Normal file
View File

@@ -0,0 +1,19 @@
---
title: stack perf
sidebar_position: 59
---
# `stack perf`
```
Usage: stack perf [OPTIONS] COMMAND [ARGS]...
Pipeline telemetry utilities.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ show Pretty-print spans from the local JSONL trace file. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

2
docs/docs/cli/pfs-cpt-ingest.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack pfs cpt-ingest title: stack pfs cpt-ingest
sidebar_position: 69 sidebar_position: 72
--- ---
# `stack pfs cpt-ingest` # `stack pfs cpt-ingest`

6
docs/docs/cli/pfs-elements.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack pfs elements title: stack pfs elements
sidebar_position: 63 sidebar_position: 66
--- ---
# `stack pfs elements` # `stack pfs elements`
@@ -14,7 +14,9 @@ Usage: stack pfs elements [OPTIONS]
│ --code TEXT HCPCS/CPT code (repeatable). │ │ --code TEXT HCPCS/CPT code (repeatable). │
│ --family TEXT Expand a registered family (CCM, APCM, …); │ │ --family TEXT Expand a registered family (CCM, APCM, …); │
│ repeatable. │ │ repeatable. │
│ --all-payable Every A/R/T code in the newest RVU year. │ │ --all-payable Every A/R/T code in the newest RVU year │
│ (experimental: ~20 min of SQL before any model │
│ call). │
│ --no-llm Skip the local-model classifier (unknown lines go │ │ --no-llm Skip the local-model classifier (unknown lines go │
│ to review). │ │ to review). │
│ --dry-run Extract and report; write nothing. │ │ --dry-run Extract and report; write nothing. │

2
docs/docs/cli/pfs-families.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack pfs families title: stack pfs families
sidebar_position: 67 sidebar_position: 68
--- ---
# `stack pfs families` # `stack pfs families`

20
docs/docs/cli/pfs-guidance.md Normal file
View File

@@ -0,0 +1,20 @@
---
title: stack pfs guidance
sidebar_position: 69
---
# `stack pfs guidance`
```
Usage: stack pfs guidance [OPTIONS]
CFR/IOM/MLN sub-regulatory crosswalk per code family (pfs.code_guidance):
which CFR sections and IOM manual chapters the family's FR paragraphs and the
CPT manual cite, with FR provenance.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --family TEXT Family key (CCM, APCM, …); repeatable. │
│ --write │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

2
docs/docs/cli/pfs-lineage.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack pfs lineage title: stack pfs lineage
sidebar_position: 64 sidebar_position: 67
--- ---
# `stack pfs lineage` # `stack pfs lineage`

30
docs/docs/cli/pfs-reaction.md Normal file
View File

@@ -0,0 +1,30 @@
---
title: stack pfs reaction
sidebar_position: 70
---
# `stack pfs reaction`
```
Usage: stack pfs reaction [OPTIONS]
Public-reaction series per code family (pfs.code_reaction): comment counts per
docket (share of the docket's commenters mentioning the family), an optional
stance sample, and the FR Comment:/Response: pair counts per rule as the
pre-2017 proxy. Always recomputed live from pgvector/bib — there is no
persisted-table fallback to read without --write, so --write only adds
persisting the result.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --family TEXT Family key (CCM, APCM, …); repeatable. │
│ --code TEXT A single code as its own pseudo-family, │
│ keyed by the code; repeatable. │
│ --stance-sample INTEGER Classify the newest N commenters per docket │
│ into support/oppose/modify/unclear via the │
│ self-hosted classifier (0 = skip; no stance │
│ sample means no LLM call at all). │
│ [default: 0] │
│ --write │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

2
docs/docs/cli/pfs-review.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack pfs review title: stack pfs review
sidebar_position: 66 sidebar_position: 71
--- ---
# `stack pfs review` # `stack pfs review`

17
docs/docs/cli/pfs.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack pfs title: stack pfs
sidebar_position: 64 sidebar_position: 65
--- ---
# `stack pfs` # `stack pfs`
@@ -20,6 +20,21 @@ Usage: stack pfs [OPTIONS] COMMAND [ARGS]...
│ cross-checked. │ │ cross-checked. │
│ families Derive families from pfs.code_element / pfs.code_event / │ │ families Derive families from pfs.code_element / pfs.code_event / │
│ pfs.rvu. │ │ pfs.rvu. │
│ guidance CFR/IOM/MLN sub-regulatory crosswalk per code family │
│ (pfs.code_guidance): which CFR sections and IOM manual chapters │
│ the │
│ family's FR paragraphs and the CPT manual cite, with FR │
│ provenance. │
│ reaction Public-reaction series per code family (pfs.code_reaction): │
│ comment │
│ counts per docket (share of the docket's commenters mentioning │
│ the │
│ family), an optional stance sample, and the FR │
│ Comment:/Response: │
│ pair counts per rule as the pre-2017 proxy. Always recomputed │
│ live │
│ from pgvector/bib — there is no persisted-table fallback to read │
│ without --write, so --write only adds persisting the result. │
│ review Element lines the classifier could not place. │ │ review Element lines the classifier could not place. │
│ cpt-ingest Parse CPT EPUB editions into │ │ cpt-ingest Parse CPT EPUB editions into │
│ pfs.cpt_section/cpt_code/cpt_instruction/ │ │ pfs.cpt_section/cpt_code/cpt_instruction/ │

29
docs/docs/cli/prisma-eligible.md Normal file
View File

@@ -0,0 +1,29 @@
---
title: stack prisma eligible
sidebar_position: 84
---
# `stack prisma eligible`
```
Usage: stack prisma eligible [OPTIONS] NAME
Stage 3 — LLM-driven full-text eligibility pass.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ * name TEXT Project slug. [required] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --limit -n INTEGER [default: 0] │
│ --max-chars INTEGER Cap per-item markdown (full text) at │
│ N chars so bounded-context local │
│ models keep the criteria in window. 0 │
│ = no cap. │
│ [default: 0] │
│ --db PATH [default: │
│ data/zotero/data/zotero.sqlite] │
│ --storage PATH [default: data/zotero/data/storage] │
│ --hold --no-hold [default: hold] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

23
docs/docs/cli/prisma-export.md Normal file
View File

@@ -0,0 +1,23 @@
---
title: stack prisma export
sidebar_position: 81
---
# `stack prisma export`
```
Usage: stack prisma export [OPTIONS] ZOT_ID
Dump one Zotero item as the LLM-ready markdown snapshot.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ * zot_id INTEGER Zotero itemID to export. [required] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --fulltext │
│ --db PATH [default: data/zotero/data/zotero.sqlite] │
│ --storage PATH [default: data/zotero/data/storage] │
│ --hold --no-hold [default: hold] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

24
docs/docs/cli/prisma-extract.md Normal file
View File

@@ -0,0 +1,24 @@
---
title: stack prisma extract
sidebar_position: 85
---
# `stack prisma extract`
```
Usage: stack prisma extract [OPTIONS] NAME
Stage 3+ — structured data extraction on included studies.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ * name TEXT Project slug. [required] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --limit -n INTEGER [default: 0] │
│ --db PATH [default: │
│ data/zotero/data/zotero.sqlite] │
│ --storage PATH [default: data/zotero/data/storage] │
│ --hold --no-hold [default: hold] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

47
docs/docs/cli/prisma-fetch.md Normal file
View File

@@ -0,0 +1,47 @@
---
title: stack prisma fetch
sidebar_position: 87
---
# `stack prisma fetch`
```
Usage: stack prisma fetch [OPTIONS] NAME
Fetch PDFs for every non-excluded item lacking an attachment.
Source cascade: Unpaywall → PMC → fallback (via the VPN droplet).
Items excluded at stage 2 are untouched — the queue filter only
passes `screen:include` and `screen:uncertain` through.
Tunnel lifecycle is automatic: if a droplet is tracked in
``.state/prisma-vpn/`` this command opens an SSH port-forward for
the duration of the fetch and closes it on exit. No sudo required.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ * name TEXT Project slug. [required] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --limit -n INTEGER [default: 0] │
│ --workers -w INTEGER Concurrent fetch workers (fallback │
│ tier stays serialized). │
│ [default: 8] │
│ --db PATH [default: │
│ data/zotero/data/zotero.sqlite] │
│ --storage PATH [default: data/zotero/data/storage] │
│ --scratch PATH [default: .state/prisma-fetch] │
│ --proxy TEXT Override PRISMA_FETCH_PROXY (else │
│ env). │
│ --vpn --no-vpn Auto-open an SSH tunnel to the │
│ provisioned droplet when one exists. │
│ --no-vpn skips the tunnel and uses │
│ only the unproxied sources. │
│ [default: vpn] │
│ --verify --no-verify Curl through the proxy once before │
│ fetching and confirm egress is │
│ non-US. │
│ [default: verify] │
│ --hold --no-hold [default: hold] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

22
docs/docs/cli/prisma-flow.md Normal file
View File

@@ -0,0 +1,22 @@
---
title: stack prisma flow
sidebar_position: 86
---
# `stack prisma flow`
```
Usage: stack prisma flow [OPTIONS] NAME
Emit the PRISMA 2020 flow diagram from current tag counts.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ * name TEXT Project slug. [required] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --mermaid --text [default: mermaid] │
│ --db PATH [default: data/zotero/data/zotero.sqlite] │
│ --hold --no-hold [default: hold] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

23
docs/docs/cli/prisma-init.md Normal file
View File

@@ -0,0 +1,23 @@
---
title: stack prisma init
sidebar_position: 80
---
# `stack prisma init`
```
Usage: stack prisma init [OPTIONS] NAME
Create (or re-load) the project's three Zotero anchor items: criteria,
extraction-template, and the reasons codebook.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ * name TEXT Project slug, e.g. skin-subs. [required] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --db PATH Zotero SQLite path. │
│ [default: data/zotero/data/zotero.sqlite] │
│ --hold --no-hold [default: hold] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

19
docs/docs/cli/prisma-ping-llm.md Normal file
View File

@@ -0,0 +1,19 @@
---
title: stack prisma ping-llm
sidebar_position: 82
---
# `stack prisma ping-llm`
```
Usage: stack prisma ping-llm [OPTIONS]
Send a cheap ping through the configured provider.
Useful for verifying ``PRISMA_LLM_PROVIDER`` + auth before kicking
off a long screening run.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

32
docs/docs/cli/prisma-run.md Normal file
View File

@@ -0,0 +1,32 @@
---
title: stack prisma run
sidebar_position: 88
---
# `stack prisma run`
```
Usage: stack prisma run [OPTIONS] NAME
Orchestrate all PRISMA stages in order.
screen → fetch → eligible → extract → flow, all inside one zotero
hold. Fetch only pulls PDFs for non-excluded items; stage 2
exclusions are terminal by design.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ * name TEXT Project slug. [required] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --limit -n INTEGER Same limit applied to every stage. │
│ [default: 0] │
│ --skip-fetch │
│ --skip-extract │
│ --db PATH [default: │
│ data/zotero/data/zotero.sqlite] │
│ --storage PATH [default: │
│ data/zotero/data/storage] │
│ --hold --no-hold [default: hold] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

34
docs/docs/cli/prisma-screen.md Normal file
View File

@@ -0,0 +1,34 @@
---
title: stack prisma screen
sidebar_position: 83
---
# `stack prisma screen`
```
Usage: stack prisma screen [OPTIONS] NAME
Stage 2 — LLM-driven title/abstract screening pass.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ * name TEXT Project slug. [required] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --limit -n INTEGER 0 = screen every unscreened item. │
│ [default: 0] │
│ --db PATH [default: │
│ data/zotero/data/zotero.sqlite] │
│ --storage PATH [default: data/zotero/data/storage] │
│ --hold --no-hold [default: hold] │
│ --hosts TEXT Comma-separated Ollama base URLs — │
│ fleet mode (#655): parallel │
│ screening via native /api/chat with │
│ per-request num_ctx; bypasses │
│ PRISMA_LLM_* env config. │
│ --fleet-model TEXT Model name on every fleet host. │
│ [default: qwen2.5:14b] │
│ --num-ctx INTEGER Context window per fleet request. │
│ [default: 8192] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

24
docs/docs/cli/prisma-vpn.md Normal file
View File

@@ -0,0 +1,24 @@
---
title: stack prisma vpn
sidebar_position: 89
---
# `stack prisma vpn`
```
Usage: stack prisma vpn [OPTIONS] COMMAND [ARGS]...
DigitalOcean SOCKS5 exit for fallback fetch (SSH tunnel, no sudo needed).
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ up Provision the droplet, wait for dante, wire Zotero up. │
│ down Destroy the droplet and wipe local state. │
│ status Report whether a PRISMA VPN droplet is currently tracked. │
│ attach-zotero Start the sidecar tunnel + rewire Zotero prefs. │
│ detach-zotero Stop the sidecar and restore Zotero's pre-proxy prefs. │
│ verify Open the tunnel, curl ifconfig.co, print egress, close. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

30
docs/docs/cli/prisma.md Normal file
View File

@@ -0,0 +1,30 @@
---
title: stack prisma
sidebar_position: 79
---
# `stack prisma`
```
Usage: stack prisma [OPTIONS] COMMAND [ARGS]...
LLM-driven PRISMA screening workflow.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ init Create (or re-load) the project's three Zotero anchor items: │
│ criteria, extraction-template, and the reasons codebook. │
│ export Dump one Zotero item as the LLM-ready markdown snapshot. │
│ ping-llm Send a cheap ping through the configured provider. │
│ screen Stage 2 — LLM-driven title/abstract screening pass. │
│ eligible Stage 3 — LLM-driven full-text eligibility pass. │
│ extract Stage 3+ — structured data extraction on included studies. │
│ flow Emit the PRISMA 2020 flow diagram from current tag counts. │
│ fetch Fetch PDFs for every non-excluded item lacking an attachment. │
│ run Orchestrate all PRISMA stages in order. │
│ vpn DigitalOcean SOCKS5 exit for fallback fetch (SSH tunnel, no sudo │
│ needed). │
╰──────────────────────────────────────────────────────────────────────────────╯
```

16
docs/docs/cli/rec-list.md Normal file
View File

@@ -0,0 +1,16 @@
---
title: stack rec list
sidebar_position: 62
---
# `stack rec list`
```
Usage: stack rec list [OPTIONS]
List registered pricers and their compare columns.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

21
docs/docs/cli/rec-opps.md Normal file
View File

@@ -0,0 +1,21 @@
---
title: stack rec opps
sidebar_position: 64
---
# `stack rec opps`
```
Usage: stack rec opps [OPTIONS]
Reconcile OPPS calculated rates against Addendum B.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --year -y INTEGER Year to reconcile; omit for all. │
│ --format -f TEXT Output format: md | json. [default: md] │
│ --output -o PATH Write to file instead of stdout. │
│ --tolerance -t INTEGER Cents of tolerance for an 'exact' match. │
│ [default: 0] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

21
docs/docs/cli/rec-pfs.md Normal file
View File

@@ -0,0 +1,21 @@
---
title: stack rec pfs
sidebar_position: 63
---
# `stack rec pfs`
```
Usage: stack rec pfs [OPTIONS]
Reconcile PFS calculated fees against carrier_locality.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --year -y INTEGER Year to reconcile; omit for all. │
│ --format -f TEXT Output format: md | json. [default: md] │
│ --output -o PATH Write to file instead of stdout. │
│ --tolerance -t INTEGER Cents of tolerance for an 'exact' match. │
│ [default: 0] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

21
docs/docs/cli/rec.md Normal file
View File

@@ -0,0 +1,21 @@
---
title: stack rec
sidebar_position: 61
---
# `stack rec`
```
Usage: stack rec [OPTIONS] COMMAND [ARGS]...
Reconcile calculated payments against CMS ground-truth files.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ list List registered pricers and their compare columns. │
│ pfs Reconcile PFS calculated fees against carrier_locality. │
│ opps Reconcile OPPS calculated rates against Addendum B. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

10
docs/docs/cli/run.md
View File

@@ -14,9 +14,11 @@ Usage: stack run [OPTIONS] NAME
│ * name TEXT Pipeline name (e.g. readmissions). [required] │ │ * name TEXT Pipeline name (e.g. readmissions). [required] │
╰──────────────────────────────────────────────────────────────────────────────╯ ╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮ ╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --target TEXT Target context: local, lake, trino, databricks. │ │ --target TEXT Target context: local, lake, trino, databricks, │
│ Defaults to stack.toml. │ │ spark. │
│ --save Save pipeline outputs back to the database. │ │ --catalog TEXT Unity Catalog name (overrides config). Implies │
│ --help Show this message and exit. │ │ target=spark. │
│ --save Save pipeline outputs back to the database. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯ ╰──────────────────────────────────────────────────────────────────────────────╯
``` ```

23
docs/docs/cli/zot-dump-schema.md Normal file
View File

@@ -0,0 +1,23 @@
---
title: stack zot dump-schema
sidebar_position: 74
---
# `stack zot dump-schema`
```
Usage: stack zot dump-schema [OPTIONS]
Dump itemType / field / creatorType IDs.
Regenerate after a Zotero upgrade: the output replaces TYPE_MAP,
FIELD_MAP, and CREATOR_TYPES in ``src/zot/db.py`` verbatim.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --db PATH Default: data/zotero/data/zotero.sqlite │
│ --python --json Emit paste-ready Python literals for zot/db.py │
│ (default) or JSON. │
│ [default: python] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

18
docs/docs/cli/zot-fix-dates.md Normal file
View File

@@ -0,0 +1,18 @@
---
title: stack zot fix-dates
sidebar_position: 75
---
# `stack zot fix-dates`
```
Usage: stack zot fix-dates [OPTIONS]
Normalize item timestamps to ISO 8601 + install auto-fix triggers.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --db PATH │
│ --hold --no-hold [default: hold] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

18
docs/docs/cli/zot-fix-fields.md Normal file
View File

@@ -0,0 +1,18 @@
---
title: stack zot fix-fields
sidebar_position: 77
---
# `stack zot fix-fields`
```
Usage: stack zot fix-fields [OPTIONS]
Remap base-field rows to type-specific fieldIDs; drop orphans.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --db PATH │
│ --hold --no-hold [default: hold] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

19
docs/docs/cli/zot-fix-keys.md Normal file
View File

@@ -0,0 +1,19 @@
---
title: stack zot fix-keys
sidebar_position: 76
---
# `stack zot fix-keys`
```
Usage: stack zot fix-keys [OPTIONS]
Replace invalid Zotero object keys (0/1/O/lowercase → valid).
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --db PATH │
│ --hold --no-hold [default: hold] │
│ --backup --no-backup [default: backup] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

21
docs/docs/cli/zot-verify-parity.md Normal file
View File

@@ -0,0 +1,21 @@
---
title: stack zot verify-parity
sidebar_position: 78
---
# `stack zot verify-parity`
```
Usage: stack zot verify-parity [OPTIONS]
Check that ``zot.db`` constants match the live Zotero schema.
The check runs implicitly on every ``Db(...)`` construction; this
command just exposes it explicitly so CI can fail loudly if someone
bumps Zotero without regenerating the maps.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --db PATH │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

25
docs/docs/cli/zot.md Normal file
View File

@@ -0,0 +1,25 @@
---
title: stack zot
sidebar_position: 73
---
# `stack zot`
```
Usage: stack zot [OPTIONS] COMMAND [ARGS]...
Zotero database maintenance.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ dump-schema Dump itemType / field / creatorType IDs. │
│ fix-dates Normalize item timestamps to ISO 8601 + install auto-fix │
│ triggers. │
│ fix-keys Replace invalid Zotero object keys (0/1/O/lowercase → valid). │
│ fix-fields Remap base-field rows to type-specific fieldIDs; drop │
│ orphans. │
│ verify-parity Check that ``zot.db`` constants match the live Zotero schema. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

336
notebooks/code_families.py
View File

@@ -565,40 +565,334 @@ def _(code, mo, not_built, pl, plain_years, q):
@app.cell(hide_code=True) @app.cell(hide_code=True)
def _(mo, not_built): def _(code, mo, not_built, pl, store):
# ── 5. Anchors in the corpus ── # ── 5. Anchors in the corpus ──
mo.md( import os as _os
from pfs.families import family_of as _family_of
_fam = _family_of(code)
_key = (_fam.key if _fam else code).upper()
_fam_codes = set(_fam.codes) if _fam else {code}
_intro = mo.md(
"## 5. Anchors in the corpus\n\n" "## 5. Anchors in the corpus\n\n"
"How many chunks in the RAG index — comments, guidance, Federal Register text — carry " "How many chunks in the RAG index — comments, guidance, Federal Register "
"this family's codes as metadata, and how many bibliography items carry a `family:` or " "text — carry this family's codes as metadata, and how many bibliography "
"`code:` tag. That count lives in pgvector, not the DuckDB replica this notebook reads.\n\n" "items carry a `family:` or `code:` tag (`stack bib code-tags`). Chunk "
+ not_built("stack llm restamp") f"counts live in pgvector (**{_key}**, family-membership match), not the "
"DuckDB replica this notebook otherwise reads; item tags live in the "
"bibliography SQLite."
) )
_panels = [_intro]
if not _os.environ.get("LLM_DB_PASSWORD"):
_panels.append(
mo.md(
"_`LLM_DB_PASSWORD` not set — pgvector is unreachable from this "
"process, so chunk-per-collection counts are skipped; item-tag "
"counts from the bibliography follow instead._"
)
)
else:
try:
from sqlalchemy import text as _sa_text
from llm.config import load as _load_llm_cfg
from llm.index import _engine as _llm_engine
_eng = _llm_engine(_load_llm_cfg())
with _eng.begin() as _conn:
_chunk_rows = _conn.execute(
_sa_text(
"SELECT c.name, count(*) AS n "
"FROM langchain_pg_embedding e "
"JOIN langchain_pg_collection c ON c.uuid = e.collection_id "
"WHERE string_to_array(COALESCE(e.cmetadata->>'families', ''), "
"' ') && ARRAY[:key] "
"GROUP BY c.name ORDER BY c.name"
),
{"key": _key},
).fetchall()
except Exception as e: # noqa: BLE001 — degrade, never crash the page
_panels.append(mo.md(f"_pgvector unavailable: {e}_"))
else:
_chunks = pl.DataFrame(
{
"collection": [r[0] for r in _chunk_rows],
"chunks": [r[1] for r in _chunk_rows],
}
)
_panels.append(
mo.ui.table(_chunks, label=f"pgvector chunks tagged families ∋ {_key}")
if not _chunks.is_empty()
else mo.md(f"_No pgvector chunks tagged `families ∋ {_key}` yet._")
)
if store is None:
_panels.append(mo.md("_bibliography unavailable — item-tag counts omitted._"))
else:
try:
_fam_tags = pl.DataFrame(store.list_tags(namespace="family"))
_code_tags = pl.DataFrame(
[
row
for row in store.list_tags(namespace="code")
if row["name"].removeprefix("code:") in _fam_codes
]
)
except Exception as e: # noqa: BLE001
_panels.append(mo.md(f"_item-tag counts unavailable: {e}_"))
else:
_panels.append(
mo.ui.table(_fam_tags, label="bib item tags — family:*")
if not _fam_tags.is_empty()
else mo.md(not_built("stack bib code-tags"))
)
_panels.append(
mo.ui.table(_code_tags, label=f"bib item tags — code:* for {_key}")
if not _code_tags.is_empty()
else mo.md(f"_No `code:` tags for {_key}'s member codes yet._")
)
mo.vstack(_panels)
return return
@app.cell(hide_code=True) @app.cell(hide_code=True)
def _(mo, not_built): def _(code, con, fr_md, mo, not_built, pl, store):
# ── 6. Guidance ── # ── 6. Guidance ──
mo.md( from bib.cfrlink import md_link as _md_link
"## 6. Guidance\n\n" from pfs.codetables import read_guidance as _read_guidance
"Sub-regulatory guidance — Medicare Learning Network articles, Internet-Only Manual " from pfs.families import family_of as _family_of
"sections, MACs' local coverage determinations — that cites a family's codes, with CFR "
"cross-references where the guidance implements a rule.\n\n" _fam = _family_of(code)
+ not_built("stack pfs guidance --family CCM --write") _key = _fam.key if _fam else code
)
_rows = []
if con is not None:
try:
_rows = _read_guidance(con, _key)
except Exception: # noqa: BLE001 — a missing table is "not built yet"
_rows = []
if not _rows:
_view = mo.md(
"## 6. Guidance\n\n"
"Sub-regulatory guidance — Medicare Learning Network articles, "
"Internet-Only Manual sections, MACs' local coverage determinations — "
"that cites a family's codes, with CFR cross-references where the "
"guidance implements a rule.\n\n"
+ not_built(f"stack pfs guidance --family {_key} --write")
)
else:
def _citation(r):
# CFR: a live eCFR link built straight from the stored locator.
if r.kind == "cfr":
try:
return _md_link(r.locator)
except ValueError:
return r.locator
# IOM: the manual chapter's own bib title when the citation
# resolved to a library item, else fall back to the locator.
if r.kind == "iom" and r.item_key and store is not None:
try:
title = store.get(r.item_key).title
if title:
return title
except KeyError:
pass
return r.locator # iom (unresolved) and mln both show the locator
def _provenance(r):
# CPT-manual guideline citations carry no FR paragraph
# (p_id_src=0, page_src=0 — module docstring); everything else
# anchors to the exact FR paragraph that named the code.
if r.p_id_src:
return fr_md(r.item_key_src, r.p_id_src)
return f"{r.item_key_src} (CPT manual)" if r.item_key_src else ""
_tbl = pl.DataFrame(
{
"code": [r.code for r in _rows],
"kind": [r.kind for r in _rows],
"citation": [_citation(r) for r in _rows],
"cited from": [_provenance(r) for r in _rows],
}
)
_n_cfr = sum(1 for r in _rows if r.kind == "cfr")
_n_iom = sum(1 for r in _rows if r.kind == "iom")
_n_mln = sum(1 for r in _rows if r.kind == "mln")
_view = mo.vstack(
[
mo.md(
"## 6. Guidance\n\n"
"Sub-regulatory guidance — Medicare Learning Network articles, "
"Internet-Only Manual sections, MACs' local coverage "
"determinations — that cites a family's codes, with CFR "
"cross-references where the guidance implements a rule. Every "
"row is a reference actually found in text that also names one "
"of the family's codes, resolved against the bibliography where "
"possible and always anchored to where it was cited. "
f"**{_key}**: {_n_cfr} CFR, {_n_iom} IOM, {_n_mln} MLN "
f"reference{'s' if len(_rows) != 1 else ''}."
),
mo.ui.table(
_tbl, label=f"pfs.code_guidance — {_key} ({len(_rows)} rows)"
),
]
)
_view
return return
@app.cell(hide_code=True) @app.cell(hide_code=True)
def _(mo, not_built): def _(alt, code, con, mo, not_built, pl):
# ── 7. Reaction ── # ── 7. Reaction ──
mo.md( from pfs.codetables import read_reaction as _read_reaction
"## 7. Reaction\n\n" from pfs.families import family_of as _family_of
"Public comment volume on a family's codes over time — how many comment letters mention "
"them, in which rule years, and (on a sample) whether commenters supported or opposed the " _fam = _family_of(code)
"proposal.\n\n" + not_built("stack pfs reaction --family CCM --write") _key = _fam.key if _fam else code
)
_rows = []
if con is not None:
try:
_rows = _read_reaction(con, _key)
except Exception: # noqa: BLE001 — a missing table is "not built yet"
_rows = []
if not _rows:
_view = mo.md(
"## 7. Reaction\n\n"
"Public comment volume on a family's codes over time — how many "
"comment letters mention them, in which rule years, and (on a sample) "
"whether commenters supported or opposed the proposal.\n\n"
+ not_built(f"stack pfs reaction --family {_key} --write")
)
else:
_dockets = sorted(
(r for r in _rows if r.period_kind == "docket"),
key=lambda r: (r.year, r.period),
)
_fr_rows = sorted(
(r for r in _rows if r.period_kind == "fr-pairs"), key=lambda r: r.year
)
_has_stance = any(
r.stance_support or r.stance_oppose or r.stance_modify or r.stance_unclear
for r in _dockets
)
_panels = [
mo.md(
"## 7. Reaction\n\n"
"Two independent proxies for public reaction, both counted from "
"chunk/paragraph metadata rather than hand-read. **Dockets** "
"(regulations.gov, 2017 on) count the distinct commenters "
"(`item_key`) whose comment text names one of the family's codes "
"— `n_items` out of the docket's `n_total` distinct commenters, "
"read straight from the `comments` pgvector collection; a stance "
"breakdown (support/oppose/modify/unclear) is filled in only for "
"dockets built with `--stance-sample N`, which classifies the "
"newest *N* matching commenters' first chunk with the self-hosted "
"closed-vocabulary classifier — an answer outside the four "
"stances counts as `unclear` — so it is a sample, not a census. "
"**FR pairs** (back to 2001) count `Comment:`/`Response:` "
"paragraph pairs CMS itself printed in a rule's preamble that "
"name a family code — the pre-2017 proxy, since regulations.gov "
"comment text isn't indexed that far back; `n_items` is the "
"qualifying-pair count naming the family and `n_total` the "
"rule's total `Comment:` paragraph count (family or not) — the "
"pool `n_items` is drawn out of."
+ (
""
if _has_stance
else " No docket in this family was built with "
"`--stance-sample`, so the stance columns below are all zero."
)
)
]
if _dockets:
_dk = pl.DataFrame(
{
"label": [f"{r.year} {r.period}" for r in _dockets],
"n_items": [r.n_items for r in _dockets],
"n_total": [r.n_total for r in _dockets],
"support": [r.stance_support for r in _dockets],
"oppose": [r.stance_oppose for r in _dockets],
"modify": [r.stance_modify for r in _dockets],
"unclear": [r.stance_unclear for r in _dockets],
}
)
_chart1 = (
alt.Chart(_dk.to_pandas())
.mark_bar()
.encode(
x=alt.X(
"label:N", sort=_dk["label"].to_list(), title="Docket (year)"
),
y=alt.Y("n_items:Q", title="Commenters naming the family"),
tooltip=[
"label",
"n_items",
"n_total",
"support",
"oppose",
"modify",
"unclear",
],
)
.properties(height=240, width=640)
)
_panels.append(mo.ui.altair_chart(_chart1))
else:
_panels.append(mo.md("_No docket has comment chunks for this family._"))
if _fr_rows:
_fp = pl.DataFrame(
{
"year": [r.year for r in _fr_rows],
"item_key": [r.period for r in _fr_rows],
"n_items": [r.n_items for r in _fr_rows],
}
)
_chart2 = (
alt.Chart(_fp.to_pandas())
.mark_bar()
.encode(
x=alt.X("year:O", title="Rule year"),
y=alt.Y("n_items:Q", title="Comment/Response pairs"),
tooltip=["year", "item_key", "n_items"],
)
.properties(height=220, width=640)
)
_panels.append(mo.ui.altair_chart(_chart2))
else:
_panels.append(
mo.md("_No pre-2017 Comment:/Response: pairs found for this family._")
)
_tbl = pl.DataFrame(
{
"period_kind": [r.period_kind for r in _rows],
"period": [r.period for r in _rows],
"year": [r.year for r in _rows],
"n_items": [r.n_items for r in _rows],
"n_total": [r.n_total for r in _rows],
"support": [r.stance_support for r in _rows],
"oppose": [r.stance_oppose for r in _rows],
"modify": [r.stance_modify for r in _rows],
"unclear": [r.stance_unclear for r in _rows],
}
)
_panels.append(
mo.ui.table(_tbl, label=f"pfs.code_reaction — {_key} ({len(_rows)} rows)")
)
_view = mo.vstack(_panels)
_view
return return

174
src/bib/codetags.py Normal file
View File

@@ -0,0 +1,174 @@
"""``code:``/``family:`` item tags — ``stack bib code-tags``.
Chunk-level anchors (``pfs.anchors.anchor_metadata``, stamped by
``llm.index``/``llm.restamp`` into pgvector's ``cmetadata``) already say
which HCPCS/CPT codes and code families a piece of text names. This
module rolls those per-chunk anchors up to the bib *item* they belong
to and stamps ``code:<CODE>`` / ``family:<KEY>`` tags on the item —
so ``stack bib query``/``stack bib tag`` and any other item-level tool
can filter "everything about G0556" without touching pgvector.
Two sources feed the same shape (``item_key -> (codes, families)``):
* :func:`item_codes_from_chunks` — one ``GROUP BY`` round trip per
pgvector collection (comments, rules, corpus).
* :func:`item_codes_from_anchors` — rules only, straight from
``fr_anchors.text`` via ``pfs.anchors.anchor_metadata``, no pgvector
needed (useful before/without an index run, or with
``--anchors-only``).
:func:`apply_code_tags` stamps the tags. It is additive and idempotent
— ``Store.add_tags`` only writes labels that are new, one transaction
per item — and caps the per-item ``code:`` cloud at
``max_codes_per_item`` distinct codes: an RVU file or code-list item
can name thousands of codes, and a tag cloud that size helps nobody.
Wide items still get every ``family:`` tag, plus ``code:`` for codes in
a hand-registered family (``pfs.families.HAND_FAMILIES`` —
ACP/CCM/PCM/TCM/APCM) since those are the ones anyone actually filters
by.
The cap is evaluated against an item's *merged* code set — a caller
that sources ``mapping`` from more than one collection (e.g. an item
whose chunks live in both ``rules`` and ``corpus``) must union all
sources into one ``mapping`` before calling :func:`apply_code_tags`;
calling it once per source would cap each source's codes independently
and let a straddling item slip under the cap in each call while still
being wide overall (Ruling A7).
"""
from __future__ import annotations
from typing import Any, Mapping
from sqlalchemy import text
from bib.tag import Tag
from pfs.anchors import anchor_metadata
from pfs.families import HAND_FAMILIES, Family
CodeMap = dict[str, tuple[set[str], set[str]]]
#: Codes belonging to a hand-registered family — these survive the
#: ``max_codes_per_item`` cap even on a wide item (Ruling: brief Task 4).
HAND_FAMILY_CODES: frozenset[str] = frozenset(
code for fam in HAND_FAMILIES.values() for code in fam.codes
)
SELECT_CHUNK_CODES_SQL = (
"SELECT e.cmetadata->>'item_key', "
"string_agg(e.cmetadata->>'codes', ' '), "
"string_agg(e.cmetadata->>'families', ' ') "
"FROM langchain_pg_embedding e "
"WHERE e.collection_id = (SELECT uuid FROM langchain_pg_collection WHERE name = :collection) "
"AND coalesce(e.cmetadata->>'codes', '') <> '' "
"GROUP BY 1"
)
def item_codes_from_chunks(engine: Any, *, collection: str) -> CodeMap:
"""``item_key -> (codes, families)`` for one pgvector collection, via
a single ``GROUP BY e.cmetadata->>'item_key'`` round trip."""
with engine.begin() as conn:
rows = conn.execute(
text(SELECT_CHUNK_CODES_SQL), {"collection": collection}
).fetchall()
out: CodeMap = {}
for item_key, codes_agg, families_agg in rows:
if not item_key:
continue
codes = {c for c in (codes_agg or "").split() if c}
families = {f for f in (families_agg or "").split() if f}
if not codes and not families:
continue
out[item_key] = (codes, families)
return out
def item_codes_from_anchors(
store: Any,
*,
families: Mapping[str, Family] | None = None,
code_index: Mapping[str, tuple[str, ...]] | None = None,
) -> CodeMap:
"""``item_key -> (codes, families)`` for rules, computed straight from
``fr_anchors.text`` via ``anchor_metadata`` — no pgvector needed."""
con = store._con() # noqa: SLF001
rows = con.execute(
"SELECT item_key, text FROM fr_anchors ORDER BY item_key, p_id"
).fetchall()
out: CodeMap = {}
for row in rows:
item_key = row["item_key"]
md = anchor_metadata(
row["text"] or "", families=families, code_index=code_index
)
codes = {c for c in md["codes"].split() if c}
fams = {f for f in md["families"].split() if f}
if not codes and not fams:
continue
cur_codes, cur_fams = out.setdefault(item_key, (set(), set()))
cur_codes.update(codes)
cur_fams.update(fams)
return out
def apply_code_tags(
store: Any,
mapping: CodeMap,
*,
dry_run: bool = False,
max_codes_per_item: int = 200,
) -> dict[str, int]:
"""Stamp ``code:``/``family:`` tags for every item in *mapping*.
Additive and idempotent: an item's current tags are read once and
only labels not already present count toward ``tags_added``. Under
a real run all new labels for an item are written in a single
``Store.add_tags`` call — one transaction per item, not one per
label — so a rerun over an unchanged mapping adds zero tags without
the per-tag commit cost that made the first backfill take 34
minutes. Items with more than ``max_codes_per_item`` distinct codes
get ``family:`` tags plus ``code:`` only for hand-family codes
(:data:`HAND_FAMILY_CODES`); such items count toward
``skipped_wide``. Items absent from *store* are skipped silently —
a caller can build *mapping* from a wider scope (e.g. pgvector) than
what's currently in bib.
``mapping`` must already be merged across every source an item can
appear in (see the module docstring, Ruling A7) — this function
applies the cap to whatever code set it is handed, per item, once.
Returns ``{"items": n, "tags_added": n, "skipped_wide": n}``.
"""
items = 0
tags_added = 0
skipped_wide = 0
for item_key, (codes, families) in mapping.items():
try:
current = set(store.get(item_key).tags)
except KeyError:
continue
items += 1
wide = len(codes) > max_codes_per_item
if wide:
skipped_wide += 1
tag_codes = codes & HAND_FAMILY_CODES
else:
tag_codes = codes
labels = [Tag.code(c).label for c in tag_codes] + [
Tag.family(f).label for f in families
]
new_labels = [label for label in labels if label not in current]
if not new_labels:
continue
if dry_run:
tags_added += len(new_labels)
else:
tags_added += store.add_tags(item_key, new_labels)
return {"items": items, "tags_added": tags_added, "skipped_wide": skipped_wide}

33
src/bib/store.py
View File

@@ -24,7 +24,7 @@ import shutil
import sqlite3 import sqlite3
from importlib import resources from importlib import resources
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any, Iterable
from bib.dockets import Docket from bib.dockets import Docket
from bib.item import Item from bib.item import Item
@@ -560,6 +560,37 @@ class Store:
self._stamp(item_key) self._stamp(item_key)
con.commit() con.commit()
def add_tags(self, item_key: str, tags: Iterable[str]) -> int:
"""Add multiple tags to an item in one transaction.
Same ``INSERT OR IGNORE`` semantics as :meth:`add_tag` — an
already-present tag is a no-op — but batched into a single
commit instead of one per tag. A backfill stamping ~486k
``code:``/``family:`` labels one-at-a-time via ``add_tag`` took
34 minutes, almost entirely commit/fsync overhead
(``bib.codetags.apply_code_tags``).
Returns the number of tags newly added (0 if every tag in
*tags* was already on the item).
"""
con = self._con()
row = con.execute("SELECT id FROM items WHERE key = ?", (item_key,)).fetchone()
if row is None:
raise KeyError(f"Item not found: {item_key}")
item_id = row["id"]
added = 0
for tag in tags:
tag_id = self._ensure_tag(tag)
cur = con.execute(
"INSERT OR IGNORE INTO item_tags (item_id, tag_id) VALUES (?, ?)",
(item_id, tag_id),
)
added += cur.rowcount
if added: # re-tagging with already-present tags only changes nothing
self._stamp(item_key)
con.commit()
return added
def remove_tag(self, item_key: str, tag: str) -> None: def remove_tag(self, item_key: str, tag: str) -> None:
"""Remove a tag from an item.""" """Remove a tag from an item."""
con = self._con() con = self._con()

44
src/bib/tag.py
View File

@@ -49,6 +49,17 @@ Namespaces
``fn:cclf.stg_beneficiary_xref``, ``fn:cclf.stg_beneficiary_xref``,
``fn:readmissions.int_encounter`` ``fn:readmissions.int_encounter``
``code``
An HCPCS/CPT code the item names (upper-case), rolled up from
chunk-level anchors (``pfs.anchors.anchor_metadata``) or FR
paragraph text — see ``bib.codetags``:
``code:99490``, ``code:G0556``
``family``
A ``pfs.code_family``/``pfs.families`` family key the item touches
— the key exactly as registered, never lower-cased:
``family:CCM``, ``family:APCM``
Usage:: Usage::
from bib.tag import Tag from bib.tag import Tag
@@ -238,6 +249,39 @@ class Tag(BaseModel):
value = f"{item_key}/{locator}" if locator else item_key value = f"{item_key}/{locator}" if locator else item_key
return cls(namespace="pin", value=value) return cls(namespace="pin", value=value)
@classmethod
def code(cls, hcpcs: str) -> Tag:
"""Tag for an HCPCS/CPT code the item names.
Parameters
----------
hcpcs : str
A code, e.g. ``99490``, ``G0556``. Upper-cased.
Examples::
Tag.code("g0556") # code:G0556
"""
return cls(namespace="code", value=hcpcs.upper())
@classmethod
def family(cls, key: str) -> Tag:
"""Tag for a ``pfs.families``/``pfs.code_family`` family key.
Parameters
----------
key : str
The family key exactly as registered (e.g. ``CCM``,
``APCM``, or a derived ``pfs.code_family`` key) — never
lower-cased, since derived keys aren't necessarily
uppercase-safe round trips.
Examples::
Tag.family("CCM") # family:CCM
"""
return cls(namespace="family", value=key)
@classmethod @classmethod
def fn(cls, qualified_name: str) -> Tag: def fn(cls, qualified_name: str) -> Tag:
"""Tag linking a bibliography item to an express function. """Tag linking a bibliography item to an express function.

118
src/cli/bib.py
View File

@@ -912,3 +912,121 @@ def cfr_jump(
typer.echo(cfrlink.md_link(ref)) typer.echo(cfrlink.md_link(ref))
else: else:
typer.echo(cfrlink.url(cfrlink.parse_cite(ref))) typer.echo(cfrlink.url(cfrlink.parse_cite(ref)))
@app.command(name="code-tags")
def code_tags(
collection: str = typer.Option(
"all", "--collection", help="Which collection: all | comments | rules | corpus."
),
anchors_only: bool = typer.Option(
False,
"--anchors-only",
help="Rules only, straight from fr_anchors text — skip pgvector entirely.",
),
dry_run: bool = typer.Option(
False,
"--dry-run",
help="Compute the merged item→codes mapping and report what would "
"be tagged; write nothing.",
),
max_codes_per_item: int = typer.Option(
200,
"--max-codes-per-item",
help="Items with more distinct codes get family: tags only (+ hand-family codes).",
),
) -> None:
"""Stamp code:/family: item tags from chunk anchors and FR paragraphs.
Rolls up ``pfs.anchors.anchor_metadata`` — already stamped on
pgvector chunks by ``stack llm index``/``stack llm restamp`` — to
the bib item level: every code an item's chunks name becomes a
``code:<CODE>`` tag, every family a ``family:<KEY>`` tag. Rules also
get a pgvector-free source (``fr_anchors.text``, the FR paragraph
anchor map from ``stack bib fr-grab``); pass ``--anchors-only`` to
use only that and skip pgvector for rules (and skip comments/corpus
entirely — they have no other source). Additive and idempotent
(``bib.codetags.apply_code_tags``) — safe to rerun after every
``stack llm restamp``. Every selected source is merged into one
``item_key -> (codes, families)`` mapping before tagging (Ruling
A7), so an item that straddles two collections (e.g. rules and
corpus) is capped by ``--max-codes-per-item`` once, on its combined
code set — the printed per-source lines are informational only;
``tags_added``/``skipped_wide`` in the total come from that one
merged pass.
"""
import os
from bib import connect
from bib.codetags import (
apply_code_tags,
item_codes_from_anchors,
item_codes_from_chunks,
)
from llm import config as llm_config
from llm.index import _engine
from pfs.anchors import code_family_index
from pfs.families import FAMILIES, refresh_from
collections = ("comments", "rules", "corpus")
if collection not in ("all", *collections):
raise typer.BadParameter("collection must be all, comments, rules or corpus")
targets = collections if collection == "all" else (collection,)
store = connect()
cfg = llm_config.load()
if os.path.exists(cfg.duckdb_replica):
import duckdb
con = duckdb.connect(cfg.duckdb_replica, read_only=True)
try:
refresh_from(con)
finally:
con.close()
code_index = code_family_index(FAMILIES)
engine = None
merged: dict[str, tuple[set[str], set[str]]] = {}
processed_any = False
def _merge(source: dict[str, tuple[set[str], set[str]]]) -> None:
for key, (codes, fams) in source.items():
cur_codes, cur_fams = merged.setdefault(key, (set(), set()))
merged[key] = (cur_codes | codes, cur_fams | fams)
for target in targets:
if anchors_only and target != "rules":
typer.echo(f"{target}: skipped (--anchors-only)")
continue
if target == "rules":
anchors_map = item_codes_from_anchors(store, code_index=code_index)
typer.echo(f"{target} (anchors): items={len(anchors_map)}")
_merge(anchors_map)
processed_any = True
if not anchors_only:
if engine is None:
engine = _engine(cfg)
chunk_map = item_codes_from_chunks(engine, collection=target)
typer.echo(f"{target} (chunks): items={len(chunk_map)}")
_merge(chunk_map)
else:
if engine is None:
engine = _engine(cfg)
chunk_map = item_codes_from_chunks(engine, collection=target)
typer.echo(f"{target} (chunks): items={len(chunk_map)}")
_merge(chunk_map)
processed_any = True
if processed_any:
stats = apply_code_tags(
store, merged, dry_run=dry_run, max_codes_per_item=max_codes_per_item
)
else:
stats = {"items": 0, "tags_added": 0, "skipped_wide": 0}
prefix = "would-" if dry_run else ""
typer.echo(
f"total: {prefix}items={stats['items']} {prefix}tags_added={stats['tags_added']} "
f"capped_wide={stats['skipped_wide']}"
)

222
src/cli/pfs.py
View File

@@ -5,6 +5,8 @@
uv run stack pfs elements --all-payable --no-llm --dry-run uv run stack pfs elements --all-payable --no-llm --dry-run
uv run stack pfs lineage --code G2058 [--write] uv run stack pfs lineage --code G2058 [--write]
uv run stack pfs families [--write] uv run stack pfs families [--write]
uv run stack pfs guidance --family CCM [--family …] [--write]
uv run stack pfs reaction --family CCM [--code G2211] [--stance-sample 30] [--write]
uv run stack pfs review [--code G0556] uv run stack pfs review [--code G0556]
uv run stack pfs cpt-ingest --edition 2024 [--edition 2022 …] [--dry-run] uv run stack pfs cpt-ingest --edition 2024 [--edition 2022 …] [--dry-run]
uv run stack pfs cpt-ingest --all uv run stack pfs cpt-ingest --all
@@ -29,14 +31,19 @@ from pfs.codetables import (
read_cpt_codes, read_cpt_codes,
read_cpt_instructions, read_cpt_instructions,
read_cpt_sections, read_cpt_sections,
read_guidance,
write_elements, write_elements,
write_events, write_events,
write_families, write_families,
write_guidance,
write_reaction,
) )
from pfs.cpt_load import ingest as cpt_ingest from pfs.cpt_load import ingest as cpt_ingest
from pfs.extract import extract_code from pfs.extract import extract_code
from pfs.families import FAMILIES, HAND_FAMILIES, derive_families, refresh_from from pfs.families import FAMILIES, HAND_FAMILIES, derive_families, refresh_from
from pfs.guidance import build as build_guidance
from pfs.lineage import lineage from pfs.lineage import lineage
from pfs.reaction import series as reaction_series
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
@@ -88,6 +95,17 @@ def _classifier() -> Any:
return closed_vocab_classifier(cfg, HostPool.from_config(cfg)) return closed_vocab_classifier(cfg, HostPool.from_config(cfg))
def _engine() -> Any:
# llm.index/llm.evidence expose no public "give me a bare SQLAlchemy
# engine over the pgvector DB" helper — cli/bib.py and cli/llm.py
# both already reach into llm.index._engine for the same reason, so
# this is the established reuse, not a new private import.
from llm import config as llm_config
from llm.index import _engine as pgvector_engine
return pgvector_engine(llm_config.load())
def _run_elements( def _run_elements(
con: Any, store: Any, targets: list[str], classify: Any, *, write: bool con: Any, store: Any, targets: list[str], classify: Any, *, write: bool
) -> None: ) -> None:
@@ -343,6 +361,210 @@ def families(
con.close() con.close()
def _print_guidance(rows: list[Any]) -> None:
for r in sorted(rows, key=lambda r: (r.family, r.code, r.kind, r.locator)):
item = r.item_key or "—"
typer.echo(
f"{r.family} {r.kind} {r.locator} {item} ← {r.item_key_src}{r.p_id_src}"
)
def _resolve_guidance_families(con: Any, keys: list[str]) -> list[tuple[str, Any]]:
"""(key, Family) per requested key, refreshed from *con* — raises
``typer.BadParameter`` on the first unknown key, before any family's
rows are built or any write happens (mirrors ``_reaction_targets``'s
resolve-everything-up-front discipline)."""
refresh_from(con)
out: list[tuple[str, Any]] = []
for key in keys:
fam = FAMILIES.get(key)
if fam is None:
raise typer.BadParameter(
f"unknown family {key!r}; known: {', '.join(HAND_FAMILIES)}"
)
out.append((key, fam))
return out
@app.command()
def guidance(
family: list[str] = typer.Option(
[], "--family", help="Family key (CCM, APCM, …); repeatable."
),
write: bool = typer.Option(False, "--write"),
) -> None:
"""CFR/IOM/MLN sub-regulatory crosswalk per code family
(pfs.code_guidance): which CFR sections and IOM manual chapters the
family's FR paragraphs and the CPT manual cite, with FR provenance."""
if not family:
raise typer.BadParameter("pass --family (repeatable)")
keys = [f.upper() for f in family]
store = _store()
if write:
# Ruling A13 (mirrors `stack pfs reaction`): resolve every family
# key up front and build every family's rows against a plain
# DuckDB read (I4) — build_guidance only reads pfs.cpt_* — never
# the write lock a notebook may be holding (#508-#514). The batch
# below only ever calls write_guidance and publishes.
con = _read()
try:
targets = _resolve_guidance_families(con, keys)
by_family: dict[str, list[Any]] = {}
rows: list[Any] = []
for key, fam in targets:
fam_rows = build_guidance(con, store, fam.codes, family=key)
by_family[key] = fam_rows
rows.extend(fam_rows)
finally:
con.close()
with _batch() as con:
ensure_tables(con)
for key, fam_rows in by_family.items():
write_guidance(con, key, fam_rows)
_publish()
_print_guidance(rows)
return
# No --write: read-only never needs the DuckDB single-writer lock a
# notebook may be holding (#508-#514) — read the replica (I4).
con = _read()
try:
refresh_from(con)
rows = []
for key in keys:
if key not in FAMILIES:
raise typer.BadParameter(
f"unknown family {key!r}; known: {', '.join(HAND_FAMILIES)}"
)
try:
rows.extend(read_guidance(con, key))
except Exception as exc:
if _missing_table(exc):
typer.echo(f"{key}: no derived guidance yet — run with --write")
continue
raise
_print_guidance(rows)
finally:
con.close()
def _print_reaction(rows: list[Any]) -> None:
for r in sorted(rows, key=lambda r: (r.family, r.period_kind, r.year, r.period)):
if r.period_kind == "docket":
stance = ""
if (
r.stance_support
or r.stance_oppose
or r.stance_modify
or r.stance_unclear
):
stance = (
f" stance support={r.stance_support} oppose={r.stance_oppose} "
f"modify={r.stance_modify} unclear={r.stance_unclear}"
)
typer.echo(
f"{r.family} {r.period} ({r.year}) {r.n_items}/{r.n_total} commenters{stance}"
)
else:
typer.echo(
f"{r.family} {r.period} CY{r.year} "
f"{r.n_items} comment/response pairs"
)
def _reaction_targets(
con: Any, family: list[str], code: list[str]
) -> list[tuple[str, tuple[str, ...], str | None]]:
"""(key, codes, display name) per target — *name* is the registered
family's own name for a ``--family`` target, ``None`` for a
``--code`` pseudo-family (``pfs.reaction.series`` falls back to the
key itself, which is already the right label for a bare code)."""
refresh_from(con)
targets: list[tuple[str, tuple[str, ...], str | None]] = []
for f in family:
key = f.upper()
fam = FAMILIES.get(key)
if fam is None:
raise typer.BadParameter(
f"unknown family {key!r}; known: {', '.join(HAND_FAMILIES)}"
)
targets.append((key, fam.codes, fam.name))
for c in code:
key = c.upper()
targets.append((key, (key,), None))
return targets
@app.command()
def reaction(
family: list[str] = typer.Option(
[], "--family", help="Family key (CCM, APCM, …); repeatable."
),
code: list[str] = typer.Option(
[],
"--code",
help="A single code as its own pseudo-family, keyed by the code; repeatable.",
),
stance_sample: int = typer.Option(
0,
"--stance-sample",
help="Classify the newest N commenters per docket into support/oppose/"
"modify/unclear via the self-hosted classifier (0 = skip; no stance "
"sample means no LLM call at all).",
),
write: bool = typer.Option(False, "--write"),
) -> None:
"""Public-reaction series per code family (pfs.code_reaction): comment
counts per docket (share of the docket's commenters mentioning the
family), an optional stance sample, and the FR Comment:/Response:
pair counts per rule as the pre-2017 proxy. Always recomputed live
from pgvector/bib — there is no persisted-table fallback to read
without --write, so --write only adds persisting the result."""
if not family and not code:
raise typer.BadParameter("pass --family or --code (repeatable)")
store = _store()
# Resolving family keys is a plain DuckDB read (I4) — never the
# write lock a notebook may be holding (#508-#514) — and it's the
# only DuckDB touch this command needs before writing.
con = _read()
try:
targets = _reaction_targets(con, family, code)
finally:
con.close()
# Ruling A13: the classifier and every family's series (docket
# counts, the stance sample — ~1-2s/sample — and FR pairs) all run
# against pgvector/bib, not DuckDB, so they run BEFORE any DuckDB
# batch is opened. The batch below only ever calls write_reaction
# and publishes — it never holds the single-writer lock (#508-#514)
# while the slow work happens.
classify = _classifier() if stance_sample > 0 else None
engine = _engine()
by_family: dict[str, list[Any]] = {}
rows: list[Any] = []
for key, codes, name in targets:
fam_rows = reaction_series(
engine,
store,
key,
codes,
classify=classify,
stance_sample_n=stance_sample,
name=name,
)
by_family[key] = fam_rows
rows.extend(fam_rows)
if write:
with _batch() as con:
ensure_tables(con)
for key, fam_rows in by_family.items():
write_reaction(con, key, fam_rows)
_publish()
_print_reaction(rows)
@app.command() @app.command()
def review(code: str = typer.Option("", "--code")) -> None: def review(code: str = typer.Option("", "--code")) -> None:
"""Element lines the classifier could not place.""" """Element lines the classifier could not place."""

15
src/llm/config.py
View File

@@ -47,6 +47,8 @@ class LlmConfig:
duckdb_replica: str = "data/replica/aco.ro.duckdb" duckdb_replica: str = "data/replica/aco.ro.duckdb"
valuation_years: int = 4 valuation_years: int = 4
code_cited_per_code: int = 3 code_cited_per_code: int = 3
code_cited_collections: tuple[str, ...] = ("rules", "comments", "corpus")
code_cited_max: int = 12
def parse_hosts(spec: str) -> tuple[tuple[str, float], ...]: def parse_hosts(spec: str) -> tuple[tuple[str, float], ...]:
@@ -71,6 +73,15 @@ def _opt(section: Any, key: str, default: Any) -> Any:
return section[key] if key in section else default return section[key] if key in section else default
def _str_tuple(val: Any) -> tuple[str, ...]:
"""*val* as a tuple of strings — a bare scalar string in stack.toml
(``code_cited_collections = "rules"``) becomes a one-element tuple
rather than ``tuple("rules")`` exploding it into its characters."""
if isinstance(val, str):
return (val,)
return tuple(val)
def load() -> LlmConfig: def load() -> LlmConfig:
"""Read the [llm] section; env vars override host-ish values.""" """Read the [llm] section; env vars override host-ish values."""
from conf import cfg from conf import cfg
@@ -105,6 +116,10 @@ def load() -> LlmConfig:
or str(_opt(section, "duckdb_replica", "data/replica/aco.ro.duckdb")), or str(_opt(section, "duckdb_replica", "data/replica/aco.ro.duckdb")),
valuation_years=int(_opt(section, "valuation_years", 4)), valuation_years=int(_opt(section, "valuation_years", 4)),
code_cited_per_code=int(_opt(section, "code_cited_per_code", 3)), code_cited_per_code=int(_opt(section, "code_cited_per_code", 3)),
code_cited_collections=_str_tuple(
_opt(section, "code_cited_collections", ["rules", "comments", "corpus"])
),
code_cited_max=int(_opt(section, "code_cited_max", 12)),
embed_dim=int(section.embed_dim), embed_dim=int(section.embed_dim),
build_ann_index=build_ann_index, build_ann_index=build_ann_index,
pg_host=os.environ.get("LLM_PG_HOST", str(section.pg_host)), pg_host=os.environ.get("LLM_PG_HOST", str(section.pg_host)),

215
src/llm/evidence.py
View File

@@ -3,9 +3,12 @@
``valuation_evidence`` detects HCPCS/CPT codes (and registered families) ``valuation_evidence`` detects HCPCS/CPT codes (and registered families)
in the question, reads their RVUs from the DuckDB read replica and in the question, reads their RVUs from the DuckDB read replica and
attaches conversion factors from ``pfs.rules``. ``code_cited_sources`` attaches conversion factors from ``pfs.rules``. ``code_cited_sources``
pulls rule chunks whose ``codes`` metadata mention those codes straight pulls chunks — from rules, comments and the reference corpus — whose
from pgvector so the answer can cite where CMS valued them. Both degrade ``codes`` metadata mention those codes, and (when a family was detected)
to "no evidence" with a warning — the chat never fails because of them. chunks whose ``families`` metadata mentions the family key, straight
from pgvector so the answer can cite where CMS valued them and where the
library discusses them. Both degrade to "no evidence" with a warning —
the chat never fails because of them.
""" """
from __future__ import annotations from __future__ import annotations
@@ -181,46 +184,182 @@ _CITED_SQL = text(
"ORDER BY cmetadata->>'date' DESC NULLS LAST, cmetadata->>'item_key', cmetadata->>'p_id'" "ORDER BY cmetadata->>'date' DESC NULLS LAST, cmetadata->>'item_key', cmetadata->>'p_id'"
) )
#: Matches on the chunk's ``families`` metadata via array overlap — this is
#: how a CCM question pulls the CPT manual's "Chronic Care Management
#: Services" guideline chunk even though that chunk never spells out 99490.
#: Two-stage windowing so the result is breadth across ITEMS, not several
#: chunks of whichever single document is newest: stage one keeps one chunk
#: per (family, item_key) — the earliest chunk in the item, by paragraph id
#: for rules or sequence number for comments/corpus; stage two then ranks
#: the surviving (one-per-item) rows per family by date and keeps the
#: newest :window.
_FAMILY_CITED_SQL = text(
"SELECT document, cmetadata FROM ( "
"SELECT document, cmetadata, family, item_key, ordkey, "
"row_number() OVER ( "
"PARTITION BY family "
"ORDER BY cmetadata->>'date' DESC NULLS LAST, item_key, ordkey "
") AS rn "
"FROM ( "
"SELECT e.document, e.cmetadata, w.family AS family, "
"e.cmetadata->>'item_key' AS item_key, "
"COALESCE(NULLIF(e.cmetadata->>'p_id', '')::int, "
"NULLIF(e.cmetadata->>'seq', '')::int) AS ordkey, "
"row_number() OVER ( "
"PARTITION BY w.family, e.cmetadata->>'item_key' "
"ORDER BY COALESCE(NULLIF(e.cmetadata->>'p_id', '')::int, "
"NULLIF(e.cmetadata->>'seq', '')::int) ASC NULLS LAST "
") AS item_rn "
"FROM langchain_pg_embedding e "
"JOIN unnest(CAST(:families AS text[])) AS w(family) "
"ON string_to_array(COALESCE(e.cmetadata->>'families', ''), ' ') && ARRAY[w.family] "
"WHERE e.collection_id = (SELECT uuid FROM langchain_pg_collection "
"WHERE name = :collection) "
") one_per_item "
"WHERE item_rn = 1 "
") t "
"WHERE rn <= :window "
"ORDER BY cmetadata->>'date' DESC NULLS LAST, item_key, ordkey"
)
def _dedupe_key(md: dict[str, str]) -> tuple[str, str]:
"""(item_key, p_id) for rule chunks — several paragraphs share an
item_key — else (item_key, seq), since comment/corpus chunks carry a
sequence number instead of a paragraph id."""
second = md.get("p_id", "") if md.get("kind") == "rule" else md.get("seq", "")
return (md.get("item_key", ""), second)
def _collect(
engine: Any,
sql: Any,
key: str,
wanted: Sequence[str],
*,
per: int,
collections: Sequence[str],
seen: set[tuple[str, str]],
) -> dict[str, list[dict]]:
"""Run *sql* once per collection in *collections*, capping hits per
*key* value (a code or a family) per collection, deduping across the
whole call via the shared *seen* set. *key* names both the query
parameter (``:codes``/``:families``) and the chunk metadata field.
Returns each collection's own hit list, keyed by collection name (a
collection with no hits is simply absent), so a caller can
round-robin interleave across collections instead of only ever
getting them concatenated."""
upper = [w.upper() for w in wanted]
if not upper:
return {}
out: dict[str, list[dict]] = {}
for collection in collections:
try:
with engine.begin() as conn:
rows = conn.execute(
sql,
{"collection": collection, key: upper, "window": per * 2},
).fetchall()
except Exception as e: # noqa: BLE001
log.warning("code-cited sources skipped (%s/%s): %s", collection, key, e)
continue
count: dict[str, int] = {w: 0 for w in upper}
collected: list[dict] = []
for document, md in rows:
md = {k: str(v) for k, v in (md or {}).items()}
dkey = _dedupe_key(md)
if dkey in seen:
continue
hits = [w for w in md.get(key, "").split() if w in count and count[w] < per]
if not hits:
continue
seen.add(dkey)
for w in hits:
count[w] += 1
collected.append(as_source(md, document[:250], 0.0))
if collected:
out[collection] = collected
return out
def _interleave(
per_collection: dict[str, list[dict]], collections: Sequence[str]
) -> list[dict]:
"""Round-robin merge of *per_collection*'s lists in *collections*
order — one row from each collection in turn, looping until every
list is exhausted, so a collection with many hits can't crowd the
others out of the front of the list ahead of a ``max_total`` cut."""
out: list[dict] = []
idx = {c: 0 for c in collections}
progressed = True
while progressed:
progressed = False
for c in collections:
rows = per_collection.get(c, [])
i = idx[c]
if i < len(rows):
out.append(rows[i])
idx[c] = i + 1
progressed = True
return out
def code_cited_sources( def code_cited_sources(
engine: Any, codes: Sequence[str], *, per_code: int, collection: str = "rules" engine: Any,
codes: Sequence[str],
*,
per_code: int,
collections: Sequence[str] = ("rules",),
families: Sequence[str] = (),
max_total: int = 12,
) -> list[dict]: ) -> list[dict]:
"""Chunks in *collection* whose ``codes`` metadata mention any of """Chunks across *collections* whose ``codes`` metadata mention any of
*codes* — at most *per_code* per code, newest rule first, deduped on *codes* — at most *per_code* per code per collection, newest first,
(item_key, p_id). Scores are 0.0: these are additive, not ranked.""" deduped on (item_key, p_id) for rules or (item_key, seq) otherwise.
wanted = [c.upper() for c in codes]
if not wanted: When *families* is non-empty, chunks whose ``families`` metadata
mentions one of them are matched too (same per-collection cap) —
skipped entirely, no query, when *families* is empty. Scores are
0.0: these are additive, not ranked.
After the per-collection caps and dedupe above, the code-cited rows
are round-robin interleaved across *collections* in the order given
(one from the first collection, one from the second, … looping back
until every collection's list is exhausted) so no single collection
crowds the others out; the family-cited rows are interleaved the
same way and appended after. The combined list is then truncated to
*max_total* — ``<= 0`` means unlimited.
"""
if not codes and not families:
return [] return []
try:
with engine.begin() as conn:
rows = conn.execute(
_CITED_SQL,
{
"collection": collection,
"codes": wanted,
"window": per_code * 2,
},
).fetchall()
except Exception as e: # noqa: BLE001
log.warning("code-cited sources skipped: %s", e)
return []
out: list[dict] = []
seen: set[tuple[str, str]] = set() seen: set[tuple[str, str]] = set()
count: dict[str, int] = {c: 0 for c in wanted} out = _interleave(
for document, md in rows: _collect(
md = {k: str(v) for k, v in (md or {}).items()} engine,
key = (md.get("item_key", ""), md.get("p_id", "")) _CITED_SQL,
if key in seen: "codes",
continue codes,
hit_codes = [ per=per_code,
c for c in md.get("codes", "").split() if c in count and count[c] < per_code collections=collections,
] seen=seen,
if not hit_codes: ),
continue collections,
seen.add(key) )
for c in hit_codes: if families:
count[c] += 1 out += _interleave(
out.append(as_source(md, document[:250], 0.0)) _collect(
engine,
_FAMILY_CITED_SQL,
"families",
families,
per=per_code,
collections=collections,
seen=seen,
),
collections,
)
if max_total > 0:
out = out[:max_total]
return out return out

19
src/llm/rag.py
View File

@@ -9,9 +9,9 @@ source gets a deep link (``llm.links``). Generation streams from the
largest live Ollama host (``HostPool.acquire_generation``) with the largest live Ollama host (``HostPool.acquire_generation``) with the
model tier that host can hold (``pick_model``). When the question model tier that host can hold (``pick_model``). When the question
cites HCPCS/CPT codes, ``llm.evidence`` supplies authoritative cites HCPCS/CPT codes, ``llm.evidence`` supplies authoritative
valuation rows and rule chunks that cite those codes; the SSE event valuation rows and code- or family-cited chunks from rules, comments and
order is ``valuation`` (only when evidence is found) → ``token``* → the reference corpus; the SSE event order is ``valuation`` (only when
``sources`` → ``done``. evidence is found) → ``token``* → ``sources`` → ``done``.
""" """
from __future__ import annotations from __future__ import annotations
@@ -139,8 +139,10 @@ def stream_answer(
"""Retrieve, then stream a grounded answer from the largest live host. """Retrieve, then stream a grounded answer from the largest live host.
When the question cites HCPCS/CPT codes, yields a ``valuation`` event When the question cites HCPCS/CPT codes, yields a ``valuation`` event
first (evidence's ``payload()``) and merges rule chunks that cite first (evidence's ``payload()``) and merges chunks from rules,
those codes into ``sources``. Then yields ``{"type":"token","text":…}`` comments and the corpus that cite those codes — or, for a detected
family, its family key — into ``sources``. Then yields
``{"type":"token","text":…}``
events as the model generates, then one events as the model generates, then one
``{"type":"sources", "sources": […], "model": …, "host": …}`` and a ``{"type":"sources", "sources": […], "model": …, "host": …}`` and a
final ``{"type":"done"}``. final ``{"type":"done"}``.
@@ -149,7 +151,12 @@ def stream_answer(
evidence = valuation_evidence(question, cfg) evidence = valuation_evidence(question, cfg)
if evidence is not None: if evidence is not None:
cited = code_cited_sources( cited = code_cited_sources(
_engine(cfg), evidence.codes, per_code=cfg.code_cited_per_code _engine(cfg),
evidence.codes,
per_code=cfg.code_cited_per_code,
collections=tuple(cfg.code_cited_collections),
families=evidence.families,
max_total=cfg.code_cited_max,
) )
sources = merge_sources(sources, cited) sources = merge_sources(sources, cited)
yield evidence.payload() yield evidence.payload()

87
src/pfs/codetables.py
View File

@@ -94,6 +94,57 @@ class FamilyRow:
note: str = "" note: str = ""
@dataclass(frozen=True)
class GuidanceRow:
"""One CFR/IOM/MLN reference a family's codes are discussed under
(``pfs.guidance``, #689) — ``kind`` is ``"cfr"``, ``"iom"`` or
``"mln"``; ``locator`` is the canonical human-readable cite
("42 CFR 410.78(a)(3)" | "100-04 ch.12 §30.6.4" | "MLN 907166");
``item_key`` is the resolved bib item (``""`` when unresolved —
most CFR/IOM citations have no matching library item on hand);
``item_key_src``/``p_id_src``/``page_src`` are the citing
paragraph's own provenance (an FR paragraph, or the CPT edition
item with ``p_id_src=0``, ``page_src=0``)."""
family: str
code: str
kind: str # "cfr" | "iom" | "mln"
locator: str
item_key: str
item_key_src: str
p_id_src: int
page_src: int
@dataclass(frozen=True)
class ReactionRow:
"""One period's public-reaction counts for a code family
(``pfs.code_reaction``, #690) — ``period_kind`` is ``"docket"``
(``period`` is a regulations.gov docket id, ``year`` the docket's
``comment_end_date`` year) or ``"fr-pairs"`` (``period`` is the FR
rule ``item_key``, ``year`` its ``rule_year_of``). For a docket row,
``n_items``/``n_total`` are distinct commenters mentioning the
family / total distinct commenters in that docket. For an fr-pairs
row, ``n_items`` is the qualifying Comment:/Response: paragraph
pairs naming the family and ``n_total`` is the rule's total
``Comment:`` paragraph count — every one in the rule, family or not
— the pool ``n_items`` is drawn out of. ``stance_*`` and
``sample_json`` (``[{"item_key": …, "stance": …}]``) stay zero/
``"[]"`` unless a stance sample was classified for that docket."""
family: str
period: str
period_kind: str # "docket" | "fr-pairs"
year: int
n_items: int
n_total: int
stance_support: int
stance_oppose: int
stance_modify: int
stance_unclear: int
sample_json: str
@dataclass(frozen=True) @dataclass(frozen=True)
class CptSectionRow: class CptSectionRow:
edition_year: int edition_year: int
@@ -192,6 +243,13 @@ CREATE TABLE IF NOT EXISTS pfs.code_event (
CREATE TABLE IF NOT EXISTS pfs.code_family ( CREATE TABLE IF NOT EXISTS pfs.code_family (
key VARCHAR, name VARCHAR, code VARCHAR, role VARCHAR, since INTEGER, until INTEGER, key VARCHAR, name VARCHAR, code VARCHAR, role VARCHAR, since INTEGER, until INTEGER,
item_key VARCHAR, p_id INTEGER, note VARCHAR); item_key VARCHAR, p_id INTEGER, note VARCHAR);
CREATE TABLE IF NOT EXISTS pfs.code_guidance (
family VARCHAR, code VARCHAR, kind VARCHAR, locator VARCHAR, item_key VARCHAR,
item_key_src VARCHAR, p_id_src INTEGER, page_src INTEGER);
CREATE TABLE IF NOT EXISTS pfs.code_reaction (
family VARCHAR, period VARCHAR, period_kind VARCHAR, year INTEGER,
n_items INTEGER, n_total INTEGER, stance_support INTEGER, stance_oppose INTEGER,
stance_modify INTEGER, stance_unclear INTEGER, sample_json VARCHAR);
CREATE TABLE IF NOT EXISTS pfs.cpt_section ( CREATE TABLE IF NOT EXISTS pfs.cpt_section (
edition_year INTEGER, item_key VARCHAR, sec_id VARCHAR, level INTEGER, title VARCHAR, edition_year INTEGER, item_key VARCHAR, sec_id VARCHAR, level INTEGER, title VARCHAR,
path VARCHAR[], path_key VARCHAR, code_lo VARCHAR, code_hi VARCHAR, guideline VARCHAR); path VARCHAR[], path_key VARCHAR, code_lo VARCHAR, code_hi VARCHAR, guideline VARCHAR);
@@ -321,6 +379,35 @@ def read_families(con: Any) -> list[FamilyRow]:
return [FamilyRow(*r) for r in rows] return [FamilyRow(*r) for r in rows]
def write_guidance(con: Any, family: str, rows: Sequence[GuidanceRow]) -> int:
ensure_tables(con)
con.execute("DELETE FROM pfs.code_guidance WHERE family = ?", [family])
return _insert(con, "pfs.code_guidance", rows)
def read_guidance(con: Any, family: str) -> list[GuidanceRow]:
rows = con.execute(
"SELECT * FROM pfs.code_guidance WHERE family = ? ORDER BY code, kind, locator",
[family],
).fetchall()
return [GuidanceRow(*r) for r in rows]
def write_reaction(con: Any, family: str, rows: Sequence[ReactionRow]) -> int:
ensure_tables(con)
con.execute("DELETE FROM pfs.code_reaction WHERE family = ?", [family])
return _insert(con, "pfs.code_reaction", rows)
def read_reaction(con: Any, family: str) -> list[ReactionRow]:
rows = con.execute(
"SELECT * FROM pfs.code_reaction WHERE family = ? "
"ORDER BY period_kind, year, period",
[family],
).fetchall()
return [ReactionRow(*r) for r in rows]
_CPT_TABLES = ( _CPT_TABLES = (
"pfs.cpt_section", "pfs.cpt_section",
"pfs.cpt_code", "pfs.cpt_code",

19
src/pfs/families.py
View File

@@ -38,6 +38,25 @@ def find_codes(text: str) -> tuple[str, ...]:
return tuple(sorted({m.group(0).upper() for m in CODE_RE.finditer(text)})) return tuple(sorted({m.group(0).upper() for m in CODE_RE.finditer(text)}))
def code_pattern(codes: Sequence[str]) -> re.Pattern[str]:
"""A word-boundary alternation matching any of *codes* (upper-cased).
Shared by ``pfs.guidance`` and ``pfs.reaction`` — both scan FR
paragraph text for a specific family's codes, not every code
``CODE_RE`` would match."""
alts = "|".join(re.escape(c.upper()) for c in codes)
return re.compile(rf"\b(?:{alts})\b")
def codes_in(text: str, pattern: re.Pattern[str]) -> tuple[str, ...]:
"""Codes matching *pattern* literally present in *text*, with FR
citations ("91 FR 43842") stripped first — a citation's page number
reads as a CPT code and must never fire a false match (mirrors
``find_codes``)."""
stripped = _FR_CITE_RE.sub(" ", text)
return tuple(sorted({m.group(0) for m in pattern.finditer(stripped)}))
@dataclass(frozen=True) @dataclass(frozen=True)
class Family: class Family:
key: str key: str

337
src/pfs/guidance.py Normal file
View File

@@ -0,0 +1,337 @@
"""Sub-regulatory crosswalk: CFR sections and IOM manual chapters/sections
a code family's rules (and the CPT manual) cite, with FR paragraph
provenance (#689, task 6 of the 2026-09-09 code-family-anchors plan).
Two sources, one extraction pass over each hit's text:
1. ``harvest`` — FR paragraphs (``fr_anchors`` in the bib SQLite) that
name any of a family's codes. Cheap and needs no environment (the
bib sqlite has no ``.env`` dependency), unlike the pgvector rule
chunks the controller notes as an equally-valid but heavier
alternative source.
2. ``harvest_cpt`` — the CPT codebook's own guideline text
(``pfs.cpt_section.guideline``, the heading that owns the family's
codes) and ``see``/``other`` instructions (``pfs.cpt_instruction``)
that mention Medicare policy, from the newest ingested edition.
Both funnel through ``_extract``, which runs three regex families
(CFR, IOM, MLN) over one paragraph/guideline's text and resolves each
hit against the bib (``resolve_cfr``/``resolve_iom``) — unresolved
references keep an empty ``item_key`` (the canonical locator is kept
either way, Resolutions §fr_anchors). Rows are written by
``pfs.codetables.write_guidance``/``read_guidance`` into
``pfs.code_guidance``, deduped on ``(family, code, kind, locator,
item_key_src, p_id_src)`` — the same citation named in the same
paragraph for two codes in one family (a common shape: "99490 and
99439 ... see § 410.78") legitimately yields two rows, but a citation
matched twice by overlapping regex branches (e.g. both the titled and
bare CFR forms firing on the same span) collapses to one.
"""
from __future__ import annotations
import re
from typing import Any, Sequence
from bib.cfrlink import canonical, item_for, parse_cite
from pfs.codetables import GuidanceRow, is_missing_table_error
from pfs.families import code_pattern, codes_in
# manual display name (lower-cased, as it appears in FR prose) -> CMS
# publication number. Controller ruling (task-6-context.md): only these
# six IOM manuals are in scope for #689.
_MANUAL_PUB = {
"claims processing": "100-04",
"benefit policy": "100-02",
"program integrity": "100-08",
"national coverage determinations": "100-03",
"managed care": "100-16",
"financial management": "100-06",
}
# "42 CFR 410.78(a)(3)" / "42 C.F.R. § 425.400" / "42 CFR Part 425" /
# "42 CFR parts 405, 414, and 426" — title-bearing form. The paragraph
# suffix and part-only form are left to bib.cfrlink.parse_cite (fed
# "<title> CFR <section>") to interpret, so this regex just has to
# isolate title/partword/part/dec/paras. Named groups (Ruling A10): the
# "parts" keyword capture lets ``_cfr_refs`` tell a plural part-list
# head ("parts 405, ...") from a singular one-off part cite ("Part
# 425") — only the plural head triggers the bare-part-number
# continuation grammar below.
CFR_RE = re.compile(
r"\b(?P<title>\d{1,2})\s*C\.?F\.?R\.?\s*"
r"(?:(?P<partword>[Pp]arts?)\s*)?(?P<part>\d{2,4})"
r"(?:\.(?P<dec>\d+[a-z]?)(?P<paras>(?:\([^)\s]+\))*))?"
)
#: Bare "§ 410.78(a)(3)" — no title present. Every paragraph this module
#: scans is a PFS rule (or the CPT manual's own Medicare-policy prose),
#: so the title is inferred as 42 (Resolutions: kept in memory only —
#: the stored ``locator`` is always the canonical "42 CFR ..." form,
#: never a flag).
BARE_CFR_RE = re.compile(r"§\s*(\d{2,4})\.(\d+[a-z]?)((?:\([^)\s]+\))*)")
#: "§§ 410.26 and 410.32" — the two-section plural form (Ruling A10:
#: the only bare-§ list shape handled; a longer comma-separated bare
#: list is not, since "§" alone doesn't carry the same list-continuation
#: convention FR drafting uses after a titled "CFR" cite).
_BARE_CFR_PLURAL_RE = re.compile(
r"§§\s*(\d{2,4}\.\d+[a-z]?(?:\([^)\s]+\))*)\s+and\s+"
r"(\d{2,4}\.\d+[a-z]?(?:\([^)\s]+\))*)"
)
#: Separator between list items in a CFR citation run: ", ", ", and ",
#: " and ", " or ", " through ", etc.
_CFR_LIST_SEP = r"(?:\s*,\s*(?:and\s+|or\s+|through\s+)?|\s+(?:and|or|through)\s+)"
#: A continuation token after a titled section-form head ("410.26",
#: "410.26(a)(3)") — no "CFR" keyword, so it must immediately follow a
#: list separator (Ruling A10) or it is prose, not a citation.
# No "^" anchor: Pattern.match(text, pos) already requires the match to
# start exactly at pos — "^" would instead (and wrongly) require pos to
# be the real start of the whole string (or MULTILINE-after-newline),
# so every continuation past the first character would silently fail.
_CFR_CONT_SECTION_RE = re.compile(
_CFR_LIST_SEP + r"(\d{2,4}\.\d+[a-z]?(?:\([^)\s]+\))*)"
)
#: A continuation token after a plural "parts" head ("405, 414, and
#: 426") — bare part numbers, no decimal (the negative lookahead keeps
#: "414.5" from being misread as a bare part).
_CFR_CONT_PART_RE = re.compile(_CFR_LIST_SEP + r"(\d{2,4})(?!\.\d)")
#: "Pub. 100-04, chapter 12" / "Publication 100-04 ... Chapter 12" (the
#: pub-number form) or "Medicare Claims Processing Manual ... Chapter
#: 12, Section 30.6.4" (the manual-name form) — either may open the
#: match; both close on an optional trailing "Section N.N".
IOM_RE = re.compile(
r"(?:Pub(?:lication)?\.?\s*100-0(\d)"
r"|(Claims Processing|Benefit Policy|Program Integrity|"
r"National Coverage Determinations|Managed Care|Financial Management)"
r"\s+Manual)"
r"[^.]{0,120}?[Cc]hapter\s*(\d+[A-Z]?)"
r"(?:[^.]{0,120}?[Ss]ection\s*([\d.]+))?"
)
#: "MLN907166" / "MLN 907166" booklet numbers (#689: optional/best-effort
#: — no title match against bib items, just the number surfaced as a
#: locator; always unresolved).
MLN_RE = re.compile(r"\bMLN\s*(\d{6,7})\b", re.IGNORECASE)
def resolve_cfr(store: Any, title: str, section: str) -> str:
"""The bib ``regulation`` item covering "*title* CFR *section*", or
``""`` when the library has no covering item (699 eCFR sections on
hand; most of the U.S. Code of Federal Regulations is not)."""
cite = parse_cite(f"{title} CFR {section}")
return item_for(cite, store)
def resolve_iom(store: Any, pub: str, chapter: str) -> str:
"""The bib item tagged ``pub:<pub>`` whose title names ``Chapter
<chapter>:`` (``bib.iom`` stamps chapter titles as "<manual> —
Chapter <N>: <title>"), or ``""`` when unresolved."""
marker = f"Chapter {chapter}:"
for item in store.list_items(tag=f"pub:{pub}"):
if marker in item.title:
return item.key
return ""
def _cfr_continuations(text: str, pos: int, pattern: re.Pattern[str]) -> list[str]:
"""Every list-continuation token starting at *pos* (Ruling A10) —
``pattern`` is ``_CFR_CONT_SECTION_RE`` or ``_CFR_CONT_PART_RE``.
Stops at the first position that isn't a separator + token (a
sentence continuing in prose, e.g. "... 410.78 and the physician",
never captures "the")."""
out: list[str] = []
while True:
m = pattern.match(text, pos)
if m is None:
break
out.append(m.group(1))
pos = m.end()
return out
def _cfr_refs(text: str) -> list[tuple[str, str]]:
"""(title, section) pairs — *section* already shaped for
``bib.cfrlink.parse_cite``'s ``"<title> CFR <section>"`` grammar
("410.78(a)(3)" or "Part 425") — for every CFR reference in *text*,
including every section/part named in a list continuation after a
titled head (Ruling A10: "42 CFR 410.20, 410.26, and 410.32" / "...
410.26(a)(3) and 410.26(b) ..." / "42 CFR parts 405, 414, and
426")."""
out: list[tuple[str, str]] = []
for m in CFR_RE.finditer(text):
title = m.group("title")
part, dec, paras = m.group("part"), m.group("dec"), m.group("paras") or ""
if dec:
out.append((title, f"{part}.{dec}{paras}"))
for token in _cfr_continuations(text, m.end(), _CFR_CONT_SECTION_RE):
out.append((title, token))
else:
out.append((title, f"Part {part}"))
if (m.group("partword") or "").lower() == "parts":
for token in _cfr_continuations(text, m.end(), _CFR_CONT_PART_RE):
out.append((title, f"Part {token}"))
consumed: list[tuple[int, int]] = []
for m in _BARE_CFR_PLURAL_RE.finditer(text):
out.append(("42", m.group(1)))
out.append(("42", m.group(2)))
consumed.append(m.span())
for m in BARE_CFR_RE.finditer(text):
if any(start <= m.start() < end for start, end in consumed):
continue # already captured by the §§ plural form
part, dec, paras = m.group(1), m.group(2), m.group(3) or ""
out.append(("42", f"{part}.{dec}{paras}"))
return out
def _iom_refs(text: str) -> list[tuple[str, str, str]]:
"""(pub, chapter, section) for every IOM reference in *text* —
*section* is ``""`` when the sentence names a chapter but no
section. A manual name the pub map doesn't cover is dropped rather
than resolved with a guessed number."""
out: list[tuple[str, str, str]] = []
for m in IOM_RE.finditer(text):
pub_digit, manual_name, chapter, section = m.groups()
pub = (
f"100-0{pub_digit}"
if pub_digit
else _MANUAL_PUB.get((manual_name or "").lower(), "")
)
if not pub:
continue
# A sentence-final "Section 10.2.5.2." swallows the closing
# period into [\d.]+ — trim it; a real section number never
# ends in ".".
out.append((pub, chapter.upper(), (section or "").rstrip(".")))
return out
def _extract(text: str, store: Any) -> list[tuple[str, str, str]]:
"""(kind, locator, item_key) for every CFR/IOM/MLN reference in
*text*, resolved against *store* — ``item_key`` is ``""`` when
unresolved."""
out: list[tuple[str, str, str]] = []
for title, section in _cfr_refs(text):
cite = parse_cite(f"{title} CFR {section}")
out.append(("cfr", canonical(cite), resolve_cfr(store, title, section)))
for pub, chapter, section in _iom_refs(text):
locator = f"{pub} ch.{chapter}" + (f" §{section}" if section else "")
out.append(("iom", locator, resolve_iom(store, pub, chapter)))
for m in MLN_RE.finditer(text):
out.append(("mln", f"MLN {m.group(1)}", ""))
return out
def dedupe(rows: Sequence[GuidanceRow]) -> list[GuidanceRow]:
"""First-seen-wins de-duplication on ``(family, code, kind,
locator, item_key_src, p_id_src)`` — see the module docstring."""
seen: set[tuple[str, str, str, str, str, int]] = set()
out: list[GuidanceRow] = []
for r in rows:
key = (r.family, r.code, r.kind, r.locator, r.item_key_src, r.p_id_src)
if key in seen:
continue
seen.add(key)
out.append(r)
return out
def harvest(store: Any, codes: Sequence[str], *, family: str) -> list[GuidanceRow]:
"""FR paragraphs (``fr_anchors``) naming any of *codes* -> the
CFR/IOM/MLN references they cite, anchored to that paragraph
(``item_key_src``/``p_id_src``/``page_src``)."""
if not codes:
return []
con = store._con() # noqa: SLF001 — same pattern as pfs.descriptors
pattern = code_pattern(codes)
like_clause = " OR ".join("text LIKE ?" for _ in codes)
params = [f"%{c.upper()}%" for c in codes]
rows = con.execute(
f"SELECT item_key, p_id, page, text FROM fr_anchors WHERE {like_clause}",
params,
).fetchall()
out: list[GuidanceRow] = []
for item_key, p_id, page, text in rows:
present = codes_in(text, pattern)
if not present:
continue
for kind, locator, resolved in _extract(text, store):
for code in present:
out.append(
GuidanceRow(
family, code, kind, locator, resolved, item_key, p_id, page
)
)
return dedupe(out)
def harvest_cpt(
con: Any, store: Any, codes: Sequence[str], *, family: str
) -> list[GuidanceRow]:
"""The newest ingested CPT edition's guideline text and ``see``/
``other`` instructions for *codes* -> the CFR/IOM/MLN references
they cite, anchored to the CPT edition item (``p_id_src=0``,
``page_src=0`` — no FR paragraph; locating a page inside the PDF
chapter is out of scope, #689). ``[]`` on a replica with no
``pfs.cpt_*`` tables yet (I4: not a bug, just nothing ingested)."""
from pfs.codetables import (
cpt_years,
read_cpt_codes,
read_cpt_instructions,
read_cpt_sections,
)
if not codes:
return []
try:
years = cpt_years(con)
except Exception as exc:
if is_missing_table_error(exc):
return []
raise
if not years:
return []
year = max(years)
codes_u = {c.upper() for c in codes}
sections = {s.sec_id: s for s in read_cpt_sections(con, year)}
codes_by_sec: dict[str, set[str]] = {}
for c in read_cpt_codes(con, year):
if c.code in codes_u:
codes_by_sec.setdefault(c.sec_id, set()).add(c.code)
out: list[GuidanceRow] = []
for sec_id, fam_codes in codes_by_sec.items():
section = sections.get(sec_id)
if section is None or not section.guideline:
continue
for kind, locator, resolved in _extract(section.guideline, store):
for code in sorted(fam_codes):
out.append(
GuidanceRow(
family, code, kind, locator, resolved, section.item_key, 0, 0
)
)
for instr in read_cpt_instructions(con, year):
if instr.code not in codes_u or instr.kind not in ("see", "other"):
continue
if not any(k in instr.text for k in ("CFR", "Medicare", "Chapter", "chapter")):
continue
for kind, locator, resolved in _extract(instr.text, store):
out.append(
GuidanceRow(
family, instr.code, kind, locator, resolved, instr.item_key, 0, 0
)
)
return dedupe(out)
def build(
con: Any, store: Any, codes: Sequence[str], *, family: str
) -> list[GuidanceRow]:
"""Every guidance row for one family — FR (``harvest``) + CPT
(``harvest_cpt``), deduped across both sources."""
rows = harvest(store, codes, family=family)
rows.extend(harvest_cpt(con, store, codes, family=family))
return dedupe(rows)

333
src/pfs/reaction.py Normal file
View File

@@ -0,0 +1,333 @@
"""Public-reaction series per code family (``pfs.code_reaction``, #690,
task 7 of the 2026-09-09 code-family-anchors plan).
Two period kinds, one ``ReactionRow`` shape:
* ``"docket"`` — one row per regulations.gov docket (2017-2026, ten
sealed dockets, ``bib.Store.dockets()``): how many distinct
commenters (``item_key``) mentioned the family, out of the docket's
total commenters, straight from the ``comments`` pgvector
collection's chunk metadata (``docket``, ``item_key``, ``codes``,
``families``) — never from ``Docket.counts_json``, which counts raw
comment documents, not distinct commenters who named the family.
When ``--stance-sample N`` is passed, the newest *N* commenters' first
matching chunk is classified into ``STANCES`` by the self-hosted
closed-vocabulary classifier (``llm.classify``); an answer outside
the vocabulary counts as ``"unclear"``.
* ``"fr-pairs"`` — one row per FR rule item, back to 2001: how many
``Comment:``/``Response:`` paragraph pairs in that rule's
``fr_anchors`` name a family code (``n_items``), out of the rule's
total ``Comment:`` paragraph count — family or not (``n_total``) —
the pre-2017 proxy, since regulations.gov comment text itself isn't
indexed for rules that old.
``series`` builds both halves for one family (or one ``--code``
pseudo-family, a single-code family keyed by the code itself) and
returns them as one list, dockets first (docket id order) then FR pairs
(rule item_key order); the CLI (``stack pfs reaction``) prints and
optionally writes them.
"""
from __future__ import annotations
import json
import re
from collections import Counter
from typing import Any, Callable, Sequence
from sqlalchemy import text
from pfs.descriptors import rule_year_of
from pfs.families import code_pattern, codes_in
#: Closed vocabulary the stance classifier answers from — anything else
#: (a refusal, a hedge, garbage) counts as "unclear" (Resolutions).
STANCES = ("support", "oppose", "modify", "unclear")
#: One docket's distinct-commenter count, with vs. without the family —
#: matches by literal code presence (``codes`` metadata) OR family
#: membership (``families`` metadata), the same two-way match
#: ``llm.evidence``'s code-cited/family-cited sources use, so a chunk
#: the indexer only stamped via family membership (no literal code
#: string in that particular chunk) still counts.
_DOCKET_COUNTS_SQL = text(
"SELECT docket, "
"count(DISTINCT item_key) FILTER (WHERE matched) AS n_items, "
"count(DISTINCT item_key) AS n_total "
"FROM ( "
"SELECT e.cmetadata->>'docket' AS docket, e.cmetadata->>'item_key' AS item_key, "
"(string_to_array(COALESCE(e.cmetadata->>'codes', ''), ' ') && CAST(:codes AS text[])) "
"OR (string_to_array(COALESCE(e.cmetadata->>'families', ''), ' ') && ARRAY[:family_key]) "
"AS matched "
"FROM langchain_pg_embedding e "
"WHERE e.collection_id = (SELECT uuid FROM langchain_pg_collection WHERE name = :collection) "
") t "
"WHERE docket IS NOT NULL AND docket <> '' "
"GROUP BY docket ORDER BY docket"
)
#: The newest N distinct items' first matching chunk in one docket — the
#: same one-chunk-per-item windowing ``llm.evidence``'s family-cited SQL
#: uses (stage one: lowest ``seq`` per item_key; stage two: newest
#: ``date`` first), text truncated to 1,500 chars for the classifier
#: (Resolutions).
_STANCE_SAMPLE_SQL = text(
"SELECT item_key, LEFT(document, 1500) FROM ( "
"SELECT e.cmetadata->>'item_key' AS item_key, e.document AS document, "
"e.cmetadata->>'date' AS date, "
"row_number() OVER ( "
"PARTITION BY e.cmetadata->>'item_key' "
"ORDER BY COALESCE(NULLIF(e.cmetadata->>'seq', '')::int, 0) ASC "
") AS item_rn "
"FROM langchain_pg_embedding e "
"WHERE e.collection_id = (SELECT uuid FROM langchain_pg_collection WHERE name = :collection) "
"AND e.cmetadata->>'docket' = :docket "
"AND ( "
"(string_to_array(COALESCE(e.cmetadata->>'codes', ''), ' ') && CAST(:codes AS text[])) "
"OR (string_to_array(COALESCE(e.cmetadata->>'families', ''), ' ') && ARRAY[:family_key]) "
") "
") one_per_item "
"WHERE item_rn = 1 "
"ORDER BY date DESC NULLS LAST "
"LIMIT :n"
)
_COMMENTS_COLLECTION = "comments"
def docket_counts(
engine: Any, family_key: str, codes: Sequence[str]
) -> list[tuple[str, int, int]]:
"""``(docket, n_items_with_family, n_items_total)`` for every docket
that has any chunks in the ``comments`` collection — one round trip,
``GROUP BY docket``."""
upper = [c.upper() for c in codes]
with engine.begin() as conn:
rows = conn.execute(
_DOCKET_COUNTS_SQL,
{
"codes": upper,
"family_key": family_key.upper(),
"collection": _COMMENTS_COLLECTION,
},
).fetchall()
return [
(docket, int(n_items or 0), int(n_total or 0))
for docket, n_items, n_total in rows
]
def stance_sample(
engine: Any, family_key: str, codes: Sequence[str], docket: str, *, n: int
) -> list[tuple[str, str]]:
"""``(item_key, text)`` for the newest *n* distinct commenters in
*docket* who mentioned the family, first matching chunk, newest
first, text capped to 1,500 chars."""
if n <= 0:
return []
upper = [c.upper() for c in codes]
with engine.begin() as conn:
rows = conn.execute(
_STANCE_SAMPLE_SQL,
{
"codes": upper,
"family_key": family_key.upper(),
"collection": _COMMENTS_COLLECTION,
"docket": docket,
"n": n,
},
).fetchall()
return [(item_key, text or "") for item_key, text in rows]
def _prompt_text(family_key: str, codes: Sequence[str], chunk: str) -> str:
return (
f"Comment on Medicare payment for {family_key} "
f"(codes {' '.join(codes)}):\n{chunk}"
)
def _stance_of(text: str, classify: Callable[[str, Sequence[str]], str | None]) -> str:
stance = classify(text, STANCES)
return stance if stance in STANCES else "unclear"
def classify_stances(
sample: Sequence[tuple[str, str]],
classify: Callable[[str, Sequence[str]], str | None],
) -> Counter:
"""Tally ``STANCES`` for a stance sample — an answer outside the
closed vocabulary counts as ``"unclear"`` (Resolutions)."""
return Counter(_stance_of(text, classify) for _, text in sample)
#: "Comment:" / "Response:" paragraph openers — a docket-comment excerpt
#: reproduced verbatim inside an FR rule's own preamble, always at the
#: very start of the paragraph. Tightened to no leading whitespace and no
#: space before the colon (case-insensitive, as before) so these agree
#: exactly with the ``LIKE 'Comment:%'``/``LIKE 'Response:%'`` SQL
#: prefilter in ``fr_pairs`` — a looser Python regex than the SQL LIKE
#: would count paragraphs the SQL never even fetched.
_COMMENT_RE = re.compile(r"^Comment:", re.I)
_RESPONSE_RE = re.compile(r"^Response:", re.I)
def fr_pairs(store: Any, codes: Sequence[str]) -> list[tuple[str, int, int, int]]:
"""``(item_key, rule_year, n_pairs, n_comment_paras)`` per rule item —
the pre-2017 proxy for public reaction, since regulations.gov comment
text isn't indexed that far back. ``n_pairs`` is the count of
qualifying ``Comment:``/``Response:`` pairs naming one of *codes*
(the family's share); ``n_comment_paras`` is the item's TOTAL count
of ``Comment:`` paragraphs — every one in the rule, family or not,
paired or not — the pool the family's share is drawn out of (the
same n_items-out-of-n_total shape the docket rows use).
Ruling A12: a ``Comment:`` paragraph pairs with the *next*
``Response:`` paragraph in the same item that comes before the next
``Comment:`` paragraph — intervening continuation paragraphs (the
comment's own prose, wrapped across more than one ``fr_anchors``
row) are allowed between them, and the code match runs over the
whole span (comment + continuations + response), not just the
comment/response paragraphs themselves. A ``Comment:`` with no
``Response:`` before the next ``Comment:`` (or the end of the item)
is unpaired and doesn't count toward ``n_pairs`` — it still counts
toward ``n_comment_paras``, which is unconditional. Rules with zero
qualifying pairs are omitted (``n_comment_paras`` alone never keeps a
rule in the output).
One SQL per family, scoped to ``Comment:``/``Response:`` openers —
unconditionally, every item's, which is what makes ``n_comment_paras``
a true item-wide total even though the query is family-scoped — plus
any paragraph naming one of *codes* (the same LIKE-prefilter idiom
``pfs.guidance.harvest`` uses) — safe under the pairing rule above: a
dropped paragraph is prose that names no code and opens neither a
comment nor a response, so it can neither create nor hide a pair.
``_COMMENT_RE``/``_RESPONSE_RE`` are anchored exactly like the SQL's
``LIKE 'Comment:%'``/``LIKE 'Response:%'`` (no leading whitespace, no
space before the colon) so the Python pass never counts a paragraph
the SQL wouldn't have fetched as an opener in the first place.
"""
if not codes:
return []
con = store._con() # noqa: SLF001 — same pattern as pfs.descriptors/pfs.guidance
code_likes = " OR ".join("a.text LIKE ?" for _ in codes)
params = [f"%{c.upper()}%" for c in codes]
rows = con.execute(
"SELECT a.item_key, a.p_id, a.text, i.title, i.date_published "
"FROM fr_anchors a JOIN items i ON i.key = a.item_key "
f"WHERE a.text LIKE 'Comment:%' OR a.text LIKE 'Response:%' OR ({code_likes}) "
"ORDER BY a.item_key, a.p_id",
params,
).fetchall()
pattern = code_pattern(codes)
by_item: dict[str, list[str]] = {}
meta: dict[str, tuple[str, str]] = {}
for item_key, _p_id, para_text, title, date_published in rows:
by_item.setdefault(item_key, []).append(para_text or "")
meta[item_key] = (title, date_published)
out: list[tuple[str, int, int, int]] = []
for item_key, paras in by_item.items():
n_pairs = 0
span: list[str] | None = None # None: no comment currently open
for para_text in paras:
if _COMMENT_RE.match(para_text):
# A new Comment: abandons any still-open, unresolved one
# (Ruling A12: unpaired, never counted).
span = [para_text]
continue
if _RESPONSE_RE.match(para_text):
if span is not None:
span.append(para_text)
if codes_in(" ".join(span), pattern):
n_pairs += 1
span = None
continue
if span is not None:
span.append(para_text) # continuation paragraph
if n_pairs:
n_comment_paras = sum(1 for p in paras if _COMMENT_RE.match(p))
title, date_published = meta[item_key]
out.append(
(
item_key,
rule_year_of(title, date_published),
n_pairs,
n_comment_paras,
)
)
return sorted(out, key=lambda r: (r[1], r[0]))
def series(
engine: Any,
store: Any,
family: str,
codes: Sequence[str],
*,
classify: Callable[[str, Sequence[str]], str | None] | None = None,
stance_sample_n: int = 0,
name: str | None = None,
) -> list[Any]:
"""Every ``ReactionRow`` for one family (or ``--code`` pseudo-family):
a docket row per docket with any comment chunks, then an fr-pairs row
per rule with qualifying Comment/Response pairs. Stance columns stay
zero (and ``sample_json`` ``"[]"``) unless *classify* is given and
*stance_sample_n* > 0. *name* is the family's display name for the
stance prompt (``FAMILIES[key].name`` for a registered family); it
defaults to *family* itself, which is already the right label for a
``--code`` pseudo-family (there is no separate display name for a
single code)."""
from pfs.codetables import ReactionRow
label = name or family
docket_years = {d.id: d.comment_end_date[:4] for d in store.dockets()}
rows: list[ReactionRow] = []
for docket, n_items, n_total in docket_counts(engine, family, codes):
year_str = docket_years.get(docket, "")
year = int(year_str) if year_str.isdigit() else 0
stance_counts: Counter = Counter()
sample_json = "[]"
if classify is not None and stance_sample_n > 0:
sample = stance_sample(engine, family, codes, docket, n=stance_sample_n)
stances = [
(item_key, _stance_of(_prompt_text(label, codes, chunk), classify))
for item_key, chunk in sample
]
stance_counts = Counter(s for _, s in stances)
sample_json = json.dumps([{"item_key": k, "stance": s} for k, s in stances])
rows.append(
ReactionRow(
family=family,
period=docket,
period_kind="docket",
year=year,
n_items=n_items,
n_total=n_total,
stance_support=stance_counts.get("support", 0),
stance_oppose=stance_counts.get("oppose", 0),
stance_modify=stance_counts.get("modify", 0),
stance_unclear=stance_counts.get("unclear", 0),
sample_json=sample_json,
)
)
for item_key, rule_year, n_pairs, n_comment_paras in fr_pairs(store, codes):
rows.append(
ReactionRow(
family=family,
period=item_key,
period_kind="fr-pairs",
year=rule_year,
n_items=n_pairs,
n_total=n_comment_paras,
stance_support=0,
stance_oppose=0,
stance_modify=0,
stance_unclear=0,
sample_json="[]",
)
)
return rows

Some files were not shown because too many files have changed in this diff Show More