homelab/stack
1
0
Fork 0
Code Issues 247 Pull Requests 2 Actions Packages Projects 1 Releases Wiki Activity

feat(notebooks,docs): code_families sections 5-7 — anchors, guidance, reaction; CLI docs (refs #688-#690)

Browse Source
This commit is contained in:
kert
2026-09-10 00:28:58 -04:00
parent 9b173d3443
commit 110aeccb89
89 changed files with 1948 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/docs/cli/api-serve.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack api serve title: stack api serve
sidebar_position: 28 sidebar_position: 58
--- ---
# `stack api serve` # `stack api serve`

2
docs/docs/cli/api.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack api title: stack api
sidebar_position: 27 sidebar_position: 57
--- ---
# `stack api` # `stack api`

21
docs/docs/cli/bib-attach-iom.md Normal file
View File

@@ -0,0 +1,21 @@
---
title: stack bib attach-iom
sidebar_position: 21
---
# `stack bib attach-iom`
```
Usage: stack bib attach-iom [OPTIONS]
Download the PDF for every Manual and attach it to the record.
Idempotent by SHA-256 — only downloads files that have changed on
CMS since the last run. Safe to schedule on a timer.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --pub -p TEXT Limit to specific pub numbers. Repeat for multiple. │
│ --force Re-download even if local SHA-256 matches. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

17
docs/docs/cli/bib-attach-oig.md Normal file
View File

@@ -0,0 +1,17 @@
---
title: stack bib attach-oig
sidebar_position: 26
---
# `stack bib attach-oig`
```
Usage: stack bib attach-oig [OPTIONS]
Download and attach the PDF/HTML for every OIG guidance record.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --force Re-download even if unchanged. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

49
docs/docs/cli/bib-backfill-comments.md Normal file
View File

@@ -0,0 +1,49 @@
---
title: stack bib backfill-comments
sidebar_position: 19
---
# `stack bib backfill-comments`
```
Usage: stack bib backfill-comments [OPTIONS]
Enrich every reg-gov comment stub with body + attachments.
Walks items tagged ``source:regulations-gov`` whose ``abstract`` is
still empty and hits ``/v4/comments/{id}?include=attachments`` for
each one. Resumable: stop any time, start again, it picks up where
it left off by skipping already-enriched items. At 1.3s/req the API
calls run at ~2700/hr; attachment downloads are concurrent on the
CDN and don't count against the rate limit.
``--mirror`` swaps the API for the Mirrulations public S3 mirror
(#662): no API key, no cap, and mirror-only comments get created.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --docket TEXT Limit to one docket (e.g. CMS-2026-2377). Without │
│ it the walk covers every un-enriched stub in the │
│ store — a multi-day crawl at the reg.gov rate │
│ cap. │
│ --limit -n INTEGER Cap items processed this run. 0 = no cap │
│ (multi-day crawl). │
│ [default: 0] │
│ --sleep FLOAT Seconds between API calls. 3.7s ≈ 970/hr — just │
│ under the 1000/hr reg.gov hourly cap (the binding │
│ limit; the 50/min burst cap is looser). CDN │
│ downloads are concurrent and don't count. │
│ [default: 3.7] │
│ --log PATH Append-only progress log (survives restarts). │
│ [default: /tmp/bib-backfill-comments.log] │
│ --mirror Pull from the Mirrulations S3 mirror instead of │
│ the reg.gov API — bulk speed, no rate cap, also │
│ ingests comments the original farm never listed. │
│ Requires --docket. │
│ --force Run even if the docket is sealed. Bypasses the │
│ docket seal only: items already tagged │
│ enriched:ok (or enriched:gone) are still skipped, │
│ so this resumes an interrupted enrichment — it │
│ never re-enriches an item. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

25
docs/docs/cli/bib-cfr-jump.md Normal file
View File

@@ -0,0 +1,25 @@
---
title: stack bib cfr-jump
sidebar_position: 31
---
# `stack bib cfr-jump`
```
Usage: stack bib cfr-jump [OPTIONS] [REF]
Bidirectional CFR cite ↔ eCFR URL (#640); optionally place the link.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ ref [REF] CFR cite: "42 CFR 425.400(c)(1)(iv)" or "42 CFR Part 425". │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --url TEXT Reverse mode: print the cite for an eCFR URL. │
│ --md Print a markdown link. │
│ --place Attach the link to its bib regulation item (fr_links) │
│ so the next zotero sync carries it as a child link │
│ attachment. │
│ --label TEXT Label for the placed link. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

44
docs/docs/cli/bib-code-tags.md Normal file
View File

@@ -0,0 +1,44 @@
---
title: stack bib code-tags
sidebar_position: 32
---
# `stack bib code-tags`
```
Usage: stack bib code-tags [OPTIONS]
Stamp code:/family: item tags from chunk anchors and FR paragraphs.
Rolls up ``pfs.anchors.anchor_metadata`` — already stamped on
pgvector chunks by ``stack llm index``/``stack llm restamp`` — to
the bib item level: every code an item's chunks name becomes a
``code:<CODE>`` tag, every family a ``family:<KEY>`` tag. Rules also
get a pgvector-free source (``fr_anchors.text``, the FR paragraph
anchor map from ``stack bib fr-grab``); pass ``--anchors-only`` to
use only that and skip pgvector for rules (and skip comments/corpus
entirely — they have no other source). Additive and idempotent
(``bib.codetags.apply_code_tags``) — safe to rerun after every
``stack llm restamp``. Every selected source is merged into one
``item_key -> (codes, families)`` mapping before tagging (Ruling
A7), so an item that straddles two collections (e.g. rules and
corpus) is capped by ``--max-codes-per-item`` once, on its combined
code set — the printed per-source lines are informational only;
``tags_added``/``skipped_wide`` in the total come from that one
merged pass.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --collection TEXT Which collection: all | comments | │
│ rules | corpus. │
│ [default: all] │
│ --anchors-only Rules only, straight from fr_anchors │
│ text — skip pgvector entirely. │
│ --dry-run Report per-collection item/tag counts; │
│ write nothing. │
│ --max-codes-per-item INTEGER Items with more distinct codes get │
│ family: tags only (+ hand-family │
│ codes). │
│ [default: 200] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

24
docs/docs/cli/bib-discover-pfs-rules.md Normal file
View File

@@ -0,0 +1,24 @@
---
title: stack bib discover-pfs-rules
sidebar_position: 15
---
# `stack bib discover-pfs-rules`
```
Usage: stack bib discover-pfs-rules [OPTIONS]
Walk the Federal Register API for every PFS rule since a date.
Upserts each as a bib ``Rule`` record tagged ``module:pfs``,
``source:federal-register``, ``rule:<cms-id>``. Cheap — the FR API
has no auth and no rate charge we can trigger at this scale
(~20 rules since 2017).
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --since TEXT [default: 2017-01-01] │
│ --until TEXT │
│ --dry-run │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

32
docs/docs/cli/bib-fetch-docket-comments.md Normal file
View File

@@ -0,0 +1,32 @@
---
title: stack bib fetch-docket-comments
sidebar_position: 16
---
# `stack bib fetch-docket-comments`
```
Usage: stack bib fetch-docket-comments [OPTIONS] DOCKET
Walk one docket's comments from its stored watermark; upsert as Source items.
The first run discovers the docket's commentable FR document (one
listing call) and stores it; later runs make no discovery calls.
Sealed dockets are skipped unless --force.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ * docket TEXT reg.gov docket id, e.g. CMS-2026-2377 [required] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --cms-id TEXT Associate comments with a specific CMS rule │
│ tag. │
│ --limit -n INTEGER Stop after N comments. 0 = all. [default: 0] │
│ --attachments Also download each comment's PDF/DOCX │
│ attachments. │
│ --sleep FLOAT Seconds between API calls (rate budget). │
│ [default: 1.3] │
│ --force Walk from page 1 even if sealed / │
│ watermarked. Never unseals. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

32
docs/docs/cli/bib-fetch-pfs-comments.md Normal file
View File

@@ -0,0 +1,32 @@
---
title: stack bib fetch-pfs-comments
sidebar_position: 17
---
# `stack bib fetch-pfs-comments`
```
Usage: stack bib fetch-pfs-comments [OPTIONS]
Discover every PFS proposed rule since *since* and pull new comments on each
docket from its stored watermark.
Sealed dockets (comment period closed + quiet period + an empty pull)
cost nothing: no rule-metadata fetch, no resolve call, no walk.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --since TEXT [default: 2017-01-01] │
│ --until TEXT │
│ --attachments │
│ --per-docket-limit INTEGER Cap comments fetched per docket. 0 = │
│ unlimited. │
│ [default: 0] │
│ --sleep FLOAT [default: 1.3] │
│ --docket TEXT Only pull this reg.gov docket (e.g. │
│ CMS-2026-2377); other dockets are skipped │
│ before any API call once known. │
│ --force Walk every docket from page 1, sealed or │
│ not. Never unseals. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

22
docs/docs/cli/bib-fr-grab.md Normal file
View File

@@ -0,0 +1,22 @@
---
title: stack bib fr-grab
sidebar_position: 29
---
# `stack bib fr-grab`
```
Usage: stack bib fr-grab [OPTIONS]
Grab federalregister.gov paragraph/page anchor maps into bib (#634).
One body-HTML fetch per rule; anchors persist in ``fr_anchors`` /
``fr_anchor_docs`` for offline jump-link resolution.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --key TEXT Grab one rule item by bib key. │
│ --all Backfill every rule item with an FR URL. │
│ --force Re-grab even if an anchor map already exists. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

26
docs/docs/cli/bib-fr-jump.md Normal file
View File

@@ -0,0 +1,26 @@
---
title: stack bib fr-jump
sidebar_position: 30
---
# `stack bib fr-jump`
```
Usage: stack bib fr-jump [OPTIONS] REF
Resolve an FR citation to its web jump link (#637); optionally place it.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ * ref TEXT FR reference: "91 FR 44218", "91 FR 44218 ¶3", "p-3600", │
│ or a quote. │
│ [required] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --key TEXT Bib item key to scope resolution. │
│ --md Print a markdown link. │
│ --place Record the link in fr_links so the next zotero sync │
│ carries it as a child link attachment. │
│ --label TEXT Label for the placed link. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

22
docs/docs/cli/bib-ingest-iom.md Normal file
View File

@@ -0,0 +1,22 @@
---
title: stack bib ingest-iom
sidebar_position: 20
---
# `stack bib ingest-iom`
```
Usage: stack bib ingest-iom [OPTIONS]
Crawl the CMS IOM index and upsert every chapter as a Manual.
Idempotent: re-running only adds missing chapters (dedup is by URL
in the store). Without --pub, ingests every manual the CMS index
advertises.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --pub -p TEXT Limit to specific pub numbers (e.g. 100-02). Repeat │
│ for multiple. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

28
docs/docs/cli/bib-ingest-mail.md Normal file
View File

@@ -0,0 +1,28 @@
---
title: stack bib ingest-mail
sidebar_position: 18
---
# `stack bib ingest-mail`
```
Usage: stack bib ingest-mail [OPTIONS]
Pull UNSEEN mail from a mailbox via IMAPS, upsert as bib Sources.
Idempotent: marks each ingested message ``\Seen`` on the server,
so re-runs only fetch new mail. Attachments ride along into bib
storage. After this, ``stack bib sync-zotero`` routes everything
tagged ``source:email`` into the appropriate Zotero collection.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --user TEXT Mailbox address to poll (must have a password in │
│ .state/mail/credentials.json). │
│ [default: cmsupdates@mail.fhirworx.io] │
│ --host TEXT [default: mail.fhirworx.io] │
│ --port INTEGER [default: 993] │
│ --folder TEXT [default: INBOX] │
│ --limit -n INTEGER [default: 0] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

18
docs/docs/cli/bib-ingest-oig.md Normal file
View File

@@ -0,0 +1,18 @@
---
title: stack bib ingest-oig
sidebar_position: 25
---
# `stack bib ingest-oig`
```
Usage: stack bib ingest-oig [OPTIONS]
Crawl OIG Compliance Program Guidance + fraud alerts indexes.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --kind -k TEXT Restrict to a subset: 'cpg' and/or 'alerts'. Repeat │
│ for multiple. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

30
docs/docs/cli/bib-refresh-iom.md Normal file
View File

@@ -0,0 +1,30 @@
---
title: stack bib refresh-iom
sidebar_position: 28
---
# `stack bib refresh-iom`
```
Usage: stack bib refresh-iom [OPTIONS]
One-shot: ingest-iom → watch-iom → attach-iom → sync-zotero.
Safe to put behind a cron. Idempotent at every step:
- ingest: upserts by URL, prunes stale records
- watch: re-attaches futurepdf.pdf only when its SHA-256 changes
- attach: skips PDFs whose checksum matches the stored copy
- sync: skips Zotero items whose URL already exists
Zotero is held (stopped) during the sync phase only.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --pub -p TEXT │
│ --file-issue --no-file-issue When watch-iom detects a │
│ futurepdf.pdf change, auto-file a │
│ Gitea issue tagged 'cms-update'. │
│ Disable for local/dev runs. │
│ [default: file-issue] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

16
docs/docs/cli/bib-refresh-oig.md Normal file
View File

@@ -0,0 +1,16 @@
---
title: stack bib refresh-oig
sidebar_position: 27
---
# `stack bib refresh-oig`
```
Usage: stack bib refresh-oig [OPTIONS]
One-shot: ingest-oig → attach-oig → sync-zotero (OIG tag).
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

51
docs/docs/cli/bib-sync-zotero.md Normal file
View File

@@ -0,0 +1,51 @@
---
title: stack bib sync-zotero
sidebar_position: 24
---
# `stack bib sync-zotero`
```
Usage: stack bib sync-zotero [OPTIONS]
Push bib items (and their attachments) to the Zotero database.
Zotero holds an exclusive write lock on its SQLite file while
running. By default we stop the 'zotero' compose service for the
duration of the sync and start it again afterward.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --tag TEXT Only sync items matching a │
│ tag. │
│ --type TEXT Only sync one item_type. │
│ --hold-zotero --no-hold Stop the 'zotero' container │
│ during sync so Zotero │
│ releases its exclusive │
│ SQLite lock; restart │
│ afterwards. │
│ [default: hold-zotero] │
│ --dedupe-collection TEXT DEPRECATED — use │
│ --merge-dupes instead, │
│ which merges rather than │
│ deletes. After the push, │
│ remove title+date │
│ duplicates from this Zotero │
│ collection (and from bib, │
│ so the next sync can't │
│ resurrect them). CMS │
│ listservs re-send identical │
│ mail with fresh │
│ Message-IDs, so duplicates │
│ otherwise accumulate — e.g. │
│ 'Cmsupdates'. │
│ --merge-dupes --no-merge-dupes After the push, merge │
│ same-URL duplicate items │
│ library-wide (union │
│ tags/collections/notes/att… │
│ keep-earliest). Supersedes │
│ --dedupe-collection, which │
│ deletes without merging. │
│ [default: no-merge-dupes] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

21
docs/docs/cli/bib-watch-iom.md Normal file
View File

@@ -0,0 +1,21 @@
---
title: stack bib watch-iom
sidebar_position: 22
---
# `stack bib watch-iom`
```
Usage: stack bib watch-iom [OPTIONS]
Poll the CMS futurepdf.pdf schedule; attach if changed.
CMS publishes upcoming IOM transmittals at
``/manuals/downloads/futurepdf.pdf``. When its checksum changes,
chapters in the main manuals are probably about to update — run
``ingest-iom`` + ``attach-iom`` afterward to pick them up.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

2
docs/docs/cli/bib.md
View File

@@ -52,5 +52,7 @@ Usage: stack bib [OPTIONS] COMMAND [ARGS]...
│ optionally place it. │ │ optionally place it. │
│ cfr-jump Bidirectional CFR cite ↔ eCFR URL (#640); optionally │ │ cfr-jump Bidirectional CFR cite ↔ eCFR URL (#640); optionally │
│ place the link. │ │ place the link. │
│ code-tags Stamp code:/family: item tags from chunk anchors and │
│ FR paragraphs. │
╰──────────────────────────────────────────────────────────────────────────────╯ ╰──────────────────────────────────────────────────────────────────────────────╯
``` ```

18
docs/docs/cli/comments-dockets.md Normal file
View File

@@ -0,0 +1,18 @@
---
title: stack comments dockets
sidebar_position: 37
---
# `stack comments dockets`
```
Usage: stack comments dockets [OPTIONS]
List dockets: id, rule, close date, watermark, counts, seal.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --discover Populate rows for every reg-docket: tag in bib that has │
│ no dockets row (one reg.gov listing call each). │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

27
docs/docs/cli/comments-extract-ocr.md Normal file
View File

@@ -0,0 +1,27 @@
---
title: stack comments extract-ocr
sidebar_position: 35
---
# `stack comments extract-ocr`
```
Usage: stack comments extract-ocr [OPTIONS]
OCR pass for ``ocr_needed`` attachments (#663).
Finds comment dirs whose combined.md frontmatter flags any
attachment ``ocr_needed`` and re-extracts them with the RapidOCR
engine (rewrites combined.md + sibling transcripts, keeps the flag
on anything OCR can't clear). CPU-only, pip-only:
uv run --with rapidocr-onnxruntime stack comments extract-ocr
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --root PATH Comments root dir. [default: .state/comments] │
│ --docket TEXT Limit to one docket (e.g. CMS-2023-0121). │
│ --limit INTEGER Cap comment dirs processed this run. 0 = no cap. │
│ [default: 0] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

37
docs/docs/cli/comments-extract.md Normal file
View File

@@ -0,0 +1,37 @@
---
title: stack comments extract
sidebar_position: 34
---
# `stack comments extract`
```
Usage: stack comments extract [OPTIONS]
Walk comment dirs and write combined.md (PDF/DOCX → text).
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --root PATH Comments root dir. │
│ [default: .state/comments] │
│ --docket TEXT Limit to one docket (e.g. │
│ CMS-2023-0121). │
│ --limit INTEGER Cap dirs processed this run. 0 = no │
│ cap. │
│ [default: 0] │
│ --workers INTEGER Concurrent extraction threads. │
│ [default: 8] │
│ --force Re-extract dirs that already have │
│ combined.md. │
│ --bib --no-bib Look up inline body from bib.sqlite │
│ items.abstract. │
│ [default: bib] │
│ --attach --no-attach Attach combined.md back to its bib │
│ item (so it flows to Zotero via `bib │
│ sync`). Implies --bib; no-op when │
│ --no-bib. │
│ [default: attach] │
│ --reattach Also attach combined.md notes for dirs │
│ that were already current (repair). │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

20
docs/docs/cli/comments-seal.md Normal file
View File

@@ -0,0 +1,20 @@
---
title: stack comments seal
sidebar_position: 38
---
# `stack comments seal`
```
Usage: stack comments seal [OPTIONS] DOCKET
Mark a docket complete: every stage skips it until unsealed or --force.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ * docket TEXT reg.gov docket id [required] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --reason TEXT [default: manual] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

17
docs/docs/cli/comments-stats.md Normal file
View File

@@ -0,0 +1,17 @@
---
title: stack comments stats
sidebar_position: 36
---
# `stack comments stats`
```
Usage: stack comments stats [OPTIONS]
Rebuild the index and print per-status counts.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --root PATH Comments root dir. [default: .state/comments] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

19
docs/docs/cli/comments-unseal.md Normal file
View File

@@ -0,0 +1,19 @@
---
title: stack comments unseal
sidebar_position: 39
---
# `stack comments unseal`
```
Usage: stack comments unseal [OPTIONS] DOCKET
Reopen a sealed docket.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ * docket TEXT reg.gov docket id [required] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

25
docs/docs/cli/comments.md Normal file
View File

@@ -0,0 +1,25 @@
---
title: stack comments
sidebar_position: 33
---
# `stack comments`
```
Usage: stack comments [OPTIONS] COMMAND [ARGS]...
CMS rulemaking comment text extraction & analysis.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ extract Walk comment dirs and write combined.md (PDF/DOCX → text). │
│ extract-ocr OCR pass for ``ocr_needed`` attachments (#663). │
│ stats Rebuild the index and print per-status counts. │
│ dockets List dockets: id, rule, close date, watermark, counts, seal. │
│ seal Mark a docket complete: every stage skips it until unsealed or │
│ --force. │
│ unseal Reopen a sealed docket. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

2
docs/docs/cli/db-comment.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack db comment title: stack db comment
sidebar_position: 21 sidebar_position: 51
--- ---
# `stack db comment` # `stack db comment`

2
docs/docs/cli/db-inspect.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack db inspect title: stack db inspect
sidebar_position: 22 sidebar_position: 52
--- ---
# `stack db inspect` # `stack db inspect`

2
docs/docs/cli/db.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack db title: stack db
sidebar_position: 20 sidebar_position: 50
--- ---
# `stack db` # `stack db`

2
docs/docs/cli/docs-build.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack docs build title: stack docs build
sidebar_position: 24 sidebar_position: 54
--- ---
# `stack docs build` # `stack docs build`

2
docs/docs/cli/docs-generate.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack docs generate title: stack docs generate
sidebar_position: 26 sidebar_position: 56
--- ---
# `stack docs generate` # `stack docs generate`

2
docs/docs/cli/docs-serve.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack docs serve title: stack docs serve
sidebar_position: 25 sidebar_position: 55
--- ---
# `stack docs serve` # `stack docs serve`

2
docs/docs/cli/docs.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack docs title: stack docs
sidebar_position: 23 sidebar_position: 53
--- ---
# `stack docs` # `stack docs`

2
docs/docs/cli/lake-deploy.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack lake deploy title: stack lake deploy
sidebar_position: 16 sidebar_position: 41
--- ---
# `stack lake deploy` # `stack lake deploy`

2
docs/docs/cli/lake-load.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack lake load title: stack lake load
sidebar_position: 17 sidebar_position: 42
--- ---
# `stack lake load` # `stack lake load`

2
docs/docs/cli/lake-parity.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack lake parity title: stack lake parity
sidebar_position: 19 sidebar_position: 44
--- ---
# `stack lake parity` # `stack lake parity`

2
docs/docs/cli/lake-validate.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack lake validate title: stack lake validate
sidebar_position: 18 sidebar_position: 43
--- ---
# `stack lake validate` # `stack lake validate`

2
docs/docs/cli/lake.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack lake title: stack lake
sidebar_position: 15 sidebar_position: 40
--- ---
# `stack lake` # `stack lake`

20
docs/docs/cli/llm-hosts.md Normal file
View File

@@ -0,0 +1,20 @@
---
title: stack llm hosts
sidebar_position: 48
---
# `stack llm hosts`
```
Usage: stack llm hosts [OPTIONS]
Show the Ollama fleet: declared VRAM, liveness, models, and which host + model
would answer a chat right now.
Source .env first (`set -a; . ./.env; set +a`) so LLM_OLLAMA_HOSTS
lists the whole fleet; the compose service gets it from its environment.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

31
docs/docs/cli/llm-index.md Normal file
View File

@@ -0,0 +1,31 @@
---
title: stack llm index
sidebar_position: 46
---
# `stack llm index`
```
Usage: stack llm index [OPTIONS]
Embed comments/rules/corpus into pgvector (incremental, resumable).
Comments are processed newest-posted first; rules come from their FR
paragraph anchor maps (run `stack bib fr-grab` first for exact links);
corpus = every non-comment bib item incl. Zotero-only storage PDFs.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --collection TEXT Which collection: comments | rules | │
│ corpus | all. │
│ [default: comments] │
│ --docket TEXT Limit comments to one docket id. │
│ --key TEXT Limit rules/corpus to these item │
│ keys. │
│ --force --no-force Re-embed even when unchanged. │
│ [default: no-force] │
│ --limit INTEGER Stop after N docs per collection (0 = │
│ all). │
│ [default: 0] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

31
docs/docs/cli/llm-restamp.md Normal file
View File

@@ -0,0 +1,31 @@
---
title: stack llm restamp
sidebar_position: 47
---
# `stack llm restamp`
```
Usage: stack llm restamp [OPTIONS]
Backfill codes/families/elements onto already-indexed chunks.
Metadata-only: computes `pfs.anchors.anchor_metadata` from each row's
stored document text and merges it into `cmetadata` with a jsonb `||`,
never touching the embedding — no GPU time, no re-chunking. Safe to
re-run: rows already stamped with the current values are skipped.
F6: never rebuilds the HNSW index (`ensure_hnsw` ALTERs the vector
column and takes an ACCESS EXCLUSIVE lock for the life of the
rebuild) — only the cheap metadata GIN indexes, and not even those
under `--dry-run`.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --collection TEXT Which collection: comments | rules | corpus | │
│ all. │
│ [default: comments] │
│ --batch INTEGER Rows per transaction. [default: 2000] │
│ --dry-run Scan and report without writing. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

18
docs/docs/cli/llm-serve.md Normal file
View File

@@ -0,0 +1,18 @@
---
title: stack llm serve
sidebar_position: 49
---
# `stack llm serve`
```
Usage: stack llm serve [OPTIONS]
Serve the SSO-guarded chat UI (llm.api:app).
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --host TEXT Bind address. [default: 127.0.0.1] │
│ --port INTEGER Port. [default: 8000] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

23
docs/docs/cli/llm.md Normal file
View File

@@ -0,0 +1,23 @@
---
title: stack llm
sidebar_position: 45
---
# `stack llm`
```
Usage: stack llm [OPTIONS] COMMAND [ARGS]...
Local RAG — index, search, tag.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ index Embed comments/rules/corpus into pgvector (incremental, resumable). │
│ restamp Backfill codes/families/elements onto already-indexed chunks. │
│ hosts Show the Ollama fleet: declared VRAM, liveness, models, and which │
│ host + model would answer a chat right now. │
│ serve Serve the SSO-guarded chat UI (llm.api:app). │
╰──────────────────────────────────────────────────────────────────────────────╯
```

16
docs/docs/cli/mail-attach-smarthost.md Normal file
View File

@@ -0,0 +1,16 @@
---
title: stack mail attach-smarthost
sidebar_position: 97
---
# `stack mail attach-smarthost`
```
Usage: stack mail attach-smarthost [OPTIONS]
Register at Resend, push DNS, install smarthost on droplet.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

16
docs/docs/cli/mail-dkim-export.md Normal file
View File

@@ -0,0 +1,16 @@
---
title: stack mail dkim-export
sidebar_position: 96
---
# `stack mail dkim-export`
```
Usage: stack mail dkim-export [OPTIONS]
Publish Maddy's DKIM public key to Cloudflare. Idempotent.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

19
docs/docs/cli/mail-dns.md Normal file
View File

@@ -0,0 +1,19 @@
---
title: stack mail dns
sidebar_position: 95
---
# `stack mail dns`
```
Usage: stack mail dns [OPTIONS]
Upsert mail DNS records in Cloudflare. Idempotent.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --claim-apex Also write apex MX/SPF/DMARC. Skip when another │
│ provider (Proton, Fastmail, Workspace) already owns │
│ the apex. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

17
docs/docs/cli/mail-down.md Normal file
View File

@@ -0,0 +1,17 @@
---
title: stack mail down
sidebar_position: 93
---
# `stack mail down`
```
Usage: stack mail down [OPTIONS]
Destroy the mail droplet. Idempotent no-op if none exists.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --yes Skip confirmation. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

18
docs/docs/cli/mail-provision.md Normal file
View File

@@ -0,0 +1,18 @@
---
title: stack mail provision
sidebar_position: 91
---
# `stack mail provision`
```
Usage: stack mail provision [OPTIONS]
End-to-end idempotent: up → wait → DNS → DKIM → smarthost → git env.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --region TEXT DO region slug (one your ISP can reach). │
│ [default: nyc3] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

20
docs/docs/cli/mail-rotate-creds.md Normal file
View File

@@ -0,0 +1,20 @@
---
title: stack mail rotate-creds
sidebar_position: 98
---
# `stack mail rotate-creds`
```
Usage: stack mail rotate-creds [OPTIONS] USER
Mint a new password for *user*@fhirworx.io and push to droplet.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ * user TEXT Mailbox to rotate (e.g. git, postmaster, cmsupdates). │
│ [required] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

20
docs/docs/cli/mail-seed-mailboxes.md Normal file
View File

@@ -0,0 +1,20 @@
---
title: stack mail seed-mailboxes
sidebar_position: 99
---
# `stack mail seed-mailboxes`
```
Usage: stack mail seed-mailboxes [OPTIONS]
Ensure the standard set of mailboxes exists (postmaster, git, cmsupdates).
Idempotent: only creates missing ones; existing mailboxes keep
their cached passwords. Useful after adding a new entry to
``DEFAULT_MAILBOXES`` in mail/droplet.py.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

16
docs/docs/cli/mail-status.md Normal file
View File

@@ -0,0 +1,16 @@
---
title: stack mail status
sidebar_position: 94
---
# `stack mail status`
```
Usage: stack mail status [OPTIONS]
Show droplet state from DO + PTR check.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

17
docs/docs/cli/mail-up.md Normal file
View File

@@ -0,0 +1,17 @@
---
title: stack mail up
sidebar_position: 92
---
# `stack mail up`
```
Usage: stack mail up [OPTIONS]
Adopt-or-create the mail droplet. Idempotent.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --region TEXT DO region slug. [default: nyc3] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

19
docs/docs/cli/mail-wire-git.md Normal file
View File

@@ -0,0 +1,19 @@
---
title: stack mail wire-git
sidebar_position: 100
---
# `stack mail wire-git`
```
Usage: stack mail wire-git [OPTIONS]
Write .state/git/mailer.env from the mail droplet's credentials.
Then run ``docker compose up -d git`` to apply (a plain ``docker
restart`` won't re-read the new env_file directive in compose.yml).
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

33
docs/docs/cli/mail.md Normal file
View File

@@ -0,0 +1,33 @@
---
title: stack mail
sidebar_position: 90
---
# `stack mail`
```
Usage: stack mail [OPTIONS] COMMAND [ARGS]...
Maddy mail server (DO + Resend smarthost).
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ provision End-to-end idempotent: up → wait → DNS → DKIM → smarthost │
│ → git env. │
│ up Adopt-or-create the mail droplet. Idempotent. │
│ down Destroy the mail droplet. Idempotent no-op if none exists. │
│ status Show droplet state from DO + PTR check. │
│ dns Upsert mail DNS records in Cloudflare. Idempotent. │
│ dkim-export Publish Maddy's DKIM public key to Cloudflare. Idempotent. │
│ attach-smarthost Register at Resend, push DNS, install smarthost on │
│ droplet. │
│ rotate-creds Mint a new password for *user*@fhirworx.io and push to │
│ droplet. │
│ seed-mailboxes Ensure the standard set of mailboxes exists (postmaster, │
│ git, cmsupdates). │
│ wire-git Write .state/git/mailer.env from the mail droplet's │
│ credentials. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

19
docs/docs/cli/perf-show.md Normal file
View File

@@ -0,0 +1,19 @@
---
title: stack perf show
sidebar_position: 60
---
# `stack perf show`
```
Usage: stack perf show [OPTIONS]
Pretty-print spans from the local JSONL trace file.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --path TEXT Path to JSONL span file. │
│ [default: traces/spans.jsonl] │
│ --last INTEGER Show last N spans. [default: 20] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

19
docs/docs/cli/perf.md Normal file
View File

@@ -0,0 +1,19 @@
---
title: stack perf
sidebar_position: 59
---
# `stack perf`
```
Usage: stack perf [OPTIONS] COMMAND [ARGS]...
Pipeline telemetry utilities.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ show Pretty-print spans from the local JSONL trace file. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

2
docs/docs/cli/pfs-cpt-ingest.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack pfs cpt-ingest title: stack pfs cpt-ingest
sidebar_position: 69 sidebar_position: 72
--- ---
# `stack pfs cpt-ingest` # `stack pfs cpt-ingest`

6
docs/docs/cli/pfs-elements.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack pfs elements title: stack pfs elements
sidebar_position: 63 sidebar_position: 66
--- ---
# `stack pfs elements` # `stack pfs elements`
@@ -14,7 +14,9 @@ Usage: stack pfs elements [OPTIONS]
│ --code TEXT HCPCS/CPT code (repeatable). │ │ --code TEXT HCPCS/CPT code (repeatable). │
│ --family TEXT Expand a registered family (CCM, APCM, …); │ │ --family TEXT Expand a registered family (CCM, APCM, …); │
│ repeatable. │ │ repeatable. │
│ --all-payable Every A/R/T code in the newest RVU year. │ │ --all-payable Every A/R/T code in the newest RVU year │
│ (experimental: ~20 min of SQL before any model │
│ call). │
│ --no-llm Skip the local-model classifier (unknown lines go │ │ --no-llm Skip the local-model classifier (unknown lines go │
│ to review). │ │ to review). │
│ --dry-run Extract and report; write nothing. │ │ --dry-run Extract and report; write nothing. │

2
docs/docs/cli/pfs-families.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack pfs families title: stack pfs families
sidebar_position: 67 sidebar_position: 68
--- ---
# `stack pfs families` # `stack pfs families`

20
docs/docs/cli/pfs-guidance.md Normal file
View File

@@ -0,0 +1,20 @@
---
title: stack pfs guidance
sidebar_position: 69
---
# `stack pfs guidance`
```
Usage: stack pfs guidance [OPTIONS]
CFR/IOM/MLN sub-regulatory crosswalk per code family (pfs.code_guidance):
which CFR sections and IOM manual chapters the family's FR paragraphs and the
CPT manual cite, with FR provenance.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --family TEXT Family key (CCM, APCM, …); repeatable. │
│ --write │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

2
docs/docs/cli/pfs-lineage.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack pfs lineage title: stack pfs lineage
sidebar_position: 64 sidebar_position: 67
--- ---
# `stack pfs lineage` # `stack pfs lineage`

30
docs/docs/cli/pfs-reaction.md Normal file
View File

@@ -0,0 +1,30 @@
---
title: stack pfs reaction
sidebar_position: 70
---
# `stack pfs reaction`
```
Usage: stack pfs reaction [OPTIONS]
Public-reaction series per code family (pfs.code_reaction): comment counts per
docket (share of the docket's commenters mentioning the family), an optional
stance sample, and the FR Comment:/Response: pair counts per rule as the
pre-2017 proxy. Always recomputed live from pgvector/bib — there is no
persisted-table fallback to read without --write, so --write only adds
persisting the result.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --family TEXT Family key (CCM, APCM, …); repeatable. │
│ --code TEXT A single code as its own pseudo-family, │
│ keyed by the code; repeatable. │
│ --stance-sample INTEGER Classify the newest N commenters per docket │
│ into support/oppose/modify/unclear via the │
│ self-hosted classifier (0 = skip; no stance │
│ sample means no LLM call at all). │
│ [default: 0] │
│ --write │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

2
docs/docs/cli/pfs-review.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack pfs review title: stack pfs review
sidebar_position: 66 sidebar_position: 71
--- ---
# `stack pfs review` # `stack pfs review`

17
docs/docs/cli/pfs.md
View File

@@ -1,6 +1,6 @@
--- ---
title: stack pfs title: stack pfs
sidebar_position: 64 sidebar_position: 65
--- ---
# `stack pfs` # `stack pfs`
@@ -20,6 +20,21 @@ Usage: stack pfs [OPTIONS] COMMAND [ARGS]...
│ cross-checked. │ │ cross-checked. │
│ families Derive families from pfs.code_element / pfs.code_event / │ │ families Derive families from pfs.code_element / pfs.code_event / │
│ pfs.rvu. │ │ pfs.rvu. │
│ guidance CFR/IOM/MLN sub-regulatory crosswalk per code family │
│ (pfs.code_guidance): which CFR sections and IOM manual chapters │
│ the │
│ family's FR paragraphs and the CPT manual cite, with FR │
│ provenance. │
│ reaction Public-reaction series per code family (pfs.code_reaction): │
│ comment │
│ counts per docket (share of the docket's commenters mentioning │
│ the │
│ family), an optional stance sample, and the FR │
│ Comment:/Response: │
│ pair counts per rule as the pre-2017 proxy. Always recomputed │
│ live │
│ from pgvector/bib — there is no persisted-table fallback to read │
│ without --write, so --write only adds persisting the result. │
│ review Element lines the classifier could not place. │ │ review Element lines the classifier could not place. │
│ cpt-ingest Parse CPT EPUB editions into │ │ cpt-ingest Parse CPT EPUB editions into │
│ pfs.cpt_section/cpt_code/cpt_instruction/ │ │ pfs.cpt_section/cpt_code/cpt_instruction/ │

29
docs/docs/cli/prisma-eligible.md Normal file
View File

@@ -0,0 +1,29 @@
---
title: stack prisma eligible
sidebar_position: 84
---
# `stack prisma eligible`
```
Usage: stack prisma eligible [OPTIONS] NAME
Stage 3 — LLM-driven full-text eligibility pass.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ * name TEXT Project slug. [required] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --limit -n INTEGER [default: 0] │
│ --max-chars INTEGER Cap per-item markdown (full text) at │
│ N chars so bounded-context local │
│ models keep the criteria in window. 0 │
│ = no cap. │
│ [default: 0] │
│ --db PATH [default: │
│ data/zotero/data/zotero.sqlite] │
│ --storage PATH [default: data/zotero/data/storage] │
│ --hold --no-hold [default: hold] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

23
docs/docs/cli/prisma-export.md Normal file
View File

@@ -0,0 +1,23 @@
---
title: stack prisma export
sidebar_position: 81
---
# `stack prisma export`
```
Usage: stack prisma export [OPTIONS] ZOT_ID
Dump one Zotero item as the LLM-ready markdown snapshot.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ * zot_id INTEGER Zotero itemID to export. [required] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --fulltext │
│ --db PATH [default: data/zotero/data/zotero.sqlite] │
│ --storage PATH [default: data/zotero/data/storage] │
│ --hold --no-hold [default: hold] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

24
docs/docs/cli/prisma-extract.md Normal file
View File

@@ -0,0 +1,24 @@
---
title: stack prisma extract
sidebar_position: 85
---
# `stack prisma extract`
```
Usage: stack prisma extract [OPTIONS] NAME
Stage 3+ — structured data extraction on included studies.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ * name TEXT Project slug. [required] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --limit -n INTEGER [default: 0] │
│ --db PATH [default: │
│ data/zotero/data/zotero.sqlite] │
│ --storage PATH [default: data/zotero/data/storage] │
│ --hold --no-hold [default: hold] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

47
docs/docs/cli/prisma-fetch.md Normal file
View File

@@ -0,0 +1,47 @@
---
title: stack prisma fetch
sidebar_position: 87
---
# `stack prisma fetch`
```
Usage: stack prisma fetch [OPTIONS] NAME
Fetch PDFs for every non-excluded item lacking an attachment.
Source cascade: Unpaywall → PMC → fallback (via the VPN droplet).
Items excluded at stage 2 are untouched — the queue filter only
passes `screen:include` and `screen:uncertain` through.
Tunnel lifecycle is automatic: if a droplet is tracked in
``.state/prisma-vpn/`` this command opens an SSH port-forward for
the duration of the fetch and closes it on exit. No sudo required.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ * name TEXT Project slug. [required] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --limit -n INTEGER [default: 0] │
│ --workers -w INTEGER Concurrent fetch workers (fallback │
│ tier stays serialized). │
│ [default: 8] │
│ --db PATH [default: │
│ data/zotero/data/zotero.sqlite] │
│ --storage PATH [default: data/zotero/data/storage] │
│ --scratch PATH [default: .state/prisma-fetch] │
│ --proxy TEXT Override PRISMA_FETCH_PROXY (else │
│ env). │
│ --vpn --no-vpn Auto-open an SSH tunnel to the │
│ provisioned droplet when one exists. │
│ --no-vpn skips the tunnel and uses │
│ only the unproxied sources. │
│ [default: vpn] │
│ --verify --no-verify Curl through the proxy once before │
│ fetching and confirm egress is │
│ non-US. │
│ [default: verify] │
│ --hold --no-hold [default: hold] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

22
docs/docs/cli/prisma-flow.md Normal file
View File

@@ -0,0 +1,22 @@
---
title: stack prisma flow
sidebar_position: 86
---
# `stack prisma flow`
```
Usage: stack prisma flow [OPTIONS] NAME
Emit the PRISMA 2020 flow diagram from current tag counts.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ * name TEXT Project slug. [required] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --mermaid --text [default: mermaid] │
│ --db PATH [default: data/zotero/data/zotero.sqlite] │
│ --hold --no-hold [default: hold] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

23
docs/docs/cli/prisma-init.md Normal file
View File

@@ -0,0 +1,23 @@
---
title: stack prisma init
sidebar_position: 80
---
# `stack prisma init`
```
Usage: stack prisma init [OPTIONS] NAME
Create (or re-load) the project's three Zotero anchor items: criteria,
extraction-template, and the reasons codebook.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ * name TEXT Project slug, e.g. skin-subs. [required] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --db PATH Zotero SQLite path. │
│ [default: data/zotero/data/zotero.sqlite] │
│ --hold --no-hold [default: hold] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

19
docs/docs/cli/prisma-ping-llm.md Normal file
View File

@@ -0,0 +1,19 @@
---
title: stack prisma ping-llm
sidebar_position: 82
---
# `stack prisma ping-llm`
```
Usage: stack prisma ping-llm [OPTIONS]
Send a cheap ping through the configured provider.
Useful for verifying ``PRISMA_LLM_PROVIDER`` + auth before kicking
off a long screening run.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

32
docs/docs/cli/prisma-run.md Normal file
View File

@@ -0,0 +1,32 @@
---
title: stack prisma run
sidebar_position: 88
---
# `stack prisma run`
```
Usage: stack prisma run [OPTIONS] NAME
Orchestrate all PRISMA stages in order.
screen → fetch → eligible → extract → flow, all inside one zotero
hold. Fetch only pulls PDFs for non-excluded items; stage 2
exclusions are terminal by design.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ * name TEXT Project slug. [required] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --limit -n INTEGER Same limit applied to every stage. │
│ [default: 0] │
│ --skip-fetch │
│ --skip-extract │
│ --db PATH [default: │
│ data/zotero/data/zotero.sqlite] │
│ --storage PATH [default: │
│ data/zotero/data/storage] │
│ --hold --no-hold [default: hold] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

34
docs/docs/cli/prisma-screen.md Normal file
View File

@@ -0,0 +1,34 @@
---
title: stack prisma screen
sidebar_position: 83
---
# `stack prisma screen`
```
Usage: stack prisma screen [OPTIONS] NAME
Stage 2 — LLM-driven title/abstract screening pass.
╭─ Arguments ──────────────────────────────────────────────────────────────────╮
│ * name TEXT Project slug. [required] │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --limit -n INTEGER 0 = screen every unscreened item. │
│ [default: 0] │
│ --db PATH [default: │
│ data/zotero/data/zotero.sqlite] │
│ --storage PATH [default: data/zotero/data/storage] │
│ --hold --no-hold [default: hold] │
│ --hosts TEXT Comma-separated Ollama base URLs — │
│ fleet mode (#655): parallel │
│ screening via native /api/chat with │
│ per-request num_ctx; bypasses │
│ PRISMA_LLM_* env config. │
│ --fleet-model TEXT Model name on every fleet host. │
│ [default: qwen2.5:14b] │
│ --num-ctx INTEGER Context window per fleet request. │
│ [default: 8192] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

24
docs/docs/cli/prisma-vpn.md Normal file
View File

@@ -0,0 +1,24 @@
---
title: stack prisma vpn
sidebar_position: 89
---
# `stack prisma vpn`
```
Usage: stack prisma vpn [OPTIONS] COMMAND [ARGS]...
DigitalOcean SOCKS5 exit for fallback fetch (SSH tunnel, no sudo needed).
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ up Provision the droplet, wait for dante, wire Zotero up. │
│ down Destroy the droplet and wipe local state. │
│ status Report whether a PRISMA VPN droplet is currently tracked. │
│ attach-zotero Start the sidecar tunnel + rewire Zotero prefs. │
│ detach-zotero Stop the sidecar and restore Zotero's pre-proxy prefs. │
│ verify Open the tunnel, curl ifconfig.co, print egress, close. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

30
docs/docs/cli/prisma.md Normal file
View File

@@ -0,0 +1,30 @@
---
title: stack prisma
sidebar_position: 79
---
# `stack prisma`
```
Usage: stack prisma [OPTIONS] COMMAND [ARGS]...
LLM-driven PRISMA screening workflow.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ init Create (or re-load) the project's three Zotero anchor items: │
│ criteria, extraction-template, and the reasons codebook. │
│ export Dump one Zotero item as the LLM-ready markdown snapshot. │
│ ping-llm Send a cheap ping through the configured provider. │
│ screen Stage 2 — LLM-driven title/abstract screening pass. │
│ eligible Stage 3 — LLM-driven full-text eligibility pass. │
│ extract Stage 3+ — structured data extraction on included studies. │
│ flow Emit the PRISMA 2020 flow diagram from current tag counts. │
│ fetch Fetch PDFs for every non-excluded item lacking an attachment. │
│ run Orchestrate all PRISMA stages in order. │
│ vpn DigitalOcean SOCKS5 exit for fallback fetch (SSH tunnel, no sudo │
│ needed). │
╰──────────────────────────────────────────────────────────────────────────────╯
```

16
docs/docs/cli/rec-list.md Normal file
View File

@@ -0,0 +1,16 @@
---
title: stack rec list
sidebar_position: 62
---
# `stack rec list`
```
Usage: stack rec list [OPTIONS]
List registered pricers and their compare columns.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

21
docs/docs/cli/rec-opps.md Normal file
View File

@@ -0,0 +1,21 @@
---
title: stack rec opps
sidebar_position: 64
---
# `stack rec opps`
```
Usage: stack rec opps [OPTIONS]
Reconcile OPPS calculated rates against Addendum B.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --year -y INTEGER Year to reconcile; omit for all. │
│ --format -f TEXT Output format: md | json. [default: md] │
│ --output -o PATH Write to file instead of stdout. │
│ --tolerance -t INTEGER Cents of tolerance for an 'exact' match. │
│ [default: 0] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

21
docs/docs/cli/rec-pfs.md Normal file
View File

@@ -0,0 +1,21 @@
---
title: stack rec pfs
sidebar_position: 63
---
# `stack rec pfs`
```
Usage: stack rec pfs [OPTIONS]
Reconcile PFS calculated fees against carrier_locality.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --year -y INTEGER Year to reconcile; omit for all. │
│ --format -f TEXT Output format: md | json. [default: md] │
│ --output -o PATH Write to file instead of stdout. │
│ --tolerance -t INTEGER Cents of tolerance for an 'exact' match. │
│ [default: 0] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

21
docs/docs/cli/rec.md Normal file
View File

@@ -0,0 +1,21 @@
---
title: stack rec
sidebar_position: 61
---
# `stack rec`
```
Usage: stack rec [OPTIONS] COMMAND [ARGS]...
Reconcile calculated payments against CMS ground-truth files.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ list List registered pricers and their compare columns. │
│ pfs Reconcile PFS calculated fees against carrier_locality. │
│ opps Reconcile OPPS calculated rates against Addendum B. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

6
docs/docs/cli/run.md
View File

@@ -14,8 +14,10 @@ Usage: stack run [OPTIONS] NAME
│ * name TEXT Pipeline name (e.g. readmissions). [required] │ │ * name TEXT Pipeline name (e.g. readmissions). [required] │
╰──────────────────────────────────────────────────────────────────────────────╯ ╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────╮ ╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --target TEXT Target context: local, lake, trino, databricks. │ │ --target TEXT Target context: local, lake, trino, databricks, │
│ Defaults to stack.toml. │ │ spark. │
│ --catalog TEXT Unity Catalog name (overrides config). Implies │
│ target=spark. │
│ --save Save pipeline outputs back to the database. │ │ --save Save pipeline outputs back to the database. │
│ --help Show this message and exit. │ │ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯ ╰──────────────────────────────────────────────────────────────────────────────╯

23
docs/docs/cli/zot-dump-schema.md Normal file
View File

@@ -0,0 +1,23 @@
---
title: stack zot dump-schema
sidebar_position: 74
---
# `stack zot dump-schema`
```
Usage: stack zot dump-schema [OPTIONS]
Dump itemType / field / creatorType IDs.
Regenerate after a Zotero upgrade: the output replaces TYPE_MAP,
FIELD_MAP, and CREATOR_TYPES in ``src/zot/db.py`` verbatim.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --db PATH Default: data/zotero/data/zotero.sqlite │
│ --python --json Emit paste-ready Python literals for zot/db.py │
│ (default) or JSON. │
│ [default: python] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

18
docs/docs/cli/zot-fix-dates.md Normal file
View File

@@ -0,0 +1,18 @@
---
title: stack zot fix-dates
sidebar_position: 75
---
# `stack zot fix-dates`
```
Usage: stack zot fix-dates [OPTIONS]
Normalize item timestamps to ISO 8601 + install auto-fix triggers.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --db PATH │
│ --hold --no-hold [default: hold] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

18
docs/docs/cli/zot-fix-fields.md Normal file
View File

@@ -0,0 +1,18 @@
---
title: stack zot fix-fields
sidebar_position: 77
---
# `stack zot fix-fields`
```
Usage: stack zot fix-fields [OPTIONS]
Remap base-field rows to type-specific fieldIDs; drop orphans.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --db PATH │
│ --hold --no-hold [default: hold] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

19
docs/docs/cli/zot-fix-keys.md Normal file
View File

@@ -0,0 +1,19 @@
---
title: stack zot fix-keys
sidebar_position: 76
---
# `stack zot fix-keys`
```
Usage: stack zot fix-keys [OPTIONS]
Replace invalid Zotero object keys (0/1/O/lowercase → valid).
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --db PATH │
│ --hold --no-hold [default: hold] │
│ --backup --no-backup [default: backup] │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

21
docs/docs/cli/zot-verify-parity.md Normal file
View File

@@ -0,0 +1,21 @@
---
title: stack zot verify-parity
sidebar_position: 78
---
# `stack zot verify-parity`
```
Usage: stack zot verify-parity [OPTIONS]
Check that ``zot.db`` constants match the live Zotero schema.
The check runs implicitly on every ``Db(...)`` construction; this
command just exposes it explicitly so CI can fail loudly if someone
bumps Zotero without regenerating the maps.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --db PATH │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

25
docs/docs/cli/zot.md Normal file
View File

@@ -0,0 +1,25 @@
---
title: stack zot
sidebar_position: 73
---
# `stack zot`
```
Usage: stack zot [OPTIONS] COMMAND [ARGS]...
Zotero database maintenance.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ dump-schema Dump itemType / field / creatorType IDs. │
│ fix-dates Normalize item timestamps to ISO 8601 + install auto-fix │
│ triggers. │
│ fix-keys Replace invalid Zotero object keys (0/1/O/lowercase → valid). │
│ fix-fields Remap base-field rows to type-specific fieldIDs; drop │
│ orphans. │
│ verify-parity Check that ``zot.db`` constants match the live Zotero schema. │
╰──────────────────────────────────────────────────────────────────────────────╯
```

324
notebooks/code_families.py
View File

@@ -565,40 +565,334 @@ def _(code, mo, not_built, pl, plain_years, q):
@app.cell(hide_code=True) @app.cell(hide_code=True)
def _(mo, not_built): def _(code, mo, not_built, pl, store):
# ── 5. Anchors in the corpus ── # ── 5. Anchors in the corpus ──
mo.md( import os as _os
from pfs.families import family_of as _family_of
_fam = _family_of(code)
_key = (_fam.key if _fam else code).upper()
_fam_codes = set(_fam.codes) if _fam else {code}
_intro = mo.md(
"## 5. Anchors in the corpus\n\n" "## 5. Anchors in the corpus\n\n"
"How many chunks in the RAG index — comments, guidance, Federal Register text — carry " "How many chunks in the RAG index — comments, guidance, Federal Register "
"this family's codes as metadata, and how many bibliography items carry a `family:` or " "text — carry this family's codes as metadata, and how many bibliography "
"`code:` tag. That count lives in pgvector, not the DuckDB replica this notebook reads.\n\n" "items carry a `family:` or `code:` tag (`stack bib code-tags`). Chunk "
+ not_built("stack llm restamp") f"counts live in pgvector (**{_key}**, family-membership match), not the "
"DuckDB replica this notebook otherwise reads; item tags live in the "
"bibliography SQLite."
) )
_panels = [_intro]
if not _os.environ.get("LLM_DB_PASSWORD"):
_panels.append(
mo.md(
"_`LLM_DB_PASSWORD` not set — pgvector is unreachable from this "
"process, so chunk-per-collection counts are skipped; item-tag "
"counts from the bibliography follow instead._"
)
)
else:
try:
from sqlalchemy import text as _sa_text
from llm.config import load as _load_llm_cfg
from llm.index import _engine as _llm_engine
_eng = _llm_engine(_load_llm_cfg())
with _eng.begin() as _conn:
_chunk_rows = _conn.execute(
_sa_text(
"SELECT c.name, count(*) AS n "
"FROM langchain_pg_embedding e "
"JOIN langchain_pg_collection c ON c.uuid = e.collection_id "
"WHERE string_to_array(COALESCE(e.cmetadata->>'families', ''), "
"' ') && ARRAY[:key] "
"GROUP BY c.name ORDER BY c.name"
),
{"key": _key},
).fetchall()
except Exception as e: # noqa: BLE001 — degrade, never crash the page
_panels.append(mo.md(f"_pgvector unavailable: {e}_"))
else:
_chunks = pl.DataFrame(
{
"collection": [r[0] for r in _chunk_rows],
"chunks": [r[1] for r in _chunk_rows],
}
)
_panels.append(
mo.ui.table(_chunks, label=f"pgvector chunks tagged families ∋ {_key}")
if not _chunks.is_empty()
else mo.md(f"_No pgvector chunks tagged `families ∋ {_key}` yet._")
)
if store is None:
_panels.append(mo.md("_bibliography unavailable — item-tag counts omitted._"))
else:
try:
_fam_tags = pl.DataFrame(store.list_tags(namespace="family"))
_code_tags = pl.DataFrame(
[
row
for row in store.list_tags(namespace="code")
if row["name"].removeprefix("code:") in _fam_codes
]
)
except Exception as e: # noqa: BLE001
_panels.append(mo.md(f"_item-tag counts unavailable: {e}_"))
else:
_panels.append(
mo.ui.table(_fam_tags, label="bib item tags — family:*")
if not _fam_tags.is_empty()
else mo.md(not_built("stack bib code-tags"))
)
_panels.append(
mo.ui.table(_code_tags, label=f"bib item tags — code:* for {_key}")
if not _code_tags.is_empty()
else mo.md(f"_No `code:` tags for {_key}'s member codes yet._")
)
mo.vstack(_panels)
return return
@app.cell(hide_code=True) @app.cell(hide_code=True)
def _(mo, not_built): def _(code, con, fr_md, mo, not_built, pl, store):
# ── 6. Guidance ── # ── 6. Guidance ──
from bib.cfrlink import md_link as _md_link
from pfs.codetables import read_guidance as _read_guidance
from pfs.families import family_of as _family_of
_fam = _family_of(code)
_key = _fam.key if _fam else code
_rows = []
if con is not None:
try:
_rows = _read_guidance(con, _key)
except Exception: # noqa: BLE001 — a missing table is "not built yet"
_rows = []
if not _rows:
_view = mo.md(
"## 6. Guidance\n\n"
"Sub-regulatory guidance — Medicare Learning Network articles, "
"Internet-Only Manual sections, MACs' local coverage determinations — "
"that cites a family's codes, with CFR cross-references where the "
"guidance implements a rule.\n\n"
+ not_built(f"stack pfs guidance --family {_key} --write")
)
else:
def _citation(r):
# CFR: a live eCFR link built straight from the stored locator.
if r.kind == "cfr":
try:
return _md_link(r.locator)
except ValueError:
return r.locator
# IOM: the manual chapter's own bib title when the citation
# resolved to a library item, else fall back to the locator.
if r.kind == "iom" and r.item_key and store is not None:
try:
title = store.get(r.item_key).title
if title:
return title
except KeyError:
pass
return r.locator # iom (unresolved) and mln both show the locator
def _provenance(r):
# CPT-manual guideline citations carry no FR paragraph
# (p_id_src=0, page_src=0 — module docstring); everything else
# anchors to the exact FR paragraph that named the code.
if r.p_id_src:
return fr_md(r.item_key_src, r.p_id_src)
return f"{r.item_key_src} (CPT manual)" if r.item_key_src else ""
_tbl = pl.DataFrame(
{
"code": [r.code for r in _rows],
"kind": [r.kind for r in _rows],
"citation": [_citation(r) for r in _rows],
"cited from": [_provenance(r) for r in _rows],
}
)
_n_cfr = sum(1 for r in _rows if r.kind == "cfr")
_n_iom = sum(1 for r in _rows if r.kind == "iom")
_n_mln = sum(1 for r in _rows if r.kind == "mln")
_view = mo.vstack(
[
mo.md( mo.md(
"## 6. Guidance\n\n" "## 6. Guidance\n\n"
"Sub-regulatory guidance — Medicare Learning Network articles, Internet-Only Manual " "Sub-regulatory guidance — Medicare Learning Network articles, "
"sections, MACs' local coverage determinations — that cites a family's codes, with CFR " "Internet-Only Manual sections, MACs' local coverage "
"cross-references where the guidance implements a rule.\n\n" "determinations — that cites a family's codes, with CFR "
+ not_built("stack pfs guidance --family CCM --write") "cross-references where the guidance implements a rule. Every "
"row is a reference actually found in text that also names one "
"of the family's codes, resolved against the bibliography where "
"possible and always anchored to where it was cited. "
f"**{_key}**: {_n_cfr} CFR, {_n_iom} IOM, {_n_mln} MLN "
f"reference{'s' if len(_rows) != 1 else ''}."
),
mo.ui.table(
_tbl, label=f"pfs.code_guidance — {_key} ({len(_rows)} rows)"
),
]
) )
_view
return return
@app.cell(hide_code=True) @app.cell(hide_code=True)
def _(mo, not_built): def _(alt, code, con, mo, not_built, pl):
# ── 7. Reaction ── # ── 7. Reaction ──
from pfs.codetables import read_reaction as _read_reaction
from pfs.families import family_of as _family_of
_fam = _family_of(code)
_key = _fam.key if _fam else code
_rows = []
if con is not None:
try:
_rows = _read_reaction(con, _key)
except Exception: # noqa: BLE001 — a missing table is "not built yet"
_rows = []
if not _rows:
_view = mo.md(
"## 7. Reaction\n\n"
"Public comment volume on a family's codes over time — how many "
"comment letters mention them, in which rule years, and (on a sample) "
"whether commenters supported or opposed the proposal.\n\n"
+ not_built(f"stack pfs reaction --family {_key} --write")
)
else:
_dockets = sorted(
(r for r in _rows if r.period_kind == "docket"),
key=lambda r: (r.year, r.period),
)
_fr_rows = sorted(
(r for r in _rows if r.period_kind == "fr-pairs"), key=lambda r: r.year
)
_has_stance = any(
r.stance_support or r.stance_oppose or r.stance_modify or r.stance_unclear
for r in _dockets
)
_panels = [
mo.md( mo.md(
"## 7. Reaction\n\n" "## 7. Reaction\n\n"
"Public comment volume on a family's codes over time — how many comment letters mention " "Two independent proxies for public reaction, both counted from "
"them, in which rule years, and (on a sample) whether commenters supported or opposed the " "chunk/paragraph metadata rather than hand-read. **Dockets** "
"proposal.\n\n" + not_built("stack pfs reaction --family CCM --write") "(regulations.gov, 2017 on) count the distinct commenters "
"(`item_key`) whose comment text names one of the family's codes "
"— `n_items` out of the docket's `n_total` distinct commenters, "
"read straight from the `comments` pgvector collection; a stance "
"breakdown (support/oppose/modify/unclear) is filled in only for "
"dockets built with `--stance-sample N`, which classifies the "
"newest *N* matching commenters' first chunk with the self-hosted "
"closed-vocabulary classifier — an answer outside the four "
"stances counts as `unclear` — so it is a sample, not a census. "
"**FR pairs** (back to 2001) count `Comment:`/`Response:` "
"paragraph pairs CMS itself printed in a rule's preamble that "
"name a family code — the pre-2017 proxy, since regulations.gov "
"comment text isn't indexed that far back; `n_items` is the "
"qualifying-pair count and `n_total` the matching `Response:` "
"paragraphs (always equal to `n_items`, since a pair requires "
"both)."
+ (
""
if _has_stance
else " No docket in this family was built with "
"`--stance-sample`, so the stance columns below are all zero."
) )
)
]
if _dockets:
_dk = pl.DataFrame(
{
"label": [f"{r.year} {r.period}" for r in _dockets],
"n_items": [r.n_items for r in _dockets],
"n_total": [r.n_total for r in _dockets],
"support": [r.stance_support for r in _dockets],
"oppose": [r.stance_oppose for r in _dockets],
"modify": [r.stance_modify for r in _dockets],
"unclear": [r.stance_unclear for r in _dockets],
}
)
_chart1 = (
alt.Chart(_dk.to_pandas())
.mark_bar()
.encode(
x=alt.X(
"label:N", sort=_dk["label"].to_list(), title="Docket (year)"
),
y=alt.Y("n_items:Q", title="Commenters naming the family"),
tooltip=[
"label",
"n_items",
"n_total",
"support",
"oppose",
"modify",
"unclear",
],
)
.properties(height=240, width=640)
)
_panels.append(mo.ui.altair_chart(_chart1))
else:
_panels.append(mo.md("_No docket has comment chunks for this family._"))
if _fr_rows:
_fp = pl.DataFrame(
{
"year": [r.year for r in _fr_rows],
"item_key": [r.period for r in _fr_rows],
"n_items": [r.n_items for r in _fr_rows],
}
)
_chart2 = (
alt.Chart(_fp.to_pandas())
.mark_bar()
.encode(
x=alt.X("year:O", title="Rule year"),
y=alt.Y("n_items:Q", title="Comment/Response pairs"),
tooltip=["year", "item_key", "n_items"],
)
.properties(height=220, width=640)
)
_panels.append(mo.ui.altair_chart(_chart2))
else:
_panels.append(
mo.md("_No pre-2017 Comment:/Response: pairs found for this family._")
)
_tbl = pl.DataFrame(
{
"period_kind": [r.period_kind for r in _rows],
"period": [r.period for r in _rows],
"year": [r.year for r in _rows],
"n_items": [r.n_items for r in _rows],
"n_total": [r.n_total for r in _rows],
"support": [r.stance_support for r in _rows],
"oppose": [r.stance_oppose for r in _rows],
"modify": [r.stance_modify for r in _rows],
"unclear": [r.stance_unclear for r in _rows],
}
)
_panels.append(
mo.ui.table(_tbl, label=f"pfs.code_reaction — {_key} ({len(_rows)} rows)")
)
_view = mo.vstack(_panels)
_view
return return

14
tests/notebooks/test_code_families_nb.py
View File

@@ -47,3 +47,17 @@ def test_headless_run_degrades_without_data(monkeypatch, tmp_path):
) )
result = mod.app.run() # marimo runs all cells; mo.stop cascades are fine result = mod.app.run() # marimo runs all cells; mo.stop cascades are fine
assert result is not None assert result is not None
def test_section5_tolerates_missing_llm_db_password(monkeypatch):
"""Section 5's pgvector chunk-count lookup must degrade to a note,
never raise, when LLM_DB_PASSWORD is unset — the bib item-tag counts
still render (task-8-context.md: "the section-5 cell must render the
bib tag counts with a note when it is missing, and never raise")."""
monkeypatch.delenv("LLM_DB_PASSWORD", raising=False)
mod = _load()
outputs, _defs = mod.app.run() # real replica/bib — data symlinks are in place
rendered = "\n".join(o._repr_html_() for o in outputs if hasattr(o, "_repr_html_"))
assert "LLM_DB_PASSWORD" in rendered and "not set" in rendered
assert "pgvector is unreachable" in rendered
assert "family:" in rendered