bib: ingest Medicare Benefits Policy Manual (IOM Pub 100-02) into library #304

New Issue

kert · 2026-03-27T04:46:57Z

kert commented

2026-03-27 04:46:57 +00:00

The Medicare Benefits Policy Manual (CMS Pub 100-02) defines what Medicare Part A and Part B cover. It is the authoritative source for coverage determinations.

Chapters needed:

Ch. 1: Inpatient Hospital Services
Ch. 6: Hospital Services Covered Under Part B
Ch. 15: Covered Medical and Other Health Services (skin subs, wound care)
Ch. 16: General Exclusions from Coverage

Source: https://www.cms.gov/regulations-and-guidance/guidance/manuals/internet-only-manuals-ioms-items/cms012673

Each chapter is a separate PDF. Create Manual items in bib.sqlite with pub_number=100-02, chapter=N, attach the PDFs.

The Medicare Benefits Policy Manual (CMS Pub 100-02) defines what Medicare Part A and Part B cover. It is the authoritative source for coverage determinations. Chapters needed: - Ch. 1: Inpatient Hospital Services - Ch. 6: Hospital Services Covered Under Part B - Ch. 15: Covered Medical and Other Health Services (skin subs, wound care) - Ch. 16: General Exclusions from Coverage Source: https://www.cms.gov/regulations-and-guidance/guidance/manuals/internet-only-manuals-ioms-items/cms012673 Each chapter is a separate PDF. Create Manual items in bib.sqlite with pub_number=100-02, chapter=N, attach the PDFs.

kert added this to the P32: CMS Manual Library — benefits, fraud, and program integrity references milestone 2026-03-27 04:46:57 +00:00

kert commented

2026-04-15 04:23:44 +00:00

Closed via #343-tracked sync pipeline. 100-02 (Benefit Policy Manual): 16 chapter PDFs ingested, attached, and synced to Zotero under Healthcare Data Platform / Manuals.

Architecture

Landed a full CMS IOM pipeline — not just this manual, every IOM CMS publishes (20 pubs, 177 chapters total).

Modules

src/bib/iom.py (new) — scraper + orchestrator.
- fetch_index(client) scrapes the CMS IOM index table.
- fetch_chapters(client, entry) scrapes each manual's landing page. Handles single-file manuals (100-17/18/19/20/21) via _WHOLE_PUB regex fallback; handles split chapters ("Chapter 1 Part 2") via _CHAPTER_PART fold into composite IDs like 1P2; skips supplementals (crosswalks, addenda).
- ingest_all(store, client, *, pubs=None, prune=True) — idempotent: re-upserts by URL and prunes Manual records whose URL is no longer in the live CMS list, cleaning up stale filename-rename dupes.
- download_attachments(store, client) — SHA-256-keyed; skips files whose on-disk checksum matches the stored copy.
- check_future_updates(store, client) — polls futurepdf.pdf (CMS upcoming-transmittal schedule); attaches only when checksum changes; designed as the trigger signal for out-of-band refresh cycles.
src/bib/sync.py — extended with attachment sync + collection routing.
- push_to_zotero(items, store=..., ...) now walks store.attachments and copies PDFs into Zotero's data/zotero/data/storage/<key>/ + creates itemAttachments rows with linkMode=1 (IMPORTED_URL).
- _zotero_collection_path(item) derives the target hierarchy; Manual items route to Healthcare Data Platform / Manuals / <manual_name>, ensured via Db.ensure_collection on the fly.
- replace_fields(item_id, fields) — wipes existing itemData before rewriting to clean stale garbage from prior buggy syncs.
src/cli/bib.py — new commands, all idempotent:
- stack bib ingest-iom [--pub X]
- stack bib attach-iom [--pub X] [--force]
- stack bib watch-iom
- stack bib sync-zotero [--tag X] [--type manual] (wraps docker stop/start zotero)
- stack bib refresh-iom — one-shot: ingest → watch → attach → sync. Safe to cron (follow-up #343).

Schema parity fix

Root cause of earlier Zotero corruption was stale TYPE_MAP / FIELD_MAP / CREATOR_TYPES constants in src/zot/db.py authored against an older Zotero schema. Modern Zotero 7 assigns different numeric IDs, so manual → 15 landed as "email" instead of "report" and attachments as "document" instead of "attachment", tripping Zotero's getFilePath() consistency checks.

Constants rewritten from the live Zotero 7 DB.
_verify_schema_parity() runs on every Db.__init__ and raises immediately if IDs drift — future Zotero upgrades can't silently corrupt data.
src/zot/schema.sql replaced with a fresh dump (2,499 lines) — the old file had triggers like itemTypeID != 14 for attachments that were valid only in the pre-7 schema.
New src/zot/ops.py packages dump_schema, fix_dates, fix_keys, fix_fields as importable + CLI-exposed (stack zot {…}). Old dev/scripts/*.py became thin shims that call into the module.

Data flow

CMS IOM index (HTML)
  │  fetch_index
  ▼
IOMEntry[] (pub, title, landing_url)
  │  fetch_chapters per entry
  ▼
ChapterEntry[] (pub, manual, chapter, title, url)
  │  cms_manual() + store.upsert()
  ▼
bib.sqlite items (item_type=manual)
  │  download_attachments → data/storage/<key>/<file>.pdf
  ▼
bib.sqlite attachments
  │  push_to_zotero(store=...)
  ▼
Zotero: report items + attachment children + collection memberships

Verified clean state

177 Manual records across 20 pubs in bib.sqlite
177 PDFs attached (one per chapter)
177 Zotero items typed report, 177 attachment children typed attachment (linkMode=1)
PRAGMA integrity_check = ok, PRAGMA foreign_key_check = ok
All (itemTypeID, fieldID) pairs valid per itemTypeFieldsCombined

Gotchas recorded for future agents

Carve-outs in .dockerignore are required (!infra/marimo/theme/, !infra/gitea/) or the build context lacks theme files.
Zotero holds an exclusive SQLite lock while running; every command that writes to the DB calls docker stop zotero / start zotero automatically (--hold default on).
The filename column in bib.attachments is a display label, not a filesystem name. Use Path(storage_path).name when copying into Zotero storage.

**Closed via #343-tracked sync pipeline.** 100-02 (Benefit Policy Manual): 16 chapter PDFs ingested, attached, and synced to Zotero under `Healthcare Data Platform / Manuals`. ## Architecture Landed a full CMS IOM pipeline — not just this manual, every IOM CMS publishes (20 pubs, 177 chapters total). ### Modules - **`src/bib/iom.py`** (new) — scraper + orchestrator. - `fetch_index(client)` scrapes the CMS IOM index table. - `fetch_chapters(client, entry)` scrapes each manual's landing page. Handles single-file manuals (100-17/18/19/20/21) via `_WHOLE_PUB` regex fallback; handles split chapters ("Chapter 1 Part 2") via `_CHAPTER_PART` fold into composite IDs like `1P2`; skips supplementals (crosswalks, addenda). - `ingest_all(store, client, *, pubs=None, prune=True)` — idempotent: re-upserts by URL and prunes Manual records whose URL is no longer in the live CMS list, cleaning up stale filename-rename dupes. - `download_attachments(store, client)` — SHA-256-keyed; skips files whose on-disk checksum matches the stored copy. - `check_future_updates(store, client)` — polls `futurepdf.pdf` (CMS upcoming-transmittal schedule); attaches only when checksum changes; designed as the trigger signal for out-of-band refresh cycles. - **`src/bib/sync.py`** — extended with attachment sync + collection routing. - `push_to_zotero(items, store=..., ...)` now walks `store.attachments` and copies PDFs into Zotero's `data/zotero/data/storage/<key>/` + creates `itemAttachments` rows with `linkMode=1` (IMPORTED_URL). - `_zotero_collection_path(item)` derives the target hierarchy; Manual items route to `Healthcare Data Platform / Manuals / <manual_name>`, ensured via `Db.ensure_collection` on the fly. - `replace_fields(item_id, fields)` — wipes existing itemData before rewriting to clean stale garbage from prior buggy syncs. - **`src/cli/bib.py`** — new commands, all idempotent: - `stack bib ingest-iom [--pub X]` - `stack bib attach-iom [--pub X] [--force]` - `stack bib watch-iom` - `stack bib sync-zotero [--tag X] [--type manual]` (wraps `docker stop/start zotero`) - `stack bib refresh-iom` — one-shot: ingest → watch → attach → sync. Safe to cron (follow-up #343). ### Schema parity fix Root cause of earlier Zotero corruption was stale `TYPE_MAP` / `FIELD_MAP` / `CREATOR_TYPES` constants in `src/zot/db.py` authored against an older Zotero schema. Modern Zotero 7 assigns different numeric IDs, so `manual → 15` landed as "email" instead of "report" and attachments as "document" instead of "attachment", tripping Zotero's `getFilePath()` consistency checks. - Constants rewritten from the live Zotero 7 DB. - `_verify_schema_parity()` runs on every `Db.__init__` and raises immediately if IDs drift — future Zotero upgrades can't silently corrupt data. - `src/zot/schema.sql` replaced with a fresh dump (2,499 lines) — the old file had triggers like `itemTypeID != 14` for attachments that were valid only in the pre-7 schema. - New `src/zot/ops.py` packages `dump_schema`, `fix_dates`, `fix_keys`, `fix_fields` as importable + CLI-exposed (`stack zot {…}`). Old `dev/scripts/*.py` became thin shims that call into the module. ### Data flow ``` CMS IOM index (HTML) │ fetch_index ▼ IOMEntry[] (pub, title, landing_url) │ fetch_chapters per entry ▼ ChapterEntry[] (pub, manual, chapter, title, url) │ cms_manual() + store.upsert() ▼ bib.sqlite items (item_type=manual) │ download_attachments → data/storage/<key>/<file>.pdf ▼ bib.sqlite attachments │ push_to_zotero(store=...) ▼ Zotero: report items + attachment children + collection memberships ``` ### Verified clean state - 177 Manual records across 20 pubs in bib.sqlite - 177 PDFs attached (one per chapter) - 177 Zotero items typed `report`, 177 attachment children typed `attachment` (linkMode=1) - `PRAGMA integrity_check` = ok, `PRAGMA foreign_key_check` = ok - All `(itemTypeID, fieldID)` pairs valid per `itemTypeFieldsCombined` ### Gotchas recorded for future agents - Carve-outs in `.dockerignore` are required (`!infra/marimo/theme/`, `!infra/gitea/`) or the build context lacks theme files. - Zotero holds an exclusive SQLite lock while running; every command that writes to the DB calls `docker stop zotero` / `start zotero` automatically (`--hold` default on). - The `filename` column in `bib.attachments` is a display label, not a filesystem name. Use `Path(storage_path).name` when copying into Zotero storage.

kert closed this issue

2026-04-15 04:23:44 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#304