stack/docs/superpowers/plans/2026-08-17-cy2027-notebook-gpci-pqm-p39.md

# P39: CY2027 Notebook — GPCI & Primary-Care Deep-Dive Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Close the CY2027 NPRM notebook's two confessed gaps — no GPCI/locality analysis and no service-level story — with real lake-backed sections covering proposed GPCIs and all 109 MSSP-attribution primary care services, near-term and long-term.

**Architecture:** Data flows the established path: NPRM Addendum E → bib registration (`fetch_nprm_addenda`) → `pfs.nprm.load_gpci_proposed` into the aco monolith with `cms.ingest_log` provenance → `publish_reference_to_lake.py` (notebooks container) → DuckLake → notebook. The 109-code attribution list already sits in the monolith and only needs a renamed publish. Notebook edits come last, gated on the production-container integration harness.

**Tech Stack:** polars, duckdb, marimo, altair; Gitea tracker; `nb_integration.py` gate.

**Spec:** `docs/superpowers/specs/2026-08-17-cy2027-notebook-gpci-pqm-design.md` (committed). Tracker (real links):
- Milestone [P39](https://git.fhirworx.io/homelab/stack/milestone/40)
- [#629 — pfs: NPRM Addendum E loader → pfs.gpci_proposed](https://git.fhirworx.io/homelab/stack/issues/629)
- [#630 — lake: publish primary-care codes + gpci_proposed](https://git.fhirworx.io/homelab/stack/issues/630)
- [#631 — notebooks: GPCI & locality section](https://git.fhirworx.io/homelab/stack/issues/631)
- [#632 — notebooks: primary-care per-service analysis](https://git.fhirworx.io/homelab/stack/issues/632)
- [#633 — deps: duckdb 1.4.4 vs migrated DuckLake catalog](https://git.fhirworx.io/homelab/stack/issues/633)

## Global Constraints

- Conventional commits, `(closes #N)`/`(refs #N)`; **no Co-Authored-By trailer**.
- Monolith writes (`ingest_pfs.py`) run on the **host** (lock-preflighted, #508 — check no marimo notebook holds `aco.duckdb` first); lake publishes and notebook validation run **inside the notebooks container** (only venv that attaches the migrated catalog until #633).
- Load Addendum E values **as published** (proposed PW GPCIs have no 1.0 floor); labeling floor status is the notebook's job, never the loader's.
- Notebook gate: `nb_integration.py --set ci-smoke` with a config listing only `cy2027_pfs_proposed_rule.py` must report `pass=1` in the prod notebooks container.
- Load the `dataviz` skill before writing any new chart cell (Tasks 4–5).
- `pfs.gpci_proposed` target columns, in order: `mac, state, locality, locality_name, work_gpci, pe_gpci, mp_gpci, gpci_year, cms_rule_id`.

---

### Task 1: Register Addendum E in bib ([#629](https://git.fhirworx.io/homelab/stack/issues/629), part 1)

**Files:**
- Modify: `dev/scripts/fetch_nprm_addenda.py` (YearConfig + CONFIGS + register)

**Interfaces:**
- Produces: bib attachments titled `CY{year} PFS NPRM Addendum E (proposed GPCIs)` on rule items `2KVJ2HKX` (2026) / `5ITGVDJV` (2027), discoverable via tag `sup:{year}_PFS_NPRM`. Task 2's `_discover_addendum_e` depends on these titles/filenames matching `(?i)addendum[_ ]?e`.

- [x] **Step 1:** Add to `YearConfig`: `addendum_e_title: str`. Add to both CONFIGS entries: `addendum_e_title="CY2026 PFS NPRM Addendum E (proposed GPCIs)"` / `...CY2027...`. In `register()`, after the Addendum B attach block, repeat the same guarded attach for the Addendum E file (locate it in `cfg.dest_dir` via `re.compile(r"(?i)addendum[_ ]?e").search(p.name)` over `dest_dir.glob("*.xlsx")`, excluding `__MACOSX`); reuse `_existing_attachment` with the E title.
- [x] **Step 2:** Run `uv run python dev/scripts/fetch_nprm_addenda.py --year 2026 --register && uv run python dev/scripts/fetch_nprm_addenda.py --year 2027 --register`. Expected: Addendum B prints `SKIP attach` (guard, from P38); Addendum E prints `Attached ...` first run. Re-run 2027 once more: E now prints `SKIP attach` (guard proven).
- [x] **Step 3:** Commit: `git add dev/scripts/fetch_nprm_addenda.py && git commit -m "feat(pfs): register NPRM Addendum E (proposed GPCIs) in bib (refs #629)"`

### Task 2: `load_gpci_proposed` loader + tests ([#629](https://git.fhirworx.io/homelab/stack/issues/629), part 2)

**Files:**
- Modify: `src/pfs/nprm.py`, `dev/scripts/ingest_pfs.py`
- Test: `tests/pfs/test_nprm.py`

**Interfaces:**
- Consumes: `_source_for(year)` (existing), bib attachment discovery (generalize `_discover_addendum_b(tag)` → `_discover_attachment(tag, pattern)`; keep `_discover_addendum_b` delegating so existing tests/callers stand).
- Produces: `load_gpci_proposed(con, *, year: int) -> dict` returning `{"rows", "source_file", "cms_rule_id", "fr_citation", "pincite_key"}` — same contract as `load_rvu_proposed`, table `pfs.gpci_proposed` per Global Constraints. `ingest_pfs.py --nprm` now loads B **and** E per year, logging one `cms.ingest_log` row each (`table_name='pfs.gpci_proposed'`).

- [x] **Step 1 (failing tests):** add to `tests/pfs/test_nprm.py`, mirroring the file's `_patch_discovery` idiom (patch `nprm._discover_addendum_e`):
  - `test_load_gpci_proposed_cy2027_single_year` — xlsx-like CSV fixture with header row `Medicare Administrative Contractor,State,Locality Number,Locality Name,2027 PW GPCI (without 1.0 Floor),2027 PE GPCI,2027 MP GPCI`, two locality rows + one footnote row (`*Note...` in MAC column); assert 2 rows, `gpci_year==2027`, footnote dropped, locality `"00"` preserved as text.
  - `test_load_gpci_proposed_cy2026_wide_melts_two_years` — fixture with 2025/2026/2027 triplets; assert rows = localities × 2 (gpci_year 2026 and 2027, no 2025), spot-check one PE value per year.
  - `test_load_gpci_proposed_partitions_isolated` — load 2026 then 2027, reload 2027; assert CMS-1832-P partition untouched, no duplicate 1848-P rows.
- [x] **Step 2:** `uv run pytest tests/pfs/test_nprm.py -q` — new tests FAIL (`load_gpci_proposed` missing), old ones PASS.
- [x] **Step 3 (implement):** in `src/pfs/nprm.py`: `_ADDENDUM_E_RE = re.compile(r"(?i)addendum[_ ]?e")`; `_discover_attachment(tag, regex, kind)` generalization; `_GPCI_TARGET_COLUMNS` list per Global Constraints; `_gpci_schema_matches(con)` (copy `_schema_matches_target` pattern against `gpci_proposed`); `load_gpci_proposed(con, *, year)`:
  read xlsx via `pl.read_excel(..., has_header=False)`, find the header row (first row whose col-0 startswith "Medicare Administrative"), name columns from it; melt: for each year-triplet `{yy} PW/PE/MP` with `yy >= year`, emit rows with `gpci_year=yy`; strip+cast GPCIs to float (strict=False); filter rows where MAC fails `^\d{5}$`; keep `state`, `locality` (zero-padded text), `locality_name` as-is; add `cms_rule_id`; create-schema/drop-stale/create-if-missing/delete-partition/insert exactly as `load_rvu_proposed` does.
- [x] **Step 4:** `uv run pytest tests/pfs/test_nprm.py -q` — all PASS. Then a real-file smoke on host (monolith untouched): `uv run python -c "import duckdb; from pfs import nprm; con=duckdb.connect(); print(nprm.load_gpci_proposed(con, year=2027)); print(nprm.load_gpci_proposed(con, year=2026)); print(con.execute('SELECT cms_rule_id, gpci_year, count(*) FROM pfs.gpci_proposed GROUP BY ALL ORDER BY ALL').fetchall())"` — expect ≈113 rows per (rule, year) group, three groups.
- [x] **Step 5:** wire `ingest_pfs.py`: inside the existing `--nprm` year loop, after the Addendum B call, call `load_gpci_proposed` + `ingest_log.log_ingest(..., table_name="pfs.gpci_proposed", ...)` with its returned fields (same run_id).
- [x] **Step 6:** Commit: `git add src/pfs/nprm.py tests/pfs/test_nprm.py dev/scripts/ingest_pfs.py && git commit -m "feat(pfs): NPRM Addendum E loader -> pfs.gpci_proposed + ingest wiring (closes #629)"`

### Task 3: Monolith ingest + lake publish ([#630](https://git.fhirworx.io/homelab/stack/issues/630))

**Files:**
- Modify: `dev/scripts/publish_reference_to_lake.py`

**Interfaces:**
- Produces (lake): `pfs.gpci_proposed` (~339 rows) and `cms.primary_care_service_code` (109 rows, columns `hcpcs_code`, `description`) readable via `connect.ducklake()`; refreshed `cms.ingest_log`. Tasks 4–5 read these names verbatim.

- [x] **Step 1:** Check the DuckDB single-writer hazard (no marimo session holding aco.duckdb), then run `uv run python dev/scripts/ingest_pfs.py --nprm` on the host. Expected: per-year lines for `pfs.rvu_proposed` AND `pfs.gpci_proposed`, and replica republish.
- [x] **Step 2:** Extend the publisher: add `"gpci_proposed"` to the `pfs` tuple; add `RENAMED: tuple[tuple[str, str], ...] = (("cms_provider_attribution._primary_care_hcpcs_codes", "cms.primary_care_service_code"),)` and a loop after the SCHEMAS loop that reads `src`, saves to the destination name, and read-back-verifies exactly like the main loop.
- [x] **Step 3:** Publish from the notebooks container (script's documented invocation): `docker cp dev/scripts/publish_reference_to_lake.py notebooks:/tmp/ && docker exec -e POSTGRES_PASSWORD="$(grep '^POSTGRES_PASSWORD=' .env | cut -d= -f2)" notebooks env PYTHONPATH=/home/kert/src uv run --project /home/kert/workspace python /tmp/publish_reference_to_lake.py --schemas pfs cms`. Expected: every table `[OK]`, incl. the two new names.
- [x] **Step 4:** Verify from the notebook's exact read path (notebooks container): `SELECT count(*) FROM cms.primary_care_service_code` → 109; `SELECT cms_rule_id, gpci_year, count(*) FROM pfs.gpci_proposed GROUP BY ALL` → three groups ≈113; `SELECT count(*) FROM cms.ingest_log WHERE table_name='pfs.gpci_proposed'` ≥ 2.
- [x] **Step 5:** Commit: `git add dev/scripts/publish_reference_to_lake.py && git commit -m "feat(lake): publish gpci_proposed + primary_care_service_code to DuckLake (closes #630)"`. Comment the verification counts on [#630](https://git.fhirworx.io/homelab/stack/issues/630).

### Task 4: Notebook — GPCI & Locality section ([#631](https://git.fhirworx.io/homelab/stack/issues/631))

**Files:**
- Modify: `notebooks/cy2027_pfs_proposed_rule.py` (new Section 3; renumber later sections; intro + provenance cells)

**Interfaces:**
- Consumes: `pfs.gpci` (final, 2016–2026), `pfs.gpci_proposed`, `pfs.calcs.payment.payment(rvu, gpci, cf)` (narwhals frames, joins on `mac`+`locality`), `proposed_for(2027)` CFs.
- Produces: section anchor cells Tasks 5 reuses (`gpci_2027p` polars frame: one row per locality with final-2026 and proposed-2027 work/pe/mp).

- [x] **Step 1:** Load the `dataviz` skill (chart cells ahead).
- [x] **Step 2:** Build the section (marimo cells, `hide_code=True`, matching the notebook's voice and sourcing conventions):
  1. Intro cell: what GPCIs are; CY2027 proposes the year-2 phase-in values vs CMS-1832-P's projection; **work-floor callout** (proposed PW published without the 1.0 floor; the CY2026-final column carries the floor).
  2. `gpci_2027p` frame: join `pfs.gpci_proposed` (CMS-1848-P, gpci_year 2027) to `pfs.gpci` (year 2026) on `mac, locality`; deltas for work/pe/mp.
  3. Movers table + diverging bar (top ±15 localities by GAF-proxy mean delta), up/down/flat counts in prose.
  4. Cross-NPRM check: 1832-P's projected-2027 vs 1848-P's proposed-2027 (self-join on locality; count of localities where the projection held vs moved).
  5. Long-term: line chart of work/pe/mp GPCI 2016–2026 + 2027p (dashed/annotated "final rule pending") for the extreme localities (max/min mean GPCI 2026) + a national median trace.
  6. Payment effect: `pfs.calcs.payment` on the 99213/27447/70553 trio at best/worst locality vs national (CF = `proposed_for(2027)` non-QP), small table.
  7. Intro cell of the notebook: replace the "does not join pfs.gpci" disclaimer with a pointer to this section; keep the "national unadjusted unless stated" scope note. Provenance cell: extend the `table_name IN (...)` filter with `'pfs.gpci','pfs.gpci_proposed'` and drop the parenthetical that excludes gpci.
- [x] **Step 3:** Gate: copy the single-notebook config trick and run `nb_integration.py --set ci-smoke` in the notebooks container → `pass=1`.
- [x] **Step 4:** Commit: `git add notebooks/cy2027_pfs_proposed_rule.py && git commit -m "feat(notebooks): CY2027 NPRM GPCI & locality section (closes #631)"`

### Task 5: Notebook — Primary-care per-service section ([#632](https://git.fhirworx.io/homelab/stack/issues/632))

**Files:**
- Modify: `notebooks/cy2027_pfs_proposed_rule.py` (new Section 4, after GPCI)

**Interfaces:**
- Consumes: `cms.primary_care_service_code`, `pfs.rvu` 2015–2026, `pfs.rvu_proposed` (CMS-1848-P), `RULES[y].conversion_factor` 2015–2026, `proposed_for(2027)` CFs, `gpci_2027p` from Task 4.
- Produces: final notebook; provenance covers every table used.

- [x] **Step 1:** Definition cell: the 109-code § 425.400(c) list read live; disclosures (current-law list applied retrospectively; unweighted). In-notebook sanity assert: `assert pcs.height == 109`.
- [x] **Step 2:** Near-term frame `pqm_near`: LEFT JOIN codes → `pfs.rvu_proposed` (1848-P, mod-less rows) and → `pfs.rvu` (2026, mod-less); per code: work/total RVU both sides, national payment both sides (`total × CF`, CY2026 final CF vs CY2027 proposed non-QP CF), `pct_change`; classify `both / new_in_2027 / missing_from_2027`. Full sortable `mo.ui.table` (all 109 rows) + bar chart ranked by % payment change (both-side codes) + explicit callout listing one-sided codes (APCM family etc.).
- [x] **Step 3:** Long-term frame `pqm_long`: codes × `pfs.rvu` 2015–2026 total RVU (mod-less) + 2027 proposed point; dollar column `total_rvu × RULES[year].conversion_factor` (2027p × proposed CF, labeled). Charts: code × year heatmap of YoY % RVU change; small-multiple dollar lines for the 12 largest cumulative movers; prose calling out the E/M-2021 step and the G-code cohorts that have no early-year history.
- [x] **Step 4:** GPCI cross-cut cell: 99213 (or G0556 if only proposed) priced at the Task-4 extreme localities under the CY2027 proposed CF — table tying the two sections together.
- [x] **Step 5:** Renumber later sections (APM → 5, Provenance → 6) and their headers; provenance SQL now covers `pfs.rvu, pfs.rvu_proposed, pfs.gpci, pfs.gpci_proposed`; add a prose line sourcing `cms.primary_care_service_code` (CCLF/attribution reference, not FR-ingested).
- [x] **Step 6:** Gate: `nb_integration.py` single-notebook run → `pass=1`.
- [x] **Step 7:** Commit: `git add notebooks/cy2027_pfs_proposed_rule.py && git commit -m "feat(notebooks): CY2027 NPRM primary-care (ACO attribution) per-service analysis (closes #632)"`

### Task 6: duckdb pin alignment ([#633](https://git.fhirworx.io/homelab/stack/issues/633))

**Files:**
- Modify: `pyproject.toml`, `uv.lock`

- [x] **Step 1:** `sed`-bump every `duckdb>=1.0.0` floor in `pyproject.toml` to `duckdb>=1.5.4`; `uv lock` (confirm the lock resolves 1.5.x); `uv sync`.
- [x] **Step 2:** Verify: host `uv run python -c "from conf import connect; print(connect.duckdb('aco').execute('select 1').fetchone())"` (monolith replica opens under 1.5.x); host ducklake attach still fails only on DNS (compose-internal), not version — acceptable proof is running the attach inside the **api** container after a rebuild is NOT in scope; the host-venv check suffices.
- [x] **Step 3:** Full test suite (the pre-commit hook runs it on commit — that IS the run; budget ~6 min). If red on duckdb-related failures: `git checkout pyproject.toml uv.lock`, comment findings on [#633](https://git.fhirworx.io/homelab/stack/issues/633), leave the issue open, and continue to Task 7.
- [x] **Step 4:** Commit (hook-gated): `git add pyproject.toml uv.lock && git commit -m "fix(deps): duckdb >=1.5.4 — attach migrated DuckLake catalog from host (closes #633)"`

### Task 7: Close the loop

- [x] **Step 1:** `git push`; confirm CI green on HEAD (stale auto-filed CI issues are known noise — check the actual run).
- [x] **Step 2:** Verify all of #629–#633 closed (auto-close on push) or close with comments; comment final row-counts + `pass=1` evidence on [#631](https://git.fhirworx.io/homelab/stack/issues/631)/[#632](https://git.fhirworx.io/homelab/stack/issues/632); close [milestone P39](https://git.fhirworx.io/homelab/stack/milestone/40).
- [x] **Step 3:** Append "P39 build outcomes" to the spec doc; commit `docs(spec): P39 build outcomes (refs #629-#633)`.
- [x] **Step 4:** Memory: note in `zotero_rule_sync_incident.md`-adjacent project memory only if something non-obvious emerged (candidate: the Addendum E wide-format melt and the publish rename map).

## Self-Review

- Coverage: #629→Tasks 1–2, #630→Task 3, #631→Task 4, #632→Task 5, #633→Task 6, closure→Task 7. Spec D1–D4 all mapped.
- Placeholders: none — every code step names exact columns, regexes, commands, expected counts.
- Consistency: `_GPCI_TARGET_COLUMNS` order matches Global Constraints and Task 3/4 SQL; `gpci_2027p` produced in Task 4 Step 2.2 and consumed in Task 5 Step 4; discovery generalization keeps `_discover_addendum_b` name alive for existing tests.