stack/docs/superpowers/plans/2026-08-17-cy2027-notebook-gpci-pqm-p39.md

P39: CY2027 Notebook — GPCI & Primary-Care Deep-Dive Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Close the CY2027 NPRM notebook's two confessed gaps — no GPCI/locality analysis and no service-level story — with real lake-backed sections covering proposed GPCIs and all 109 MSSP-attribution primary care services, near-term and long-term.

Architecture: Data flows the established path: NPRM Addendum E → bib registration (fetch_nprm_addenda) → pfs.nprm.load_gpci_proposed into the aco monolith with cms.ingest_log provenance → publish_reference_to_lake.py (notebooks container) → DuckLake → notebook. The 109-code attribution list already sits in the monolith and only needs a renamed publish. Notebook edits come last, gated on the production-container integration harness.

Tech Stack: polars, duckdb, marimo, altair; Gitea tracker; nb_integration.py gate.

Spec: docs/superpowers/specs/2026-08-17-cy2027-notebook-gpci-pqm-design.md (committed). Tracker (real links):

• Milestone P39
• #629 — pfs: NPRM Addendum E loader → pfs.gpci_proposed
• #630 — lake: publish primary-care codes + gpci_proposed
• #631 — notebooks: GPCI & locality section
• #632 — notebooks: primary-care per-service analysis
• #633 — deps: duckdb 1.4.4 vs migrated DuckLake catalog

Global Constraints

• Conventional commits, (closes #N)/(refs #N); no Co-Authored-By trailer.
• Monolith writes (ingest_pfs.py) run on the host (lock-preflighted, #508 — check no marimo notebook holds aco.duckdb first); lake publishes and notebook validation run inside the notebooks container (only venv that attaches the migrated catalog until #633).
• Load Addendum E values as published (proposed PW GPCIs have no 1.0 floor); labeling floor status is the notebook's job, never the loader's.
• Notebook gate: nb_integration.py --set ci-smoke with a config listing only cy2027_pfs_proposed_rule.py must report pass=1 in the prod notebooks container.
• Load the dataviz skill before writing any new chart cell (Tasks 4–5).
• pfs.gpci_proposed target columns, in order: mac, state, locality, locality_name, work_gpci, pe_gpci, mp_gpci, gpci_year, cms_rule_id.

---

Task 1: Register Addendum E in bib (#629, part 1)

Files:

• Modify: dev/scripts/fetch_nprm_addenda.py (YearConfig + CONFIGS + register)

Interfaces:

• Produces: bib attachments titled CY{year} PFS NPRM Addendum E (proposed GPCIs) on rule items 2KVJ2HKX (2026) / 5ITGVDJV (2027), discoverable via tag sup:{year}_PFS_NPRM. Task 2's _discover_addendum_e depends on these titles/filenames matching (?i)addendum[_ ]?e.

• Step 1: Add to YearConfig: addendum_e_title: str. Add to both CONFIGS entries: addendum_e_title="CY2026 PFS NPRM Addendum E (proposed GPCIs)" / ...CY2027.... In register(), after the Addendum B attach block, repeat the same guarded attach for the Addendum E file (locate it in cfg.dest_dir via re.compile(r"(?i)addendum[_ ]?e").search(p.name) over dest_dir.glob("*.xlsx"), excluding __MACOSX); reuse _existing_attachment with the E title.

• Step 2: Run uv run python dev/scripts/fetch_nprm_addenda.py --year 2026 --register && uv run python dev/scripts/fetch_nprm_addenda.py --year 2027 --register. Expected: Addendum B prints SKIP attach (guard, from P38); Addendum E prints Attached ... first run. Re-run 2027 once more: E now prints SKIP attach (guard proven).

• Step 3: Commit: git add dev/scripts/fetch_nprm_addenda.py && git commit -m "feat(pfs): register NPRM Addendum E (proposed GPCIs) in bib (refs #629)"

Task 2: load_gpci_proposed loader + tests (#629, part 2)

Files:

• Modify: src/pfs/nprm.py, dev/scripts/ingest_pfs.py
• Test: tests/pfs/test_nprm.py

Interfaces:

• Consumes: _source_for(year) (existing), bib attachment discovery (generalize _discover_addendum_b(tag) → _discover_attachment(tag, pattern); keep _discover_addendum_b delegating so existing tests/callers stand).

• Produces: load_gpci_proposed(con, *, year: int) -> dict returning {"rows", "source_file", "cms_rule_id", "fr_citation", "pincite_key"} — same contract as load_rvu_proposed, table pfs.gpci_proposed per Global Constraints. ingest_pfs.py --nprm now loads B and E per year, logging one cms.ingest_log row each (table_name='pfs.gpci_proposed').

• Step 1 (failing tests): add to tests/pfs/test_nprm.py, mirroring the file's _patch_discovery idiom (patch nprm._discover_addendum_e):

  • test_load_gpci_proposed_cy2027_single_year — xlsx-like CSV fixture with header row Medicare Administrative Contractor,State,Locality Number,Locality Name,2027 PW GPCI (without 1.0 Floor),2027 PE GPCI,2027 MP GPCI, two locality rows + one footnote row (*Note... in MAC column); assert 2 rows, gpci_year==2027, footnote dropped, locality "00" preserved as text.
  • test_load_gpci_proposed_cy2026_wide_melts_two_years — fixture with 2025/2026/2027 triplets; assert rows = localities × 2 (gpci_year 2026 and 2027, no 2025), spot-check one PE value per year.
  • test_load_gpci_proposed_partitions_isolated — load 2026 then 2027, reload 2027; assert CMS-1832-P partition untouched, no duplicate 1848-P rows.

• Step 2: uv run pytest tests/pfs/test_nprm.py -q — new tests FAIL (load_gpci_proposed missing), old ones PASS.

• Step 3 (implement): in src/pfs/nprm.py: _ADDENDUM_E_RE = re.compile(r"(?i)addendum[_ ]?e"); _discover_attachment(tag, regex, kind) generalization; _GPCI_TARGET_COLUMNS list per Global Constraints; _gpci_schema_matches(con) (copy _schema_matches_target pattern against gpci_proposed); load_gpci_proposed(con, *, year): read xlsx via pl.read_excel(..., has_header=False), find the header row (first row whose col-0 startswith "Medicare Administrative"), name columns from it; melt: for each year-triplet {yy} PW/PE/MP with yy >= year, emit rows with gpci_year=yy; strip+cast GPCIs to float (strict=False); filter rows where MAC fails ^\d{5}$; keep state, locality (zero-padded text), locality_name as-is; add cms_rule_id; create-schema/drop-stale/create-if-missing/delete-partition/insert exactly as load_rvu_proposed does.

• Step 4: uv run pytest tests/pfs/test_nprm.py -q — all PASS. Then a real-file smoke on host (monolith untouched): uv run python -c "import duckdb; from pfs import nprm; con=duckdb.connect(); print(nprm.load_gpci_proposed(con, year=2027)); print(nprm.load_gpci_proposed(con, year=2026)); print(con.execute('SELECT cms_rule_id, gpci_year, count(*) FROM pfs.gpci_proposed GROUP BY ALL ORDER BY ALL').fetchall())" — expect ≈113 rows per (rule, year) group, three groups.

• Step 5: wire ingest_pfs.py: inside the existing --nprm year loop, after the Addendum B call, call load_gpci_proposed + ingest_log.log_ingest(..., table_name="pfs.gpci_proposed", ...) with its returned fields (same run_id).

• Step 6: Commit: git add src/pfs/nprm.py tests/pfs/test_nprm.py dev/scripts/ingest_pfs.py && git commit -m "feat(pfs): NPRM Addendum E loader -> pfs.gpci_proposed + ingest wiring (closes #629)"

Task 3: Monolith ingest + lake publish (#630)

Files:

• Modify: dev/scripts/publish_reference_to_lake.py

Interfaces:

• Produces (lake): pfs.gpci_proposed (~339 rows) and cms.primary_care_service_code (109 rows, columns hcpcs_code, description) readable via connect.ducklake(); refreshed cms.ingest_log. Tasks 4–5 read these names verbatim.

• Step 1: Check the DuckDB single-writer hazard (no marimo session holding aco.duckdb), then run uv run python dev/scripts/ingest_pfs.py --nprm on the host. Expected: per-year lines for pfs.rvu_proposed AND pfs.gpci_proposed, and replica republish.

• Step 2: Extend the publisher: add "gpci_proposed" to the pfs tuple; add RENAMED: tuple[tuple[str, str], ...] = (("cms_provider_attribution._primary_care_hcpcs_codes", "cms.primary_care_service_code"),) and a loop after the SCHEMAS loop that reads src, saves to the destination name, and read-back-verifies exactly like the main loop.

• Step 3: Publish from the notebooks container (script's documented invocation): docker cp dev/scripts/publish_reference_to_lake.py notebooks:/tmp/ && docker exec -e POSTGRES_PASSWORD="$(grep '^POSTGRES_PASSWORD=' .env | cut -d= -f2)" notebooks env PYTHONPATH=/home/kert/src uv run --project /home/kert/workspace python /tmp/publish_reference_to_lake.py --schemas pfs cms. Expected: every table [OK], incl. the two new names.

• Step 4: Verify from the notebook's exact read path (notebooks container): SELECT count(*) FROM cms.primary_care_service_code → 109; SELECT cms_rule_id, gpci_year, count(*) FROM pfs.gpci_proposed GROUP BY ALL → three groups ≈113; SELECT count(*) FROM cms.ingest_log WHERE table_name='pfs.gpci_proposed' ≥ 2.

• Step 5: Commit: git add dev/scripts/publish_reference_to_lake.py && git commit -m "feat(lake): publish gpci_proposed + primary_care_service_code to DuckLake (closes #630)". Comment the verification counts on #630.

Task 4: Notebook — GPCI & Locality section (#631)

Files:

• Modify: notebooks/cy2027_pfs_proposed_rule.py (new Section 3; renumber later sections; intro + provenance cells)

Interfaces:

• Consumes: pfs.gpci (final, 2016–2026), pfs.gpci_proposed, pfs.calcs.payment.payment(rvu, gpci, cf) (narwhals frames, joins on mac+locality), proposed_for(2027) CFs.

• Produces: section anchor cells Tasks 5 reuses (gpci_2027p polars frame: one row per locality with final-2026 and proposed-2027 work/pe/mp).

• Step 1: Load the dataviz skill (chart cells ahead).

• Step 2: Build the section (marimo cells, hide_code=True, matching the notebook's voice and sourcing conventions):

  1. Intro cell: what GPCIs are; CY2027 proposes the year-2 phase-in values vs CMS-1832-P's projection; work-floor callout (proposed PW published without the 1.0 floor; the CY2026-final column carries the floor).
  2. gpci_2027p frame: join pfs.gpci_proposed (CMS-1848-P, gpci_year 2027) to pfs.gpci (year 2026) on mac, locality; deltas for work/pe/mp.
  3. Movers table + diverging bar (top ±15 localities by GAF-proxy mean delta), up/down/flat counts in prose.
  4. Cross-NPRM check: 1832-P's projected-2027 vs 1848-P's proposed-2027 (self-join on locality; count of localities where the projection held vs moved).
  5. Long-term: line chart of work/pe/mp GPCI 2016–2026 + 2027p (dashed/annotated "final rule pending") for the extreme localities (max/min mean GPCI 2026) + a national median trace.
  6. Payment effect: pfs.calcs.payment on the 99213/27447/70553 trio at best/worst locality vs national (CF = proposed_for(2027) non-QP), small table.
  7. Intro cell of the notebook: replace the "does not join pfs.gpci" disclaimer with a pointer to this section; keep the "national unadjusted unless stated" scope note. Provenance cell: extend the table_name IN (...) filter with 'pfs.gpci','pfs.gpci_proposed' and drop the parenthetical that excludes gpci.

• Step 3: Gate: copy the single-notebook config trick and run nb_integration.py --set ci-smoke in the notebooks container → pass=1.

• Step 4: Commit: git add notebooks/cy2027_pfs_proposed_rule.py && git commit -m "feat(notebooks): CY2027 NPRM GPCI & locality section (closes #631)"

Task 5: Notebook — Primary-care per-service section (#632)

Files:

• Modify: notebooks/cy2027_pfs_proposed_rule.py (new Section 4, after GPCI)

Interfaces:

• Consumes: cms.primary_care_service_code, pfs.rvu 2015–2026, pfs.rvu_proposed (CMS-1848-P), RULES[y].conversion_factor 2015–2026, proposed_for(2027) CFs, gpci_2027p from Task 4.

• Produces: final notebook; provenance covers every table used.

• Step 1: Definition cell: the 109-code § 425.400(c) list read live; disclosures (current-law list applied retrospectively; unweighted). In-notebook sanity assert: assert pcs.height == 109.

• Step 2: Near-term frame pqm_near: LEFT JOIN codes → pfs.rvu_proposed (1848-P, mod-less rows) and → pfs.rvu (2026, mod-less); per code: work/total RVU both sides, national payment both sides (total × CF, CY2026 final CF vs CY2027 proposed non-QP CF), pct_change; classify both / new_in_2027 / missing_from_2027. Full sortable mo.ui.table (all 109 rows) + bar chart ranked by % payment change (both-side codes) + explicit callout listing one-sided codes (APCM family etc.).

• Step 3: Long-term frame pqm_long: codes × pfs.rvu 2015–2026 total RVU (mod-less) + 2027 proposed point; dollar column total_rvu × RULES[year].conversion_factor (2027p × proposed CF, labeled). Charts: code × year heatmap of YoY % RVU change; small-multiple dollar lines for the 12 largest cumulative movers; prose calling out the E/M-2021 step and the G-code cohorts that have no early-year history.

• Step 4: GPCI cross-cut cell: 99213 (or G0556 if only proposed) priced at the Task-4 extreme localities under the CY2027 proposed CF — table tying the two sections together.

• Step 5: Renumber later sections (APM → 5, Provenance → 6) and their headers; provenance SQL now covers pfs.rvu, pfs.rvu_proposed, pfs.gpci, pfs.gpci_proposed; add a prose line sourcing cms.primary_care_service_code (CCLF/attribution reference, not FR-ingested).

• Step 6: Gate: nb_integration.py single-notebook run → pass=1.

• Step 7: Commit: git add notebooks/cy2027_pfs_proposed_rule.py && git commit -m "feat(notebooks): CY2027 NPRM primary-care (ACO attribution) per-service analysis (closes #632)"

Task 6: duckdb pin alignment (#633)

Files:

• Modify: pyproject.toml, uv.lock

• Step 1: sed-bump every duckdb>=1.0.0 floor in pyproject.toml to duckdb>=1.5.4; uv lock (confirm the lock resolves 1.5.x); uv sync.

• Step 2: Verify: host uv run python -c "from conf import connect; print(connect.duckdb('aco').execute('select 1').fetchone())" (monolith replica opens under 1.5.x); host ducklake attach still fails only on DNS (compose-internal), not version — acceptable proof is running the attach inside the api container after a rebuild is NOT in scope; the host-venv check suffices.

• Step 3: Full test suite (the pre-commit hook runs it on commit — that IS the run; budget ~6 min). If red on duckdb-related failures: git checkout pyproject.toml uv.lock, comment findings on #633, leave the issue open, and continue to Task 7.

• Step 4: Commit (hook-gated): git add pyproject.toml uv.lock && git commit -m "fix(deps): duckdb >=1.5.4 — attach migrated DuckLake catalog from host (closes #633)"

Task 7: Close the loop

• Step 1: git push; confirm CI green on HEAD (stale auto-filed CI issues are known noise — check the actual run).
• Step 2: Verify all of #629–#633 closed (auto-close on push) or close with comments; comment final row-counts + pass=1 evidence on #631/#632; close milestone P39.
• Step 3: Append "P39 build outcomes" to the spec doc; commit docs(spec): P39 build outcomes (refs #629-#633).
• Step 4: Memory: note in zotero_rule_sync_incident.md-adjacent project memory only if something non-obvious emerged (candidate: the Addendum E wide-format melt and the publish rename map).

Self-Review

• Coverage: #629→Tasks 1–2, #630→Task 3, #631→Task 4, #632→Task 5, #633→Task 6, closure→Task 7. Spec D1–D4 all mapped.
• Placeholders: none — every code step names exact columns, regexes, commands, expected counts.
• Consistency: _GPCI_TARGET_COLUMNS order matches Global Constraints and Task 3/4 SQL; gpci_2027p produced in Task 4 Step 2.2 and consumed in Task 5 Step 4; discovery generalization keeps _discover_addendum_b name alive for existing tests.