stack/docs/superpowers/specs/2026-08-17-cy2027-notebook-gpci-pqm-design.md

CY2027 NPRM Notebook — GPCI & Primary-Care (PQM) Deep-Dive (P39)

Date: 2026-08-17. Milestone: P39. Requested as: "fix the notebook, and also ensure that the notebook focuses both on the GPCI AND focuses on more in-depth analysis of the PQM services (i.e. long-term and near-term comparison of each individual service that is also a designated primary care service for purposes of ACO attribution)."

Problem

notebooks/cy2027_pfs_proposed_rule.py executes clean (verified via nb_integration.py in the production notebooks container, 2026-08-17) but is analytically incomplete in exactly the two ways the request names:

1. GPCI. Every dollar figure is national/unadjusted — the intro discloses "GPCI = 1.0 … real payment varies by locality via pfs.gpci, which this notebook does not join." The CY2027 NPRM's proposed GPCIs (Addendum E) were captured to disk in P37 (data/cms/pfs_nprm/2027/) but never loaded to any table; the lake's pfs.gpci holds final-rule GPCIs 2016–2026 only.
2. Primary-care service-level analysis. No section examines individual services. "PQM services" is defined by the request's own gloss as the designated primary care services used for MSSP/ACO beneficiary assignment (42 CFR § 425.400(c)). The authoritative in-repo list is cms_provider_attribution._primary_care_hcpcs_codes (aco monolith, 109 HCPCS codes, current-law list incl. APCM G0556–G0558, CHI/PIN G0019–G0024, G2211, G3002–G3003).

"Fix the notebook" therefore means: close those two gaps, and update the intro + provenance sections that currently disclaim them.

Data facts (verified 2026-08-17)

• pfs.rvu (lake): 2015–2026, ~16–19k rows/year.
• pfs.rvu_proposed (lake): CMS-1832-P (CY2026 NPRM) + CMS-1848-P (CY2027 NPRM) partitions.
• pfs.gpci (lake): 2016–2026; columns mac, locality, locality_name, work_gpci, pe_gpci, mp_gpci, year, state; ~109–112 localities/year.
• pfs.rules.RULES: 2014–2026 incl. conversion_factor for every year 2015–2026; proposed_for(2027) has both CY2027 proposed CFs.
• NPRM Addendum E workbooks on disk for 2026 and 2027 (from fetch_nprm_addenda.py); not attached in bib, not loaded.
• cms_provider_attribution._primary_care_hcpcs_codes: 109 rows (hcpcs_code, description), in aco.duckdb / aco.ro.duckdb only — not in the lake, so the notebook cannot read it today.
• Pipeline shape: ingest → aco.duckdb monolith (host) → dev/scripts/publish_reference_to_lake.py (runs inside the notebooks container) → DuckLake → notebooks read via connect.ducklake().
• pfs.calcs.payment.payment(rvu, gpci, cf) already implements the locality formula (work×work_gpci + pe×pe_gpci + mp×mp_gpci) × CF.
• Landmine: the DuckLake catalog metadata has been migrated to a version only duckdb ≥1.5 attaches. The notebooks venv (1.5.4) works; the main project venv and api container (1.4.4, from uv.lock) fail with "Only DuckLake versions 0.1–0.3 supported." Nothing on the critical path uses the broken combination today, but host-side connect.ducklake() is dead until the pin moves.

Design

D1 — pfs.gpci_proposed (new lake table)

Mirror of the rvu_proposed pattern: one partition per cms_rule_id (CMS-1832-P, CMS-1848-P), loaded from the NPRM Addendum E workbook. Verified layouts: the CY2027 file carries one proposed year (2027 PW/PE/MP); the CY2026 file is wide — 2025 baseline plus proposed 2026 and 2027 columns (GPCIs phase in biennially). Load long: columns mac, state, locality, locality_name, work_gpci, pe_gpci, mp_gpci, gpci_year, cms_rule_id, taking every year-column ≥ the NPRM's rule year (CMS-1832-P → gpci_year 2026, 2027; CMS-1848-P → gpci_year 2027; the 2025 baseline column is a restatement, skipped). That makes the "1832-P's projected 2027 vs 1848-P's actually-proposed 2027" comparison a plain self-join. Footnote rows (non-numeric MAC) are filtered; proposed PW GPCIs are published without the statutory 1.0 work floor — values load as published and the notebook labels the floor status rather than silently re-flooring. Loader lives in src/pfs/nprm.py beside load_rvu_proposed, discovers the workbook through the bib attachment tagged sup:<year>_PFS_NPRM (same mechanism as Addendum B), writes cms.ingest_log provenance, and deletes/reinserts only its own partition. fetch_nprm_addenda.py CONFIGS gain an Addendum E attach (title CY{year} PFS NPRM Addendum E (proposed GPCIs)) using the existing title-guarded idempotent register.

Addendum D (GAFs) is not loaded — the GPCI components are the primitive; GAF adds no information the notebook needs (YAGNI).

D2 — primary-care code list to the lake

publish_reference_to_lake.py currently publishes same-named tables from three schemas. Extend it with an explicit rename map and publish cms_provider_attribution._primary_care_hcpcs_codes → cms.primary_care_service_code (109 rows, hcpcs_code, description). pfs.gpci_proposed joins the existing pfs list. The publish runs in the notebooks container as documented in that script's header.

D3 — notebook changes

New Section: GPCI & Locality (between current Sections 2 and 3):

• Near-term: CY2027-proposed vs CY2026-final work/pe/mp GPCI deltas per locality — table of biggest movers + diverging-bar chart; counts of localities up/down/flat.
• Long-term: GPCI trend 2016–2026 (+2027 proposed, visually flagged "final rule pending") for the highest/lowest-GPCI localities and the floor story (work GPCI floor).
• Payment effect: locality-adjusted payment via pfs.calcs.payment for example codes (reusing the notebook's existing 99213/27447/70553 trio), best vs worst locality vs national.

New Section: Primary Care Services (ACO attribution set):

• Definition callout: the 109-code § 425.400(c) list, read live from cms.primary_care_service_code, with the disclosure that the current-law designation list is applied retrospectively across years (the list itself is not versioned in-repo).
• Near-term, every code individually: CY2026-final vs CY2027-proposed work/total RVUs and national payment (both CFs known), full 109-row sortable table + chart ranked by % payment change; codes present in only one side surfaced explicitly (new APCM-family codes have no CY2026 final RVU).
• Long-term, per code: total-RVU and dollar (RVU × that year's final CF) trajectories 2015–2026 with the 2027 proposed point appended and labeled; heatmap (code × year, % change) plus small-multiple lines for the codes with the largest cumulative moves.
• GPCI cross-cut: the near-term payment delta for a chosen primary-care code shown at the extreme localities (joins D1).
• Aggregation is unweighted (no utilization data in the lake); disclosed inline.

Fixes to existing cells: intro rewritten (GPCI = 1.0 disclaimer replaced by a pointer to the GPCI section; the remaining national-unadjusted figures keep a scoped note), provenance section extended to pfs.gpci, pfs.gpci_proposed ingest-log rows and the primary-care list's source note. Everything else (CF walk, RVU deltas, APM section) stays.

D4 — duckdb/DuckLake version alignment

Bump the project's duckdb to ≥1.5.4 (uv lock), so host tooling can attach the migrated catalog again; verify the test suite and that the monolith (written by 1.4.4) opens cleanly under 1.5.4. If the bump breaks tests, defer with findings on the issue rather than forcing it — nothing in P39's critical path requires it (loads target the monolith; publish + notebook run in the notebooks container).

Testing

• Loader: unit tests beside the existing rvu_proposed NPRM tests — header detection on the real CY2027 Addendum E layout, partition isolation (loading 2027 leaves the 1832-P partition untouched), numeric coercion.
• Publisher: rename-map unit test (pure function).
• Notebook: nb_integration.py single-notebook run in the production notebooks container must stay green; data-sanity assertions inside the notebook (e.g. 109 codes read, locality counts match) fail loud in that harness.

Out of scope

• Utilization-weighted impact (no claims-volume reference in the lake).
• Addendum D / GAF ingestion.
• Versioning the primary-care designation list by rule year.
• Retiring the monolith (#514) — P39 adds one publisher entry, no more.

P39 build outcomes (2026-08-17)

Issues #629–#633, milestone P39 (5 commits, 39dfaa2..afeec59, direct on main — mail-poller live-mounts this checkout and Tasks ran against the live monolith/lake). All closed same-day; notebook gate pass=1 in the prod notebooks container after each notebook change.

• #629 (39dfaa2 + the Addendum-E register commit): load_gpci_proposed → pfs.gpci_proposed (long format with gpci_year; CMS-1832-P carries 2026+2027, CMS-1848-P carries 2027 — 109 localities each; 2025 baseline restatement skipped; PW loaded floor-free as published). Addendum E attached in bib for both years (title-guarded, canonical storage paths). 3 new loader tests; ingest_pfs.py --nprm logs pfs.gpci_proposed ingest rows.
• #630 (fc26d5f): publisher gained gpci_proposed + a rename map; cms_provider_attribution._primary_care_hcpcs_codes → cms.primary_care_service_code (109 rows). Verified from connect.ducklake() in the notebooks container.
• #631/#632 (8c4e13c, one combined commit — the sections were a single splice): new Sections 3 (GPCI & locality) and 4 (primary care / attribution set); APM → 5, Provenance → 6; intro's "GPCI=1.0, does not join pfs.gpci" disclaimer replaced; provenance covers the gpci tables and sources the attribution list narratively.
• #633 (afeec59): duckdb floor 1.0 → ≥1.5.4 (resolved 1.5.5); full suite green under the new pin; monolith + replica open cleanly; host-side connect.ducklake() no longer version-blocked (DNS to the compose-internal catalog remains, by design).
• Findings worth keeping: the CY2027 NPRM re-proposes the CY2026 NPRM's 2027 GPCI phase-in for 103 of 109 localities (6 revised); every designated primary-care code that prices anywhere prices in both CY2026-final and CY2027-proposed tables — the 12 unmatched are all deleted legacy codes still on the § 425.400(c) list (the notebook's one-sided narrative was corrected against live data before commit).
• Deviation: #631/#632 shipped as one commit; Task-3 ingest ran via a targeted driver (same duckdb_batch/ingest_log/replica plumbing) instead of the full ingest_pfs.py --nprm reload, to avoid an unnecessary multi-minute load_all pass.