stack/docs/superpowers/specs/2026-08-17-cy2027-notebook-gpci-pqm-design.md

# CY2027 NPRM Notebook — GPCI & Primary-Care (PQM) Deep-Dive (P39)

Date: 2026-08-17. Milestone: P39. Requested as: "fix the notebook, and
also ensure that the notebook focuses both on the GPCI AND focuses on
more in-depth analysis of the PQM services (i.e. long-term and
near-term comparison of each individual service that is also a
designated primary care service for purposes of ACO attribution)."

## Problem

`notebooks/cy2027_pfs_proposed_rule.py` executes clean (verified via
`nb_integration.py` in the production notebooks container, 2026-08-17)
but is analytically incomplete in exactly the two ways the request
names:

1. **GPCI.** Every dollar figure is national/unadjusted — the intro
   discloses "GPCI = 1.0 … real payment varies by locality via
   `pfs.gpci`, which this notebook does not join." The CY2027 NPRM's
   proposed GPCIs (Addendum E) were captured to disk in P37
   (`data/cms/pfs_nprm/2027/`) but never loaded to any table; the
   lake's `pfs.gpci` holds final-rule GPCIs 2016–2026 only.
2. **Primary-care service-level analysis.** No section examines
   individual services. "PQM services" is defined by the request's own
   gloss as the **designated primary care services used for MSSP/ACO
   beneficiary assignment** (42 CFR § 425.400(c)). The authoritative
   in-repo list is `cms_provider_attribution._primary_care_hcpcs_codes`
   (aco monolith, 109 HCPCS codes, current-law list incl. APCM
   G0556–G0558, CHI/PIN G0019–G0024, G2211, G3002–G3003).

"Fix the notebook" therefore means: close those two gaps, and update
the intro + provenance sections that currently disclaim them.

## Data facts (verified 2026-08-17)

- `pfs.rvu` (lake): 2015–2026, ~16–19k rows/year.
- `pfs.rvu_proposed` (lake): CMS-1832-P (CY2026 NPRM) + CMS-1848-P
  (CY2027 NPRM) partitions.
- `pfs.gpci` (lake): 2016–2026; columns `mac, locality, locality_name,
  work_gpci, pe_gpci, mp_gpci, year, state`; ~109–112 localities/year.
- `pfs.rules.RULES`: 2014–2026 incl. `conversion_factor` for every
  year 2015–2026; `proposed_for(2027)` has both CY2027 proposed CFs.
- NPRM Addendum E workbooks on disk for 2026 and 2027 (from
  `fetch_nprm_addenda.py`); **not** attached in bib, **not** loaded.
- `cms_provider_attribution._primary_care_hcpcs_codes`: 109 rows
  (`hcpcs_code`, `description`), in `aco.duckdb` / `aco.ro.duckdb`
  only — not in the lake, so the notebook cannot read it today.
- Pipeline shape: ingest → `aco.duckdb` monolith (host) →
  `dev/scripts/publish_reference_to_lake.py` (runs **inside the
  notebooks container**) → DuckLake → notebooks read via
  `connect.ducklake()`.
- `pfs.calcs.payment.payment(rvu, gpci, cf)` already implements the
  locality formula `(work×work_gpci + pe×pe_gpci + mp×mp_gpci) × CF`.
- **Landmine:** the DuckLake catalog metadata has been migrated to a
  version only duckdb ≥1.5 attaches. The notebooks venv (1.5.4) works;
  the main project venv and api container (1.4.4, from `uv.lock`) fail
  with "Only DuckLake versions 0.1–0.3 supported." Nothing on the
  critical path uses the broken combination today, but host-side
  `connect.ducklake()` is dead until the pin moves.

## Design

### D1 — `pfs.gpci_proposed` (new lake table)

Mirror of the `rvu_proposed` pattern: one partition per `cms_rule_id`
(CMS-1832-P, CMS-1848-P), loaded from the NPRM **Addendum E** workbook.
Verified layouts: the CY2027 file carries one proposed year (2027
PW/PE/MP); the CY2026 file is wide — 2025 baseline plus proposed 2026
**and** 2027 columns (GPCIs phase in biennially). Load long: columns
`mac, state, locality, locality_name, work_gpci, pe_gpci, mp_gpci,
gpci_year, cms_rule_id`, taking every year-column ≥ the NPRM's rule
year (CMS-1832-P → gpci_year 2026, 2027; CMS-1848-P → gpci_year 2027;
the 2025 baseline column is a restatement, skipped). That makes the
"1832-P's projected 2027 vs 1848-P's actually-proposed 2027"
comparison a plain self-join. Footnote rows (non-numeric MAC) are
filtered; proposed PW GPCIs are published **without** the statutory
1.0 work floor — values load as published and the notebook labels the
floor status rather than silently re-flooring. Loader lives in
`src/pfs/nprm.py` beside
`load_rvu_proposed`, discovers the workbook through the bib attachment
tagged `sup:<year>_PFS_NPRM` (same mechanism as Addendum B), writes
`cms.ingest_log` provenance, and deletes/reinserts only its own
partition. `fetch_nprm_addenda.py` CONFIGS gain an Addendum E
attach (title `CY{year} PFS NPRM Addendum E (proposed GPCIs)`) using
the existing title-guarded idempotent register.

Addendum D (GAFs) is **not** loaded — the GPCI components are the
primitive; GAF adds no information the notebook needs (YAGNI).

### D2 — primary-care code list to the lake

`publish_reference_to_lake.py` currently publishes same-named tables
from three schemas. Extend it with an explicit rename map and publish
`cms_provider_attribution._primary_care_hcpcs_codes` →
**`cms.primary_care_service_code`** (109 rows, `hcpcs_code`,
`description`). `pfs.gpci_proposed` joins the existing `pfs` list.
The publish runs in the notebooks container as documented in that
script's header.

### D3 — notebook changes

New **Section: GPCI & Locality** (between current Sections 2 and 3):
- Near-term: CY2027-proposed vs CY2026-final `work/pe/mp` GPCI deltas
  per locality — table of biggest movers + diverging-bar chart; counts
  of localities up/down/flat.
- Long-term: GPCI trend 2016–2026 (+2027 proposed, visually flagged
  "final rule pending") for the highest/lowest-GPCI localities and the
  floor story (work GPCI floor).
- Payment effect: locality-adjusted payment via `pfs.calcs.payment`
  for example codes (reusing the notebook's existing 99213/27447/70553
  trio), best vs worst locality vs national.

New **Section: Primary Care Services (ACO attribution set)**:
- Definition callout: the 109-code § 425.400(c) list, read live from
  `cms.primary_care_service_code`, with the disclosure that the
  current-law designation list is applied retrospectively across
  years (the list itself is not versioned in-repo).
- Near-term, **every code individually**: CY2026-final vs
  CY2027-proposed work/total RVUs and national payment (both CFs
  known), full 109-row sortable table + chart ranked by % payment
  change; codes present in only one side surfaced explicitly (new
  APCM-family codes have no CY2026 final RVU).
- Long-term, per code: total-RVU and dollar (RVU × that year's final
  CF) trajectories 2015–2026 with the 2027 proposed point appended and
  labeled; heatmap (code × year, % change) plus small-multiple lines
  for the codes with the largest cumulative moves.
- GPCI cross-cut: the near-term payment delta for a chosen
  primary-care code shown at the extreme localities (joins D1).
- Aggregation is unweighted (no utilization data in the lake);
  disclosed inline.

**Fixes to existing cells:** intro rewritten (GPCI = 1.0 disclaimer
replaced by a pointer to the GPCI section; the remaining
national-unadjusted figures keep a scoped note), provenance section
extended to `pfs.gpci`, `pfs.gpci_proposed` ingest-log rows and the
primary-care list's source note. Everything else (CF walk, RVU deltas,
APM section) stays.

### D4 — duckdb/DuckLake version alignment

Bump the project's duckdb to ≥1.5.4 (`uv lock`), so host tooling can
attach the migrated catalog again; verify the test suite and that the
monolith (written by 1.4.4) opens cleanly under 1.5.4. If the bump
breaks tests, defer with findings on the issue rather than forcing it
— nothing in P39's critical path requires it (loads target the
monolith; publish + notebook run in the notebooks container).

## Testing

- Loader: unit tests beside the existing `rvu_proposed` NPRM tests —
  header detection on the real CY2027 Addendum E layout, partition
  isolation (loading 2027 leaves the 1832-P partition untouched),
  numeric coercion.
- Publisher: rename-map unit test (pure function).
- Notebook: `nb_integration.py` single-notebook run in the production
  notebooks container must stay green; data-sanity assertions inside
  the notebook (e.g. 109 codes read, locality counts match) fail loud
  in that harness.

## Out of scope

- Utilization-weighted impact (no claims-volume reference in the lake).
- Addendum D / GAF ingestion.
- Versioning the primary-care designation list by rule year.
- Retiring the monolith (#514) — P39 adds one publisher entry, no more.

## P39 build outcomes (2026-08-17)

Issues #629–#633, milestone P39 (5 commits, 39dfaa2..afeec59, direct on
main — mail-poller live-mounts this checkout and Tasks ran against the
live monolith/lake). All closed same-day; notebook gate `pass=1` in the
prod notebooks container after each notebook change.

- **#629 (39dfaa2 + the Addendum-E register commit)**:
  `load_gpci_proposed` → `pfs.gpci_proposed` (long format with
  `gpci_year`; CMS-1832-P carries 2026+2027, CMS-1848-P carries 2027 —
  109 localities each; 2025 baseline restatement skipped; PW loaded
  floor-free as published). Addendum E attached in bib for both years
  (title-guarded, canonical storage paths). 3 new loader tests;
  `ingest_pfs.py --nprm` logs `pfs.gpci_proposed` ingest rows.
- **#630 (fc26d5f)**: publisher gained `gpci_proposed` + a rename map;
  `cms_provider_attribution._primary_care_hcpcs_codes` →
  `cms.primary_care_service_code` (109 rows). Verified from
  `connect.ducklake()` in the notebooks container.
- **#631/#632 (8c4e13c, one combined commit — the sections were a
  single splice)**: new Sections 3 (GPCI & locality) and 4 (primary
  care / attribution set); APM → 5, Provenance → 6; intro's "GPCI=1.0,
  does not join pfs.gpci" disclaimer replaced; provenance covers the
  gpci tables and sources the attribution list narratively.
- **#633 (afeec59)**: duckdb floor 1.0 → ≥1.5.4 (resolved 1.5.5); full
  suite green under the new pin; monolith + replica open cleanly;
  host-side `connect.ducklake()` no longer version-blocked (DNS to the
  compose-internal catalog remains, by design).
- **Findings worth keeping**: the CY2027 NPRM re-proposes the CY2026
  NPRM's 2027 GPCI phase-in for 103 of 109 localities (6 revised);
  every designated primary-care code that prices anywhere prices in
  *both* CY2026-final and CY2027-proposed tables — the 12 unmatched
  are all deleted legacy codes still on the § 425.400(c) list (the
  notebook's one-sided narrative was corrected against live data
  before commit).
- **Deviation**: #631/#632 shipped as one commit; Task-3 ingest ran via
  a targeted driver (same duckdb_batch/ingest_log/replica plumbing)
  instead of the full `ingest_pfs.py --nprm` reload, to avoid an
  unnecessary multi-minute `load_all` pass.