kert 71861d9d32 fix(lake): M5 covered OPPS only — build out PFS (refs #514) ...

The M5 close-out missed half the issue's scope: #514 says 'OPPS/PFS
reference data' and I cut over only OPPS, leaving PFS — the largest
reference domain, 23.5M rows across 8 tables — entirely on the
monolith, including pfs.* queries in the very notebook whose OPPS
query was migrated. This completes PFS the same way:

- publish_opps_to_lake.py → publish_reference_to_lake.py with a
  schema registry (opps: 3 tables, pfs: 8); host-side docker-exec
  wrapper extracted to dev/scripts/_lake.py, shared by the ingests.
- PFS published to the lake and read-back verified: carrier_locality
  21,863,770 rows in 10.1s, plus rvu/gpci/clinical_labor/medical_
  equipment/medical_supply/physician_work_time/zip_carrier_locality.
- New dev/scripts/ingest_pfs.py wraps pfs.pipe.load_all (previously
  ad-hoc, no entrypoint) with the standard plumbing: duckdb_batch
  preflight, replica refresh, lake publish.
- 5 notebooks migrated: pfs_calcs, pfs_reconciliation,
  skin_sub_budget_neutrality read the lake as their primary
  connection; skin_sub_pricing and skin_sub_cost_sharing switch their
  pure-pfs cells to the lake. The one cross-source join
  (pfs × skin_subs) stays on the monolith mirror, annotated.
  All 5 headless-verified in prod: zero cell errors.
- pfs_calcs leaves the pre-commit host-run safe list (the lake catalog
  is compose-internal); the nightly integration covers it in-container.

2026-07-10 23:47:51 -04:00

1.8 KiB

Raw Blame History

View Raw