M5: migrate notebooks to lake reads; retire the monolith for reference data #514

New Issue

kert · 2026-07-08T19:41:25Z

kert commented

2026-07-08 19:41:25 +00:00

Point notebooks at the lake (DuckDB iceberg extension or Trino) for OPPS/PFS
reference data, cut ingestion over to the lake, and retire the local DuckDB
monolith as the reference-data store. Concurrent read/write becomes the norm;
the lock class of failures disappears.

Background: docs/superpowers/specs/2026-07-08-duckdb-concurrency-streaming.md (M5). Depends on M4.

Point notebooks at the lake (DuckDB iceberg extension or Trino) for OPPS/PFS reference data, cut ingestion over to the lake, and retire the local DuckDB monolith as the reference-data store. Concurrent read/write becomes the norm; the lock class of failures disappears. Background: [`docs/superpowers/specs/2026-07-08-duckdb-concurrency-streaming.md`](https://git.fhirworx.io/homelab/stack/src/branch/main/docs/superpowers/specs/2026-07-08-duckdb-concurrency-streaming.md) (M5). Depends on M4.

kert added the lakehouse label 2026-07-08 19:41:25 +00:00

kert referenced this issue

2026-07-11 02:47:04 +00:00

M2: split aco.duckdb into per-domain files + ATTACH #511

kert referenced this issue

2026-07-11 03:04:36 +00:00

M4: implement aco.lake write path + pilot OPPS ingestion to the lake #513

kert referenced this issue from a commit

2026-07-11 03:30:51 +00:00

feat(lake): M5 — notebooks read OPPS from the lake; ingest publishes it (closes #514)

kert closed this issue

2026-07-11 03:30:51 +00:00

kert commented

2026-07-11 03:31:42 +00:00

Implemented in 39e0d2a143 (on DuckLake per the M3 decision #512, not Trino/Iceberg as originally sketched):

Notebook reads come from the lake: conf.connect.ducklake() — read-only DuckDB connection with the lake as the default database (SELECT … FROM opps.addendum_b unchanged); auth via a dedicated select-only ducklake_ro postgres role in the notebooks container. Both OPPS-reading notebooks (skin_sub_pricing, skin_sub_cost_sharing) migrated and headless-verified in prod with zero cell errors. Concurrent read/write is now the norm for OPPS: catalog transactions + snapshot isolation, no file lock anywhere in the path.
Ingestion cut over: ingest_opps.py publishes to the lake as its final step (docker-exec'd, --no-lake opt-out). Verified end-to-end: one command does monolith merge → replica refresh → lake publish with read-back checks (219,665 / 7,776 / 2,520 rows OK).
Monolith retirement: the lake is authoritative for OPPS. The monolith's opps schema is retained as a deprecated mirror for two remaining consumers — the aco.pipe analytics graph (reads opps.apc_weight/wage_index in-monolith) and the read replica. Removal condition documented in docs/docs/duckdb-concurrency.md; dropping it is a small follow-up once aco.pipe reads the lake (it can use the same DuckLakeContext).

That completes the M1–M5 arc from 2026-07-08-duckdb-concurrency-streaming.md: preflight (#508), --year merge (#509), read replica (#510), split superseded (#511), DuckLake decision (#512), write path + pilot (#513), and this cutover.

Implemented in https://git.fhirworx.io/homelab/stack/commit/39e0d2a143ad27533c11a1c447dfa2eddd66ccd8 (on DuckLake per the M3 decision #512, not Trino/Iceberg as originally sketched): - **Notebook reads come from the lake**: `conf.connect.ducklake()` — read-only DuckDB connection with the lake as the default database (`SELECT … FROM opps.addendum_b` unchanged); auth via a dedicated select-only `ducklake_ro` postgres role in the notebooks container. Both OPPS-reading notebooks (skin_sub_pricing, skin_sub_cost_sharing) migrated and headless-verified in prod with zero cell errors. Concurrent read/write is now the norm for OPPS: catalog transactions + snapshot isolation, no file lock anywhere in the path. - **Ingestion cut over**: `ingest_opps.py` publishes to the lake as its final step (docker-exec'd, `--no-lake` opt-out). Verified end-to-end: one command does monolith merge → replica refresh → lake publish with read-back checks (219,665 / 7,776 / 2,520 rows OK). - **Monolith retirement**: the lake is authoritative for OPPS. The monolith's `opps` schema is retained as a *deprecated mirror* for two remaining consumers — the `aco.pipe` analytics graph (reads opps.apc_weight/wage_index in-monolith) and the read replica. Removal condition documented in [`docs/docs/duckdb-concurrency.md`](https://git.fhirworx.io/homelab/stack/src/branch/main/docs/docs/duckdb-concurrency.md); dropping it is a small follow-up once `aco.pipe` reads the lake (it can use the same `DuckLakeContext`). That completes the M1–M5 arc from `2026-07-08-duckdb-concurrency-streaming.md`: preflight (#508), --year merge (#509), read replica (#510), split superseded (#511), DuckLake decision (#512), write path + pilot (#513), and this cutover.

kert reopened this issue

2026-07-11 03:39:05 +00:00

kert referenced this issue from a commit

2026-07-11 03:48:01 +00:00

fix(lake): M5 covered OPPS only — build out PFS (refs #514)

kert commented

2026-08-13 23:46:55 +00:00

Status audit (2026-08-13, repo-verified): PARTIAL. Lake reads migrated for 4 notebooks (pfs_calcs, pfs_reconciliation, cy2026_pfs_proposed_rule, skin_sub_budget_neutrality); 6 still open connect.duckdb() (incl. _template.py:23) and 2 hybrid; ingestion remains monolith-first (ingest_opps.py writes DuckDB then publishes; publish_reference_to_lake.py is the M4 pilot shape). Body requires notebook migration + ingestion cutover + monolith retirement — roughly 1 of 3 complete.

**Status audit (2026-08-13, repo-verified):** PARTIAL. Lake reads migrated for 4 notebooks (pfs_calcs, pfs_reconciliation, cy2026_pfs_proposed_rule, skin_sub_budget_neutrality); 6 still open connect.duckdb() (incl. _template.py:23) and 2 hybrid; ingestion remains monolith-first (ingest_opps.py writes DuckDB then publishes; publish_reference_to_lake.py is the M4 pilot shape). Body requires notebook migration + ingestion cutover + monolith retirement — roughly 1 of 3 complete.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#514