kert 39e0d2a143 feat(lake): M5 — notebooks read OPPS from the lake; ingest publishes it (closes #514) ...

Read path: conf.connect.ducklake() returns a read-only DuckDB
connection with the lake attached as the default database, so
'SELECT ... FROM opps.addendum_b' works unchanged. Auth is a dedicated
select-only postgres role (ducklake_ro, DUCKLAKE_RO_PASSWORD in the
notebooks container env) — notebooks never hold superuser creds, and
read-only is enforced at both the attach and the role. Both OPPS-
reading notebooks (skin_sub_pricing, skin_sub_cost_sharing) migrated
to lake queries; headless-verified zero cell errors in prod.

Write path cutover: ingest_opps.py publishes the OPPS tables to the
lake as its final step (docker-exec'd publish_opps_to_lake.py — the
postgres catalog and RustFS are compose-internal; --no-lake to skip).
Verified end-to-end: one host command now does monolith merge →
replica refresh → lake publish with read-back checks (219,665 +
7,776 + 2,520 rows, all OK).

The monolith's opps schema stays as a deprecated mirror for the
aco.pipe analytics graph and the read replica; documented removal
condition in docs/docs/duckdb-concurrency.md.

2026-07-10 23:30:36 -04:00

14 KiB

Raw Blame History

View Raw