M4: implement aco.lake write path + pilot OPPS ingestion to the lake #513

New Issue

kert · 2026-07-08T19:41:25Z

kert commented

2026-07-08 19:41:25 +00:00

Implement the aco.lake Context write path (currently Context.load/save and
_execute_transpiled raise NotImplementedError) and pilot OPPS Addendum B
ingestion to the lake store chosen in M3 (Iceberg via Nessie/RustFS, or
DuckLake). Concurrent notebook readers + batch writers coexist via snapshot
isolation — no file lock.

Background: docs/superpowers/specs/2026-07-08-duckdb-concurrency-streaming.md (option E). Depends on M3. Also closes the standing
aco.lake remote-execution gap.

Implement the `aco.lake` `Context` write path (currently `Context.load/save` and `_execute_transpiled` raise `NotImplementedError`) and pilot OPPS Addendum B ingestion to the lake store chosen in M3 (Iceberg via Nessie/RustFS, or DuckLake). Concurrent notebook readers + batch writers coexist via snapshot isolation — no file lock. Background: `docs/superpowers/specs/2026-07-08-duckdb-concurrency-streaming.md` (option E). Depends on M3. Also closes the standing `aco.lake` remote-execution gap.

kert added the lakehouse label 2026-07-08 19:41:25 +00:00

kert referenced this issue

2026-07-11 02:57:27 +00:00

M3: spike DuckLake vs Iceberg for concurrent reference-data storage #512

kert referenced this issue from a commit

2026-07-11 03:04:08 +00:00

feat(lake): M4 — DuckLakeContext write path + OPPS lake pilot (closes #513)

kert closed this issue

2026-07-11 03:04:08 +00:00

kert commented

2026-07-11 03:04:36 +00:00

Implemented in 446f1b9398, per the M3 decision (#512: DuckLake, not Iceberg):

Write path: aco.lake.DuckLakeContext — subclasses DuckDBContext, so load/save (append/replace, catalog mapping, narwhals frames) are inherited; only the connection wiring differs (ducklake + httpfs extensions, RustFS S3 secret, ATTACH the catalog as the default database). Postgres DSN in prod, file catalog in tests. This resolves the standing write-path gap: there is now a concrete, write-capable lake context.
Pilot ran against prod: dev/scripts/publish_opps_to_lake.py published the OPPS reference tables through Context.save(mode="replace") to the ducklake postgres catalog + s3://lakehouse/ducklake/ — addendum_b 219,665 rows in 2.6s, apc_weight 7,776, skin_sub_addendum_b 2,520, all read-back verified through Context.load.
Config: stack.toml [lake.ducklake]; 6 tests on a local file catalog (same code paths, no services); CI preinstalls the ducklake extension (same race class as #515).

Notes: the issue mentioned Iceberg via Nessie as one option — the spike chose DuckLake, and Nessie's REST endpoint was fixed as a fallback (#512). engine._execute_transpiled (remote Trino/Databricks execution) is orthogonal to this path — DuckLake executes locally in DuckDB — and stays tracked by the enterprise-context work. Next: M5 #514 (notebooks read the lake, cut ingestion over, retire the monolith for reference data).

Implemented in https://git.fhirworx.io/homelab/stack/commit/446f1b939850f466fbdaeb9fcc0eb1e1b77a6376, per the M3 decision (#512: DuckLake, not Iceberg): - **Write path**: `aco.lake.DuckLakeContext` — subclasses `DuckDBContext`, so `load`/`save` (append/replace, catalog mapping, narwhals frames) are inherited; only the connection wiring differs (ducklake + httpfs extensions, RustFS S3 secret, ATTACH the catalog as the default database). Postgres DSN in prod, file catalog in tests. This resolves the standing write-path gap: there is now a concrete, write-capable lake context. - **Pilot ran against prod**: `dev/scripts/publish_opps_to_lake.py` published the OPPS reference tables through `Context.save(mode="replace")` to the `ducklake` postgres catalog + `s3://lakehouse/ducklake/` — `addendum_b` 219,665 rows in 2.6s, `apc_weight` 7,776, `skin_sub_addendum_b` 2,520, all read-back verified through `Context.load`. - Config: `stack.toml [lake.ducklake]`; 6 tests on a local file catalog (same code paths, no services); CI preinstalls the ducklake extension (same race class as #515). Notes: the issue mentioned Iceberg via Nessie as one option — the spike chose DuckLake, and Nessie's REST endpoint was fixed as a fallback (#512). `engine._execute_transpiled` (remote Trino/Databricks execution) is orthogonal to this path — DuckLake executes locally in DuckDB — and stays tracked by the enterprise-context work. Next: M5 #514 (notebooks read the lake, cut ingestion over, retire the monolith for reference data).

kert referenced this issue

2026-07-11 03:31:42 +00:00

M5: migrate notebooks to lake reads; retire the monolith for reference data #514

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#513