homelab/stack
1
0
Fork 0
Code Issues 79 Pull Requests 1 Actions 1 Packages Projects 1 Releases Wiki Activity

M3: spike DuckLake vs Iceberg for concurrent reference-data storage #512

New Issue
Closed
opened 2026-07-08 19:41:25 +00:00 by kert · 1 comment
kert commented 2026-07-08 19:41:25 +00:00
Owner
Copy Link

Evaluate the two concurrent-read/write options for CMS reference data
(OPPS/PFS/etc.), both on the already-deployed RustFS object store:

  • Iceberg via the existing Nessie + Trino + Polaris lake context
    (stack.toml [context.lake]): snapshot isolation, optimistic concurrency.
  • DuckLake: DuckDB's catalog + Parquet lakehouse — ACID multi-writer while
    keeping the DuckDB SQL interface (lighter than full Iceberg/Trino).

Deliver a decision record: which becomes the concurrent store, and why.

Background: docs/superpowers/specs/2026-07-08-duckdb-concurrency-streaming.md (options E, F). Feeds M4.

Evaluate the two concurrent-read/write options for CMS reference data (OPPS/PFS/etc.), both on the already-deployed RustFS object store: - **Iceberg** via the existing Nessie + Trino + Polaris lake context (`stack.toml [context.lake]`): snapshot isolation, optimistic concurrency. - **DuckLake**: DuckDB's catalog + Parquet lakehouse — ACID multi-writer while keeping the DuckDB SQL interface (lighter than full Iceberg/Trino). Deliver a decision record: which becomes the concurrent store, and why. Background: `docs/superpowers/specs/2026-07-08-duckdb-concurrency-streaming.md` (options E, F). Feeds M4.
kert added the lakehouseinfra labels 2026-07-08 19:41:25 +00:00
kert closed this issue 2026-07-11 02:56:56 +00:00
kert commented 2026-07-11 02:57:27 +00:00
Author
Owner
Copy Link

Spike complete — decision: DuckLake (postgres catalog + Parquet on RustFS). Full decision record: docs/superpowers/specs/2026-07-10-ducklake-vs-iceberg-decision.md (b4b25a3bc2).

Hands-on results (prod notebooks container, real OPPS data):

DuckLake Iceberg (Nessie REST)
219k-row write to RustFS 1.7s, one SQL statement 3.1s via pyiceberg (arrow API)
2 concurrent writers PASS, 0 retries (catalog txns) PASS, 2 retries (optimistic, client retry loops)
notebook-native duckdb read ✓ ✗ 403 — RustFS rejects Nessie-vended creds
snapshots / time travel ✓ ✓
moving parts postgres (already deployed) + RustFS Nessie + Trino + Polaris + per-client S3 config

Bonus finding: the Nessie Iceberg REST endpoint had never worked — two compose config bugs (warehouse must be a defined name, not the s3 URI; S3 creds must be a secrets-manager URN). Both fixed and verified in the same commit, so the Iceberg path now genuinely works as the fallback.

Feeds M4 (#513): implement the aco.lake write path against DuckLake, ducklake catalog db in the existing postgres, pilot OPPS Addendum B.

Spike complete — **decision: DuckLake** (postgres catalog + Parquet on RustFS). Full decision record: `docs/superpowers/specs/2026-07-10-ducklake-vs-iceberg-decision.md` (https://git.fhirworx.io/homelab/stack/commit/b4b25a3bc2ab21fdcd66e11b3bf2265f9ec28670). Hands-on results (prod notebooks container, real OPPS data): | | DuckLake | Iceberg (Nessie REST) | |---|---|---| | 219k-row write to RustFS | **1.7s, one SQL statement** | 3.1s via pyiceberg (arrow API) | | 2 concurrent writers | **PASS, 0 retries** (catalog txns) | PASS, 2 retries (optimistic, client retry loops) | | notebook-native duckdb read | ✓ | ✗ 403 — RustFS rejects Nessie-vended creds | | snapshots / time travel | ✓ | ✓ | | moving parts | postgres (already deployed) + RustFS | Nessie + Trino + Polaris + per-client S3 config | Bonus finding: the Nessie Iceberg REST endpoint had **never worked** — two compose config bugs (warehouse must be a defined *name*, not the s3 URI; S3 creds must be a secrets-manager URN). Both fixed and verified in the same commit, so the Iceberg path now genuinely works as the fallback. Feeds M4 (#513): implement the `aco.lake` write path against DuckLake, `ducklake` catalog db in the existing postgres, pilot OPPS Addendum B.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
api
REST API (FastAPI)
 auth
Authentication and authorization
 bib
Bibliography, Zotero, citations
 ci
CI/CD pipelines and Woodpecker plugins
 cli
CLI commands and UX
 cloud
Cloud provider abstraction (AWS, GCP, Azure, self-hosted)
 config
Configuration and wiring
 data
Data loading, CCLF, BCDA, seeds
 databricks
Databricks jobs and deployment
 docs
Documentation and generated references
 image:api
image:docs
image:mc
image:notebooks
image:zotero
infra
lakehouse
Nessie, Polaris, Trino, Iceberg
 llm
llm module — local RAG over comments + Zotero corpus
 notebooks
marimo notebook errors (auto-filed by nb-watcher/nb-integration)
 packaging
Python packaging, optional deps, build system
 pfs
pipeline
Pipeline execution and validation
 pipeline:ci
pipeline:deploy
pipeline:harden
pipeline:infra-ci
pipeline:rebuild-all
pipeline:release
qpp
quality
Testing, coverage, error handling
 step:build
step:deploy
step:hadolint
step:lint
step:publish
step:scan
step:test
step:validate
styles
Charts, figures, visual theming
 tree
Project directory structure and file organization
No Label infra lakehouse
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
kert renovate-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: homelab/stack#512
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?