b4b25a3bc2
Some checks failed
CI / lint (push) Successful in 37s
CI / notebooks-smoke (push) Successful in 1m25s
Deploy / notebooks (push) Has been skipped
Deploy / zotero (push) Has been skipped
Deploy / docs (push) Has been skipped
Deploy / api (push) Has been skipped
Deploy / mc (push) Has been skipped
Infra CI / notebooks (push) Successful in 50s
Infra CI / zotero (push) Successful in 13s
Infra CI / docs (push) Successful in 15s
Infra CI / api (push) Successful in 13s
Infra CI / mc (push) Successful in 19s
Deploy / report (push) Successful in 16s
CI / test (push) Has been cancelled
Hands-on spike in the prod notebooks container against real OPPS data (full table in docs/superpowers/specs/2026-07-10-ducklake-vs-iceberg- decision.md): - DuckLake (postgres catalog + Parquet on RustFS): 219k-row write in 1.7s via one SQL statement; TRUE multi-writer (2 concurrent writers, 0 retries — catalog transactions serialize commits); native snapshots/time travel; same DuckDB SQL interface notebooks and ingests already use. - Iceberg (Nessie REST): works for pyiceberg/Trino, but optimistic concurrency needs client retry loops, and the notebook-native duckdb iceberg read path 403s — RustFS rejects Nessie-vended credentials. Decision: DuckLake. Risks (young format, no external-engine interop) accepted — data is plain Parquet on S3 and the Iceberg path stays as fallback. Also fixes two Nessie compose bugs the spike surfaced (the Iceberg REST endpoint had NEVER worked): default-warehouse must be a NAME referencing a defined warehouse, and S3 credentials must go through the secrets manager as a URN (the ACCESS_KEY_ID env style is ignored by current Nessie). Verified: warehouse config served, pyiceberg create/append/scan and Trino reads all work now.