feat(lake): M3 decision record — DuckLake over Iceberg (closes #512); fix Nessie Iceberg REST

Hands-on spike in the prod notebooks container against real OPPS data
(full table in docs/superpowers/specs/2026-07-10-ducklake-vs-iceberg-
decision.md):

- DuckLake (postgres catalog + Parquet on RustFS): 219k-row write in
  1.7s via one SQL statement; TRUE multi-writer (2 concurrent writers,
  0 retries — catalog transactions serialize commits); native
  snapshots/time travel; same DuckDB SQL interface notebooks and
  ingests already use.
- Iceberg (Nessie REST): works for pyiceberg/Trino, but optimistic
  concurrency needs client retry loops, and the notebook-native duckdb
  iceberg read path 403s — RustFS rejects Nessie-vended credentials.

Decision: DuckLake. Risks (young format, no external-engine interop)
accepted — data is plain Parquet on S3 and the Iceberg path stays as
fallback.

Also fixes two Nessie compose bugs the spike surfaced (the Iceberg
REST endpoint had NEVER worked): default-warehouse must be a NAME
referencing a defined warehouse, and S3 credentials must go through
the secrets manager as a URN (the ACCESS_KEY_ID env style is ignored
by current Nessie). Verified: warehouse config served, pyiceberg
create/append/scan and Trino reads all work now.

compose.yml

@@ -435,11 +435,21 @@ services:
       - QUARKUS_DATASOURCE_JDBC_URL=jdbc:postgresql://postgres:5432/nessie
       - QUARKUS_DATASOURCE_USERNAME=nessie
       - QUARKUS_DATASOURCE_PASSWORD=${NESSIE_DB_PASSWORD}
-      # Iceberg REST Catalog settings
+      # Iceberg REST Catalog settings.
-      - NESSIE_CATALOG_DEFAULT_WAREHOUSE=${S3_WAREHOUSE:-s3://lakehouse/}
+      # default-warehouse is a NAME that must reference a defined
+      # warehouse; pointing it straight at the s3 URI leaves the REST
+      # endpoint 500ing with "Default warehouse ... is not defined".
+      - NESSIE_CATALOG_DEFAULT_WAREHOUSE=warehouse
+      - nessie.catalog.warehouses.warehouse.location=${S3_WAREHOUSE:-s3://lakehouse/}
       - NESSIE_CATALOG_SERVICE_S3_DEFAULT_OPTIONS_ENDPOINT=${S3_ENDPOINT:-http://rustfs:9000}
-      - NESSIE_CATALOG_SERVICE_S3_DEFAULT_OPTIONS_ACCESS_KEY_ID=${NESSIE_S3_ACCESS_KEY}
+      # S3 credentials go through Nessie's secrets manager: the option
-      - NESSIE_CATALOG_SERVICE_S3_DEFAULT_OPTIONS_SECRET_ACCESS_KEY=${NESSIE_S3_SECRET_KEY}
+      # takes a URN referencing quarkus config keys with .name/.secret.
+      # (The former ACCESS_KEY_ID/SECRET_ACCESS_KEY env style is ignored
+      # by current Nessie — "Missing access key and secret for STATIC
+      # authentication mode".)
+      - nessie.catalog.service.s3.default-options.access-key=urn:nessie-secret:quarkus:s3creds
+      - s3creds.name=${NESSIE_S3_ACCESS_KEY}
+      - s3creds.secret=${NESSIE_S3_SECRET_KEY}
       - NESSIE_CATALOG_SERVICE_S3_DEFAULT_OPTIONS_PATH_STYLE_ACCESS=true
       - NESSIE_CATALOG_SERVICE_S3_DEFAULT_OPTIONS_REGION=${S3_REGION:-us-east-1}
       # OpenTelemetry tracing

docs/superpowers/specs/2026-07-10-ducklake-vs-iceberg-decision.md

@@ -0,0 +1,61 @@
+# DuckLake vs Iceberg for concurrent reference-data storage — decision record
+
+**Status:** Decided — **DuckLake** (postgres catalog + Parquet on RustFS)
+**Date:** 2026-07-10
+**Issue:** #512 (M3 of the DuckDB concurrency plan,
+`2026-07-08-duckdb-concurrency-streaming.md`)
+**Method:** hands-on spikes run inside the prod notebooks container against the
+deployed RustFS/Nessie/Trino/Postgres stack, using real OPPS reference data.
+
+## What was tested
+
+| Test | DuckLake | Iceberg (Nessie REST) |
+|---|---|---|
+| Attach/connect | 0.10 s (postgres catalog) | 0.02 s (REST config) |
+| Bulk write (`opps.addendum_b`, 219,665 rows → RustFS) | **1.7 s**, one SQL statement | create+append 3.1 s via pyiceberg (arrow API, not SQL) |
+| Read back | instant, plain DuckDB SQL | pyiceberg 0.04 s ✓ · Trino 0.6 s ✓ · **duckdb iceberg ATTACH ✗** (403: RustFS rejects Nessie-vended credentials) |
+| Concurrent writers (2 procs × 5 batches × 1000 rows) | **PASS, 0 retries** — catalog transactions serialize commits | PASS, but optimistic: 2 `CommitFailedException` retries needed client-side retry loops |
+| Concurrent reader during writes | never errored | never errored |
+| Snapshots / time travel | native (`lake.snapshots()`, `AT (VERSION => n)`) | native (Iceberg snapshots) |
+
+Spike scripts: `spike_ducklake.py`, `spike_ducklake_concurrent.py`,
+`spike_iceberg.py`, `spike_iceberg2.py` (session scratchpad; trivially
+reproducible from this table).
+
+## Decision: DuckLake
+
+1. **Same interface everywhere.** Ingests and notebooks already speak DuckDB
+   SQL. With DuckLake, an ingest is `INSERT INTO lake.addendum_b SELECT …` and
+   a notebook is `SELECT … FROM lake.addendum_b` — no new client library, no
+   API split between read and write paths. Iceberg's write path is pyiceberg
+   (arrow-level) or Trino SQL, both foreign to the existing codebase.
+2. **Multi-writer is stronger in practice.** DuckLake serializes commits
+   through catalog-database transactions: two concurrent bulk writers finished
+   with zero conflicts. Iceberg's optimistic concurrency worked but pushed
+   retry loops into every writer.
+3. **The notebook-native Iceberg read path is broken on this stack.** duckdb's
+   iceberg extension authenticates via catalog-vended credentials, which
+   RustFS rejects (403 on metadata avro). pyiceberg/Trino reads work, but the
+   whole point of M5 is notebooks reading the lake through DuckDB.
+4. **Fewer moving parts.** DuckLake needs postgres (already deployed — Nessie's
+   own store) + RustFS. The Iceberg path needs Nessie + Trino + Polaris +
+   per-client S3 config — and the spike found its REST endpoint had **never
+   worked** (two compose config bugs, fixed in this change: warehouse
+   definition + S3 secret URN).
+
+**Risks accepted:** DuckLake is a young format (DuckDB Labs, 2025) with a
+smaller ecosystem than Iceberg; external engines (Spark/Trino) can't query it
+natively today. Mitigations: data is plain Parquet on S3 (readable without the
+catalog in a pinch); the postgres catalog is tiny and rebuildable; Nessie/
+Trino/Polaris stay deployed, and the now-fixed Iceberg REST path remains a
+fallback if engine interop becomes a requirement.
+
+## Consequences (M4, #513)
+
+- `aco.lake` write path implements against DuckLake: `ATTACH
+  'ducklake:postgres:dbname=ducklake …' AS lake (DATA_PATH
+  's3://lakehouse/ducklake/')`.
+- Catalog database: `ducklake` in the existing postgres (the spike used
+  `ducklake_spike`; create the real one at M4).
+- Pilot: OPPS Addendum B ingestion dual-writes monolith + lake, then notebooks
+  cut reads over (M5, #514).