stack/docs/superpowers/specs/2026-07-08-duckdb-concurrency-streaming.md

DuckDB concurrency & data streaming — strategy

Status: Draft (awaiting user approval) Date: 2026-07-08 Trigger: The OPPS Addendum B re-ingest (dev/scripts/ingest_opps.py) failed with IOException: Could not set lock on file "data/aco.duckdb": Conflicting lock is held ... because a long-running marimo notebook kernel held a read connection to the shared DuckDB file.

Problem

data/aco.duckdb is a single ~3.2 GB DuckDB file shared by every schema (aco, opps, pfs, bib, …). DuckDB's process-level locking allows either one read-write connection or multiple read-only connections — a read-write open requires exclusive access. So any open handle, even a read-only one, blocks a writer.

Our readers are long-lived: the marimo edit server (:2718, uptime measured in days) spawns a kernel per open notebook, and a notebook cell that calls conf.connect.duckdb() (default read_only=True) keeps that connection for the lifetime of the kernel. Result: every batch ingest or pipeline write (ingest_opps.py, PFS ingest, aco.pipe runs) fails whenever any notebook is open against the DB. This is not OPPS-specific — it blocks all data-refresh workflows, and it will recur.

Root causes

1. Monolith + broad locking. One file for all schemas means a reader of any schema blocks a writer of any other schema.
2. Connection lifetime. conf.connect.duckdb() returns a bare connection; notebooks hold it open indefinitely instead of using a short-lived scope.
3. No writer coordination. Ingest scripts open read-write directly and surface DuckDB's raw IOException with no detection, retry, or guidance.

Options considered

#	Option	Effort	Concurrency	Notes
A	Ingest-then-swap the whole file	M	none	Would copy 3.2 GB per ingest; readers keep stale inode until reconnect. Rejected for the monolith.
B	Preflight lock detection + retry + connection hygiene	S	none	Turns cryptic failure into guidance; shrinks the lock window. Necessary regardless.
C	Notebook read-replica (aco.ro.duckdb) refreshed post-ingest	M	reader/writer decoupled	Notebooks never block writers; staleness bounded by refresh cadence. Cheap, effective.
D	Split aco.duckdb into per-domain files (opps, pfs, …) + ATTACH	M	narrows blast radius	A domain ingest only conflicts with readers of that domain.
E	Iceberg lake context (Nessie + RustFS + Trino/Polaris — already deployed)	L	full snapshot isolation	The stack's intended concurrent read/write path. Depends on completing aco.lake write path (currently NotImplementedError).
F	DuckLake (DuckDB catalog + Parquet on S3/RustFS)	L	ACID multi-writer	Keeps the DuckDB SQL interface; lighter than full Iceberg/Trino. Spike vs E.
G	MotherDuck (hosted)	—	full	External SaaS; conflicts with the self-hosted ethos. Rejected.

Recommended path

• Now (unblock + stop the bleeding): B — a lock-aware preflight in the ingest scripts (detect the holder via lsof, retry with backoff, and emit an actionable message naming the notebook/PID) plus a short-lived conf.connect context manager and notebook guidance.
• Near term (decouple readers): C — publish a read-only snapshot (aco.ro.duckdb) after each ingest and point notebooks / conf.connect.duckdb at it. Optionally D to narrow write contention further.
• Strategic (true concurrency): spike F (DuckLake) vs E (Iceberg), then implement the aco.lake write path and pilot OPPS ingestion to the lake, finally migrate notebooks to read from the lake and retire the monolith for reference data. This also unblocks the already-flagged aco.lake / _execute_transpiled NotImplementedError work.

Milestones → issues

• M1 / #A — Ingest preflight: lock detection, retry-with-backoff, actionable error; conf.connect.duckdb context manager + notebook connection guidance. (S)
• M1 / #B — Fix ingest_opps.py --year footgun: with --year it DROPs and rebuilds opps.addendum_b from only that year, wiping the others. Make it a per-year merge/upsert (delete-that-year + insert), not a full replace. (S)
• M2 / #C — Notebook read-replica aco.ro.duckdb: publish post-ingest, point notebooks at it, define refresh cadence. (M)
• M2 / #D — Split aco.duckdb into per-domain files + ATTACH for cross-domain queries; update conf.path/conf.connect. (M)
• M3 / #E — Spike DuckLake vs Iceberg for concurrent reference-data storage; decision record. (M)
• M4 / #F — Implement aco.lake Context write path (resolve NotImplementedError) + pilot OPPS ingestion to Iceberg/DuckLake via Nessie/RustFS. (L)
• M5 / #G — Migrate notebooks to read OPPS/PFS from the lake (iceberg/trino); cut over ingestion; retire the monolith for reference data. (L)

Interim operating procedure (until M1 lands)

Before a reference-data ingest, ensure no notebook holds the DB:

lsof data/aco.duckdb          # find the holder (usually a marimo kernel)
# close the notebook tab, or stop the specific kernel PID, then:
uv run python dev/scripts/ingest_opps.py

The discrete issues above are ready to file via dev/scripts/file_concurrency_issues.py (needs GITEA_TOKEN).