stack/docs/superpowers/specs/2026-07-08-duckdb-concurrency-streaming.md

# DuckDB concurrency & data streaming — strategy

**Status:** Draft (awaiting user approval)
**Date:** 2026-07-08
**Trigger:** The OPPS Addendum B re-ingest (`dev/scripts/ingest_opps.py`) failed
with `IOException: Could not set lock on file "data/aco.duckdb": Conflicting
lock is held ...` because a long-running marimo notebook kernel held a
read connection to the shared DuckDB file.

## Problem

`data/aco.duckdb` is a single ~3.2 GB DuckDB file shared by every schema
(`aco`, `opps`, `pfs`, `bib`, …). DuckDB's process-level locking allows
**either** one read-write connection **or** multiple read-only connections —
a read-write open requires exclusive access. So **any** open handle, even a
read-only one, blocks a writer.

Our readers are long-lived: the `marimo edit` server (`:2718`, uptime measured
in **days**) spawns a kernel per open notebook, and a notebook cell that calls
`conf.connect.duckdb()` (default `read_only=True`) keeps that connection for
the **lifetime of the kernel**. Result: every batch ingest or pipeline write
(`ingest_opps.py`, PFS ingest, `aco.pipe` runs) fails whenever any notebook is
open against the DB. This is not OPPS-specific — it blocks **all** data-refresh
workflows, and it will recur.

### Root causes
1. **Monolith + broad locking.** One file for all schemas means a reader of
   *any* schema blocks a writer of *any other* schema.
2. **Connection lifetime.** `conf.connect.duckdb()` returns a bare connection;
   notebooks hold it open indefinitely instead of using a short-lived scope.
3. **No writer coordination.** Ingest scripts open read-write directly and
   surface DuckDB's raw `IOException` with no detection, retry, or guidance.

## Options considered

| # | Option | Effort | Concurrency | Notes |
|---|--------|--------|-------------|-------|
| A | Ingest-then-swap the whole file | M | none | Would copy 3.2 GB per ingest; readers keep stale inode until reconnect. Rejected for the monolith. |
| B | Preflight lock detection + retry + connection hygiene | S | none | Turns cryptic failure into guidance; shrinks the lock window. Necessary regardless. |
| C | Notebook **read-replica** (`aco.ro.duckdb`) refreshed post-ingest | M | reader/writer decoupled | Notebooks never block writers; staleness bounded by refresh cadence. Cheap, effective. |
| D | **Split** `aco.duckdb` into per-domain files (`opps`, `pfs`, …) + `ATTACH` | M | narrows blast radius | A domain ingest only conflicts with readers of that domain. |
| E | **Iceberg** lake context (Nessie + RustFS + Trino/Polaris — already deployed) | L | full snapshot isolation | The stack's intended concurrent read/write path. Depends on completing `aco.lake` write path (currently `NotImplementedError`). |
| F | **DuckLake** (DuckDB catalog + Parquet on S3/RustFS) | L | ACID multi-writer | Keeps the DuckDB SQL interface; lighter than full Iceberg/Trino. Spike vs E. |
| G | MotherDuck (hosted) | — | full | External SaaS; conflicts with the self-hosted ethos. Rejected. |

## Recommended path

- **Now (unblock + stop the bleeding):** **B** — a lock-aware preflight in the
  ingest scripts (detect the holder via `lsof`, retry with backoff, and emit an
  actionable message naming the notebook/PID) plus a short-lived
  `conf.connect` context manager and notebook guidance.
- **Near term (decouple readers):** **C** — publish a read-only snapshot
  (`aco.ro.duckdb`) after each ingest and point notebooks / `conf.connect.duckdb`
  at it. Optionally **D** to narrow write contention further.
- **Strategic (true concurrency):** spike **F (DuckLake)** vs **E (Iceberg)**,
  then implement the `aco.lake` write path and pilot OPPS ingestion to the lake,
  finally migrate notebooks to read from the lake and retire the monolith for
  reference data. This also unblocks the already-flagged
  `aco.lake` / `_execute_transpiled` `NotImplementedError` work.

## Milestones → issues

- **M1 / #A** — Ingest preflight: lock detection, retry-with-backoff, actionable
  error; `conf.connect.duckdb` context manager + notebook connection guidance. (S)
- **M1 / #B** — Fix `ingest_opps.py --year` footgun: with `--year` it `DROP`s and
  rebuilds `opps.addendum_b` from only that year, wiping the others. Make it a
  per-year merge/upsert (delete-that-year + insert), not a full replace. (S)
- **M2 / #C** — Notebook read-replica `aco.ro.duckdb`: publish post-ingest, point
  notebooks at it, define refresh cadence. (M)
- **M2 / #D** — Split `aco.duckdb` into per-domain files + `ATTACH` for
  cross-domain queries; update `conf.path`/`conf.connect`. (M)
- **M3 / #E** — Spike DuckLake vs Iceberg for concurrent reference-data storage;
  decision record. (M)
- **M4 / #F** — Implement `aco.lake` `Context` write path (resolve
  `NotImplementedError`) + pilot OPPS ingestion to Iceberg/DuckLake via
  Nessie/RustFS. (L)
- **M5 / #G** — Migrate notebooks to read OPPS/PFS from the lake (iceberg/trino);
  cut over ingestion; retire the monolith for reference data. (L)

## Interim operating procedure (until M1 lands)

Before a reference-data ingest, ensure no notebook holds the DB:

```bash
lsof data/aco.duckdb          # find the holder (usually a marimo kernel)
# close the notebook tab, or stop the specific kernel PID, then:
uv run python dev/scripts/ingest_opps.py
```

The discrete issues above are ready to file via
`dev/scripts/file_concurrency_issues.py` (needs `GITEA_TOKEN`).