Files
stack/docs/docs/duckdb-concurrency.md
kert 0bc53ccb7d fix(conf,llm,compose): the DuckDB replica lives in data/replica/ (refs P48)
The llm container mounted the whole ./data directory read-only just to
read one snapshot. Snapshots now get their own directory, published with
mkdir -p, and compose mounts only ./data/replica — the primary databases
are no longer visible inside the container.
2026-09-08 23:40:22 -04:00

4.4 KiB
Raw Blame History

title, sidebar_position
title sidebar_position
DuckDB concurrency & connection hygiene 90

DuckDB concurrency & connection hygiene

data/aco.duckdb is a single-writer store: one process holding any connection — even read-only — blocks every writer. The classic failure is a batch ingest dying with:

IOException: Could not set lock on file "data/aco.duckdb"

while a marimo notebook kernel quietly holds a connection from a cell that ran hours ago.

Writing: use conf.connect.duckdb_batch

Batch ingests should not call duckdb.connect() directly. The duckdb_batch() context manager preflights the lock — retrying with exponential backoff, and if the file is still held, failing with the holder PIDs/commands (via lsof) instead of a raw IOException:

from conf.connect import duckdb_batch

with duckdb_batch("aco") as con:      # retries, then names the lock holder
    con.execute("CREATE SCHEMA IF NOT EXISTS opps")
    ...
# connection is always closed here — the lock is released even on error

dev/scripts/ingest_opps.py and dev/scripts/ingest_asp.py use this.

Reading in notebooks: keep connections short-lived

  • conf.connect.duckdb() opens read-only by default — keep it that way in notebooks; never pass read_only=False from a notebook.
  • A read-only handle still blocks writers. Don't keep a module-level connection alive for the life of the kernel: open, query, and close within the cell, or wrap access in a small helper that closes after each query.
  • If an ingest reports the lock is held by a marimo PID, close that notebook tab (or stop the PID) and re-run — the notebook loses nothing; it reconnects on the next cell run.

Notebooks read a replica, not the primary

Read-only opens through conf.connect.duckdb() resolve to the snapshot data/replica/aco.ro.duckdb when it exists — so notebook kernels never hold the primary's lock at all, and ingests stop caring how many notebooks are open.

  • Refresh cadence: every ingest republishes the snapshot as its last step (conf.connect.publish_replica("aco") — holds the write lock, CHECKPOINTs, copies, then swaps atomically). Staleness is therefore bounded by ingest frequency; to refresh manually: uv run python -c "from conf.connect import publish_replica; publish_replica()"
  • Reading the live primary instead: pass replica=False to conf.connect.duckdb(), or set STACK_DUCKDB_REPLICA=0.
  • Kernels holding the old snapshot keep a valid file handle after a swap; re-running the connect cell picks up the fresh one.

Reference data (OPPS + PFS) is authoritative in the lake

Since M5 (#514), the OPPS and PFS reference tables live in DuckLake — postgres catalog (ducklake db) + Parquet on RustFS (s3://lakehouse/ducklake/), per the M3 decision record. Concurrency there is structural: the catalog serializes writers transactionally and readers get snapshot isolation.

  • Reading (notebooks / in-container code): conf.connect.ducklake() — read-only by default via the ducklake_ro postgres role (DUCKLAKE_RO_PASSWORD, in the notebooks container env). The lake is the connection's default database, so SELECT … FROM opps.addendum_b or pfs.rvu works unchanged. Compose-internal only — host-side code goes through docker exec.
  • Writing: aco.lake.DuckLakeContext (see dev/scripts/ publish_reference_to_lake.py). ingest_opps.py and ingest_pfs.py publish to the lake as their final step (--no-lake to skip).
  • The monolith's opps/pfs schemas remain as a deprecated mirror for the aco.pipe analytics graph, the read replica, and cross-source joins (e.g. pfs × skin_subs); drop them once those consumers migrate.
  • Cross-source queries (lake schema joined with a monolith-only schema like skin_subs) stay on the monolith connection until the joined schema also moves.

Per-year ingests merge, not wipe

ingest_opps.py --year YYYY used to DROP TABLE + CREATE ... AS from only that year's files, silently erasing every other year (#509). It now merges: existing rows for other years are preserved and the filtered year is replaced.

Roadmap

The strategic fixes (per-domain DB files, read replica, lake storage with snapshot isolation) are tracked as milestones M2–M5 in issues #510–#514, per docs/superpowers/specs/2026-07-08-duckdb-concurrency-streaming.md.