The llm container mounted the whole ./data directory read-only just to read one snapshot. Snapshots now get their own directory, published with mkdir -p, and compose mounts only ./data/replica — the primary databases are no longer visible inside the container.
4.4 KiB
title, sidebar_position
| title | sidebar_position |
|---|---|
| DuckDB concurrency & connection hygiene | 90 |
DuckDB concurrency & connection hygiene
data/aco.duckdb is a single-writer store: one process holding any
connection — even read-only — blocks every writer. The classic failure is a
batch ingest dying with:
IOException: Could not set lock on file "data/aco.duckdb"
while a marimo notebook kernel quietly holds a connection from a cell that ran hours ago.
Writing: use conf.connect.duckdb_batch
Batch ingests should not call duckdb.connect() directly. The
duckdb_batch() context manager preflights the lock — retrying with
exponential backoff, and if the file is still held, failing with the holder
PIDs/commands (via lsof) instead of a raw IOException:
from conf.connect import duckdb_batch
with duckdb_batch("aco") as con: # retries, then names the lock holder
con.execute("CREATE SCHEMA IF NOT EXISTS opps")
...
# connection is always closed here — the lock is released even on error
dev/scripts/ingest_opps.py and dev/scripts/ingest_asp.py use this.
Reading in notebooks: keep connections short-lived
conf.connect.duckdb()opens read-only by default — keep it that way in notebooks; never passread_only=Falsefrom a notebook.- A read-only handle still blocks writers. Don't keep a module-level connection alive for the life of the kernel: open, query, and close within the cell, or wrap access in a small helper that closes after each query.
- If an ingest reports the lock is held by a
marimoPID, close that notebook tab (or stop the PID) and re-run — the notebook loses nothing; it reconnects on the next cell run.
Notebooks read a replica, not the primary
Read-only opens through conf.connect.duckdb() resolve to the snapshot
data/replica/aco.ro.duckdb when it exists — so notebook kernels never hold the
primary's lock at all, and ingests stop caring how many notebooks are open.
- Refresh cadence: every ingest republishes the snapshot as its last step
(
conf.connect.publish_replica("aco")— holds the write lock,CHECKPOINTs, copies, then swaps atomically). Staleness is therefore bounded by ingest frequency; to refresh manually:uv run python -c "from conf.connect import publish_replica; publish_replica()" - Reading the live primary instead: pass
replica=Falsetoconf.connect.duckdb(), or setSTACK_DUCKDB_REPLICA=0. - Kernels holding the old snapshot keep a valid file handle after a swap; re-running the connect cell picks up the fresh one.
Reference data (OPPS + PFS) is authoritative in the lake
Since M5 (#514), the OPPS and PFS reference tables live in DuckLake —
postgres catalog (ducklake db) + Parquet on RustFS
(s3://lakehouse/ducklake/), per the M3 decision record. Concurrency
there is structural: the catalog serializes writers transactionally and
readers get snapshot isolation.
- Reading (notebooks / in-container code):
conf.connect.ducklake()— read-only by default via theducklake_ropostgres role (DUCKLAKE_RO_PASSWORD, in the notebooks container env). The lake is the connection's default database, soSELECT … FROM opps.addendum_borpfs.rvuworks unchanged. Compose-internal only — host-side code goes throughdocker exec. - Writing:
aco.lake.DuckLakeContext(seedev/scripts/ publish_reference_to_lake.py).ingest_opps.pyandingest_pfs.pypublish to the lake as their final step (--no-laketo skip). - The monolith's
opps/pfsschemas remain as a deprecated mirror for theaco.pipeanalytics graph, the read replica, and cross-source joins (e.g. pfs × skin_subs); drop them once those consumers migrate. - Cross-source queries (lake schema joined with a monolith-only schema like
skin_subs) stay on the monolith connection until the joined schema also moves.
Per-year ingests merge, not wipe
ingest_opps.py --year YYYY used to DROP TABLE + CREATE ... AS from only
that year's files, silently erasing every other year (#509). It now merges:
existing rows for other years are preserved and the filtered year is replaced.
Roadmap
The strategic fixes (per-domain DB files, read replica, lake storage with
snapshot isolation) are tracked as milestones M2–M5 in issues
#510–#514,
per docs/superpowers/specs/2026-07-08-duckdb-concurrency-streaming.md.