71861d9d32
All checks were successful
CI / lint (push) Successful in 29s
CI / notebooks-smoke (push) Successful in 1m25s
Deploy / notebooks (push) Has been skipped
Deploy / zotero (push) Has been skipped
Deploy / docs (push) Has been skipped
Deploy / api (push) Has been skipped
Deploy / mc (push) Has been skipped
Infra CI / notebooks (push) Successful in 56s
Infra CI / zotero (push) Successful in 13s
Infra CI / docs (push) Successful in 1m14s
Infra CI / api (push) Successful in 50s
Infra CI / mc (push) Successful in 19s
Deploy / report (push) Successful in 13s
CI / test (push) Successful in 14m27s
Harden / build-scan-report (push) Successful in 26m15s
Renovate / renovate (push) Successful in 15s
Notebooks Integration / notebooks-integration (push) Successful in 7m16s
Zotero Sync / zotero-sync (push) Successful in 53s
Package Supply Chain / pkg-supply-chain (push) Successful in 58s
The M5 close-out missed half the issue's scope: #514 says 'OPPS/PFS reference data' and I cut over only OPPS, leaving PFS — the largest reference domain, 23.5M rows across 8 tables — entirely on the monolith, including pfs.* queries in the very notebook whose OPPS query was migrated. This completes PFS the same way: - publish_opps_to_lake.py → publish_reference_to_lake.py with a schema registry (opps: 3 tables, pfs: 8); host-side docker-exec wrapper extracted to dev/scripts/_lake.py, shared by the ingests. - PFS published to the lake and read-back verified: carrier_locality 21,863,770 rows in 10.1s, plus rvu/gpci/clinical_labor/medical_ equipment/medical_supply/physician_work_time/zip_carrier_locality. - New dev/scripts/ingest_pfs.py wraps pfs.pipe.load_all (previously ad-hoc, no entrypoint) with the standard plumbing: duckdb_batch preflight, replica refresh, lake publish. - 5 notebooks migrated: pfs_calcs, pfs_reconciliation, skin_sub_budget_neutrality read the lake as their primary connection; skin_sub_pricing and skin_sub_cost_sharing switch their pure-pfs cells to the lake. The one cross-source join (pfs × skin_subs) stays on the monolith mirror, annotated. All 5 headless-verified in prod: zero cell errors. - pfs_calcs leaves the pre-commit host-run safe list (the lake catalog is compose-internal); the nightly integration covers it in-container.
100 lines
4.4 KiB
Markdown
100 lines
4.4 KiB
Markdown
---
|
||
title: DuckDB concurrency & connection hygiene
|
||
sidebar_position: 90
|
||
---
|
||
|
||
# DuckDB concurrency & connection hygiene
|
||
|
||
`data/aco.duckdb` is a **single-writer** store: one process holding *any*
|
||
connection — even read-only — blocks every writer. The classic failure is a
|
||
batch ingest dying with:
|
||
|
||
```
|
||
IOException: Could not set lock on file "data/aco.duckdb"
|
||
```
|
||
|
||
while a marimo notebook kernel quietly holds a connection from a cell that ran
|
||
hours ago.
|
||
|
||
## Writing: use `conf.connect.duckdb_batch`
|
||
|
||
Batch ingests should not call `duckdb.connect()` directly. The
|
||
`duckdb_batch()` context manager preflights the lock — retrying with
|
||
exponential backoff, and if the file is still held, failing with the holder
|
||
PIDs/commands (via `lsof`) instead of a raw `IOException`:
|
||
|
||
```python
|
||
from conf.connect import duckdb_batch
|
||
|
||
with duckdb_batch("aco") as con: # retries, then names the lock holder
|
||
con.execute("CREATE SCHEMA IF NOT EXISTS opps")
|
||
...
|
||
# connection is always closed here — the lock is released even on error
|
||
```
|
||
|
||
`dev/scripts/ingest_opps.py` and `dev/scripts/ingest_asp.py` use this.
|
||
|
||
## Reading in notebooks: keep connections short-lived
|
||
|
||
- `conf.connect.duckdb()` opens **read-only by default** — keep it that way in
|
||
notebooks; never pass `read_only=False` from a notebook.
|
||
- A read-only handle still blocks writers. Don't keep a module-level
|
||
connection alive for the life of the kernel: open, query, and close within
|
||
the cell, or wrap access in a small helper that closes after each query.
|
||
- If an ingest reports the lock is held by a `marimo` PID, close that notebook
|
||
tab (or stop the PID) and re-run — the notebook loses nothing; it reconnects
|
||
on the next cell run.
|
||
|
||
## Notebooks read a replica, not the primary
|
||
|
||
Read-only opens through `conf.connect.duckdb()` resolve to the snapshot
|
||
`data/aco.ro.duckdb` when it exists — so notebook kernels never hold the
|
||
primary's lock at all, and ingests stop caring how many notebooks are open.
|
||
|
||
- **Refresh cadence**: every ingest republishes the snapshot as its last step
|
||
(`conf.connect.publish_replica("aco")` — holds the write lock, `CHECKPOINT`s,
|
||
copies, then swaps atomically). Staleness is therefore bounded by ingest
|
||
frequency; to refresh manually:
|
||
`uv run python -c "from conf.connect import publish_replica; publish_replica()"`
|
||
- **Reading the live primary instead**: pass `replica=False` to
|
||
`conf.connect.duckdb()`, or set `STACK_DUCKDB_REPLICA=0`.
|
||
- Kernels holding the *old* snapshot keep a valid file handle after a swap;
|
||
re-running the connect cell picks up the fresh one.
|
||
|
||
## Reference data (OPPS + PFS) is authoritative in the lake
|
||
|
||
Since M5 (#514), the OPPS and PFS reference tables live in **DuckLake** —
|
||
postgres catalog (`ducklake` db) + Parquet on RustFS
|
||
(`s3://lakehouse/ducklake/`), per the M3 decision record. Concurrency
|
||
there is structural: the catalog serializes writers transactionally and
|
||
readers get snapshot isolation.
|
||
|
||
- **Reading (notebooks / in-container code)**: `conf.connect.ducklake()` —
|
||
read-only by default via the `ducklake_ro` postgres role
|
||
(`DUCKLAKE_RO_PASSWORD`, in the notebooks container env). The lake is the
|
||
connection's default database, so `SELECT … FROM opps.addendum_b` or
|
||
`pfs.rvu` works unchanged. Compose-internal only — host-side code goes
|
||
through `docker exec`.
|
||
- **Writing**: `aco.lake.DuckLakeContext` (see `dev/scripts/
|
||
publish_reference_to_lake.py`). `ingest_opps.py` and `ingest_pfs.py`
|
||
publish to the lake as their final step (`--no-lake` to skip).
|
||
- The monolith's `opps`/`pfs` schemas remain as a **deprecated mirror** for
|
||
the `aco.pipe` analytics graph, the read replica, and cross-source joins
|
||
(e.g. pfs × skin_subs); drop them once those consumers migrate.
|
||
- Cross-source queries (lake schema joined with a monolith-only schema like
|
||
`skin_subs`) stay on the monolith connection until the joined schema also
|
||
moves.
|
||
|
||
## Per-year ingests merge, not wipe
|
||
|
||
`ingest_opps.py --year YYYY` used to `DROP TABLE` + `CREATE ... AS` from only
|
||
that year's files, silently erasing every other year (#509). It now merges:
|
||
existing rows for other years are preserved and the filtered year is replaced.
|
||
|
||
## Roadmap
|
||
|
||
The strategic fixes (per-domain DB files, read replica, lake storage with
|
||
snapshot isolation) are tracked as milestones M2–M5 in issues
|
||
[#510](https://git.fhirworx.io/homelab/stack/issues/510)–[#514](https://git.fhirworx.io/homelab/stack/issues/514),
|
||
per `docs/superpowers/specs/2026-07-08-duckdb-concurrency-streaming.md`.
|