M2: notebook read-replica (aco.ro.duckdb) to decouple readers from writers #510

New Issue

kert · 2026-07-08T19:41:25Z

kert commented

2026-07-08 19:41:25 +00:00

Decouple long-running notebook readers from batch writers: after each ingest,
publish a read-only snapshot aco.ro.duckdb and point notebooks /
conf.connect.duckdb(read_only=True) at the replica. Writers own the primary;
notebooks never block them. Staleness is bounded by the refresh cadence.

Scope

Post-ingest snapshot step (copy or EXPORT/ATTACH + COPY).
conf switch so notebook reads resolve to the replica.
Define/refresh cadence + document it.

Background & alternatives: docs/superpowers/specs/2026-07-08-duckdb-concurrency-streaming.md (option C). Complements the per-domain split
(M2 / #D).

Decouple long-running notebook readers from batch writers: after each ingest, publish a read-only snapshot `aco.ro.duckdb` and point notebooks / `conf.connect.duckdb(read_only=True)` at the replica. Writers own the primary; notebooks never block them. Staleness is bounded by the refresh cadence. **Scope** - Post-ingest snapshot step (copy or `EXPORT`/`ATTACH` + `COPY`). - `conf` switch so notebook reads resolve to the replica. - Define/refresh cadence + document it. Background & alternatives: `docs/superpowers/specs/2026-07-08-duckdb-concurrency-streaming.md` (option C). Complements the per-domain split (M2 / #D).

kert added the infra label 2026-07-08 19:41:25 +00:00

kert referenced this issue from a commit

2026-07-11 02:24:09 +00:00

feat(conf): notebook read-replica aco.ro.duckdb (closes #510)

kert closed this issue

2026-07-11 02:24:09 +00:00

kert commented

2026-07-11 02:25:49 +00:00

Implemented in 579d17beb4 and live in prod:

Post-ingest snapshot: conf.connect.publish_replica("aco") — holds the write lock, CHECKPOINTs the WAL, copies to data/aco.ro.duckdb, swaps atomically. Wired as the final step of ingest_opps.py and ingest_asp.py, so staleness is bounded by ingest cadence (the issue's "refresh cadence").
conf switch: conf.connect.duckdb() read-only opens now resolve to the replica when it exists — notebooks never hold the primary's single-writer lock. Opt-outs: replica=False or STACK_DUCKDB_REPLICA=0; write opens always use the primary.
Docs: docs/docs/duckdb-concurrency.md (cadence, manual refresh one-liner, opt-outs, swap semantics).

Verified live: replica published (3.1 GB) and the notebooks container resolves conf.connect.duckdb() to /home/kert/data/aco.ro.duckdb with current data visible. 82 conf tests cover snapshot consistency, divergence visibility, opt-outs, and atomic republish. Writers and notebook readers are now fully decoupled — the lock class of ingest failures is gone in day-to-day use; #511–#514 remain for the storage-level redesign.

Implemented in https://git.fhirworx.io/homelab/stack/commit/579d17beb4f5c2c5b50e30ea244f44b969dde7ac and live in prod: - **Post-ingest snapshot**: `conf.connect.publish_replica("aco")` — holds the write lock, `CHECKPOINT`s the WAL, copies to `data/aco.ro.duckdb`, swaps atomically. Wired as the final step of `ingest_opps.py` and `ingest_asp.py`, so staleness is bounded by ingest cadence (the issue's "refresh cadence"). - **conf switch**: `conf.connect.duckdb()` read-only opens now resolve to the replica when it exists — notebooks never hold the primary's single-writer lock. Opt-outs: `replica=False` or `STACK_DUCKDB_REPLICA=0`; write opens always use the primary. - **Docs**: `docs/docs/duckdb-concurrency.md` (cadence, manual refresh one-liner, opt-outs, swap semantics). Verified live: replica published (3.1 GB) and the notebooks container resolves `conf.connect.duckdb()` to `/home/kert/data/aco.ro.duckdb` with current data visible. 82 conf tests cover snapshot consistency, divergence visibility, opt-outs, and atomic republish. Writers and notebook readers are now fully decoupled — the lock class of ingest failures is gone in day-to-day use; #511–#514 remain for the storage-level redesign.

kert referenced this issue

2026-07-11 02:47:04 +00:00

M2: split aco.duckdb into per-domain files + ATTACH #511

kert referenced this issue

2026-07-11 03:31:42 +00:00

M5: migrate notebooks to lake reads; retire the monolith for reference data #514

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#510