M2: split aco.duckdb into per-domain files + ATTACH #511

New Issue

kert · 2026-07-08T19:41:25Z

kert commented

2026-07-08 19:41:25 +00:00

data/aco.duckdb is one ~3.2 GB file for all schemas (aco, opps, pfs,
bib, …), so a reader of any schema blocks a writer of any other. Split into
per-domain DB files (opps.duckdb, pfs.duckdb, …) and ATTACH them for
cross-domain queries. A domain ingest then only contends with readers of that
domain.

Scope: conf.path/conf.connect for per-domain DBs; ATTACH helper;
migrate the ingest scripts; keep aco.duckdb as the analytics/joined store.

Background: docs/superpowers/specs/2026-07-08-duckdb-concurrency-streaming.md (option D).

`data/aco.duckdb` is one ~3.2 GB file for all schemas (`aco`, `opps`, `pfs`, `bib`, …), so a reader of any schema blocks a writer of any other. Split into per-domain DB files (`opps.duckdb`, `pfs.duckdb`, …) and `ATTACH` them for cross-domain queries. A domain ingest then only contends with readers of that domain. **Scope**: `conf.path`/`conf.connect` for per-domain DBs; `ATTACH` helper; migrate the ingest scripts; keep `aco.duckdb` as the analytics/joined store. Background: `docs/superpowers/specs/2026-07-08-duckdb-concurrency-streaming.md` (option D).

kert added the infra label 2026-07-08 19:41:25 +00:00

kert commented

2026-07-11 02:47:04 +00:00

Closing as superseded / deferred. The two shipped M1/M2 fixes removed this issue's driving pain:

#510: read-only opens resolve to the aco.ro.duckdb snapshot — notebook readers never hold the primary's lock, so reader-vs-writer contention (the recurring failure) is gone.
#508: any residual collision surfaces as an actionable holder-naming error with retry, not a raw IOException.

What a split would still buy is writer-vs-writer concurrency (parallel domain ingests + pipe runs) and cheaper per-domain replica publishing — real but modest, while the cost is high: migrating ~9 skin_subs writer scripts plus the PFS load path, and a required destructive DROP SCHEMA on the monolith (an in-monolith schema shadows an attached catalog of the same name, so keeping both silently serves stale data).

Meanwhile the strategic path (M3 #512 → M5 #514) moves reference data to the lake with true multi-writer snapshot isolation and retires the monolith for reference data anyway — making this split a throwaway intermediate. Proceeding directly to the #512 DuckLake-vs-Iceberg spike.

Reopen if: overlapping writers become a real operational need before the lake path lands (e.g. scheduled ingests start colliding with aco.pipe runs).

Closing as **superseded / deferred**. The two shipped M1/M2 fixes removed this issue's driving pain: - #510: read-only opens resolve to the `aco.ro.duckdb` snapshot — notebook readers never hold the primary's lock, so reader-vs-writer contention (the recurring failure) is gone. - #508: any residual collision surfaces as an actionable holder-naming error with retry, not a raw IOException. What a split would still buy is writer-vs-writer concurrency (parallel domain ingests + pipe runs) and cheaper per-domain replica publishing — real but modest, while the cost is high: migrating ~9 skin_subs writer scripts plus the PFS load path, and a required destructive `DROP SCHEMA` on the monolith (an in-monolith schema shadows an attached catalog of the same name, so keeping both silently serves stale data). Meanwhile the strategic path (M3 #512 → M5 #514) moves reference data to the lake with true multi-writer snapshot isolation and retires the monolith for reference data anyway — making this split a throwaway intermediate. Proceeding directly to the #512 DuckLake-vs-Iceberg spike. **Reopen if**: overlapping writers become a real operational need before the lake path lands (e.g. scheduled ingests start colliding with `aco.pipe` runs).

kert closed this issue

2026-07-11 02:47:04 +00:00

kert referenced this issue

2026-07-11 03:31:42 +00:00

M5: migrate notebooks to lake reads; retire the monolith for reference data #514

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#511