M1: ingest_opps.py --year wipes other years (full-replace footgun) #509
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: homelab/stack#509
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
dev/scripts/ingest_opps.py --year YYYYprocesses only that year's dir, thenruns
DROP TABLE opps.addendum_b; CREATE TABLE ... AS SELECT * FROM <that year>— which wipes every other year from the table. The only safe way to keep all
years is a full ingest with no
--year.Fix: make
--yeara per-year merge —DELETE FROM opps.addendum_b WHERE year = ?then insert the new rows — leaving other years intact. Same forapc_weight/skin_sub_addendum_b.Background:
docs/superpowers/specs/2026-07-08-duckdb-concurrency-streaming.md(M1 / #B).Fixed in
d809a45f92.--yearis now a per-year merge: the load step reads back the existing table's other-year rows and unions them with the newly parsed frames before rebuilding (pandas concat also unions columns, so CMS header drift between years stays handled).skin_sub_addendum_bis derived from the mergedaddendum_btable instead of the run's frames, so it keeps all years too.Validated against the real store: a
--year 2026run preserved identical per-year row counts for 2014–2025 acrossapc_weight,addendum_b, andskin_sub_addendum_bwhile rebuilding 2026.