kert d809a45f92 fix(ingest): M1 duckdb concurrency — lock preflight + per-year merge (closes #508, #509) ...

#508: conf.connect.duckdb_batch() — short-lived RW connection with
exponential-backoff retry on the single-writer lock; on exhaustion
raises naming the holder PIDs/commands via lsof ('close the notebook
tab or stop that PID') instead of a raw IOException. Wired into
ingest_opps.py and ingest_asp.py. Connection-hygiene doc at
docs/docs/duckdb-concurrency.md.

#509: ingest_opps.py --year no longer wipes other years. The load step
now merges: existing other-year rows are read back and unioned with the
new frames before the rebuild (pandas concat also unions columns, so
CMS format drift stays handled). skin_sub_addendum_b is derived from
the merged addendum_b table instead of this run's frames.

Validated against the real store: --year 2026 run preserved identical
per-year counts for 2014-2025 across apc_weight (9 years), addendum_b
and skin_sub_addendum_b (13 years) while rebuilding 2026.

2026-07-10 16:48:08 -04:00

530 B

Raw Blame History

View Raw