homelab/stack
1
0
Fork 0
Code Issues 79 Pull Requests 1 Actions 1 Packages Projects 1 Releases Wiki Activity

bib: Store.upsert clobbers curated tags/collections on URL-dedupe update (wipe-and-replace _sync_tags) #624

New Issue
Closed
opened 2026-08-17 14:57:48 +00:00 by kert · 0 comments
kert commented 2026-08-17 14:57:48 +00:00
Owner
Copy Link

Root cause of the CY2027-NPRM-missing-from-Zotero incident (see the milestone for the full chain).

Store.upsert (src/bib/store.py:207) dedupes by URL and, on the existing-item path, calls self.update(ekey, tags=item.tags, collections=item.collections) → _sync_tags (src/bib/store.py:423), which does DELETE FROM item_tags WHERE item_id = ? and rewrites only the incoming set. The # Merge tags comment merges the tags= parameter into the incoming item — it never merges with the tags already stored on the existing row. _sync_collections has the identical wipe-and-replace hazard.

Observed damage (2026-08-14): the comment-farm re-ingest (store.upsert(rule) at src/cli/bib.py:241,251) rebuilt rule items from the FR API and wiped curated tags:

  • 5ITGVDJV (CY2027 NPRM) lost sup:2027_PFS_NPRM + file:rvu + year:2027 at 05:58:13Z — added 5h earlier by fetch_nprm_addenda --year 2027 --register
  • 2KVJ2HKX (CY2026 NPRM) lost sup:2026_PFS_NPRM + file:rvu + year:2026 at 00:27:43Z
  • both sup:*_PFS_NPRM tags now have zero item associations in bib; Zotero still shows sup:2026_PFS_NPRM on 2KVJ2HKX only as a stale pre-wipe snapshot
  • this is the same drift that surfaced as the docs test failure in #623 (fixed there by making the export hermetic — symptom, not cause)
  • follow-on breakage: pfs.nprm._addendum_b_path discovers Addendum B by sup:<year>_PFS_NPRM tag (NPRM_SOURCES), so load_rvu_proposed re-runs now raise "No Addendum B attachment found" for both years

Fix: on the upsert dedupe path, union the existing row's tags and collections with the incoming ones before calling update(). Store.update(tags=...) keeps its explicit replace semantics (documented); deliberate removal stays remove_tag. Regression tests: existing curated tag survives a URL-dedupe upsert; same for collection membership.

Data repair is #625; Zotero catch-up sync is #626; nightly prevention is #627.

**Root cause of the CY2027-NPRM-missing-from-Zotero incident** (see the milestone for the full chain). `Store.upsert` (src/bib/store.py:207) dedupes by URL and, on the existing-item path, calls `self.update(ekey, tags=item.tags, collections=item.collections)` → `_sync_tags` (src/bib/store.py:423), which does `DELETE FROM item_tags WHERE item_id = ?` and rewrites only the incoming set. The `# Merge tags` comment merges the `tags=` *parameter* into the incoming item — it never merges with the tags already stored on the existing row. `_sync_collections` has the identical wipe-and-replace hazard. **Observed damage** (2026-08-14): the comment-farm re-ingest (`store.upsert(rule)` at src/cli/bib.py:241,251) rebuilt rule items from the FR API and wiped curated tags: - `5ITGVDJV` (CY2027 NPRM) lost `sup:2027_PFS_NPRM` + `file:rvu` + `year:2027` at 05:58:13Z — added 5h earlier by `fetch_nprm_addenda --year 2027 --register` - `2KVJ2HKX` (CY2026 NPRM) lost `sup:2026_PFS_NPRM` + `file:rvu` + `year:2026` at 00:27:43Z - both `sup:*_PFS_NPRM` tags now have **zero** item associations in bib; Zotero still shows `sup:2026_PFS_NPRM` on 2KVJ2HKX only as a stale pre-wipe snapshot - this is the same drift that surfaced as the docs test failure in #623 (fixed there by making the export hermetic — symptom, not cause) - follow-on breakage: `pfs.nprm._addendum_b_path` discovers Addendum B by `sup:<year>_PFS_NPRM` tag (`NPRM_SOURCES`), so `load_rvu_proposed` re-runs now raise "No Addendum B attachment found" for both years **Fix**: on the upsert dedupe path, union the existing row's tags and collections with the incoming ones before calling `update()`. `Store.update(tags=...)` keeps its explicit replace semantics (documented); deliberate removal stays `remove_tag`. Regression tests: existing curated tag survives a URL-dedupe upsert; same for collection membership. Data repair is #625; Zotero catch-up sync is #626; nightly prevention is #627.
kert added this to the P38: Zotero Rule Sync — upsert tag merge, tag repair, nightly rules scope milestone 2026-08-17 14:57:48 +00:00
kert added the bibquality labels 2026-08-17 14:57:48 +00:00
kert closed this issue 2026-08-17 15:13:22 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
api
REST API (FastAPI)
 auth
Authentication and authorization
 bib
Bibliography, Zotero, citations
 ci
CI/CD pipelines and Woodpecker plugins
 cli
CLI commands and UX
 cloud
Cloud provider abstraction (AWS, GCP, Azure, self-hosted)
 config
Configuration and wiring
 data
Data loading, CCLF, BCDA, seeds
 databricks
Databricks jobs and deployment
 docs
Documentation and generated references
 image:api
image:docs
image:mc
image:notebooks
image:zotero
infra
lakehouse
Nessie, Polaris, Trino, Iceberg
 llm
llm module — local RAG over comments + Zotero corpus
 notebooks
marimo notebook errors (auto-filed by nb-watcher/nb-integration)
 packaging
Python packaging, optional deps, build system
 pfs
pipeline
Pipeline execution and validation
 pipeline:ci
pipeline:deploy
pipeline:harden
pipeline:infra-ci
pipeline:rebuild-all
pipeline:release
qpp
quality
Testing, coverage, error handling
 step:build
step:deploy
step:hadolint
step:lint
step:publish
step:scan
step:test
step:validate
styles
Charts, figures, visual theming
 tree
Project directory structure and file organization
No Label bib quality
Milestone
No items
No Milestone P38: Zotero Rule Sync — upsert tag merge, tag repair, nightly rules scope
Projects
Clear projects
No project
Assignees
Clear assignees
kert renovate-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: homelab/stack#624
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?