docs(plan): Task 3b data-preserving upsert + abstract recovery step after the 2026-09-08 clobber (refs #615)
This commit is contained in:
@@ -668,6 +668,82 @@ git commit -m "feat(bib): Store.upsert_status — identical re-upsert writes not
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
### Task 3b: Data-preserving upsert — never blank a stored value
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `src/bib/store.py` (`upsert_status`)
|
||||||
|
- Test: `tests/bib/test_store_upsert_noop.py` (extend)
|
||||||
|
|
||||||
|
**Why (incident 2026-09-08):** the fetch list walk builds a `Source` whose `abstract` is empty (the list endpoint has no body) and re-upserts every comment it sees. With the old upsert that overwrote 17,607 enriched bodies in CMS-2026-2377 and 1,115 in CMS-2017-0092. Task 3's comparison alone would still call that "updated" and write the empty abstract. Upsert must merge, not replace: an empty incoming value carries no information.
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Consumes: `Store.upsert_status` from Task 3.
|
||||||
|
- Produces: same signature; for every column in `_COMPARE_COLS` (except `extra_json`) an empty incoming string keeps the stored value. `extra_json` is compared/written as today (it is always a full serialization).
|
||||||
|
|
||||||
|
- [ ] **Step 1: Write the failing tests** (append to `tests/bib/test_store_upsert_noop.py`)
|
||||||
|
|
||||||
|
```python
|
||||||
|
def test_empty_incoming_abstract_keeps_stored_body():
|
||||||
|
"""A list-walk row (no body) must not blank an enriched comment."""
|
||||||
|
s = _store()
|
||||||
|
key, _ = s.upsert_status(_item(abstract="enriched body"))
|
||||||
|
_, status = s.upsert_status(_item(abstract=""))
|
||||||
|
assert status == "unchanged"
|
||||||
|
assert s.get(key).abstract == "enriched body"
|
||||||
|
|
||||||
|
|
||||||
|
def test_empty_incoming_title_keeps_stored_title():
|
||||||
|
s = _store()
|
||||||
|
it = _item(); it.title = "Org: CMS-2026-2377-1"
|
||||||
|
key, _ = s.upsert_status(it)
|
||||||
|
it2 = _item(); it2.title = ""
|
||||||
|
_, status = s.upsert_status(it2)
|
||||||
|
assert status == "unchanged"
|
||||||
|
assert s.get(key).title == "Org: CMS-2026-2377-1"
|
||||||
|
|
||||||
|
|
||||||
|
def test_non_empty_incoming_still_updates():
|
||||||
|
s = _store()
|
||||||
|
key, _ = s.upsert_status(_item(abstract="v1"))
|
||||||
|
_, status = s.upsert_status(_item(abstract="v2"))
|
||||||
|
assert status == "updated" and s.get(key).abstract == "v2"
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Run to verify failure**
|
||||||
|
|
||||||
|
Run: `uv run --no-sync pytest tests/bib/test_store_upsert_noop.py -q -p no:cacheprovider`
|
||||||
|
Expected: the first two new tests FAIL (`status == "updated"`, abstract/title blanked).
|
||||||
|
|
||||||
|
- [ ] **Step 3: Implement**
|
||||||
|
|
||||||
|
In `upsert_status`, after `cur_row = current.to_row()` and before the `same_cols` check:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Merge, don't replace: an empty incoming value carries no
|
||||||
|
# information (a list-walk row has no body), so the stored value
|
||||||
|
# wins. extra_json is always a full serialization and is exempt.
|
||||||
|
for c in self._COMPARE_COLS:
|
||||||
|
if c != "extra_json" and not new_row.get(c) and cur_row.get(c):
|
||||||
|
new_row[c] = cur_row[c]
|
||||||
|
setattr(item, c, cur_row[c])
|
||||||
|
```
|
||||||
|
|
||||||
|
`setattr(item, c, …)` keeps the later `item.to_row()` (used to build the written row) consistent with the merged values. `Item` is a pydantic model; plain attribute assignment is allowed on these fields.
|
||||||
|
|
||||||
|
- [ ] **Step 4: Run to verify pass**
|
||||||
|
|
||||||
|
Run: `uv run --no-sync pytest tests/bib -q -p no:cacheprovider`
|
||||||
|
Expected: pass.
|
||||||
|
|
||||||
|
- [ ] **Step 5: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add src/bib/store.py tests/bib/test_store_upsert_noop.py
|
||||||
|
git commit -m "fix(bib): upsert never blanks a stored value — empty incoming columns keep the stored ones (refs #615)"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
### Task 4: Idempotent `attach_file`
|
### Task 4: Idempotent `attach_file`
|
||||||
|
|
||||||
**Files:**
|
**Files:**
|
||||||
@@ -3409,6 +3485,19 @@ uv run stack comments dockets # reopens the store
|
|||||||
sqlite3 data/bib.sqlite "SELECT name FROM sqlite_master WHERE name='idx_attachments_item_filename'"
|
sqlite3 data/bib.sqlite "SELECT name FROM sqlite_master WHERE name='idx_attachments_item_filename'"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2b: Recover the abstracts blanked on 2026-09-08**
|
||||||
|
|
||||||
|
The old upsert overwrote 17,607 enriched bodies in CMS-2026-2377 and 1,115 in CMS-2017-0092 with '' (incident in the SDD ledger). They still carry `enriched:ok`, so backfill skips them. Clear the tag on those rows and re-enrich from the mirror (idempotent attach + data-preserving upsert are in place by now):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
for d in CMS-2026-2377 CMS-2017-0092; do
|
||||||
|
sqlite3 data/bib.sqlite "DELETE FROM item_tags WHERE tag_id=(SELECT id FROM tags WHERE name='enriched:ok') AND item_id IN (SELECT id FROM items WHERE url LIKE 'https://www.regulations.gov/comment/$d-%' AND coalesce(abstract,'')='')"
|
||||||
|
uv run stack bib backfill-comments --docket $d --mirror
|
||||||
|
done
|
||||||
|
sqlite3 data/bib.sqlite "SELECT substr(url,37,13), count(*), sum(coalesce(abstract,'')<>'') FROM items WHERE url LIKE 'https://www.regulations.gov/comment/CMS-2026-2377-%' OR url LIKE 'https://www.regulations.gov/comment/CMS-2017-0092-%' GROUP BY 1"
|
||||||
|
```
|
||||||
|
Expected: non-empty abstracts ≈ total for both dockets (a few genuinely empty bodies are fine). Then re-run `stack comments extract` and `stack llm index --collection comments` so any comment whose text was lost from combined.md/index is restored (the index still holds pre-wipe text; fingerprints change because `updated_at` moved, hash-skip absorbs unchanged text).
|
||||||
|
|
||||||
- [ ] **Step 3: Seal the historical dockets**
|
- [ ] **Step 3: Seal the historical dockets**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|||||||
Reference in New Issue
Block a user