bib: two rule items per FR document for 2019-16041 and 2024-14828 — merge to one key #836

New Issue

kert · 2026-09-25T00:32:01Z

kert commented

2026-09-25 00:32:01 +00:00

Found 2026-09-24 while de-duplicating search results: two Federal Register document numbers have two bib rule items each, both with fr_anchors, so every paragraph of those rules is indexed twice in the rules collection under different item keys and cited twice (same label).

2024-14828 (CY2025 PFS proposed): VXSUDWCR (url dated 2024/08/02, fr_page 61382, cms_id '') and XFGGRBDH (url dated 2024/07/31, fr_page 61596, cms_id 0938-AV33) — the public-inspection vs publication URL of the same document.
2019-16041: Q8R92EHT and MAE2WDTG.

Retrieval side is patched (llm.search.dedupe_by_label, and the chat merges by label), but the duplicates still cost index space, double the code_event anchors that reference both keys (e.g. 99441/99490 lineage rows cite both VXSUDWCR and XFGGRBDH), and split tags. Fix at the source: pick the publication-date item as canonical, re-point fr_anchors, fr_anchor_docs, pfs.code_event/code_element/code_guidance rows, tags and Zotero links to it, delete the twin, then stack llm index --collection rules --force --key <canonical> and stack llm prune-rules. Add a uniqueness check on extra_json.document_number for rule items in stack bib discover-pfs-rules so it cannot recur.

Found 2026-09-24 while de-duplicating search results: two Federal Register document numbers have two bib `rule` items each, both with `fr_anchors`, so every paragraph of those rules is indexed twice in the `rules` collection under different item keys and cited twice (same label). - 2024-14828 (CY2025 PFS proposed): `VXSUDWCR` (url dated 2024/08/02, fr_page 61382, cms_id '') and `XFGGRBDH` (url dated 2024/07/31, fr_page 61596, cms_id 0938-AV33) — the public-inspection vs publication URL of the same document. - 2019-16041: `Q8R92EHT` and `MAE2WDTG`. Retrieval side is patched (`llm.search.dedupe_by_label`, and the chat merges by label), but the duplicates still cost index space, double the `code_event` anchors that reference both keys (e.g. 99441/99490 lineage rows cite both VXSUDWCR and XFGGRBDH), and split tags. Fix at the source: pick the publication-date item as canonical, re-point `fr_anchors`, `fr_anchor_docs`, `pfs.code_event`/`code_element`/`code_guidance` rows, tags and Zotero links to it, delete the twin, then `stack llm index --collection rules --force --key <canonical>` and `stack llm prune-rules`. Add a uniqueness check on `extra_json.document_number` for rule items in `stack bib discover-pfs-rules` so it cannot recur.

kert added the quality bib labels 2026-09-25 00:32:01 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#836