llm/bib: normalize FR rule text at capture, not at every index run #617
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: homelab/stack#617
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Follow-up to #595/#589. The FR full-text endpoint serves HTML-wrapped text (
<html><pre>…, inline anchors, entities, stray NULs), and today the cleanup lives at index time:src/llm/source.pystrips wrapper/tags/entities andsrc/llm/chunk.pystrips control chars — re-executed for every doc on every (re)index. Cleaner design: normalize once at capture —dev/scripts/fetch_fr_attachments.pywrites cleaned plain text to the stored.txt(raw bytes kept alongside as.html), one-time migration cleans the existingdata/fr_downloads/*.txt, and the index-time strippers stay as cheap idempotent guards rather than load-bearing transforms. Shared cleaner lives inrex(llm already depends on rex extractors).Done in
d726dca.clean_fr_textinsrc/rex/frtext.py(wrapper+inline-tag strip, entity unescape, NUL/C0 removal,[[Page N]]preserved, idempotent — 16 TDD tests);dev/scripts/fetch_fr_attachments.pycleans at download (raw kept as.html); migration cleaned the only 2 HTML-wrapped files (2025-13271, 2025-19787 — the older 52 were already plain text);src/llm/source.pynow delegates to the shared cleaner as an idempotent guard. Rules collection re-embedded once for the hash bump (5,781 chunks), steady-stateindexed=0 skipped=2confirmed — no recurring re-scrub.