llm: RAG tagging chain + batch runner #575
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: homelab/stack#575
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Per comment: retrieve top-k corpus passages → closed-vocab structured output with confidence. This implements the thematic-tagging half of #254 on the llm engine. ~4 GPU-days at 2s/comment on one card — batch, resumable, docket-scoped, multi-host, run off-peak on the rack card.
Scope
--docketscoping, uses multi-host dispatchAcceptance
Refs #254.
Spec:
docs/superpowers/specs/2026-07-16-llm-module-design.mdStatus audit (2026-08-13, repo-verified): NOT DONE. No tagging chain/batch runner in src/llm (P35 not started).
Done — merged to main as
5e37641(commit65f8d78). Pilot running on CMS-2026-2377.llm.tagger: for one comment, its already-indexed chunks (text + vector fromlangchain_pg_embedding, no re-embedding) are scored by cosine against the 57 theme cards (embedded once per run), the top 8 are shortlisted with the chunk that scored each, and each candidate gets one yes/no judgement on the largest live host at temperature 0 with that chunk as evidence (make_judge, thellm.classifyroute). Up to 3 themes whose shortlist similarity ≥ 0.60 are accepted (defaults from the #577 sweep), confidence = that similarity. State on the bib item (extra_json.llm_tags: version, model, content hash, tags with evidence chunk, the full shortlist with verdicts) makes runs resumable and idempotent — unchanged comments are skipped,--forceredoes them. Fan-out usesHostPool(dead hosts dropped, #796). Every collaborator is injected; 14 unit tests without Ollama or pgvector, 7 CLI tests.stack llm tag --docket D [--limit N] [--force] [--dry-run] [--top 8] [--max-tags 3] [--min-confidence 0.60] [--verbose]. Throughput on the laptop 5070 Ti (rig down): ~13 s per comment (8 judgements) — the pilot's newest 300 comments take about an hour; the whole docket (43k) would be ~6 days on one card, so the multi-host pool matters for fan-out.Implements #254's thematic-tagging half; stance stays with the P43 tooling.