docs(spec): llm whole-library retrieval, recency ranking, exact evidence links, largest-GPU routing (refs #571 #654 #615)

docs/superpowers/specs/2026-09-03-llm-corpus-recency-links-gpu-design.md

@@ -0,0 +1,256 @@
+# llm service — whole-library retrieval, recency ranking, exact-paragraph evidence links, largest-GPU routing
+
+**Status:** Draft — awaiting approval
+**Date:** 2026-09-03
+**Milestones:** P34 (#571 grounded generation with citations), P45 (#654 fan-out to
+generation), plus #615 (re-index of CMS-2026-2377)
+**Builds on:** `docs/superpowers/specs/2026-07-16-llm-module-design.md`,
+`2026-07-17-llm-chat-ui-design.md`, `2026-08-17-fr-jump-links-design.md`
+
+## Goal
+
+Four changes to the `llm` chat service (`src/llm`, `llm.fhirworx.io`):
+
+1. **Whole-library retrieval.** Answers draw on the entire Zotero/bib library —
+   public comments, Federal Register rules, and the reference corpus (PubMed
+   records, manuals, regulations, downloads, mail sources) — not just the
+   `comments` collection.
+2. **Recency-first.** Retrieval prefers the most recently posted comments and
+   the newest rules/sources when relevance is comparable, and the answer states
+   dates.
+3. **Exact evidence links.** Every source shown under an answer carries a deep
+   link that lands on the exact passage: the FR paragraph anchor plus a
+   scroll-to-text highlight for rules, the comment page or the attachment PDF
+   page for comments, and the item URL (with PDF page when known) for corpus
+   items. The FR jump-link resolver gains the same highlight so notebook
+   links land on the passage too.
+4. **Largest live GPU.** Generation runs on the largest GPU that is up at
+   request time (today the rig 4090), with a bigger instruct model when that
+   host can hold it. Embedding keeps fanning out across every live host.
+
+All inference stays on the local Ollama fleet — no cloud APIs.
+
+## Current state (measured 2026-09-03)
+
+| Fact | Value |
+|---|---|
+| pgvector collections | `comments` 732,716 chunks / 166,919 items (last indexed 2026-08-14); `rules` 7,884 chunks / **3 of 80** rules; no `corpus` collection |
+| bib comments | 196,739 (≈30k not yet indexed); `date_published` = regulations.gov posted date, latest 2026-08-19 |
+| bib non-comment items | 22,470 (15,242 with abstracts; 848 with local attachments, 3,402 files: txt/xlsx/pdf/csv) |
+| Zotero-only PDFs | 1,129 bib-keyed Zotero parents hold a storage PDF; 848 of those are in bib `attachments` — the rest (PRISMA stage-3 fetches) live only in `data/zotero/data/storage` |
+| FR anchor maps | 80 rules, 193,220 paragraph anchors in `fr_anchors` (p_id, page, ordinal, text) |
+| Ollama hosts | rack 3060 12 GB (`ollama`, 0.32.1), laptop 5070 Ti 12 GB (`notebook.local`), rig 4090 24 GB (`rig.local`); all three up; `qwen2.5:14b` on all, `qwen2.5:32b` only on rig, `llama3.1:8b` (current `instruct_model`) only on rack |
+| Generation routing today | `pool.acquire()` picks least in-flight, so the rack 3060 always wins; `check(instruct_model)` drops every host but the rack anyway |
+| Rule text | indexed from the FR `.txt` attachment — chunk → paragraph mapping is lost |
+| Comment attachment text | `combined.md` sections per attachment (`## attachment_1.pdf`); PDF pages joined with `\n\n`, page boundaries lost |
+
+The FR live page (browser UA) carries `id="p-N"` on every paragraph and
+`id="page-NNNNN"` markers; a non-browser UA gets a 10 KB "Request Access"
+page. `#p-N` anchors are valid but land on the paragraph's top edge under the
+site header, and page cites (`91 FR 44242` → `#page-44242`) land on a page
+marker that sits mid-paragraph — that is the "not quite the exact paragraph"
+symptom.
+
+## Decisions
+
+| Decision | Choice | Why |
+|---|---|---|
+| Corpus scope | Three collections: `comments`, `rules` (rebuilt from `fr_anchors`), `corpus` (every non-comment bib item; abstract + local attachment text + Zotero-storage PDF text) | Matches the library as the user sees it in Zotero; `fr_anchors` gives exact paragraph provenance for free |
+| Cross-collection retrieval | Query each collection, over-fetch, merge, re-rank in Python | Three PGVector stores already exist; no schema change; per-kind `k` keeps rules from drowning comments |
+| Recency | Soft boost: `score = similarity × (1 − w) + w × recency`, recency = `exp(−age / half_life)`; optional hard `since` date on the API | Relevance must still win on clearly better hits; a hard cutoff alone would hide older authoritative rules |
+| Rule chunking | One chunk per FR paragraph (packing tiny neighbours up to ~2,000 chars); metadata `p_id`, `page`, `html_url` | Chunk == anchor, so the evidence link is exact by construction |
+| Comment page mapping | At index time, locate each attachment-derived chunk's opening text in the local PDF's per-page text (pymupdf); store `attachment`, `page` | No re-extraction of 28k comments; PDFs are already on disk |
+| Link rendering | `llm/links.py` builds `url` + `label` per source; FR links get a `:~:text=` scroll-to-text fragment; `frlink.resolve(..., highlight=True)` gains the same | Text fragments highlight and centre the passage in Chromium/Safari/Firefox ≥131 and fall back to `#p-N` elsewhere |
+| GPU routing | Hosts declare VRAM inline (`http://rig.local:11434@24`); generation goes to the largest live host serving the model; embeds stay least-loaded | Ollama exposes no GPU-size API; declared size + live probe is the honest "largest available at any time" |
+| Models | `instruct_model = qwen2.5:14b` everywhere; `instruct_model_large = qwen2.5:32b` when the chosen host declares ≥ `large_min_vram_gb` (20) and serves it; `num_ctx` passed per request (8,192) | Both already pulled; 32b is VRAM-resident on the 4090 (43 tok/s measured); no more Modelfile ctx variants |
+| Rollout | Update `.env` host lists, `docker compose build llm && up -d llm`; re-index on the host with nohup across the fleet | Same procedure as every other roll (see deploy_rollout memory) |
+
+## Architecture
+
+```
+bib.sqlite ──┬─ comments (combined.md / abstract) ─┐
+             ├─ fr_anchors paragraphs (rules)      ├─> llm.source → llm.chunk → embed (fleet fan-out) → pgvector
+             └─ corpus items (+ Zotero storage PDFs)┘         metadata: kind, date, url, p_id/page/attachment
+
+POST /chat {question, since?}
+   └─ rag.retrieve: embed question once → 3 × similarity_search_with_score (over-fetch)
+        → rerank.recency_blend → top-N mixed sources
+   └─ links.for_source: kind-specific deep link + label
+   └─ pool.acquire_generation(): largest live host by declared VRAM → model tier → /api/chat (stream)
+   └─ SSE: token* , sources (with url/label/date/kind), done
+```
+
+## Components
+
+### `llm/config.py`
+- `ollama_hosts` becomes `tuple[Host, ...]` where `Host(url, vram_gb)`;
+  parse `url@vram` (missing `@` → `vram_gb=0`, sorts last).
+- New fields with `[llm]` defaults: `instruct_model_large = ""`,
+  `large_min_vram_gb = 20`, `chat_num_ctx = 8192`,
+  `recency_half_life_days = 365`, `recency_weight = 0.3`,
+  `k_per_kind = {"comment": 8, "rule": 4, "corpus": 4}`, `top_n = 8`.
+- `LLM_OLLAMA_HOSTS` env keeps precedence (annotated form).
+
+### `llm/pool.py`
+- `HostPool` stores `vram_gb` per host.
+- `acquire_generation(model)` → context manager yielding the alive host
+  with the largest `vram_gb` that serves `model` (tie → least in-flight);
+  raises the existing "no host serves" error when none does.
+- `pick_model(cfg, host)` → `instruct_model_large` if
+  `host.vram_gb >= large_min_vram_gb` and the host lists it, else
+  `instruct_model`. Liveness is re-probed on every chat request (already
+  the case via `check`), so a rig outage falls back to the laptop/rack.
+- Fix the 5080 → 5070 Ti docstring (#653 note).
+
+### `llm/source.py`
+- `iter_rule_docs` rebuilt on `fr_anchors`: yields one `Doc` per rule whose
+  text is the anchor paragraphs in `p_id` order, separated by a sentinel
+  the chunker respects, with a parallel `paragraphs` list so
+  `chunk_rule_doc` can pack whole paragraphs and stamp `p_id`, `page`,
+  `html_url` per chunk. Rules without an anchor map fall back to the
+  current `.txt` path (metadata without `p_id`).
+- `iter_comment_docs` keeps `combined.md`; the chunker's section split
+  already isolates each `## attachment_N.ext` — a new
+  `attachment_pages(root, docket, cid)` helper returns per-page text for
+  PDF attachments; chunk metadata gains `attachment` and `page` (empty when
+  not located).
+- `iter_corpus_docs` gains `title`, `url`, `date`, `item_type`,
+  `project` (from `project:` tags) metadata and a Zotero-storage PDF
+  fallback: a `ZoteroPdfIndex` built once per run from a snapshot copy of
+  `zotero.sqlite` (never the live file), mapping bib key → storage PDF
+  paths; text via the existing rex PDF extractor. Non-PDF attachment text
+  keeps the current path.
+- Every doc carries `date` (ISO, `date_published`) and `kind`
+  (`comment` | `rule` | `corpus`).
+- Comment iteration order becomes newest-first (`ORDER BY date_published
+  DESC`) so incremental runs surface the latest first.
+
+### `llm/chunk.py`
+- `chunk_doc` unchanged for comments/corpus.
+- New `chunk_paragraphs(doc, paragraphs)` for rules: greedy pack of whole
+  paragraphs up to `target_chars`; a paragraph longer than the target is
+  hard-wrapped with overlap but keeps its `p_id`. Metadata per chunk:
+  `p_id` (first paragraph), `p_id_last`, `page`.
+
+### `llm/rerank.py` (new, pure)
+- `recency(date, now, half_life_days) -> float` in `[0,1]`; undated → 0.
+- `blend(hits, *, weight, half_life_days, now) -> list[hit]`: converts
+  pgvector cosine distance to similarity, blends, sorts desc, dedupes by
+  item key (best chunk per item), returns `top_n`.
+- `filter_since(hits, since)` for the hard cutoff.
+
+### `llm/links.py` (new, pure over metadata)
+- `for_source(md) -> (url, label)`:
+  - comment: `https://www.regulations.gov/comment/{cid}`; when
+    `attachment` is set: `https://downloads.regulations.gov/{cid}/{attachment}`
+    plus `#page={page}` for PDFs. Label `CMS-2026-2377-3438` (+ ` p.4`).
+  - rule: `{html_url}#p-{p_id}` + `:~:text={first ~8 words of snippet}`;
+    label `91 FR 43949 ¶4` (page + ordinal looked up from metadata).
+  - corpus: item `url` (+ `#page=` when the chunk came from a PDF and the
+    URL ends in `.pdf`); label = short title (≤ 60 chars) + year.
+- `text_fragment(text) -> str`: first sentence clipped to ~80 chars,
+  percent-encoded per the URL Fragment Text Directives spec (`-`, `,`, `&`
+  escaped).
+
+### `bib/frlink.py` (tweak)
+- `resolve(..., highlight=False)`: when true, paragraph-resolved links
+  (`¶k`, raw anchor, quote) append `:~:text=` built from the paragraph's
+  opening words; page-only cites additionally upgrade to the first
+  paragraph *starting* on that page when one exists (so `91 FR 44242`
+  lands on a paragraph, not the mid-paragraph page marker). `md_link`
+  passes `highlight` through; `Pincite.jump_url` uses `highlight=True`.
+- Quote resolution: when several paragraphs match, prefer the one whose
+  text starts with the quote, else the shortest; raise only when still
+  tied. Existing single-match behaviour unchanged.
+
+### `llm/rag.py`
+- `retrieve(question, *, cfg, pool, since=None)`: one query embedding,
+  three stores, `k_per_kind × 3` over-fetch each, `rerank.blend`, then
+  `links.for_source` → source dicts
+  `{id, kind, label, url, title, date, docket, snippet, score}`.
+- `build_messages`: excerpts rendered as `[label] (kind, date) snippet`;
+  system prompt tells the model to cite `[label]`, prefer the most recent
+  comments when excerpts conflict, and name the year when it matters.
+- `stream_answer`: `pool.acquire_generation` + `pick_model`, passes
+  `options.num_ctx`; the `sources` event carries the enriched dicts and
+  `model`/`host` for the UI footer.
+
+### `llm/api.py` + `web/chat.html`
+- `ChatRequest` gains optional `since: str | None`.
+- `/health` returns live hosts with `vram_gb` and the host/model generation
+  would pick right now.
+- UI: sources render as links (`<a target=_blank>`) with kind badge, date,
+  and label; a "last 12 months" checkbox sets `since`; the footer shows
+  `model @ host`.
+
+### `cli/llm.py`
+- `index --collection all` runs comments → rules → corpus; `--newest-first`
+  is the default order for comments.
+- `stack llm hosts` prints the pool with VRAM, liveness, models, and the
+  generation pick (operator check for "which GPU will answer").
+
+### Compose / env
+- `.env`: `LLM_OLLAMA_HOSTS` and `LLM_OLLAMA_HOSTS_IN_CONTAINER` gain `@12`,
+  `@12`, `@24` annotations; `stack.toml [llm]`: `instruct_model =
+  "qwen2.5:14b"`, `instruct_model_large = "qwen2.5:32b"`, new knobs above.
+- No compose topology change; the container already reaches both remotes
+  (verified `docker exec llm curl 192.168.1.222:11434`).
+
+## Data flow for one question
+
+1. Embed the question once on the least-loaded live host.
+2. Similarity search `comments` (k=24), `rules` (k=12), `corpus` (k=12).
+3. Convert distance → similarity, blend with recency, apply `since`,
+   dedupe per item, keep top 8.
+4. Build deep links; render excerpts; stream from the largest live host
+   with the tier-appropriate model at `num_ctx` 8,192.
+5. Emit sources (with links) and done.
+
+## Error handling
+
+- A host down at request time is dropped by `check`; if the rig is down the
+  laptop answers with `qwen2.5:14b`. If no host serves the model the
+  stream emits the existing `error` event.
+- Missing metadata never breaks a link: `links.for_source` degrades
+  rule → page link → FR html_url; comment → comment page; corpus → item
+  URL; undated hits get recency 0 (pure similarity).
+- Zotero snapshot copy failure logs and skips the Zotero-PDF fallback;
+  bib attachments still index.
+- Page location failures leave `page` empty; the link falls back to the
+  attachment root.
+
+## Testing
+
+- Unit (TDD, `tests/llm`): host annotation parsing; `acquire_generation`
+  ordering and fallback; `pick_model` tiers; `chunk_paragraphs` packing +
+  p_id stamping; `rerank.blend` maths and dedupe; `links.for_source` for
+  all three kinds incl. degraded cases; `text_fragment` encoding;
+  `retrieve` merging three mocked stores; `build_messages` rendering;
+  comment page location on a synthetic 2-page PDF fixture.
+- `tests/bib/test_frlink.py`: highlight fragment, page-cite paragraph
+  upgrade, quote tie-break.
+- Live verification after rollout: `stack llm hosts` shows rig as the pick;
+  `/health` from inside the compose net; one chat via `docker exec git
+  curl` streams from `rig`; three sample rule links opened in the
+  playwright Chromium image scroll to the highlighted paragraph.
+
+## Rollout + re-index
+
+1. Merge, bump `COMMIT_SHA`, `docker compose build llm && docker compose
+   up -d llm`.
+2. On the host: `nohup stack llm index --collection all` (fleet fan-out on
+   all three GPUs). Estimated volume: ~30k new comments, 80 rules
+   (~193k paragraphs → ~80k chunks), 22k corpus items; a few hours on
+   `nomic-embed-text`. The old `rules` chunks are replaced per item
+   (`_delete_old_chunks`), no downtime for `comments`.
+3. Note for later: a nightly `stack llm index --collection comments` step
+   in the existing sync workflow keeps recency honest (out of scope here).
+
+## Out of scope
+
+- Hybrid BM25 + vector retrieval and cross-encoder reranking (better for
+  citation-heavy questions; separate issue).
+- Multi-turn chat memory; the service stays single-shot.
+- P35 tagging chain.
+- Moving the `llm` container itself off the rack — it needs postgres and
+  bib.sqlite; only inference moves.