stack/docs/superpowers/specs/2026-09-03-llm-corpus-recency-links-gpu-design.md

# llm service — whole-library retrieval, recency ranking, exact-paragraph evidence links, largest-GPU routing

**Status:** Draft — awaiting approval
**Date:** 2026-09-03
**Milestones:** P34 (#571 grounded generation with citations), P45 (#654 fan-out to
generation), plus #615 (re-index of CMS-2026-2377)
**Builds on:** `docs/superpowers/specs/2026-07-16-llm-module-design.md`,
`2026-07-17-llm-chat-ui-design.md`, `2026-08-17-fr-jump-links-design.md`

## Goal

Four changes to the `llm` chat service (`src/llm`, `llm.fhirworx.io`):

1. **Whole-library retrieval.** Answers draw on the entire Zotero/bib library —
   public comments, Federal Register rules, and the reference corpus (PubMed
   records, manuals, regulations, downloads, mail sources) — not just the
   `comments` collection.
2. **Recency-first.** Retrieval prefers the most recently posted comments and
   the newest rules/sources when relevance is comparable, and the answer states
   dates.
3. **Exact evidence links.** Every source shown under an answer carries a deep
   link that lands on the exact passage: the FR paragraph anchor plus a
   scroll-to-text highlight for rules, the comment page or the attachment PDF
   page for comments, and the item URL (with PDF page when known) for corpus
   items. The FR jump-link resolver gains the same highlight so notebook
   links land on the passage too.
4. **Largest live GPU.** Generation runs on the largest GPU that is up at
   request time (today the rig 4090), with a bigger instruct model when that
   host can hold it. Embedding keeps fanning out across every live host.

All inference stays on the local Ollama fleet — no cloud APIs.

## Current state (measured 2026-09-03)

| Fact | Value |
|---|---|
| pgvector collections | `comments` 732,716 chunks / 166,919 items (last indexed 2026-08-14); `rules` 7,884 chunks / **3 of 80** rules; no `corpus` collection |
| bib comments | 196,739 (≈30k not yet indexed); `date_published` = regulations.gov posted date, latest 2026-08-19 |
| bib non-comment items | 22,470 (15,242 with abstracts; 848 with local attachments, 3,402 files: txt/xlsx/pdf/csv) |
| Zotero-only PDFs | 1,129 bib-keyed Zotero parents hold a storage PDF; 848 of those are in bib `attachments` — the rest (PRISMA stage-3 fetches) live only in `data/zotero/data/storage` |
| FR anchor maps | 80 rules, 193,220 paragraph anchors in `fr_anchors` (p_id, page, ordinal, text) |
| Ollama hosts | rack 3060 12 GB (`ollama`, 0.32.1), laptop 5070 Ti 12 GB (`notebook.local`), rig 4090 24 GB (`rig.local`); all three up; `qwen2.5:14b` on all, `qwen2.5:32b` only on rig, `llama3.1:8b` (current `instruct_model`) only on rack |
| Generation routing today | `pool.acquire()` picks least in-flight, so the rack 3060 always wins; `check(instruct_model)` drops every host but the rack anyway |
| Rule text | indexed from the FR `.txt` attachment — chunk → paragraph mapping is lost |
| Comment attachment text | `combined.md` sections per attachment (`## attachment_1.pdf`); PDF pages joined with `\n\n`, page boundaries lost |

The FR live page (browser UA) carries `id="p-N"` on every paragraph and
`id="page-NNNNN"` markers; a non-browser UA gets a 10 KB "Request Access"
page. `#p-N` anchors are valid but land on the paragraph's top edge under the
site header, and page cites (`91 FR 44242` → `#page-44242`) land on a page
marker that sits mid-paragraph — that is the "not quite the exact paragraph"
symptom.

## Decisions

| Decision | Choice | Why |
|---|---|---|
| Corpus scope | Three collections: `comments`, `rules` (rebuilt from `fr_anchors`), `corpus` (every non-comment bib item; abstract + local attachment text + Zotero-storage PDF text) | Matches the library as the user sees it in Zotero; `fr_anchors` gives exact paragraph provenance for free |
| Cross-collection retrieval | Query each collection, over-fetch, merge, re-rank in Python | Three PGVector stores already exist; no schema change; per-kind `k` keeps rules from drowning comments |
| Recency | Soft boost: `score = similarity × (1 − w) + w × recency`, recency = `exp(−age / half_life)`; optional hard `since` date on the API | Relevance must still win on clearly better hits; a hard cutoff alone would hide older authoritative rules |
| Rule chunking | One chunk per FR paragraph (packing tiny neighbours up to ~2,000 chars); metadata `p_id`, `page`, `html_url` | Chunk == anchor, so the evidence link is exact by construction |
| Comment page mapping | At index time, locate each attachment-derived chunk's opening text in the local PDF's per-page text (pymupdf); store `attachment`, `page` | No re-extraction of 28k comments; PDFs are already on disk |
| Link rendering | `llm/links.py` builds `url` + `label` per source; FR links get a `:~:text=` scroll-to-text fragment; `frlink.resolve(..., highlight=True)` gains the same | Text fragments highlight and centre the passage in Chromium/Safari/Firefox ≥131 and fall back to `#p-N` elsewhere |
| GPU routing | Hosts declare VRAM inline (`http://rig.local:11434@24`); generation goes to the largest live host serving the model; embeds stay least-loaded | Ollama exposes no GPU-size API; declared size + live probe is the honest "largest available at any time" |
| Models | `instruct_model = qwen2.5:14b` everywhere; `instruct_model_large = qwen2.5:32b` when the chosen host declares ≥ `large_min_vram_gb` (20) and serves it; `num_ctx` passed per request (8,192) | Both already pulled; 32b is VRAM-resident on the 4090 (43 tok/s measured); no more Modelfile ctx variants |
| Rollout | Update `.env` host lists, `docker compose build llm && up -d llm`; re-index on the host with nohup across the fleet | Same procedure as every other roll (see deploy_rollout memory) |

## Architecture

```
bib.sqlite ──┬─ comments (combined.md / abstract) ─┐
             ├─ fr_anchors paragraphs (rules)      ├─> llm.source → llm.chunk → embed (fleet fan-out) → pgvector
             └─ corpus items (+ Zotero storage PDFs)┘         metadata: kind, date, url, p_id/page/attachment

POST /chat {question, since?}
   └─ rag.retrieve: embed question once → 3 × similarity_search_with_score (over-fetch)
        → rerank.recency_blend → top-N mixed sources
   └─ links.for_source: kind-specific deep link + label
   └─ pool.acquire_generation(): largest live host by declared VRAM → model tier → /api/chat (stream)
   └─ SSE: token* , sources (with url/label/date/kind), done
```

## Components

### `llm/config.py`
- `ollama_hosts` becomes `tuple[Host, ...]` where `Host(url, vram_gb)`;
  parse `url@vram` (missing `@` → `vram_gb=0`, sorts last).
- New fields with `[llm]` defaults: `instruct_model_large = ""`,
  `large_min_vram_gb = 20`, `chat_num_ctx = 8192`,
  `recency_half_life_days = 365`, `recency_weight = 0.3`,
  `k_per_kind = {"comment": 8, "rule": 4, "corpus": 4}`, `top_n = 8`.
- `LLM_OLLAMA_HOSTS` env keeps precedence (annotated form).

### `llm/pool.py`
- `HostPool` stores `vram_gb` per host.
- `acquire_generation(model)` → context manager yielding the alive host
  with the largest `vram_gb` that serves `model` (tie → least in-flight);
  raises the existing "no host serves" error when none does.
- `pick_model(cfg, host)` → `instruct_model_large` if
  `host.vram_gb >= large_min_vram_gb` and the host lists it, else
  `instruct_model`. Liveness is re-probed on every chat request (already
  the case via `check`), so a rig outage falls back to the laptop/rack.
- Fix the 5080 → 5070 Ti docstring (#653 note).

### `llm/source.py`
- `iter_rule_docs` rebuilt on `fr_anchors`: yields one `Doc` per rule whose
  text is the anchor paragraphs in `p_id` order, separated by a sentinel
  the chunker respects, with a parallel `paragraphs` list so
  `chunk_rule_doc` can pack whole paragraphs and stamp `p_id`, `page`,
  `html_url` per chunk. Rules without an anchor map fall back to the
  current `.txt` path (metadata without `p_id`).
- `iter_comment_docs` keeps `combined.md`; the chunker's section split
  already isolates each `## attachment_N.ext` — a new
  `attachment_pages(root, docket, cid)` helper returns per-page text for
  PDF attachments; chunk metadata gains `attachment` and `page` (empty when
  not located).
- `iter_corpus_docs` gains `title`, `url`, `date`, `item_type`,
  `project` (from `project:` tags) metadata and a Zotero-storage PDF
  fallback: a `ZoteroPdfIndex` built once per run from a snapshot copy of
  `zotero.sqlite` (never the live file), mapping bib key → storage PDF
  paths; text via the existing rex PDF extractor. Non-PDF attachment text
  keeps the current path.
- Every doc carries `date` (ISO, `date_published`) and `kind`
  (`comment` | `rule` | `corpus`).
- Comment iteration order becomes newest-first (`ORDER BY date_published
  DESC`) so incremental runs surface the latest first.

### `llm/chunk.py`
- `chunk_doc` unchanged for comments/corpus.
- New `chunk_paragraphs(doc, paragraphs)` for rules: greedy pack of whole
  paragraphs up to `target_chars`; a paragraph longer than the target is
  hard-wrapped with overlap but keeps its `p_id`. Metadata per chunk:
  `p_id` (first paragraph), `p_id_last`, `page`.

### `llm/rerank.py` (new, pure)
- `recency(date, now, half_life_days) -> float` in `[0,1]`; undated → 0.
- `blend(hits, *, weight, half_life_days, now) -> list[hit]`: converts
  pgvector cosine distance to similarity, blends, sorts desc, dedupes by
  item key (best chunk per item), returns `top_n`.
- `filter_since(hits, since)` for the hard cutoff.

### `llm/links.py` (new, pure over metadata)
- `for_source(md) -> (url, label)`:
  - comment: `https://www.regulations.gov/comment/{cid}`; when
    `attachment` is set: `https://downloads.regulations.gov/{cid}/{attachment}`
    plus `#page={page}` for PDFs. Label `CMS-2026-2377-3438` (+ ` p.4`).
  - rule: `{html_url}#p-{p_id}` + `:~:text={first ~8 words of snippet}`;
    label `91 FR 43949 ¶4` (page + ordinal looked up from metadata).
  - corpus: item `url` (+ `#page=` when the chunk came from a PDF and the
    URL ends in `.pdf`); label = short title (≤ 60 chars) + year.
- `text_fragment(text) -> str`: first sentence clipped to ~80 chars,
  percent-encoded per the URL Fragment Text Directives spec (`-`, `,`, `&`
  escaped).

### `bib/frlink.py` (tweak)
- `resolve(..., highlight=False)`: when true, paragraph-resolved links
  (`¶k`, raw anchor, quote) append `:~:text=` built from the paragraph's
  opening words; page-only cites additionally upgrade to the first
  paragraph *starting* on that page when one exists (so `91 FR 44242`
  lands on a paragraph, not the mid-paragraph page marker). `md_link`
  passes `highlight` through; `Pincite.jump_url` uses `highlight=True`.
- Quote resolution: when several paragraphs match, prefer the one whose
  text starts with the quote, else the shortest; raise only when still
  tied. Existing single-match behaviour unchanged.

### `llm/rag.py`
- `retrieve(question, *, cfg, pool, since=None)`: one query embedding,
  three stores, `k_per_kind × 3` over-fetch each, `rerank.blend`, then
  `links.for_source` → source dicts
  `{id, kind, label, url, title, date, docket, snippet, score}`.
- `build_messages`: excerpts rendered as `[label] (kind, date) snippet`;
  system prompt tells the model to cite `[label]`, prefer the most recent
  comments when excerpts conflict, and name the year when it matters.
- `stream_answer`: `pool.acquire_generation` + `pick_model`, passes
  `options.num_ctx`; the `sources` event carries the enriched dicts and
  `model`/`host` for the UI footer.

### `llm/api.py` + `web/chat.html`
- `ChatRequest` gains optional `since: str | None`.
- `/health` returns live hosts with `vram_gb` and the host/model generation
  would pick right now.
- UI: sources render as links (`<a target=_blank>`) with kind badge, date,
  and label; a "last 12 months" checkbox sets `since`; the footer shows
  `model @ host`.

### `cli/llm.py`
- `index --collection all` runs comments → rules → corpus; `--newest-first`
  is the default order for comments.
- `stack llm hosts` prints the pool with VRAM, liveness, models, and the
  generation pick (operator check for "which GPU will answer").

### Compose / env
- `.env`: `LLM_OLLAMA_HOSTS` and `LLM_OLLAMA_HOSTS_IN_CONTAINER` gain `@12`,
  `@12`, `@24` annotations; `stack.toml [llm]`: `instruct_model =
  "qwen2.5:14b"`, `instruct_model_large = "qwen2.5:32b"`, new knobs above.
- No compose topology change; the container already reaches both remotes
  (verified `docker exec llm curl 192.168.1.222:11434`).

## Data flow for one question

1. Embed the question once on the least-loaded live host.
2. Similarity search `comments` (k=24), `rules` (k=12), `corpus` (k=12).
3. Convert distance → similarity, blend with recency, apply `since`,
   dedupe per item, keep top 8.
4. Build deep links; render excerpts; stream from the largest live host
   with the tier-appropriate model at `num_ctx` 8,192.
5. Emit sources (with links) and done.

## Error handling

- A host down at request time is dropped by `check`; if the rig is down the
  laptop answers with `qwen2.5:14b`. If no host serves the model the
  stream emits the existing `error` event.
- Missing metadata never breaks a link: `links.for_source` degrades
  rule → page link → FR html_url; comment → comment page; corpus → item
  URL; undated hits get recency 0 (pure similarity).
- Zotero snapshot copy failure logs and skips the Zotero-PDF fallback;
  bib attachments still index.
- Page location failures leave `page` empty; the link falls back to the
  attachment root.

## Testing

- Unit (TDD, `tests/llm`): host annotation parsing; `acquire_generation`
  ordering and fallback; `pick_model` tiers; `chunk_paragraphs` packing +
  p_id stamping; `rerank.blend` maths and dedupe; `links.for_source` for
  all three kinds incl. degraded cases; `text_fragment` encoding;
  `retrieve` merging three mocked stores; `build_messages` rendering;
  comment page location on a synthetic 2-page PDF fixture.
- `tests/bib/test_frlink.py`: highlight fragment, page-cite paragraph
  upgrade, quote tie-break.
- Live verification after rollout: `stack llm hosts` shows rig as the pick;
  `/health` from inside the compose net; one chat via `docker exec git
  curl` streams from `rig`; three sample rule links opened in the
  playwright Chromium image scroll to the highlighted paragraph.

## Rollout + re-index

1. Merge, bump `COMMIT_SHA`, `docker compose build llm && docker compose
   up -d llm`.
2. On the host: `nohup stack llm index --collection all` (fleet fan-out on
   all three GPUs). Estimated volume: ~30k new comments, 80 rules
   (~193k paragraphs → ~80k chunks), 22k corpus items; a few hours on
   `nomic-embed-text`. The old `rules` chunks are replaced per item
   (`_delete_old_chunks`), no downtime for `comments`.
3. Note for later: a nightly `stack llm index --collection comments` step
   in the existing sync workflow keeps recency honest (out of scope here).

## Out of scope

- Hybrid BM25 + vector retrieval and cross-encoder reranking (better for
  citation-heavy questions; separate issue).
- Multi-turn chat memory; the service stays single-shot.
- P35 tagging chain.
- Moving the `llm` container itself off the rack — it needs postgres and
  bib.sqlite; only inference moves.