107 lines
5.4 KiB
Markdown
107 lines
5.4 KiB
Markdown
# llm chat UI — Gitea-SSO-guarded RAG chat
|
|
|
|
**Status:** Designed — approved, implementing on branch `llm-chat-ui`
|
|
**Date:** 2026-07-17
|
|
**Part of:** P34 (LLM Query). Builds on the P33 foundation (`src/llm`: config,
|
|
pool, chunk, source, index) and the AVX-512 pgvector fix (#580).
|
|
|
|
## Goal
|
|
|
|
A web chat at `llm.fhirworx.io`, guarded by the existing Gitea SSO, where a
|
|
signed-in user asks a question and gets a grounded answer synthesized from the
|
|
indexed regulations.gov comments, with the source comments cited. Single-shot
|
|
(stateless) per question; answers stream token-by-token.
|
|
|
|
## Architecture
|
|
|
|
A new `llm` FastAPI service (`src/llm/api.py`, run as `uvicorn llm.api:app`),
|
|
its own image (`infra/images/llm.Dockerfile`, `[images.llm]`), and compose
|
|
service `llm` on the `gateway` + `storage` + `data` networks (gateway for
|
|
Traefik, storage to reach `postgres:5432`, data to reach `ollama:11434`).
|
|
Traefik routes `llm.fhirworx.io` → `http://llm:8000` with the `git-sso`
|
|
middleware, exactly like `notebooks`.
|
|
|
|
```
|
|
browser ──TLS──> Traefik ──git-sso(forwardAuth→auth-handler→oauth2-proxy/Gitea)──> llm:8000
|
|
│
|
|
POST /chat {question} ▼
|
|
└─ rag.retrieve: embed (ollama pool) → pgvector similarity_search('comments', k)
|
|
rag.stream_answer: grounded prompt → stream llama3.1:8b (ollama /api/chat)
|
|
└─ SSE: token* , then sources , then done
|
|
```
|
|
|
|
## Components (all `src/llm/`)
|
|
|
|
- **`rag.py`** — pure RAG logic, no web:
|
|
- `retrieve(question, *, cfg, pool, k=6) -> list[Source]` where
|
|
`Source = {comment_id, docket, snippet, score}`. Embeds the question with
|
|
the pool and runs `PGVector.similarity_search_with_score` on the
|
|
`comments` collection.
|
|
- `build_messages(question, sources) -> list[dict]` — a system message
|
|
("answer only from these comment excerpts; cite ids like
|
|
`[CMS-2017-0092-1306]`; if they don't cover it, say so") plus the user
|
|
question with the excerpts.
|
|
- `stream_answer(question, *, cfg, pool) -> Iterator[Event]` — retrieve,
|
|
build messages, then stream `cfg.instruct_model` from a pool host via
|
|
Ollama `POST /api/chat` (`stream: true`, NDJSON). Yields
|
|
`{"type":"token","text":…}` events, then one
|
|
`{"type":"sources","sources":[…]}`, then `{"type":"done"}`. On no
|
|
retrieval hits it still answers (the model abstains per the prompt).
|
|
- **`api.py`** — FastAPI app: `GET /health` (ok + ollama/pg reachability),
|
|
`GET /` (serves `web/chat.html`), `GET /whoami` (returns the
|
|
`X-Auth-Request-User` header value, `""` if absent), `POST /chat`
|
|
(`{"question": str}` → `StreamingResponse` of SSE `data:` lines from
|
|
`stream_answer`; errors become a `{"type":"error","message":…}` event).
|
|
- **`web/chat.html`** — one self-contained page (inline CSS/JS, no framework).
|
|
On load calls `/whoami` to greet the user; a text box + Send POSTs to
|
|
`/chat` and reads the streamed body, appending tokens and rendering the
|
|
cited sources under the answer. The fhirworx favicon/CSS are injected by
|
|
Traefik (`theme: true`).
|
|
- **`infra/images/llm.Dockerfile`** — mirrors `api.Dockerfile`: uv base,
|
|
`uv sync --no-dev --extra llm`, `COPY stack.toml`, `CMD uvicorn
|
|
llm.api:app --host 0.0.0.0 --port 8000`; curl healthcheck on `/health`.
|
|
- **CLI** — `stack llm serve` (uvicorn locally) alongside `stack llm index`.
|
|
|
|
## Wiring (routing + SSO)
|
|
|
|
- `infra/traefik/dynamic/services.yml` `$reef`: add
|
|
`"llm" (dict "port" "8000" "theme" true "mw" "git-sso,secure-headers")`.
|
|
- `compose.yml`: `llm` service (container_name `llm`) on gateway/storage/data;
|
|
env `LLM_OLLAMA_HOSTS=http://ollama:11434`, `LLM_PG_HOST=postgres`,
|
|
`LLM_DB_PASSWORD`; `depends_on` postgres(healthy)+ollama; promtail label;
|
|
build section + `[images.llm]` in stack.toml.
|
|
- Register the `llm` subdomain in `stack.toml [platform].subdomains`,
|
|
`dev/scripts/bootstrap_certs.py` `HOSTS_SUBDOMAINS`, and
|
|
`dev/scripts/bootstrap_sso.py` `SUBDOMAINS`. Wildcard `*.fhirworx.io` cert
|
|
and `.fhirworx.io` oauth2-proxy cookie already cover it — no cert/oauth2
|
|
change. Regenerate CoreDNS via `gen_config.py`.
|
|
|
|
## Identity
|
|
|
|
The chain forwards only `X-Auth-Request-User` / `X-Auth-Request-Email`. The UI
|
|
shows the username; the app does not re-authenticate (Traefik already
|
|
enforced SSO — anyone reaching the app is authenticated).
|
|
|
|
## Error handling
|
|
|
|
- Ollama or pgvector unreachable → `/chat` emits an `error` event and the UI
|
|
shows it in the transcript; `/health` reports degraded.
|
|
- Empty question → 400. Retrieval returns nothing → the model answers that it
|
|
has nothing indexed on the topic (grounded-prompt abstention).
|
|
|
|
## Testing
|
|
|
|
- `rag`: retrieval assembles `Source`s from mocked pool+vectorstore;
|
|
`build_messages` includes ids and the abstention instruction; `stream_answer`
|
|
yields token→sources→done given a mocked Ollama NDJSON stream.
|
|
- `api`: `TestClient` over `/health`, `/`, `/whoami` (header parsed), and
|
|
`/chat` (mocked `stream_answer`, asserts SSE framing + error path).
|
|
- 99% coverage bar; DB/ollama seams behind `# pragma: no cover` only where a
|
|
live server is required.
|
|
|
|
## Scope
|
|
|
|
v1 answers over the `comments` collection as indexed (pilot docket, ~15.9k
|
|
chunks). Multi-turn history, corpus collection, and a collection selector are
|
|
out of scope — coverage grows by running `stack llm index`, no UI change.
|