stack/docs/superpowers/specs/2026-07-17-llm-chat-ui-design.md

llm chat UI — Gitea-SSO-guarded RAG chat

Status: Designed — approved, implementing on branch llm-chat-ui Date: 2026-07-17 Part of: P34 (LLM Query). Builds on the P33 foundation (src/llm: config, pool, chunk, source, index) and the AVX-512 pgvector fix (#580).

Goal

A web chat at llm.fhirworx.io, guarded by the existing Gitea SSO, where a signed-in user asks a question and gets a grounded answer synthesized from the indexed regulations.gov comments, with the source comments cited. Single-shot (stateless) per question; answers stream token-by-token.

Architecture

A new llm FastAPI service (src/llm/api.py, run as uvicorn llm.api:app), its own image (infra/images/llm.Dockerfile, [images.llm]), and compose service llm on the gateway + storage + data networks (gateway for Traefik, storage to reach postgres:5432, data to reach ollama:11434). Traefik routes llm.fhirworx.io → http://llm:8000 with the git-sso middleware, exactly like notebooks.

browser ──TLS──> Traefik ──git-sso(forwardAuth→auth-handler→oauth2-proxy/Gitea)──> llm:8000
                                                                                     │
   POST /chat {question}                                                             ▼
        └─ rag.retrieve: embed (ollama pool) → pgvector similarity_search('comments', k)
           rag.stream_answer: grounded prompt → stream llama3.1:8b (ollama /api/chat)
           └─ SSE: token* , then sources , then done

Components (all src/llm/)

• rag.py — pure RAG logic, no web:
  • retrieve(question, *, cfg, pool, k=6) -> list[Source] where Source = {comment_id, docket, snippet, score}. Embeds the question with the pool and runs PGVector.similarity_search_with_score on the comments collection.
  • build_messages(question, sources) -> list[dict] — a system message ("answer only from these comment excerpts; cite ids like [CMS-2017-0092-1306]; if they don't cover it, say so") plus the user question with the excerpts.
  • stream_answer(question, *, cfg, pool) -> Iterator[Event] — retrieve, build messages, then stream cfg.instruct_model from a pool host via Ollama POST /api/chat (stream: true, NDJSON). Yields {"type":"token","text":…} events, then one {"type":"sources","sources":[…]}, then {"type":"done"}. On no retrieval hits it still answers (the model abstains per the prompt).
• api.py — FastAPI app: GET /health (ok + ollama/pg reachability), GET / (serves web/chat.html), GET /whoami (returns the X-Auth-Request-User header value, "" if absent), POST /chat ({"question": str} → StreamingResponse of SSE data: lines from stream_answer; errors become a {"type":"error","message":…} event).
• web/chat.html — one self-contained page (inline CSS/JS, no framework). On load calls /whoami to greet the user; a text box + Send POSTs to /chat and reads the streamed body, appending tokens and rendering the cited sources under the answer. The fhirworx favicon/CSS are injected by Traefik (theme: true).
• infra/images/llm.Dockerfile — mirrors api.Dockerfile: uv base, uv sync --no-dev --extra llm, COPY stack.toml, CMD uvicorn llm.api:app --host 0.0.0.0 --port 8000; curl healthcheck on /health.
• CLI — stack llm serve (uvicorn locally) alongside stack llm index.

Wiring (routing + SSO)

• infra/traefik/dynamic/services.yml $reef: add "llm" (dict "port" "8000" "theme" true "mw" "git-sso,secure-headers").
• compose.yml: llm service (container_name llm) on gateway/storage/data; env LLM_OLLAMA_HOSTS=http://ollama:11434, LLM_PG_HOST=postgres, LLM_DB_PASSWORD; depends_on postgres(healthy)+ollama; promtail label; build section + [images.llm] in stack.toml.
• Register the llm subdomain in stack.toml [platform].subdomains, dev/scripts/bootstrap_certs.py HOSTS_SUBDOMAINS, and dev/scripts/bootstrap_sso.py SUBDOMAINS. Wildcard *.fhirworx.io cert and .fhirworx.io oauth2-proxy cookie already cover it — no cert/oauth2 change. Regenerate CoreDNS via gen_config.py.

Identity

The chain forwards only X-Auth-Request-User / X-Auth-Request-Email. The UI shows the username; the app does not re-authenticate (Traefik already enforced SSO — anyone reaching the app is authenticated).

Error handling

• Ollama or pgvector unreachable → /chat emits an error event and the UI shows it in the transcript; /health reports degraded.
• Empty question → 400. Retrieval returns nothing → the model answers that it has nothing indexed on the topic (grounded-prompt abstention).

Testing

• rag: retrieval assembles Sources from mocked pool+vectorstore; build_messages includes ids and the abstention instruction; stream_answer yields token→sources→done given a mocked Ollama NDJSON stream.
• api: TestClient over /health, /, /whoami (header parsed), and /chat (mocked stream_answer, asserts SSE framing + error path).
• 99% coverage bar; DB/ollama seams behind # pragma: no cover only where a live server is required.

Scope

v1 answers over the comments collection as indexed (pilot docket, ~15.9k chunks). Multi-turn history, corpus collection, and a collection selector are out of scope — coverage grows by running stack llm index, no UI change.