llm: extend HostPool fan-out from embeddings to chat/generation #654

New Issue

kert · 2026-08-18T19:24:20Z

kert commented

2026-08-18 19:24:20 +00:00

llm.pool.HostPool does least-loaded fan-out with health + model checks, but only embed_texts uses it (raw /api/embed). Add the generation analog:

generate_texts/chat fan-out over /api/chat (or the OpenAI-compat /v1/chat/completions) with the same least-loaded acquire() discipline, per-host in-flight limit (1-2 — Ollama serializes per-model anyway), timeouts, and tool-call passthrough.
Per-host model check before dispatch (a host missing the model is dropped, not errored).
Heterogeneous throughput is the point: the pool's least-loaded policy naturally gives the 4090 more work than the 3060 — no static sharding needed.
Consumers: prisma screening (companion issue), llm RAG tagging batch runner (#575), #615 post-close comment re-index, #577 golden-set eval.

`llm.pool.HostPool` does least-loaded fan-out with health + model checks, but only `embed_texts` uses it (raw `/api/embed`). Add the generation analog: - `generate_texts`/`chat` fan-out over `/api/chat` (or the OpenAI-compat `/v1/chat/completions`) with the same least-loaded `acquire()` discipline, per-host in-flight limit (1-2 — Ollama serializes per-model anyway), timeouts, and tool-call passthrough. - Per-host model check before dispatch (a host missing the model is dropped, not errored). - Heterogeneous throughput is the point: the pool's least-loaded policy naturally gives the 4090 more work than the 3060 — no static sharding needed. - Consumers: prisma screening (companion issue), llm RAG tagging batch runner (#575), #615 post-close comment re-index, #577 golden-set eval.

kert added this to the P45: GPU Fleet — pooled inference across rack 3060 / rig 4090 / laptop 5070 Ti milestone 2026-08-18 19:24:20 +00:00

kert added the infra llm labels 2026-08-18 19:24:20 +00:00

kert commented

2026-08-18 19:28:19 +00:00

Design refinement: the generation fan-out should call /api/chat with options: {num_ctx: 8192} (per-request) instead of requiring the custom qwen2.5:14b-ctx8k Modelfile variant on every host. Then each fleet member needs only a vanilla ollama pull qwen2.5:14b, and context size travels with the request rather than with per-host model state. (The OpenAI-compat /v1 endpoint cannot carry num_ctx, so the pool should speak native /api/chat — it already speaks native /api/embed for the same reason.)

Design refinement: the generation fan-out should call `/api/chat` with `options: {num_ctx: 8192}` (per-request) instead of requiring the custom `qwen2.5:14b-ctx8k` Modelfile variant on every host. Then each fleet member needs only a vanilla `ollama pull qwen2.5:14b`, and context size travels with the request rather than with per-host model state. (The OpenAI-compat `/v1` endpoint cannot carry `num_ctx`, so the pool should speak native `/api/chat` — it already speaks native `/api/embed` for the same reason.)