llm: multi-host Ollama dispatch (3060/4090/5080) #568

New Issue

kert · 2026-07-16T19:26:08Z

kert commented

2026-07-16 19:26:08 +00:00

Remote GPUs are just Ollama endpoints — rig 4090 and laptop 5080 run ollama serve; batch runners fan out over LAN/tailscale. All bib/pgvector I/O stays on the server; remote boxes contribute pure compute. This is the lightweight version of the fan-out the April extraction spec anticipated for inference workloads.

Scope

LLM_OLLAMA_HOSTS (list) config; least-loaded dispatch across hosts
per-host model availability check at startup; missing host degrades gracefully to remaining pool
works for both embedding (P33) and tagging (P35) runners
docket partitioning via --docket remains the manual fallback

Acceptance

indexer saturates 2+ hosts concurrently; unplugging a host mid-run degrades without data loss

Refs .claude/specs/2026-04-23-comments-extraction-design.md (distributed-GPU notes).

Spec: docs/superpowers/specs/2026-07-16-llm-module-design.md

Remote GPUs are just Ollama endpoints — rig 4090 and laptop 5080 run `ollama serve`; batch runners fan out over LAN/tailscale. All bib/pgvector I/O stays on the server; remote boxes contribute pure compute. This is the lightweight version of the fan-out the April extraction spec anticipated for inference workloads. **Scope** - [ ] `LLM_OLLAMA_HOSTS` (list) config; least-loaded dispatch across hosts - [ ] per-host model availability check at startup; missing host degrades gracefully to remaining pool - [ ] works for both embedding (P33) and tagging (P35) runners - [ ] docket partitioning via `--docket` remains the manual fallback **Acceptance** - indexer saturates 2+ hosts concurrently; unplugging a host mid-run degrades without data loss Refs `.claude/specs/2026-04-23-comments-extraction-design.md` (distributed-GPU notes). Spec: `docs/superpowers/specs/2026-07-16-llm-module-design.md`

kert added this to the P33: LLM Foundation — Ollama, pgvector, embedding index milestone 2026-07-16 19:26:08 +00:00

kert added the infra llm labels 2026-07-16 19:26:08 +00:00

kert commented

2026-07-17 19:37:21 +00:00

Done in P33 — merged to main as 14d279b (branch llm-p33). Pilot docket CMS-2017-0092 indexed end-to-end with exact search; full test gate green at 99.75%.

Done in P33 — merged to main as 14d279b (branch llm-p33). Pilot docket CMS-2017-0092 indexed end-to-end with exact search; full test gate green at 99.75%.

kert closed this issue

2026-07-17 19:37:21 +00:00

kert referenced this issue from a commit

2026-07-17 21:13:19 +00:00

feat(llm): multi-host Ollama pool — least-loaded embed fan-out (refs #568)

kert referenced this issue from a commit

2026-07-17 21:13:19 +00:00

fix(llm): HostPool.check merges in-flight counts; guard release (refs #568)

kert referenced this issue from a commit

2026-07-17 21:13:19 +00:00

test(llm): close P33 coverage gap; widen pool host probe; doc chunk guard (refs #563 #566 #567 #568)

Sign in to join this conversation.