fleet: expose + register rig 4090 and laptop 5070 Ti Ollama endpoints #653
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: homelab/stack#653
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Make the remote GPUs reachable, addressable, and health-checkable from the rack:
ollama serveon rig (4090) and laptop (5070 Ti), bound so the rack can reach them (LAN/tailnet address; document which). Neither host is currently resolvable from the rack (no tailscale, norighostname — verified 2026-08-18 during the palliative screening run, which is why it ran on the 3060 alone).LLM_OLLAMA_HOSTSin.env(comma-separated, beats[llm].ollamaperllm/config.py) and/or a[llm].hostslist in stack.toml with per-host comments (rack/rig/laptop, VRAM).stack llm hosts(or similar) CLI that prints each host's reachability, loaded models, and VRAM — thin wrapper overHostPool.check().ollama pullthe working set on each host (currentlyqwen2.5:14b-ctx8k— note it is a local Modelfile variant withnum_ctx 8192; the create step must be repeated per host — plusnomic-embed-text/bge-m3for the embed pool).llm/pool.py(says "laptop 5080"; the laptop is a 5070 Ti 16GB).Laptop caveat to document: it sleeps/roams — the pool's
check()already tolerates a missing host, but batch runners should re-check between batches, not only at startup.Design decision (user direction 2026-08-18): the laptop joins as a Docker Swarm worker running the
ollama/ollamaimage, not a bare endpoint.--gpus; usedefault-runtime: nvidiain the laptop daemon.json (nvidia-container-toolkit) +NVIDIA_VISIBLE_DEVICES=all, pinned vianode.labels.gpu==5070ti. (Alternative:node-generic-resourcesGPU advertisement +generic_resourcesreservation — more ceremony, unneeded for a one-GPU node.)11435:11434and the rack reaches the laptop model athttp://127.0.0.1:11435regardless of the laptop's DHCP address —LLM_OLLAMA_HOSTSgets a stable local URL.docker swarm initis additive next to the compose stack; open 2377/tcp 7946/tcp+udp 4789/udp between the machines).check()between batches already tolerates); Windows/Docker Desktop cannot do swarm+GPU (no default-runtime, no device requests in services) — if the laptop is not Linux, fall back to native Ollama.ollama_laptoppersists pulls across service restarts; pulls can be driven remotely viaPOST /api/pullthrough the mesh port.Swarm approach invalidated — the rack runs ROOTLESS Docker (rootlesskit + slirp4netns,
unix:///run/user/1000/docker.sock). Swarm mode does not function rootless:swarm initappears to succeed (node ls shows Leader) but the 2377 control listener stays inside the rootless netns (only published container ports are forwarded; laptop join → connection refused) and the VXLAN overlay data plane cannot work there. Phantom swarm state removed (docker swarm leave --force).Revised design of record: containerized-but-swarmless —
ollama/ollamaimage with the same nvidia device-request GPU mechanism the rack compose service uses (proven under rootless on this OS):docker run --gpus 1 -p 11434:11434 -e OLLAMA_KEEP_ALIVE=5m -v ollama_models:/root/.ollama ollama/ollama(+nvidia-ctk config --set nvidia-container-cli.no-cgroups=truefor rootless);notebook.localresolves from the rack (verified, 192.168.1.7, ~2ms) —LLM_OLLAMA_HOSTS=http://127.0.0.1:11434,http://notebook.local:11434;Laptop is operational in the fleet (2026-08-18):
ollama/ollamaimage + nvidia device-request mechanism as the rack compose service; container seesCUDA0: RTX 5070 Ti Laptop GPU— note it is the Laptop variant with 11.5 GiB usable VRAM, not the desktop 16 GB (32B-class models remain a rig-4090-only option).notebook.local(mDNS) + ufw scoped to exactly the rack (allow from 192.168.1.5 to any port 11434 proto tcp). Rack IP should get a DHCP reservation so the rule stays valid.qwen2.5:14bpulled remotely via/api/pull; placement verified fully resident: 9,803/9,803 MiB in VRAM.LLM_OLLAMA_HOSTS=http://127.0.0.1:11434,http://notebook.local:11434in .env;stack prisma screen --hosts ...(fleet driver, #655) consumes it.Remaining for this issue: the rig 4090 (same recipe),
stack llm hostshealthcheck CLI, stale pool.py docstring.Rig 4090 is in the fleet (2026-08-18): same container recipe,
rig.localmDNS, ufw scoped to the rack.qwen2.5:14b+qwen2.5:32bpulled remotely; 32B fully VRAM-resident (20,756 MiB).Measured decode (identical request, num_ctx 8192):
Combined 14B fleet ≈ 5.4x a lone 3060. First 4090 job: 32B adjudication of the 116 stage-2 uncertains + 2 no-tool-call stragglers (fleet driver, single-host mode).
LLM_OLLAMA_HOSTSnow lists all three hosts. Remaining here:stack llm hostshealthcheck CLI + pool.py docstring fix.Closing as delivered. State on 2026-09-11 (stack llm hosts, live): rack http://127.0.0.1:11434 12 GB up (bge-m3, llama3.1:8b, nomic-embed-text, qwen2.5:14b, qwen2.5:14b-ctx8k); rig http://rig.local:11434 24 GB up (nomic-embed-text, qwen2.5:14b, qwen2.5:14b-ctx32k, qwen2.5:32b) — chats generate there; laptop http://notebook.local:11434 registered, DOWN at the moment (it sleeps/roams — the documented caveat; HostPool.check tolerates it and pick_model falls back to the rack). LLM_OLLAMA_HOSTS carries url@vram for all three; the laptop's declared VRAM corrected from 12 to 16 GB (5070 Ti). stack llm hosts prints reachability, VRAM and models and names the generation host/model; the /hosts API endpoint does the same for the service. The stale '5080' comment in llm/pool.py was already fixed. Remaining operational note, not code: when the laptop is up, pull the working set there (qwen2.5:14b-ctx8k is a local Modelfile variant — create it per host; nomic-embed-text/bge-m3 for the embed pool), and batch runners should re-check hosts between batches (#655).