llm: embed fan-out aborts the whole index run when one pool host dies mid-run #796

New Issue

kert · 2026-09-22T19:51:05Z

kert commented

2026-09-22 19:51:05 +00:00

Seen 2026-09-22 in the scheduled CMS-2026-2377 re-farm (#615): stack llm index --collection comments --docket CMS-2026-2377 died with ConnectError: [Errno -2] Name or service not known raised from llm.pool.embed_texts → client.post(f"{host}/api/embed"). HostPool.check() had already filtered the pool to live hosts at start, but rig.local (mDNS) stopped resolving partway through, and embed_texts.run has no recovery: the first transport error on any batch propagates through future.result() and the whole run exits 1 — every remaining batch, on the hosts that were still fine, is abandoned. The same happened on 2026-09-04 (comments: ConnectError: [Errno -2] Name or service not known in the runner output, #615 thread).

Fix: on httpx.TransportError (ConnectError / ConnectTimeout / ReadTimeout / RemoteProtocolError) from a host, drop that host from the pool and retry the batch on another live host; only when no host is left raise a clear RuntimeError. HostPool.acquire() on an empty pool must raise the same error instead of min()'s ValueError. Keep raise_for_status failures (a host that answers 500) as-is — that is a model/config problem, not a dead host.

Done when: tests/llm/test_pool.py covers (a) a batch that hits a dying host completes on the survivor and the dead host is gone from pool.hosts, (b) every host dying raises RuntimeError naming the last failure; and the re-farm's index step survives a host disappearing mid-run.

Seen 2026-09-22 in the scheduled CMS-2026-2377 re-farm (#615): `stack llm index --collection comments --docket CMS-2026-2377` died with `ConnectError: [Errno -2] Name or service not known` raised from `llm.pool.embed_texts` → `client.post(f"{host}/api/embed")`. `HostPool.check()` had already filtered the pool to live hosts at start, but `rig.local` (mDNS) stopped resolving partway through, and `embed_texts.run` has no recovery: the first transport error on any batch propagates through `future.result()` and the whole run exits 1 — every remaining batch, on the hosts that were still fine, is abandoned. The same happened on 2026-09-04 (`comments: ConnectError: [Errno -2] Name or service not known` in the runner output, #615 thread). **Fix:** on `httpx.TransportError` (ConnectError / ConnectTimeout / ReadTimeout / RemoteProtocolError) from a host, drop that host from the pool and retry the batch on another live host; only when no host is left raise a clear `RuntimeError`. `HostPool.acquire()` on an empty pool must raise the same error instead of `min()`'s `ValueError`. Keep `raise_for_status` failures (a host that answers 500) as-is — that is a model/config problem, not a dead host. **Done when:** `tests/llm/test_pool.py` covers (a) a batch that hits a dying host completes on the survivor and the dead host is gone from `pool.hosts`, (b) every host dying raises `RuntimeError` naming the last failure; and the re-farm's index step survives a host disappearing mid-run.

kert added the llm quality labels 2026-09-22 19:51:05 +00:00

kert referenced this issue from a commit

2026-09-22 19:52:30 +00:00

fix(llm): drop a pool host that dies mid-run and retry the batch on the survivors (refs #796)

kert referenced this issue from a commit

2026-09-22 19:52:30 +00:00

merge: #796 — drop a pool host that dies mid-run and retry the batch on the survivors (refs #796)

kert commented

2026-09-22 19:53:04 +00:00

Fixed in cbc55c9, merged to main as d83e955. embed_texts.run now catches httpx.TransportError from a host, calls the new HostPool.drop(host) and retries the batch on the remaining hosts; when none are left it raises RuntimeError("every Ollama host failed; last <host>: <exc>"). acquire()/acquire_generation() on an empty pool raise the same clear error instead of ValueError from min()/max(). HTTP error responses still surface through raise_for_status. Tests: dying host dropped + batch completes on the survivor (in-flight map clean), every host dying raises, empty-pool acquire is a clear error — tests/llm/test_pool.py 24 passed; llm suite green in the pre-commit run. The deployed llm image needs a roll to pick this up for chat-side embeds; the host-side stack llm index (what the re-farm cron runs) uses it as soon as the checkout is on main.

Fixed in cbc55c9, merged to main as d83e955. `embed_texts.run` now catches `httpx.TransportError` from a host, calls the new `HostPool.drop(host)` and retries the batch on the remaining hosts; when none are left it raises `RuntimeError("every Ollama host failed; last <host>: <exc>")`. `acquire()`/`acquire_generation()` on an empty pool raise the same clear error instead of `ValueError` from `min()`/`max()`. HTTP error responses still surface through `raise_for_status`. Tests: dying host dropped + batch completes on the survivor (in-flight map clean), every host dying raises, empty-pool acquire is a clear error — `tests/llm/test_pool.py` 24 passed; llm suite green in the pre-commit run. The deployed `llm` image needs a roll to pick this up for chat-side embeds; the host-side `stack llm index` (what the re-farm cron runs) uses it as soon as the checkout is on main.

kert closed this issue

2026-09-22 19:53:04 +00:00

kert referenced this issue

2026-09-22 21:09:22 +00:00

llm: RAG tagging chain + batch runner #575

kert referenced this issue from a commit

2026-09-22 21:11:47 +00:00

fix(llm): the tagging judge drops a host that dies mid-run and retries on the next largest, like embed_texts (refs #575, #796)

kert referenced this issue

2026-09-22 22:03:25 +00:00

comments: post-close re-farm + re-index of CMS-2026-2377 #615

Sign in to join this conversation.