[nb] api_explorer.py: exception: [Errno 111] Connection refused #546

New Issue

kert · 2026-07-10T13:51:20Z

kert commented

2026-07-10 13:51:20 +00:00

Auto-filed by the notebook error filer (nb-sig:828fcadc8dc3 — do not edit this marker; dedup and auto-close key on it).

Notebook: api_explorer.py
Error: exception: [Errno 111] Connection refused
Source: nb-watcher

Traceback (most recent call last):
  File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
    yield
  File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 250, in handle_request
    resp = self._pool.handle_request(req)
  File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request
    raise exc from None
  File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
    response = connection.handle_request(
        pool_request.request
    )
  File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpcore/_sync/connection.py", line 101, in handle_request
    raise exc
  File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpcore/_sync/connection.py", line 78, in handle_request
    stream = self._connect(request)
  File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpcore/_sync/connection.py", line 124, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
  File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpcore/_backends/sync.py", line 207, in connect_tcp
    with map_exceptions(exc_map):
         ~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/kert/.local/share/uv/python/cpython-3.13.14-linux-x86_64-gnu/lib/python3.13/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tmp/marimo_10259/__marimo__cell_bkHC_.py", line 1, in <module>
    _r = client.get("/health")
  File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpx/_client.py", line 1053, in get
    return self.request(
           ~~~~~~~~~~~~^
        "GET",
        ^^^^^^
    ...<7 lines>...
        extensions=extensions,
        ^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpx/_client.py", line 825, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
        request,
    ...<2 lines>...
        history=[],
    )
  File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
        request,
        follow_redirects=follow_redirects,
        history=history,
    )
  File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpx/_client.py", line 1014, in _send_single_request
    response = transport.handle_request(request)
  File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 249, in handle_request
    with map_httpcore_exceptions():
         ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/kert/.local/share/uv/python/cpython-3.13.14-linux-x86_64-gnu/lib/python3.13/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [Errno 111] Connection refused

This issue is closed automatically when the error stops occurring.

Auto-filed by the notebook error filer (`nb-sig:828fcadc8dc3` — do not edit this marker; dedup and auto-close key on it). - **Notebook:** `api_explorer.py` - **Error:** `exception`: [Errno 111] Connection refused - **Source:** nb-watcher ``` Traceback (most recent call last): File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions yield File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 250, in handle_request resp = self._pool.handle_request(req) File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request raise exc from None File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request response = connection.handle_request( pool_request.request ) File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpcore/_sync/connection.py", line 101, in handle_request raise exc File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpcore/_sync/connection.py", line 78, in handle_request stream = self._connect(request) File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpcore/_sync/connection.py", line 124, in _connect stream = self._network_backend.connect_tcp(**kwargs) File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpcore/_backends/sync.py", line 207, in connect_tcp with map_exceptions(exc_map): ~~~~~~~~~~~~~~^^^^^^^^^ File "/home/kert/.local/share/uv/python/cpython-3.13.14-linux-x86_64-gnu/lib/python3.13/contextlib.py", line 162, in __exit__ self.gen.throw(value) ~~~~~~~~~~~~~~^^^^^^^ File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions raise to_exc(exc) from exc httpcore.ConnectError: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/tmp/marimo_10259/__marimo__cell_bkHC_.py", line 1, in <module> _r = client.get("/health") File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpx/_client.py", line 1053, in get return self.request( ~~~~~~~~~~~~^ "GET", ^^^^^^ ...<7 lines>... extensions=extensions, ^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpx/_client.py", line 825, in request return self.send(request, auth=auth, follow_redirects=follow_redirects) ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpx/_client.py", line 914, in send response = self._send_handling_auth( request, ...<2 lines>... history=[], ) File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpx/_client.py", line 942, in _send_handling_auth response = self._send_handling_redirects( request, follow_redirects=follow_redirects, history=history, ) File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpx/_client.py", line 979, in _send_handling_redirects response = self._send_single_request(request) File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpx/_client.py", line 1014, in _send_single_request response = transport.handle_request(request) File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 249, in handle_request with map_httpcore_exceptions(): ~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/kert/.local/share/uv/python/cpython-3.13.14-linux-x86_64-gnu/lib/python3.13/contextlib.py", line 162, in __exit__ self.gen.throw(value) ~~~~~~~~~~~~~~^^^^^^^ File "/home/kert/workspace/.venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions raise mapped_exc(message) from exc httpx.ConnectError: [Errno 111] Connection refused ``` This issue is closed automatically when the error stops occurring.

kert added the notebooks label 2026-07-10 14:07:30 +00:00

kert referenced this issue from a commit

2026-07-10 18:17:19 +00:00

fix(notebooks): api_explorer target URL + zotero_tutorial polars schema inference

kert commented

2026-07-10 18:37:34 +00:00

Fixed in 9b58e2dfb2 + 81dba5ef2a. Root cause: the api service was wedged — /health → _check_bib() → list_items()[:1] hydrated the entire bib store (one get() per row, ~140 s at the 181k items the Zotero sync reached) and leaked the connection, pinning a threadpool thread per hit until the pool (40) was exhausted; the container then sat unhealthy for ~12 h with ~1500 leaked healthcheck zombie processes. Fixes: SQL-level limit pushdown, count() as a single COUNT query, explicit Store.close() in the health check, curl --max-time 4 in the image healthcheck, and the image now installs the cli extra bundle (a bare uv sync --no-dev shipped no uvicorn). Redeployed as fhirworx/api:9b58e2df — /health answers in <300 ms from the notebooks container. Separately, the notebook itself pointed at http://localhost:8080 (a host-side dev server that never exists in the container) — fixed in e8f6385bc5 to default to http://api:8000 with a STACK_API_URL override. Verified: headless export in the prod container now has zero cell errors.

Fixed in https://git.fhirworx.io/homelab/stack/commit/9b58e2dfb2646b0e8f0eace4b2939657c03518e1 + https://git.fhirworx.io/homelab/stack/commit/81dba5ef2a22c24ffff1e7687f1b5813d32a49bc. Root cause: the api service was wedged — `/health` → `_check_bib()` → `list_items()[:1]` hydrated the entire bib store (one `get()` per row, ~140 s at the 181k items the Zotero sync reached) and leaked the connection, pinning a threadpool thread per hit until the pool (40) was exhausted; the container then sat unhealthy for ~12 h with ~1500 leaked healthcheck zombie processes. Fixes: SQL-level `limit` pushdown, `count()` as a single COUNT query, explicit `Store.close()` in the health check, `curl --max-time 4` in the image healthcheck, and the image now installs the `cli` extra bundle (a bare `uv sync --no-dev` shipped no uvicorn). Redeployed as `fhirworx/api:9b58e2df` — `/health` answers in <300 ms from the notebooks container. Separately, the notebook itself pointed at `http://localhost:8080` (a host-side dev server that never exists in the container) — fixed in https://git.fhirworx.io/homelab/stack/commit/e8f6385bc547f8d808ee9acbe0597daa26a663e8 to default to `http://api:8000` with a `STACK_API_URL` override. Verified: headless export in the prod container now has zero cell errors.

kert closed this issue

2026-07-10 18:37:34 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#546