llm: observability — metrics + Grafana panel #579

New Issue

kert · 2026-07-16T19:26:09Z

kert commented

2026-07-16 19:26:09 +00:00

Per stack convention (P26 pipeline telemetry).

Scope

Prometheus metrics: embed/tag throughput, queue depth, per-host dispatch counts, request latency, abstain rate
Grafana panel; alerts optional (GPU covered by nvidia-exporter already)

Acceptance

metrics scraped; panel shows a live tagging run

Spec: docs/superpowers/specs/2026-07-16-llm-module-design.md

Per stack convention (P26 pipeline telemetry). **Scope** - [ ] Prometheus metrics: embed/tag throughput, queue depth, per-host dispatch counts, request latency, abstain rate - [ ] Grafana panel; alerts optional (GPU covered by nvidia-exporter already) **Acceptance** - metrics scraped; panel shows a live tagging run Spec: [`docs/superpowers/specs/2026-07-16-llm-module-design.md`](https://git.fhirworx.io/homelab/stack/src/branch/main/docs/superpowers/specs/2026-07-16-llm-module-design.md)

kert added this to the P35: LLM Tagging — closed-vocab RAG tagging into bib/Zotero milestone 2026-07-16 19:26:09 +00:00

kert added the llm infra labels 2026-07-16 19:26:09 +00:00

kert commented

2026-08-13 23:46:55 +00:00

Status audit (2026-08-13, repo-verified): NOT DONE. No llm-specific Prometheus metrics (no Counter/Histogram in src/llm); infra/grafana/dashboards has 8 dashboards, none llm/ollama. Only generic OTel via perf.

**Status audit (2026-08-13, repo-verified):** NOT DONE. No llm-specific Prometheus metrics (no Counter/Histogram in src/llm); infra/grafana/dashboards has 8 dashboards, none llm/ollama. Only generic OTel via perf.

kert referenced this issue from a commit

2026-09-11 22:59:53 +00:00

feat(llm,infra): OTel metrics for dispatch/embeddings/chat/indexing + LLM Grafana dashboard (refs #579)

kert referenced this issue from a commit

2026-09-11 22:59:53 +00:00

merge: #579 — OTel metrics for dispatch/embeddings/chat/indexing + LLM Grafana dashboard (refs #579)

kert referenced this issue from a commit

2026-09-11 22:59:53 +00:00

fix(perf,infra): actually export the metrics — /metrics route, perf extra in the api/llm images, scrape targets, telemetry on (refs #579)

kert commented

2026-09-11 23:00:37 +00:00

Done: 60018f4 (instruments + dashboard, merged 53fb28e) and 1843c0a (the export path), rolled to llm/api:1843c0a.

Instruments (src/llm/metrics.py, lazy off perf.meter('stack.llm'), no-op when telemetry is off): stack_llm_dispatch_total{host,kind}, stack_llm_inflight{host}, stack_llm_dispatch_seconds{host,kind}, stack_llm_embedded_texts_total{host}, stack_llm_embed_batch_seconds{host}, stack_llm_chat_seconds{stage}, stack_llm_chat_total{outcome}, stack_llm_chat_tokens_total{direction}, stack_llm_indexed_chunks_total{collection}, stack_llm_indexed_items_total{collection,outcome}. Tag throughput and abstain rate are not instrumented because the tagging chain (#575) does not exist yet — add them there.

Dashboard infra/grafana/dashboards/llm.json (uid llm, 11 panels: request rate by status, chat outcomes, dispatch rate and in-flight per host, dispatch p50/p95 by host+kind, chat latency by stage, embedding throughput, chat tokens, indexed chunks/items, log tail).

The finding that mattered: nothing from the P26 telemetry convention had ever reached Prometheus. setup_meter_provider installs only an in-process PrometheusMetricReader; no route served the registry (api and llm answered 404 on /metrics), the api/llm images never installed the perf extra, STACK_TELEMETRY was off, and there was no scrape target — so the data-pipelines request-rate panel had been empty since it was written, and its metric name (stack_http_server_requests_total) never existed. Fixed: perf.middleware mounts GET /metrics, both images install --extra perf, compose enables telemetry for api and llm, services.yml scrapes api:8000 and llm:8000 directly (bypassing the collector's namespace re-prefix), both dashboards query http_server_duration_milliseconds_count{job=…}. Verified live: Prometheus targets api/llm are up and stack_llm_dispatch_total{job="llm",host="http://ollama:11434",kind="embed"} is queryable after one /search call. 'Panel shows a live tagging run' is deferred to #575 with the tagging instruments.

Done: 60018f4 (instruments + dashboard, merged 53fb28e) and 1843c0a (the export path), rolled to llm/api:1843c0a. **Instruments** (src/llm/metrics.py, lazy off perf.meter('stack.llm'), no-op when telemetry is off): stack_llm_dispatch_total{host,kind}, stack_llm_inflight{host}, stack_llm_dispatch_seconds{host,kind}, stack_llm_embedded_texts_total{host}, stack_llm_embed_batch_seconds{host}, stack_llm_chat_seconds{stage}, stack_llm_chat_total{outcome}, stack_llm_chat_tokens_total{direction}, stack_llm_indexed_chunks_total{collection}, stack_llm_indexed_items_total{collection,outcome}. Tag throughput and abstain rate are not instrumented because the tagging chain (#575) does not exist yet — add them there. **Dashboard** infra/grafana/dashboards/llm.json (uid llm, 11 panels: request rate by status, chat outcomes, dispatch rate and in-flight per host, dispatch p50/p95 by host+kind, chat latency by stage, embedding throughput, chat tokens, indexed chunks/items, log tail). **The finding that mattered:** nothing from the P26 telemetry convention had ever reached Prometheus. setup_meter_provider installs only an in-process PrometheusMetricReader; no route served the registry (api and llm answered 404 on /metrics), the api/llm images never installed the perf extra, STACK_TELEMETRY was off, and there was no scrape target — so the data-pipelines request-rate panel had been empty since it was written, and its metric name (stack_http_server_requests_total) never existed. Fixed: perf.middleware mounts GET /metrics, both images install --extra perf, compose enables telemetry for api and llm, services.yml scrapes api:8000 and llm:8000 directly (bypassing the collector's namespace re-prefix), both dashboards query http_server_duration_milliseconds_count{job=…}. Verified live: Prometheus targets api/llm are up and stack_llm_dispatch_total{job="llm",host="http://ollama:11434",kind="embed"} is queryable after one /search call. 'Panel shows a live tagging run' is deferred to #575 with the tagging instruments.

kert closed this issue

2026-09-11 23:00:37 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#579