merge: remove cadvisor from the observability stack (chore/remove-cadvisor)

compose.yml

@@ -790,27 +790,6 @@ services:
       - no-new-privileges:true
     restart: unless-stopped
 
-  cadvisor:
-    image: gcr.io/cadvisor/cadvisor:latest
-    container_name: cadvisor
-    networks:
-      - observability
-    privileged: true
-    volumes:
-      - /:/rootfs:ro
-      - ${DOCKER_SOCK:-/run/user/1000/docker.sock}:/var/run/docker.sock:ro
-      - /sys:/sys:ro
-      - /dev/disk/:/dev/disk:ro
-    healthcheck:
-      test: ["CMD", "wget", "-q", "--spider", "http://localhost:8080/healthz"]
-      interval: 30s
-      timeout: 5s
-      retries: 5
-      start_period: 30s
-    labels:
-      - "promtail=true"
-    restart: unless-stopped
-
   nvidia-exporter:
     image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.9-3.6.1-ubuntu22.04
     # Prefixed: corwins.media compose also defines a "nvidia-exporter"

docs/superpowers/plans/2026-05-01-observability-port.md

@@ -2,11 +2,11 @@
 
 > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
 
-**Goal:** Replace Jaeger with Tempo, add cadvisor + nvidia-exporter, switch promtail from filesystem scraping to docker_sd opt-in, port per-service log pipelines to the stack's services, enable Loki retention, wire Grafana SSO via Gitea, and provision six per-service dashboards.
+**Goal:** Replace Jaeger with Tempo, add nvidia-exporter, switch promtail from filesystem scraping to docker_sd opt-in, port per-service log pipelines to the stack's services, enable Loki retention, wire Grafana SSO via Gitea, and provision six per-service dashboards.
 
-**Architecture:** Compose-native — every change lands in `compose.yml`, an `infra/<component>/<config>.yml`, or a JSON dashboard. The full stack runs on Podman/Docker with the existing networks (`gateway`, `observability`, `storage`, `data`, `ci`). New services live on `observability`; tempo + cadvisor + nvidia-exporter only need docker.sock and host filesystem reads, so the network's `internal: true` constraint is preserved. Grafana already has `gateway + observability`, which is exactly what's needed for OAuth token exchange to `git:3000` plus internal datasource access.
+**Architecture:** Compose-native — every change lands in `compose.yml`, an `infra/<component>/<config>.yml`, or a JSON dashboard. The full stack runs on Podman/Docker with the existing networks (`gateway`, `observability`, `storage`, `data`, `ci`). New services live on `observability`; tempo + nvidia-exporter only need docker.sock and host filesystem reads, so the network's `internal: true` constraint is preserved. Grafana already has `gateway + observability`, which is exactly what's needed for OAuth token exchange to `git:3000` plus internal datasource access.
 
-**Tech Stack:** Docker Compose, Loki, Promtail, Grafana, Prometheus, Tempo, OpenTelemetry Collector, cAdvisor, DCGM-exporter (NVIDIA), Traefik, Gitea (OAuth provider).
+**Tech Stack:** Docker Compose, Loki, Promtail, Grafana, Prometheus, Tempo, OpenTelemetry Collector, DCGM-exporter (NVIDIA), Traefik, Gitea (OAuth provider).
 
 **Spec:** `docs/superpowers/specs/2026-05-01-observability-port-design.md`
 
@@ -27,11 +27,11 @@
 - `tests/test_observability.py` — compose invariant tests for the new layout
 
 **Modified:**
-- `compose.yml` — add tempo/cadvisor/nvidia-exporter, drop jaeger, label all services with `promtail=true`, Traefik access logs, Grafana OAuth env
+- `compose.yml` — add tempo/nvidia-exporter, drop jaeger, label all services with `promtail=true`, Traefik access logs, Grafana OAuth env
 - `infra/loki/loki-config.yml` — retention + compactor
 - `infra/loki/promtail-config.yml` — full rewrite (docker_sd + per-service)
 - `infra/otel/otel-collector.yml` — traces → tempo, add memory_limiter
-- `infra/prometheus/targets/services.yml` — add cadvisor/nvidia/tempo, drop jaeger
+- `infra/prometheus/targets/services.yml` — add nvidia/tempo, drop jaeger
 - `infra/grafana/provisioning/datasources/datasources.yml` — Tempo replaces Jaeger; Loki default; correlation
 - `infra/traefik/dynamic/services.yml` — swap jaeger → tempo route
 - `stack.toml` — `subdomains` swap jaeger → tempo
@@ -224,89 +224,6 @@ git commit -m "feat(observability): add Tempo trace backend"
 
 ---
 
-## Task 2: cadvisor service
-
-**Files:**
-- Modify: `compose.yml`
-- Test: `tests/test_observability.py`
-
-- [ ] **Step 2.1: Write the failing test**
-
-Append to `tests/test_observability.py`:
-
-```python
-class TestCadvisor:
-    def test_cadvisor_service_present(self):
-        svcs = _services()
-        assert "cadvisor" in svcs
-
-    def test_cadvisor_privileged(self):
-        # cadvisor needs privileged for cgroup reads
-        assert _services()["cadvisor"].get("privileged") is True
-
-    def test_cadvisor_mounts(self):
-        svc = _services()["cadvisor"]
-        sources = []
-        for v in svc["volumes"]:
-            if isinstance(v, str):
-                sources.append(v.split(":")[0])
-        # rootfs, sys, /dev/disk needed for resource introspection
-        assert any(s == "/" for s in sources)
-        assert any(s == "/sys" for s in sources)
-```
-
-- [ ] **Step 2.2: Run test to verify it fails**
-
-```bash
-uv run pytest tests/test_observability.py::TestCadvisor -v
-```
-Expected: FAIL.
-
-- [ ] **Step 2.3: Add cadvisor to compose.yml**
-
-Insert after the `tempo` service block:
-
-```yaml
-  cadvisor:
-    image: gcr.io/cadvisor/cadvisor:latest
-    container_name: cadvisor
-    networks:
-      - observability
-    privileged: true
-    volumes:
-      - /:/rootfs:ro
-      - ${DOCKER_SOCK:-/run/user/1000/docker.sock}:/var/run/docker.sock:ro
-      - /sys:/sys:ro
-      - /dev/disk/:/dev/disk:ro
-    healthcheck:
-      test: ["CMD", "wget", "-q", "--spider", "http://localhost:8080/healthz"]
-      interval: 30s
-      timeout: 5s
-      retries: 5
-      start_period: 30s
-    restart: unless-stopped
-```
-
-- [ ] **Step 2.4: Run tests, bring service up, verify**
-
-```bash
-uv run pytest tests/test_observability.py::TestCadvisor -v
-docker compose up -d cadvisor
-docker exec cadvisor wget -q -O- http://localhost:8080/healthz
-# expected: "ok"
-docker exec cadvisor wget -q -O- http://localhost:8080/metrics | grep -c '^container_cpu_usage_seconds_total'
-# expected: > 0
-```
-
-- [ ] **Step 2.5: Commit**
-
-```bash
-git add compose.yml tests/test_observability.py
-git commit -m "feat(observability): add cadvisor for container metrics"
-```
-
----
-
 ## Task 3: nvidia-exporter (DCGM) service
 
 **Files:**
@@ -342,7 +259,7 @@ Expected: FAIL.
 
 - [ ] **Step 3.3: Add service**
 
-Insert after the `cadvisor` service block:
+Insert after the `tempo` service block:
 
 ```yaml
   nvidia-exporter:
@@ -972,7 +889,6 @@ class TestPrometheusTargets:
         targets = yaml.safe_load(path.read_text())
         jobs = {entry["labels"]["job"] for entry in targets if "labels" in entry}
         assert "tempo" in jobs
-        assert "cadvisor" in jobs
         assert "nvidia-gpu" in jobs
         assert "jaeger" not in jobs
 ```
@@ -1008,11 +924,6 @@ Expected: FAIL.
   labels:
     job: loki
 
-# cAdvisor
-- targets: ['cadvisor:8080']
-  labels:
-    job: cadvisor
-
 # NVIDIA GPU (DCGM)
 - targets: ['nvidia-exporter:9400']
   labels:
@@ -1049,14 +960,14 @@ uv run pytest tests/test_observability.py::TestPrometheusTargets -v
 # Hot-reload (web.enable-lifecycle is on)
 docker exec prometheus wget -q --post-data= -O- http://localhost:9090/-/reload
 docker exec prometheus wget -q -O- 'http://localhost:9090/api/v1/targets?state=active' | grep -o '"job":"[^"]*"' | sort -u
-# expected: cadvisor, nvidia-gpu, tempo, traefik, loki, etc.
+# expected: nvidia-gpu, tempo, traefik, loki, etc.
 ```
 
 - [ ] **Step 8.5: Commit**
 
 ```bash
 git add infra/prometheus/targets/services.yml tests/test_observability.py
-git commit -m "feat(prom): add cadvisor/nvidia/tempo targets, drop jaeger"
+git commit -m "feat(prom): add nvidia/tempo targets, drop jaeger"
 ```
 
 ---
@@ -1177,7 +1088,7 @@ PROMTAIL_LABELED = {
     "notebooks", "zotero", "webdav", "nessie", "trino", "polaris",
     "dashboard", "docs", "api", "mail-poller", "auth-handler",
     "oauth2-proxy", "cloudflared", "loki", "promtail", "tempo",
-    "cadvisor", "otel-collector", "prometheus", "grafana",
+    "otel-collector", "prometheus", "grafana",
     "nvidia-exporter",
 }
 
@@ -1240,7 +1151,7 @@ need the entry appended.
 Special cases:
 - `promtail` itself gets `promtail=true` (it scrapes its own logs)
 - `mc` and `wire` (the `tools` profile) get **no** label — see test
-- `tempo`, `cadvisor`, `nvidia-exporter` were created in tasks 1–3; add the
+- `tempo`, `nvidia-exporter` were created in tasks 1–3; add the
   label to those service blocks too
 
 - [ ] **Step 10.4: Enable Traefik JSON access logs**
@@ -1874,7 +1785,7 @@ Cross-check against the spec sections:
 
 | Spec section | Plan task |
 |---|---|
-| Compose changes — add tempo/cadvisor/nvidia-exporter | Tasks 1–3 |
+| Compose changes — add tempo/nvidia-exporter | Tasks 1–3 |
 | Compose changes — remove jaeger | Task 13 |
 | Compose changes — promtail=true labels | Task 10 |
 | Compose changes — Traefik access logs | Task 10 |

docs/superpowers/specs/2026-05-01-observability-port-design.md

@@ -6,7 +6,7 @@
 ## Goal
 
 Port the corwins.media observability approach to the stack repo: replace Jaeger
-with Tempo, add cadvisor + nvidia-exporter, switch promtail from filesystem
+with Tempo, add nvidia-exporter, switch promtail from filesystem
 log scraping to `docker_sd_configs` with `promtail=true` opt-in, add per-service
 log pipeline stages tailored to the stack's services, enable Loki retention,
 wire Grafana OAuth via Gitea, and provision six thematic dashboards.
@@ -28,7 +28,6 @@ wire Grafana OAuth via Gitea, and provision six thematic dashboards.
 ### Compose changes
 
 **Add services** (all on `observability` network):
-- `cadvisor` — container CPU/mem/disk/net metrics, scrapes `/`, `/sys`, docker.sock; runs `privileged: true` to read cgroup data
 - `tempo` — OTLP trace backend, replaces Jaeger
 - `nvidia-exporter` — DCGM GPU metrics, `runtime: nvidia`, GPU device reservation
 
@@ -43,7 +42,7 @@ wire Grafana OAuth via Gitea, and provision six thematic dashboards.
 
 **Volumes:** add `tempo_data`. Existing `loki_data`, `prometheus_data`, `grafana_data` unchanged.
 
-**Networks:** unchanged. `observability` stays `internal: true`. cadvisor, tempo, nvidia-exporter, promtail need only docker.sock + host filesystems, not internet egress.
+**Networks:** unchanged. `observability` stays `internal: true`. tempo, nvidia-exporter, promtail need only docker.sock + host filesystems, not internet egress.
 
 ### Subdomain / route changes
 - `stack.toml` `subdomains`: replace `"jaeger"` with `"tempo"`
@@ -78,7 +77,6 @@ wire Grafana OAuth via Gitea, and provision six thematic dashboards.
 | prometheus | yes | self-monitoring |
 | grafana | yes | self-monitoring |
 | otel-collector | yes | self-monitoring |
-| cadvisor | yes | self-monitoring |
 | mc | no | sleep-infinity sidecar, no useful logs |
 | wire | no | one-shot bootstrap, profile=tools |
 
@@ -271,11 +269,6 @@ Replace traces exporter:
 +    job: tempo
 +    __metrics_path__: /metrics
 +
-+# cadvisor
-+- targets: ['cadvisor:8080']
-+  labels:
-+    job: cadvisor
-+
 +# NVIDIA GPU
 +- targets: ['nvidia-exporter:9400']
 +  labels:
@@ -376,7 +369,6 @@ Six thematic dashboards under `infra/grafana/dashboards/` (auto-loaded by the ex
 
 ### 1. `homelab-overview.json`
 - **Stat row:** total containers up, total log volume (Loki) /1m, total error rate (Loki `level="error"`) /1m
-- **Time series:** container CPU% from cadvisor (`rate(container_cpu_usage_seconds_total{container_label_promtail="true"}[1m])`), container memory from cadvisor
 - **Logs:** top error sources by container (Loki bar chart)
 
 ### 2. `gateway-auth.json`
@@ -404,7 +396,6 @@ Six thematic dashboards under `infra/grafana/dashboards/` (auto-loaded by the ex
 
 ### 6. `gpu-notebooks.json`
 - **Time series:** GPU utilization, memory used, temperature, power (DCGM `DCGM_FI_DEV_*`)
-- **Time series:** notebooks + zotero container CPU/mem (cadvisor)
 
 The existing `pipeline-performance.json` is folded into `data-pipelines.json` if its panels overlap; otherwise kept.
 
@@ -412,7 +403,7 @@ The existing `pipeline-performance.json` is folded into `data-pipelines.json` if
 
 | Step | Description | Verification |
 |---|---|---|
-| 1 | Add `tempo`, `cadvisor`, `nvidia-exporter` services. Don't remove jaeger yet. | `docker compose config` parses |
+| 1 | Add `tempo`, `nvidia-exporter` services. Don't remove jaeger yet. | `docker compose config` parses |
 | 2 | Write new `loki-config.yml` (retention/compactor) and `tempo.yml`. | `docker compose up -d loki tempo` healthy |
 | 3 | Rewrite `promtail-config.yml` with docker_sd + per-service stages. | promtail container starts; `curl loki:3100/loki/api/v1/labels` shows expected labels |
 | 4 | Update `otel-collector.yml`: traces → tempo (keep jaeger receiver until step 6). | `curl tempo:3200/ready` returns ready; trace appears in Grafana Explore |

infra/grafana/dashboards/gpu-notebooks.json

@@ -10,8 +10,6 @@
     {"id": 1, "title": "GPU utilization %", "type": "timeseries", "gridPos": {"x": 0, "y": 0, "w": 12, "h": 8}, "datasource": {"uid": "prometheus"}, "targets": [{"expr": "DCGM_FI_DEV_GPU_UTIL"}]},
     {"id": 2, "title": "GPU memory used (MiB)", "type": "timeseries", "gridPos": {"x": 12, "y": 0, "w": 12, "h": 8}, "datasource": {"uid": "prometheus"}, "targets": [{"expr": "DCGM_FI_DEV_FB_USED"}]},
     {"id": 3, "title": "GPU temp (°C)", "type": "timeseries", "gridPos": {"x": 0, "y": 8, "w": 12, "h": 8}, "datasource": {"uid": "prometheus"}, "targets": [{"expr": "DCGM_FI_DEV_GPU_TEMP"}]},
-    {"id": 4, "title": "GPU power (W)", "type": "timeseries", "gridPos": {"x": 12, "y": 8, "w": 12, "h": 8}, "datasource": {"uid": "prometheus"}, "targets": [{"expr": "DCGM_FI_DEV_POWER_USAGE"}]},
+    {"id": 4, "title": "GPU power (W)", "type": "timeseries", "gridPos": {"x": 12, "y": 8, "w": 12, "h": 8}, "datasource": {"uid": "prometheus"}, "targets": [{"expr": "DCGM_FI_DEV_POWER_USAGE"}]}
-    {"id": 5, "title": "Notebooks/Zotero CPU %", "type": "timeseries", "gridPos": {"x": 0, "y": 16, "w": 12, "h": 8}, "datasource": {"uid": "prometheus"}, "targets": [{"expr": "rate(container_cpu_usage_seconds_total{name=~\"notebooks|zotero\"}[1m]) * 100", "legendFormat": "{{name}}"}]},
-    {"id": 6, "title": "Notebooks/Zotero memory", "type": "timeseries", "gridPos": {"x": 12, "y": 16, "w": 12, "h": 8}, "datasource": {"uid": "prometheus"}, "targets": [{"expr": "container_memory_working_set_bytes{name=~\"notebooks|zotero\"} / 1024 / 1024", "legendFormat": "{{name}}"}]}
   ]
 }

infra/grafana/dashboards/homelab-overview.json

@@ -7,11 +7,8 @@
   "time": {"from": "now-6h", "to": "now"},
   "refresh": "30s",
   "panels": [
-    {"id": 1, "title": "Containers up", "type": "stat", "gridPos": {"x": 0, "y": 0, "w": 4, "h": 4}, "datasource": {"uid": "prometheus"}, "targets": [{"expr": "count(container_last_seen{name!=\"\"})"}]},
+    {"id": 2, "title": "Log volume / 1m", "type": "stat", "gridPos": {"x": 0, "y": 0, "w": 6, "h": 4}, "datasource": {"uid": "loki"}, "targets": [{"expr": "sum(rate({container=~\".+\"}[1m]))"}]},
-    {"id": 2, "title": "Log volume / 1m", "type": "stat", "gridPos": {"x": 4, "y": 0, "w": 4, "h": 4}, "datasource": {"uid": "loki"}, "targets": [{"expr": "sum(rate({container=~\".+\"}[1m]))"}]},
+    {"id": 3, "title": "Error rate / 1m", "type": "stat", "gridPos": {"x": 6, "y": 0, "w": 6, "h": 4}, "datasource": {"uid": "loki"}, "targets": [{"expr": "sum(rate({level=\"error\"}[1m]))"}]},
-    {"id": 3, "title": "Error rate / 1m", "type": "stat", "gridPos": {"x": 8, "y": 0, "w": 4, "h": 4}, "datasource": {"uid": "loki"}, "targets": [{"expr": "sum(rate({level=\"error\"}[1m]))"}]},
+    {"id": 6, "title": "Top error sources (1h)", "type": "logs", "gridPos": {"x": 0, "y": 4, "w": 24, "h": 8}, "datasource": {"uid": "loki"}, "targets": [{"expr": "{level=\"error\"} | json"}]}
-    {"id": 4, "title": "Container CPU %", "type": "timeseries", "gridPos": {"x": 0, "y": 4, "w": 12, "h": 8}, "datasource": {"uid": "prometheus"}, "targets": [{"expr": "rate(container_cpu_usage_seconds_total{name!=\"\"}[1m]) * 100", "legendFormat": "{{name}}"}]},
-    {"id": 5, "title": "Container memory (MiB)", "type": "timeseries", "gridPos": {"x": 12, "y": 4, "w": 12, "h": 8}, "datasource": {"uid": "prometheus"}, "targets": [{"expr": "container_memory_working_set_bytes{name!=\"\"} / 1024 / 1024", "legendFormat": "{{name}}"}]},
-    {"id": 6, "title": "Top error sources (1h)", "type": "logs", "gridPos": {"x": 0, "y": 12, "w": 24, "h": 8}, "datasource": {"uid": "loki"}, "targets": [{"expr": "{level=\"error\"} | json"}]}
   ]
 }

infra/prometheus/targets/services.yml

@@ -19,11 +19,6 @@
   labels:
     job: loki
 
-# cAdvisor
-- targets: ['cadvisor:8080']
-  labels:
-    job: cadvisor
-
 # NVIDIA GPU (DCGM)
 - targets: ['nvidia-exporter:9400']
   labels:

tests/test_compose_mounts.py

@@ -136,7 +136,6 @@ class TestSecurityOpts:
             "docs",  # static site
             "promtail",  # needs host log access
             "cloudflared",  # tunnel agent
-            "cadvisor",  # privileged: true for host metrics
             "nvidia-exporter",  # upstream dcgm-exporter, GPU runtime + SYS_ADMIN
         }
         compose = _load_compose()

tests/test_observability.py

@@ -42,26 +42,6 @@ class TestTempo:
         assert any("./infra/tempo/tempo.yml" in v for v in mounts)
 
 
-class TestCadvisor:
-    def test_cadvisor_service_present(self):
-        svcs = _services()
-        assert "cadvisor" in svcs
-
-    def test_cadvisor_privileged(self):
-        # cadvisor needs privileged for cgroup reads
-        assert _services()["cadvisor"].get("privileged") is True
-
-    def test_cadvisor_mounts(self):
-        svc = _services()["cadvisor"]
-        sources = []
-        for v in svc["volumes"]:
-            if isinstance(v, str):
-                sources.append(v.split(":")[0])
-        # rootfs, sys, /dev/disk needed for resource introspection
-        assert any(s == "/" for s in sources)
-        assert any(s == "/sys" for s in sources)
-
-
 class TestNvidiaExporter:
     def test_nvidia_exporter_present(self):
         svcs = _services()
@@ -155,7 +135,6 @@ class TestPrometheusTargets:
         targets = yaml.safe_load(path.read_text())
         jobs = {entry["labels"]["job"] for entry in targets if "labels" in entry}
         assert "tempo" in jobs
-        assert "cadvisor" in jobs
         assert "nvidia-gpu" in jobs
         assert "jaeger" not in jobs
 
@@ -195,7 +174,6 @@ PROMTAIL_LABELED = {
     "loki",
     "promtail",
     "tempo",
-    "cadvisor",
     "otel-collector",
     "prometheus",
     "grafana",