docs: drop the cadvisor task and mentions from the observability plan and spec
Some checks failed
CI / lint (push) Has been cancelled
CI / test (push) Has been cancelled
CI / notebooks-smoke (push) Has been cancelled
Infra CI / notebooks (push) Has been cancelled
Infra CI / zotero (push) Has been cancelled
Infra CI / docs (push) Has been cancelled
Infra CI / api (push) Has been cancelled
Infra CI / llm (push) Has been cancelled
Infra CI / mc (push) Has been cancelled
CI / lint (pull_request) Successful in 33s
CI / notebooks-smoke (pull_request) Successful in 1m32s
Infra CI / notebooks (pull_request) Successful in 1m12s
Infra CI / zotero (pull_request) Successful in 14s
Infra CI / docs (pull_request) Successful in 13s
Infra CI / api (pull_request) Successful in 12s
Infra CI / llm (pull_request) Successful in 17s
Infra CI / mc (pull_request) Successful in 13s
CI / test (pull_request) Successful in 17m57s

This commit is contained in:
kert
2026-09-08 21:23:33 -04:00
parent caf43846c5
commit aadf037b5e
3 changed files with 13 additions and 112 deletions

View File

@@ -2,11 +2,11 @@
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Replace Jaeger with Tempo, add cadvisor + nvidia-exporter, switch promtail from filesystem scraping to docker_sd opt-in, port per-service log pipelines to the stack's services, enable Loki retention, wire Grafana SSO via Gitea, and provision six per-service dashboards.
**Goal:** Replace Jaeger with Tempo, add nvidia-exporter, switch promtail from filesystem scraping to docker_sd opt-in, port per-service log pipelines to the stack's services, enable Loki retention, wire Grafana SSO via Gitea, and provision six per-service dashboards.
**Architecture:** Compose-native — every change lands in `compose.yml`, an `infra/<component>/<config>.yml`, or a JSON dashboard. The full stack runs on Podman/Docker with the existing networks (`gateway`, `observability`, `storage`, `data`, `ci`). New services live on `observability`; tempo + cadvisor + nvidia-exporter only need docker.sock and host filesystem reads, so the network's `internal: true` constraint is preserved. Grafana already has `gateway + observability`, which is exactly what's needed for OAuth token exchange to `git:3000` plus internal datasource access.
**Tech Stack:** Docker Compose, Loki, Promtail, Grafana, Prometheus, Tempo, OpenTelemetry Collector, cAdvisor, DCGM-exporter (NVIDIA), Traefik, Gitea (OAuth provider).
**Tech Stack:** Docker Compose, Loki, Promtail, Grafana, Prometheus, Tempo, OpenTelemetry Collector, DCGM-exporter (NVIDIA), Traefik, Gitea (OAuth provider).
**Spec:** `docs/superpowers/specs/2026-05-01-observability-port-design.md`
@@ -27,11 +27,11 @@
- `tests/test_observability.py` — compose invariant tests for the new layout
**Modified:**
- `compose.yml` — add tempo/cadvisor/nvidia-exporter, drop jaeger, label all services with `promtail=true`, Traefik access logs, Grafana OAuth env
- `compose.yml` — add tempo/nvidia-exporter, drop jaeger, label all services with `promtail=true`, Traefik access logs, Grafana OAuth env
- `infra/loki/loki-config.yml` — retention + compactor
- `infra/loki/promtail-config.yml` — full rewrite (docker_sd + per-service)
- `infra/otel/otel-collector.yml` — traces → tempo, add memory_limiter
- `infra/prometheus/targets/services.yml` — add cadvisor/nvidia/tempo, drop jaeger
- `infra/prometheus/targets/services.yml` — add nvidia/tempo, drop jaeger
- `infra/grafana/provisioning/datasources/datasources.yml` — Tempo replaces Jaeger; Loki default; correlation
- `infra/traefik/dynamic/services.yml` — swap jaeger → tempo route
- `stack.toml` — `subdomains` swap jaeger → tempo
@@ -224,89 +224,6 @@ git commit -m "feat(observability): add Tempo trace backend"
---
## Task 2: cadvisor service
**Files:**
- Modify: `compose.yml`
- Test: `tests/test_observability.py`
- [ ] **Step 2.1: Write the failing test**
Append to `tests/test_observability.py`:
```python
class TestCadvisor:
def test_cadvisor_service_present(self):
svcs = _services()
assert "cadvisor" in svcs
def test_cadvisor_privileged(self):
# cadvisor needs privileged for cgroup reads
assert _services()["cadvisor"].get("privileged") is True
def test_cadvisor_mounts(self):
svc = _services()["cadvisor"]
sources = []
for v in svc["volumes"]:
if isinstance(v, str):
sources.append(v.split(":")[0])
# rootfs, sys, /dev/disk needed for resource introspection
assert any(s == "/" for s in sources)
assert any(s == "/sys" for s in sources)
```
- [ ] **Step 2.2: Run test to verify it fails**
```bash
uv run pytest tests/test_observability.py::TestCadvisor -v
```
Expected: FAIL.
- [ ] **Step 2.3: Add cadvisor to compose.yml**
Insert after the `tempo` service block:
```yaml
cadvisor:
image: gcr.io/cadvisor/cadvisor:latest
container_name: cadvisor
networks:
- observability
privileged: true
volumes:
- /:/rootfs:ro
- ${DOCKER_SOCK:-/run/user/1000/docker.sock}:/var/run/docker.sock:ro
- /sys:/sys:ro
- /dev/disk/:/dev/disk:ro
healthcheck:
test: ["CMD", "wget", "-q", "--spider", "http://localhost:8080/healthz"]
interval: 30s
timeout: 5s
retries: 5
start_period: 30s
restart: unless-stopped
```
- [ ] **Step 2.4: Run tests, bring service up, verify**
```bash
uv run pytest tests/test_observability.py::TestCadvisor -v
docker compose up -d cadvisor
docker exec cadvisor wget -q -O- http://localhost:8080/healthz
# expected: "ok"
docker exec cadvisor wget -q -O- http://localhost:8080/metrics | grep -c '^container_cpu_usage_seconds_total'
# expected: > 0
```
- [ ] **Step 2.5: Commit**
```bash
git add compose.yml tests/test_observability.py
git commit -m "feat(observability): add cadvisor for container metrics"
```
---
## Task 3: nvidia-exporter (DCGM) service
**Files:**
@@ -342,7 +259,7 @@ Expected: FAIL.
- [ ] **Step 3.3: Add service**
Insert after the `cadvisor` service block:
Insert after the `tempo` service block:
```yaml
nvidia-exporter:
@@ -972,7 +889,6 @@ class TestPrometheusTargets:
targets = yaml.safe_load(path.read_text())
jobs = {entry["labels"]["job"] for entry in targets if "labels" in entry}
assert "tempo" in jobs
assert "cadvisor" in jobs
assert "nvidia-gpu" in jobs
assert "jaeger" not in jobs
```
@@ -1008,11 +924,6 @@ Expected: FAIL.
labels:
job: loki
# cAdvisor
- targets: ['cadvisor:8080']
labels:
job: cadvisor
# NVIDIA GPU (DCGM)
- targets: ['nvidia-exporter:9400']
labels:
@@ -1049,14 +960,14 @@ uv run pytest tests/test_observability.py::TestPrometheusTargets -v
# Hot-reload (web.enable-lifecycle is on)
docker exec prometheus wget -q --post-data= -O- http://localhost:9090/-/reload
docker exec prometheus wget -q -O- 'http://localhost:9090/api/v1/targets?state=active' | grep -o '"job":"[^"]*"' | sort -u
# expected: cadvisor, nvidia-gpu, tempo, traefik, loki, etc.
# expected: nvidia-gpu, tempo, traefik, loki, etc.
```
- [ ] **Step 8.5: Commit**
```bash
git add infra/prometheus/targets/services.yml tests/test_observability.py
git commit -m "feat(prom): add cadvisor/nvidia/tempo targets, drop jaeger"
git commit -m "feat(prom): add nvidia/tempo targets, drop jaeger"
```
---
@@ -1177,7 +1088,7 @@ PROMTAIL_LABELED = {
"notebooks", "zotero", "webdav", "nessie", "trino", "polaris",
"dashboard", "docs", "api", "mail-poller", "auth-handler",
"oauth2-proxy", "cloudflared", "loki", "promtail", "tempo",
"cadvisor", "otel-collector", "prometheus", "grafana",
"otel-collector", "prometheus", "grafana",
"nvidia-exporter",
}
@@ -1240,7 +1151,7 @@ need the entry appended.
Special cases:
- `promtail` itself gets `promtail=true` (it scrapes its own logs)
- `mc` and `wire` (the `tools` profile) get **no** label — see test
- `tempo`, `cadvisor`, `nvidia-exporter` were created in tasks 1–3; add the
- `tempo`, `nvidia-exporter` were created in tasks 1–3; add the
label to those service blocks too
- [ ] **Step 10.4: Enable Traefik JSON access logs**
@@ -1874,7 +1785,7 @@ Cross-check against the spec sections:
| Spec section | Plan task |
|---|---|
| Compose changes — add tempo/cadvisor/nvidia-exporter | Tasks 1–3 |
| Compose changes — add tempo/nvidia-exporter | Tasks 1–3 |
| Compose changes — remove jaeger | Task 13 |
| Compose changes — promtail=true labels | Task 10 |
| Compose changes — Traefik access logs | Task 10 |

View File

@@ -6,7 +6,7 @@
## Goal
Port the corwins.media observability approach to the stack repo: replace Jaeger
with Tempo, add cadvisor + nvidia-exporter, switch promtail from filesystem
with Tempo, add nvidia-exporter, switch promtail from filesystem
log scraping to `docker_sd_configs` with `promtail=true` opt-in, add per-service
log pipeline stages tailored to the stack's services, enable Loki retention,
wire Grafana OAuth via Gitea, and provision six thematic dashboards.
@@ -28,7 +28,6 @@ wire Grafana OAuth via Gitea, and provision six thematic dashboards.
### Compose changes
**Add services** (all on `observability` network):
- `cadvisor` — container CPU/mem/disk/net metrics, scrapes `/`, `/sys`, docker.sock; runs `privileged: true` to read cgroup data
- `tempo` — OTLP trace backend, replaces Jaeger
- `nvidia-exporter` — DCGM GPU metrics, `runtime: nvidia`, GPU device reservation
@@ -43,7 +42,7 @@ wire Grafana OAuth via Gitea, and provision six thematic dashboards.
**Volumes:** add `tempo_data`. Existing `loki_data`, `prometheus_data`, `grafana_data` unchanged.
**Networks:** unchanged. `observability` stays `internal: true`. cadvisor, tempo, nvidia-exporter, promtail need only docker.sock + host filesystems, not internet egress.
**Networks:** unchanged. `observability` stays `internal: true`. tempo, nvidia-exporter, promtail need only docker.sock + host filesystems, not internet egress.
### Subdomain / route changes
- `stack.toml` `subdomains`: replace `"jaeger"` with `"tempo"`
@@ -78,7 +77,6 @@ wire Grafana OAuth via Gitea, and provision six thematic dashboards.
| prometheus | yes | self-monitoring |
| grafana | yes | self-monitoring |
| otel-collector | yes | self-monitoring |
| cadvisor | yes | self-monitoring |
| mc | no | sleep-infinity sidecar, no useful logs |
| wire | no | one-shot bootstrap, profile=tools |
@@ -271,11 +269,6 @@ Replace traces exporter:
+ job: tempo
+ __metrics_path__: /metrics
+
+# cadvisor
+- targets: ['cadvisor:8080']
+ labels:
+ job: cadvisor
+
+# NVIDIA GPU
+- targets: ['nvidia-exporter:9400']
+ labels:
@@ -376,7 +369,6 @@ Six thematic dashboards under `infra/grafana/dashboards/` (auto-loaded by the ex
### 1. `homelab-overview.json`
- **Stat row:** total containers up, total log volume (Loki) /1m, total error rate (Loki `level="error"`) /1m
- **Time series:** container CPU% from cadvisor (`rate(container_cpu_usage_seconds_total{container_label_promtail="true"}[1m])`), container memory from cadvisor
- **Logs:** top error sources by container (Loki bar chart)
### 2. `gateway-auth.json`
@@ -404,7 +396,6 @@ Six thematic dashboards under `infra/grafana/dashboards/` (auto-loaded by the ex
### 6. `gpu-notebooks.json`
- **Time series:** GPU utilization, memory used, temperature, power (DCGM `DCGM_FI_DEV_*`)
- **Time series:** notebooks + zotero container CPU/mem (cadvisor)
The existing `pipeline-performance.json` is folded into `data-pipelines.json` if its panels overlap; otherwise kept.
@@ -412,7 +403,7 @@ The existing `pipeline-performance.json` is folded into `data-pipelines.json` if
| Step | Description | Verification |
|---|---|---|
| 1 | Add `tempo`, `cadvisor`, `nvidia-exporter` services. Don't remove jaeger yet. | `docker compose config` parses |
| 1 | Add `tempo`, `nvidia-exporter` services. Don't remove jaeger yet. | `docker compose config` parses |
| 2 | Write new `loki-config.yml` (retention/compactor) and `tempo.yml`. | `docker compose up -d loki tempo` healthy |
| 3 | Rewrite `promtail-config.yml` with docker_sd + per-service stages. | promtail container starts; `curl loki:3100/loki/api/v1/labels` shows expected labels |
| 4 | Update `otel-collector.yml`: traces → tempo (keep jaeger receiver until step 6). | `curl tempo:3200/ready` returns ready; trace appears in Grafana Explore |

View File

@@ -135,7 +135,6 @@ class TestPrometheusTargets:
targets = yaml.safe_load(path.read_text())
jobs = {entry["labels"]["job"] for entry in targets if "labels" in entry}
assert "tempo" in jobs
assert "cadvisor" not in jobs
assert "nvidia-gpu" in jobs
assert "jaeger" not in jobs