docs(observability): port plan from corwins.media to stack

Design doc covering: Jaeger→Tempo swap, cadvisor + nvidia-exporter, promtail docker_sd opt-in, per-service log pipelines, Loki retention, Grafana OAuth via Gitea, and six per-service dashboards. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 10:12:31 -04:00
parent db7ea65287
commit 52af0a4f8e
1 changed files with 434 additions and 0 deletions
--- a/docs/superpowers/specs/2026-05-01-observability-port-design.md
+++ b/docs/superpowers/specs/2026-05-01-observability-port-design.md
@@ -0,0 +1,434 @@
+# Observability port from corwins.media → stack
+
+**Date:** 2026-05-01
+**Status:** Draft (awaiting user approval before implementation planning)
+
+## Goal
+
+Port the corwins.media observability approach to the stack repo: replace Jaeger
+with Tempo, add cadvisor + nvidia-exporter, switch promtail from filesystem
+log scraping to `docker_sd_configs` with `promtail=true` opt-in, add per-service
+log pipeline stages tailored to the stack's services, enable Loki retention,
+wire Grafana OAuth via Gitea, and provision six thematic dashboards.
+
+## Reference: corwins.media baseline
+
+| Component | corwins.media file |
+|---|---|
+| Loki | `infra/observability/loki.yaml` |
+| Promtail | `infra/observability/promtail.yaml` |
+| Prometheus | `infra/observability/prometheus.yaml` |
+| OTel collector | `infra/observability/otel-collector.yaml` |
+| Tempo | `infra/observability/tempo.yaml` |
+| Grafana datasources | `infra/observability/grafana/provisioning/datasources/datasources.yaml` |
+| Grafana dashboards | `infra/observability/grafana/dashboards/*.json` |
+
+## Architecture
+
+### Compose changes
+
+**Add services** (all on `observability` network):
+- `cadvisor` — container CPU/mem/disk/net metrics, scrapes `/`, `/sys`, docker.sock; runs `privileged: true` to read cgroup data
+- `tempo` — OTLP trace backend, replaces Jaeger
+- `nvidia-exporter` — DCGM GPU metrics, `runtime: nvidia`, GPU device reservation
+
+**Remove services:**
+- `jaeger`
+
+**Modify services:**
+- `traefik`: enable JSON access logs (`--accesslog=true --accesslog.format=json --accesslog.fields.headers.defaultMode=keep`)
+- `promtail`: drop the `/var/lib/docker/containers` bind mount, mount `/var/run/docker.sock` (read-only), add `group_add: ["${DOCKER_GID:-999}"]`
+- All eligible services (see table below): add `labels: ["promtail=true"]`
+- `grafana`: add OAuth env vars + `env_file: .state/gitea/grafana.env (required: false)`
+
+**Volumes:** add `tempo_data`. Existing `loki_data`, `prometheus_data`, `grafana_data` unchanged.
+
+**Networks:** unchanged. `observability` stays `internal: true`. cadvisor, tempo, nvidia-exporter, promtail need only docker.sock + host filesystems, not internet egress.
+
+### Subdomain / route changes
+- `stack.toml` `subdomains`: replace `"jaeger"` with `"tempo"`
+- `infra/traefik/dynamic/`: replace jaeger router with tempo router (port `3200`)
+- `bootstrap_sso.py` `SUBDOMAINS`: replace `"jaeger"` with `"tempo"`
+
+### `promtail=true` label coverage
+
+| Container | Label | Notes |
+|---|---|---|
+| traefik | yes | gateway access logs |
+| coredns | yes | DNS resolution events |
+| rustfs | yes | S3 op logs |
+| postgres | yes | DB events |
+| git (gitea) | yes | login/push/oauth/webhook |
+| act-runner | yes | CI job logs |
+| notebooks | yes | marimo events |
+| zotero | yes | rare manual debug |
+| webdav | yes | rclone access |
+| nessie | yes | Iceberg catalog |
+| trino | yes | query lifecycle |
+| polaris | yes | Iceberg catalog |
+| dashboard | yes | nginx access |
+| docs | yes | docusaurus access |
+| api | yes | FastAPI events |
+| mail-poller | yes | poll cycle / ingest |
+| auth-handler | yes | oauth2 flow |
+| oauth2-proxy | yes | AuthSuccess/Failure |
+| cloudflared | yes | tunnel events |
+| jaeger → tempo | yes | self-monitoring |
+| loki | yes | self-monitoring |
+| prometheus | yes | self-monitoring |
+| grafana | yes | self-monitoring |
+| otel-collector | yes | self-monitoring |
+| cadvisor | yes | self-monitoring |
+| mc | no | sleep-infinity sidecar, no useful logs |
+| wire | no | one-shot bootstrap, profile=tools |
+
+## Configs
+
+### `infra/loki/loki-config.yml`
+
+Add retention + compactor (port from corwins.media `loki.yaml`):
+
+```yaml
+limits_config:
+  retention_period: 168h            # 7d
+  reject_old_samples: true
+  reject_old_samples_max_age: 168h
+  ingestion_rate_mb: 16
+  ingestion_burst_size_mb: 32
+  max_query_series: 5000
+  allow_structured_metadata: true   # keep stack-specific
+
+compactor:
+  working_directory: /loki/compactor
+  compaction_interval: 10m
+  retention_enabled: true
+  retention_delete_delay: 2h
+  delete_request_store: filesystem
+```
+
+Drop `query_range.results_cache` (corwins doesn't use it; saves a bit of memory). Keep `analytics.reporting_enabled: false`.
+
+### `infra/loki/promtail-config.yml` (full rewrite)
+
+Top-level:
+
+```yaml
+server:
+  http_listen_port: 9080
+
+positions:
+  filename: /tmp/positions.yaml
+
+clients:
+  - url: http://loki:3100/loki/api/v1/push
+
+scrape_configs:
+  - job_name: docker
+    docker_sd_configs:
+      - host: unix:///var/run/docker.sock
+        refresh_interval: 15s
+    relabel_configs:
+      - source_labels: ['__meta_docker_container_label_promtail']
+        regex: 'true'
+        action: keep
+      - source_labels: ['__meta_docker_container_name']
+        regex: '/?(.*)'
+        target_label: container
+      - source_labels: ['__meta_docker_container_name']
+        regex: '/?(.*)'
+        target_label: job
+      - source_labels: ['__meta_docker_container_label_com_docker_compose_service']
+        target_label: service
+      - source_labels: ['__meta_docker_container_label_com_docker_compose_project']
+        target_label: project
+    pipeline_stages:
+      - docker: {}
+      # Generic: log level
+      - regex:
+          expression: '(?i)(?P<level>error|warn|info|debug|fatal|critical|trace)'
+      - labels:
+          level:
+      # Generic: HTTP access line (works for any container that emits common-style logs)
+      - regex:
+          expression: '(?P<remote_ip>\d+\.\d+\.\d+\.\d+) .* "(?P<http_method>GET|POST|PUT|DELETE|PATCH|HEAD|OPTIONS) (?P<http_path>[^ ]*) [^"]*" (?P<http_status>\d{3}) (?P<http_bytes>\d+)'
+      - labels:
+          http_method:
+          http_status:
+
+      # --- Per-service stages (selectors below) ---
+```
+
+Per-service stages (each is a `match: { selector: '{container="X"}', stages: [...] }` block):
+
+| Container | Extracted labels |
+|---|---|
+| `traefik` | json stage parsing access log → `tf_router`, `tf_service`, `http_status`, `http_method`; regex stage for non-access lines: `tf_event` (TLS handshake / certResolver / EntryPoint / router) |
+| `git` | `gitea_event` ∈ {login, logout, register, push, pull, branch, tag, issue, webhook, oauth, created, deleted, merged} |
+| `oauth2-proxy` | `oauth_event` ∈ {AuthSuccess, AuthFailure, OAuthStart, OAuthCallback, OAuthError, Invalid, Expired, Authenticated, Forbidden} |
+| `postgres` | `pg_event` ∈ {FATAL, ERROR, WARNING, checkpoint, autovacuum, connection authorized, connection received, statement} |
+| `trino` | `trino_event` ∈ {QUERY_CREATED, QUERY_STARTED, QUERY_COMPLETED, QUERY_FAILED, SPLIT_COMPLETED}; `query_id` from `\b([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})\b` |
+| `nessie`, `polaris` | Quarkus standard access — `quarkus_route` (path prefix), `http_status` |
+| `rustfs` | `s3_op` ∈ {PutObject, GetObject, DeleteObject, ListObjectsV2, HeadObject, CompleteMultipart}; `s3_bucket` |
+| `api` | `api_event` ∈ {pipeline, ingest, health, error}; `pipeline_name` if present |
+| `mail-poller` | `mail_event` ∈ {ingest, poll, error}; `mail_count` (numeric) |
+| `act-runner` | `ci_event` ∈ {job started, job finished, job failed, runner registered}; `job_id` |
+| `cloudflared` | `cf_event` ∈ {connection, registered, unregistered, reconnect, quic, error, tunnel, origin}; `cf_origin_status` |
+
+All other promtail-eligible containers receive only the generic stages.
+
+### `infra/tempo/tempo.yml` (new)
+
+Port from corwins.media `tempo.yaml`:
+
+```yaml
+server:
+  http_listen_port: 3200
+
+distributor:
+  receivers:
+    otlp:
+      protocols:
+        grpc: { endpoint: 0.0.0.0:4317 }
+        http: { endpoint: 0.0.0.0:4318 }
+
+ingester:
+  trace_idle_period: 30s
+  max_block_bytes: 1048576
+  max_block_duration: 5m
+
+storage:
+  trace:
+    backend: local
+    local:
+      path: /var/tempo/traces
+    wal:
+      path: /var/tempo/wal
+```
+
+### `infra/otel/otel-collector.yml`
+
+Replace traces exporter:
+
+```diff
+ exporters:
+-  otlp_grpc/jaeger:
+-    endpoint: jaeger:4317
+-    tls:
+-      insecure: true
+  otlp_http/tempo:
+    endpoint: http://tempo:4318
+    tls:
+      insecure: true
+```
+
+```diff
+ processors:
+   batch:
+     timeout: 5s
+     send_batch_size: 1024
+  memory_limiter:
+    check_interval: 5s
+    limit_mib: 256
+    spike_limit_mib: 64
+   resource:
+     attributes:
+       - key: deployment.environment
+         value: homelab
+         action: upsert
+```
+
+```diff
+ service:
+   pipelines:
+     traces:
+       receivers: [otlp]
+-      processors: [batch, resource]
+-      exporters: [otlp_grpc/jaeger]
+      processors: [memory_limiter, batch, resource]
+      exporters: [otlp_http/tempo]
+     metrics:
+       receivers: [otlp]
+-      processors: [batch]
+      processors: [memory_limiter, batch]
+       exporters: [prometheus]
+     logs:
+       receivers: [otlp]
+-      processors: [batch, resource]
+      processors: [memory_limiter, batch, resource]
+       exporters: [otlp_http/loki]
+```
+
+### `infra/prometheus/targets/services.yml`
+
+```diff
+-# Jaeger
+-- targets: ['jaeger:14269']
+-  labels:
+-    job: jaeger
+# Tempo
+- targets: ['tempo:3200']
+  labels:
+    job: tempo
+    __metrics_path__: /metrics
+
+# cadvisor
+- targets: ['cadvisor:8080']
+  labels:
+    job: cadvisor
+
+# NVIDIA GPU
+- targets: ['nvidia-exporter:9400']
+  labels:
+    job: nvidia-gpu
+```
+
+### `infra/grafana/provisioning/datasources/datasources.yml`
+
+```yaml
+apiVersion: 1
+datasources:
+  - name: Loki
+    type: loki
+    access: proxy
+    url: http://loki:3100
+    uid: loki
+    isDefault: true
+    jsonData:
+      maxLines: 1000
+
+  - name: Prometheus
+    type: prometheus
+    access: proxy
+    url: http://prometheus:9090
+    uid: prometheus
+    jsonData:
+      httpMethod: POST
+
+  - name: Tempo
+    type: tempo
+    access: proxy
+    url: http://tempo:3200
+    uid: tempo
+    jsonData:
+      tracesToLogsV2:
+        datasourceUid: loki
+        filterByTraceID: true
+      serviceMap:
+        datasourceUid: prometheus
+```
+
+### Grafana OAuth via Gitea
+
+Extend `dev/scripts/bootstrap_sso.py` to register a second OAuth app and write `.state/gitea/grafana.env`:
+
+```python
+GRAFANA_ENV = STATE / "grafana.env"
+
+def _write_grafana_env(client_id: str, client_secret: str) -> bool:
+    return _write_env_file(
+        GRAFANA_ENV,
+        (
+            f"GF_AUTH_GENERIC_OAUTH_CLIENT_ID={client_id}\n"
+            f"GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET={client_secret}\n"
+        ),
+    )
+
+# in main(), after the oauth2-proxy app is provisioned:
+grafana_creds = _ensure_oauth_app(
+    client,
+    token,
+    name="grafana",
+    redirect_uri=f"https://grafana.{DOMAIN}/login/generic_oauth",
+)
+if grafana_creds:
+    _write_grafana_env(*grafana_creds)
+```
+
+Compose changes for `grafana`:
+
+```yaml
+grafana:
+  # ... existing fields ...
+  env_file:
+    - path: .state/gitea/grafana.env
+      required: false
+  environment:
+    # existing GF_SECURITY_ADMIN_*, GF_USERS_*, GF_AUTH_ANONYMOUS_*, GF_SERVER_ROOT_URL
+    # ... plus:
+    - GF_AUTH_GENERIC_OAUTH_ENABLED=true
+    - GF_AUTH_GENERIC_OAUTH_NAME=Gitea
+    - GF_AUTH_GENERIC_OAUTH_SCOPES=openid profile email
+    - GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://git.${DOMAIN:-fhirworx.io}/login/oauth/authorize
+    - GF_AUTH_GENERIC_OAUTH_TOKEN_URL=http://git:3000/login/oauth/access_token
+    - GF_AUTH_GENERIC_OAUTH_API_URL=http://git:3000/api/v1/user
+    - GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=true
+    - GF_AUTH_GENERIC_OAUTH_AUTO_LOGIN=false
+    - GF_AUTH_OAUTH_ALLOW_INSECURE_EMAIL_LOOKUP=true
+    - GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH=is_admin && 'GrafanaAdmin'
+    - GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_GRAFANA_ADMIN=true
+```
+
+Grafana already has both `gateway` and `observability` networks, which is exactly what's needed: TOKEN_URL/API_URL traverse `gateway` to reach `git:3000`.
+
+## Per-service Grafana dashboards
+
+Six thematic dashboards under `infra/grafana/dashboards/` (auto-loaded by the existing provisioning config):
+
+### 1. `homelab-overview.json`
+- **Stat row:** total containers up, total log volume (Loki) /1m, total error rate (Loki `level="error"`) /1m
+- **Time series:** container CPU% from cadvisor (`rate(container_cpu_usage_seconds_total{container_label_promtail="true"}[1m])`), container memory from cadvisor
+- **Logs:** top error sources by container (Loki bar chart)
+
+### 2. `gateway-auth.json`
+- **Time series:** Traefik request rate by status code, p50/p95/p99 latency from `traefik_service_request_duration_seconds`
+- **Time series:** oauth2-proxy AuthSuccess vs AuthFailure rate (from Loki `oauth_event`)
+- **Time series:** Gitea login + push event rate (from Loki `gitea_event`)
+- **Logs:** failed-auth tail (Loki: `{container=~"oauth2-proxy|git"} |~ "(?i)(failure|forbidden|401|403)"`)
+
+### 3. `data-lake.json`
+- **Stat:** Trino queries running / completed last hour / failed last hour
+- **Time series:** Trino query duration histogram (from `trino_event="QUERY_COMPLETED"`)
+- **Time series:** Nessie + Polaris HTTP request rate + error rate (from Quarkus exporter `/q/metrics`)
+- **Time series:** RustFS S3 op count by `s3_op` (from Loki)
+
+### 4. `data-pipelines.json`
+- **Time series:** api FastAPI request rate + status (Prometheus `stack_*` from otel-collector)
+- **Time series:** mail-poller poll cycles + ingest counts (from Loki `mail_event`, `mail_count`)
+- **Time series:** pipeline-run rate by `pipeline_name`
+- **Logs:** pipeline error tail
+
+### 5. `ci.json`
+- **Stat:** act-runner active jobs, jobs in last 24h, fail rate
+- **Time series:** act-runner `ci_event` by status
+- **Time series:** Gitea push + webhook events (Loki `gitea_event`)
+
+### 6. `gpu-notebooks.json`
+- **Time series:** GPU utilization, memory used, temperature, power (DCGM `DCGM_FI_DEV_*`)
+- **Time series:** notebooks + zotero container CPU/mem (cadvisor)
+
+The existing `pipeline-performance.json` is folded into `data-pipelines.json` if its panels overlap; otherwise kept.
+
+## Migration plan
+
+| Step | Description | Verification |
+|---|---|---|
+| 1 | Add `tempo`, `cadvisor`, `nvidia-exporter` services. Don't remove jaeger yet. | `docker compose config` parses |
+| 2 | Write new `loki-config.yml` (retention/compactor) and `tempo.yml`. | `docker compose up -d loki tempo` healthy |
+| 3 | Rewrite `promtail-config.yml` with docker_sd + per-service stages. | promtail container starts; `curl loki:3100/loki/api/v1/labels` shows expected labels |
+| 4 | Update `otel-collector.yml`: traces → tempo (keep jaeger receiver until step 6). | `curl tempo:3200/ready` returns ready; trace appears in Grafana Explore |
+| 5 | Add `promtail=true` labels to all services in `compose.yml` + enable Traefik access logs. | `curl 'loki:3100/loki/api/v1/label/container/values'` returns full container set |
+| 6 | Update `datasources.yml` (Loki default, Tempo replaces Jaeger, correlation). Update `prometheus/targets/services.yml`. Update `traefik/dynamic/` route. Update `stack.toml` and `bootstrap_sso.py` SUBDOMAINS. | Grafana Explore: query each datasource; click trace span → loki query opens |
+| 7 | Extend `bootstrap_sso.py` with grafana OAuth app. Add OAuth env vars + `env_file` to `grafana` service. Run `docker compose run --rm wire`. | Grafana login page shows "Sign in with Gitea"; admin user logs in via Gitea |
+| 8 | Provision six dashboards under `infra/grafana/dashboards/`. | All dashboards visible and rendering |
+| 9 | Remove `jaeger` service, jaeger volume references, jaeger route. | `docker compose up -d` succeeds with no jaeger references |
+
+## Out of scope
+
+- Alert rules (Loki ruler / Prometheus alertmanager) — separate effort
+- Long-term metrics storage (Mimir / remote write) — 7d Prometheus retention is sufficient for now
+- Log archival to S3 (rustfs) — Loki retention deletes old data; archival can be a follow-up
+- Tempo metrics-from-traces (span metrics) — requires Mimir or remote write target
+
+## Open questions
+
+None — all design decisions are settled by the user's answers in the brainstorming dialogue.