stack/docs/superpowers/specs/2026-05-01-observability-port-design.md

# Observability port from corwins.media → stack

**Date:** 2026-05-01
**Status:** Draft (awaiting user approval before implementation planning)

## Goal

Port the corwins.media observability approach to the stack repo: replace Jaeger
with Tempo, add nvidia-exporter, switch promtail from filesystem
log scraping to `docker_sd_configs` with `promtail=true` opt-in, add per-service
log pipeline stages tailored to the stack's services, enable Loki retention,
wire Grafana OAuth via Gitea, and provision six thematic dashboards.

## Reference: corwins.media baseline

| Component | corwins.media file |
|---|---|
| Loki | `infra/observability/loki.yaml` |
| Promtail | `infra/observability/promtail.yaml` |
| Prometheus | `infra/observability/prometheus.yaml` |
| OTel collector | `infra/observability/otel-collector.yaml` |
| Tempo | `infra/observability/tempo.yaml` |
| Grafana datasources | `infra/observability/grafana/provisioning/datasources/datasources.yaml` |
| Grafana dashboards | `infra/observability/grafana/dashboards/*.json` |

## Architecture

### Compose changes

**Add services** (all on `observability` network):
- `tempo` — OTLP trace backend, replaces Jaeger
- `nvidia-exporter` — DCGM GPU metrics, `runtime: nvidia`, GPU device reservation

**Remove services:**
- `jaeger`

**Modify services:**
- `traefik`: enable JSON access logs (`--accesslog=true --accesslog.format=json --accesslog.fields.headers.defaultMode=keep`)
- `promtail`: drop the `/var/lib/docker/containers` bind mount, mount `/var/run/docker.sock` (read-only), add `group_add: ["${DOCKER_GID:-999}"]`
- All eligible services (see table below): add `labels: ["promtail=true"]`
- `grafana`: add OAuth env vars + `env_file: .state/gitea/grafana.env (required: false)`

**Volumes:** add `tempo_data`. Existing `loki_data`, `prometheus_data`, `grafana_data` unchanged.

**Networks:** unchanged. `observability` stays `internal: true`. tempo, nvidia-exporter, promtail need only docker.sock + host filesystems, not internet egress.

### Subdomain / route changes
- `stack.toml` `subdomains`: replace `"jaeger"` with `"tempo"`
- `infra/traefik/dynamic/`: replace jaeger router with tempo router (port `3200`)
- `bootstrap_sso.py` `SUBDOMAINS`: replace `"jaeger"` with `"tempo"`

### `promtail=true` label coverage

| Container | Label | Notes |
|---|---|---|
| traefik | yes | gateway access logs |
| coredns | yes | DNS resolution events |
| rustfs | yes | S3 op logs |
| postgres | yes | DB events |
| git (gitea) | yes | login/push/oauth/webhook |
| act-runner | yes | CI job logs |
| notebooks | yes | marimo events |
| zotero | yes | rare manual debug |
| webdav | yes | rclone access |
| nessie | yes | Iceberg catalog |
| trino | yes | query lifecycle |
| polaris | yes | Iceberg catalog |
| dashboard | yes | nginx access |
| docs | yes | docusaurus access |
| api | yes | FastAPI events |
| mail-poller | yes | poll cycle / ingest |
| auth-handler | yes | oauth2 flow |
| oauth2-proxy | yes | AuthSuccess/Failure |
| cloudflared | yes | tunnel events |
| jaeger → tempo | yes | self-monitoring |
| loki | yes | self-monitoring |
| prometheus | yes | self-monitoring |
| grafana | yes | self-monitoring |
| otel-collector | yes | self-monitoring |
| mc | no | sleep-infinity sidecar, no useful logs |
| wire | no | one-shot bootstrap, profile=tools |

## Configs

### `infra/loki/loki-config.yml`

Add retention + compactor (port from corwins.media `loki.yaml`):

```yaml
limits_config:
  retention_period: 168h            # 7d
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  ingestion_rate_mb: 16
  ingestion_burst_size_mb: 32
  max_query_series: 5000
  allow_structured_metadata: true   # keep stack-specific

compactor:
  working_directory: /loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  delete_request_store: filesystem
```

Drop `query_range.results_cache` (corwins doesn't use it; saves a bit of memory). Keep `analytics.reporting_enabled: false`.

### `infra/loki/promtail-config.yml` (full rewrite)

Top-level:

```yaml
server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 15s
    relabel_configs:
      - source_labels: ['__meta_docker_container_label_promtail']
        regex: 'true'
        action: keep
      - source_labels: ['__meta_docker_container_name']
        regex: '/?(.*)'
        target_label: container
      - source_labels: ['__meta_docker_container_name']
        regex: '/?(.*)'
        target_label: job
      - source_labels: ['__meta_docker_container_label_com_docker_compose_service']
        target_label: service
      - source_labels: ['__meta_docker_container_label_com_docker_compose_project']
        target_label: project
    pipeline_stages:
      - docker: {}
      # Generic: log level
      - regex:
          expression: '(?i)(?P<level>error|warn|info|debug|fatal|critical|trace)'
      - labels:
          level:
      # Generic: HTTP access line (works for any container that emits common-style logs)
      - regex:
          expression: '(?P<remote_ip>\d+\.\d+\.\d+\.\d+) .* "(?P<http_method>GET|POST|PUT|DELETE|PATCH|HEAD|OPTIONS) (?P<http_path>[^ ]*) [^"]*" (?P<http_status>\d{3}) (?P<http_bytes>\d+)'
      - labels:
          http_method:
          http_status:

      # --- Per-service stages (selectors below) ---
```

Per-service stages (each is a `match: { selector: '{container="X"}', stages: [...] }` block):

| Container | Extracted labels |
|---|---|
| `traefik` | json stage parsing access log → `tf_router`, `tf_service`, `http_status`, `http_method`; regex stage for non-access lines: `tf_event` (TLS handshake / certResolver / EntryPoint / router) |
| `git` | `gitea_event` ∈ {login, logout, register, push, pull, branch, tag, issue, webhook, oauth, created, deleted, merged} |
| `oauth2-proxy` | `oauth_event` ∈ {AuthSuccess, AuthFailure, OAuthStart, OAuthCallback, OAuthError, Invalid, Expired, Authenticated, Forbidden} |
| `postgres` | `pg_event` ∈ {FATAL, ERROR, WARNING, checkpoint, autovacuum, connection authorized, connection received, statement} |
| `trino` | `trino_event` ∈ {QUERY_CREATED, QUERY_STARTED, QUERY_COMPLETED, QUERY_FAILED, SPLIT_COMPLETED}; `query_id` from `\b([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})\b` |
| `nessie`, `polaris` | Quarkus standard access — `quarkus_route` (path prefix), `http_status` |
| `rustfs` | `s3_op` ∈ {PutObject, GetObject, DeleteObject, ListObjectsV2, HeadObject, CompleteMultipart}; `s3_bucket` |
| `api` | `api_event` ∈ {pipeline, ingest, health, error}; `pipeline_name` if present |
| `mail-poller` | `mail_event` ∈ {ingest, poll, error}; `mail_count` (numeric) |
| `act-runner` | `ci_event` ∈ {job started, job finished, job failed, runner registered}; `job_id` |
| `cloudflared` | `cf_event` ∈ {connection, registered, unregistered, reconnect, quic, error, tunnel, origin}; `cf_origin_status` |

All other promtail-eligible containers receive only the generic stages.

### `infra/tempo/tempo.yml` (new)

Port from corwins.media `tempo.yaml`:

```yaml
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc: { endpoint: 0.0.0.0:4317 }
        http: { endpoint: 0.0.0.0:4318 }

ingester:
  trace_idle_period: 30s
  max_block_bytes: 1048576
  max_block_duration: 5m

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/traces
    wal:
      path: /var/tempo/wal
```

### `infra/otel/otel-collector.yml`

Replace traces exporter:

```diff
 exporters:
-  otlp_grpc/jaeger:
-    endpoint: jaeger:4317
-    tls:
-      insecure: true
+  otlp_http/tempo:
+    endpoint: http://tempo:4318
+    tls:
+      insecure: true
```

```diff
 processors:
   batch:
     timeout: 5s
     send_batch_size: 1024
+  memory_limiter:
+    check_interval: 5s
+    limit_mib: 256
+    spike_limit_mib: 64
   resource:
     attributes:
       - key: deployment.environment
         value: homelab
         action: upsert
```

```diff
 service:
   pipelines:
     traces:
       receivers: [otlp]
-      processors: [batch, resource]
-      exporters: [otlp_grpc/jaeger]
+      processors: [memory_limiter, batch, resource]
+      exporters: [otlp_http/tempo]
     metrics:
       receivers: [otlp]
-      processors: [batch]
+      processors: [memory_limiter, batch]
       exporters: [prometheus]
     logs:
       receivers: [otlp]
-      processors: [batch, resource]
+      processors: [memory_limiter, batch, resource]
       exporters: [otlp_http/loki]
```

### `infra/prometheus/targets/services.yml`

```diff
-# Jaeger
-- targets: ['jaeger:14269']
-  labels:
-    job: jaeger
+# Tempo
+- targets: ['tempo:3200']
+  labels:
+    job: tempo
+    __metrics_path__: /metrics
+
+# NVIDIA GPU
+- targets: ['nvidia-exporter:9400']
+  labels:
+    job: nvidia-gpu
```

### `infra/grafana/provisioning/datasources/datasources.yml`

```yaml
apiVersion: 1
datasources:
  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    uid: loki
    isDefault: true
    jsonData:
      maxLines: 1000

  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    uid: prometheus
    jsonData:
      httpMethod: POST

  - name: Tempo
    type: tempo
    access: proxy
    url: http://tempo:3200
    uid: tempo
    jsonData:
      tracesToLogsV2:
        datasourceUid: loki
        filterByTraceID: true
      serviceMap:
        datasourceUid: prometheus
```

### Grafana OAuth via Gitea

Extend `dev/scripts/bootstrap_sso.py` to register a second OAuth app and write `.state/gitea/grafana.env`:

```python
GRAFANA_ENV = STATE / "grafana.env"

def _write_grafana_env(client_id: str, client_secret: str) -> bool:
    return _write_env_file(
        GRAFANA_ENV,
        (
            f"GF_AUTH_GENERIC_OAUTH_CLIENT_ID={client_id}\n"
            f"GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET={client_secret}\n"
        ),
    )

# in main(), after the oauth2-proxy app is provisioned:
grafana_creds = _ensure_oauth_app(
    client,
    token,
    name="grafana",
    redirect_uri=f"https://grafana.{DOMAIN}/login/generic_oauth",
)
if grafana_creds:
    _write_grafana_env(*grafana_creds)
```

Compose changes for `grafana`:

```yaml
grafana:
  # ... existing fields ...
  env_file:
    - path: .state/gitea/grafana.env
      required: false
  environment:
    # existing GF_SECURITY_ADMIN_*, GF_USERS_*, GF_AUTH_ANONYMOUS_*, GF_SERVER_ROOT_URL
    # ... plus:
    - GF_AUTH_GENERIC_OAUTH_ENABLED=true
    - GF_AUTH_GENERIC_OAUTH_NAME=Gitea
    - GF_AUTH_GENERIC_OAUTH_SCOPES=openid profile email
    - GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://git.${DOMAIN:-fhirworx.io}/login/oauth/authorize
    - GF_AUTH_GENERIC_OAUTH_TOKEN_URL=http://git:3000/login/oauth/access_token
    - GF_AUTH_GENERIC_OAUTH_API_URL=http://git:3000/api/v1/user
    - GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=true
    - GF_AUTH_GENERIC_OAUTH_AUTO_LOGIN=false
    - GF_AUTH_OAUTH_ALLOW_INSECURE_EMAIL_LOOKUP=true
    - GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH=is_admin && 'GrafanaAdmin'
    - GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_GRAFANA_ADMIN=true
```

Grafana already has both `gateway` and `observability` networks, which is exactly what's needed: TOKEN_URL/API_URL traverse `gateway` to reach `git:3000`.

## Per-service Grafana dashboards

Six thematic dashboards under `infra/grafana/dashboards/` (auto-loaded by the existing provisioning config):

### 1. `homelab-overview.json`
- **Stat row:** total containers up, total log volume (Loki) /1m, total error rate (Loki `level="error"`) /1m
- **Logs:** top error sources by container (Loki bar chart)

### 2. `gateway-auth.json`
- **Time series:** Traefik request rate by status code, p50/p95/p99 latency from `traefik_service_request_duration_seconds`
- **Time series:** oauth2-proxy AuthSuccess vs AuthFailure rate (from Loki `oauth_event`)
- **Time series:** Gitea login + push event rate (from Loki `gitea_event`)
- **Logs:** failed-auth tail (Loki: `{container=~"oauth2-proxy|git"} |~ "(?i)(failure|forbidden|401|403)"`)

### 3. `data-lake.json`
- **Stat:** Trino queries running / completed last hour / failed last hour
- **Time series:** Trino query duration histogram (from `trino_event="QUERY_COMPLETED"`)
- **Time series:** Nessie + Polaris HTTP request rate + error rate (from Quarkus exporter `/q/metrics`)
- **Time series:** RustFS S3 op count by `s3_op` (from Loki)

### 4. `data-pipelines.json`
- **Time series:** api FastAPI request rate + status (Prometheus `stack_*` from otel-collector)
- **Time series:** mail-poller poll cycles + ingest counts (from Loki `mail_event`, `mail_count`)
- **Time series:** pipeline-run rate by `pipeline_name`
- **Logs:** pipeline error tail

### 5. `ci.json`
- **Stat:** act-runner active jobs, jobs in last 24h, fail rate
- **Time series:** act-runner `ci_event` by status
- **Time series:** Gitea push + webhook events (Loki `gitea_event`)

### 6. `gpu-notebooks.json`
- **Time series:** GPU utilization, memory used, temperature, power (DCGM `DCGM_FI_DEV_*`)

The existing `pipeline-performance.json` is folded into `data-pipelines.json` if its panels overlap; otherwise kept.

## Migration plan

| Step | Description | Verification |
|---|---|---|
| 1 | Add `tempo`, `nvidia-exporter` services. Don't remove jaeger yet. | `docker compose config` parses |
| 2 | Write new `loki-config.yml` (retention/compactor) and `tempo.yml`. | `docker compose up -d loki tempo` healthy |
| 3 | Rewrite `promtail-config.yml` with docker_sd + per-service stages. | promtail container starts; `curl loki:3100/loki/api/v1/labels` shows expected labels |
| 4 | Update `otel-collector.yml`: traces → tempo (keep jaeger receiver until step 6). | `curl tempo:3200/ready` returns ready; trace appears in Grafana Explore |
| 5 | Add `promtail=true` labels to all services in `compose.yml` + enable Traefik access logs. | `curl 'loki:3100/loki/api/v1/label/container/values'` returns full container set |
| 6 | Update `datasources.yml` (Loki default, Tempo replaces Jaeger, correlation). Update `prometheus/targets/services.yml`. Update `traefik/dynamic/` route. Update `stack.toml` and `bootstrap_sso.py` SUBDOMAINS. | Grafana Explore: query each datasource; click trace span → loki query opens |
| 7 | Extend `bootstrap_sso.py` with grafana OAuth app. Add OAuth env vars + `env_file` to `grafana` service. Run `docker compose run --rm wire`. | Grafana login page shows "Sign in with Gitea"; admin user logs in via Gitea |
| 8 | Provision six dashboards under `infra/grafana/dashboards/`. | All dashboards visible and rendering |
| 9 | Remove `jaeger` service, jaeger volume references, jaeger route. | `docker compose up -d` succeeds with no jaeger references |

## Out of scope

- Alert rules (Loki ruler / Prometheus alertmanager) — separate effort
- Long-term metrics storage (Mimir / remote write) — 7d Prometheus retention is sufficient for now
- Log archival to S3 (rustfs) — Loki retention deletes old data; archival can be a follow-up
- Tempo metrics-from-traces (span metrics) — requires Mimir or remote write target

## Open questions

None — all design decisions are settled by the user's answers in the brainstorming dialogue.