stack/docs/superpowers/specs/2026-05-01-observability-port-design.md

Observability port from corwins.media → stack

Date: 2026-05-01 Status: Draft (awaiting user approval before implementation planning)

Goal

Port the corwins.media observability approach to the stack repo: replace Jaeger with Tempo, add nvidia-exporter, switch promtail from filesystem log scraping to docker_sd_configs with promtail=true opt-in, add per-service log pipeline stages tailored to the stack's services, enable Loki retention, wire Grafana OAuth via Gitea, and provision six thematic dashboards.

Reference: corwins.media baseline

Component	corwins.media file
Loki	infra/observability/loki.yaml
Promtail	infra/observability/promtail.yaml
Prometheus	infra/observability/prometheus.yaml
OTel collector	infra/observability/otel-collector.yaml
Tempo	infra/observability/tempo.yaml
Grafana datasources	infra/observability/grafana/provisioning/datasources/datasources.yaml
Grafana dashboards	infra/observability/grafana/dashboards/*.json

Architecture

Compose changes

Add services (all on observability network):

• tempo — OTLP trace backend, replaces Jaeger
• nvidia-exporter — DCGM GPU metrics, runtime: nvidia, GPU device reservation

Remove services:

• jaeger

Modify services:

• traefik: enable JSON access logs (--accesslog=true --accesslog.format=json --accesslog.fields.headers.defaultMode=keep)
• promtail: drop the /var/lib/docker/containers bind mount, mount /var/run/docker.sock (read-only), add group_add: ["${DOCKER_GID:-999}"]
• All eligible services (see table below): add labels: ["promtail=true"]
• grafana: add OAuth env vars + env_file: .state/gitea/grafana.env (required: false)

Volumes: add tempo_data. Existing loki_data, prometheus_data, grafana_data unchanged.

Networks: unchanged. observability stays internal: true. tempo, nvidia-exporter, promtail need only docker.sock + host filesystems, not internet egress.

Subdomain / route changes

• stack.toml subdomains: replace "jaeger" with "tempo"
• infra/traefik/dynamic/: replace jaeger router with tempo router (port 3200)
• bootstrap_sso.py SUBDOMAINS: replace "jaeger" with "tempo"

promtail=true label coverage

Container	Label	Notes
traefik	yes	gateway access logs
coredns	yes	DNS resolution events
rustfs	yes	S3 op logs
postgres	yes	DB events
git (gitea)	yes	login/push/oauth/webhook
act-runner	yes	CI job logs
notebooks	yes	marimo events
zotero	yes	rare manual debug
webdav	yes	rclone access
nessie	yes	Iceberg catalog
trino	yes	query lifecycle
polaris	yes	Iceberg catalog
dashboard	yes	nginx access
docs	yes	docusaurus access
api	yes	FastAPI events
mail-poller	yes	poll cycle / ingest
auth-handler	yes	oauth2 flow
oauth2-proxy	yes	AuthSuccess/Failure
cloudflared	yes	tunnel events
jaeger → tempo	yes	self-monitoring
loki	yes	self-monitoring
prometheus	yes	self-monitoring
grafana	yes	self-monitoring
otel-collector	yes	self-monitoring
mc	no	sleep-infinity sidecar, no useful logs
wire	no	one-shot bootstrap, profile=tools

Configs

infra/loki/loki-config.yml

Add retention + compactor (port from corwins.media loki.yaml):

limits_config:
  retention_period: 168h            # 7d
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  ingestion_rate_mb: 16
  ingestion_burst_size_mb: 32
  max_query_series: 5000
  allow_structured_metadata: true   # keep stack-specific

compactor:
  working_directory: /loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  delete_request_store: filesystem

Drop query_range.results_cache (corwins doesn't use it; saves a bit of memory). Keep analytics.reporting_enabled: false.

infra/loki/promtail-config.yml (full rewrite)

Top-level:

server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 15s
    relabel_configs:
      - source_labels: ['__meta_docker_container_label_promtail']
        regex: 'true'
        action: keep
      - source_labels: ['__meta_docker_container_name']
        regex: '/?(.*)'
        target_label: container
      - source_labels: ['__meta_docker_container_name']
        regex: '/?(.*)'
        target_label: job
      - source_labels: ['__meta_docker_container_label_com_docker_compose_service']
        target_label: service
      - source_labels: ['__meta_docker_container_label_com_docker_compose_project']
        target_label: project
    pipeline_stages:
      - docker: {}
      # Generic: log level
      - regex:
          expression: '(?i)(?P<level>error|warn|info|debug|fatal|critical|trace)'
      - labels:
          level:
      # Generic: HTTP access line (works for any container that emits common-style logs)
      - regex:
          expression: '(?P<remote_ip>\d+\.\d+\.\d+\.\d+) .* "(?P<http_method>GET|POST|PUT|DELETE|PATCH|HEAD|OPTIONS) (?P<http_path>[^ ]*) [^"]*" (?P<http_status>\d{3}) (?P<http_bytes>\d+)'
      - labels:
          http_method:
          http_status:

      # --- Per-service stages (selectors below) ---

Per-service stages (each is a match: { selector: '{container="X"}', stages: [...] } block):

Container	Extracted labels
traefik	json stage parsing access log → tf_router, tf_service, http_status, http_method; regex stage for non-access lines: tf_event (TLS handshake / certResolver / EntryPoint / router)
git	gitea_event ∈ {login, logout, register, push, pull, branch, tag, issue, webhook, oauth, created, deleted, merged}
oauth2-proxy	oauth_event ∈ {AuthSuccess, AuthFailure, OAuthStart, OAuthCallback, OAuthError, Invalid, Expired, Authenticated, Forbidden}
postgres	pg_event ∈ {FATAL, ERROR, WARNING, checkpoint, autovacuum, connection authorized, connection received, statement}
trino	trino_event ∈ {QUERY_CREATED, QUERY_STARTED, QUERY_COMPLETED, QUERY_FAILED, SPLIT_COMPLETED}; query_id from \b([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})\b
nessie, polaris	Quarkus standard access — quarkus_route (path prefix), http_status
rustfs	s3_op ∈ {PutObject, GetObject, DeleteObject, ListObjectsV2, HeadObject, CompleteMultipart}; s3_bucket
api	api_event ∈ {pipeline, ingest, health, error}; pipeline_name if present
mail-poller	mail_event ∈ {ingest, poll, error}; mail_count (numeric)
act-runner	ci_event ∈ {job started, job finished, job failed, runner registered}; job_id
cloudflared	cf_event ∈ {connection, registered, unregistered, reconnect, quic, error, tunnel, origin}; cf_origin_status

All other promtail-eligible containers receive only the generic stages.

infra/tempo/tempo.yml (new)

Port from corwins.media tempo.yaml:

server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc: { endpoint: 0.0.0.0:4317 }
        http: { endpoint: 0.0.0.0:4318 }

ingester:
  trace_idle_period: 30s
  max_block_bytes: 1048576
  max_block_duration: 5m

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/traces
    wal:
      path: /var/tempo/wal

infra/otel/otel-collector.yml

Replace traces exporter:

 exporters:
-  otlp_grpc/jaeger:
-    endpoint: jaeger:4317
-    tls:
-      insecure: true
+  otlp_http/tempo:
+    endpoint: http://tempo:4318
+    tls:
+      insecure: true

 processors:
   batch:
     timeout: 5s
     send_batch_size: 1024
+  memory_limiter:
+    check_interval: 5s
+    limit_mib: 256
+    spike_limit_mib: 64
   resource:
     attributes:
       - key: deployment.environment
         value: homelab
         action: upsert

 service:
   pipelines:
     traces:
       receivers: [otlp]
-      processors: [batch, resource]
-      exporters: [otlp_grpc/jaeger]
+      processors: [memory_limiter, batch, resource]
+      exporters: [otlp_http/tempo]
     metrics:
       receivers: [otlp]
-      processors: [batch]
+      processors: [memory_limiter, batch]
       exporters: [prometheus]
     logs:
       receivers: [otlp]
-      processors: [batch, resource]
+      processors: [memory_limiter, batch, resource]
       exporters: [otlp_http/loki]

infra/prometheus/targets/services.yml

-# Jaeger
-- targets: ['jaeger:14269']
-  labels:
-    job: jaeger
+# Tempo
+- targets: ['tempo:3200']
+  labels:
+    job: tempo
+    __metrics_path__: /metrics
+
+# NVIDIA GPU
+- targets: ['nvidia-exporter:9400']
+  labels:
+    job: nvidia-gpu

infra/grafana/provisioning/datasources/datasources.yml

apiVersion: 1
datasources:
  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    uid: loki
    isDefault: true
    jsonData:
      maxLines: 1000

  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    uid: prometheus
    jsonData:
      httpMethod: POST

  - name: Tempo
    type: tempo
    access: proxy
    url: http://tempo:3200
    uid: tempo
    jsonData:
      tracesToLogsV2:
        datasourceUid: loki
        filterByTraceID: true
      serviceMap:
        datasourceUid: prometheus

Grafana OAuth via Gitea

Extend dev/scripts/bootstrap_sso.py to register a second OAuth app and write .state/gitea/grafana.env:

GRAFANA_ENV = STATE / "grafana.env"

def _write_grafana_env(client_id: str, client_secret: str) -> bool:
    return _write_env_file(
        GRAFANA_ENV,
        (
            f"GF_AUTH_GENERIC_OAUTH_CLIENT_ID={client_id}\n"
            f"GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET={client_secret}\n"
        ),
    )

# in main(), after the oauth2-proxy app is provisioned:
grafana_creds = _ensure_oauth_app(
    client,
    token,
    name="grafana",
    redirect_uri=f"https://grafana.{DOMAIN}/login/generic_oauth",
)
if grafana_creds:
    _write_grafana_env(*grafana_creds)

Compose changes for grafana:

grafana:
  # ... existing fields ...
  env_file:
    - path: .state/gitea/grafana.env
      required: false
  environment:
    # existing GF_SECURITY_ADMIN_*, GF_USERS_*, GF_AUTH_ANONYMOUS_*, GF_SERVER_ROOT_URL
    # ... plus:
    - GF_AUTH_GENERIC_OAUTH_ENABLED=true
    - GF_AUTH_GENERIC_OAUTH_NAME=Gitea
    - GF_AUTH_GENERIC_OAUTH_SCOPES=openid profile email
    - GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://git.${DOMAIN:-fhirworx.io}/login/oauth/authorize
    - GF_AUTH_GENERIC_OAUTH_TOKEN_URL=http://git:3000/login/oauth/access_token
    - GF_AUTH_GENERIC_OAUTH_API_URL=http://git:3000/api/v1/user
    - GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=true
    - GF_AUTH_GENERIC_OAUTH_AUTO_LOGIN=false
    - GF_AUTH_OAUTH_ALLOW_INSECURE_EMAIL_LOOKUP=true
    - GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH=is_admin && 'GrafanaAdmin'
    - GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_GRAFANA_ADMIN=true

Grafana already has both gateway and observability networks, which is exactly what's needed: TOKEN_URL/API_URL traverse gateway to reach git:3000.

Per-service Grafana dashboards

Six thematic dashboards under infra/grafana/dashboards/ (auto-loaded by the existing provisioning config):

1. homelab-overview.json

• Stat row: total containers up, total log volume (Loki) /1m, total error rate (Loki level="error") /1m
• Logs: top error sources by container (Loki bar chart)

2. gateway-auth.json

• Time series: Traefik request rate by status code, p50/p95/p99 latency from traefik_service_request_duration_seconds
• Time series: oauth2-proxy AuthSuccess vs AuthFailure rate (from Loki oauth_event)
• Time series: Gitea login + push event rate (from Loki gitea_event)
• Logs: failed-auth tail (Loki: {container=~"oauth2-proxy|git"} |~ "(?i)(failure|forbidden|401|403)")

3. data-lake.json

• Stat: Trino queries running / completed last hour / failed last hour
• Time series: Trino query duration histogram (from trino_event="QUERY_COMPLETED")
• Time series: Nessie + Polaris HTTP request rate + error rate (from Quarkus exporter /q/metrics)
• Time series: RustFS S3 op count by s3_op (from Loki)

4. data-pipelines.json

• Time series: api FastAPI request rate + status (Prometheus stack_* from otel-collector)
• Time series: mail-poller poll cycles + ingest counts (from Loki mail_event, mail_count)
• Time series: pipeline-run rate by pipeline_name
• Logs: pipeline error tail

5. ci.json

• Stat: act-runner active jobs, jobs in last 24h, fail rate
• Time series: act-runner ci_event by status
• Time series: Gitea push + webhook events (Loki gitea_event)

6. gpu-notebooks.json

• Time series: GPU utilization, memory used, temperature, power (DCGM DCGM_FI_DEV_*)

The existing pipeline-performance.json is folded into data-pipelines.json if its panels overlap; otherwise kept.

Migration plan

Step	Description	Verification
1	Add tempo, nvidia-exporter services. Don't remove jaeger yet.	docker compose config parses
2	Write new loki-config.yml (retention/compactor) and tempo.yml.	docker compose up -d loki tempo healthy
3	Rewrite promtail-config.yml with docker_sd + per-service stages.	promtail container starts; curl loki:3100/loki/api/v1/labels shows expected labels
4	Update otel-collector.yml: traces → tempo (keep jaeger receiver until step 6).	curl tempo:3200/ready returns ready; trace appears in Grafana Explore
5	Add promtail=true labels to all services in compose.yml + enable Traefik access logs.	curl 'loki:3100/loki/api/v1/label/container/values' returns full container set
6	Update datasources.yml (Loki default, Tempo replaces Jaeger, correlation). Update prometheus/targets/services.yml. Update traefik/dynamic/ route. Update stack.toml and bootstrap_sso.py SUBDOMAINS.	Grafana Explore: query each datasource; click trace span → loki query opens
7	Extend bootstrap_sso.py with grafana OAuth app. Add OAuth env vars + env_file to grafana service. Run docker compose run --rm wire.	Grafana login page shows "Sign in with Gitea"; admin user logs in via Gitea
8	Provision six dashboards under infra/grafana/dashboards/.	All dashboards visible and rendering
9	Remove jaeger service, jaeger volume references, jaeger route.	docker compose up -d succeeds with no jaeger references

Out of scope

• Alert rules (Loki ruler / Prometheus alertmanager) — separate effort
• Long-term metrics storage (Mimir / remote write) — 7d Prometheus retention is sufficient for now
• Log archival to S3 (rustfs) — Loki retention deletes old data; archival can be a follow-up
• Tempo metrics-from-traces (span metrics) — requires Mimir or remote write target

Open questions

None — all design decisions are settled by the user's answers in the brainstorming dialogue.