stack/docs/superpowers/plans/2026-05-01-observability-port.md

Observability Port (corwins.media → stack) Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Replace Jaeger with Tempo, add nvidia-exporter, switch promtail from filesystem scraping to docker_sd opt-in, port per-service log pipelines to the stack's services, enable Loki retention, wire Grafana SSO via Gitea, and provision six per-service dashboards.

Architecture: Compose-native — every change lands in compose.yml, an infra/<component>/<config>.yml, or a JSON dashboard. The full stack runs on Podman/Docker with the existing networks (gateway, observability, storage, data, ci). New services live on observability; tempo + nvidia-exporter only need docker.sock and host filesystem reads, so the network's internal: true constraint is preserved. Grafana already has gateway + observability, which is exactly what's needed for OAuth token exchange to git:3000 plus internal datasource access.

Tech Stack: Docker Compose, Loki, Promtail, Grafana, Prometheus, Tempo, OpenTelemetry Collector, DCGM-exporter (NVIDIA), Traefik, Gitea (OAuth provider).

Spec: docs/superpowers/specs/2026-05-01-observability-port-design.md

Working pattern for verification: stack uses pytest tests/test_compose_mounts.py to validate compose, plus live docker compose up + curl probes for runtime checks. We'll add tests/test_observability.py for the compose-level invariants and use bash deploy.sh helpers (wait_healthy) where appropriate.

---

File Structure

Created:

• infra/tempo/tempo.yml — tempo runtime config
• infra/grafana/dashboards/homelab-overview.json — fleet health dashboard
• infra/grafana/dashboards/gateway-auth.json — Traefik + oauth2-proxy + Gitea
• infra/grafana/dashboards/data-lake.json — Trino + Nessie + Polaris + RustFS
• infra/grafana/dashboards/data-pipelines.json — api + mail-poller + pipelines
• infra/grafana/dashboards/ci.json — act-runner + Gitea push
• infra/grafana/dashboards/gpu-notebooks.json — DCGM + notebooks/zotero
• tests/test_observability.py — compose invariant tests for the new layout

Modified:

• compose.yml — add tempo/nvidia-exporter, drop jaeger, label all services with promtail=true, Traefik access logs, Grafana OAuth env
• infra/loki/loki-config.yml — retention + compactor
• infra/loki/promtail-config.yml — full rewrite (docker_sd + per-service)
• infra/otel/otel-collector.yml — traces → tempo, add memory_limiter
• infra/prometheus/targets/services.yml — add nvidia/tempo, drop jaeger
• infra/grafana/provisioning/datasources/datasources.yml — Tempo replaces Jaeger; Loki default; correlation
• infra/traefik/dynamic/services.yml — swap jaeger → tempo route
• stack.toml — subdomains swap jaeger → tempo
• dev/scripts/bootstrap_sso.py — SUBDOMAINS swap jaeger → tempo; add Grafana OAuth app + grafana.env writer

---

Pre-flight

• Step 0.1: Confirm prerequisites

# verify GPU is present (notebooks already uses CUDA)
nvidia-smi >/dev/null && echo "GPU OK"

# verify docker GID for promtail group_add
getent group docker | cut -d: -f3
# Note this value — you'll set DOCKER_GID in .env if it isn't already 999

• Step 0.2: Create the test scaffolding file

Create tests/test_observability.py:

"""Compose-level invariants for the observability port.

refs docs/superpowers/specs/2026-05-01-observability-port-design.md
"""

from __future__ import annotations

from pathlib import Path

import yaml

ROOT = Path(__file__).resolve().parents[1]
COMPOSE = ROOT / "compose.yml"


def _load() -> dict:
    return yaml.safe_load(COMPOSE.read_text())


def _services() -> dict:
    return _load().get("services", {})

• Step 0.3: Commit scaffolding

git add tests/test_observability.py
git commit -m "test(observability): add compose-invariant test scaffold"

---

Task 1: Tempo service + config

Files:

• Create: infra/tempo/tempo.yml

• Modify: compose.yml (add tempo service, add tempo_data volume)

• Test: tests/test_observability.py

• Step 1.1: Write the failing test

Append to tests/test_observability.py:

class TestTempo:
    def test_tempo_service_present(self):
        svcs = _services()
        assert "tempo" in svcs
        assert svcs["tempo"]["networks"] == ["observability"]

    def test_tempo_volume_declared(self):
        vols = _load().get("volumes", {})
        assert "tempo_data" in vols

    def test_tempo_config_mounted(self):
        svc = _services()["tempo"]
        mounts = [v for v in svc["volumes"] if "/etc/tempo" in v]
        assert any("./infra/tempo/tempo.yml" in v for v in mounts)

• Step 1.2: Run test to verify it fails

uv run pytest tests/test_observability.py::TestTempo -v

Expected: FAIL with KeyError: 'tempo' or AssertionError.

• Step 1.3: Create the tempo config file

Write infra/tempo/tempo.yml:

server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

ingester:
  trace_idle_period: 30s
  max_block_bytes: 1048576
  max_block_duration: 5m

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/traces
    wal:
      path: /var/tempo/wal

• Step 1.4: Add tempo service to compose.yml

Insert this block in compose.yml after the loki service (before promtail):

  tempo:
    image: grafana/tempo:2.7.2
    container_name: tempo
    networks:
      - observability
    volumes:
      - ./infra/tempo/tempo.yml:/etc/tempo/config.yaml:ro
      - tempo_data:/var/tempo
    command: -config.file=/etc/tempo/config.yaml
    healthcheck:
      test: ["CMD", "wget", "-q", "--spider", "http://localhost:3200/ready"]
      interval: 30s
      timeout: 5s
      retries: 5
      start_period: 30s
    security_opt:
      - no-new-privileges:true
    restart: unless-stopped

Add tempo_data: to the bottom volumes: block (alphabetised next to prometheus_data:):

volumes:
  postgres_data:
  rustfs_data:
  rustfs_logs:
  gitea_data:
  gitea_config:
  act_runner_data:
  loki_data:
  prometheus_data:
  tempo_data:
  grafana_data:

• Step 1.5: Run tests to verify they pass

uv run pytest tests/test_observability.py::TestTempo -v

Expected: PASS.

• Step 1.6: Verify compose parses and tempo comes up

docker compose config -q
docker compose up -d tempo
docker inspect --format='{{.State.Health.Status}}' tempo
# expected: "healthy" within 30-60s
curl -sf http://localhost:3200/ready  # may need to publish port temporarily; otherwise:
docker exec tempo wget -q -O- http://localhost:3200/ready
# expected: "ready"

• Step 1.7: Commit

git add infra/tempo/tempo.yml compose.yml tests/test_observability.py
git commit -m "feat(observability): add Tempo trace backend"

---

Task 3: nvidia-exporter (DCGM) service

Files:

• Modify: compose.yml

• Test: tests/test_observability.py

• Step 3.1: Write the failing test

Append to tests/test_observability.py:

class TestNvidiaExporter:
    def test_nvidia_exporter_present(self):
        svcs = _services()
        assert "nvidia-exporter" in svcs

    def test_nvidia_runtime(self):
        svc = _services()["nvidia-exporter"]
        assert svc.get("runtime") == "nvidia"

    def test_nvidia_gpu_reservation(self):
        svc = _services()["nvidia-exporter"]
        devices = svc["deploy"]["resources"]["reservations"]["devices"]
        assert any(d.get("driver") == "nvidia" for d in devices)

• Step 3.2: Run test to verify it fails

uv run pytest tests/test_observability.py::TestNvidiaExporter -v

Expected: FAIL.

• Step 3.3: Add service

Insert after the tempo service block:

  nvidia-exporter:
    image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.9-3.6.1-ubuntu22.04
    container_name: nvidia-exporter
    networks:
      - observability
    runtime: nvidia
    environment:
      - DCGM_EXPORTER_NO_HOSTNAME=1
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    cap_add:
      - SYS_ADMIN
    restart: unless-stopped

• Step 3.4: Run tests, bring service up, verify

uv run pytest tests/test_observability.py::TestNvidiaExporter -v
docker compose up -d nvidia-exporter
docker exec nvidia-exporter curl -sf http://localhost:9400/metrics | grep -c '^DCGM_FI_DEV_GPU_UTIL'
# expected: > 0

• Step 3.5: Commit

git add compose.yml tests/test_observability.py
git commit -m "feat(observability): add DCGM nvidia-exporter for GPU metrics"

---

Task 4: Loki retention + compactor

Files:

• Modify: infra/loki/loki-config.yml

• Test: tests/test_observability.py

• Step 4.1: Write the failing test

Append to tests/test_observability.py:

class TestLokiRetention:
    def test_loki_compactor_configured(self):
        cfg = yaml.safe_load((ROOT / "infra/loki/loki-config.yml").read_text())
        assert cfg["compactor"]["retention_enabled"] is True
        assert cfg["limits_config"]["retention_period"] == "168h"

• Step 4.2: Run test to verify it fails

uv run pytest tests/test_observability.py::TestLokiRetention -v

Expected: FAIL with KeyError: 'compactor'.

• Step 4.3: Update loki config

Replace the contents of infra/loki/loki-config.yml with:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

limits_config:
  retention_period: 168h
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  ingestion_rate_mb: 16
  ingestion_burst_size_mb: 32
  max_query_series: 5000
  allow_structured_metadata: true

compactor:
  working_directory: /loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  delete_request_store: filesystem

analytics:
  reporting_enabled: false

• Step 4.4: Run tests, restart loki, verify

uv run pytest tests/test_observability.py::TestLokiRetention -v
docker compose up -d --force-recreate loki
docker logs loki 2>&1 | tail -30
# expected: no "config error", "ready" within ~15s
docker exec loki wget -q -O- http://localhost:3100/ready
# expected: "ready"

• Step 4.5: Commit

git add infra/loki/loki-config.yml tests/test_observability.py
git commit -m "feat(loki): enable 7d retention via compactor"

---

Task 5: Promtail rewrite — docker_sd + generic stages

Files:

• Modify: compose.yml (promtail service: drop containers bind, add docker.sock + group_add)

• Modify: infra/loki/promtail-config.yml (full rewrite, generic stages only)

• Test: tests/test_observability.py

• Step 5.1: Write the failing test

Append to tests/test_observability.py:

class TestPromtailRewire:
    def test_promtail_uses_docker_sock(self):
        svc = _services()["promtail"]
        sources = [v.split(":")[0] for v in svc["volumes"] if isinstance(v, str)]
        # New: docker.sock; old filesystem mount must be gone.
        assert any("docker.sock" in s for s in sources)
        assert not any("/var/lib/docker/containers" in v for v in svc["volumes"])

    def test_promtail_has_docker_gid(self):
        svc = _services()["promtail"]
        assert "group_add" in svc

    def test_promtail_config_is_docker_sd(self):
        cfg = yaml.safe_load(
            (ROOT / "infra/loki/promtail-config.yml").read_text()
        )
        jobs = cfg["scrape_configs"]
        docker_jobs = [j for j in jobs if j.get("job_name") == "docker"]
        assert docker_jobs, "expected job_name=docker"
        assert "docker_sd_configs" in docker_jobs[0]

• Step 5.2: Run test to verify it fails

uv run pytest tests/test_observability.py::TestPromtailRewire -v

Expected: FAIL.

• Step 5.3: Replace promtail config (generic stages only — per-service comes in Task 6)

Write infra/loki/promtail-config.yml:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 15s
    relabel_configs:
      - source_labels: ['__meta_docker_container_label_promtail']
        regex: 'true'
        action: keep
      - source_labels: ['__meta_docker_container_name']
        regex: '/?(.*)'
        target_label: container
      - source_labels: ['__meta_docker_container_name']
        regex: '/?(.*)'
        target_label: job
      - source_labels: ['__meta_docker_container_label_com_docker_compose_service']
        target_label: service
      - source_labels: ['__meta_docker_container_label_com_docker_compose_project']
        target_label: project
    pipeline_stages:
      - docker: {}
      - regex:
          expression: '(?i)(?P<level>error|warn|info|debug|fatal|critical|trace)'
      - labels:
          level:
      - regex:
          expression: '(?P<remote_ip>\d+\.\d+\.\d+\.\d+) .* "(?P<http_method>GET|POST|PUT|DELETE|PATCH|HEAD|OPTIONS) (?P<http_path>[^ ]*) [^"]*" (?P<http_status>\d{3}) (?P<http_bytes>\d+)'
      - labels:
          http_method:
          http_status:

• Step 5.4: Update the promtail service in compose.yml

Replace the existing promtail: service block with:

  promtail:
    image: grafana/promtail:latest
    container_name: promtail
    networks:
      - observability
    volumes:
      - ./infra/loki/promtail-config.yml:/etc/promtail/config.yml:ro
      - ${DOCKER_SOCK:-/run/user/1000/docker.sock}:/var/run/docker.sock:ro
    command: -config.file=/etc/promtail/config.yml
    depends_on:
      - loki
    group_add:
      - "${DOCKER_GID:-985}"
    security_opt:
      - no-new-privileges:true
    restart: unless-stopped

(If getent group docker returned a different number than 985 in step 0.1, set DOCKER_GID in .env.)

• Step 5.5: Run tests, recreate promtail, verify scraping works

uv run pytest tests/test_observability.py::TestPromtailRewire -v

# Add a temporary promtail=true label to one container so scraping has *something* to find:
# We'll batch-label everything in Task 10. For this verification, label loki itself.
docker compose stop promtail
docker compose rm -f promtail
docker compose up -d promtail
docker logs promtail 2>&1 | tail -20
# expected: "Adding target", no auth/socket errors

# After Task 10 labels are applied, the next probe is meaningful. For now:
docker exec promtail wget -q -O- 'http://loki:3100/loki/api/v1/labels' | head
# expected: JSON with at least {"data":[...]} — empty list is acceptable here pre-labels

• Step 5.6: Commit

git add infra/loki/promtail-config.yml compose.yml tests/test_observability.py
git commit -m "feat(promtail): switch to docker_sd opt-in scraping"

---

Task 6: Per-service log pipeline stages

Files:

• Modify: infra/loki/promtail-config.yml

• Test: tests/test_observability.py

• Step 6.1: Write the failing test

Append to tests/test_observability.py:

class TestPerServiceStages:
    """Each named container must have a matching pipeline stage that emits at
    least one service-specific label."""

    EXPECTED = {
        "traefik": "tf_event",
        "git": "gitea_event",
        "oauth2-proxy": "oauth_event",
        "postgres": "pg_event",
        "trino": "trino_event",
        "rustfs": "s3_op",
        "api": "api_event",
        "mail-poller": "mail_event",
        "act-runner": "ci_event",
        "cloudflared": "cf_event",
    }

    def test_each_service_has_a_match_stage(self):
        cfg = yaml.safe_load(
            (ROOT / "infra/loki/promtail-config.yml").read_text()
        )
        stages = cfg["scrape_configs"][0]["pipeline_stages"]
        text = yaml.safe_dump(stages)
        for container, label in self.EXPECTED.items():
            assert f'container="{container}"' in text, f"no match stage for {container}"
            assert label in text, f"no {label} extraction for {container}"

• Step 6.2: Run test to verify it fails

uv run pytest tests/test_observability.py::TestPerServiceStages -v

Expected: FAIL.

• Step 6.3: Append per-service stages to promtail config

Append the following to infra/loki/promtail-config.yml (under the existing pipeline_stages: block of the docker job — keep the generic stages first):

      # --- Traefik ---
      - match:
          selector: '{container="traefik"}'
          stages:
            - json:
                expressions:
                  RouterName: RouterName
                  ServiceName: ServiceName
                  DownstreamStatus: DownstreamStatus
                  RequestMethod: RequestMethod
            - labels:
                tf_router: RouterName
                tf_service: ServiceName
                http_status: DownstreamStatus
                http_method: RequestMethod
            - regex:
                expression: '(?P<tf_event>TLS handshake|EntryPoint|router|certResolver|middleware|tls)'
            - labels:
                tf_event:

      # --- Gitea ---
      - match:
          selector: '{container="git"}'
          stages:
            - regex:
                expression: '(?P<gitea_event>login|logout|register|push|pull|branch|tag|issue|webhook|oauth|created|deleted|merged)'
            - labels:
                gitea_event:

      # --- oauth2-proxy ---
      - match:
          selector: '{container="oauth2-proxy"}'
          stages:
            - regex:
                expression: '(?P<oauth_event>AuthSuccess|AuthFailure|OAuthStart|OAuthCallback|OAuthError|Invalid|Expired|Authenticated|Forbidden)'
            - labels:
                oauth_event:

      # --- Postgres ---
      - match:
          selector: '{container="postgres"}'
          stages:
            - regex:
                expression: '(?P<pg_event>FATAL|ERROR|WARNING|checkpoint|autovacuum|connection authorized|connection received|statement)'
            - labels:
                pg_event:

      # --- Trino ---
      - match:
          selector: '{container="trino"}'
          stages:
            - regex:
                expression: '(?P<trino_event>QUERY_CREATED|QUERY_STARTED|QUERY_COMPLETED|QUERY_FAILED|SPLIT_COMPLETED)'
            - regex:
                expression: '\b(?P<query_id>[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})\b'
            - labels:
                trino_event:
                query_id:

      # --- Nessie / Polaris (Quarkus) ---
      - match:
          selector: '{container=~"nessie|polaris"}'
          stages:
            - regex:
                expression: '"(?P<http_method>GET|POST|PUT|DELETE) (?P<quarkus_route>/[^ ?]+)'
            - labels:
                quarkus_route:

      # --- RustFS ---
      - match:
          selector: '{container="rustfs"}'
          stages:
            - regex:
                expression: '(?P<s3_op>PutObject|GetObject|DeleteObject|ListObjectsV2|HeadObject|CompleteMultipart)'
            - regex:
                expression: 'bucket=(?P<s3_bucket>[A-Za-z0-9._-]+)'
            - labels:
                s3_op:
                s3_bucket:

      # --- API (FastAPI/stack) ---
      - match:
          selector: '{container="api"}'
          stages:
            - regex:
                expression: '(?P<api_event>pipeline|ingest|health|error)'
            - regex:
                expression: 'pipeline=(?P<pipeline_name>[A-Za-z0-9_.-]+)'
            - labels:
                api_event:
                pipeline_name:

      # --- mail-poller ---
      - match:
          selector: '{container="mail-poller"}'
          stages:
            - regex:
                expression: '(?P<mail_event>ingest|poll|error)'
            - regex:
                expression: 'count=(?P<mail_count>\d+)'
            - labels:
                mail_event:
                mail_count:

      # --- act-runner ---
      - match:
          selector: '{container="act-runner"}'
          stages:
            - regex:
                expression: '(?P<ci_event>job started|job finished|job failed|runner registered)'
            - regex:
                expression: 'job_id=(?P<job_id>\S+)'
            - labels:
                ci_event:
                job_id:

      # --- cloudflared ---
      - match:
          selector: '{container="cloudflared"}'
          stages:
            - regex:
                expression: '(?P<cf_event>connection|registered|unregistered|reconnect|quic|error|tunnel|origin)'
            - regex:
                expression: 'status=(?P<cf_origin_status>\d{3})'
            - labels:
                cf_event:
                cf_origin_status:

• Step 6.4: Run tests, restart promtail, verify

uv run pytest tests/test_observability.py::TestPerServiceStages -v
docker compose restart promtail
docker logs promtail 2>&1 | tail -20
# expected: no "yaml: line N" parse errors

• Step 6.5: Commit

git add infra/loki/promtail-config.yml tests/test_observability.py
git commit -m "feat(promtail): per-service log pipeline stages"

---

Task 7: OTel collector — traces to Tempo

Files:

• Modify: infra/otel/otel-collector.yml

• Test: tests/test_observability.py

• Step 7.1: Write the failing test

Append to tests/test_observability.py:

class TestOtelTempo:
    def test_otel_exports_traces_to_tempo(self):
        cfg = yaml.safe_load(
            (ROOT / "infra/otel/otel-collector.yml").read_text()
        )
        exporters = cfg["exporters"]
        assert "otlp_http/tempo" in exporters
        assert exporters["otlp_http/tempo"]["endpoint"] == "http://tempo:4318"
        assert "otlp_grpc/jaeger" not in exporters
        traces = cfg["service"]["pipelines"]["traces"]
        assert "otlp_http/tempo" in traces["exporters"]
        assert "memory_limiter" in traces["processors"]

• Step 7.2: Run test to verify it fails

uv run pytest tests/test_observability.py::TestOtelTempo -v

Expected: FAIL.

• Step 7.3: Replace infra/otel/otel-collector.yml

# OpenTelemetry Collector configuration
# Fan-out: traces → Tempo, metrics → Prometheus scrape, logs → Loki

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024

  memory_limiter:
    check_interval: 5s
    limit_mib: 256
    spike_limit_mib: 64

  resource:
    attributes:
      - key: deployment.environment
        value: homelab
        action: upsert

exporters:
  otlp_http/tempo:
    endpoint: http://tempo:4318
    tls:
      insecure: true

  otlp_http/loki:
    endpoint: http://loki:3100/otlp

  prometheus:
    endpoint: 0.0.0.0:8889
    namespace: stack
    resource_to_telemetry_conversion:
      enabled: true

extensions:
  health_check:
    endpoint: 0.0.0.0:13133

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch, resource]
      exporters: [otlp_http/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch, resource]
      exporters: [otlp_http/loki]

• Step 7.4: Update otel-collector compose dependency

In compose.yml, change otel-collector.depends_on:

  otel-collector:
    # ... existing fields ...
    depends_on:
      - tempo
      - loki

(Replace the previous - jaeger entry with - tempo.)

• Step 7.5: Run tests, restart otel-collector, verify

uv run pytest tests/test_observability.py::TestOtelTempo -v
docker compose up -d --force-recreate otel-collector
docker logs otel-collector 2>&1 | tail -15
# expected: "Everything is ready", no exporter errors
docker exec otel-collector wget -q -O- http://localhost:13133/
# expected: "Server available" or 200 OK

• Step 7.6: Commit

git add infra/otel/otel-collector.yml compose.yml tests/test_observability.py
git commit -m "feat(otel): export traces to Tempo, add memory_limiter"

---

Task 8: Prometheus targets

Files:

• Modify: infra/prometheus/targets/services.yml

• Test: tests/test_observability.py

• Step 8.1: Write the failing test

Append to tests/test_observability.py:

class TestPrometheusTargets:
    def test_targets_swap_jaeger_for_tempo_and_add_exporters(self):
        path = ROOT / "infra/prometheus/targets/services.yml"
        targets = yaml.safe_load(path.read_text())
        jobs = {entry["labels"]["job"] for entry in targets if "labels" in entry}
        assert "tempo" in jobs
        assert "nvidia-gpu" in jobs
        assert "jaeger" not in jobs

• Step 8.2: Run test to verify it fails

uv run pytest tests/test_observability.py::TestPrometheusTargets -v

Expected: FAIL.

• Step 8.3: Update infra/prometheus/targets/services.yml

# ── Scrape Target Registry ───────────────────────────
# To add a service: append one entry below.
# Pattern: container_name:port, job label, optional __metrics_path__.

# Traefik
- targets: ['traefik:8080']
  labels:
    job: traefik
    __metrics_path__: /metrics

# Tempo
- targets: ['tempo:3200']
  labels:
    job: tempo
    __metrics_path__: /metrics

# Loki
- targets: ['loki:3100']
  labels:
    job: loki

# NVIDIA GPU (DCGM)
- targets: ['nvidia-exporter:9400']
  labels:
    job: nvidia-gpu

# Nessie
- targets: ['nessie:9000']
  labels:
    job: nessie
    __metrics_path__: /q/metrics

# Polaris
- targets: ['polaris:8182']
  labels:
    job: polaris
    __metrics_path__: /q/metrics

# Trino
- targets: ['trino:8080']
  labels:
    job: trino
    __metrics_path__: /v1/status

# OTel Collector
- targets: ['otel-collector:8889']
  labels:
    job: otel-collector

• Step 8.4: Run tests, reload prometheus, verify

uv run pytest tests/test_observability.py::TestPrometheusTargets -v
# Hot-reload (web.enable-lifecycle is on)
docker exec prometheus wget -q --post-data= -O- http://localhost:9090/-/reload
docker exec prometheus wget -q -O- 'http://localhost:9090/api/v1/targets?state=active' | grep -o '"job":"[^"]*"' | sort -u
# expected: nvidia-gpu, tempo, traefik, loki, etc.

• Step 8.5: Commit

git add infra/prometheus/targets/services.yml tests/test_observability.py
git commit -m "feat(prom): add nvidia/tempo targets, drop jaeger"

---

Task 9: Grafana datasources — Tempo + correlation

Files:

• Modify: infra/grafana/provisioning/datasources/datasources.yml

• Test: tests/test_observability.py

• Step 9.1: Write the failing test

Append to tests/test_observability.py:

class TestGrafanaDatasources:
    def test_datasources_have_tempo_loki_default_correlation(self):
        path = ROOT / "infra/grafana/provisioning/datasources/datasources.yml"
        cfg = yaml.safe_load(path.read_text())
        ds = {d["name"]: d for d in cfg["datasources"]}
        assert "Tempo" in ds and "Jaeger" not in ds
        assert ds["Loki"].get("isDefault") is True
        tempo = ds["Tempo"]
        assert tempo["jsonData"]["tracesToLogsV2"]["datasourceUid"] == "loki"
        assert tempo["jsonData"]["serviceMap"]["datasourceUid"] == "prometheus"

• Step 9.2: Run test to verify it fails

uv run pytest tests/test_observability.py::TestGrafanaDatasources -v

Expected: FAIL.

• Step 9.3: Replace datasources file

Write infra/grafana/provisioning/datasources/datasources.yml:

apiVersion: 1
datasources:
  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    uid: loki
    isDefault: true
    jsonData:
      maxLines: 1000

  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    uid: prometheus
    jsonData:
      httpMethod: POST

  - name: Tempo
    type: tempo
    access: proxy
    url: http://tempo:3200
    uid: tempo
    jsonData:
      tracesToLogsV2:
        datasourceUid: loki
        filterByTraceID: true
      serviceMap:
        datasourceUid: prometheus

• Step 9.4: Update grafana compose depends_on

In compose.yml, edit the grafana.depends_on: list:

  grafana:
    # ... existing fields ...
    depends_on:
      - loki
      - tempo
      - prometheus

(Replace - jaeger with - tempo.)

• Step 9.5: Run tests, restart grafana, verify

uv run pytest tests/test_observability.py::TestGrafanaDatasources -v
docker compose up -d --force-recreate grafana
docker logs grafana 2>&1 | grep -i "datasource" | tail
# expected: "successfully provisioned" lines for Loki / Prometheus / Tempo

• Step 9.6: Commit

git add infra/grafana/provisioning/datasources/datasources.yml compose.yml tests/test_observability.py
git commit -m "feat(grafana): swap Jaeger→Tempo, enable trace↔log correlation"

---

Task 10: promtail=true labels + Traefik access logs

Files:

• Modify: compose.yml

• Test: tests/test_observability.py

• Step 10.1: Write the failing test

Append to tests/test_observability.py:

PROMTAIL_LABELED = {
    "coredns", "traefik", "rustfs", "postgres", "git", "act-runner",
    "notebooks", "zotero", "webdav", "nessie", "trino", "polaris",
    "dashboard", "docs", "api", "mail-poller", "auth-handler",
    "oauth2-proxy", "cloudflared", "loki", "promtail", "tempo",
    "otel-collector", "prometheus", "grafana",
    "nvidia-exporter",
}

PROMTAIL_NOT_LABELED = {"mc", "wire"}


class TestPromtailLabels:
    def test_eligible_services_have_promtail_label(self):
        svcs = _services()
        missing = []
        for name in PROMTAIL_LABELED:
            if name not in svcs:
                missing.append(f"  {name}: not in compose")
                continue
            labels = svcs[name].get("labels", []) or []
            if "promtail=true" not in labels:
                missing.append(f"  {name}: missing promtail=true label")
        assert not missing, "\n".join(missing)

    def test_excluded_services_do_not_have_label(self):
        svcs = _services()
        for name in PROMTAIL_NOT_LABELED:
            if name not in svcs:
                continue
            labels = svcs[name].get("labels", []) or []
            assert "promtail=true" not in labels, f"{name} should NOT be labeled"

    def test_traefik_access_logs_enabled(self):
        traefik = _services()["traefik"]
        cmd = traefik.get("command", [])
        if isinstance(cmd, str):
            cmd = cmd.split()
        flat = " ".join(str(x) for x in cmd)
        assert "--accesslog" in flat or any(
            "TRAEFIK_ACCESSLOG" in str(e) for e in traefik.get("environment", [])
        )

• Step 10.2: Run test to verify it fails

uv run pytest tests/test_observability.py::TestPromtailLabels -v

Expected: FAIL.

• Step 10.3: Add promtail=true to every eligible service

For each service listed in PROMTAIL_LABELED above (do them one at a time so commits are reviewable), add this block under the service definition:

    labels:
      - "promtail=true"

Tip: services with no existing labels: key need a new one; services that already have labels: (none in the current compose, but possible in future) need the entry appended.

Special cases:

• promtail itself gets promtail=true (it scrapes its own logs)

• mc and wire (the tools profile) get no label — see test

• tempo, nvidia-exporter were created in tasks 1–3; add the label to those service blocks too

• Step 10.4: Enable Traefik JSON access logs

In compose.yml, replace the traefik service block's command: (currently absent — Traefik runs its baked default) by adding:

  traefik:
    image: traefik:v3.3
    container_name: traefik
    networks:
      - gateway
      - observability
      - storage
      - ci
    ports:
      - "80:80"
      - "443:443"
      - "8081:8080"
    volumes:
      - ./infra/traefik/traefik.yml:/etc/traefik/traefik.yml:ro
      - ./infra/traefik/dynamic:/etc/traefik/dynamic:ro
      - ./infra/traefik/certs:/etc/traefik/certs:ro
      - ./infra/traefik/plugins:/plugins-local:ro
    environment:
      - DOMAIN=${DOMAIN:-fhirworx.io}
      - OTEL_SERVICE_NAME=traefik
      - TRAEFIK_ACCESSLOG=true
      - TRAEFIK_ACCESSLOG_FORMAT=json
      - TRAEFIK_ACCESSLOG_FIELDS_DEFAULTMODE=keep
      - TRAEFIK_ACCESSLOG_FIELDS_HEADERS_DEFAULTMODE=keep
    labels:
      - "promtail=true"
    security_opt:
      - no-new-privileges:true
    restart: unless-stopped

(The existing fields are preserved; only environment: gains four TRAEFIK_ACCESSLOG_* vars and labels: is added.)

• Step 10.5: Run tests, recreate the affected services, verify

uv run pytest tests/test_observability.py::TestPromtailLabels -v

# Recreate everything so labels register on running containers
docker compose up -d

# Promtail should now have many targets
docker exec promtail wget -q -O- 'http://loki:3100/loki/api/v1/label/container/values' | python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d.get('data',[])), 'containers'); print(d['data'])"
# expected: count ≥ 20

# Hit a Traefik route to generate an access log
curl -sk https://traefik:8081/ping >/dev/null || true
docker exec promtail wget -q -O- 'http://loki:3100/loki/api/v1/query?query={container="traefik"}' | head -c 500
# expected: JSON access-log entries

• Step 10.6: Commit

git add compose.yml tests/test_observability.py
git commit -m "feat(observability): label all services for promtail; enable Traefik access logs"

---

Task 11: Grafana OAuth via Gitea

Files:

• Modify: dev/scripts/bootstrap_sso.py

• Modify: compose.yml (grafana service)

• Test: tests/test_observability.py

• Step 11.1: Write the failing test

Append to tests/test_observability.py:

class TestGrafanaOAuth:
    def test_grafana_loads_state_env_file(self):
        env_files = _services()["grafana"].get("env_file", [])
        # accepts either a list of dicts or a list of strings
        joined = yaml.safe_dump(env_files)
        assert ".state/gitea/grafana.env" in joined

    def test_grafana_oauth_env_vars(self):
        envs = _services()["grafana"].get("environment", [])
        text = "\n".join(envs)
        assert "GF_AUTH_GENERIC_OAUTH_ENABLED=true" in text
        assert "GF_AUTH_GENERIC_OAUTH_NAME=Gitea" in text
        assert "GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH=is_admin && 'GrafanaAdmin'" in text

    def test_bootstrap_sso_provisions_grafana(self):
        src = (ROOT / "dev/scripts/bootstrap_sso.py").read_text()
        assert "GRAFANA_ENV" in src
        assert 'name="grafana"' in src
        assert "/login/generic_oauth" in src

• Step 11.2: Run test to verify it fails

uv run pytest tests/test_observability.py::TestGrafanaOAuth -v

Expected: FAIL.

• Step 11.3: Extend dev/scripts/bootstrap_sso.py

Add after the existing OAUTH2_PROXY_ENV = STATE / "oauth2-proxy.env" line:

GRAFANA_ENV = STATE / "grafana.env"

Add this helper function near _write_oauth2_proxy_env:

def _write_grafana_env(client_id: str, client_secret: str) -> bool:
    return _write_env_file(
        GRAFANA_ENV,
        (
            f"GF_AUTH_GENERIC_OAUTH_CLIENT_ID={client_id}\n"
            f"GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET={client_secret}\n"
        ),
    )

In main(), after the existing oauth2-proxy block (creds = _ensure_oauth_app(... name="oauth2-proxy" ...); _write_oauth2_proxy_env(*creds)), append:

        grafana_creds = _ensure_oauth_app(
            client,
            token,
            name="grafana",
            redirect_uri=f"https://grafana.{DOMAIN}/login/generic_oauth",
        )
        if grafana_creds:
            _write_grafana_env(*grafana_creds)

• Step 11.4: Update grafana service in compose.yml

Replace the existing grafana service block with:

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    networks:
      - gateway
      - observability
    env_file:
      - path: .state/gitea/grafana.env
        required: false
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=${GF_ADMIN_PASSWORD:-admin}
      - GF_USERS_ALLOW_SIGN_UP=false
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer
      - GF_SERVER_ROOT_URL=https://grafana.${DOMAIN:-fhirworx.io}
      - GF_AUTH_GENERIC_OAUTH_ENABLED=true
      - GF_AUTH_GENERIC_OAUTH_NAME=Gitea
      - GF_AUTH_GENERIC_OAUTH_SCOPES=openid profile email
      - GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://git.${DOMAIN:-fhirworx.io}/login/oauth/authorize
      - GF_AUTH_GENERIC_OAUTH_TOKEN_URL=http://git:3000/login/oauth/access_token
      - GF_AUTH_GENERIC_OAUTH_API_URL=http://git:3000/api/v1/user
      - GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=true
      - GF_AUTH_GENERIC_OAUTH_AUTO_LOGIN=false
      - GF_AUTH_OAUTH_ALLOW_INSECURE_EMAIL_LOOKUP=true
      - GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH=is_admin && 'GrafanaAdmin'
      - GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_GRAFANA_ADMIN=true
    volumes:
      - ./infra/grafana/provisioning:/etc/grafana/provisioning:ro
      - ./infra/grafana/dashboards:/var/lib/grafana/dashboards:ro
      - grafana_data:/var/lib/grafana
      - ./assets/icons/favicon.svg:/usr/share/grafana/public/img/grafana_icon.svg:ro
      - ./assets/icons/fav32.png:/usr/share/grafana/public/img/fav32.png:ro
      - ./assets/icons/apple-touch-icon.png:/usr/share/grafana/public/img/apple-touch-icon.png:ro
    depends_on:
      - loki
      - tempo
      - prometheus
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "wget --no-verbose --tries=1 --spider http://localhost:3000/api/health || exit 1",
        ]
      interval: 15s
      timeout: 5s
      retries: 5
      start_period: 30s
    labels:
      - "promtail=true"
    security_opt:
      - no-new-privileges:true
    restart: unless-stopped

• Step 11.5: Run tests + run wire + restart grafana

uv run pytest tests/test_observability.py::TestGrafanaOAuth -v

# Provision the OAuth app (Gitea must be running)
docker compose run --rm wire
# expected output includes: ok: OAuth2 app 'grafana' created

# Verify the env file was written
test -f .state/gitea/grafana.env && head -1 .state/gitea/grafana.env

docker compose up -d --force-recreate grafana
docker logs grafana 2>&1 | grep -i "oauth" | tail
# expected: lines mentioning generic_oauth setup

• Step 11.6: Manual SSO smoke test

Open https://grafana.${DOMAIN} in a browser (or https://localhost:3000 via hosts override). Click Sign in with Gitea. Confirm:

• Redirects to Gitea login

• After Gitea auth, returns to Grafana

• Logged-in user has GrafanaAdmin role (top-right user → preferences)

• Step 11.7: Commit

git add dev/scripts/bootstrap_sso.py compose.yml tests/test_observability.py
git commit -m "feat(grafana): Gitea OAuth SSO via bootstrap_sso.py"

---

Task 12: Subdomains + Traefik route swap (jaeger → tempo)

Files:

• Modify: stack.toml

• Modify: dev/scripts/bootstrap_sso.py (SUBDOMAINS)

• Modify: infra/traefik/dynamic/services.yml

• Test: tests/test_observability.py

• Step 12.1: Write the failing test

Append to tests/test_observability.py:

class TestSubdomainSwap:
    def test_stack_toml_subdomains(self):
        import tomllib
        cfg = tomllib.loads((ROOT / "stack.toml").read_text())
        subs = cfg["platform"]["subdomains"]
        assert "tempo" in subs
        assert "jaeger" not in subs

    def test_bootstrap_sso_subdomains(self):
        src = (ROOT / "dev/scripts/bootstrap_sso.py").read_text()
        assert '"tempo"' in src
        assert '"jaeger"' not in src

    def test_traefik_route_swap(self):
        src = (ROOT / "infra/traefik/dynamic/services.yml").read_text()
        assert '"tempo"' in src
        assert '"jaeger"' not in src

• Step 12.2: Run test to verify it fails

uv run pytest tests/test_observability.py::TestSubdomainSwap -v

Expected: FAIL.

• Step 12.3: Edit stack.toml

In stack.toml, replace "jaeger" with "tempo" in the subdomains list:

subdomains = [
    "dashboard", "docs", "git", "ci", "notebooks", "zotero",
    "webdav", "api", "nessie", "trino", "polaris",
    "grafana", "prometheus", "tempo", "loki",
    "s3", "s3console",
]

• Step 12.4: Edit dev/scripts/bootstrap_sso.py

In the SUBDOMAINS list, replace "jaeger" with "tempo".

• Step 12.5: Edit infra/traefik/dynamic/services.yml

Find the line "jaeger" (dict "port" "16686" "theme" true "mw" "git-sso,infra-headers") and change it to:

  "tempo"              (dict "port" "3200"  "theme" true  "mw" "git-sso,infra-headers")

• Step 12.6: Run tests, reload traefik, verify route

uv run pytest tests/test_observability.py::TestSubdomainSwap -v
# Traefik picks up dynamic config without restart, but force a reload to be safe:
docker compose restart traefik
# Hit the new endpoint via Traefik (note: requires DNS/hosts entry for tempo.${DOMAIN})
curl -sk https://tempo.${DOMAIN:-fhirworx.io}/ready
# expected: "ready"

# Re-run wire to push the new tunnel ingress to Cloudflare
docker compose run --rm wire

• Step 12.7: Commit

git add stack.toml dev/scripts/bootstrap_sso.py infra/traefik/dynamic/services.yml tests/test_observability.py
git commit -m "feat(routing): swap jaeger→tempo subdomain and Traefik route"

---

Task 13: Remove Jaeger

Files:

• Modify: compose.yml

• Test: tests/test_observability.py

• Step 13.1: Write the failing test

Append to tests/test_observability.py:

class TestJaegerRemoved:
    def test_jaeger_service_gone(self):
        assert "jaeger" not in _services()

• Step 13.2: Run test to verify it fails

uv run pytest tests/test_observability.py::TestJaegerRemoved -v

Expected: FAIL.

• Step 13.3: Remove the jaeger: service block from compose.yml

Delete the entire block — including the comment # Observability Stack if Jaeger was the first entry under it (the comment can stay since loki/tempo etc. follow).

• Step 13.4: Run tests, recreate stack, verify nothing broke

uv run pytest tests/test_observability.py -v   # full suite
docker compose down jaeger 2>/dev/null || true   # remove the running container if present
docker compose up -d
docker compose ps   # no jaeger entry

• Step 13.5: Commit

git add compose.yml tests/test_observability.py
git commit -m "chore(observability): remove Jaeger (replaced by Tempo)"

---

Task 14: Six per-service Grafana dashboards

Files:

• Create: infra/grafana/dashboards/homelab-overview.json
• Create: infra/grafana/dashboards/gateway-auth.json
• Create: infra/grafana/dashboards/data-lake.json
• Create: infra/grafana/dashboards/data-pipelines.json
• Create: infra/grafana/dashboards/ci.json
• Create: infra/grafana/dashboards/gpu-notebooks.json
• Test: tests/test_observability.py

For each dashboard: scaffold the JSON using the homelab-overview.json template below, then set title, uid, and replace the panels with the queries listed for that dashboard. Each panels[] entry needs id, title, type, gridPos {x,y,w,h}, datasource {uid}, and a targets[] array. Use type: "timeseries" for time series, "stat" for single-stat, "logs" for log panels.

• Step 14.1: Write the failing test

Append to tests/test_observability.py:

import json

EXPECTED_DASHBOARDS = {
    "homelab-overview", "gateway-auth", "data-lake",
    "data-pipelines", "ci", "gpu-notebooks",
}


class TestDashboards:
    def test_all_dashboards_exist_and_parse(self):
        dash_dir = ROOT / "infra/grafana/dashboards"
        files = {p.stem for p in dash_dir.glob("*.json")}
        missing = EXPECTED_DASHBOARDS - files
        assert not missing, f"missing dashboards: {missing}"
        for name in EXPECTED_DASHBOARDS:
            with (dash_dir / f"{name}.json").open() as f:
                data = json.load(f)
            assert "panels" in data, f"{name}: no panels[]"
            assert data["panels"], f"{name}: empty panels[]"

• Step 14.2: Run test to verify it fails

uv run pytest tests/test_observability.py::TestDashboards -v

Expected: FAIL.

• Step 14.3: Create homelab-overview.json (template)

{
  "title": "Homelab Overview",
  "uid": "homelab-overview",
  "tags": ["homelab", "stack"],
  "timezone": "browser",
  "schemaVersion": 39,
  "time": {"from": "now-6h", "to": "now"},
  "refresh": "30s",
  "panels": [
    {
      "id": 1,
      "title": "Containers up",
      "type": "stat",
      "gridPos": {"x": 0, "y": 0, "w": 4, "h": 4},
      "datasource": {"uid": "prometheus"},
      "targets": [{"expr": "count(container_last_seen{name!=\"\"})"}]
    },
    {
      "id": 2,
      "title": "Log volume / 1m",
      "type": "stat",
      "gridPos": {"x": 4, "y": 0, "w": 4, "h": 4},
      "datasource": {"uid": "loki"},
      "targets": [{"expr": "sum(rate({container=~\".+\"}[1m]))"}]
    },
    {
      "id": 3,
      "title": "Error rate / 1m",
      "type": "stat",
      "gridPos": {"x": 8, "y": 0, "w": 4, "h": 4},
      "datasource": {"uid": "loki"},
      "targets": [{"expr": "sum(rate({level=\"error\"}[1m]))"}]
    },
    {
      "id": 4,
      "title": "Container CPU %",
      "type": "timeseries",
      "gridPos": {"x": 0, "y": 4, "w": 12, "h": 8},
      "datasource": {"uid": "prometheus"},
      "targets": [
        {"expr": "rate(container_cpu_usage_seconds_total{name!=\"\"}[1m]) * 100", "legendFormat": "{{name}}"}
      ]
    },
    {
      "id": 5,
      "title": "Container memory (MiB)",
      "type": "timeseries",
      "gridPos": {"x": 12, "y": 4, "w": 12, "h": 8},
      "datasource": {"uid": "prometheus"},
      "targets": [
        {"expr": "container_memory_working_set_bytes{name!=\"\"} / 1024 / 1024", "legendFormat": "{{name}}"}
      ]
    },
    {
      "id": 6,
      "title": "Top error sources (1h)",
      "type": "logs",
      "gridPos": {"x": 0, "y": 12, "w": 24, "h": 8},
      "datasource": {"uid": "loki"},
      "targets": [{"expr": "{level=\"error\"} | json"}]
    }
  ]
}

• Step 14.4: Create gateway-auth.json

Reuse the template from 14.3, change title to "Gateway & Auth" and uid to "gateway-auth", and set panels to (replace ids/positions/queries):

[
  {"title": "Traefik request rate by status",      "datasource": "prometheus", "expr": "sum by (code) (rate(traefik_service_requests_total[1m]))"},
  {"title": "Traefik latency p95 (s)",             "datasource": "prometheus", "expr": "histogram_quantile(0.95, sum by (le) (rate(traefik_service_request_duration_seconds_bucket[5m])))"},
  {"title": "oauth2-proxy auth events",            "datasource": "loki",       "expr": "sum by (oauth_event) (rate({container=\"oauth2-proxy\"} | json [1m]))"},
  {"title": "Gitea login + push events",           "datasource": "loki",       "expr": "sum by (gitea_event) (rate({container=\"git\"} | regexp `(?P<gitea_event>login|push|webhook)` [1m]))"},
  {"title": "Failed auth tail",                    "datasource": "loki",       "expr": "{container=~\"oauth2-proxy|git\"} |~ \"(?i)(failure|forbidden|401|403)\"", "type": "logs"}
]

Translate each entry into a full panel object with the same shape as in 14.3 (id 1..N incrementing, gridPos 12-col grid two panels per row).

• Step 14.5: Create data-lake.json

title "Data Lake", uid "data-lake", panels:

[
  {"title": "Trino queries (status)",       "datasource": "loki",       "expr": "sum by (trino_event) (rate({container=\"trino\"}[1m]))"},
  {"title": "Trino query duration p95",     "datasource": "prometheus", "expr": "histogram_quantile(0.95, sum by (le) (rate(trino_query_execution_seconds_bucket[5m])))"},
  {"title": "Nessie/Polaris HTTP rate",     "datasource": "prometheus", "expr": "sum by (job) (rate(http_server_requests_seconds_count{job=~\"nessie|polaris\"}[1m]))"},
  {"title": "Nessie/Polaris error rate",    "datasource": "prometheus", "expr": "sum by (job) (rate(http_server_requests_seconds_count{job=~\"nessie|polaris\",status=~\"5..\"}[1m]))"},
  {"title": "RustFS S3 ops by op",          "datasource": "loki",       "expr": "sum by (s3_op) (rate({container=\"rustfs\"} | json [1m]))"}
]

• Step 14.6: Create data-pipelines.json

title "Data Pipelines", uid "data-pipelines", panels:

[
  {"title": "API request rate (status)",   "datasource": "prometheus", "expr": "sum by (status) (rate(stack_http_server_requests_total[1m]))"},
  {"title": "Mail-poller poll cycles",     "datasource": "loki",       "expr": "sum(rate({container=\"mail-poller\", mail_event=\"poll\"}[5m]))"},
  {"title": "Mail-poller ingest count",    "datasource": "loki",       "expr": "sum(rate({container=\"mail-poller\", mail_event=\"ingest\"}[5m]))"},
  {"title": "Pipeline errors",             "datasource": "loki",       "expr": "sum by (pipeline_name) (rate({container=\"api\", api_event=\"error\"}[5m]))"},
  {"title": "Pipeline error tail",         "datasource": "loki",       "expr": "{container=\"api\", level=\"error\"}", "type": "logs"}
]

If infra/grafana/dashboards/pipeline-performance.json exists and overlaps, fold its panels into this file and delete pipeline-performance.json in the same step.

• Step 14.7: Create ci.json

title "CI", uid "ci", panels:

[
  {"title": "Active CI jobs",              "datasource": "loki",       "expr": "count_over_time({container=\"act-runner\", ci_event=\"job started\"}[24h])"},
  {"title": "Job outcomes (24h)",          "datasource": "loki",       "expr": "sum by (ci_event) (count_over_time({container=\"act-runner\", ci_event=~\"job (started|finished|failed)\"}[24h]))"},
  {"title": "Gitea push events",           "datasource": "loki",       "expr": "sum(rate({container=\"git\", gitea_event=\"push\"}[5m]))"},
  {"title": "Webhook events",              "datasource": "loki",       "expr": "sum(rate({container=\"git\", gitea_event=\"webhook\"}[5m]))"}
]

• Step 14.8: Create gpu-notebooks.json

title "GPU & Notebooks", uid "gpu-notebooks", panels:

[
  {"title": "GPU utilization %",           "datasource": "prometheus", "expr": "DCGM_FI_DEV_GPU_UTIL"},
  {"title": "GPU memory used (MiB)",       "datasource": "prometheus", "expr": "DCGM_FI_DEV_FB_USED"},
  {"title": "GPU temp (°C)",               "datasource": "prometheus", "expr": "DCGM_FI_DEV_GPU_TEMP"},
  {"title": "GPU power (W)",               "datasource": "prometheus", "expr": "DCGM_FI_DEV_POWER_USAGE"},
  {"title": "Notebooks/Zotero CPU %",      "datasource": "prometheus", "expr": "rate(container_cpu_usage_seconds_total{name=~\"notebooks|zotero\"}[1m]) * 100", "legendFormat": "{{name}}"},
  {"title": "Notebooks/Zotero memory",     "datasource": "prometheus", "expr": "container_memory_working_set_bytes{name=~\"notebooks|zotero\"} / 1024 / 1024",   "legendFormat": "{{name}}"}
]

• Step 14.9: Run tests + verify Grafana picks them up

uv run pytest tests/test_observability.py::TestDashboards -v
docker compose restart grafana
docker logs grafana 2>&1 | grep -i "dashboard" | tail -20
# expected: "finished provisioning" entries for each new file

# In a browser at https://grafana.${DOMAIN} → Dashboards, all six should appear

• Step 14.10: Commit

git add infra/grafana/dashboards/*.json tests/test_observability.py
git commit -m "feat(grafana): six per-service dashboards"

---

Task 15: End-to-end verification

• Step 15.1: Full suite green

uv run pytest tests/test_observability.py tests/test_compose_mounts.py -v
# expected: all green

• Step 15.2: All services up + healthy

docker compose up -d
sleep 60
docker compose ps --format 'table {{.Name}}\t{{.State}}\t{{.Status}}'
# expected: every service in "running"; healthchecked services show "healthy"

• Step 15.3: Cross-signal correlation works

# Generate a trace by hitting the api (which is OTel-instrumented)
curl -sk https://api.${DOMAIN:-fhirworx.io}/health >/dev/null

# In Grafana → Explore:
#   1. Switch datasource to Tempo, run "service.name = api"
#   2. Click any span → Logs panel should auto-open with matching service.name in Loki
#   3. Open the gpu-notebooks dashboard — DCGM panels populated
#   4. Open data-lake dashboard — RustFS S3 ops panel populates after `mc ls` against rustfs
docker exec mc mc ls local/ >/dev/null

• Step 15.4: Final commit if any cleanup landed

If any of the above steps required a tweak (legend formatting, panel sizing, etc.), commit it now:

git add -A
git status   # confirm only intended changes
git commit -m "fix(observability): post-verification tweaks"   # if applicable

---

Self-Review Notes

Cross-check against the spec sections:

Spec section	Plan task
Compose changes — add tempo/nvidia-exporter	Tasks 1–3
Compose changes — remove jaeger	Task 13
Compose changes — promtail=true labels	Task 10
Compose changes — Traefik access logs	Task 10
Compose changes — promtail docker_sd	Task 5
Subdomain map (stack.toml, bootstrap_sso, traefik)	Task 12
Loki retention/compactor	Task 4
Tempo config	Task 1
OTel collector swap	Task 7
Prometheus targets	Task 8
Grafana datasources + correlation	Task 9
Grafana OAuth via Gitea	Task 11
Six per-service dashboards	Task 14
Migration verification (Grafana login, trace→logs)	Task 15

All spec sections covered. No placeholders. Method/label names consistent across tasks (promtail=true, tempo_data, STATE / "grafana.env").