stack/docs/superpowers/plans/2026-05-01-observability-port.md

# Observability Port (corwins.media → stack) Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Replace Jaeger with Tempo, add nvidia-exporter, switch promtail from filesystem scraping to docker_sd opt-in, port per-service log pipelines to the stack's services, enable Loki retention, wire Grafana SSO via Gitea, and provision six per-service dashboards.

**Architecture:** Compose-native — every change lands in `compose.yml`, an `infra/<component>/<config>.yml`, or a JSON dashboard. The full stack runs on Podman/Docker with the existing networks (`gateway`, `observability`, `storage`, `data`, `ci`). New services live on `observability`; tempo + nvidia-exporter only need docker.sock and host filesystem reads, so the network's `internal: true` constraint is preserved. Grafana already has `gateway + observability`, which is exactly what's needed for OAuth token exchange to `git:3000` plus internal datasource access.

**Tech Stack:** Docker Compose, Loki, Promtail, Grafana, Prometheus, Tempo, OpenTelemetry Collector, DCGM-exporter (NVIDIA), Traefik, Gitea (OAuth provider).

**Spec:** `docs/superpowers/specs/2026-05-01-observability-port-design.md`

**Working pattern for verification:** stack uses `pytest tests/test_compose_mounts.py` to validate compose, plus live `docker compose up` + `curl` probes for runtime checks. We'll add `tests/test_observability.py` for the compose-level invariants and use `bash deploy.sh` helpers (`wait_healthy`) where appropriate.

---

## File Structure

**Created:**
- `infra/tempo/tempo.yml` — tempo runtime config
- `infra/grafana/dashboards/homelab-overview.json` — fleet health dashboard
- `infra/grafana/dashboards/gateway-auth.json` — Traefik + oauth2-proxy + Gitea
- `infra/grafana/dashboards/data-lake.json` — Trino + Nessie + Polaris + RustFS
- `infra/grafana/dashboards/data-pipelines.json` — api + mail-poller + pipelines
- `infra/grafana/dashboards/ci.json` — act-runner + Gitea push
- `infra/grafana/dashboards/gpu-notebooks.json` — DCGM + notebooks/zotero
- `tests/test_observability.py` — compose invariant tests for the new layout

**Modified:**
- `compose.yml` — add tempo/nvidia-exporter, drop jaeger, label all services with `promtail=true`, Traefik access logs, Grafana OAuth env
- `infra/loki/loki-config.yml` — retention + compactor
- `infra/loki/promtail-config.yml` — full rewrite (docker_sd + per-service)
- `infra/otel/otel-collector.yml` — traces → tempo, add memory_limiter
- `infra/prometheus/targets/services.yml` — add nvidia/tempo, drop jaeger
- `infra/grafana/provisioning/datasources/datasources.yml` — Tempo replaces Jaeger; Loki default; correlation
- `infra/traefik/dynamic/services.yml` — swap jaeger → tempo route
- `stack.toml` — `subdomains` swap jaeger → tempo
- `dev/scripts/bootstrap_sso.py` — `SUBDOMAINS` swap jaeger → tempo; add Grafana OAuth app + grafana.env writer

---

## Pre-flight

- [ ] **Step 0.1: Confirm prerequisites**

```bash
# verify GPU is present (notebooks already uses CUDA)
nvidia-smi >/dev/null && echo "GPU OK"

# verify docker GID for promtail group_add
getent group docker | cut -d: -f3
# Note this value — you'll set DOCKER_GID in .env if it isn't already 999
```

- [ ] **Step 0.2: Create the test scaffolding file**

Create `tests/test_observability.py`:

```python
"""Compose-level invariants for the observability port.

refs docs/superpowers/specs/2026-05-01-observability-port-design.md
"""

from __future__ import annotations

from pathlib import Path

import yaml

ROOT = Path(__file__).resolve().parents[1]
COMPOSE = ROOT / "compose.yml"


def _load() -> dict:
    return yaml.safe_load(COMPOSE.read_text())


def _services() -> dict:
    return _load().get("services", {})
```

- [ ] **Step 0.3: Commit scaffolding**

```bash
git add tests/test_observability.py
git commit -m "test(observability): add compose-invariant test scaffold"
```

---

## Task 1: Tempo service + config

**Files:**
- Create: `infra/tempo/tempo.yml`
- Modify: `compose.yml` (add `tempo` service, add `tempo_data` volume)
- Test: `tests/test_observability.py`

- [ ] **Step 1.1: Write the failing test**

Append to `tests/test_observability.py`:

```python
class TestTempo:
    def test_tempo_service_present(self):
        svcs = _services()
        assert "tempo" in svcs
        assert svcs["tempo"]["networks"] == ["observability"]

    def test_tempo_volume_declared(self):
        vols = _load().get("volumes", {})
        assert "tempo_data" in vols

    def test_tempo_config_mounted(self):
        svc = _services()["tempo"]
        mounts = [v for v in svc["volumes"] if "/etc/tempo" in v]
        assert any("./infra/tempo/tempo.yml" in v for v in mounts)
```

- [ ] **Step 1.2: Run test to verify it fails**

```bash
uv run pytest tests/test_observability.py::TestTempo -v
```
Expected: FAIL with `KeyError: 'tempo'` or `AssertionError`.

- [ ] **Step 1.3: Create the tempo config file**

Write `infra/tempo/tempo.yml`:

```yaml
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

ingester:
  trace_idle_period: 30s
  max_block_bytes: 1048576
  max_block_duration: 5m

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/traces
    wal:
      path: /var/tempo/wal
```

- [ ] **Step 1.4: Add tempo service to compose.yml**

Insert this block in `compose.yml` after the `loki` service (before `promtail`):

```yaml
  tempo:
    image: grafana/tempo:2.7.2
    container_name: tempo
    networks:
      - observability
    volumes:
      - ./infra/tempo/tempo.yml:/etc/tempo/config.yaml:ro
      - tempo_data:/var/tempo
    command: -config.file=/etc/tempo/config.yaml
    healthcheck:
      test: ["CMD", "wget", "-q", "--spider", "http://localhost:3200/ready"]
      interval: 30s
      timeout: 5s
      retries: 5
      start_period: 30s
    security_opt:
      - no-new-privileges:true
    restart: unless-stopped
```

Add `tempo_data:` to the bottom `volumes:` block (alphabetised next to `prometheus_data:`):

```yaml
volumes:
  postgres_data:
  rustfs_data:
  rustfs_logs:
  gitea_data:
  gitea_config:
  act_runner_data:
  loki_data:
  prometheus_data:
  tempo_data:
  grafana_data:
```

- [ ] **Step 1.5: Run tests to verify they pass**

```bash
uv run pytest tests/test_observability.py::TestTempo -v
```
Expected: PASS.

- [ ] **Step 1.6: Verify compose parses and tempo comes up**

```bash
docker compose config -q
docker compose up -d tempo
docker inspect --format='{{.State.Health.Status}}' tempo
# expected: "healthy" within 30-60s
curl -sf http://localhost:3200/ready  # may need to publish port temporarily; otherwise:
docker exec tempo wget -q -O- http://localhost:3200/ready
# expected: "ready"
```

- [ ] **Step 1.7: Commit**

```bash
git add infra/tempo/tempo.yml compose.yml tests/test_observability.py
git commit -m "feat(observability): add Tempo trace backend"
```

---

## Task 3: nvidia-exporter (DCGM) service

**Files:**
- Modify: `compose.yml`
- Test: `tests/test_observability.py`

- [ ] **Step 3.1: Write the failing test**

Append to `tests/test_observability.py`:

```python
class TestNvidiaExporter:
    def test_nvidia_exporter_present(self):
        svcs = _services()
        assert "nvidia-exporter" in svcs

    def test_nvidia_runtime(self):
        svc = _services()["nvidia-exporter"]
        assert svc.get("runtime") == "nvidia"

    def test_nvidia_gpu_reservation(self):
        svc = _services()["nvidia-exporter"]
        devices = svc["deploy"]["resources"]["reservations"]["devices"]
        assert any(d.get("driver") == "nvidia" for d in devices)
```

- [ ] **Step 3.2: Run test to verify it fails**

```bash
uv run pytest tests/test_observability.py::TestNvidiaExporter -v
```
Expected: FAIL.

- [ ] **Step 3.3: Add service**

Insert after the `tempo` service block:

```yaml
  nvidia-exporter:
    image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.9-3.6.1-ubuntu22.04
    container_name: nvidia-exporter
    networks:
      - observability
    runtime: nvidia
    environment:
      - DCGM_EXPORTER_NO_HOSTNAME=1
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    cap_add:
      - SYS_ADMIN
    restart: unless-stopped
```

- [ ] **Step 3.4: Run tests, bring service up, verify**

```bash
uv run pytest tests/test_observability.py::TestNvidiaExporter -v
docker compose up -d nvidia-exporter
docker exec nvidia-exporter curl -sf http://localhost:9400/metrics | grep -c '^DCGM_FI_DEV_GPU_UTIL'
# expected: > 0
```

- [ ] **Step 3.5: Commit**

```bash
git add compose.yml tests/test_observability.py
git commit -m "feat(observability): add DCGM nvidia-exporter for GPU metrics"
```

---

## Task 4: Loki retention + compactor

**Files:**
- Modify: `infra/loki/loki-config.yml`
- Test: `tests/test_observability.py`

- [ ] **Step 4.1: Write the failing test**

Append to `tests/test_observability.py`:

```python
class TestLokiRetention:
    def test_loki_compactor_configured(self):
        cfg = yaml.safe_load((ROOT / "infra/loki/loki-config.yml").read_text())
        assert cfg["compactor"]["retention_enabled"] is True
        assert cfg["limits_config"]["retention_period"] == "168h"
```

- [ ] **Step 4.2: Run test to verify it fails**

```bash
uv run pytest tests/test_observability.py::TestLokiRetention -v
```
Expected: FAIL with `KeyError: 'compactor'`.

- [ ] **Step 4.3: Update loki config**

Replace the contents of `infra/loki/loki-config.yml` with:

```yaml
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

limits_config:
  retention_period: 168h
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  ingestion_rate_mb: 16
  ingestion_burst_size_mb: 32
  max_query_series: 5000
  allow_structured_metadata: true

compactor:
  working_directory: /loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  delete_request_store: filesystem

analytics:
  reporting_enabled: false
```

- [ ] **Step 4.4: Run tests, restart loki, verify**

```bash
uv run pytest tests/test_observability.py::TestLokiRetention -v
docker compose up -d --force-recreate loki
docker logs loki 2>&1 | tail -30
# expected: no "config error", "ready" within ~15s
docker exec loki wget -q -O- http://localhost:3100/ready
# expected: "ready"
```

- [ ] **Step 4.5: Commit**

```bash
git add infra/loki/loki-config.yml tests/test_observability.py
git commit -m "feat(loki): enable 7d retention via compactor"
```

---

## Task 5: Promtail rewrite — docker_sd + generic stages

**Files:**
- Modify: `compose.yml` (promtail service: drop containers bind, add docker.sock + group_add)
- Modify: `infra/loki/promtail-config.yml` (full rewrite, generic stages only)
- Test: `tests/test_observability.py`

- [ ] **Step 5.1: Write the failing test**

Append to `tests/test_observability.py`:

```python
class TestPromtailRewire:
    def test_promtail_uses_docker_sock(self):
        svc = _services()["promtail"]
        sources = [v.split(":")[0] for v in svc["volumes"] if isinstance(v, str)]
        # New: docker.sock; old filesystem mount must be gone.
        assert any("docker.sock" in s for s in sources)
        assert not any("/var/lib/docker/containers" in v for v in svc["volumes"])

    def test_promtail_has_docker_gid(self):
        svc = _services()["promtail"]
        assert "group_add" in svc

    def test_promtail_config_is_docker_sd(self):
        cfg = yaml.safe_load(
            (ROOT / "infra/loki/promtail-config.yml").read_text()
        )
        jobs = cfg["scrape_configs"]
        docker_jobs = [j for j in jobs if j.get("job_name") == "docker"]
        assert docker_jobs, "expected job_name=docker"
        assert "docker_sd_configs" in docker_jobs[0]
```

- [ ] **Step 5.2: Run test to verify it fails**

```bash
uv run pytest tests/test_observability.py::TestPromtailRewire -v
```
Expected: FAIL.

- [ ] **Step 5.3: Replace promtail config (generic stages only — per-service comes in Task 6)**

Write `infra/loki/promtail-config.yml`:

```yaml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 15s
    relabel_configs:
      - source_labels: ['__meta_docker_container_label_promtail']
        regex: 'true'
        action: keep
      - source_labels: ['__meta_docker_container_name']
        regex: '/?(.*)'
        target_label: container
      - source_labels: ['__meta_docker_container_name']
        regex: '/?(.*)'
        target_label: job
      - source_labels: ['__meta_docker_container_label_com_docker_compose_service']
        target_label: service
      - source_labels: ['__meta_docker_container_label_com_docker_compose_project']
        target_label: project
    pipeline_stages:
      - docker: {}
      - regex:
          expression: '(?i)(?P<level>error|warn|info|debug|fatal|critical|trace)'
      - labels:
          level:
      - regex:
          expression: '(?P<remote_ip>\d+\.\d+\.\d+\.\d+) .* "(?P<http_method>GET|POST|PUT|DELETE|PATCH|HEAD|OPTIONS) (?P<http_path>[^ ]*) [^"]*" (?P<http_status>\d{3}) (?P<http_bytes>\d+)'
      - labels:
          http_method:
          http_status:
```

- [ ] **Step 5.4: Update the promtail service in compose.yml**

Replace the existing `promtail:` service block with:

```yaml
  promtail:
    image: grafana/promtail:latest
    container_name: promtail
    networks:
      - observability
    volumes:
      - ./infra/loki/promtail-config.yml:/etc/promtail/config.yml:ro
      - ${DOCKER_SOCK:-/run/user/1000/docker.sock}:/var/run/docker.sock:ro
    command: -config.file=/etc/promtail/config.yml
    depends_on:
      - loki
    group_add:
      - "${DOCKER_GID:-985}"
    security_opt:
      - no-new-privileges:true
    restart: unless-stopped
```

(If `getent group docker` returned a different number than 985 in step 0.1, set `DOCKER_GID` in `.env`.)

- [ ] **Step 5.5: Run tests, recreate promtail, verify scraping works**

```bash
uv run pytest tests/test_observability.py::TestPromtailRewire -v

# Add a temporary promtail=true label to one container so scraping has *something* to find:
# We'll batch-label everything in Task 10. For this verification, label loki itself.
docker compose stop promtail
docker compose rm -f promtail
docker compose up -d promtail
docker logs promtail 2>&1 | tail -20
# expected: "Adding target", no auth/socket errors

# After Task 10 labels are applied, the next probe is meaningful. For now:
docker exec promtail wget -q -O- 'http://loki:3100/loki/api/v1/labels' | head
# expected: JSON with at least {"data":[...]} — empty list is acceptable here pre-labels
```

- [ ] **Step 5.6: Commit**

```bash
git add infra/loki/promtail-config.yml compose.yml tests/test_observability.py
git commit -m "feat(promtail): switch to docker_sd opt-in scraping"
```

---

## Task 6: Per-service log pipeline stages

**Files:**
- Modify: `infra/loki/promtail-config.yml`
- Test: `tests/test_observability.py`

- [ ] **Step 6.1: Write the failing test**

Append to `tests/test_observability.py`:

```python
class TestPerServiceStages:
    """Each named container must have a matching pipeline stage that emits at
    least one service-specific label."""

    EXPECTED = {
        "traefik": "tf_event",
        "git": "gitea_event",
        "oauth2-proxy": "oauth_event",
        "postgres": "pg_event",
        "trino": "trino_event",
        "rustfs": "s3_op",
        "api": "api_event",
        "mail-poller": "mail_event",
        "act-runner": "ci_event",
        "cloudflared": "cf_event",
    }

    def test_each_service_has_a_match_stage(self):
        cfg = yaml.safe_load(
            (ROOT / "infra/loki/promtail-config.yml").read_text()
        )
        stages = cfg["scrape_configs"][0]["pipeline_stages"]
        text = yaml.safe_dump(stages)
        for container, label in self.EXPECTED.items():
            assert f'container="{container}"' in text, f"no match stage for {container}"
            assert label in text, f"no {label} extraction for {container}"
```

- [ ] **Step 6.2: Run test to verify it fails**

```bash
uv run pytest tests/test_observability.py::TestPerServiceStages -v
```
Expected: FAIL.

- [ ] **Step 6.3: Append per-service stages to promtail config**

Append the following to `infra/loki/promtail-config.yml` (under the existing
`pipeline_stages:` block of the `docker` job — keep the generic stages first):

```yaml
      # --- Traefik ---
      - match:
          selector: '{container="traefik"}'
          stages:
            - json:
                expressions:
                  RouterName: RouterName
                  ServiceName: ServiceName
                  DownstreamStatus: DownstreamStatus
                  RequestMethod: RequestMethod
            - labels:
                tf_router: RouterName
                tf_service: ServiceName
                http_status: DownstreamStatus
                http_method: RequestMethod
            - regex:
                expression: '(?P<tf_event>TLS handshake|EntryPoint|router|certResolver|middleware|tls)'
            - labels:
                tf_event:

      # --- Gitea ---
      - match:
          selector: '{container="git"}'
          stages:
            - regex:
                expression: '(?P<gitea_event>login|logout|register|push|pull|branch|tag|issue|webhook|oauth|created|deleted|merged)'
            - labels:
                gitea_event:

      # --- oauth2-proxy ---
      - match:
          selector: '{container="oauth2-proxy"}'
          stages:
            - regex:
                expression: '(?P<oauth_event>AuthSuccess|AuthFailure|OAuthStart|OAuthCallback|OAuthError|Invalid|Expired|Authenticated|Forbidden)'
            - labels:
                oauth_event:

      # --- Postgres ---
      - match:
          selector: '{container="postgres"}'
          stages:
            - regex:
                expression: '(?P<pg_event>FATAL|ERROR|WARNING|checkpoint|autovacuum|connection authorized|connection received|statement)'
            - labels:
                pg_event:

      # --- Trino ---
      - match:
          selector: '{container="trino"}'
          stages:
            - regex:
                expression: '(?P<trino_event>QUERY_CREATED|QUERY_STARTED|QUERY_COMPLETED|QUERY_FAILED|SPLIT_COMPLETED)'
            - regex:
                expression: '\b(?P<query_id>[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})\b'
            - labels:
                trino_event:
                query_id:

      # --- Nessie / Polaris (Quarkus) ---
      - match:
          selector: '{container=~"nessie|polaris"}'
          stages:
            - regex:
                expression: '"(?P<http_method>GET|POST|PUT|DELETE) (?P<quarkus_route>/[^ ?]+)'
            - labels:
                quarkus_route:

      # --- RustFS ---
      - match:
          selector: '{container="rustfs"}'
          stages:
            - regex:
                expression: '(?P<s3_op>PutObject|GetObject|DeleteObject|ListObjectsV2|HeadObject|CompleteMultipart)'
            - regex:
                expression: 'bucket=(?P<s3_bucket>[A-Za-z0-9._-]+)'
            - labels:
                s3_op:
                s3_bucket:

      # --- API (FastAPI/stack) ---
      - match:
          selector: '{container="api"}'
          stages:
            - regex:
                expression: '(?P<api_event>pipeline|ingest|health|error)'
            - regex:
                expression: 'pipeline=(?P<pipeline_name>[A-Za-z0-9_.-]+)'
            - labels:
                api_event:
                pipeline_name:

      # --- mail-poller ---
      - match:
          selector: '{container="mail-poller"}'
          stages:
            - regex:
                expression: '(?P<mail_event>ingest|poll|error)'
            - regex:
                expression: 'count=(?P<mail_count>\d+)'
            - labels:
                mail_event:
                mail_count:

      # --- act-runner ---
      - match:
          selector: '{container="act-runner"}'
          stages:
            - regex:
                expression: '(?P<ci_event>job started|job finished|job failed|runner registered)'
            - regex:
                expression: 'job_id=(?P<job_id>\S+)'
            - labels:
                ci_event:
                job_id:

      # --- cloudflared ---
      - match:
          selector: '{container="cloudflared"}'
          stages:
            - regex:
                expression: '(?P<cf_event>connection|registered|unregistered|reconnect|quic|error|tunnel|origin)'
            - regex:
                expression: 'status=(?P<cf_origin_status>\d{3})'
            - labels:
                cf_event:
                cf_origin_status:
```

- [ ] **Step 6.4: Run tests, restart promtail, verify**

```bash
uv run pytest tests/test_observability.py::TestPerServiceStages -v
docker compose restart promtail
docker logs promtail 2>&1 | tail -20
# expected: no "yaml: line N" parse errors
```

- [ ] **Step 6.5: Commit**

```bash
git add infra/loki/promtail-config.yml tests/test_observability.py
git commit -m "feat(promtail): per-service log pipeline stages"
```

---

## Task 7: OTel collector — traces to Tempo

**Files:**
- Modify: `infra/otel/otel-collector.yml`
- Test: `tests/test_observability.py`

- [ ] **Step 7.1: Write the failing test**

Append to `tests/test_observability.py`:

```python
class TestOtelTempo:
    def test_otel_exports_traces_to_tempo(self):
        cfg = yaml.safe_load(
            (ROOT / "infra/otel/otel-collector.yml").read_text()
        )
        exporters = cfg["exporters"]
        assert "otlp_http/tempo" in exporters
        assert exporters["otlp_http/tempo"]["endpoint"] == "http://tempo:4318"
        assert "otlp_grpc/jaeger" not in exporters
        traces = cfg["service"]["pipelines"]["traces"]
        assert "otlp_http/tempo" in traces["exporters"]
        assert "memory_limiter" in traces["processors"]
```

- [ ] **Step 7.2: Run test to verify it fails**

```bash
uv run pytest tests/test_observability.py::TestOtelTempo -v
```
Expected: FAIL.

- [ ] **Step 7.3: Replace `infra/otel/otel-collector.yml`**

```yaml
# OpenTelemetry Collector configuration
# Fan-out: traces → Tempo, metrics → Prometheus scrape, logs → Loki

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024

  memory_limiter:
    check_interval: 5s
    limit_mib: 256
    spike_limit_mib: 64

  resource:
    attributes:
      - key: deployment.environment
        value: homelab
        action: upsert

exporters:
  otlp_http/tempo:
    endpoint: http://tempo:4318
    tls:
      insecure: true

  otlp_http/loki:
    endpoint: http://loki:3100/otlp

  prometheus:
    endpoint: 0.0.0.0:8889
    namespace: stack
    resource_to_telemetry_conversion:
      enabled: true

extensions:
  health_check:
    endpoint: 0.0.0.0:13133

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch, resource]
      exporters: [otlp_http/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch, resource]
      exporters: [otlp_http/loki]
```

- [ ] **Step 7.4: Update otel-collector compose dependency**

In `compose.yml`, change `otel-collector.depends_on`:

```yaml
  otel-collector:
    # ... existing fields ...
    depends_on:
      - tempo
      - loki
```

(Replace the previous `- jaeger` entry with `- tempo`.)

- [ ] **Step 7.5: Run tests, restart otel-collector, verify**

```bash
uv run pytest tests/test_observability.py::TestOtelTempo -v
docker compose up -d --force-recreate otel-collector
docker logs otel-collector 2>&1 | tail -15
# expected: "Everything is ready", no exporter errors
docker exec otel-collector wget -q -O- http://localhost:13133/
# expected: "Server available" or 200 OK
```

- [ ] **Step 7.6: Commit**

```bash
git add infra/otel/otel-collector.yml compose.yml tests/test_observability.py
git commit -m "feat(otel): export traces to Tempo, add memory_limiter"
```

---

## Task 8: Prometheus targets

**Files:**
- Modify: `infra/prometheus/targets/services.yml`
- Test: `tests/test_observability.py`

- [ ] **Step 8.1: Write the failing test**

Append to `tests/test_observability.py`:

```python
class TestPrometheusTargets:
    def test_targets_swap_jaeger_for_tempo_and_add_exporters(self):
        path = ROOT / "infra/prometheus/targets/services.yml"
        targets = yaml.safe_load(path.read_text())
        jobs = {entry["labels"]["job"] for entry in targets if "labels" in entry}
        assert "tempo" in jobs
        assert "nvidia-gpu" in jobs
        assert "jaeger" not in jobs
```

- [ ] **Step 8.2: Run test to verify it fails**

```bash
uv run pytest tests/test_observability.py::TestPrometheusTargets -v
```
Expected: FAIL.

- [ ] **Step 8.3: Update `infra/prometheus/targets/services.yml`**

```yaml
# ── Scrape Target Registry ───────────────────────────
# To add a service: append one entry below.
# Pattern: container_name:port, job label, optional __metrics_path__.

# Traefik
- targets: ['traefik:8080']
  labels:
    job: traefik
    __metrics_path__: /metrics

# Tempo
- targets: ['tempo:3200']
  labels:
    job: tempo
    __metrics_path__: /metrics

# Loki
- targets: ['loki:3100']
  labels:
    job: loki

# NVIDIA GPU (DCGM)
- targets: ['nvidia-exporter:9400']
  labels:
    job: nvidia-gpu

# Nessie
- targets: ['nessie:9000']
  labels:
    job: nessie
    __metrics_path__: /q/metrics

# Polaris
- targets: ['polaris:8182']
  labels:
    job: polaris
    __metrics_path__: /q/metrics

# Trino
- targets: ['trino:8080']
  labels:
    job: trino
    __metrics_path__: /v1/status

# OTel Collector
- targets: ['otel-collector:8889']
  labels:
    job: otel-collector
```

- [ ] **Step 8.4: Run tests, reload prometheus, verify**

```bash
uv run pytest tests/test_observability.py::TestPrometheusTargets -v
# Hot-reload (web.enable-lifecycle is on)
docker exec prometheus wget -q --post-data= -O- http://localhost:9090/-/reload
docker exec prometheus wget -q -O- 'http://localhost:9090/api/v1/targets?state=active' | grep -o '"job":"[^"]*"' | sort -u
# expected: nvidia-gpu, tempo, traefik, loki, etc.
```

- [ ] **Step 8.5: Commit**

```bash
git add infra/prometheus/targets/services.yml tests/test_observability.py
git commit -m "feat(prom): add nvidia/tempo targets, drop jaeger"
```

---

## Task 9: Grafana datasources — Tempo + correlation

**Files:**
- Modify: `infra/grafana/provisioning/datasources/datasources.yml`
- Test: `tests/test_observability.py`

- [ ] **Step 9.1: Write the failing test**

Append to `tests/test_observability.py`:

```python
class TestGrafanaDatasources:
    def test_datasources_have_tempo_loki_default_correlation(self):
        path = ROOT / "infra/grafana/provisioning/datasources/datasources.yml"
        cfg = yaml.safe_load(path.read_text())
        ds = {d["name"]: d for d in cfg["datasources"]}
        assert "Tempo" in ds and "Jaeger" not in ds
        assert ds["Loki"].get("isDefault") is True
        tempo = ds["Tempo"]
        assert tempo["jsonData"]["tracesToLogsV2"]["datasourceUid"] == "loki"
        assert tempo["jsonData"]["serviceMap"]["datasourceUid"] == "prometheus"
```

- [ ] **Step 9.2: Run test to verify it fails**

```bash
uv run pytest tests/test_observability.py::TestGrafanaDatasources -v
```
Expected: FAIL.

- [ ] **Step 9.3: Replace datasources file**

Write `infra/grafana/provisioning/datasources/datasources.yml`:

```yaml
apiVersion: 1
datasources:
  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    uid: loki
    isDefault: true
    jsonData:
      maxLines: 1000

  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    uid: prometheus
    jsonData:
      httpMethod: POST

  - name: Tempo
    type: tempo
    access: proxy
    url: http://tempo:3200
    uid: tempo
    jsonData:
      tracesToLogsV2:
        datasourceUid: loki
        filterByTraceID: true
      serviceMap:
        datasourceUid: prometheus
```

- [ ] **Step 9.4: Update grafana compose `depends_on`**

In `compose.yml`, edit the `grafana.depends_on:` list:

```yaml
  grafana:
    # ... existing fields ...
    depends_on:
      - loki
      - tempo
      - prometheus
```

(Replace `- jaeger` with `- tempo`.)

- [ ] **Step 9.5: Run tests, restart grafana, verify**

```bash
uv run pytest tests/test_observability.py::TestGrafanaDatasources -v
docker compose up -d --force-recreate grafana
docker logs grafana 2>&1 | grep -i "datasource" | tail
# expected: "successfully provisioned" lines for Loki / Prometheus / Tempo
```

- [ ] **Step 9.6: Commit**

```bash
git add infra/grafana/provisioning/datasources/datasources.yml compose.yml tests/test_observability.py
git commit -m "feat(grafana): swap Jaeger→Tempo, enable trace↔log correlation"
```

---

## Task 10: `promtail=true` labels + Traefik access logs

**Files:**
- Modify: `compose.yml`
- Test: `tests/test_observability.py`

- [ ] **Step 10.1: Write the failing test**

Append to `tests/test_observability.py`:

```python
PROMTAIL_LABELED = {
    "coredns", "traefik", "rustfs", "postgres", "git", "act-runner",
    "notebooks", "zotero", "webdav", "nessie", "trino", "polaris",
    "dashboard", "docs", "api", "mail-poller", "auth-handler",
    "oauth2-proxy", "cloudflared", "loki", "promtail", "tempo",
    "otel-collector", "prometheus", "grafana",
    "nvidia-exporter",
}

PROMTAIL_NOT_LABELED = {"mc", "wire"}


class TestPromtailLabels:
    def test_eligible_services_have_promtail_label(self):
        svcs = _services()
        missing = []
        for name in PROMTAIL_LABELED:
            if name not in svcs:
                missing.append(f"  {name}: not in compose")
                continue
            labels = svcs[name].get("labels", []) or []
            if "promtail=true" not in labels:
                missing.append(f"  {name}: missing promtail=true label")
        assert not missing, "\n".join(missing)

    def test_excluded_services_do_not_have_label(self):
        svcs = _services()
        for name in PROMTAIL_NOT_LABELED:
            if name not in svcs:
                continue
            labels = svcs[name].get("labels", []) or []
            assert "promtail=true" not in labels, f"{name} should NOT be labeled"

    def test_traefik_access_logs_enabled(self):
        traefik = _services()["traefik"]
        cmd = traefik.get("command", [])
        if isinstance(cmd, str):
            cmd = cmd.split()
        flat = " ".join(str(x) for x in cmd)
        assert "--accesslog" in flat or any(
            "TRAEFIK_ACCESSLOG" in str(e) for e in traefik.get("environment", [])
        )
```

- [ ] **Step 10.2: Run test to verify it fails**

```bash
uv run pytest tests/test_observability.py::TestPromtailLabels -v
```
Expected: FAIL.

- [ ] **Step 10.3: Add `promtail=true` to every eligible service**

For **each** service listed in `PROMTAIL_LABELED` above (do them one at a time
so commits are reviewable), add this block under the service definition:

```yaml
    labels:
      - "promtail=true"
```

Tip: services with no existing `labels:` key need a new one; services that
already have `labels:` (none in the current compose, but possible in future)
need the entry appended.

Special cases:
- `promtail` itself gets `promtail=true` (it scrapes its own logs)
- `mc` and `wire` (the `tools` profile) get **no** label — see test
- `tempo`, `nvidia-exporter` were created in tasks 1–3; add the
  label to those service blocks too

- [ ] **Step 10.4: Enable Traefik JSON access logs**

In `compose.yml`, replace the `traefik` service block's `command:` (currently
absent — Traefik runs its baked default) by adding:

```yaml
  traefik:
    image: traefik:v3.3
    container_name: traefik
    networks:
      - gateway
      - observability
      - storage
      - ci
    ports:
      - "80:80"
      - "443:443"
      - "8081:8080"
    volumes:
      - ./infra/traefik/traefik.yml:/etc/traefik/traefik.yml:ro
      - ./infra/traefik/dynamic:/etc/traefik/dynamic:ro
      - ./infra/traefik/certs:/etc/traefik/certs:ro
      - ./infra/traefik/plugins:/plugins-local:ro
    environment:
      - DOMAIN=${DOMAIN:-fhirworx.io}
      - OTEL_SERVICE_NAME=traefik
      - TRAEFIK_ACCESSLOG=true
      - TRAEFIK_ACCESSLOG_FORMAT=json
      - TRAEFIK_ACCESSLOG_FIELDS_DEFAULTMODE=keep
      - TRAEFIK_ACCESSLOG_FIELDS_HEADERS_DEFAULTMODE=keep
    labels:
      - "promtail=true"
    security_opt:
      - no-new-privileges:true
    restart: unless-stopped
```

(The existing fields are preserved; only `environment:` gains four `TRAEFIK_ACCESSLOG_*` vars and `labels:` is added.)

- [ ] **Step 10.5: Run tests, recreate the affected services, verify**

```bash
uv run pytest tests/test_observability.py::TestPromtailLabels -v

# Recreate everything so labels register on running containers
docker compose up -d

# Promtail should now have many targets
docker exec promtail wget -q -O- 'http://loki:3100/loki/api/v1/label/container/values' | python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d.get('data',[])), 'containers'); print(d['data'])"
# expected: count ≥ 20

# Hit a Traefik route to generate an access log
curl -sk https://traefik:8081/ping >/dev/null || true
docker exec promtail wget -q -O- 'http://loki:3100/loki/api/v1/query?query={container="traefik"}' | head -c 500
# expected: JSON access-log entries
```

- [ ] **Step 10.6: Commit**

```bash
git add compose.yml tests/test_observability.py
git commit -m "feat(observability): label all services for promtail; enable Traefik access logs"
```

---

## Task 11: Grafana OAuth via Gitea

**Files:**
- Modify: `dev/scripts/bootstrap_sso.py`
- Modify: `compose.yml` (grafana service)
- Test: `tests/test_observability.py`

- [ ] **Step 11.1: Write the failing test**

Append to `tests/test_observability.py`:

```python
class TestGrafanaOAuth:
    def test_grafana_loads_state_env_file(self):
        env_files = _services()["grafana"].get("env_file", [])
        # accepts either a list of dicts or a list of strings
        joined = yaml.safe_dump(env_files)
        assert ".state/gitea/grafana.env" in joined

    def test_grafana_oauth_env_vars(self):
        envs = _services()["grafana"].get("environment", [])
        text = "\n".join(envs)
        assert "GF_AUTH_GENERIC_OAUTH_ENABLED=true" in text
        assert "GF_AUTH_GENERIC_OAUTH_NAME=Gitea" in text
        assert "GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH=is_admin && 'GrafanaAdmin'" in text

    def test_bootstrap_sso_provisions_grafana(self):
        src = (ROOT / "dev/scripts/bootstrap_sso.py").read_text()
        assert "GRAFANA_ENV" in src
        assert 'name="grafana"' in src
        assert "/login/generic_oauth" in src
```

- [ ] **Step 11.2: Run test to verify it fails**

```bash
uv run pytest tests/test_observability.py::TestGrafanaOAuth -v
```
Expected: FAIL.

- [ ] **Step 11.3: Extend `dev/scripts/bootstrap_sso.py`**

Add after the existing `OAUTH2_PROXY_ENV = STATE / "oauth2-proxy.env"` line:

```python
GRAFANA_ENV = STATE / "grafana.env"
```

Add this helper function near `_write_oauth2_proxy_env`:

```python
def _write_grafana_env(client_id: str, client_secret: str) -> bool:
    return _write_env_file(
        GRAFANA_ENV,
        (
            f"GF_AUTH_GENERIC_OAUTH_CLIENT_ID={client_id}\n"
            f"GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET={client_secret}\n"
        ),
    )
```

In `main()`, after the existing oauth2-proxy block (`creds = _ensure_oauth_app(... name="oauth2-proxy" ...); _write_oauth2_proxy_env(*creds)`), append:

```python
        grafana_creds = _ensure_oauth_app(
            client,
            token,
            name="grafana",
            redirect_uri=f"https://grafana.{DOMAIN}/login/generic_oauth",
        )
        if grafana_creds:
            _write_grafana_env(*grafana_creds)
```

- [ ] **Step 11.4: Update `grafana` service in compose.yml**

Replace the existing `grafana` service block with:

```yaml
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    networks:
      - gateway
      - observability
    env_file:
      - path: .state/gitea/grafana.env
        required: false
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=${GF_ADMIN_PASSWORD:-admin}
      - GF_USERS_ALLOW_SIGN_UP=false
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer
      - GF_SERVER_ROOT_URL=https://grafana.${DOMAIN:-fhirworx.io}
      - GF_AUTH_GENERIC_OAUTH_ENABLED=true
      - GF_AUTH_GENERIC_OAUTH_NAME=Gitea
      - GF_AUTH_GENERIC_OAUTH_SCOPES=openid profile email
      - GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://git.${DOMAIN:-fhirworx.io}/login/oauth/authorize
      - GF_AUTH_GENERIC_OAUTH_TOKEN_URL=http://git:3000/login/oauth/access_token
      - GF_AUTH_GENERIC_OAUTH_API_URL=http://git:3000/api/v1/user
      - GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=true
      - GF_AUTH_GENERIC_OAUTH_AUTO_LOGIN=false
      - GF_AUTH_OAUTH_ALLOW_INSECURE_EMAIL_LOOKUP=true
      - GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH=is_admin && 'GrafanaAdmin'
      - GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_GRAFANA_ADMIN=true
    volumes:
      - ./infra/grafana/provisioning:/etc/grafana/provisioning:ro
      - ./infra/grafana/dashboards:/var/lib/grafana/dashboards:ro
      - grafana_data:/var/lib/grafana
      - ./assets/icons/favicon.svg:/usr/share/grafana/public/img/grafana_icon.svg:ro
      - ./assets/icons/fav32.png:/usr/share/grafana/public/img/fav32.png:ro
      - ./assets/icons/apple-touch-icon.png:/usr/share/grafana/public/img/apple-touch-icon.png:ro
    depends_on:
      - loki
      - tempo
      - prometheus
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "wget --no-verbose --tries=1 --spider http://localhost:3000/api/health || exit 1",
        ]
      interval: 15s
      timeout: 5s
      retries: 5
      start_period: 30s
    labels:
      - "promtail=true"
    security_opt:
      - no-new-privileges:true
    restart: unless-stopped
```

- [ ] **Step 11.5: Run tests + run wire + restart grafana**

```bash
uv run pytest tests/test_observability.py::TestGrafanaOAuth -v

# Provision the OAuth app (Gitea must be running)
docker compose run --rm wire
# expected output includes: ok: OAuth2 app 'grafana' created

# Verify the env file was written
test -f .state/gitea/grafana.env && head -1 .state/gitea/grafana.env

docker compose up -d --force-recreate grafana
docker logs grafana 2>&1 | grep -i "oauth" | tail
# expected: lines mentioning generic_oauth setup
```

- [ ] **Step 11.6: Manual SSO smoke test**

Open `https://grafana.${DOMAIN}` in a browser (or `https://localhost:3000` via
hosts override). Click **Sign in with Gitea**. Confirm:
- Redirects to Gitea login
- After Gitea auth, returns to Grafana
- Logged-in user has `GrafanaAdmin` role (top-right user → preferences)

- [ ] **Step 11.7: Commit**

```bash
git add dev/scripts/bootstrap_sso.py compose.yml tests/test_observability.py
git commit -m "feat(grafana): Gitea OAuth SSO via bootstrap_sso.py"
```

---

## Task 12: Subdomains + Traefik route swap (jaeger → tempo)

**Files:**
- Modify: `stack.toml`
- Modify: `dev/scripts/bootstrap_sso.py` (`SUBDOMAINS`)
- Modify: `infra/traefik/dynamic/services.yml`
- Test: `tests/test_observability.py`

- [ ] **Step 12.1: Write the failing test**

Append to `tests/test_observability.py`:

```python
class TestSubdomainSwap:
    def test_stack_toml_subdomains(self):
        import tomllib
        cfg = tomllib.loads((ROOT / "stack.toml").read_text())
        subs = cfg["platform"]["subdomains"]
        assert "tempo" in subs
        assert "jaeger" not in subs

    def test_bootstrap_sso_subdomains(self):
        src = (ROOT / "dev/scripts/bootstrap_sso.py").read_text()
        assert '"tempo"' in src
        assert '"jaeger"' not in src

    def test_traefik_route_swap(self):
        src = (ROOT / "infra/traefik/dynamic/services.yml").read_text()
        assert '"tempo"' in src
        assert '"jaeger"' not in src
```

- [ ] **Step 12.2: Run test to verify it fails**

```bash
uv run pytest tests/test_observability.py::TestSubdomainSwap -v
```
Expected: FAIL.

- [ ] **Step 12.3: Edit `stack.toml`**

In `stack.toml`, replace `"jaeger"` with `"tempo"` in the `subdomains` list:

```toml
subdomains = [
    "dashboard", "docs", "git", "ci", "notebooks", "zotero",
    "webdav", "api", "nessie", "trino", "polaris",
    "grafana", "prometheus", "tempo", "loki",
    "s3", "s3console",
]
```

- [ ] **Step 12.4: Edit `dev/scripts/bootstrap_sso.py`**

In the `SUBDOMAINS` list, replace `"jaeger"` with `"tempo"`.

- [ ] **Step 12.5: Edit `infra/traefik/dynamic/services.yml`**

Find the line `"jaeger" (dict "port" "16686" "theme" true  "mw" "git-sso,infra-headers")` and change it to:

```yaml
  "tempo"              (dict "port" "3200"  "theme" true  "mw" "git-sso,infra-headers")
```

- [ ] **Step 12.6: Run tests, reload traefik, verify route**

```bash
uv run pytest tests/test_observability.py::TestSubdomainSwap -v
# Traefik picks up dynamic config without restart, but force a reload to be safe:
docker compose restart traefik
# Hit the new endpoint via Traefik (note: requires DNS/hosts entry for tempo.${DOMAIN})
curl -sk https://tempo.${DOMAIN:-fhirworx.io}/ready
# expected: "ready"

# Re-run wire to push the new tunnel ingress to Cloudflare
docker compose run --rm wire
```

- [ ] **Step 12.7: Commit**

```bash
git add stack.toml dev/scripts/bootstrap_sso.py infra/traefik/dynamic/services.yml tests/test_observability.py
git commit -m "feat(routing): swap jaeger→tempo subdomain and Traefik route"
```

---

## Task 13: Remove Jaeger

**Files:**
- Modify: `compose.yml`
- Test: `tests/test_observability.py`

- [ ] **Step 13.1: Write the failing test**

Append to `tests/test_observability.py`:

```python
class TestJaegerRemoved:
    def test_jaeger_service_gone(self):
        assert "jaeger" not in _services()
```

- [ ] **Step 13.2: Run test to verify it fails**

```bash
uv run pytest tests/test_observability.py::TestJaegerRemoved -v
```
Expected: FAIL.

- [ ] **Step 13.3: Remove the `jaeger:` service block from `compose.yml`**

Delete the entire block — including the comment `# Observability Stack` if Jaeger was the first entry under it (the comment can stay since loki/tempo etc. follow).

- [ ] **Step 13.4: Run tests, recreate stack, verify nothing broke**

```bash
uv run pytest tests/test_observability.py -v   # full suite
docker compose down jaeger 2>/dev/null || true   # remove the running container if present
docker compose up -d
docker compose ps   # no jaeger entry
```

- [ ] **Step 13.5: Commit**

```bash
git add compose.yml tests/test_observability.py
git commit -m "chore(observability): remove Jaeger (replaced by Tempo)"
```

---

## Task 14: Six per-service Grafana dashboards

**Files:**
- Create: `infra/grafana/dashboards/homelab-overview.json`
- Create: `infra/grafana/dashboards/gateway-auth.json`
- Create: `infra/grafana/dashboards/data-lake.json`
- Create: `infra/grafana/dashboards/data-pipelines.json`
- Create: `infra/grafana/dashboards/ci.json`
- Create: `infra/grafana/dashboards/gpu-notebooks.json`
- Test: `tests/test_observability.py`

> **For each dashboard:** scaffold the JSON using the `homelab-overview.json`
> template below, then set `title`, `uid`, and replace the panels with the
> queries listed for that dashboard. Each `panels[]` entry needs `id`,
> `title`, `type`, `gridPos {x,y,w,h}`, `datasource {uid}`, and a `targets[]`
> array. Use `type: "timeseries"` for time series, `"stat"` for single-stat,
> `"logs"` for log panels.

- [ ] **Step 14.1: Write the failing test**

Append to `tests/test_observability.py`:

```python
import json

EXPECTED_DASHBOARDS = {
    "homelab-overview", "gateway-auth", "data-lake",
    "data-pipelines", "ci", "gpu-notebooks",
}


class TestDashboards:
    def test_all_dashboards_exist_and_parse(self):
        dash_dir = ROOT / "infra/grafana/dashboards"
        files = {p.stem for p in dash_dir.glob("*.json")}
        missing = EXPECTED_DASHBOARDS - files
        assert not missing, f"missing dashboards: {missing}"
        for name in EXPECTED_DASHBOARDS:
            with (dash_dir / f"{name}.json").open() as f:
                data = json.load(f)
            assert "panels" in data, f"{name}: no panels[]"
            assert data["panels"], f"{name}: empty panels[]"
```

- [ ] **Step 14.2: Run test to verify it fails**

```bash
uv run pytest tests/test_observability.py::TestDashboards -v
```
Expected: FAIL.

- [ ] **Step 14.3: Create `homelab-overview.json` (template)**

```json
{
  "title": "Homelab Overview",
  "uid": "homelab-overview",
  "tags": ["homelab", "stack"],
  "timezone": "browser",
  "schemaVersion": 39,
  "time": {"from": "now-6h", "to": "now"},
  "refresh": "30s",
  "panels": [
    {
      "id": 1,
      "title": "Containers up",
      "type": "stat",
      "gridPos": {"x": 0, "y": 0, "w": 4, "h": 4},
      "datasource": {"uid": "prometheus"},
      "targets": [{"expr": "count(container_last_seen{name!=\"\"})"}]
    },
    {
      "id": 2,
      "title": "Log volume / 1m",
      "type": "stat",
      "gridPos": {"x": 4, "y": 0, "w": 4, "h": 4},
      "datasource": {"uid": "loki"},
      "targets": [{"expr": "sum(rate({container=~\".+\"}[1m]))"}]
    },
    {
      "id": 3,
      "title": "Error rate / 1m",
      "type": "stat",
      "gridPos": {"x": 8, "y": 0, "w": 4, "h": 4},
      "datasource": {"uid": "loki"},
      "targets": [{"expr": "sum(rate({level=\"error\"}[1m]))"}]
    },
    {
      "id": 4,
      "title": "Container CPU %",
      "type": "timeseries",
      "gridPos": {"x": 0, "y": 4, "w": 12, "h": 8},
      "datasource": {"uid": "prometheus"},
      "targets": [
        {"expr": "rate(container_cpu_usage_seconds_total{name!=\"\"}[1m]) * 100", "legendFormat": "{{name}}"}
      ]
    },
    {
      "id": 5,
      "title": "Container memory (MiB)",
      "type": "timeseries",
      "gridPos": {"x": 12, "y": 4, "w": 12, "h": 8},
      "datasource": {"uid": "prometheus"},
      "targets": [
        {"expr": "container_memory_working_set_bytes{name!=\"\"} / 1024 / 1024", "legendFormat": "{{name}}"}
      ]
    },
    {
      "id": 6,
      "title": "Top error sources (1h)",
      "type": "logs",
      "gridPos": {"x": 0, "y": 12, "w": 24, "h": 8},
      "datasource": {"uid": "loki"},
      "targets": [{"expr": "{level=\"error\"} | json"}]
    }
  ]
}
```

- [ ] **Step 14.4: Create `gateway-auth.json`**

Reuse the template from 14.3, change `title` to `"Gateway & Auth"` and `uid` to `"gateway-auth"`, and set `panels` to (replace ids/positions/queries):

```json
[
  {"title": "Traefik request rate by status",      "datasource": "prometheus", "expr": "sum by (code) (rate(traefik_service_requests_total[1m]))"},
  {"title": "Traefik latency p95 (s)",             "datasource": "prometheus", "expr": "histogram_quantile(0.95, sum by (le) (rate(traefik_service_request_duration_seconds_bucket[5m])))"},
  {"title": "oauth2-proxy auth events",            "datasource": "loki",       "expr": "sum by (oauth_event) (rate({container=\"oauth2-proxy\"} | json [1m]))"},
  {"title": "Gitea login + push events",           "datasource": "loki",       "expr": "sum by (gitea_event) (rate({container=\"git\"} | regexp `(?P<gitea_event>login|push|webhook)` [1m]))"},
  {"title": "Failed auth tail",                    "datasource": "loki",       "expr": "{container=~\"oauth2-proxy|git\"} |~ \"(?i)(failure|forbidden|401|403)\"", "type": "logs"}
]
```

Translate each entry into a full panel object with the same shape as in 14.3
(`id` 1..N incrementing, `gridPos` 12-col grid two panels per row).

- [ ] **Step 14.5: Create `data-lake.json`**

`title` `"Data Lake"`, `uid` `"data-lake"`, panels:

```json
[
  {"title": "Trino queries (status)",       "datasource": "loki",       "expr": "sum by (trino_event) (rate({container=\"trino\"}[1m]))"},
  {"title": "Trino query duration p95",     "datasource": "prometheus", "expr": "histogram_quantile(0.95, sum by (le) (rate(trino_query_execution_seconds_bucket[5m])))"},
  {"title": "Nessie/Polaris HTTP rate",     "datasource": "prometheus", "expr": "sum by (job) (rate(http_server_requests_seconds_count{job=~\"nessie|polaris\"}[1m]))"},
  {"title": "Nessie/Polaris error rate",    "datasource": "prometheus", "expr": "sum by (job) (rate(http_server_requests_seconds_count{job=~\"nessie|polaris\",status=~\"5..\"}[1m]))"},
  {"title": "RustFS S3 ops by op",          "datasource": "loki",       "expr": "sum by (s3_op) (rate({container=\"rustfs\"} | json [1m]))"}
]
```

- [ ] **Step 14.6: Create `data-pipelines.json`**

`title` `"Data Pipelines"`, `uid` `"data-pipelines"`, panels:

```json
[
  {"title": "API request rate (status)",   "datasource": "prometheus", "expr": "sum by (status) (rate(stack_http_server_requests_total[1m]))"},
  {"title": "Mail-poller poll cycles",     "datasource": "loki",       "expr": "sum(rate({container=\"mail-poller\", mail_event=\"poll\"}[5m]))"},
  {"title": "Mail-poller ingest count",    "datasource": "loki",       "expr": "sum(rate({container=\"mail-poller\", mail_event=\"ingest\"}[5m]))"},
  {"title": "Pipeline errors",             "datasource": "loki",       "expr": "sum by (pipeline_name) (rate({container=\"api\", api_event=\"error\"}[5m]))"},
  {"title": "Pipeline error tail",         "datasource": "loki",       "expr": "{container=\"api\", level=\"error\"}", "type": "logs"}
]
```

If `infra/grafana/dashboards/pipeline-performance.json` exists and overlaps, fold its panels into this file and delete `pipeline-performance.json` in the same step.

- [ ] **Step 14.7: Create `ci.json`**

`title` `"CI"`, `uid` `"ci"`, panels:

```json
[
  {"title": "Active CI jobs",              "datasource": "loki",       "expr": "count_over_time({container=\"act-runner\", ci_event=\"job started\"}[24h])"},
  {"title": "Job outcomes (24h)",          "datasource": "loki",       "expr": "sum by (ci_event) (count_over_time({container=\"act-runner\", ci_event=~\"job (started|finished|failed)\"}[24h]))"},
  {"title": "Gitea push events",           "datasource": "loki",       "expr": "sum(rate({container=\"git\", gitea_event=\"push\"}[5m]))"},
  {"title": "Webhook events",              "datasource": "loki",       "expr": "sum(rate({container=\"git\", gitea_event=\"webhook\"}[5m]))"}
]
```

- [ ] **Step 14.8: Create `gpu-notebooks.json`**

`title` `"GPU & Notebooks"`, `uid` `"gpu-notebooks"`, panels:

```json
[
  {"title": "GPU utilization %",           "datasource": "prometheus", "expr": "DCGM_FI_DEV_GPU_UTIL"},
  {"title": "GPU memory used (MiB)",       "datasource": "prometheus", "expr": "DCGM_FI_DEV_FB_USED"},
  {"title": "GPU temp (°C)",               "datasource": "prometheus", "expr": "DCGM_FI_DEV_GPU_TEMP"},
  {"title": "GPU power (W)",               "datasource": "prometheus", "expr": "DCGM_FI_DEV_POWER_USAGE"},
  {"title": "Notebooks/Zotero CPU %",      "datasource": "prometheus", "expr": "rate(container_cpu_usage_seconds_total{name=~\"notebooks|zotero\"}[1m]) * 100", "legendFormat": "{{name}}"},
  {"title": "Notebooks/Zotero memory",     "datasource": "prometheus", "expr": "container_memory_working_set_bytes{name=~\"notebooks|zotero\"} / 1024 / 1024",   "legendFormat": "{{name}}"}
]
```

- [ ] **Step 14.9: Run tests + verify Grafana picks them up**

```bash
uv run pytest tests/test_observability.py::TestDashboards -v
docker compose restart grafana
docker logs grafana 2>&1 | grep -i "dashboard" | tail -20
# expected: "finished provisioning" entries for each new file

# In a browser at https://grafana.${DOMAIN} → Dashboards, all six should appear
```

- [ ] **Step 14.10: Commit**

```bash
git add infra/grafana/dashboards/*.json tests/test_observability.py
git commit -m "feat(grafana): six per-service dashboards"
```

---

## Task 15: End-to-end verification

- [ ] **Step 15.1: Full suite green**

```bash
uv run pytest tests/test_observability.py tests/test_compose_mounts.py -v
# expected: all green
```

- [ ] **Step 15.2: All services up + healthy**

```bash
docker compose up -d
sleep 60
docker compose ps --format 'table {{.Name}}\t{{.State}}\t{{.Status}}'
# expected: every service in "running"; healthchecked services show "healthy"
```

- [ ] **Step 15.3: Cross-signal correlation works**

```bash
# Generate a trace by hitting the api (which is OTel-instrumented)
curl -sk https://api.${DOMAIN:-fhirworx.io}/health >/dev/null

# In Grafana → Explore:
#   1. Switch datasource to Tempo, run "service.name = api"
#   2. Click any span → Logs panel should auto-open with matching service.name in Loki
#   3. Open the gpu-notebooks dashboard — DCGM panels populated
#   4. Open data-lake dashboard — RustFS S3 ops panel populates after `mc ls` against rustfs
docker exec mc mc ls local/ >/dev/null
```

- [ ] **Step 15.4: Final commit if any cleanup landed**

If any of the above steps required a tweak (legend formatting, panel sizing,
etc.), commit it now:

```bash
git add -A
git status   # confirm only intended changes
git commit -m "fix(observability): post-verification tweaks"   # if applicable
```

---

## Self-Review Notes

Cross-check against the spec sections:

| Spec section | Plan task |
|---|---|
| Compose changes — add tempo/nvidia-exporter | Tasks 1–3 |
| Compose changes — remove jaeger | Task 13 |
| Compose changes — promtail=true labels | Task 10 |
| Compose changes — Traefik access logs | Task 10 |
| Compose changes — promtail docker_sd | Task 5 |
| Subdomain map (`stack.toml`, bootstrap_sso, traefik) | Task 12 |
| Loki retention/compactor | Task 4 |
| Tempo config | Task 1 |
| OTel collector swap | Task 7 |
| Prometheus targets | Task 8 |
| Grafana datasources + correlation | Task 9 |
| Grafana OAuth via Gitea | Task 11 |
| Six per-service dashboards | Task 14 |
| Migration verification (Grafana login, trace→logs) | Task 15 |

All spec sections covered. No placeholders. Method/label names consistent across tasks (`promtail=true`, `tempo_data`, `STATE / "grafana.env"`).