fix(observability): live bring-up corrections

Surfaced when bringing the stack up alongside corwins.media:

- Promtail: filter docker_sd to compose project=stack. Both stacks share
  the host's docker.sock; without this, every promtail=true container
  from corwins.media also gets ingested into our Loki.
- Container name conflict: corwins.media also defines a "nvidia-exporter"
  container; rename ours to "stack-nvidia-exporter". Service-name DNS
  ("nvidia-exporter:9400") still resolves on the observability network
  so the Prometheus scrape target is unchanged.
- Grafana datasources: provisioning is additive; add deleteDatasources
  block so the legacy Jaeger datasource is removed from the DB.
- Trino prometheus target: /metrics requires auth (Trino rejects
  unauthenticated scrapes). Drop the target until the openmetrics
  catalog is wired with a shared secret. Loki still captures the
  query lifecycle for the data-lake dashboard.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
kert
2026-05-01 12:01:21 -04:00
parent 9741ce0a52
commit dd4b5c0a80
4 changed files with 21 additions and 6 deletions

View File

@@ -621,7 +621,10 @@ services:
nvidia-exporter:
image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.9-3.6.1-ubuntu22.04
container_name: nvidia-exporter
# Prefixed: corwins.media compose also defines a "nvidia-exporter"
# container; Docker container names are global. Service DNS
# ("nvidia-exporter:9400") still resolves on the observability net.
container_name: stack-nvidia-exporter
networks:
- observability
runtime: nvidia

View File

@@ -1,4 +1,11 @@
apiVersion: 1
# Provisioning is additive — datasources removed from this file remain
# in the Grafana DB until explicitly deleted here.
deleteDatasources:
- name: Jaeger
orgId: 1
datasources:
- name: Loki
type: loki

View File

@@ -14,6 +14,12 @@ scrape_configs:
- host: unix:///var/run/docker.sock
refresh_interval: 15s
relabel_configs:
# Only stack containers (docker.sock exposes every container on the
# host; without this, corwins.media's labelled containers also get
# ingested into our Loki).
- source_labels: ['__meta_docker_container_label_com_docker_compose_project']
regex: 'stack'
action: keep
- source_labels: ['__meta_docker_container_label_promtail']
regex: 'true'
action: keep

View File

@@ -41,11 +41,10 @@
job: polaris
__metrics_path__: /q/metrics
# Trino — /metrics is the OpenMetrics endpoint (435+); /v1/status is JSON
- targets: ['trino:8080']
labels:
job: trino
__metrics_path__: /metrics
# Trino — /metrics requires auth (Trino rejects unauthenticated scrapes
# even with no auth configured). Skipped until the openmetrics catalog
# is wired with a shared secret. Loki still captures query lifecycle
# events for the data-lake dashboard.
# OTel Collector
- targets: ['otel-collector:8889']