fix(observability): live bring-up corrections
Surfaced when bringing the stack up alongside corwins.media:
- Promtail: filter docker_sd to compose project=stack. Both stacks share
the host's docker.sock; without this, every promtail=true container
from corwins.media also gets ingested into our Loki.
- Container name conflict: corwins.media also defines a "nvidia-exporter"
container; rename ours to "stack-nvidia-exporter". Service-name DNS
("nvidia-exporter:9400") still resolves on the observability network
so the Prometheus scrape target is unchanged.
- Grafana datasources: provisioning is additive; add deleteDatasources
block so the legacy Jaeger datasource is removed from the DB.
- Trino prometheus target: /metrics requires auth (Trino rejects
unauthenticated scrapes). Drop the target until the openmetrics
catalog is wired with a shared secret. Loki still captures the
query lifecycle for the data-lake dashboard.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -621,7 +621,10 @@ services:
|
||||
|
||||
nvidia-exporter:
|
||||
image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.9-3.6.1-ubuntu22.04
|
||||
container_name: nvidia-exporter
|
||||
# Prefixed: corwins.media compose also defines a "nvidia-exporter"
|
||||
# container; Docker container names are global. Service DNS
|
||||
# ("nvidia-exporter:9400") still resolves on the observability net.
|
||||
container_name: stack-nvidia-exporter
|
||||
networks:
|
||||
- observability
|
||||
runtime: nvidia
|
||||
|
||||
@@ -1,4 +1,11 @@
|
||||
apiVersion: 1
|
||||
|
||||
# Provisioning is additive — datasources removed from this file remain
|
||||
# in the Grafana DB until explicitly deleted here.
|
||||
deleteDatasources:
|
||||
- name: Jaeger
|
||||
orgId: 1
|
||||
|
||||
datasources:
|
||||
- name: Loki
|
||||
type: loki
|
||||
|
||||
@@ -14,6 +14,12 @@ scrape_configs:
|
||||
- host: unix:///var/run/docker.sock
|
||||
refresh_interval: 15s
|
||||
relabel_configs:
|
||||
# Only stack containers (docker.sock exposes every container on the
|
||||
# host; without this, corwins.media's labelled containers also get
|
||||
# ingested into our Loki).
|
||||
- source_labels: ['__meta_docker_container_label_com_docker_compose_project']
|
||||
regex: 'stack'
|
||||
action: keep
|
||||
- source_labels: ['__meta_docker_container_label_promtail']
|
||||
regex: 'true'
|
||||
action: keep
|
||||
|
||||
@@ -41,11 +41,10 @@
|
||||
job: polaris
|
||||
__metrics_path__: /q/metrics
|
||||
|
||||
# Trino — /metrics is the OpenMetrics endpoint (435+); /v1/status is JSON
|
||||
- targets: ['trino:8080']
|
||||
labels:
|
||||
job: trino
|
||||
__metrics_path__: /metrics
|
||||
# Trino — /metrics requires auth (Trino rejects unauthenticated scrapes
|
||||
# even with no auth configured). Skipped until the openmetrics catalog
|
||||
# is wired with a shared secret. Loki still captures query lifecycle
|
||||
# events for the data-lake dashboard.
|
||||
|
||||
# OTel Collector
|
||||
- targets: ['otel-collector:8889']
|
||||
|
||||
Reference in New Issue
Block a user