fix(observability): live bring-up corrections
Surfaced when bringing the stack up alongside corwins.media:
- Promtail: filter docker_sd to compose project=stack. Both stacks share
the host's docker.sock; without this, every promtail=true container
from corwins.media also gets ingested into our Loki.
- Container name conflict: corwins.media also defines a "nvidia-exporter"
container; rename ours to "stack-nvidia-exporter". Service-name DNS
("nvidia-exporter:9400") still resolves on the observability network
so the Prometheus scrape target is unchanged.
- Grafana datasources: provisioning is additive; add deleteDatasources
block so the legacy Jaeger datasource is removed from the DB.
- Trino prometheus target: /metrics requires auth (Trino rejects
unauthenticated scrapes). Drop the target until the openmetrics
catalog is wired with a shared secret. Loki still captures the
query lifecycle for the data-lake dashboard.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -621,7 +621,10 @@ services:
|
|||||||
|
|
||||||
nvidia-exporter:
|
nvidia-exporter:
|
||||||
image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.9-3.6.1-ubuntu22.04
|
image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.9-3.6.1-ubuntu22.04
|
||||||
container_name: nvidia-exporter
|
# Prefixed: corwins.media compose also defines a "nvidia-exporter"
|
||||||
|
# container; Docker container names are global. Service DNS
|
||||||
|
# ("nvidia-exporter:9400") still resolves on the observability net.
|
||||||
|
container_name: stack-nvidia-exporter
|
||||||
networks:
|
networks:
|
||||||
- observability
|
- observability
|
||||||
runtime: nvidia
|
runtime: nvidia
|
||||||
|
|||||||
@@ -1,4 +1,11 @@
|
|||||||
apiVersion: 1
|
apiVersion: 1
|
||||||
|
|
||||||
|
# Provisioning is additive — datasources removed from this file remain
|
||||||
|
# in the Grafana DB until explicitly deleted here.
|
||||||
|
deleteDatasources:
|
||||||
|
- name: Jaeger
|
||||||
|
orgId: 1
|
||||||
|
|
||||||
datasources:
|
datasources:
|
||||||
- name: Loki
|
- name: Loki
|
||||||
type: loki
|
type: loki
|
||||||
|
|||||||
@@ -14,6 +14,12 @@ scrape_configs:
|
|||||||
- host: unix:///var/run/docker.sock
|
- host: unix:///var/run/docker.sock
|
||||||
refresh_interval: 15s
|
refresh_interval: 15s
|
||||||
relabel_configs:
|
relabel_configs:
|
||||||
|
# Only stack containers (docker.sock exposes every container on the
|
||||||
|
# host; without this, corwins.media's labelled containers also get
|
||||||
|
# ingested into our Loki).
|
||||||
|
- source_labels: ['__meta_docker_container_label_com_docker_compose_project']
|
||||||
|
regex: 'stack'
|
||||||
|
action: keep
|
||||||
- source_labels: ['__meta_docker_container_label_promtail']
|
- source_labels: ['__meta_docker_container_label_promtail']
|
||||||
regex: 'true'
|
regex: 'true'
|
||||||
action: keep
|
action: keep
|
||||||
|
|||||||
@@ -41,11 +41,10 @@
|
|||||||
job: polaris
|
job: polaris
|
||||||
__metrics_path__: /q/metrics
|
__metrics_path__: /q/metrics
|
||||||
|
|
||||||
# Trino — /metrics is the OpenMetrics endpoint (435+); /v1/status is JSON
|
# Trino — /metrics requires auth (Trino rejects unauthenticated scrapes
|
||||||
- targets: ['trino:8080']
|
# even with no auth configured). Skipped until the openmetrics catalog
|
||||||
labels:
|
# is wired with a shared secret. Loki still captures query lifecycle
|
||||||
job: trino
|
# events for the data-lake dashboard.
|
||||||
__metrics_path__: /metrics
|
|
||||||
|
|
||||||
# OTel Collector
|
# OTel Collector
|
||||||
- targets: ['otel-collector:8889']
|
- targets: ['otel-collector:8889']
|
||||||
|
|||||||
Reference in New Issue
Block a user