homelab/stack
1
0
Fork 0
Code Issues 70 Pull Requests 1 Actions Packages Projects 1 Releases Wiki Activity

infra: Grafana pipeline performance dashboard #212

New Issue
Closed
opened 2026-03-24 23:50:07 +00:00 by kert · 2 comments
kert commented 2026-03-24 23:50:07 +00:00
Owner
Copy Link

Ship a provisioned dashboard in infra/grafana/dashboards/pipeline-performance.json.

Panels

  • Pipeline run duration over time (Prometheus histogram)
  • Step-by-step duration heatmap (per-pipeline breakdown)
  • Cache hit rate gauge (hits / total steps)
  • Memory RSS over time (psutil gauge)
  • CPU utilization over time (psutil gauge)
  • Row throughput per step (rows/sec)
  • Trace waterfall link (Jaeger deep link from pipeline run)
  • Recent pipeline runs table (last 20 runs with status, duration, step count)
  • Log panel (Loki query filtered by trace_id)

Tasks

  • Build dashboard JSON with variables: $pipeline, $timerange
  • Add to Grafana provisioning so it auto-loads
  • Test: dashboard loads with sample data from a pipeline run
Ship a provisioned dashboard in `infra/grafana/dashboards/pipeline-performance.json`. ## Panels - [ ] Pipeline run duration over time (Prometheus histogram) - [ ] Step-by-step duration heatmap (per-pipeline breakdown) - [ ] Cache hit rate gauge (hits / total steps) - [ ] Memory RSS over time (psutil gauge) - [ ] CPU utilization over time (psutil gauge) - [ ] Row throughput per step (rows/sec) - [ ] Trace waterfall link (Jaeger deep link from pipeline run) - [ ] Recent pipeline runs table (last 20 runs with status, duration, step count) - [ ] Log panel (Loki query filtered by trace_id) ## Tasks - [ ] Build dashboard JSON with variables: `$pipeline`, `$timerange` - [ ] Add to Grafana provisioning so it auto-loads - [ ] Test: dashboard loads with sample data from a pipeline run
kert added this to the P26: Pipeline Telemetry — OTel instrumentation, perf module, Grafana dashboards milestone 2026-03-24 23:50:07 +00:00
kert commented 2026-03-25 00:13:51 +00:00
Author
Owner
Copy Link

Starting. Creating infra/grafana/dashboards/pipeline-performance.json with panels for:

  • Pipeline run duration over time
  • Step duration heatmap
  • Cache hit rate
  • RSS + CPU gauges
  • Row throughput
  • Trace waterfall link
  • Recent runs table
  • Log panel (Loki)
Starting. Creating `infra/grafana/dashboards/pipeline-performance.json` with panels for: - Pipeline run duration over time - Step duration heatmap - Cache hit rate - RSS + CPU gauges - Row throughput - Trace waterfall link - Recent runs table - Log panel (Loki)
kert commented 2026-03-25 00:19:16 +00:00
Author
Owner
Copy Link

Complete.

Created: infra/grafana/dashboards/pipeline-performance.json

8 panels:

  • Pipeline Run Duration (p50/p95 histograms)
  • Step Duration Heatmap
  • Cache Hit Rate gauge (red→yellow→green thresholds)
  • Memory RSS timeseries
  • CPU Utilization timeseries
  • Row Throughput per Step (bar gauge)
  • Recent Pipeline Traces (Jaeger search table)
  • Pipeline Logs (Loki, filtered by trace_id)

Variables: $pipeline (multi-select, auto-populated from Prometheus labels)
Datasources: Prometheus, Jaeger, Loki (all 3 already provisioned in Grafana)
Auto-loads via existing dashboard provisioner (watches /var/lib/grafana/dashboards)

Tests: 8 in test_dashboard.py

  • File exists, valid JSON, required keys, UID, panel count+names, variable, datasources, tags

✅ 8/8 passed. Full suite: 12,026 passed, zero failures.

Complete. **Created:** `infra/grafana/dashboards/pipeline-performance.json` **8 panels:** - Pipeline Run Duration (p50/p95 histograms) - Step Duration Heatmap - Cache Hit Rate gauge (red→yellow→green thresholds) - Memory RSS timeseries - CPU Utilization timeseries - Row Throughput per Step (bar gauge) - Recent Pipeline Traces (Jaeger search table) - Pipeline Logs (Loki, filtered by trace_id) **Variables:** `$pipeline` (multi-select, auto-populated from Prometheus labels) **Datasources:** Prometheus, Jaeger, Loki (all 3 already provisioned in Grafana) **Auto-loads** via existing dashboard provisioner (watches `/var/lib/grafana/dashboards`) **Tests:** 8 in `test_dashboard.py` - File exists, valid JSON, required keys, UID, panel count+names, variable, datasources, tags ✅ 8/8 passed. Full suite: **12,026 passed**, zero failures.
kert closed this issue 2026-03-25 00:19:16 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
api
REST API (FastAPI)
 auth
Authentication and authorization
 bib
Bibliography, Zotero, citations
 ci
CI/CD pipelines and Woodpecker plugins
 cli
CLI commands and UX
 cloud
Cloud provider abstraction (AWS, GCP, Azure, self-hosted)
 config
Configuration and wiring
 data
Data loading, CCLF, BCDA, seeds
 databricks
Databricks jobs and deployment
 docs
Documentation and generated references
 image:api
image:docs
image:mc
image:notebooks
image:zotero
infra
lakehouse
Nessie, Polaris, Trino, Iceberg
 llm
llm module — local RAG over comments + Zotero corpus
 notebooks
marimo notebook errors (auto-filed by nb-watcher/nb-integration)
 packaging
Python packaging, optional deps, build system
 pfs
pipeline
Pipeline execution and validation
 pipeline:ci
pipeline:deploy
pipeline:harden
pipeline:infra-ci
pipeline:rebuild-all
pipeline:release
qpp
quality
Testing, coverage, error handling
 step:build
step:deploy
step:hadolint
step:lint
step:publish
step:scan
step:test
step:validate
styles
Charts, figures, visual theming
 tree
Project directory structure and file organization
No Label
Milestone
No items
No Milestone P26: Pipeline Telemetry — OTel instrumentation, perf module, Grafana dashboards
Projects
Clear projects
No project Skin Subs
Assignees
Clear assignees
kert renovate-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: homelab/stack#212
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?