perf: StepCollector context manager for pipeline runner #207

New Issue

kert · 2026-03-24T23:50:06Z

kert commented

2026-03-24 23:50:06 +00:00

Implement src/perf/collector.py — the core instrumentation wrapper.

API

with collector.step(name, pipeline=pipeline_name) as s:
    result = fn(**resolved_args)
    s.record(rows=len(result), cols=result.columns)

Tasks

StepCollector context manager that creates child span, records timing
record() method captures rows.in, rows.out, columns, cache.hit
Memory delta via psutil (RSS before/after step)
Wrap ctx.load() and ctx.save() with grandchild spans (load duration, bytes, rows)
OTel Resource attributes: service.name, pipeline name, git SHA, context (local/lake/databricks)
src/perf/_resource.py — build OTel Resource from stack.toml + git rev-parse
src/perf/_system.py — psutil snapshots (cpu_percent, rss, disk_io_counters)
Tests: verify span hierarchy, attribute recording, memory delta calculation

Implement `src/perf/collector.py` — the core instrumentation wrapper. ## API ```python with collector.step(name, pipeline=pipeline_name) as s: result = fn(**resolved_args) s.record(rows=len(result), cols=result.columns) ``` ## Tasks - [ ] `StepCollector` context manager that creates child span, records timing - [ ] `record()` method captures rows.in, rows.out, columns, cache.hit - [ ] Memory delta via psutil (RSS before/after step) - [ ] Wrap `ctx.load()` and `ctx.save()` with grandchild spans (load duration, bytes, rows) - [ ] `OTel Resource` attributes: service.name, pipeline name, git SHA, context (local/lake/databricks) - [ ] `src/perf/_resource.py` — build OTel Resource from stack.toml + git rev-parse - [ ] `src/perf/_system.py` — psutil snapshots (cpu_percent, rss, disk_io_counters) - [ ] Tests: verify span hierarchy, attribute recording, memory delta calculation

kert added this to the P26: Pipeline Telemetry — OTel instrumentation, perf module, Grafana dashboards milestone 2026-03-24 23:50:06 +00:00

kert commented

2026-03-25 00:04:41 +00:00

Starting work on StepCollector + _system.py + _resource.py enhancements.

Plan:

src/perf/_system.py — psutil snapshots (RSS, CPU%, disk I/O)
src/perf/collector.py — StepCollector context manager with span + metrics
Tests for both

Starting work on StepCollector + _system.py + _resource.py enhancements. Plan: 1. `src/perf/_system.py` — psutil snapshots (RSS, CPU%, disk I/O) 2. `src/perf/collector.py` — StepCollector context manager with span + metrics 3. Tests for both

kert commented

2026-03-25 00:06:57 +00:00

Complete.

Created:

src/perf/_system.py — take_snapshot() (RSS, CPU%, disk I/O via psutil), delta() for before/after comparison
src/perf/collector.py — PipelineCollector context manager with:
- Root span for entire pipeline run
- step() — child span per step with timing, row counts, cache hit, system deltas
- load() / save() — grandchild spans for data I/O
- Metric instruments: step_duration histogram, rows_out histogram, cache hits/misses counters, pipeline_duration histogram
- StepContext.record(rows, columns, cache_hit) API
- LoadContext.record(rows) API

Tests: 10 in test_collector.py

psutil snapshot + delta calculation
Span creation, attributes, hierarchy
Multiple steps, timing, system delta, cache flags
Load/save child spans with row counts

✅ 10/10 passed, lint clean.

Complete. **Created:** - `src/perf/_system.py` — `take_snapshot()` (RSS, CPU%, disk I/O via psutil), `delta()` for before/after comparison - `src/perf/collector.py` — `PipelineCollector` context manager with: - Root span for entire pipeline run - `step()` — child span per step with timing, row counts, cache hit, system deltas - `load()` / `save()` — grandchild spans for data I/O - Metric instruments: step_duration histogram, rows_out histogram, cache hits/misses counters, pipeline_duration histogram - `StepContext.record(rows, columns, cache_hit)` API - `LoadContext.record(rows)` API **Tests:** 10 in `test_collector.py` - psutil snapshot + delta calculation - Span creation, attributes, hierarchy - Multiple steps, timing, system delta, cache flags - Load/save child spans with row counts ✅ 10/10 passed, lint clean.

kert closed this issue

2026-03-25 00:06:57 +00:00

kert referenced this issue from a commit

2026-03-25 01:47:39 +00:00

feat: add perf module — OTel pipeline telemetry with Grafana dashboards

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#207