dab: enrich Pipeline model with dependency and cluster metadata #170

New Issue

kert · 2026-03-24T05:49:22Z

kert commented

2026-03-24 05:49:22 +00:00

Goal

Extend aco.pipe.base.Pipeline so each pipeline declares its own upstream dependencies and compute requirements as structured metadata — not inferred from function signatures.

Why

The current databricks.yml hardcodes depends_on per task. The generator needs this from the pipeline itself. Function-signature inference is fragile (cross-schema refs like terminology are ambiguous). Each pipeline should declare:

upstream: list of pipeline names it depends on (for DAB depends_on)
cluster_profile: compute tier hint (e.g. default, large, gpu) so the generator can map to cluster configs

Scope

Add upstream: list[str] = [] and cluster_profile: str = "default" to Pipeline
Populate upstream in each of the 13 pipeline modules using the known dependency graph
Validate: no cycles, all referenced upstreams exist in registry
Tests: verify the DAG is a valid topological order

Acceptance

Pipeline.upstream is populated for all 13 pipelines
from aco.pipe import registry; assert all(p.upstream is not None for p in registry.values())
Topological sort produces a valid execution order

## Goal Extend `aco.pipe.base.Pipeline` so each pipeline declares its own upstream dependencies and compute requirements as structured metadata — not inferred from function signatures. ## Why The current databricks.yml hardcodes `depends_on` per task. The generator needs this from the pipeline itself. Function-signature inference is fragile (cross-schema refs like `terminology` are ambiguous). Each pipeline should declare: - `upstream`: list of pipeline names it depends on (for DAB `depends_on`) - `cluster_profile`: compute tier hint (e.g. `default`, `large`, `gpu`) so the generator can map to cluster configs ## Scope - Add `upstream: list[str] = []` and `cluster_profile: str = "default"` to `Pipeline` - Populate `upstream` in each of the 13 pipeline modules using the known dependency graph - Validate: no cycles, all referenced upstreams exist in registry - Tests: verify the DAG is a valid topological order ## Acceptance - [ ] `Pipeline.upstream` is populated for all 13 pipelines - [ ] `from aco.pipe import registry; assert all(p.upstream is not None for p in registry.values())` - [ ] Topological sort produces a valid execution order

kert added this to the P22: DAB Generator — pluggable databricks.yml from pipeline registry milestone 2026-03-24 05:49:22 +00:00

kert added the config pipeline labels 2026-03-24 05:49:22 +00:00

kert referenced this issue from a commit

2026-03-24 05:58:38 +00:00

add DAB generator: databricks.yml from pipeline registry + stack.toml

kert referenced a pull request that will close this issue

2026-03-24 05:59:05 +00:00

feat: DAB generator — databricks.yml from pipeline registry + stack.toml #180

kert closed this issue

2026-03-24 05:59:05 +00:00

Sign in to join this conversation.