dab: implement gen_dab.py emitter that produces databricks.yml from registry + config #172

New Issue

kert · 2026-03-24T05:49:22Z

kert commented

2026-03-24 05:49:22 +00:00

Goal

A generator script that reads the pipeline registry, stack.toml [databricks], and table catalog to emit a valid databricks.yml — zero hardcoded pipeline names, task lists, or dependencies.

Why

When a new pipeline is added to aco.pipe, the DAB config should update automatically (via gen_config.py --check in CI). The current databricks.yml is hand-maintained and will drift.

Scope

New module: dev/scripts/backends/databricks.py (follows gitea.py / github.py pattern)
Reads registry dict → generates resources.jobs.stack_pipelines.tasks with correct depends_on from Pipeline.upstream
Reads stack.toml [databricks] → generates bundle, variables, workspace, targets, job_clusters
Reads table catalog → generates Unity Catalog schemas and volumes resources
Output: databricks.yml at repo root
Wire into gen_config.py so uv run python dev/scripts/gen_config.py also generates databricks.yml
--check mode validates databricks.yml is up-to-date

Acceptance

uv run python dev/scripts/gen_config.py produces a databricks.yml identical to current (modulo formatting)
Adding a new pipeline module + running gen_config.py adds the task automatically
databricks bundle validate passes on the generated file

## Goal A generator script that reads the pipeline registry, stack.toml [databricks], and table catalog to emit a valid databricks.yml — zero hardcoded pipeline names, task lists, or dependencies. ## Why When a new pipeline is added to `aco.pipe`, the DAB config should update automatically (via `gen_config.py --check` in CI). The current databricks.yml is hand-maintained and will drift. ## Scope - New module: `dev/scripts/backends/databricks.py` (follows gitea.py / github.py pattern) - Reads `registry` dict → generates `resources.jobs.stack_pipelines.tasks` with correct `depends_on` from `Pipeline.upstream` - Reads `stack.toml [databricks]` → generates `bundle`, `variables`, `workspace`, `targets`, `job_clusters` - Reads table catalog → generates Unity Catalog `schemas` and `volumes` resources - Output: `databricks.yml` at repo root - Wire into `gen_config.py` so `uv run python dev/scripts/gen_config.py` also generates databricks.yml - `--check` mode validates databricks.yml is up-to-date ## Acceptance - [ ] `uv run python dev/scripts/gen_config.py` produces a databricks.yml identical to current (modulo formatting) - [ ] Adding a new pipeline module + running gen_config.py adds the task automatically - [ ] `databricks bundle validate` passes on the generated file

kert added this to the P22: DAB Generator — pluggable databricks.yml from pipeline registry milestone 2026-03-24 05:49:22 +00:00

kert added the databricks pipeline labels 2026-03-24 05:49:22 +00:00

kert referenced this issue from a commit

2026-03-24 05:58:38 +00:00

add DAB generator: databricks.yml from pipeline registry + stack.toml

kert referenced this issue

2026-03-24 05:59:05 +00:00

feat: DAB generator — databricks.yml from pipeline registry + stack.toml #180

kert closed this issue

2026-03-24 05:59:05 +00:00

Sign in to join this conversation.