feat: DAB generator — databricks.yml from pipeline registry + stack.toml #180

Merged

kert merged 1 commits from feat/dab-generator into main 2026-03-24 06:03:24 +00:00

Conversation 0 Commits 1 Files Changed 19 +967 -150

kert commented 2026-03-24 05:59:05 +00:00

Owner

Copy Link

Summary

Generates databricks.yml entirely from the pipeline registry and stack.toml — zero hardcoded pipeline names, task lists, or dependencies.

• Pipeline metadata: upstream and cluster_profile fields on all 13 Pipeline models
• Config: [databricks] section in stack.toml with clusters, targets, schedule
• Core emitter: backends/databricks.py with DabPlugin protocol
• Resources: 13 job tasks + 30 Unity Catalog schemas + staging volume + wheel artifact
• CI gate: gen_config.py --check catches stale databricks.yml
• Tests: 23 tests covering round-trip, completeness, deps, idempotency, extensibility

How it works

Add a new pipeline module to aco.pipe → run gen_config.py → task appears in databricks.yml with correct depends_on. No file to hand-edit.

Plugin architecture

New DAB resource types (quality monitors, dashboards, DLT) can be added by implementing DabPlugin.resources() — no changes to the core generator.

Test plan

• 23 DAB-specific tests pass
• All 11,954 tests pass (23 new)
• gen_config.py --check passes
• Generated databricks.yml matches expected structure
• Idempotent: two runs produce identical output

Closes #170, #171, #172, #173, #174, #175, #176, #177, #178, #179

## Summary Generates `databricks.yml` entirely from the pipeline registry and `stack.toml` — zero hardcoded pipeline names, task lists, or dependencies. - **Pipeline metadata**: `upstream` and `cluster_profile` fields on all 13 `Pipeline` models - **Config**: `[databricks]` section in `stack.toml` with clusters, targets, schedule - **Core emitter**: `backends/databricks.py` with `DabPlugin` protocol - **Resources**: 13 job tasks + 30 Unity Catalog schemas + staging volume + wheel artifact - **CI gate**: `gen_config.py --check` catches stale databricks.yml - **Tests**: 23 tests covering round-trip, completeness, deps, idempotency, extensibility ### How it works Add a new pipeline module to `aco.pipe` → run `gen_config.py` → task appears in databricks.yml with correct `depends_on`. No file to hand-edit. ### Plugin architecture New DAB resource types (quality monitors, dashboards, DLT) can be added by implementing `DabPlugin.resources()` — no changes to the core generator. ## Test plan - [x] 23 DAB-specific tests pass - [x] All 11,954 tests pass (23 new) - [x] gen_config.py --check passes - [x] Generated databricks.yml matches expected structure - [x] Idempotent: two runs produce identical output Closes #170, #171, #172, #173, #174, #175, #176, #177, #178, #179

kert added this to the P22: DAB Generator — pluggable databricks.yml from pipeline registry milestone 2026-03-24 05:59:05 +00:00

kert added 1 commit 2026-03-24 05:59:05 +00:00

add DAB generator: databricks.yml from pipeline registry + stack.toml ... f2e9dabadb

closes #170, closes #171, closes #172, closes #173, closes #174,
closes #175, closes #176, closes #177, closes #178, closes #179

Pipeline metadata (#170):
- Add upstream and cluster_profile to Pipeline model
- Populate upstream for all 13 pipeline modules

Config (#171):
- Add [databricks] section to stack.toml with clusters, targets

Core emitter (#172):
- backends/databricks.py generates databricks.yml from registry
- All 13 tasks with correct depends_on, no hardcoding
- Wired into gen_config.py (generate + --check)

Resources (#173, #174):
- 30 Unity Catalog schemas from aco.lake.catalog.Catalog
- Staging volume for data upload
- Cluster profiles: only referenced profiles emitted

Artifacts (#175): wheel build via uv build
Permissions (#176): production targets support run_as
CI gate (#177): gen_config.py --check covers databricks.yml
Plugin architecture (#178): DabPlugin protocol, 3 built-in plugins
Tests (#179): 23 tests covering round-trip, completeness, deps, idempotency

kert merged commit fbede316d9 into main 2026-03-24 06:03:24 +00:00

kert referenced this issue from a commit 2026-03-24 06:03:25 +00:00

Merge pull request 'feat: DAB generator — databricks.yml from pipeline registry + stack.toml' (#180) from feat/dab-generator into main

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

api

REST API (FastAPI)

auth

Authentication and authorization

bib

Bibliography, Zotero, citations

ci

CI/CD pipelines and Woodpecker plugins

cli

CLI commands and UX

cloud

Cloud provider abstraction (AWS, GCP, Azure, self-hosted)

config

Configuration and wiring

data

Data loading, CCLF, BCDA, seeds

databricks

Databricks jobs and deployment

docs

Documentation and generated references

image:api

image:docs

image:mc

image:notebooks

image:zotero

infra

lakehouse

Nessie, Polaris, Trino, Iceberg

llm

llm module — local RAG over comments + Zotero corpus

notebooks

marimo notebook errors (auto-filed by nb-watcher/nb-integration)

packaging

Python packaging, optional deps, build system

pfs

pipeline

Pipeline execution and validation

pipeline:ci

pipeline:deploy

pipeline:harden

pipeline:infra-ci

pipeline:rebuild-all

pipeline:release

qpp

quality

Testing, coverage, error handling

step:build

step:deploy

step:hadolint

step:lint

step:publish

step:scan

step:test

step:validate

styles

Charts, figures, visual theming

tree

Project directory structure and file organization

No Label

Milestone

No items

No Milestone P22: DAB Generator — pluggable databricks.yml from pipeline registry

Projects

Clear projects

No project

Assignees

Clear assignees

kert renovate-bot

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#180