Create Databricks Asset Bundle (DAB) for all pipelines #18

New Issue

kert · 2026-03-12T17:40:14Z

kert commented

2026-03-12 17:40:14 +00:00

Single DAB project that deploys all 12 pipelines as Databricks jobs:

One multi-task workflow with task per pipeline in dependency order
Use existing make_databricks_loader for Unity Catalog access
Bundle definition in databricks.yml at repo root
Parameterize catalog name via bundle variables
stack.toml [databricks] section for host, token, catalog

Single DAB project that deploys all 12 pipelines as Databricks jobs: - One multi-task workflow with task per pipeline in dependency order - Use existing make_databricks_loader for Unity Catalog access - Bundle definition in databricks.yml at repo root - Parameterize catalog name via bundle variables - stack.toml [databricks] section for host, token, catalog

kert added this to the P5: Databricks — Jobs, Unity Catalog, parity milestone 2026-03-12 17:40:14 +00:00

kert added the databricks label 2026-03-12 17:40:14 +00:00

kert closed this issue

2026-03-12 21:16:19 +00:00

kert referenced this issue from a commit

2026-03-12 23:29:17 +00:00

implement P5 Databricks: DAB bundle, Unity deploy, config targets, parity (fix #18, fix #19, fix #20, fix #21)

kert reopened this issue

2026-03-24 03:43:38 +00:00

kert commented

2026-03-24 20:32:19 +00:00

Completed as part of P22 (DAB Generator — #170–#179).

What exists:

databricks.yml — auto-generated from stack.toml + pipeline registry via dev/scripts/gen_config.py
dev/scripts/backends/databricks.py — 4 plugins: SQLPlugin (transpiles to CTAS), JobsPlugin (13 tasks with deps), SchemasPlugin, VolumesPlugin
stack.toml [databricks] — bundle config, 2 cluster profiles (default/large), 3 targets (dev/staging/prod)
bundle/sql/ — transpiled SQL for all 12 pipeline modules (gitignored, regenerated via --dab-sql)
Schedule: daily 6 AM ET, parameterized catalog per target

This was the last open issue from P5. Closing.

Completed as part of P22 (DAB Generator — #170–#179). **What exists:** - `databricks.yml` — auto-generated from `stack.toml` + pipeline registry via `dev/scripts/gen_config.py` - `dev/scripts/backends/databricks.py` — 4 plugins: SQLPlugin (transpiles to CTAS), JobsPlugin (13 tasks with deps), SchemasPlugin, VolumesPlugin - `stack.toml [databricks]` — bundle config, 2 cluster profiles (default/large), 3 targets (dev/staging/prod) - `bundle/sql/` — transpiled SQL for all 12 pipeline modules (gitignored, regenerated via `--dab-sql`) - Schedule: daily 6 AM ET, parameterized catalog per target This was the last open issue from P5. Closing.

kert closed this issue

2026-03-24 20:32:19 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#18