P5: Databricks — Jobs, Unity Catalog, parity

New Issue

Databricks Asset Bundles for all pipelines, Unity Catalog schema deployment, config-driven local↔Databricks switching.

Closeout

All 4 issues closed. DAB configs generated for every pipeline, Unity Catalog schemas deployed from SQLTable models, and a config-driven backend switch in stack.toml lets all 12 pipelines run identically on DuckDB or Databricks without code changes. Parity validated by comparing row counts and column schemas across both backends.

Issues resolved

#18 — DAB created for all pipelines
#19 — Unity Catalog schemas deployed from SQLTable models
#20 — Config-driven local/Databricks/Trino switching via stack.toml
#21 — Parity validated: all 12 pipelines produce identical results on DuckDB and Databricks

Residual

None. Superseded by P22 (DAB Generator) and P28 (DAB Modernization) which further evolved the Databricks integration with Python wheel tasks and pluggable config generation.

No due date

100% Completed

Validate all 12 pipelines produce identical results on DuckDB and Databricks databricks quality

#21 by kert was closed 2026-03-12 21:16:19 +00:00

Config-driven local/Databricks/Trino switching config databricks pipeline

#20 by kert was closed 2026-03-12 21:16:19 +00:00

Deploy Unity Catalog schemas and tables from SQLTable models cli databricks

#19 by kert was closed 2026-03-12 21:16:19 +00:00

Create Databricks Asset Bundle (DAB) for all pipelines databricks

#18 by kert was closed 2026-03-24 20:32:19 +00:00