dab: final validation — databricks bundle validate, tests, cleanup #226

New Issue

kert · 2026-03-25T03:23:18Z

kert commented

2026-03-25 03:23:18 +00:00

Tasks

Run databricks bundle validate equivalent tests on generated config
Verify all 13 pipeline tasks have correct dependencies
Verify 13 schemas (not 34) in resources
Verify no bundle/sql/ references remain
Verify stack.toml [databricks] is clean (no dead config)
Full test suite passes
Commit and push

## Tasks - [ ] Run `databricks bundle validate` equivalent tests on generated config - [ ] Verify all 13 pipeline tasks have correct dependencies - [ ] Verify 13 schemas (not 34) in resources - [ ] Verify no `bundle/sql/` references remain - [ ] Verify `stack.toml [databricks]` is clean (no dead config) - [ ] Full test suite passes - [ ] Commit and push

kert added this to the P28: DAB Modernization — Python wheel tasks, databricks-bundles, schema cleanup milestone 2026-03-25 03:23:18 +00:00

kert commented

2026-03-25 03:39:57 +00:00

Complete.

Validation results:

13 pipeline tasks with correct python_wheel_task and dependencies ✅
13 schemas (not 34) — legacy schemas excluded ✅
No bundle/sql/ references in tasks ✅
No warehouse_id variable ✅
No dead large cluster config ✅
stack.toml [databricks] clean ✅
24 DAB tests + 112 transpile tests pass ✅
12,061 total tests pass ✅

#224 (databricks-bundles Python resources) deferred — the current generator works correctly with python_wheel_task. The YAML→Python resource migration is an optimization, not a fix.

Complete. Validation results: - 13 pipeline tasks with correct `python_wheel_task` and dependencies ✅ - 13 schemas (not 34) — legacy schemas excluded ✅ - No `bundle/sql/` references in tasks ✅ - No `warehouse_id` variable ✅ - No dead `large` cluster config ✅ - `stack.toml [databricks]` clean ✅ - 24 DAB tests + 112 transpile tests pass ✅ - 12,061 total tests pass ✅ #224 (databricks-bundles Python resources) deferred — the current generator works correctly with python_wheel_task. The YAML→Python resource migration is an optimization, not a fix.

kert closed this issue

2026-03-25 03:39:58 +00:00

kert referenced this issue from a commit

2026-03-25 03:41:40 +00:00

feat: modernize DAB — Python wheel tasks, schema cleanup, dead config removal

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#226