dab: add databricks-bundles package and Python resource loader #224

New Issue

kert · 2026-03-25T03:23:18Z

kert commented

2026-03-25 03:23:18 +00:00

Problem

The 375-line OrderedDict YAML generator is brittle. Modern DAB supports Python-defined resources via databricks-bundles package.

Design

# bundle/resources/__init__.py
from databricks.bundles.jobs import Job, Task, PythonWheelTask
from aco.pipe import registry

def load_resources():
    schemas = derive_from_registry(registry)
    tasks = [Task(task_key=name, python_wheel_task=...) for name, pipe in registry.items()]
    job = Job(name="stack-pipelines", tasks=tasks, schedule=...)
    return schemas + [Volume(...), job]

Tasks

Add databricks-bundles>=0.295.0 to [project.optional-dependencies] lake
Create bundle/resources/__init__.py with load_resources()
Shrink databricks.yml to ~20 lines (bundle name, targets, python entry point)
Rewrite dev/scripts/backends/databricks.py to use load_resources() for validation
Update tests to validate typed Python resources instead of YAML parsing
Remove legacy databricks-cli>=0.18.0 from deps

## Problem The 375-line OrderedDict YAML generator is brittle. Modern DAB supports Python-defined resources via `databricks-bundles` package. ## Design ```python # bundle/resources/__init__.py from databricks.bundles.jobs import Job, Task, PythonWheelTask from aco.pipe import registry def load_resources(): schemas = derive_from_registry(registry) tasks = [Task(task_key=name, python_wheel_task=...) for name, pipe in registry.items()] job = Job(name="stack-pipelines", tasks=tasks, schedule=...) return schemas + [Volume(...), job] ``` ## Tasks - [ ] Add `databricks-bundles>=0.295.0` to `[project.optional-dependencies] lake` - [ ] Create `bundle/resources/__init__.py` with `load_resources()` - [ ] Shrink `databricks.yml` to ~20 lines (bundle name, targets, python entry point) - [ ] Rewrite `dev/scripts/backends/databricks.py` to use `load_resources()` for validation - [ ] Update tests to validate typed Python resources instead of YAML parsing - [ ] Remove legacy `databricks-cli>=0.18.0` from deps

kert added this to the P28: DAB Modernization — Python wheel tasks, databricks-bundles, schema cleanup milestone 2026-03-25 03:23:18 +00:00

kert referenced this issue

2026-03-25 03:39:57 +00:00

dab: final validation — databricks bundle validate, tests, cleanup #226

kert commented

2026-03-25 04:19:42 +00:00

Starting. Plan:

Check if databricks-bundles is installable
Create bundle/resources/init.py with load_resources()
Shrink databricks.yml to use Python entry point
Rewrite databricks.py generator to validate against typed resources
Update pyproject.toml deps
Update tests
Verify CI passes before closing

Starting. Plan: 1. Check if databricks-bundles is installable 2. Create bundle/resources/__init__.py with load_resources() 3. Shrink databricks.yml to use Python entry point 4. Rewrite databricks.py generator to validate against typed resources 5. Update pyproject.toml deps 6. Update tests 7. Verify CI passes before closing

kert referenced this issue from a commit

2026-03-25 04:30:40 +00:00

feat: databricks-bundles Python resource loader — typed DAB resources

kert commented

2026-03-25 04:39:58 +00:00

Complete. CI #105: 13/13 passed. Deploy #106: passed.

Created:

bundle/resources/__init__.py — load_resources() returns typed Resources with add_job, add_schema, add_volume
Uses Variable(path="catalog", type=str) for catalog parameterization
13 PythonWheelTask tasks, 13 schemas, 1 volume

Modified:

databricks.yml — 23 lines (was 278). Delegates to python.resources: [resources:load_resources]
pyproject.toml — replaced databricks-cli>=0.18.0 with databricks-bundles>=0.295.0
databricks.py generator — emit() now produces minimal YAML, SQL only via sql=True flag
gen_config.py — passes sql flag through

Tests: 30 DAB tests (12 new Python resource tests)

✅ CI verified. Deploy verified.

Complete. CI #105: 13/13 passed. Deploy #106: passed. **Created:** - `bundle/resources/__init__.py` — `load_resources()` returns typed `Resources` with `add_job`, `add_schema`, `add_volume` - Uses `Variable(path="catalog", type=str)` for catalog parameterization - 13 `PythonWheelTask` tasks, 13 schemas, 1 volume **Modified:** - `databricks.yml` — 23 lines (was 278). Delegates to `python.resources: [resources:load_resources]` - `pyproject.toml` — replaced `databricks-cli>=0.18.0` with `databricks-bundles>=0.295.0` - `databricks.py` generator — `emit()` now produces minimal YAML, SQL only via `sql=True` flag - `gen_config.py` — passes `sql` flag through **Tests:** 30 DAB tests (12 new Python resource tests) ✅ CI verified. Deploy verified.

kert closed this issue

2026-03-25 04:39:58 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#224