dab: replace sql_task with python_wheel_task in JobsPlugin #223

New Issue

kert · 2026-03-25T03:23:18Z

kert commented

2026-03-25 03:23:18 +00:00

Problem

Current DAB uses sql_task.file.path: bundle/sql/{pipeline}/ which requires 208 transpiled SQL files. PythonWheelTask runs the same narwhals code directly.

Design

tasks:
  - task_key: core
    python_wheel_task:
      package_name: stack
      entry_point: cli
      named_parameters:
        pipeline: core
        catalog: ${var.catalog}
    depends_on:
      - task_key: claims_preprocessing

Tasks

Update JobsPlugin.resources() to emit python_wheel_task instead of sql_task
Remove DDL task generation (schemas created by bundle, not SQL)
Add libraries section referencing the stack wheel
Remove bundle/sql/ from sync includes
Delete bundle/sql/ directory (208 files)
Update tests
Verify databricks.yml structure is valid

## Problem Current DAB uses `sql_task.file.path: bundle/sql/{pipeline}/` which requires 208 transpiled SQL files. PythonWheelTask runs the same narwhals code directly. ## Design ```yaml tasks: - task_key: core python_wheel_task: package_name: stack entry_point: cli named_parameters: pipeline: core catalog: ${var.catalog} depends_on: - task_key: claims_preprocessing ``` ## Tasks - [ ] Update `JobsPlugin.resources()` to emit `python_wheel_task` instead of `sql_task` - [ ] Remove DDL task generation (schemas created by bundle, not SQL) - [ ] Add `libraries` section referencing the stack wheel - [ ] Remove `bundle/sql/` from sync includes - [ ] Delete `bundle/sql/` directory (208 files) - [ ] Update tests - [ ] Verify `databricks.yml` structure is valid

kert added this to the P28: DAB Modernization — Python wheel tasks, databricks-bundles, schema cleanup milestone 2026-03-25 03:23:18 +00:00

kert commented

2026-03-25 03:39:34 +00:00

Complete.

Replaced sql_task with python_wheel_task in JobsPlugin
Each task: package_name=stack, entry_point=cli, named_parameters={run: <pipeline>, --target: spark, --catalog: ${var.catalog}, --save: ""}
Removed DDL task generation
Removed warehouse_id variable (only needed for SQL tasks)
Removed SQLPlugin from _DEFAULT_PLUGINS
Added libraries referencing wheel artifact
Updated tests: 24/24 passed (new tests for python_wheel_task, no_sql_tasks, no_warehouse_id, libraries)

✅ 12,061 total tests pass.

Complete. - Replaced `sql_task` with `python_wheel_task` in `JobsPlugin` - Each task: `package_name=stack, entry_point=cli, named_parameters={run: <pipeline>, --target: spark, --catalog: ${var.catalog}, --save: ""}` - Removed DDL task generation - Removed `warehouse_id` variable (only needed for SQL tasks) - Removed `SQLPlugin` from `_DEFAULT_PLUGINS` - Added `libraries` referencing wheel artifact - Updated tests: 24/24 passed (new tests for python_wheel_task, no_sql_tasks, no_warehouse_id, libraries) ✅ 12,061 total tests pass.

kert closed this issue

2026-03-25 03:39:34 +00:00

kert referenced this issue from a commit

2026-03-25 03:41:40 +00:00

feat: modernize DAB — Python wheel tasks, schema cleanup, dead config removal

kert referenced this issue

2026-03-25 04:00:41 +00:00

dab: delete bundle/sql/ — no longer deployment artifacts #228

kert referenced this issue

2026-03-25 04:04:37 +00:00

dab: delete bundle/sql/ — no longer deployment artifacts #228

Sign in to join this conversation.