feat: DAB generates transpiled SQL + DDL instead of wheel wrapper #181

kert · 2026-03-24T06:20:14Z

kert commented

2026-03-24 06:20:14 +00:00

Summary

Replaces the python_wheel_task wrapper with actual transpiled Databricks SQL.

SQLPlugin: transpiles all 208 pipeline expressions to native Databricks SQL via aco.lake.transpile (CTAS mode with ${var.catalog} variable)
DDLPlugin: generates 1,364 CREATE TABLE IF NOT EXISTS DDL files from aco.table Pydantic models
Jobs: tasks use sql_task with file paths referencing bundle/sql/<pipeline>/
gen_config.py: --dab-sql flag for full SQL generation (~4 min); --check only validates databricks.yml (12s, CI-safe)
SQL/DDL files in bundle/sql/ and bundle/ddl/ (gitignored, generated on demand)

What Databricks executes

Native SQL — no Python wheel, no CLI wrapper. Each pipeline step is a CREATE OR REPLACE TABLE statement transpiled from the narwhals expression functions.

Test plan

26 DAB-specific tests pass (SQL generation, DDL, completeness, idempotency)
11,957 total tests pass
gen_config.py --check passes in 12s

## Summary Replaces the python_wheel_task wrapper with actual transpiled Databricks SQL. - **SQLPlugin**: transpiles all 208 pipeline expressions to native Databricks SQL via `aco.lake.transpile` (CTAS mode with `${var.catalog}` variable) - **DDLPlugin**: generates 1,364 `CREATE TABLE IF NOT EXISTS` DDL files from `aco.table` Pydantic models - **Jobs**: tasks use `sql_task` with file paths referencing `bundle/sql/<pipeline>/` - **gen_config.py**: `--dab-sql` flag for full SQL generation (~4 min); `--check` only validates `databricks.yml` (12s, CI-safe) - SQL/DDL files in `bundle/sql/` and `bundle/ddl/` (gitignored, generated on demand) ## What Databricks executes Native SQL — no Python wheel, no CLI wrapper. Each pipeline step is a `CREATE OR REPLACE TABLE` statement transpiled from the narwhals expression functions. ## Test plan - [x] 26 DAB-specific tests pass (SQL generation, DDL, completeness, idempotency) - [x] 11,957 total tests pass - [x] `gen_config.py --check` passes in 12s

kert added this to the P22: DAB Generator — pluggable databricks.yml from pipeline registry milestone 2026-03-24 06:20:14 +00:00

kert added 1 commit 2026-03-24 06:20:14 +00:00

generate transpiled SQL + DDL for DAB instead of wheel wrapper tasks

CI / lint-test (push) Successful in 6m46s

Details

CI / lint-test (pull_request) Successful in 6m46s

Details

6b02a804c8

- SQLPlugin: transpiles 208 pipeline expressions to Databricks SQL
  via aco.lake.transpile (CTAS mode with catalog variable)
- DDLPlugin: generates 1,364 CREATE TABLE DDL from SQLTable models
- Jobs use sql_task with file paths instead of python_wheel_task
- SQL/DDL written to bundle/sql/ and bundle/ddl/ (gitignored)
- gen_config.py: --dab-sql flag for full SQL generation (~4 min)
  --check only validates databricks.yml (12s, CI-safe)
- 26 tests covering SQL generation, DDL, completeness, idempotency

kert merged commit c5d0f89c52 into main

2026-03-24 06:21:29 +00:00

kert referenced this issue from a commit

2026-03-24 06:21:30 +00:00

Merge pull request 'feat: DAB generates transpiled SQL + DDL instead of wheel wrapper' (#181) from feat/dab-generator into main

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#181