dab: generate Unity Catalog schema and volume resources from table catalog #173

New Issue

kert · 2026-03-24T05:49:22Z

kert commented

2026-03-24 05:49:22 +00:00

Goal

The DAB generator should emit resources.schemas and resources.volumes blocks derived from aco.lake.catalog.Catalog, so Unity Catalog structure is managed declaratively.

Why

Currently setup_unity_catalog.py creates schemas imperatively via SDK calls. DAB supports schemas and volumes as first-class resources — deploying the bundle should create/update the catalog structure automatically.

Scope

Use Catalog.schemas() to discover all schemas (core, readmissions, pharmacy, etc.)
Generate resources.schemas.<name> for each schema, referencing ${var.catalog}
Generate resources.volumes.staging for parquet upload staging area
Ensure depends_on ordering: schemas deploy before jobs run

Acceptance

Generated databricks.yml includes resources.schemas for all discovered schemas
databricks bundle validate accepts the schema resources
Deploying the bundle creates schemas in Unity Catalog (verified with --dry-run)

## Goal The DAB generator should emit `resources.schemas` and `resources.volumes` blocks derived from `aco.lake.catalog.Catalog`, so Unity Catalog structure is managed declaratively. ## Why Currently `setup_unity_catalog.py` creates schemas imperatively via SDK calls. DAB supports `schemas` and `volumes` as first-class resources — deploying the bundle should create/update the catalog structure automatically. ## Scope - Use `Catalog.schemas()` to discover all schemas (core, readmissions, pharmacy, etc.) - Generate `resources.schemas.<name>` for each schema, referencing `${var.catalog}` - Generate `resources.volumes.staging` for parquet upload staging area - Ensure `depends_on` ordering: schemas deploy before jobs run ## Acceptance - [ ] Generated databricks.yml includes `resources.schemas` for all discovered schemas - [ ] `databricks bundle validate` accepts the schema resources - [ ] Deploying the bundle creates schemas in Unity Catalog (verified with --dry-run)

kert added this to the P22: DAB Generator — pluggable databricks.yml from pipeline registry milestone 2026-03-24 05:49:22 +00:00

kert added the databricks lakehouse labels 2026-03-24 05:49:22 +00:00

kert referenced this issue from a commit

2026-03-24 05:58:38 +00:00

add DAB generator: databricks.yml from pipeline registry + stack.toml

kert referenced this issue

2026-03-24 05:59:05 +00:00

feat: DAB generator — databricks.yml from pipeline registry + stack.toml #180

kert closed this issue

2026-03-24 05:59:05 +00:00

Sign in to join this conversation.