dab: add [databricks] config section to stack.toml for bundle generation #171

New Issue

kert · 2026-03-24T05:49:22Z

kert commented

2026-03-24 05:49:22 +00:00

Goal

Centralise all DAB generation parameters in stack.toml so the generator reads from one source.

Why

Currently databricks.yml hardcodes cluster specs, spark version, schedule, and target catalogs. These should live in stack.toml alongside the rest of the platform config, following the same pattern as [ci], [images], [mirrors].

Scope

Add to stack.toml:

[databricks]
bundle_name = "stack"
package_name = "stack"
entry_point = "cli"
schedule = "0 0 6 * * ?"
timezone = "America/New_York"

[databricks.cluster]
spark_version = "15.4.x-scala2.12"
node_type_id = "i3.xlarge"
num_workers = 1

[databricks.targets.dev]
mode = "development"
default = true
catalog = "aco_dev"

[databricks.targets.staging]
catalog = "aco_staging"

[databricks.targets.prod]
mode = "production"
catalog = "aco"

Acceptance

cfg.databricks.bundle_name returns "stack"
cfg.databricks.targets is iterable with dev/staging/prod
No DAB-specific values remain hardcoded outside stack.toml

## Goal Centralise all DAB generation parameters in stack.toml so the generator reads from one source. ## Why Currently databricks.yml hardcodes cluster specs, spark version, schedule, and target catalogs. These should live in stack.toml alongside the rest of the platform config, following the same pattern as [ci], [images], [mirrors]. ## Scope Add to stack.toml: ```toml [databricks] bundle_name = "stack" package_name = "stack" entry_point = "cli" schedule = "0 0 6 * * ?" timezone = "America/New_York" [databricks.cluster] spark_version = "15.4.x-scala2.12" node_type_id = "i3.xlarge" num_workers = 1 [databricks.targets.dev] mode = "development" default = true catalog = "aco_dev" [databricks.targets.staging] catalog = "aco_staging" [databricks.targets.prod] mode = "production" catalog = "aco" ``` ## Acceptance - [ ] `cfg.databricks.bundle_name` returns `"stack"` - [ ] `cfg.databricks.targets` is iterable with dev/staging/prod - [ ] No DAB-specific values remain hardcoded outside stack.toml

kert added this to the P22: DAB Generator — pluggable databricks.yml from pipeline registry milestone 2026-03-24 05:49:22 +00:00

kert added the config label 2026-03-24 05:49:22 +00:00

kert referenced this issue from a commit

2026-03-24 05:58:38 +00:00

add DAB generator: databricks.yml from pipeline registry + stack.toml

kert referenced this issue

2026-03-24 05:59:05 +00:00

feat: DAB generator — databricks.yml from pipeline registry + stack.toml #180

kert closed this issue

2026-03-24 05:59:05 +00:00

Sign in to join this conversation.