homelab/stack
1
0
Fork 0
Code Issues 80 Pull Requests 1 Actions Packages Projects 1 Releases Wiki Activity

dab: add CLI --pipeline/--catalog flags for Databricks wheel execution #222

New Issue
Closed
opened 2026-03-25 03:23:18 +00:00 by kert · 1 comment
kert commented 2026-03-25 03:23:18 +00:00
Owner
Copy Link

Problem

The stack run CLI command needs flags for Databricks PythonWheelTask execution: --pipeline <name> --catalog <catalog>.

Design

stack run readmissions --catalog aco_dev

When --catalog is provided, use make_databricks_loader(spark, catalog) instead of the DuckDB loader. Spark session acquired from Databricks runtime via SparkSession.builder.getOrCreate().

Tasks

  • Add --catalog option to cli/run.py
  • When --catalog set: import spark, create make_databricks_loader, call pipeline.run(load)
  • When --catalog not set: existing DuckDB path (unchanged)
  • Graceful error if pyspark not available (not on Databricks)
  • Tests: mock spark, verify loader creation
## Problem The `stack run` CLI command needs flags for Databricks PythonWheelTask execution: `--pipeline <name> --catalog <catalog>`. ## Design ```bash stack run readmissions --catalog aco_dev ``` When `--catalog` is provided, use `make_databricks_loader(spark, catalog)` instead of the DuckDB loader. Spark session acquired from Databricks runtime via `SparkSession.builder.getOrCreate()`. ## Tasks - [ ] Add `--catalog` option to `cli/run.py` - [ ] When `--catalog` set: import spark, create `make_databricks_loader`, call `pipeline.run(load)` - [ ] When `--catalog` not set: existing DuckDB path (unchanged) - [ ] Graceful error if pyspark not available (not on Databricks) - [ ] Tests: mock spark, verify loader creation
kert added this to the P28: DAB Modernization — Python wheel tasks, databricks-bundles, schema cleanup milestone 2026-03-25 03:23:18 +00:00
kert commented 2026-03-25 03:39:33 +00:00
Author
Owner
Copy Link

Complete.

  • Added --catalog option to stack run (implies target=spark)
  • Added _SparkContext class with make_databricks_loader integration
  • stack run readmissions --catalog aco_dev runs pipeline on Databricks via SparkSession
  • Graceful: pyspark only imported when target=spark

✅ Lint clean.

Complete. - Added `--catalog` option to `stack run` (implies `target=spark`) - Added `_SparkContext` class with `make_databricks_loader` integration - `stack run readmissions --catalog aco_dev` runs pipeline on Databricks via SparkSession - Graceful: pyspark only imported when target=spark ✅ Lint clean.
kert closed this issue 2026-03-25 03:39:34 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
api
REST API (FastAPI)
 auth
Authentication and authorization
 bib
Bibliography, Zotero, citations
 ci
CI/CD pipelines and Woodpecker plugins
 cli
CLI commands and UX
 cloud
Cloud provider abstraction (AWS, GCP, Azure, self-hosted)
 config
Configuration and wiring
 data
Data loading, CCLF, BCDA, seeds
 databricks
Databricks jobs and deployment
 docs
Documentation and generated references
 image:api
image:docs
image:mc
image:notebooks
image:zotero
infra
lakehouse
Nessie, Polaris, Trino, Iceberg
 llm
llm module — local RAG over comments + Zotero corpus
 notebooks
marimo notebook errors (auto-filed by nb-watcher/nb-integration)
 packaging
Python packaging, optional deps, build system
 pfs
pipeline
Pipeline execution and validation
 pipeline:ci
pipeline:deploy
pipeline:harden
pipeline:infra-ci
pipeline:rebuild-all
pipeline:release
qpp
quality
Testing, coverage, error handling
 step:build
step:deploy
step:hadolint
step:lint
step:publish
step:scan
step:test
step:validate
styles
Charts, figures, visual theming
 tree
Project directory structure and file organization
No Label
Milestone
No items
No Milestone P28: DAB Modernization — Python wheel tasks, databricks-bundles, schema cleanup
Projects
Clear projects
No project
Assignees
Clear assignees
kert renovate-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: homelab/stack#222
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?