stack/cloud/databricks/README.md

BCDA manager as a Databricks App — playbook

The manager (src/bcda/ui, stack bcda ui) is a Streamlit app that today runs as a compose service behind Traefik with a local store under data/bcda/ and DuckDB as its warehouse. This playbook makes the same code run as a Databricks App for a workspace user who cannot create or manage catalogs. Everything below is organised around that constraint.

Scaffold: cloud/databricks/bcda-app/ (databricks.yml, app.yaml, requirements.txt, .python-version). Nothing in it creates a catalog, schema, volume or warehouse; it only references objects an admin provisioned once and asks for grants to the app's own service principal.

0 · TL;DR

Topic	Decision
Deployment unit	A separate bundle (cloud/databricks/bcda-app/databricks.yml) with one apps resource. The main bundle/ (jobs, schemas, volumes) stays admin-only: it creates schemas and volumes, which needs rights the app user does not have.
Identity	App service principal for background work (pull, flatten, reconcile, load). User authorization (on-behalf-of-user) for the Data page, so Unity Catalog grants, row filters and column masks of the signed-in user apply.
Store	A Unity Catalog volume reached through fsspec (dbfs:/Volumes/<catalog>/bcda/bcda_store). Store and flatten_export already take any fsspec URI. Local disk in an app is ephemeral.
Tables	bcda.*, input_layer.* land as Delta tables in <catalog>.bcda through the SQL warehouse (CREATE OR REPLACE TABLE … AS SELECT * FROM parquet.\/Volumes/…``). No DuckDB file, no replica.
Reconciliation	Phase 1: run bcda.sync as is, over partitions copied to the app's ephemeral disk (flat/ is 62 MB for the Large entity). Phase 2: port _recompute to the warehouse (the SQL is window functions + UNION ALL BY NAME, both supported by Databricks SQL).
Heavy work	The EOB baseline (2.2 GB ndjson, 6 min on the sandbox) is CPU and network work the app runtime is "optimized for UI rendering, not data processing" for. Run it in the app first (Large size), then move it to a Lakeflow job the app triggers and watches (phase 3).
Compute	Medium (2 vCPU / 6 GB, the default) for the UI; Large (4 vCPU / 12 GB) only while a baseline pull runs. Stopped apps cost nothing.
Network	BCDA is an external API. If the workspace runs a restricted serverless network policy, sandbox.bcda.cms.gov and api.bcda.cms.gov must be allow-listed, plus the Databricks and PyPI domains every app needs.
Python	Apps default to 3.11; the repo requires ≥ 3.12. The scaffold pins 3.12 through uv (.python-version + pyproject), which the platform honours for uv projects.

1 · Platform facts the design rests on

Verified against the Databricks docs on 2026-10-25 (links in §9):

• An app is a containerised service on serverless compute. In-memory state and local files do not survive a restart or redeploy.
• Runtime: Ubuntu 22.04, Python 3.11 in a dedicated venv, uv 0.10.2. Dependencies from requirements.txt (pip) or pyproject.toml (uv); pinned versions override pre-installed packages. No apt.
• app.yaml: command (a list, not run through a shell; only DATABRICKS_APP_PORT is substituted) and env entries with value or valueFrom: <resource key>. Streamlit gets STREAMLIT_SERVER_PORT, STREAMLIT_SERVER_ADDRESS=0.0.0.0, STREAMLIT_SERVER_HEADLESS=true, STREAMLIT_BROWSER_GATHER_USAGE_STATS=false set for it.
• Injected: DATABRICKS_HOST, DATABRICKS_APP_NAME, DATABRICKS_WORKSPACE_ID, DATABRICKS_APP_PORT, DATABRICKS_CLIENT_ID / DATABRICKS_CLIENT_SECRET (the app's service principal; the SDK picks them up). With app telemetry on: OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4314, OTEL_SERVICE_NAME.
• App resources (declared on the app, exposed by valueFrom): SQL warehouse → id (Can use / Can manage); secret → decrypted value (Can read / write / manage); volume → /Volumes/<catalog>/<schema>/<volume> (Can read / Can read and write); job → id (Can view / Can manage run / Can manage); table → full name (Select / Modify); Lakebase, serving, Genie, experiment, function, connection, vector-search index. Apps must use existing resources and can't create new ones; adding a resource needs Can manage on both the resource and the app.
• Two identities: the per-app service principal (shared by all users; needs explicit grants on the warehouse and each UC object) and user authorization (the user's SSO token forwarded as X-Forwarded-Access-Token; in Streamlit: st.context.headers.get("X-Forwarded-Access-Token")). Scopes are opt-in per app (sql, sql:restricted-query, files, catalog.tables, …; default is identity-only). A workspace admin can restrict or disable the allowed scopes.
• Compute: Medium up to 2 vCPU / 6 GB (default), Large up to 4 vCPU / 12 GB (compute_size: MEDIUM | LARGE | XLARGE in the bundle, XLARGE depending on workspace configuration); resizing keeps the old size until the update finishes.
• Platform contract: listen on 0.0.0.0:$DATABRICKS_APP_PORT, exit within 15 s of SIGTERM (else SIGKILL), speak H2C (TLS is theirs), log to stdout/stderr (not local files).
• Limits: 100 apps per workspace; app source files ≤ 10 MB each; Files API individual file 500 MB (workspace files row; volumes undocumented on that page).
• Sharing: CAN_USE / CAN_MANAGE on the app; users must be in the Databricks account.
• Deploy: UI, databricks apps deploy <name> --source-code-path … (after databricks sync), Git-backed (--git-branch), or a bundle (databricks bundle deploy + bundle run). The Python databricks-bundles package pinned here (0.295.0) has no apps module, so the app resource is YAML.
• Local disk size inside the container is not documented. Treat it as scratch for one job's partitions, not for the raw exports.

2 · Permissions: what the app user can and cannot do

The user deploying and running the app has no CREATE CATALOG and does not own a catalog. Unity Catalog grants are explicit — nothing is implied by a parent grant or by ownership of something else — so the split is:

2.1 Once, by a metastore/catalog admin

-- an existing catalog, e.g. the dev target in stack.toml
USE CATALOG aco_dev;

CREATE SCHEMA IF NOT EXISTS bcda COMMENT 'BCDA manager: store volume + bcda.* tables';
CREATE VOLUME IF NOT EXISTS bcda.bcda_store COMMENT 'Store root: jobs/files ledgers, exports/, flat/';

-- secrets: scope `stack` already mapped in stack.toml [databricks.secrets]
-- databricks secrets create-scope stack
-- databricks secrets put-secret stack bcda-client-id
-- databricks secrets put-secret stack bcda-client-secret

The app's service principal does not exist until the app does. After the first databricks bundle deploy, read its id from the app's Authorization tab (or databricks apps get bcda-manager), then:

-- <SP> = the app service principal's application id
GRANT USE CATALOG                      ON CATALOG aco_dev        TO `<SP>`;
GRANT USE SCHEMA, CREATE TABLE,
      MODIFY, SELECT                   ON SCHEMA  aco_dev.bcda   TO `<SP>`;
GRANT READ VOLUME, WRITE VOLUME        ON VOLUME  aco_dev.bcda.bcda_store TO `<SP>`;
-- the connector writes input_layer.* too
GRANT USE SCHEMA, CREATE TABLE, MODIFY, SELECT ON SCHEMA aco_dev.input_layer TO `<SP>`;

Plus, outside SQL: CAN USE on the SQL warehouse, READ on the two secrets, and (phase 3) CAN MANAGE RUN on the ingest job. The bundle's resources: block asks for exactly these on the app and the deploy attaches them. Attaching a volume, table or function as a uc_securable resource also grants the app USE CATALOG and USE SCHEMA on its parents automatically (bundle resources reference), so of the SQL above only the schema-level CREATE TABLE, MODIFY, SELECT (and the input_layer block) still need someone who holds MANAGE on the schema or owns it.

For the people who will use the app (the Data page runs as them):

GRANT USE CATALOG ON CATALOG aco_dev TO `bcda-analysts`;
GRANT USE SCHEMA, SELECT ON SCHEMA aco_dev.bcda TO `bcda-analysts`;
GRANT USE SCHEMA, SELECT ON SCHEMA aco_dev.input_layer TO `bcda-analysts`;

2.2 What the app user needs

• CAN_MANAGE on the app (the bundle creates it as the deployer).
• CAN USE on the warehouse, READ on the secrets, READ VOLUME / WRITE VOLUME on the volume: required to attach them as app resources (the platform checks Can manage on the resource when you add it in the UI; via the bundle the deployer must at least be able to reference them — confirm in your workspace, policies vary).
• Nothing on the catalog beyond USE CATALOG.

2.3 What not to run

• stack lake deploy --target databricks / the main bundle/: both create schemas and a volume (bundle/resources/__init__.py), which needs CREATE SCHEMA / CREATE VOLUME on the catalog.
• dev/scripts/setup_unity_catalog.py without an admin: it creates schemas (the catalog itself must already exist — the script says so).

3 · Architecture mapping

Compose service today	Databricks App
Traefik + oauth2-proxy (Gitea SSO)	Platform SSO; share with CAN_USE. No Traefik route, no DNS, no cert.
./data/bcda bind mount (Store, flat/, logs)	UC volume aco_dev.bcda.bcda_store via fsspec-databricks (dbfs:/Volumes/aco_dev/bcda/bcda_store).
BCDA_CLIENT_ID/SECRET from .env	valueFrom: secret (scope stack, keys bcda-client-id, bcda-client-secret).
DuckDB aco.duckdb + replica	Delta tables in aco_dev.bcda / aco_dev.input_layer, written and read through the SQL warehouse.
bcda.sync DuckDB recompute over flat/history/*	Phase 1 unchanged on ephemeral disk; phase 2 on the warehouse.
Connector pipeline (bcda.pipe.cclf) via DuckDBContext	EnterpriseContext (dialect databricks) or the transpiled SQL under bundle/sql/ — the same expressions already generate Databricks SQL.
/metrics on :8502 scraped by Prometheus	App telemetry: when enabled the runtime sets OTEL_EXPORTER_OTLP_ENDPOINT; perf already speaks OTLP. No second port.
data/bcda/log.jsonl	stdout (the platform captures it); keep JSON lines.
docker compose up -d bcda	databricks bundle deploy && databricks bundle run bcda_manager.

Why the store goes on a volume, not Lakebase or workspace files

The store is files: ndjson parts up to hundreds of MB, Parquet partitions, two append-only Parquet ledgers and a JSON manifest per job. Volumes are the governed file surface in Unity Catalog and the app can get exactly the READ VOLUME/WRITE VOLUME it needs on one volume. /Volumes is not mounted inside an app container (community reports; no official mount), so access goes through the Files API — fsspec-databricks wraps it with Databricks unified auth, which is what the injected service-principal credentials provide. It is a one-maintainer beta package (0.2.1, 2026-10-23); pin it, and keep the Files API via WorkspaceClient.files as the fallback for the few operations the store needs (put_file, open, exists, mkdirs, glob).

4 · Code changes the playbook assumes (small, listed so they are not surprises)

1. Store root from the environment. service.store_path() reads conf.path("storage.bcda"), which is a local Path. Add BCDA_STORE_URI (any fsspec URI) that wins when set; make_store already accepts a URI and storage_options. app.yaml sets it from the volume resource (valueFrom: bcda_store gives /Volumes/…; prefix dbfs: in code or set the value directly).
2. Register the fsspec driver at import when BCDA_STORE_URI starts with dbfs: (import fsspec_databricks; fsspec_databricks.use()).
3. bcda.sync on a remote store (phase 1). _root(store) assumes a local path. Give apply() a workdir (ephemeral /tmp/bcda-sync): copy the needed flat/history/<key>/*.parquet partitions and the current flat/*.parquet down, run the DuckDB recompute there, copy the new current tables and the ledger back. 62 MB for the Large entity; keep the raw exports on the volume only.
4. Loader target. load_bcda writes DuckDB. Add a databricks path that runs, per table, on the warehouse: CREATE OR REPLACE TABLE aco_dev.bcda.<table> AS SELECT * FROM parquet.\/Volumes/aco_dev/bcda/bcda_store/flat/.parquet`, then the connector via EnterpriseContext`. No replica publish.
5. Data page through the user's token. replica_connect() returns a DuckDB connection; on Databricks return a databricks.sql connection with access_token=st.context.headers["X-Forwarded-Access-Token"] and the warehouse id from DATABRICKS_WAREHOUSE_ID. The page already caps results (run_query(limit=1000)). Requested scopes: sql (or sql:restricted-query for read-only).
6. Logging to stdout when DATABRICKS_APP_NAME is set (bcda.log.setup currently writes data/bcda/log.jsonl).
7. Metrics: skip metrics.serve() (no second port); leave STACK_TELEMETRY to the OTLP exporter when app telemetry is on.
8. Graceful stop: the runner's threads are daemon threads; long runs must survive a redeploy by being resumable — they are (Store.resume, stack bcda resume), and the Sync page re-applies anything downloaded but not reconciled.

5 · Optimisation notes (what actually matters on this runtime)

• Keep the app a control plane. The runtime is sized for UI. The baseline EOB pull is 192 files / 2.2 GB and the flattener streams in 2,000-resource batches, so memory stays flat, but it is minutes of CPU and network in a 2-vCPU container with other users' sessions on it. Phase 3 moves export → flatten → reconcile → load into a Lakeflow job (the main bundle already builds wheel tasks for stack commands) that the app triggers with CAN MANAGE RUN and polls — the docs' own "start, then poll" pattern for long operations.
• Size for the phase you are in. Medium for day-to-day; switch to Large for the first baseline if it runs in-app, then back. The Exports page's one-job-per-type default also keeps each run small.
• Cache what is static. Dictionary and crosswalk frames (dictionary_frames) and vendor_status are per-process constants: wrap with st.cache_data. Server job lists change: ttl=60.
• One connection per session, not per widget. Create the SQL connector once per session (st.session_state), reuse it; the user's forwarded token can change between sessions, so key the cache on the token's hash and re-read the header every run (a community thread reports stale headers after a logout/login in the same tab).
• Serverless warehouse, small, auto-stop. stack.toml already says SMALL, auto_stop_mins = 10. All heavy SQL (reconcile phase 2, loads, connector) runs there, not in the app.
• Never pull a table into the app. The Data page already limits rows; the Sync and Store pages read ledgers of a few hundred rows. Keep st.dataframe on Polars frames; avoid collect() on anything that came from the warehouse unfiltered.
• Startup. Imports in service.py are already lazy; keep metrics.serve() and vendor verification out of the import path. Pre-compute nothing at boot.
• Egress policy dry-run first. If the workspace enforces a restricted network policy, test with the policy's dry-run mode: an app that cannot reach sandbox.bcda.cms.gov fails only when a user presses Start export.
• Source size. Sync only what the app imports: src/, pyproject.toml, uv.lock, stack.toml, dev/seeds/bcda_v3_*.csv (dictionary page), assets/icons/fav32.png. Not data/, dev/vendor (13 MB, has a 1.5 MB xlsx and gzip fixtures), dev/seeds as a whole (157 MB; two ndjson files exceed the 10 MB per-file limit), tuva/, infra/. The scaffold's sync.exclude does this.
• Cost. A stopped app costs nothing; running is billed per hour of compute at the chosen size. Stop it outside working hours if nobody uses it; the store is on the volume, nothing is lost.

6 · Runbook

# 0 · prerequisites (admin): §2.1 schema, volume, secrets, warehouse; you: CLI ≥ 0.229 with OAuth U2M
databricks auth login --host https://<workspace>.cloud.databricks.com

# 1 · set the three ids the scaffold references
cd cloud/databricks/bcda-app
export BCDA_WAREHOUSE_ID=<sql warehouse id>           # databricks warehouses list
#     catalog/schema/volume names are variables in databricks.yml (defaults: aco_dev / bcda / bcda_store)

# 2 · dependencies file (regenerate when uv.lock changes)
(cd ../../.. && uv export --no-dev --no-hashes --extra bcda --extra aco --extra conf \
   -o cloud/databricks/bcda-app/requirements.txt && echo "fsspec-databricks==0.2.1" >> cloud/databricks/bcda-app/requirements.txt \
   && echo "databricks-sql-connector>=4.0" >> cloud/databricks/bcda-app/requirements.txt && echo "." >> cloud/databricks/bcda-app/requirements.txt)

# 3 · validate, deploy (creates the app + its service principal), start
databricks bundle validate -t dev
databricks bundle deploy   -t dev
databricks bundle run bcda_manager -t dev

# 4 · UC grants to the new service principal (admin, §2.1), then user authorization scopes
databricks apps get bcda-manager-dev | jq .service_principal_client_id
#   App → Authorization → + Add scope: sql  (and files if the Data page should read the volume as the user)

# 5 · share
#   App → Permissions → bcda-analysts: CAN_USE   (or the permissions: block in databricks.yml)

# 6 · first run
#   Connect → preset/production creds from secrets → Exports: baseline, one job per type (Large size for this)
#   Sync → Reconcile now → Data page queries as you (OBO)

# 7 · logs and health
databricks apps logs bcda-manager-dev          # or the app's Logs tab

Redeploy = databricks bundle deploy again; in-flight runs die with the container, so finish them with stack bcda resume <key> from the Store page's key list (or wait for the next apply, which skips nothing).

7 · Phases

Phase	Scope	Done when
1	Store on the volume, local (ephemeral) reconcile, loader to Delta via warehouse, Data page via OBO, logs to stdout	A baseline and a _since delta round-trip from the app; aco_dev.bcda.sync_ledger shows both; Data page queries succeed for a bcda-analysts member who has no catalog rights
2	bcda.sync._recompute as warehouse SQL over parquet.\/Volumes/…`` partitions; keep the same ledger and invariants	tests/bcda/test_sync.py scenarios pass against a warehouse (parity test), app no longer copies partitions
3	Ingest as a Lakeflow job triggered from the Exports/Sync pages; app only polls	Baseline pull runs with the app on Medium; job run id recorded on the store job row
4	Telemetry to app OTLP; audit records of user actions (who started what) as structured logs	Traces/logs visible under the app's telemetry

8 · Open questions to verify in the target workspace

• Whether sync.paths / source_code_path may point above the bundle root (../../..) in your CLI version. If not, move cloud/databricks/bcda-app/databricks.yml to a thin wrapper at the repo root under another bundle name, or build a wheel and sync only app.yaml + requirements.txt that installs it.

• Allowed OAuth scopes for apps (admin setting) — if None, the Data page must run as the service principal and every analyst needs CAN_USE on the app only; UC filters then do not apply.

• Serverless network policy mode and whether *.bcda.cms.gov is allowed.

• Whether the deploying user can attach the warehouse/secret/volume resources without CAN MANAGE on them (the UI requires it; bundles may as well).

• Ephemeral disk headroom for phase 1 (62 MB of partitions is safe; a full raw EOB export is not meant to touch local disk).

• fsspec-databricks write path on your cloud (AWS is listed as untested by the maintainer) — fall back to WorkspaceClient.files if multipart uploads misbehave.

9 · Sources

• Databricks Apps key concepts — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/key-concepts
• Apps environment (runtime, env vars) — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/system-env
• Configure app execution (app.yaml) — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/app-runtime
• Environment variables / valueFrom — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/environment-variables
• Add resources to an app — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/resources
• Authorization (service principal, user authorization, scopes) — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/auth
• Compute size — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/compute-size
• Best practices — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/best-practices
• Networking / serverless network policies — https://docs.databricks.com/gcp/en/dev-tools/databricks-apps/networking , https://learn.microsoft.com/en-in/Azure/databricks/security/network/serverless-network-security/network-policies
• Deploy — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/deploy
• Resource limits — https://docs.databricks.com/aws/en/resources/limits
• Bundles: apps tutorial and resources reference — https://docs.databricks.com/aws/en/dev-tools/bundles/apps-tutorial , https://docs.databricks.com/aws/en/dev-tools/bundles/resources
• Unity Catalog privileges — https://docs.databricks.com/aws/en/data-governance/unity-catalog/access-control/privileges-reference , https://docs.databricks.com/aws/en/data-governance/unity-catalog/access-control/permissions-concepts
• Volume files / Files API — https://docs.databricks.com/aws/files/volumes , https://docs.databricks.com/api/files/v2/file
• fsspec-databricks — https://pypi.org/project/fsspec-databricks/
• Community reports used only as caveats: /Volumes not mounted in apps (https://community.databricks.com/t5/administration-architecture/how-to-access-unitycatalog-s-volume-inside-databricks-app/m-p/134845), stale X-Forwarded-Access-Token after re-login (https://community.databricks.com/t5/administration-architecture/x-forwarded-access-token-occasionally-return-stale-values-from-a/td-p/163074)