From 8cd4633dac73176f4e15aaef27010521e63bbf8a Mon Sep 17 00:00:00 2001 From: kert Date: Sun, 25 Oct 2026 16:19:35 -0400 Subject: [PATCH] docs(bcda): playbook for running the manager as a Databricks App without catalog rights MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit cloud/databricks/README.md: what the platform guarantees (ephemeral container, Python 3.11 default → uv pin 3.12, app.yaml command/env with valueFrom, injected service-principal credentials, user authorization via X-Forwarded-Access-Token and opt-in scopes, Medium/Large compute, 15 s SIGTERM, 10 MB per source file, restricted egress policies), the permission split for a user who cannot create or manage catalogs (admin provisions catalog/schema/volume/warehouse/secrets once; attaching a uc_securable grants USE CATALOG/USE SCHEMA automatically; only CREATE TABLE/MODIFY/SELECT on the schema need a schema manager), the compose → Databricks mapping (store on a UC volume via fsspec, Delta tables through the SQL warehouse, no DuckDB file, no second port), the eight small code changes the port needs, optimisation notes (app as control plane, size per phase, cache statics, one connection per session, never pull a table into the app, sync only what the app imports), a runbook, phases 1–4 and the open questions to verify in the target workspace. Sources listed. cloud/databricks/bcda-app/: a bundle of its own (the main bundle creates schemas and a volume, which needs catalog rights) with the app resource, its warehouse/secret/volume resources and permissions, app.yaml, .python-version and a requirements stub. refs #1139 --- cloud/databricks/README.md | 351 +++++++++++++++++++++ cloud/databricks/bcda-app/.python-version | 1 + cloud/databricks/bcda-app/app.yaml | 27 ++ cloud/databricks/bcda-app/databricks.yml | 104 ++++++ cloud/databricks/bcda-app/requirements.txt | 7 + 5 files changed, 490 insertions(+) create mode 100644 cloud/databricks/README.md create mode 100644 cloud/databricks/bcda-app/.python-version create mode 100644 cloud/databricks/bcda-app/app.yaml create mode 100644 cloud/databricks/bcda-app/databricks.yml create mode 100644 cloud/databricks/bcda-app/requirements.txt diff --git a/cloud/databricks/README.md b/cloud/databricks/README.md new file mode 100644 index 0000000..80d500d --- /dev/null +++ b/cloud/databricks/README.md @@ -0,0 +1,351 @@ +# BCDA manager as a Databricks App — playbook + +The manager (`src/bcda/ui`, `stack bcda ui`) is a Streamlit app that today +runs as a compose service behind Traefik with a local store under +`data/bcda/` and DuckDB as its warehouse. This playbook makes the same code +run as a **Databricks App** for a workspace user who **cannot create or +manage catalogs**. Everything below is organised around that constraint. + +Scaffold: `cloud/databricks/bcda-app/` (`databricks.yml`, `app.yaml`, +`requirements.txt`, `.python-version`). Nothing in it creates a catalog, +schema, volume or warehouse; it only *references* objects an admin +provisioned once and asks for grants to the app's own service principal. + +## 0 · TL;DR + +| Topic | Decision | +|---|---| +| Deployment unit | A **separate bundle** (`cloud/databricks/bcda-app/databricks.yml`) with one `apps` resource. The main `bundle/` (jobs, schemas, volumes) stays admin-only: it *creates* schemas and volumes, which needs rights the app user does not have. | +| Identity | **App service principal** for background work (pull, flatten, reconcile, load). **User authorization (on-behalf-of-user)** for the Data page, so Unity Catalog grants, row filters and column masks of the signed-in user apply. | +| Store | A **Unity Catalog volume** reached through fsspec (`dbfs:/Volumes//bcda/bcda_store`). `Store` and `flatten_export` already take any fsspec URI. Local disk in an app is ephemeral. | +| Tables | `bcda.*`, `input_layer.*` land as Delta tables in `.bcda` through the **SQL warehouse** (`CREATE OR REPLACE TABLE … AS SELECT * FROM parquet.\`/Volumes/…\``). No DuckDB file, no replica. | +| Reconciliation | Phase 1: run `bcda.sync` as is, over partitions copied to the app's ephemeral disk (`flat/` is 62 MB for the Large entity). Phase 2: port `_recompute` to the warehouse (the SQL is window functions + `UNION ALL BY NAME`, both supported by Databricks SQL). | +| Heavy work | The EOB baseline (2.2 GB ndjson, 6 min on the sandbox) is CPU and network work the app runtime is "optimized for UI rendering, not data processing" for. Run it in the app first (Large size), then move it to a **Lakeflow job** the app triggers and watches (phase 3). | +| Compute | **Medium** (2 vCPU / 6 GB, the default) for the UI; **Large** (4 vCPU / 12 GB) only while a baseline pull runs. Stopped apps cost nothing. | +| Network | BCDA is an external API. If the workspace runs a *restricted* serverless network policy, `sandbox.bcda.cms.gov` and `api.bcda.cms.gov` must be allow-listed, plus the Databricks and PyPI domains every app needs. | +| Python | Apps default to **3.11**; the repo requires **≥ 3.12**. The scaffold pins 3.12 through uv (`.python-version` + `pyproject`), which the platform honours for uv projects. | + +## 1 · Platform facts the design rests on + +Verified against the Databricks docs on 2026-10-25 (links in §9): + +- An app is a containerised service on serverless compute. **In-memory + state and local files do not survive a restart or redeploy.** +- Runtime: Ubuntu 22.04, Python 3.11 in a dedicated venv, uv 0.10.2. + Dependencies from `requirements.txt` (pip) or `pyproject.toml` (uv); + pinned versions override pre-installed packages. No `apt`. +- `app.yaml`: `command` (a list, **not run through a shell**; only + `DATABRICKS_APP_PORT` is substituted) and `env` entries with `value` or + `valueFrom: `. Streamlit gets `STREAMLIT_SERVER_PORT`, + `STREAMLIT_SERVER_ADDRESS=0.0.0.0`, `STREAMLIT_SERVER_HEADLESS=true`, + `STREAMLIT_BROWSER_GATHER_USAGE_STATS=false` set for it. +- Injected: `DATABRICKS_HOST`, `DATABRICKS_APP_NAME`, + `DATABRICKS_WORKSPACE_ID`, `DATABRICKS_APP_PORT`, + `DATABRICKS_CLIENT_ID` / `DATABRICKS_CLIENT_SECRET` (the app's service + principal; the SDK picks them up). With app telemetry on: + `OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4314`, `OTEL_SERVICE_NAME`. +- App resources (declared on the app, exposed by `valueFrom`): SQL + warehouse → id (Can use / Can manage); secret → decrypted value (Can + read / write / manage); volume → `/Volumes///` + (Can read / Can read and write); job → id (Can view / Can manage run / + Can manage); table → full name (Select / Modify); Lakebase, serving, + Genie, experiment, function, connection, vector-search index. **Apps + must use existing resources and can't create new ones**; adding a + resource needs *Can manage* on both the resource and the app. +- Two identities: the per-app **service principal** (shared by all users; + needs explicit grants on the warehouse *and* each UC object) and + **user authorization** (the user's SSO token forwarded as + `X-Forwarded-Access-Token`; in Streamlit: + `st.context.headers.get("X-Forwarded-Access-Token")`). Scopes are + opt-in per app (`sql`, `sql:restricted-query`, `files`, + `catalog.tables`, …; default is identity-only). A workspace admin can + restrict or disable the allowed scopes. +- Compute: Medium up to 2 vCPU / 6 GB (default), Large up to 4 vCPU / + 12 GB (`compute_size: MEDIUM | LARGE | XLARGE` in the bundle, XLARGE + depending on workspace configuration); resizing keeps the old size + until the update finishes. +- Platform contract: listen on `0.0.0.0:$DATABRICKS_APP_PORT`, exit + within **15 s of SIGTERM** (else SIGKILL), speak H2C (TLS is theirs), + log to **stdout/stderr** (not local files). +- Limits: 100 apps per workspace; app source files **≤ 10 MB each**; + Files API individual file 500 MB (workspace files row; volumes + undocumented on that page). +- Sharing: `CAN_USE` / `CAN_MANAGE` on the app; users must be in the + Databricks account. +- Deploy: UI, `databricks apps deploy --source-code-path …` + (after `databricks sync`), Git-backed (`--git-branch`), or a bundle + (`databricks bundle deploy` + `bundle run`). The Python + `databricks-bundles` package pinned here (0.295.0) has **no `apps` + module**, so the app resource is YAML. +- Local disk size inside the container is **not documented**. Treat it as + scratch for one job's partitions, not for the raw exports. + +## 2 · Permissions: what the app user can and cannot do + +The user deploying and running the app has **no `CREATE CATALOG` and does +not own a catalog**. Unity Catalog grants are explicit — nothing is +implied by a parent grant or by ownership of something else — so the split +is: + +### 2.1 Once, by a metastore/catalog admin + +```sql +-- an existing catalog, e.g. the dev target in stack.toml +USE CATALOG aco_dev; + +CREATE SCHEMA IF NOT EXISTS bcda COMMENT 'BCDA manager: store volume + bcda.* tables'; +CREATE VOLUME IF NOT EXISTS bcda.bcda_store COMMENT 'Store root: jobs/files ledgers, exports/, flat/'; + +-- secrets: scope `stack` already mapped in stack.toml [databricks.secrets] +-- databricks secrets create-scope stack +-- databricks secrets put-secret stack bcda-client-id +-- databricks secrets put-secret stack bcda-client-secret +``` + +The app's service principal does not exist until the app does. After the +first `databricks bundle deploy`, read its id from the app's +**Authorization** tab (or `databricks apps get bcda-manager`), then: + +```sql +-- = the app service principal's application id +GRANT USE CATALOG ON CATALOG aco_dev TO ``; +GRANT USE SCHEMA, CREATE TABLE, + MODIFY, SELECT ON SCHEMA aco_dev.bcda TO ``; +GRANT READ VOLUME, WRITE VOLUME ON VOLUME aco_dev.bcda.bcda_store TO ``; +-- the connector writes input_layer.* too +GRANT USE SCHEMA, CREATE TABLE, MODIFY, SELECT ON SCHEMA aco_dev.input_layer TO ``; +``` + +Plus, outside SQL: `CAN USE` on the SQL warehouse, `READ` on the two +secrets, and (phase 3) `CAN MANAGE RUN` on the ingest job. The bundle's +`resources:` block asks for exactly these on the app and the deploy +attaches them. Attaching a volume, table or function as a `uc_securable` +resource also grants the app `USE CATALOG` and `USE SCHEMA` on its +parents automatically (bundle resources reference), so of the SQL above +only the schema-level `CREATE TABLE, MODIFY, SELECT` (and the +`input_layer` block) still need someone who holds `MANAGE` on the schema +or owns it. + +For the people who will *use* the app (the Data page runs as them): + +```sql +GRANT USE CATALOG ON CATALOG aco_dev TO `bcda-analysts`; +GRANT USE SCHEMA, SELECT ON SCHEMA aco_dev.bcda TO `bcda-analysts`; +GRANT USE SCHEMA, SELECT ON SCHEMA aco_dev.input_layer TO `bcda-analysts`; +``` + +### 2.2 What the app user needs + +- `CAN_MANAGE` on the app (the bundle creates it as the deployer). +- `CAN USE` on the warehouse, `READ` on the secrets, `READ VOLUME` / + `WRITE VOLUME` on the volume: required to *attach* them as app + resources (the platform checks *Can manage* on the resource when you + add it in the UI; via the bundle the deployer must at least be able to + reference them — confirm in your workspace, policies vary). +- Nothing on the catalog beyond `USE CATALOG`. + +### 2.3 What not to run + +- `stack lake deploy --target databricks` / the main `bundle/`: both + create schemas and a volume (`bundle/resources/__init__.py`), which + needs `CREATE SCHEMA` / `CREATE VOLUME` on the catalog. +- `dev/scripts/setup_unity_catalog.py` without an admin: it creates + schemas (the catalog itself must already exist — the script says so). + +## 3 · Architecture mapping + +| Compose service today | Databricks App | +|---|---| +| Traefik + oauth2-proxy (Gitea SSO) | Platform SSO; share with `CAN_USE`. No Traefik route, no DNS, no cert. | +| `./data/bcda` bind mount (`Store`, `flat/`, logs) | UC volume `aco_dev.bcda.bcda_store` via fsspec-databricks (`dbfs:/Volumes/aco_dev/bcda/bcda_store`). | +| `BCDA_CLIENT_ID/SECRET` from `.env` | `valueFrom: secret` (scope `stack`, keys `bcda-client-id`, `bcda-client-secret`). | +| DuckDB `aco.duckdb` + replica | Delta tables in `aco_dev.bcda` / `aco_dev.input_layer`, written and read through the SQL warehouse. | +| `bcda.sync` DuckDB recompute over `flat/history/*` | Phase 1 unchanged on ephemeral disk; phase 2 on the warehouse. | +| Connector pipeline (`bcda.pipe.cclf`) via `DuckDBContext` | `EnterpriseContext` (dialect `databricks`) or the transpiled SQL under `bundle/sql/` — the same expressions already generate Databricks SQL. | +| `/metrics` on :8502 scraped by Prometheus | App telemetry: when enabled the runtime sets `OTEL_EXPORTER_OTLP_ENDPOINT`; `perf` already speaks OTLP. No second port. | +| `data/bcda/log.jsonl` | stdout (the platform captures it); keep JSON lines. | +| `docker compose up -d bcda` | `databricks bundle deploy && databricks bundle run bcda_manager`. | + +### Why the store goes on a volume, not Lakebase or workspace files + +The store is files: ndjson parts up to hundreds of MB, Parquet partitions, +two append-only Parquet ledgers and a JSON manifest per job. Volumes are +the governed file surface in Unity Catalog and the app can get exactly the +`READ VOLUME`/`WRITE VOLUME` it needs on one volume. `/Volumes` is **not +mounted** inside an app container (community reports; no official mount), +so access goes through the Files API — `fsspec-databricks` wraps it with +Databricks unified auth, which is what the injected service-principal +credentials provide. It is a one-maintainer beta package (0.2.1, +2026-10-23); pin it, and keep the Files API via `WorkspaceClient.files` as +the fallback for the few operations the store needs (`put_file`, `open`, +`exists`, `mkdirs`, `glob`). + +## 4 · Code changes the playbook assumes (small, listed so they are not surprises) + +1. **Store root from the environment.** `service.store_path()` reads + `conf.path("storage.bcda")`, which is a local `Path`. Add + `BCDA_STORE_URI` (any fsspec URI) that wins when set; `make_store` + already accepts a URI and `storage_options`. `app.yaml` sets it from + the volume resource (`valueFrom: bcda_store` gives `/Volumes/…`; + prefix `dbfs:` in code or set the value directly). +2. **Register the fsspec driver at import** when `BCDA_STORE_URI` starts + with `dbfs:` (`import fsspec_databricks; fsspec_databricks.use()`). +3. **`bcda.sync` on a remote store (phase 1).** `_root(store)` assumes a + local path. Give `apply()` a `workdir` (ephemeral `/tmp/bcda-sync`): + copy the needed `flat/history//*.parquet` partitions and the + current `flat/*.parquet` down, run the DuckDB recompute there, copy + the new current tables and the ledger back. 62 MB for the Large + entity; keep the raw exports on the volume only. +4. **Loader target.** `load_bcda` writes DuckDB. Add a `databricks` + path that runs, per table, on the warehouse: + `CREATE OR REPLACE TABLE aco_dev.bcda. AS SELECT * FROM + parquet.\`/Volumes/aco_dev/bcda/bcda_store/flat/
.parquet\``, + then the connector via `EnterpriseContext`. No replica publish. +5. **Data page through the user's token.** `replica_connect()` returns a + DuckDB connection; on Databricks return a `databricks.sql` connection + with `access_token=st.context.headers["X-Forwarded-Access-Token"]` + and the warehouse id from `DATABRICKS_WAREHOUSE_ID`. The page already + caps results (`run_query(limit=1000)`). Requested scopes: `sql` + (or `sql:restricted-query` for read-only). +6. **Logging to stdout** when `DATABRICKS_APP_NAME` is set + (`bcda.log.setup` currently writes `data/bcda/log.jsonl`). +7. **Metrics**: skip `metrics.serve()` (no second port); leave + `STACK_TELEMETRY` to the OTLP exporter when app telemetry is on. +8. **Graceful stop**: the runner's threads are daemon threads; long runs + must survive a redeploy by being resumable — they are + (`Store.resume`, `stack bcda resume`), and the Sync page re-applies + anything downloaded but not reconciled. + +## 5 · Optimisation notes (what actually matters on this runtime) + +- **Keep the app a control plane.** The runtime is sized for UI. The + baseline EOB pull is 192 files / 2.2 GB and the flattener streams in + 2,000-resource batches, so memory stays flat, but it is minutes of CPU + and network in a 2-vCPU container with other users' sessions on it. + Phase 3 moves `export → flatten → reconcile → load` into a Lakeflow + job (the main bundle already builds wheel tasks for `stack` commands) + that the app triggers with `CAN MANAGE RUN` and polls — the docs' own + "start, then poll" pattern for long operations. +- **Size for the phase you are in.** Medium for day-to-day; switch to + Large for the first baseline if it runs in-app, then back. The + Exports page's one-job-per-type default also keeps each run small. +- **Cache what is static.** Dictionary and crosswalk frames + (`dictionary_frames`) and `vendor_status` are per-process constants: + wrap with `st.cache_data`. Server job lists change: `ttl=60`. +- **One connection per session, not per widget.** Create the SQL + connector once per session (`st.session_state`), reuse it; the user's + forwarded token can change between sessions, so key the cache on the + token's hash and re-read the header every run (a community thread + reports stale headers after a logout/login in the same tab). +- **Serverless warehouse, small, auto-stop.** `stack.toml` already says + `SMALL`, `auto_stop_mins = 10`. All heavy SQL (reconcile phase 2, + loads, connector) runs there, not in the app. +- **Never pull a table into the app.** The Data page already limits + rows; the Sync and Store pages read ledgers of a few hundred rows. + Keep `st.dataframe` on Polars frames; avoid `collect()` on anything + that came from the warehouse unfiltered. +- **Startup.** Imports in `service.py` are already lazy; keep + `metrics.serve()` and vendor verification out of the import path. + Pre-compute nothing at boot. +- **Egress policy dry-run first.** If the workspace enforces a + restricted network policy, test with the policy's dry-run mode: an + app that cannot reach `sandbox.bcda.cms.gov` fails only when a user + presses *Start export*. +- **Source size.** Sync only what the app imports: `src/`, + `pyproject.toml`, `uv.lock`, `stack.toml`, `dev/seeds/bcda_v3_*.csv` + (dictionary page), `assets/icons/fav32.png`. Not `data/`, `dev/vendor` + (13 MB, has a 1.5 MB xlsx and gzip fixtures), `dev/seeds` as a whole + (157 MB; two ndjson files exceed the 10 MB per-file limit), `tuva/`, + `infra/`. The scaffold's `sync.exclude` does this. +- **Cost.** A stopped app costs nothing; running is billed per hour of + compute at the chosen size. Stop it outside working hours if nobody + uses it; the store is on the volume, nothing is lost. + +## 6 · Runbook + +```bash +# 0 · prerequisites (admin): §2.1 schema, volume, secrets, warehouse; you: CLI ≥ 0.229 with OAuth U2M +databricks auth login --host https://.cloud.databricks.com + +# 1 · set the three ids the scaffold references +cd cloud/databricks/bcda-app +export BCDA_WAREHOUSE_ID= # databricks warehouses list +# catalog/schema/volume names are variables in databricks.yml (defaults: aco_dev / bcda / bcda_store) + +# 2 · dependencies file (regenerate when uv.lock changes) +(cd ../../.. && uv export --no-dev --no-hashes --extra bcda --extra aco --extra conf \ + -o cloud/databricks/bcda-app/requirements.txt && echo "fsspec-databricks==0.2.1" >> cloud/databricks/bcda-app/requirements.txt \ + && echo "databricks-sql-connector>=4.0" >> cloud/databricks/bcda-app/requirements.txt && echo "." >> cloud/databricks/bcda-app/requirements.txt) + +# 3 · validate, deploy (creates the app + its service principal), start +databricks bundle validate -t dev +databricks bundle deploy -t dev +databricks bundle run bcda_manager -t dev + +# 4 · UC grants to the new service principal (admin, §2.1), then user authorization scopes +databricks apps get bcda-manager-dev | jq .service_principal_client_id +# App → Authorization → + Add scope: sql (and files if the Data page should read the volume as the user) + +# 5 · share +# App → Permissions → bcda-analysts: CAN_USE (or the permissions: block in databricks.yml) + +# 6 · first run +# Connect → preset/production creds from secrets → Exports: baseline, one job per type (Large size for this) +# Sync → Reconcile now → Data page queries as you (OBO) + +# 7 · logs and health +databricks apps logs bcda-manager-dev # or the app's Logs tab +``` + +Redeploy = `databricks bundle deploy` again; in-flight runs die with the +container, so finish them with `stack bcda resume ` from the Store +page's key list (or wait for the next apply, which skips nothing). + +## 7 · Phases + +| Phase | Scope | Done when | +|---|---|---| +| 1 | Store on the volume, local (ephemeral) reconcile, loader to Delta via warehouse, Data page via OBO, logs to stdout | A baseline and a `_since` delta round-trip from the app; `aco_dev.bcda.sync_ledger` shows both; Data page queries succeed for a `bcda-analysts` member who has no catalog rights | +| 2 | `bcda.sync._recompute` as warehouse SQL over `parquet.\`/Volumes/…\`` partitions; keep the same ledger and invariants | `tests/bcda/test_sync.py` scenarios pass against a warehouse (parity test), app no longer copies partitions | +| 3 | Ingest as a Lakeflow job triggered from the Exports/Sync pages; app only polls | Baseline pull runs with the app on Medium; job run id recorded on the store job row | +| 4 | Telemetry to app OTLP; audit records of user actions (who started what) as structured logs | Traces/logs visible under the app's telemetry | + +## 8 · Open questions to verify in the target workspace + +- Whether `sync.paths` / `source_code_path` may point above the bundle + root (`../../..`) in your CLI version. If not, move + `cloud/databricks/bcda-app/databricks.yml` to a thin wrapper at the + repo root under another bundle name, or build a wheel and sync only + `app.yaml` + `requirements.txt` that installs it. + +- Allowed OAuth scopes for apps (admin setting) — if `None`, the Data + page must run as the service principal and every analyst needs + `CAN_USE` on the app only; UC filters then do not apply. +- Serverless network policy mode and whether `*.bcda.cms.gov` is allowed. +- Whether the deploying user can attach the warehouse/secret/volume + resources without `CAN MANAGE` on them (the UI requires it; bundles may + as well). +- Ephemeral disk headroom for phase 1 (62 MB of partitions is safe; a + full raw EOB export is not meant to touch local disk). +- `fsspec-databricks` write path on your cloud (AWS is listed as + untested by the maintainer) — fall back to `WorkspaceClient.files` if + multipart uploads misbehave. + +## 9 · Sources + +- Databricks Apps key concepts — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/key-concepts +- Apps environment (runtime, env vars) — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/system-env +- Configure app execution (`app.yaml`) — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/app-runtime +- Environment variables / `valueFrom` — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/environment-variables +- Add resources to an app — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/resources +- Authorization (service principal, user authorization, scopes) — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/auth +- Compute size — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/compute-size +- Best practices — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/best-practices +- Networking / serverless network policies — https://docs.databricks.com/gcp/en/dev-tools/databricks-apps/networking , https://learn.microsoft.com/en-in/Azure/databricks/security/network/serverless-network-security/network-policies +- Deploy — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/deploy +- Resource limits — https://docs.databricks.com/aws/en/resources/limits +- Bundles: apps tutorial and resources reference — https://docs.databricks.com/aws/en/dev-tools/bundles/apps-tutorial , https://docs.databricks.com/aws/en/dev-tools/bundles/resources +- Unity Catalog privileges — https://docs.databricks.com/aws/en/data-governance/unity-catalog/access-control/privileges-reference , https://docs.databricks.com/aws/en/data-governance/unity-catalog/access-control/permissions-concepts +- Volume files / Files API — https://docs.databricks.com/aws/files/volumes , https://docs.databricks.com/api/files/v2/file +- fsspec-databricks — https://pypi.org/project/fsspec-databricks/ +- Community reports used only as caveats: `/Volumes` not mounted in apps (https://community.databricks.com/t5/administration-architecture/how-to-access-unitycatalog-s-volume-inside-databricks-app/m-p/134845), stale `X-Forwarded-Access-Token` after re-login (https://community.databricks.com/t5/administration-architecture/x-forwarded-access-token-occasionally-return-stale-values-from-a/td-p/163074) diff --git a/cloud/databricks/bcda-app/.python-version b/cloud/databricks/bcda-app/.python-version new file mode 100644 index 0000000..e4fba21 --- /dev/null +++ b/cloud/databricks/bcda-app/.python-version @@ -0,0 +1 @@ +3.12 diff --git a/cloud/databricks/bcda-app/app.yaml b/cloud/databricks/bcda-app/app.yaml new file mode 100644 index 0000000..6040822 --- /dev/null +++ b/cloud/databricks/bcda-app/app.yaml @@ -0,0 +1,27 @@ +# Runtime manifest for the BCDA manager on Databricks Apps. +# The platform sets STREAMLIT_SERVER_PORT/ADDRESS/HEADLESS for Streamlit, so +# `python -m bcda.ui` needs no port argument; it passes the fhirworx theme +# file as --theme.base itself. Commands are not run through a shell. +command: ["python", "-m", "bcda.ui"] +env: + # Store root on the Unity Catalog volume (fsspec-databricks: dbfs:/Volumes/…). + # valueFrom: bcda_store would give the bare /Volumes path; see README §4.1. + - name: BCDA_STORE_URI + value: "dbfs:/Volumes/aco_dev/bcda/bcda_store" + - name: BCDA_CLIENT_ID + valueFrom: bcda_client_id + - name: BCDA_CLIENT_SECRET + valueFrom: bcda_client_secret + - name: BCDA_BASE_URL + value: "https://sandbox.bcda.cms.gov" + - name: DATABRICKS_WAREHOUSE_ID + valueFrom: sql_warehouse + - name: BCDA_TARGET_CATALOG + value: "aco_dev" + - name: BCDA_TARGET_SCHEMA + value: "bcda" + # No second port on Apps; metrics go to app telemetry (OTLP) when enabled. + - name: STACK_TELEMETRY + value: "false" + - name: BCDA_METRICS_PORT + value: "0" diff --git a/cloud/databricks/bcda-app/databricks.yml b/cloud/databricks/bcda-app/databricks.yml new file mode 100644 index 0000000..ac7e62a --- /dev/null +++ b/cloud/databricks/bcda-app/databricks.yml @@ -0,0 +1,104 @@ +# BCDA manager as a Databricks App — a bundle of its own. +# +# Deliberately separate from the repo's main bundle (bundle/, generated by +# dev/scripts/gen_config.py), which CREATES schemas and a volume and so needs +# catalog-level rights. This one only references objects an admin provisioned +# once (cloud/databricks/README.md §2.1) and asks for grants to the app's +# service principal. Deploy from this directory: +# +# databricks bundle validate -t dev && databricks bundle deploy -t dev && databricks bundle run bcda_manager -t dev +bundle: + name: bcda-manager + +variables: + catalog: + description: Existing catalog (no CREATE CATALOG needed) + default: aco_dev + schema: + description: Existing schema holding the store volume and bcda.* tables + default: bcda + volume: + description: Existing managed volume used as the store root + default: bcda_store + warehouse_id: + description: SQL warehouse the app may use + lookup: + warehouse: stack-sql-warehouse + secret_scope: + default: stack + users_group: + description: Group that may open the app (Data page runs as them) + default: bcda-analysts + +# Only what the app imports. The whole repo is far over the 10 MB per-file +# limit (dev/seeds ndjson, uv.lock is fine) and most of it is irrelevant. +sync: + paths: + - ../../.. + include: + - ../../../src/** + - ../../../pyproject.toml + - ../../../uv.lock + - ../../../stack.toml + - ../../../README.md + - ../../../dev/seeds/bcda_v3_*.csv + - ../../../assets/icons/fav32.png + - ./app.yaml + - ./requirements.txt + - ./.python-version + exclude: + - ../../../data/** + - ../../../dev/vendor/** + - ../../../tuva/** + - ../../../infra/** + - ../../../docs/** + - ../../../notebooks/** + - ../../../.venv/** + +resources: + apps: + bcda_manager: + name: bcda-manager-${bundle.target} + description: "BCDA v3 manager — exports, store, _since reconciliation, load, crosswalk" + source_code_path: ../../.. + # MEDIUM (2 vCPU / 6 GB) for the UI; switch to LARGE while a baseline + # pull runs in-app (README §5), back afterwards. + compute_size: MEDIUM + resources: + - name: sql_warehouse + sql_warehouse: + id: ${var.warehouse_id} + permission: CAN_USE + - name: bcda_client_id + secret: + scope: ${var.secret_scope} + key: bcda-client-id + permission: READ + - name: bcda_client_secret + secret: + scope: ${var.secret_scope} + key: bcda-client-secret + permission: READ + - name: bcda_store + uc_securable: + securable_type: VOLUME + securable_full_name: ${var.catalog}.${var.schema}.${var.volume} + permission: WRITE_VOLUME + # The Data page queries as the signed-in user; sql:restricted-query is + # enough for read-only browsing. Admins may restrict allowed scopes. + user_api_scopes: + - sql:restricted-query + permissions: + - group_name: ${var.users_group} + level: CAN_USE + +targets: + dev: + mode: development + default: true + prod: + mode: production + variables: + catalog: aco + workspace: + root_path: /Workspace/Shared/.bundle/${bundle.name}/${bundle.target} diff --git a/cloud/databricks/bcda-app/requirements.txt b/cloud/databricks/bcda-app/requirements.txt new file mode 100644 index 0000000..fa8c87a --- /dev/null +++ b/cloud/databricks/bcda-app/requirements.txt @@ -0,0 +1,7 @@ +# Regenerate from the lock (README §6 step 2): +# uv export --no-dev --no-hashes --extra bcda --extra aco --extra conf -o cloud/databricks/bcda-app/requirements.txt +# then append the three lines below. Keep exact pins: the platform installs +# with pip and pinned versions override its pre-installed packages. +fsspec-databricks==0.2.1 +databricks-sql-connector>=4.0 +.