homelab/stack
1
0
Fork 0
Code Issues 131 Pull Requests 1 Actions Packages Projects 1 Releases Wiki Activity

docs(bcda): playbook for running the manager as a Databricks App without catalog rights

Browse Source 
cloud/databricks/README.md: what the platform guarantees (ephemeral
container, Python 3.11 default → uv pin 3.12, app.yaml command/env with
valueFrom, injected service-principal credentials, user authorization via
X-Forwarded-Access-Token and opt-in scopes, Medium/Large compute, 15 s
SIGTERM, 10 MB per source file, restricted egress policies), the
permission split for a user who cannot create or manage catalogs (admin
provisions catalog/schema/volume/warehouse/secrets once; attaching a
uc_securable grants USE CATALOG/USE SCHEMA automatically; only
CREATE TABLE/MODIFY/SELECT on the schema need a schema manager), the
compose → Databricks mapping (store on a UC volume via fsspec, Delta
tables through the SQL warehouse, no DuckDB file, no second port), the
eight small code changes the port needs, optimisation notes (app as
control plane, size per phase, cache statics, one connection per
session, never pull a table into the app, sync only what the app
imports), a runbook, phases 1–4 and the open questions to verify in the
target workspace. Sources listed.

cloud/databricks/bcda-app/: a bundle of its own (the main bundle creates
schemas and a volume, which needs catalog rights) with the app resource,
its warehouse/secret/volume resources and permissions, app.yaml,
.python-version and a requirements stub.

refs #1139
This commit is contained in:
kert
2026-10-25 16:19:35 -04:00
parent a05cc51916
commit 8cd4633dac
5 changed files with 490 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

351
cloud/databricks/README.md Normal file
View File

@@ -0,0 +1,351 @@
# BCDA manager as a Databricks App — playbook
The manager (`src/bcda/ui`, `stack bcda ui`) is a Streamlit app that today
runs as a compose service behind Traefik with a local store under
`data/bcda/` and DuckDB as its warehouse. This playbook makes the same code
run as a **Databricks App** for a workspace user who **cannot create or
manage catalogs**. Everything below is organised around that constraint.
Scaffold: `cloud/databricks/bcda-app/` (`databricks.yml`, `app.yaml`,
`requirements.txt`, `.python-version`). Nothing in it creates a catalog,
schema, volume or warehouse; it only *references* objects an admin
provisioned once and asks for grants to the app's own service principal.
## 0 · TL;DR
| Topic | Decision |
|---|---|
| Deployment unit | A **separate bundle** (`cloud/databricks/bcda-app/databricks.yml`) with one `apps` resource. The main `bundle/` (jobs, schemas, volumes) stays admin-only: it *creates* schemas and volumes, which needs rights the app user does not have. |
| Identity | **App service principal** for background work (pull, flatten, reconcile, load). **User authorization (on-behalf-of-user)** for the Data page, so Unity Catalog grants, row filters and column masks of the signed-in user apply. |
| Store | A **Unity Catalog volume** reached through fsspec (`dbfs:/Volumes/<catalog>/bcda/bcda_store`). `Store` and `flatten_export` already take any fsspec URI. Local disk in an app is ephemeral. |
| Tables | `bcda.*`, `input_layer.*` land as Delta tables in `<catalog>.bcda` through the **SQL warehouse** (`CREATE OR REPLACE TABLE … AS SELECT * FROM parquet.\`/Volumes/…\``). No DuckDB file, no replica. |
| Reconciliation | Phase 1: run `bcda.sync` as is, over partitions copied to the app's ephemeral disk (`flat/` is 62 MB for the Large entity). Phase 2: port `_recompute` to the warehouse (the SQL is window functions + `UNION ALL BY NAME`, both supported by Databricks SQL). |
| Heavy work | The EOB baseline (2.2 GB ndjson, 6 min on the sandbox) is CPU and network work the app runtime is "optimized for UI rendering, not data processing" for. Run it in the app first (Large size), then move it to a **Lakeflow job** the app triggers and watches (phase 3). |
| Compute | **Medium** (2 vCPU / 6 GB, the default) for the UI; **Large** (4 vCPU / 12 GB) only while a baseline pull runs. Stopped apps cost nothing. |
| Network | BCDA is an external API. If the workspace runs a *restricted* serverless network policy, `sandbox.bcda.cms.gov` and `api.bcda.cms.gov` must be allow-listed, plus the Databricks and PyPI domains every app needs. |
| Python | Apps default to **3.11**; the repo requires **≥ 3.12**. The scaffold pins 3.12 through uv (`.python-version` + `pyproject`), which the platform honours for uv projects. |
## 1 · Platform facts the design rests on
Verified against the Databricks docs on 2026-10-25 (links in §9):
- An app is a containerised service on serverless compute. **In-memory
state and local files do not survive a restart or redeploy.**
- Runtime: Ubuntu 22.04, Python 3.11 in a dedicated venv, uv 0.10.2.
Dependencies from `requirements.txt` (pip) or `pyproject.toml` (uv);
pinned versions override pre-installed packages. No `apt`.
- `app.yaml`: `command` (a list, **not run through a shell**; only
`DATABRICKS_APP_PORT` is substituted) and `env` entries with `value` or
`valueFrom: <resource key>`. Streamlit gets `STREAMLIT_SERVER_PORT`,
`STREAMLIT_SERVER_ADDRESS=0.0.0.0`, `STREAMLIT_SERVER_HEADLESS=true`,
`STREAMLIT_BROWSER_GATHER_USAGE_STATS=false` set for it.
- Injected: `DATABRICKS_HOST`, `DATABRICKS_APP_NAME`,
`DATABRICKS_WORKSPACE_ID`, `DATABRICKS_APP_PORT`,
`DATABRICKS_CLIENT_ID` / `DATABRICKS_CLIENT_SECRET` (the app's service
principal; the SDK picks them up). With app telemetry on:
`OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4314`, `OTEL_SERVICE_NAME`.
- App resources (declared on the app, exposed by `valueFrom`): SQL
warehouse → id (Can use / Can manage); secret → decrypted value (Can
read / write / manage); volume → `/Volumes/<catalog>/<schema>/<volume>`
(Can read / Can read and write); job → id (Can view / Can manage run /
Can manage); table → full name (Select / Modify); Lakebase, serving,
Genie, experiment, function, connection, vector-search index. **Apps
must use existing resources and can't create new ones**; adding a
resource needs *Can manage* on both the resource and the app.
- Two identities: the per-app **service principal** (shared by all users;
needs explicit grants on the warehouse *and* each UC object) and
**user authorization** (the user's SSO token forwarded as
`X-Forwarded-Access-Token`; in Streamlit:
`st.context.headers.get("X-Forwarded-Access-Token")`). Scopes are
opt-in per app (`sql`, `sql:restricted-query`, `files`,
`catalog.tables`, …; default is identity-only). A workspace admin can
restrict or disable the allowed scopes.
- Compute: Medium up to 2 vCPU / 6 GB (default), Large up to 4 vCPU /
12 GB (`compute_size: MEDIUM | LARGE | XLARGE` in the bundle, XLARGE
depending on workspace configuration); resizing keeps the old size
until the update finishes.
- Platform contract: listen on `0.0.0.0:$DATABRICKS_APP_PORT`, exit
within **15 s of SIGTERM** (else SIGKILL), speak H2C (TLS is theirs),
log to **stdout/stderr** (not local files).
- Limits: 100 apps per workspace; app source files **≤ 10 MB each**;
Files API individual file 500 MB (workspace files row; volumes
undocumented on that page).
- Sharing: `CAN_USE` / `CAN_MANAGE` on the app; users must be in the
Databricks account.
- Deploy: UI, `databricks apps deploy <name> --source-code-path …`
(after `databricks sync`), Git-backed (`--git-branch`), or a bundle
(`databricks bundle deploy` + `bundle run`). The Python
`databricks-bundles` package pinned here (0.295.0) has **no `apps`
module**, so the app resource is YAML.
- Local disk size inside the container is **not documented**. Treat it as
scratch for one job's partitions, not for the raw exports.
## 2 · Permissions: what the app user can and cannot do
The user deploying and running the app has **no `CREATE CATALOG` and does
not own a catalog**. Unity Catalog grants are explicit — nothing is
implied by a parent grant or by ownership of something else — so the split
is:
### 2.1 Once, by a metastore/catalog admin
```sql
-- an existing catalog, e.g. the dev target in stack.toml
USE CATALOG aco_dev;
CREATE SCHEMA IF NOT EXISTS bcda COMMENT 'BCDA manager: store volume + bcda.* tables';
CREATE VOLUME IF NOT EXISTS bcda.bcda_store COMMENT 'Store root: jobs/files ledgers, exports/, flat/';
-- secrets: scope `stack` already mapped in stack.toml [databricks.secrets]
-- databricks secrets create-scope stack
-- databricks secrets put-secret stack bcda-client-id
-- databricks secrets put-secret stack bcda-client-secret
```
The app's service principal does not exist until the app does. After the
first `databricks bundle deploy`, read its id from the app's
**Authorization** tab (or `databricks apps get bcda-manager`), then:
```sql
-- <SP> = the app service principal's application id
GRANT USE CATALOG ON CATALOG aco_dev TO `<SP>`;
GRANT USE SCHEMA, CREATE TABLE,
MODIFY, SELECT ON SCHEMA aco_dev.bcda TO `<SP>`;
GRANT READ VOLUME, WRITE VOLUME ON VOLUME aco_dev.bcda.bcda_store TO `<SP>`;
-- the connector writes input_layer.* too
GRANT USE SCHEMA, CREATE TABLE, MODIFY, SELECT ON SCHEMA aco_dev.input_layer TO `<SP>`;
```
Plus, outside SQL: `CAN USE` on the SQL warehouse, `READ` on the two
secrets, and (phase 3) `CAN MANAGE RUN` on the ingest job. The bundle's
`resources:` block asks for exactly these on the app and the deploy
attaches them. Attaching a volume, table or function as a `uc_securable`
resource also grants the app `USE CATALOG` and `USE SCHEMA` on its
parents automatically (bundle resources reference), so of the SQL above
only the schema-level `CREATE TABLE, MODIFY, SELECT` (and the
`input_layer` block) still need someone who holds `MANAGE` on the schema
or owns it.
For the people who will *use* the app (the Data page runs as them):
```sql
GRANT USE CATALOG ON CATALOG aco_dev TO `bcda-analysts`;
GRANT USE SCHEMA, SELECT ON SCHEMA aco_dev.bcda TO `bcda-analysts`;
GRANT USE SCHEMA, SELECT ON SCHEMA aco_dev.input_layer TO `bcda-analysts`;
```
### 2.2 What the app user needs
- `CAN_MANAGE` on the app (the bundle creates it as the deployer).
- `CAN USE` on the warehouse, `READ` on the secrets, `READ VOLUME` /
`WRITE VOLUME` on the volume: required to *attach* them as app
resources (the platform checks *Can manage* on the resource when you
add it in the UI; via the bundle the deployer must at least be able to
reference them — confirm in your workspace, policies vary).
- Nothing on the catalog beyond `USE CATALOG`.
### 2.3 What not to run
- `stack lake deploy --target databricks` / the main `bundle/`: both
create schemas and a volume (`bundle/resources/__init__.py`), which
needs `CREATE SCHEMA` / `CREATE VOLUME` on the catalog.
- `dev/scripts/setup_unity_catalog.py` without an admin: it creates
schemas (the catalog itself must already exist — the script says so).
## 3 · Architecture mapping
| Compose service today | Databricks App |
|---|---|
| Traefik + oauth2-proxy (Gitea SSO) | Platform SSO; share with `CAN_USE`. No Traefik route, no DNS, no cert. |
| `./data/bcda` bind mount (`Store`, `flat/`, logs) | UC volume `aco_dev.bcda.bcda_store` via fsspec-databricks (`dbfs:/Volumes/aco_dev/bcda/bcda_store`). |
| `BCDA_CLIENT_ID/SECRET` from `.env` | `valueFrom: secret` (scope `stack`, keys `bcda-client-id`, `bcda-client-secret`). |
| DuckDB `aco.duckdb` + replica | Delta tables in `aco_dev.bcda` / `aco_dev.input_layer`, written and read through the SQL warehouse. |
| `bcda.sync` DuckDB recompute over `flat/history/*` | Phase 1 unchanged on ephemeral disk; phase 2 on the warehouse. |
| Connector pipeline (`bcda.pipe.cclf`) via `DuckDBContext` | `EnterpriseContext` (dialect `databricks`) or the transpiled SQL under `bundle/sql/` — the same expressions already generate Databricks SQL. |
| `/metrics` on :8502 scraped by Prometheus | App telemetry: when enabled the runtime sets `OTEL_EXPORTER_OTLP_ENDPOINT`; `perf` already speaks OTLP. No second port. |
| `data/bcda/log.jsonl` | stdout (the platform captures it); keep JSON lines. |
| `docker compose up -d bcda` | `databricks bundle deploy && databricks bundle run bcda_manager`. |
### Why the store goes on a volume, not Lakebase or workspace files
The store is files: ndjson parts up to hundreds of MB, Parquet partitions,
two append-only Parquet ledgers and a JSON manifest per job. Volumes are
the governed file surface in Unity Catalog and the app can get exactly the
`READ VOLUME`/`WRITE VOLUME` it needs on one volume. `/Volumes` is **not
mounted** inside an app container (community reports; no official mount),
so access goes through the Files API — `fsspec-databricks` wraps it with
Databricks unified auth, which is what the injected service-principal
credentials provide. It is a one-maintainer beta package (0.2.1,
2026-10-23); pin it, and keep the Files API via `WorkspaceClient.files` as
the fallback for the few operations the store needs (`put_file`, `open`,
`exists`, `mkdirs`, `glob`).
## 4 · Code changes the playbook assumes (small, listed so they are not surprises)
1. **Store root from the environment.** `service.store_path()` reads
`conf.path("storage.bcda")`, which is a local `Path`. Add
`BCDA_STORE_URI` (any fsspec URI) that wins when set; `make_store`
already accepts a URI and `storage_options`. `app.yaml` sets it from
the volume resource (`valueFrom: bcda_store` gives `/Volumes/…`;
prefix `dbfs:` in code or set the value directly).
2. **Register the fsspec driver at import** when `BCDA_STORE_URI` starts
with `dbfs:` (`import fsspec_databricks; fsspec_databricks.use()`).
3. **`bcda.sync` on a remote store (phase 1).** `_root(store)` assumes a
local path. Give `apply()` a `workdir` (ephemeral `/tmp/bcda-sync`):
copy the needed `flat/history/<key>/*.parquet` partitions and the
current `flat/*.parquet` down, run the DuckDB recompute there, copy
the new current tables and the ledger back. 62 MB for the Large
entity; keep the raw exports on the volume only.
4. **Loader target.** `load_bcda` writes DuckDB. Add a `databricks`
path that runs, per table, on the warehouse:
`CREATE OR REPLACE TABLE aco_dev.bcda.<table> AS SELECT * FROM
parquet.\`/Volumes/aco_dev/bcda/bcda_store/flat/<table>.parquet\``,
then the connector via `EnterpriseContext`. No replica publish.
5. **Data page through the user's token.** `replica_connect()` returns a
DuckDB connection; on Databricks return a `databricks.sql` connection
with `access_token=st.context.headers["X-Forwarded-Access-Token"]`
and the warehouse id from `DATABRICKS_WAREHOUSE_ID`. The page already
caps results (`run_query(limit=1000)`). Requested scopes: `sql`
(or `sql:restricted-query` for read-only).
6. **Logging to stdout** when `DATABRICKS_APP_NAME` is set
(`bcda.log.setup` currently writes `data/bcda/log.jsonl`).
7. **Metrics**: skip `metrics.serve()` (no second port); leave
`STACK_TELEMETRY` to the OTLP exporter when app telemetry is on.
8. **Graceful stop**: the runner's threads are daemon threads; long runs
must survive a redeploy by being resumable — they are
(`Store.resume`, `stack bcda resume`), and the Sync page re-applies
anything downloaded but not reconciled.
## 5 · Optimisation notes (what actually matters on this runtime)
- **Keep the app a control plane.** The runtime is sized for UI. The
baseline EOB pull is 192 files / 2.2 GB and the flattener streams in
2,000-resource batches, so memory stays flat, but it is minutes of CPU
and network in a 2-vCPU container with other users' sessions on it.
Phase 3 moves `export → flatten → reconcile → load` into a Lakeflow
job (the main bundle already builds wheel tasks for `stack` commands)
that the app triggers with `CAN MANAGE RUN` and polls — the docs' own
"start, then poll" pattern for long operations.
- **Size for the phase you are in.** Medium for day-to-day; switch to
Large for the first baseline if it runs in-app, then back. The
Exports page's one-job-per-type default also keeps each run small.
- **Cache what is static.** Dictionary and crosswalk frames
(`dictionary_frames`) and `vendor_status` are per-process constants:
wrap with `st.cache_data`. Server job lists change: `ttl=60`.
- **One connection per session, not per widget.** Create the SQL
connector once per session (`st.session_state`), reuse it; the user's
forwarded token can change between sessions, so key the cache on the
token's hash and re-read the header every run (a community thread
reports stale headers after a logout/login in the same tab).
- **Serverless warehouse, small, auto-stop.** `stack.toml` already says
`SMALL`, `auto_stop_mins = 10`. All heavy SQL (reconcile phase 2,
loads, connector) runs there, not in the app.
- **Never pull a table into the app.** The Data page already limits
rows; the Sync and Store pages read ledgers of a few hundred rows.
Keep `st.dataframe` on Polars frames; avoid `collect()` on anything
that came from the warehouse unfiltered.
- **Startup.** Imports in `service.py` are already lazy; keep
`metrics.serve()` and vendor verification out of the import path.
Pre-compute nothing at boot.
- **Egress policy dry-run first.** If the workspace enforces a
restricted network policy, test with the policy's dry-run mode: an
app that cannot reach `sandbox.bcda.cms.gov` fails only when a user
presses *Start export*.
- **Source size.** Sync only what the app imports: `src/`,
`pyproject.toml`, `uv.lock`, `stack.toml`, `dev/seeds/bcda_v3_*.csv`
(dictionary page), `assets/icons/fav32.png`. Not `data/`, `dev/vendor`
(13 MB, has a 1.5 MB xlsx and gzip fixtures), `dev/seeds` as a whole
(157 MB; two ndjson files exceed the 10 MB per-file limit), `tuva/`,
`infra/`. The scaffold's `sync.exclude` does this.
- **Cost.** A stopped app costs nothing; running is billed per hour of
compute at the chosen size. Stop it outside working hours if nobody
uses it; the store is on the volume, nothing is lost.
## 6 · Runbook
```bash
# 0 · prerequisites (admin): §2.1 schema, volume, secrets, warehouse; you: CLI ≥ 0.229 with OAuth U2M
databricks auth login --host https://<workspace>.cloud.databricks.com
# 1 · set the three ids the scaffold references
cd cloud/databricks/bcda-app
export BCDA_WAREHOUSE_ID=<sql warehouse id> # databricks warehouses list
# catalog/schema/volume names are variables in databricks.yml (defaults: aco_dev / bcda / bcda_store)
# 2 · dependencies file (regenerate when uv.lock changes)
(cd ../../.. && uv export --no-dev --no-hashes --extra bcda --extra aco --extra conf \
-o cloud/databricks/bcda-app/requirements.txt && echo "fsspec-databricks==0.2.1" >> cloud/databricks/bcda-app/requirements.txt \
&& echo "databricks-sql-connector>=4.0" >> cloud/databricks/bcda-app/requirements.txt && echo "." >> cloud/databricks/bcda-app/requirements.txt)
# 3 · validate, deploy (creates the app + its service principal), start
databricks bundle validate -t dev
databricks bundle deploy -t dev
databricks bundle run bcda_manager -t dev
# 4 · UC grants to the new service principal (admin, §2.1), then user authorization scopes
databricks apps get bcda-manager-dev | jq .service_principal_client_id
# App → Authorization → + Add scope: sql (and files if the Data page should read the volume as the user)
# 5 · share
# App → Permissions → bcda-analysts: CAN_USE (or the permissions: block in databricks.yml)
# 6 · first run
# Connect → preset/production creds from secrets → Exports: baseline, one job per type (Large size for this)
# Sync → Reconcile now → Data page queries as you (OBO)
# 7 · logs and health
databricks apps logs bcda-manager-dev # or the app's Logs tab
```
Redeploy = `databricks bundle deploy` again; in-flight runs die with the
container, so finish them with `stack bcda resume <key>` from the Store
page's key list (or wait for the next apply, which skips nothing).
## 7 · Phases
| Phase | Scope | Done when |
|---|---|---|
| 1 | Store on the volume, local (ephemeral) reconcile, loader to Delta via warehouse, Data page via OBO, logs to stdout | A baseline and a `_since` delta round-trip from the app; `aco_dev.bcda.sync_ledger` shows both; Data page queries succeed for a `bcda-analysts` member who has no catalog rights |
| 2 | `bcda.sync._recompute` as warehouse SQL over `parquet.\`/Volumes/…\`` partitions; keep the same ledger and invariants | `tests/bcda/test_sync.py` scenarios pass against a warehouse (parity test), app no longer copies partitions |
| 3 | Ingest as a Lakeflow job triggered from the Exports/Sync pages; app only polls | Baseline pull runs with the app on Medium; job run id recorded on the store job row |
| 4 | Telemetry to app OTLP; audit records of user actions (who started what) as structured logs | Traces/logs visible under the app's telemetry |
## 8 · Open questions to verify in the target workspace
- Whether `sync.paths` / `source_code_path` may point above the bundle
root (`../../..`) in your CLI version. If not, move
`cloud/databricks/bcda-app/databricks.yml` to a thin wrapper at the
repo root under another bundle name, or build a wheel and sync only
`app.yaml` + `requirements.txt` that installs it.
- Allowed OAuth scopes for apps (admin setting) — if `None`, the Data
page must run as the service principal and every analyst needs
`CAN_USE` on the app only; UC filters then do not apply.
- Serverless network policy mode and whether `*.bcda.cms.gov` is allowed.
- Whether the deploying user can attach the warehouse/secret/volume
resources without `CAN MANAGE` on them (the UI requires it; bundles may
as well).
- Ephemeral disk headroom for phase 1 (62 MB of partitions is safe; a
full raw EOB export is not meant to touch local disk).
- `fsspec-databricks` write path on your cloud (AWS is listed as
untested by the maintainer) — fall back to `WorkspaceClient.files` if
multipart uploads misbehave.
## 9 · Sources
- Databricks Apps key concepts — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/key-concepts
- Apps environment (runtime, env vars) — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/system-env
- Configure app execution (`app.yaml`) — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/app-runtime
- Environment variables / `valueFrom` — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/environment-variables
- Add resources to an app — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/resources
- Authorization (service principal, user authorization, scopes) — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/auth
- Compute size — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/compute-size
- Best practices — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/best-practices
- Networking / serverless network policies — https://docs.databricks.com/gcp/en/dev-tools/databricks-apps/networking , https://learn.microsoft.com/en-in/Azure/databricks/security/network/serverless-network-security/network-policies
- Deploy — https://docs.databricks.com/aws/en/dev-tools/databricks-apps/deploy
- Resource limits — https://docs.databricks.com/aws/en/resources/limits
- Bundles: apps tutorial and resources reference — https://docs.databricks.com/aws/en/dev-tools/bundles/apps-tutorial , https://docs.databricks.com/aws/en/dev-tools/bundles/resources
- Unity Catalog privileges — https://docs.databricks.com/aws/en/data-governance/unity-catalog/access-control/privileges-reference , https://docs.databricks.com/aws/en/data-governance/unity-catalog/access-control/permissions-concepts
- Volume files / Files API — https://docs.databricks.com/aws/files/volumes , https://docs.databricks.com/api/files/v2/file
- fsspec-databricks — https://pypi.org/project/fsspec-databricks/
- Community reports used only as caveats: `/Volumes` not mounted in apps (https://community.databricks.com/t5/administration-architecture/how-to-access-unitycatalog-s-volume-inside-databricks-app/m-p/134845), stale `X-Forwarded-Access-Token` after re-login (https://community.databricks.com/t5/administration-architecture/x-forwarded-access-token-occasionally-return-stale-values-from-a/td-p/163074)

1
cloud/databricks/bcda-app/.python-version Normal file
View File

@@ -0,0 +1 @@
3.12

27
cloud/databricks/bcda-app/app.yaml Normal file
View File

@@ -0,0 +1,27 @@
# Runtime manifest for the BCDA manager on Databricks Apps.
# The platform sets STREAMLIT_SERVER_PORT/ADDRESS/HEADLESS for Streamlit, so
# `python -m bcda.ui` needs no port argument; it passes the fhirworx theme
# file as --theme.base itself. Commands are not run through a shell.
command: ["python", "-m", "bcda.ui"]
env:
# Store root on the Unity Catalog volume (fsspec-databricks: dbfs:/Volumes/…).
# valueFrom: bcda_store would give the bare /Volumes path; see README §4.1.
- name: BCDA_STORE_URI
value: "dbfs:/Volumes/aco_dev/bcda/bcda_store"
- name: BCDA_CLIENT_ID
valueFrom: bcda_client_id
- name: BCDA_CLIENT_SECRET
valueFrom: bcda_client_secret
- name: BCDA_BASE_URL
value: "https://sandbox.bcda.cms.gov"
- name: DATABRICKS_WAREHOUSE_ID
valueFrom: sql_warehouse
- name: BCDA_TARGET_CATALOG
value: "aco_dev"
- name: BCDA_TARGET_SCHEMA
value: "bcda"
# No second port on Apps; metrics go to app telemetry (OTLP) when enabled.
- name: STACK_TELEMETRY
value: "false"
- name: BCDA_METRICS_PORT
value: "0"

104
cloud/databricks/bcda-app/databricks.yml Normal file
View File

@@ -0,0 +1,104 @@
# BCDA manager as a Databricks App — a bundle of its own.
#
# Deliberately separate from the repo's main bundle (bundle/, generated by
# dev/scripts/gen_config.py), which CREATES schemas and a volume and so needs
# catalog-level rights. This one only references objects an admin provisioned
# once (cloud/databricks/README.md §2.1) and asks for grants to the app's
# service principal. Deploy from this directory:
#
# databricks bundle validate -t dev && databricks bundle deploy -t dev && databricks bundle run bcda_manager -t dev
bundle:
name: bcda-manager
variables:
catalog:
description: Existing catalog (no CREATE CATALOG needed)
default: aco_dev
schema:
description: Existing schema holding the store volume and bcda.* tables
default: bcda
volume:
description: Existing managed volume used as the store root
default: bcda_store
warehouse_id:
description: SQL warehouse the app may use
lookup:
warehouse: stack-sql-warehouse
secret_scope:
default: stack
users_group:
description: Group that may open the app (Data page runs as them)
default: bcda-analysts
# Only what the app imports. The whole repo is far over the 10 MB per-file
# limit (dev/seeds ndjson, uv.lock is fine) and most of it is irrelevant.
sync:
paths:
- ../../..
include:
- ../../../src/**
- ../../../pyproject.toml
- ../../../uv.lock
- ../../../stack.toml
- ../../../README.md
- ../../../dev/seeds/bcda_v3_*.csv
- ../../../assets/icons/fav32.png
- ./app.yaml
- ./requirements.txt
- ./.python-version
exclude:
- ../../../data/**
- ../../../dev/vendor/**
- ../../../tuva/**
- ../../../infra/**
- ../../../docs/**
- ../../../notebooks/**
- ../../../.venv/**
resources:
apps:
bcda_manager:
name: bcda-manager-${bundle.target}
description: "BCDA v3 manager — exports, store, _since reconciliation, load, crosswalk"
source_code_path: ../../..
# MEDIUM (2 vCPU / 6 GB) for the UI; switch to LARGE while a baseline
# pull runs in-app (README §5), back afterwards.
compute_size: MEDIUM
resources:
- name: sql_warehouse
sql_warehouse:
id: ${var.warehouse_id}
permission: CAN_USE
- name: bcda_client_id
secret:
scope: ${var.secret_scope}
key: bcda-client-id
permission: READ
- name: bcda_client_secret
secret:
scope: ${var.secret_scope}
key: bcda-client-secret
permission: READ
- name: bcda_store
uc_securable:
securable_type: VOLUME
securable_full_name: ${var.catalog}.${var.schema}.${var.volume}
permission: WRITE_VOLUME
# The Data page queries as the signed-in user; sql:restricted-query is
# enough for read-only browsing. Admins may restrict allowed scopes.
user_api_scopes:
- sql:restricted-query
permissions:
- group_name: ${var.users_group}
level: CAN_USE
targets:
dev:
mode: development
default: true
prod:
mode: production
variables:
catalog: aco
workspace:
root_path: /Workspace/Shared/.bundle/${bundle.name}/${bundle.target}

7
cloud/databricks/bcda-app/requirements.txt Normal file
View File

@@ -0,0 +1,7 @@
# Regenerate from the lock (README §6 step 2):
# uv export --no-dev --no-hashes --extra bcda --extra aco --extra conf -o cloud/databricks/bcda-app/requirements.txt
# then append the three lines below. Keep exact pins: the platform installs
# with pip and pinned versions override its pre-installed packages.
fsspec-databricks==0.2.1
databricks-sql-connector>=4.0
.