docs: document Databricks SDK integration in README
Some checks failed
CI / skinny-install (aco) (push) Successful in 52s
CI / lint-test (push) Successful in 1m16s
CI / skinny-install (api) (push) Successful in 24s
CI / skinny-install (bcda) (push) Successful in 28s
CI / skinny-install (bib) (push) Successful in 34s
CI / skinny-install (bls) (push) Successful in 27s
CI / skinny-install (ccw) (push) Successful in 34s
CI / skinny-install (cli) (push) Successful in 37s
CI / skinny-install (cms) (push) Successful in 26s
CI / skinny-install (conf) (push) Successful in 29s
CI / skinny-install (opps) (push) Successful in 38s
CI / skinny-install (perf) (push) Successful in 31s
CI / skinny-install (pfs) (push) Successful in 33s
CI / skinny-install (rex) (push) Successful in 31s
CI / skinny-install (aco) (pull_request) Successful in 46s
CI / lint-test (pull_request) Successful in 1m9s
CI / skinny-install (api) (pull_request) Successful in 30s
CI / skinny-install (bcda) (pull_request) Successful in 29s
CI / skinny-install (bib) (pull_request) Successful in 24s
CI / skinny-install (bls) (pull_request) Successful in 31s
CI / skinny-install (ccw) (pull_request) Successful in 30s
CI / skinny-install (cli) (pull_request) Successful in 34s
CI / skinny-install (cms) (pull_request) Successful in 29s
CI / skinny-install (conf) (pull_request) Successful in 28s
CI / skinny-install (opps) (pull_request) Successful in 31s
Infra CI / zotero (pull_request) Successful in 6s
Infra CI / docs (pull_request) Failing after 6s
CI / skinny-install (perf) (pull_request) Successful in 27s
CI / skinny-install (pfs) (pull_request) Successful in 34s
Infra CI / notebooks (pull_request) Successful in 6s
CI / skinny-install (rex) (pull_request) Successful in 27s
Infra CI / mc (pull_request) Successful in 7s
Infra CI / api (pull_request) Successful in 9s

Add SDK integration section to README.md.j2 with:
- 18-service table showing category, SDK service, module, purpose
- Governance example (declarative grants, drift audit, PHI schemas)
- Job orchestration example (create, run, poll)
- Secret sync usage
- Quality monitoring usage
- stack.toml config snippets

Regenerate README.md (12,320 tests, 25 services, 14 pipelines).
This commit is contained in:
kert
2026-03-26 20:02:16 -04:00
parent 97852e818c
commit 62b1771b6a
2 changed files with 159 additions and 3 deletions

View File

@@ -2,7 +2,7 @@
![coverage](assets/icons/coverage.svg)
Healthcare analytics platform on self-hosted infrastructure. Replaces dbt SQL models with narwhals DataFrame-agnostic expression functions, backed by DuckDB locally and Iceberg/Trino in the lakehouse. 25 services, 14 data pipelines, 12,291 tests at 99% coverage.
Healthcare analytics platform on self-hosted infrastructure. Replaces dbt SQL models with narwhals DataFrame-agnostic expression functions, backed by DuckDB locally and Iceberg/Trino in the lakehouse. 25 services, 14 data pipelines, 12,320 tests at 99% coverage.
## Quick start
@@ -264,6 +264,84 @@ uv run python dev/scripts/gen_config.py --dab-sql # generate SQL + DDL (~4 min
databricks bundle deploy -t dev # deploy to Databricks
```
### SDK integration
The platform uses 18 Databricks SDK services via `UnityClient` (wrapping `WorkspaceClient`):
| Category | Services | Module | Purpose |
|----------|----------|--------|---------|
| **Catalog CRUD** | `catalogs`, `schemas`, `tables`, `volumes` | `lake/unity.py` | Unity Catalog object management |
| **Data loading** | `files`, `statement_execution` | `lake/sync.py` | Parquet upload + COPY INTO |
| **Governance** | `grants` | `lake/governance.py` | Declarative schema permissions with drift audit |
| **Secrets** | `secrets` | `lake/unity.py` | Scope management, env var sync |
| **Constraints** | `table_constraints` | `lake/unity.py` | PK/FK on claims tables for lineage |
| **Jobs** | `jobs` | `lake/jobs.py` | Create/run/monitor pipeline jobs from Python |
| **Compute** | `warehouses` | `lake/unity.py` | Find/create/start/stop warehouses by name |
| **Quality** | `quality_monitors` | `lake/quality.py` | Lakehouse Monitoring profiles |
| **Audit** | `system_schemas` | `lake/unity.py` | Table lineage and audit log queries |
### Governance
`src/aco/lake/governance.py` provides declarative grant management:
```python
from aco.lake.governance import GovernancePolicy, GrantRule, apply_governance
policy = GovernancePolicy(
catalog="aco",
rules=[
GrantRule(group="analysts", schemas=["readmissions"], privileges=["SELECT"]),
GrantRule(group="pipeline_svc", schemas=["*"], privileges=["ALL_PRIVILEGES"]),
],
phi_schemas=["core", "claims_preprocessing", "cclf"],
)
# Preview changes
apply_governance(client, policy, dry_run=True)
# Detect drift between declared and actual grants
from aco.lake.governance import audit_governance
drifts = audit_governance(client, policy)
```
PHI schemas, group privilege mappings, and quality monitor config are declared in `stack.toml`:
```toml
[databricks.governance]
phi_schemas = ["core", "claims_preprocessing", "cclf", "input_layer"]
[databricks.governance.groups.analysts]
schemas = ["readmissions", "quality_measures"]
privileges = ["SELECT"]
```
### Job orchestration
`src/aco/lake/jobs.py` translates `Pipeline` objects into Databricks Jobs:
```python
from aco.lake.jobs import JobManager
mgr = JobManager(client, catalog="aco")
job_id = mgr.create_pipeline_job("readmissions", schedule="0 0 6 * * ?")
run_id = mgr.run_now(job_id)
status = mgr.get_run_status(run_id)
```
### Secret sync
```bash
uv run python dev/scripts/sync_secrets.py --dry-run # preview
uv run python dev/scripts/sync_secrets.py # push to Databricks
```
### Quality monitoring
```python
from aco.lake.quality import setup_monitors
setup_monitors(client, catalog="aco", schemas=["core", "claims_preprocessing"])
```
## Testing
### Architecture
@@ -337,7 +415,7 @@ The hook classifies staged files and runs only what is relevant:
| `src/sem/*.py` | `tests/sem/` + `test_ast_coverage.py` | Module tests + structural invariants |
| `src/aco/*.py` + `src/sem/*.py` | `tests/aco/` + `tests/sem/` + structural | Both module test dirs |
| `tests/bib/test_sync.py` | `tests/bib/` | Changed test dir |
| `pyproject.toml` or `conftest.py` | Full suite (12,291+ tests) | Infrastructure change |
| `pyproject.toml` or `conftest.py` | Full suite (12,320+ tests) | Infrastructure change |
| `notebooks/pfs_calcs.py` | `marimo check` + notebook execution | Notebook validation only |
| `README.md` only | Nothing | No testable changes |
@@ -416,7 +494,7 @@ stack/
│ ├── hooks/ Git hooks (tracked, core.hooksPath = dev/hooks)
│ ├── seeds/ Reference data (BCDA samples, CMS docs)
│ └── pipelines/ CI-agnostic pipeline specs
├── tests/ 12,291 tests at 99% coverage
├── tests/ 12,320 tests at 99% coverage
├── notebooks/ Marimo notebooks
├── docs/ Docusaurus site
├── data/ DuckDB, bib.sqlite, BCDA/CMS data, zotero (gitignored)

View File

@@ -257,6 +257,84 @@ uv run python dev/scripts/gen_config.py --dab-sql # generate SQL + DDL (~4 min
databricks bundle deploy -t dev # deploy to Databricks
```
### SDK integration
The platform uses 18 Databricks SDK services via `UnityClient` (wrapping `WorkspaceClient`):
| Category | Services | Module | Purpose |
|----------|----------|--------|---------|
| **Catalog CRUD** | `catalogs`, `schemas`, `tables`, `volumes` | `lake/unity.py` | Unity Catalog object management |
| **Data loading** | `files`, `statement_execution` | `lake/sync.py` | Parquet upload + COPY INTO |
| **Governance** | `grants` | `lake/governance.py` | Declarative schema permissions with drift audit |
| **Secrets** | `secrets` | `lake/unity.py` | Scope management, env var sync |
| **Constraints** | `table_constraints` | `lake/unity.py` | PK/FK on claims tables for lineage |
| **Jobs** | `jobs` | `lake/jobs.py` | Create/run/monitor pipeline jobs from Python |
| **Compute** | `warehouses` | `lake/unity.py` | Find/create/start/stop warehouses by name |
| **Quality** | `quality_monitors` | `lake/quality.py` | Lakehouse Monitoring profiles |
| **Audit** | `system_schemas` | `lake/unity.py` | Table lineage and audit log queries |
### Governance
`src/aco/lake/governance.py` provides declarative grant management:
```python
from aco.lake.governance import GovernancePolicy, GrantRule, apply_governance
policy = GovernancePolicy(
catalog="aco",
rules=[
GrantRule(group="analysts", schemas=["readmissions"], privileges=["SELECT"]),
GrantRule(group="pipeline_svc", schemas=["*"], privileges=["ALL_PRIVILEGES"]),
],
phi_schemas=["core", "claims_preprocessing", "cclf"],
)
# Preview changes
apply_governance(client, policy, dry_run=True)
# Detect drift between declared and actual grants
from aco.lake.governance import audit_governance
drifts = audit_governance(client, policy)
```
PHI schemas, group privilege mappings, and quality monitor config are declared in `stack.toml`:
```toml
[databricks.governance]
phi_schemas = ["core", "claims_preprocessing", "cclf", "input_layer"]
[databricks.governance.groups.analysts]
schemas = ["readmissions", "quality_measures"]
privileges = ["SELECT"]
```
### Job orchestration
`src/aco/lake/jobs.py` translates `Pipeline` objects into Databricks Jobs:
```python
from aco.lake.jobs import JobManager
mgr = JobManager(client, catalog="aco")
job_id = mgr.create_pipeline_job("readmissions", schedule="0 0 6 * * ?")
run_id = mgr.run_now(job_id)
status = mgr.get_run_status(run_id)
```
### Secret sync
```bash
uv run python dev/scripts/sync_secrets.py --dry-run # preview
uv run python dev/scripts/sync_secrets.py # push to Databricks
```
### Quality monitoring
```python
from aco.lake.quality import setup_monitors
setup_monitors(client, catalog="aco", schemas=["core", "claims_preprocessing"])
```
## Testing
### Architecture