docs: document Databricks SDK integration in README
Some checks failed
CI / skinny-install (aco) (push) Successful in 52s
CI / lint-test (push) Successful in 1m16s
CI / skinny-install (api) (push) Successful in 24s
CI / skinny-install (bcda) (push) Successful in 28s
CI / skinny-install (bib) (push) Successful in 34s
CI / skinny-install (bls) (push) Successful in 27s
CI / skinny-install (ccw) (push) Successful in 34s
CI / skinny-install (cli) (push) Successful in 37s
CI / skinny-install (cms) (push) Successful in 26s
CI / skinny-install (conf) (push) Successful in 29s
CI / skinny-install (opps) (push) Successful in 38s
CI / skinny-install (perf) (push) Successful in 31s
CI / skinny-install (pfs) (push) Successful in 33s
CI / skinny-install (rex) (push) Successful in 31s
CI / skinny-install (aco) (pull_request) Successful in 46s
CI / lint-test (pull_request) Successful in 1m9s
CI / skinny-install (api) (pull_request) Successful in 30s
CI / skinny-install (bcda) (pull_request) Successful in 29s
CI / skinny-install (bib) (pull_request) Successful in 24s
CI / skinny-install (bls) (pull_request) Successful in 31s
CI / skinny-install (ccw) (pull_request) Successful in 30s
CI / skinny-install (cli) (pull_request) Successful in 34s
CI / skinny-install (cms) (pull_request) Successful in 29s
CI / skinny-install (conf) (pull_request) Successful in 28s
CI / skinny-install (opps) (pull_request) Successful in 31s
Infra CI / zotero (pull_request) Successful in 6s
Infra CI / docs (pull_request) Failing after 6s
CI / skinny-install (perf) (pull_request) Successful in 27s
CI / skinny-install (pfs) (pull_request) Successful in 34s
Infra CI / notebooks (pull_request) Successful in 6s
CI / skinny-install (rex) (pull_request) Successful in 27s
Infra CI / mc (pull_request) Successful in 7s
Infra CI / api (pull_request) Successful in 9s
Some checks failed
CI / skinny-install (aco) (push) Successful in 52s
CI / lint-test (push) Successful in 1m16s
CI / skinny-install (api) (push) Successful in 24s
CI / skinny-install (bcda) (push) Successful in 28s
CI / skinny-install (bib) (push) Successful in 34s
CI / skinny-install (bls) (push) Successful in 27s
CI / skinny-install (ccw) (push) Successful in 34s
CI / skinny-install (cli) (push) Successful in 37s
CI / skinny-install (cms) (push) Successful in 26s
CI / skinny-install (conf) (push) Successful in 29s
CI / skinny-install (opps) (push) Successful in 38s
CI / skinny-install (perf) (push) Successful in 31s
CI / skinny-install (pfs) (push) Successful in 33s
CI / skinny-install (rex) (push) Successful in 31s
CI / skinny-install (aco) (pull_request) Successful in 46s
CI / lint-test (pull_request) Successful in 1m9s
CI / skinny-install (api) (pull_request) Successful in 30s
CI / skinny-install (bcda) (pull_request) Successful in 29s
CI / skinny-install (bib) (pull_request) Successful in 24s
CI / skinny-install (bls) (pull_request) Successful in 31s
CI / skinny-install (ccw) (pull_request) Successful in 30s
CI / skinny-install (cli) (pull_request) Successful in 34s
CI / skinny-install (cms) (pull_request) Successful in 29s
CI / skinny-install (conf) (pull_request) Successful in 28s
CI / skinny-install (opps) (pull_request) Successful in 31s
Infra CI / zotero (pull_request) Successful in 6s
Infra CI / docs (pull_request) Failing after 6s
CI / skinny-install (perf) (pull_request) Successful in 27s
CI / skinny-install (pfs) (pull_request) Successful in 34s
Infra CI / notebooks (pull_request) Successful in 6s
CI / skinny-install (rex) (pull_request) Successful in 27s
Infra CI / mc (pull_request) Successful in 7s
Infra CI / api (pull_request) Successful in 9s
Add SDK integration section to README.md.j2 with: - 18-service table showing category, SDK service, module, purpose - Governance example (declarative grants, drift audit, PHI schemas) - Job orchestration example (create, run, poll) - Secret sync usage - Quality monitoring usage - stack.toml config snippets Regenerate README.md (12,320 tests, 25 services, 14 pipelines).
This commit is contained in:
84
README.md
84
README.md
@@ -2,7 +2,7 @@
|
||||
|
||||

|
||||
|
||||
Healthcare analytics platform on self-hosted infrastructure. Replaces dbt SQL models with narwhals DataFrame-agnostic expression functions, backed by DuckDB locally and Iceberg/Trino in the lakehouse. 25 services, 14 data pipelines, 12,291 tests at 99% coverage.
|
||||
Healthcare analytics platform on self-hosted infrastructure. Replaces dbt SQL models with narwhals DataFrame-agnostic expression functions, backed by DuckDB locally and Iceberg/Trino in the lakehouse. 25 services, 14 data pipelines, 12,320 tests at 99% coverage.
|
||||
|
||||
## Quick start
|
||||
|
||||
@@ -264,6 +264,84 @@ uv run python dev/scripts/gen_config.py --dab-sql # generate SQL + DDL (~4 min
|
||||
databricks bundle deploy -t dev # deploy to Databricks
|
||||
```
|
||||
|
||||
### SDK integration
|
||||
|
||||
The platform uses 18 Databricks SDK services via `UnityClient` (wrapping `WorkspaceClient`):
|
||||
|
||||
| Category | Services | Module | Purpose |
|
||||
|----------|----------|--------|---------|
|
||||
| **Catalog CRUD** | `catalogs`, `schemas`, `tables`, `volumes` | `lake/unity.py` | Unity Catalog object management |
|
||||
| **Data loading** | `files`, `statement_execution` | `lake/sync.py` | Parquet upload + COPY INTO |
|
||||
| **Governance** | `grants` | `lake/governance.py` | Declarative schema permissions with drift audit |
|
||||
| **Secrets** | `secrets` | `lake/unity.py` | Scope management, env var sync |
|
||||
| **Constraints** | `table_constraints` | `lake/unity.py` | PK/FK on claims tables for lineage |
|
||||
| **Jobs** | `jobs` | `lake/jobs.py` | Create/run/monitor pipeline jobs from Python |
|
||||
| **Compute** | `warehouses` | `lake/unity.py` | Find/create/start/stop warehouses by name |
|
||||
| **Quality** | `quality_monitors` | `lake/quality.py` | Lakehouse Monitoring profiles |
|
||||
| **Audit** | `system_schemas` | `lake/unity.py` | Table lineage and audit log queries |
|
||||
|
||||
### Governance
|
||||
|
||||
`src/aco/lake/governance.py` provides declarative grant management:
|
||||
|
||||
```python
|
||||
from aco.lake.governance import GovernancePolicy, GrantRule, apply_governance
|
||||
|
||||
policy = GovernancePolicy(
|
||||
catalog="aco",
|
||||
rules=[
|
||||
GrantRule(group="analysts", schemas=["readmissions"], privileges=["SELECT"]),
|
||||
GrantRule(group="pipeline_svc", schemas=["*"], privileges=["ALL_PRIVILEGES"]),
|
||||
],
|
||||
phi_schemas=["core", "claims_preprocessing", "cclf"],
|
||||
)
|
||||
|
||||
# Preview changes
|
||||
apply_governance(client, policy, dry_run=True)
|
||||
|
||||
# Detect drift between declared and actual grants
|
||||
from aco.lake.governance import audit_governance
|
||||
drifts = audit_governance(client, policy)
|
||||
```
|
||||
|
||||
PHI schemas, group privilege mappings, and quality monitor config are declared in `stack.toml`:
|
||||
|
||||
```toml
|
||||
[databricks.governance]
|
||||
phi_schemas = ["core", "claims_preprocessing", "cclf", "input_layer"]
|
||||
|
||||
[databricks.governance.groups.analysts]
|
||||
schemas = ["readmissions", "quality_measures"]
|
||||
privileges = ["SELECT"]
|
||||
```
|
||||
|
||||
### Job orchestration
|
||||
|
||||
`src/aco/lake/jobs.py` translates `Pipeline` objects into Databricks Jobs:
|
||||
|
||||
```python
|
||||
from aco.lake.jobs import JobManager
|
||||
|
||||
mgr = JobManager(client, catalog="aco")
|
||||
job_id = mgr.create_pipeline_job("readmissions", schedule="0 0 6 * * ?")
|
||||
run_id = mgr.run_now(job_id)
|
||||
status = mgr.get_run_status(run_id)
|
||||
```
|
||||
|
||||
### Secret sync
|
||||
|
||||
```bash
|
||||
uv run python dev/scripts/sync_secrets.py --dry-run # preview
|
||||
uv run python dev/scripts/sync_secrets.py # push to Databricks
|
||||
```
|
||||
|
||||
### Quality monitoring
|
||||
|
||||
```python
|
||||
from aco.lake.quality import setup_monitors
|
||||
setup_monitors(client, catalog="aco", schemas=["core", "claims_preprocessing"])
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
### Architecture
|
||||
@@ -337,7 +415,7 @@ The hook classifies staged files and runs only what is relevant:
|
||||
| `src/sem/*.py` | `tests/sem/` + `test_ast_coverage.py` | Module tests + structural invariants |
|
||||
| `src/aco/*.py` + `src/sem/*.py` | `tests/aco/` + `tests/sem/` + structural | Both module test dirs |
|
||||
| `tests/bib/test_sync.py` | `tests/bib/` | Changed test dir |
|
||||
| `pyproject.toml` or `conftest.py` | Full suite (12,291+ tests) | Infrastructure change |
|
||||
| `pyproject.toml` or `conftest.py` | Full suite (12,320+ tests) | Infrastructure change |
|
||||
| `notebooks/pfs_calcs.py` | `marimo check` + notebook execution | Notebook validation only |
|
||||
| `README.md` only | Nothing | No testable changes |
|
||||
|
||||
@@ -416,7 +494,7 @@ stack/
|
||||
│ ├── hooks/ Git hooks (tracked, core.hooksPath = dev/hooks)
|
||||
│ ├── seeds/ Reference data (BCDA samples, CMS docs)
|
||||
│ └── pipelines/ CI-agnostic pipeline specs
|
||||
├── tests/ 12,291 tests at 99% coverage
|
||||
├── tests/ 12,320 tests at 99% coverage
|
||||
├── notebooks/ Marimo notebooks
|
||||
├── docs/ Docusaurus site
|
||||
├── data/ DuckDB, bib.sqlite, BCDA/CMS data, zotero (gitignored)
|
||||
|
||||
78
README.md.j2
78
README.md.j2
@@ -257,6 +257,84 @@ uv run python dev/scripts/gen_config.py --dab-sql # generate SQL + DDL (~4 min
|
||||
databricks bundle deploy -t dev # deploy to Databricks
|
||||
```
|
||||
|
||||
### SDK integration
|
||||
|
||||
The platform uses 18 Databricks SDK services via `UnityClient` (wrapping `WorkspaceClient`):
|
||||
|
||||
| Category | Services | Module | Purpose |
|
||||
|----------|----------|--------|---------|
|
||||
| **Catalog CRUD** | `catalogs`, `schemas`, `tables`, `volumes` | `lake/unity.py` | Unity Catalog object management |
|
||||
| **Data loading** | `files`, `statement_execution` | `lake/sync.py` | Parquet upload + COPY INTO |
|
||||
| **Governance** | `grants` | `lake/governance.py` | Declarative schema permissions with drift audit |
|
||||
| **Secrets** | `secrets` | `lake/unity.py` | Scope management, env var sync |
|
||||
| **Constraints** | `table_constraints` | `lake/unity.py` | PK/FK on claims tables for lineage |
|
||||
| **Jobs** | `jobs` | `lake/jobs.py` | Create/run/monitor pipeline jobs from Python |
|
||||
| **Compute** | `warehouses` | `lake/unity.py` | Find/create/start/stop warehouses by name |
|
||||
| **Quality** | `quality_monitors` | `lake/quality.py` | Lakehouse Monitoring profiles |
|
||||
| **Audit** | `system_schemas` | `lake/unity.py` | Table lineage and audit log queries |
|
||||
|
||||
### Governance
|
||||
|
||||
`src/aco/lake/governance.py` provides declarative grant management:
|
||||
|
||||
```python
|
||||
from aco.lake.governance import GovernancePolicy, GrantRule, apply_governance
|
||||
|
||||
policy = GovernancePolicy(
|
||||
catalog="aco",
|
||||
rules=[
|
||||
GrantRule(group="analysts", schemas=["readmissions"], privileges=["SELECT"]),
|
||||
GrantRule(group="pipeline_svc", schemas=["*"], privileges=["ALL_PRIVILEGES"]),
|
||||
],
|
||||
phi_schemas=["core", "claims_preprocessing", "cclf"],
|
||||
)
|
||||
|
||||
# Preview changes
|
||||
apply_governance(client, policy, dry_run=True)
|
||||
|
||||
# Detect drift between declared and actual grants
|
||||
from aco.lake.governance import audit_governance
|
||||
drifts = audit_governance(client, policy)
|
||||
```
|
||||
|
||||
PHI schemas, group privilege mappings, and quality monitor config are declared in `stack.toml`:
|
||||
|
||||
```toml
|
||||
[databricks.governance]
|
||||
phi_schemas = ["core", "claims_preprocessing", "cclf", "input_layer"]
|
||||
|
||||
[databricks.governance.groups.analysts]
|
||||
schemas = ["readmissions", "quality_measures"]
|
||||
privileges = ["SELECT"]
|
||||
```
|
||||
|
||||
### Job orchestration
|
||||
|
||||
`src/aco/lake/jobs.py` translates `Pipeline` objects into Databricks Jobs:
|
||||
|
||||
```python
|
||||
from aco.lake.jobs import JobManager
|
||||
|
||||
mgr = JobManager(client, catalog="aco")
|
||||
job_id = mgr.create_pipeline_job("readmissions", schedule="0 0 6 * * ?")
|
||||
run_id = mgr.run_now(job_id)
|
||||
status = mgr.get_run_status(run_id)
|
||||
```
|
||||
|
||||
### Secret sync
|
||||
|
||||
```bash
|
||||
uv run python dev/scripts/sync_secrets.py --dry-run # preview
|
||||
uv run python dev/scripts/sync_secrets.py # push to Databricks
|
||||
```
|
||||
|
||||
### Quality monitoring
|
||||
|
||||
```python
|
||||
from aco.lake.quality import setup_monitors
|
||||
setup_monitors(client, catalog="aco", schemas=["core", "claims_preprocessing"])
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
### Architecture
|
||||
|
||||
Reference in New Issue
Block a user