docs: document Databricks SDK integration in README
Some checks failed
CI / skinny-install (aco) (push) Successful in 52s
CI / lint-test (push) Successful in 1m16s
CI / skinny-install (api) (push) Successful in 24s
CI / skinny-install (bcda) (push) Successful in 28s
CI / skinny-install (bib) (push) Successful in 34s
CI / skinny-install (bls) (push) Successful in 27s
CI / skinny-install (ccw) (push) Successful in 34s
CI / skinny-install (cli) (push) Successful in 37s
CI / skinny-install (cms) (push) Successful in 26s
CI / skinny-install (conf) (push) Successful in 29s
CI / skinny-install (opps) (push) Successful in 38s
CI / skinny-install (perf) (push) Successful in 31s
CI / skinny-install (pfs) (push) Successful in 33s
CI / skinny-install (rex) (push) Successful in 31s
CI / skinny-install (aco) (pull_request) Successful in 46s
CI / lint-test (pull_request) Successful in 1m9s
CI / skinny-install (api) (pull_request) Successful in 30s
CI / skinny-install (bcda) (pull_request) Successful in 29s
CI / skinny-install (bib) (pull_request) Successful in 24s
CI / skinny-install (bls) (pull_request) Successful in 31s
CI / skinny-install (ccw) (pull_request) Successful in 30s
CI / skinny-install (cli) (pull_request) Successful in 34s
CI / skinny-install (cms) (pull_request) Successful in 29s
CI / skinny-install (conf) (pull_request) Successful in 28s
CI / skinny-install (opps) (pull_request) Successful in 31s
Infra CI / zotero (pull_request) Successful in 6s
Infra CI / docs (pull_request) Failing after 6s
CI / skinny-install (perf) (pull_request) Successful in 27s
CI / skinny-install (pfs) (pull_request) Successful in 34s
Infra CI / notebooks (pull_request) Successful in 6s
CI / skinny-install (rex) (pull_request) Successful in 27s
Infra CI / mc (pull_request) Successful in 7s
Infra CI / api (pull_request) Successful in 9s
Some checks failed
CI / skinny-install (aco) (push) Successful in 52s
CI / lint-test (push) Successful in 1m16s
CI / skinny-install (api) (push) Successful in 24s
CI / skinny-install (bcda) (push) Successful in 28s
CI / skinny-install (bib) (push) Successful in 34s
CI / skinny-install (bls) (push) Successful in 27s
CI / skinny-install (ccw) (push) Successful in 34s
CI / skinny-install (cli) (push) Successful in 37s
CI / skinny-install (cms) (push) Successful in 26s
CI / skinny-install (conf) (push) Successful in 29s
CI / skinny-install (opps) (push) Successful in 38s
CI / skinny-install (perf) (push) Successful in 31s
CI / skinny-install (pfs) (push) Successful in 33s
CI / skinny-install (rex) (push) Successful in 31s
CI / skinny-install (aco) (pull_request) Successful in 46s
CI / lint-test (pull_request) Successful in 1m9s
CI / skinny-install (api) (pull_request) Successful in 30s
CI / skinny-install (bcda) (pull_request) Successful in 29s
CI / skinny-install (bib) (pull_request) Successful in 24s
CI / skinny-install (bls) (pull_request) Successful in 31s
CI / skinny-install (ccw) (pull_request) Successful in 30s
CI / skinny-install (cli) (pull_request) Successful in 34s
CI / skinny-install (cms) (pull_request) Successful in 29s
CI / skinny-install (conf) (pull_request) Successful in 28s
CI / skinny-install (opps) (pull_request) Successful in 31s
Infra CI / zotero (pull_request) Successful in 6s
Infra CI / docs (pull_request) Failing after 6s
CI / skinny-install (perf) (pull_request) Successful in 27s
CI / skinny-install (pfs) (pull_request) Successful in 34s
Infra CI / notebooks (pull_request) Successful in 6s
CI / skinny-install (rex) (pull_request) Successful in 27s
Infra CI / mc (pull_request) Successful in 7s
Infra CI / api (pull_request) Successful in 9s
Add SDK integration section to README.md.j2 with: - 18-service table showing category, SDK service, module, purpose - Governance example (declarative grants, drift audit, PHI schemas) - Job orchestration example (create, run, poll) - Secret sync usage - Quality monitoring usage - stack.toml config snippets Regenerate README.md (12,320 tests, 25 services, 14 pipelines).
This commit is contained in:
84
README.md
84
README.md
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||

|

|
||||||
|
|
||||||
Healthcare analytics platform on self-hosted infrastructure. Replaces dbt SQL models with narwhals DataFrame-agnostic expression functions, backed by DuckDB locally and Iceberg/Trino in the lakehouse. 25 services, 14 data pipelines, 12,291 tests at 99% coverage.
|
Healthcare analytics platform on self-hosted infrastructure. Replaces dbt SQL models with narwhals DataFrame-agnostic expression functions, backed by DuckDB locally and Iceberg/Trino in the lakehouse. 25 services, 14 data pipelines, 12,320 tests at 99% coverage.
|
||||||
|
|
||||||
## Quick start
|
## Quick start
|
||||||
|
|
||||||
@@ -264,6 +264,84 @@ uv run python dev/scripts/gen_config.py --dab-sql # generate SQL + DDL (~4 min
|
|||||||
databricks bundle deploy -t dev # deploy to Databricks
|
databricks bundle deploy -t dev # deploy to Databricks
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### SDK integration
|
||||||
|
|
||||||
|
The platform uses 18 Databricks SDK services via `UnityClient` (wrapping `WorkspaceClient`):
|
||||||
|
|
||||||
|
| Category | Services | Module | Purpose |
|
||||||
|
|----------|----------|--------|---------|
|
||||||
|
| **Catalog CRUD** | `catalogs`, `schemas`, `tables`, `volumes` | `lake/unity.py` | Unity Catalog object management |
|
||||||
|
| **Data loading** | `files`, `statement_execution` | `lake/sync.py` | Parquet upload + COPY INTO |
|
||||||
|
| **Governance** | `grants` | `lake/governance.py` | Declarative schema permissions with drift audit |
|
||||||
|
| **Secrets** | `secrets` | `lake/unity.py` | Scope management, env var sync |
|
||||||
|
| **Constraints** | `table_constraints` | `lake/unity.py` | PK/FK on claims tables for lineage |
|
||||||
|
| **Jobs** | `jobs` | `lake/jobs.py` | Create/run/monitor pipeline jobs from Python |
|
||||||
|
| **Compute** | `warehouses` | `lake/unity.py` | Find/create/start/stop warehouses by name |
|
||||||
|
| **Quality** | `quality_monitors` | `lake/quality.py` | Lakehouse Monitoring profiles |
|
||||||
|
| **Audit** | `system_schemas` | `lake/unity.py` | Table lineage and audit log queries |
|
||||||
|
|
||||||
|
### Governance
|
||||||
|
|
||||||
|
`src/aco/lake/governance.py` provides declarative grant management:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from aco.lake.governance import GovernancePolicy, GrantRule, apply_governance
|
||||||
|
|
||||||
|
policy = GovernancePolicy(
|
||||||
|
catalog="aco",
|
||||||
|
rules=[
|
||||||
|
GrantRule(group="analysts", schemas=["readmissions"], privileges=["SELECT"]),
|
||||||
|
GrantRule(group="pipeline_svc", schemas=["*"], privileges=["ALL_PRIVILEGES"]),
|
||||||
|
],
|
||||||
|
phi_schemas=["core", "claims_preprocessing", "cclf"],
|
||||||
|
)
|
||||||
|
|
||||||
|
# Preview changes
|
||||||
|
apply_governance(client, policy, dry_run=True)
|
||||||
|
|
||||||
|
# Detect drift between declared and actual grants
|
||||||
|
from aco.lake.governance import audit_governance
|
||||||
|
drifts = audit_governance(client, policy)
|
||||||
|
```
|
||||||
|
|
||||||
|
PHI schemas, group privilege mappings, and quality monitor config are declared in `stack.toml`:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[databricks.governance]
|
||||||
|
phi_schemas = ["core", "claims_preprocessing", "cclf", "input_layer"]
|
||||||
|
|
||||||
|
[databricks.governance.groups.analysts]
|
||||||
|
schemas = ["readmissions", "quality_measures"]
|
||||||
|
privileges = ["SELECT"]
|
||||||
|
```
|
||||||
|
|
||||||
|
### Job orchestration
|
||||||
|
|
||||||
|
`src/aco/lake/jobs.py` translates `Pipeline` objects into Databricks Jobs:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from aco.lake.jobs import JobManager
|
||||||
|
|
||||||
|
mgr = JobManager(client, catalog="aco")
|
||||||
|
job_id = mgr.create_pipeline_job("readmissions", schedule="0 0 6 * * ?")
|
||||||
|
run_id = mgr.run_now(job_id)
|
||||||
|
status = mgr.get_run_status(run_id)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Secret sync
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run python dev/scripts/sync_secrets.py --dry-run # preview
|
||||||
|
uv run python dev/scripts/sync_secrets.py # push to Databricks
|
||||||
|
```
|
||||||
|
|
||||||
|
### Quality monitoring
|
||||||
|
|
||||||
|
```python
|
||||||
|
from aco.lake.quality import setup_monitors
|
||||||
|
setup_monitors(client, catalog="aco", schemas=["core", "claims_preprocessing"])
|
||||||
|
```
|
||||||
|
|
||||||
## Testing
|
## Testing
|
||||||
|
|
||||||
### Architecture
|
### Architecture
|
||||||
@@ -337,7 +415,7 @@ The hook classifies staged files and runs only what is relevant:
|
|||||||
| `src/sem/*.py` | `tests/sem/` + `test_ast_coverage.py` | Module tests + structural invariants |
|
| `src/sem/*.py` | `tests/sem/` + `test_ast_coverage.py` | Module tests + structural invariants |
|
||||||
| `src/aco/*.py` + `src/sem/*.py` | `tests/aco/` + `tests/sem/` + structural | Both module test dirs |
|
| `src/aco/*.py` + `src/sem/*.py` | `tests/aco/` + `tests/sem/` + structural | Both module test dirs |
|
||||||
| `tests/bib/test_sync.py` | `tests/bib/` | Changed test dir |
|
| `tests/bib/test_sync.py` | `tests/bib/` | Changed test dir |
|
||||||
| `pyproject.toml` or `conftest.py` | Full suite (12,291+ tests) | Infrastructure change |
|
| `pyproject.toml` or `conftest.py` | Full suite (12,320+ tests) | Infrastructure change |
|
||||||
| `notebooks/pfs_calcs.py` | `marimo check` + notebook execution | Notebook validation only |
|
| `notebooks/pfs_calcs.py` | `marimo check` + notebook execution | Notebook validation only |
|
||||||
| `README.md` only | Nothing | No testable changes |
|
| `README.md` only | Nothing | No testable changes |
|
||||||
|
|
||||||
@@ -416,7 +494,7 @@ stack/
|
|||||||
│ ├── hooks/ Git hooks (tracked, core.hooksPath = dev/hooks)
|
│ ├── hooks/ Git hooks (tracked, core.hooksPath = dev/hooks)
|
||||||
│ ├── seeds/ Reference data (BCDA samples, CMS docs)
|
│ ├── seeds/ Reference data (BCDA samples, CMS docs)
|
||||||
│ └── pipelines/ CI-agnostic pipeline specs
|
│ └── pipelines/ CI-agnostic pipeline specs
|
||||||
├── tests/ 12,291 tests at 99% coverage
|
├── tests/ 12,320 tests at 99% coverage
|
||||||
├── notebooks/ Marimo notebooks
|
├── notebooks/ Marimo notebooks
|
||||||
├── docs/ Docusaurus site
|
├── docs/ Docusaurus site
|
||||||
├── data/ DuckDB, bib.sqlite, BCDA/CMS data, zotero (gitignored)
|
├── data/ DuckDB, bib.sqlite, BCDA/CMS data, zotero (gitignored)
|
||||||
|
|||||||
78
README.md.j2
78
README.md.j2
@@ -257,6 +257,84 @@ uv run python dev/scripts/gen_config.py --dab-sql # generate SQL + DDL (~4 min
|
|||||||
databricks bundle deploy -t dev # deploy to Databricks
|
databricks bundle deploy -t dev # deploy to Databricks
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### SDK integration
|
||||||
|
|
||||||
|
The platform uses 18 Databricks SDK services via `UnityClient` (wrapping `WorkspaceClient`):
|
||||||
|
|
||||||
|
| Category | Services | Module | Purpose |
|
||||||
|
|----------|----------|--------|---------|
|
||||||
|
| **Catalog CRUD** | `catalogs`, `schemas`, `tables`, `volumes` | `lake/unity.py` | Unity Catalog object management |
|
||||||
|
| **Data loading** | `files`, `statement_execution` | `lake/sync.py` | Parquet upload + COPY INTO |
|
||||||
|
| **Governance** | `grants` | `lake/governance.py` | Declarative schema permissions with drift audit |
|
||||||
|
| **Secrets** | `secrets` | `lake/unity.py` | Scope management, env var sync |
|
||||||
|
| **Constraints** | `table_constraints` | `lake/unity.py` | PK/FK on claims tables for lineage |
|
||||||
|
| **Jobs** | `jobs` | `lake/jobs.py` | Create/run/monitor pipeline jobs from Python |
|
||||||
|
| **Compute** | `warehouses` | `lake/unity.py` | Find/create/start/stop warehouses by name |
|
||||||
|
| **Quality** | `quality_monitors` | `lake/quality.py` | Lakehouse Monitoring profiles |
|
||||||
|
| **Audit** | `system_schemas` | `lake/unity.py` | Table lineage and audit log queries |
|
||||||
|
|
||||||
|
### Governance
|
||||||
|
|
||||||
|
`src/aco/lake/governance.py` provides declarative grant management:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from aco.lake.governance import GovernancePolicy, GrantRule, apply_governance
|
||||||
|
|
||||||
|
policy = GovernancePolicy(
|
||||||
|
catalog="aco",
|
||||||
|
rules=[
|
||||||
|
GrantRule(group="analysts", schemas=["readmissions"], privileges=["SELECT"]),
|
||||||
|
GrantRule(group="pipeline_svc", schemas=["*"], privileges=["ALL_PRIVILEGES"]),
|
||||||
|
],
|
||||||
|
phi_schemas=["core", "claims_preprocessing", "cclf"],
|
||||||
|
)
|
||||||
|
|
||||||
|
# Preview changes
|
||||||
|
apply_governance(client, policy, dry_run=True)
|
||||||
|
|
||||||
|
# Detect drift between declared and actual grants
|
||||||
|
from aco.lake.governance import audit_governance
|
||||||
|
drifts = audit_governance(client, policy)
|
||||||
|
```
|
||||||
|
|
||||||
|
PHI schemas, group privilege mappings, and quality monitor config are declared in `stack.toml`:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[databricks.governance]
|
||||||
|
phi_schemas = ["core", "claims_preprocessing", "cclf", "input_layer"]
|
||||||
|
|
||||||
|
[databricks.governance.groups.analysts]
|
||||||
|
schemas = ["readmissions", "quality_measures"]
|
||||||
|
privileges = ["SELECT"]
|
||||||
|
```
|
||||||
|
|
||||||
|
### Job orchestration
|
||||||
|
|
||||||
|
`src/aco/lake/jobs.py` translates `Pipeline` objects into Databricks Jobs:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from aco.lake.jobs import JobManager
|
||||||
|
|
||||||
|
mgr = JobManager(client, catalog="aco")
|
||||||
|
job_id = mgr.create_pipeline_job("readmissions", schedule="0 0 6 * * ?")
|
||||||
|
run_id = mgr.run_now(job_id)
|
||||||
|
status = mgr.get_run_status(run_id)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Secret sync
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run python dev/scripts/sync_secrets.py --dry-run # preview
|
||||||
|
uv run python dev/scripts/sync_secrets.py # push to Databricks
|
||||||
|
```
|
||||||
|
|
||||||
|
### Quality monitoring
|
||||||
|
|
||||||
|
```python
|
||||||
|
from aco.lake.quality import setup_monitors
|
||||||
|
setup_monitors(client, catalog="aco", schemas=["core", "claims_preprocessing"])
|
||||||
|
```
|
||||||
|
|
||||||
## Testing
|
## Testing
|
||||||
|
|
||||||
### Architecture
|
### Architecture
|
||||||
|
|||||||
Reference in New Issue
Block a user