Files
stack/tests/aco/test_lake_quality.py
kert 97852e818c
Some checks failed
CI / skinny-install (api) (push) Successful in 24s
CI / lint-test (push) Successful in 1m18s
CI / skinny-install (bib) (push) Successful in 30s
CI / skinny-install (bls) (push) Successful in 28s
CI / skinny-install (aco) (push) Successful in 50s
CI / skinny-install (bcda) (push) Successful in 33s
CI / skinny-install (ccw) (push) Successful in 32s
CI / skinny-install (cms) (push) Successful in 24s
CI / skinny-install (cli) (push) Successful in 36s
CI / skinny-install (conf) (push) Successful in 33s
CI / skinny-install (opps) (push) Successful in 30s
CI / skinny-install (perf) (push) Successful in 31s
CI / skinny-install (pfs) (push) Successful in 34s
CI / skinny-install (rex) (push) Has been cancelled
CI / skinny-install (aco) (pull_request) Successful in 51s
CI / lint-test (pull_request) Successful in 1m20s
CI / skinny-install (api) (pull_request) Successful in 29s
CI / skinny-install (bcda) (pull_request) Successful in 30s
CI / skinny-install (bib) (pull_request) Successful in 32s
CI / skinny-install (bls) (pull_request) Successful in 26s
CI / skinny-install (ccw) (pull_request) Successful in 27s
CI / skinny-install (cli) (pull_request) Successful in 29s
CI / skinny-install (cms) (pull_request) Successful in 34s
CI / skinny-install (conf) (pull_request) Successful in 22s
CI / skinny-install (opps) (pull_request) Successful in 33s
CI / skinny-install (perf) (pull_request) Successful in 31s
CI / skinny-install (pfs) (pull_request) Successful in 34s
CI / skinny-install (rex) (pull_request) Successful in 32s
Infra CI / notebooks (push) Successful in 8s
Infra CI / zotero (push) Successful in 6s
Infra CI / docs (push) Failing after 12s
Infra CI / api (push) Successful in 10s
Infra CI / mc (push) Successful in 9s
Infra CI / notebooks (pull_request) Successful in 7s
Infra CI / zotero (pull_request) Successful in 6s
Infra CI / docs (pull_request) Failing after 5s
Infra CI / api (pull_request) Successful in 6s
Infra CI / mc (pull_request) Successful in 6s
feat: deep Databricks SDK integration — governance, jobs, quality, secrets
Expand Databricks SDK usage from 6 to 18 WorkspaceClient services.

Tier 1 — Governance & Compliance:
- governance.py: declarative GovernancePolicy with apply + audit drift
- UnityClient: grants CRUD (ws.grants), secret management (ws.secrets),
  table constraints PK/FK (ws.table_constraints)
- sync_secrets.py: push env vars to Databricks scopes per stack.toml

Tier 2 — Job Orchestration:
- jobs.py: JobManager translates Pipeline → Databricks Job with task
  dependencies, run_now, get_run_status, list/delete
- UnityClient: warehouse lookup by name (ws.warehouses), start/stop

Tier 3 — Data Quality & Monitoring:
- quality.py: setup_monitors creates Lakehouse Monitoring profiles,
  list_monitors, run_refresh (ws.quality_monitors)
- UnityClient: system schema access, table lineage queries, audit log
  queries (ws.system_schemas, ws.statement_execution)

Config: stack.toml gains [databricks.warehouse], [databricks.secrets],
[databricks.governance], [databricks.quality] sections.

23 new tests, all passing.
2026-03-26 19:57:51 -04:00

58 lines
2.0 KiB
Python

"""Tests for aco.lake.quality — Lakehouse Monitoring setup."""
from __future__ import annotations
from unittest.mock import MagicMock
from aco.lake.quality import MonitorProfile, MonitorResult, setup_monitors
class TestMonitorModels:
def test_profile_defaults(self):
p = MonitorProfile(table="aco.core")
assert p.output_schema == "_monitoring"
assert p.slicing_exprs == []
def test_result_model(self):
r = MonitorResult(table="aco.core.encounter", status="active")
assert r.table == "aco.core.encounter"
assert r.dashboard_id is None
class TestSetupMonitors:
def test_dry_run(self):
client = MagicMock()
table = MagicMock()
table.full_name = "aco.core.encounter"
table.name = "encounter"
client._ws.tables.list.return_value = [table]
actions = setup_monitors(client, catalog="aco", schemas=["core"], dry_run=True)
assert len(actions) == 1
assert "[dry-run]" in actions[0]
assert "aco.core.encounter" in actions[0]
# Should not call create in dry_run
client._ws.quality_monitors.create.assert_not_called()
def test_creates_output_schema(self):
client = MagicMock()
client._ws.schemas.get.side_effect = Exception("not found")
client._ws.tables.list.return_value = []
setup_monitors(client, catalog="aco", schemas=["core"], dry_run=False)
client._ws.schemas.create.assert_called_once()
def test_skips_existing_monitor(self):
client = MagicMock()
client._ws.schemas.get.return_value = MagicMock()
table = MagicMock()
table.full_name = "aco.core.encounter"
table.name = "encounter"
client._ws.tables.list.return_value = [table]
# Monitor already exists
client._ws.quality_monitors.get.return_value = MagicMock()
actions = setup_monitors(client, catalog="aco", schemas=["core"], dry_run=False)
assert any("[exists]" in a for a in actions)
client._ws.quality_monitors.create.assert_not_called()