Some checks failed
CI / skinny-install (api) (push) Successful in 24s
CI / lint-test (push) Successful in 1m18s
CI / skinny-install (bib) (push) Successful in 30s
CI / skinny-install (bls) (push) Successful in 28s
CI / skinny-install (aco) (push) Successful in 50s
CI / skinny-install (bcda) (push) Successful in 33s
CI / skinny-install (ccw) (push) Successful in 32s
CI / skinny-install (cms) (push) Successful in 24s
CI / skinny-install (cli) (push) Successful in 36s
CI / skinny-install (conf) (push) Successful in 33s
CI / skinny-install (opps) (push) Successful in 30s
CI / skinny-install (perf) (push) Successful in 31s
CI / skinny-install (pfs) (push) Successful in 34s
CI / skinny-install (rex) (push) Has been cancelled
CI / skinny-install (aco) (pull_request) Successful in 51s
CI / lint-test (pull_request) Successful in 1m20s
CI / skinny-install (api) (pull_request) Successful in 29s
CI / skinny-install (bcda) (pull_request) Successful in 30s
CI / skinny-install (bib) (pull_request) Successful in 32s
CI / skinny-install (bls) (pull_request) Successful in 26s
CI / skinny-install (ccw) (pull_request) Successful in 27s
CI / skinny-install (cli) (pull_request) Successful in 29s
CI / skinny-install (cms) (pull_request) Successful in 34s
CI / skinny-install (conf) (pull_request) Successful in 22s
CI / skinny-install (opps) (pull_request) Successful in 33s
CI / skinny-install (perf) (pull_request) Successful in 31s
CI / skinny-install (pfs) (pull_request) Successful in 34s
CI / skinny-install (rex) (pull_request) Successful in 32s
Infra CI / notebooks (push) Successful in 8s
Infra CI / zotero (push) Successful in 6s
Infra CI / docs (push) Failing after 12s
Infra CI / api (push) Successful in 10s
Infra CI / mc (push) Successful in 9s
Infra CI / notebooks (pull_request) Successful in 7s
Infra CI / zotero (pull_request) Successful in 6s
Infra CI / docs (pull_request) Failing after 5s
Infra CI / api (pull_request) Successful in 6s
Infra CI / mc (pull_request) Successful in 6s
Expand Databricks SDK usage from 6 to 18 WorkspaceClient services. Tier 1 — Governance & Compliance: - governance.py: declarative GovernancePolicy with apply + audit drift - UnityClient: grants CRUD (ws.grants), secret management (ws.secrets), table constraints PK/FK (ws.table_constraints) - sync_secrets.py: push env vars to Databricks scopes per stack.toml Tier 2 — Job Orchestration: - jobs.py: JobManager translates Pipeline → Databricks Job with task dependencies, run_now, get_run_status, list/delete - UnityClient: warehouse lookup by name (ws.warehouses), start/stop Tier 3 — Data Quality & Monitoring: - quality.py: setup_monitors creates Lakehouse Monitoring profiles, list_monitors, run_refresh (ws.quality_monitors) - UnityClient: system schema access, table lineage queries, audit log queries (ws.system_schemas, ws.statement_execution) Config: stack.toml gains [databricks.warehouse], [databricks.secrets], [databricks.governance], [databricks.quality] sections. 23 new tests, all passing.
58 lines
2.0 KiB
Python
58 lines
2.0 KiB
Python
"""Tests for aco.lake.quality — Lakehouse Monitoring setup."""
|
|
|
|
from __future__ import annotations
|
|
|
|
from unittest.mock import MagicMock
|
|
|
|
from aco.lake.quality import MonitorProfile, MonitorResult, setup_monitors
|
|
|
|
|
|
class TestMonitorModels:
|
|
def test_profile_defaults(self):
|
|
p = MonitorProfile(table="aco.core")
|
|
assert p.output_schema == "_monitoring"
|
|
assert p.slicing_exprs == []
|
|
|
|
def test_result_model(self):
|
|
r = MonitorResult(table="aco.core.encounter", status="active")
|
|
assert r.table == "aco.core.encounter"
|
|
assert r.dashboard_id is None
|
|
|
|
|
|
class TestSetupMonitors:
|
|
def test_dry_run(self):
|
|
client = MagicMock()
|
|
table = MagicMock()
|
|
table.full_name = "aco.core.encounter"
|
|
table.name = "encounter"
|
|
client._ws.tables.list.return_value = [table]
|
|
|
|
actions = setup_monitors(client, catalog="aco", schemas=["core"], dry_run=True)
|
|
assert len(actions) == 1
|
|
assert "[dry-run]" in actions[0]
|
|
assert "aco.core.encounter" in actions[0]
|
|
# Should not call create in dry_run
|
|
client._ws.quality_monitors.create.assert_not_called()
|
|
|
|
def test_creates_output_schema(self):
|
|
client = MagicMock()
|
|
client._ws.schemas.get.side_effect = Exception("not found")
|
|
client._ws.tables.list.return_value = []
|
|
|
|
setup_monitors(client, catalog="aco", schemas=["core"], dry_run=False)
|
|
client._ws.schemas.create.assert_called_once()
|
|
|
|
def test_skips_existing_monitor(self):
|
|
client = MagicMock()
|
|
client._ws.schemas.get.return_value = MagicMock()
|
|
table = MagicMock()
|
|
table.full_name = "aco.core.encounter"
|
|
table.name = "encounter"
|
|
client._ws.tables.list.return_value = [table]
|
|
# Monitor already exists
|
|
client._ws.quality_monitors.get.return_value = MagicMock()
|
|
|
|
actions = setup_monitors(client, catalog="aco", schemas=["core"], dry_run=False)
|
|
assert any("[exists]" in a for a in actions)
|
|
client._ws.quality_monitors.create.assert_not_called()
|