Merge pull request 'feat: sem — semantic coverage orchestration' (#289) from feat/semantic-coverage into main

README.md

@@ -2,7 +2,7 @@
 
 ![coverage](assets/icons/coverage.svg)
 
-Healthcare analytics platform on self-hosted infrastructure. Replaces dbt SQL models with narwhals DataFrame-agnostic expression functions, backed by DuckDB locally and Iceberg/Trino in the lakehouse. 22 services, 13 data pipelines, 11,955 tests at 99% coverage.
+Healthcare analytics platform on self-hosted infrastructure. Replaces dbt SQL models with narwhals DataFrame-agnostic expression functions, backed by DuckDB locally and Iceberg/Trino in the lakehouse. 25 services, 14 data pipelines, 12,320 tests at 99% coverage.
 
 ## Quick start
 
@@ -37,7 +37,7 @@ pip install stack[all]         # everything
 pip install stack[aco,aws]     # ACO analytics with AWS storage
 ```
 
-11 modules available as optional extras: `conf`, `aco`, `api`, `bcda`, `bib`, `bls`, `ccw`, `cli`, `cms`, `pfs`, `rex`. Cloud providers: `aws`, `gcp`, `azure`. Aggregates: `all`, `lake`.
+14 modules available as optional extras: `aco`, `api`, `bcda`, `bib`, `bls`, `ccw`, `cli`, `cms`, `conf`, `opps`, `perf`, `pfs`, `rex`, `sem`. Cloud providers: `aws`, `azure`, `gcp`. Aggregates: `all`, `lake`.
 
 ## Services
 
@@ -85,23 +85,24 @@ All services route through Traefik at `*.homelab.fhirworx.io`.
 
 ### Pipelines
 
-13 registered pipelines with 189 steps, validated by `run_pipeline(steps, load)` which enforces output schemas via pandera-style column checks.
+14 registered pipelines with 230 steps, validated by `run_pipeline(steps, load)` which enforces output schemas via pandera-style column checks.
 
-| Pipeline | Domain |
+| Pipeline | Steps | Domain |
-|----------|--------|
+|----------|------:|--------|
-| `input_layer` | Raw CCLF/BCDA ingestion |
+| `ahrq_measures` | 39 | AHRQ quality indicators |
-| `core` | Encounters, patients, providers |
+| `cclf` | 19 | CCLF connector pipeline -- raw CCLF files -> input_layer tables |
-| `claims_preprocessing` | Claim matching, encounter IDs |
+| `claims_preprocessing` | 63 | Claim matching, encounter IDs |
-| `readmissions` | 30-day readmission rates |
+| `cms_quality_measures` | 20 | CMS ACO quality metrics |
-| `pharmacy` | Part D drug utilization |
+| `core` | 12 | Encounters, patients, providers |
-| `ahrq_measures` | AHRQ quality indicators |
+| `data_quality` | 1 | Validation and completeness |
-| `cms_quality_measures` | CMS ACO quality metrics |
+| `hcc_suspecting` | 6 | HCC risk adjustment |
-| `hcc_suspecting` | HCC risk adjustment |
+| `input_layer` | 13 | Raw CCLF/BCDA ingestion |
-| `provider_attribution` | Provider-patient assignment |
+| `main` | 14 | Full pipeline orchestration |
-| `quality_measures` | Composite quality scores |
+| `opps` | 2 | Outpatient payment system |
-| `data_quality` | Validation and completeness |
+| `pharmacy` | 4 | Part D drug utilization |
-| `cclf` | CCLF file parsing |
+| `provider_attribution` | 10 | Provider-patient assignment |
-| `main` | Full pipeline orchestration |
+| `quality_measures` | 15 | Composite quality scores |
+| `readmissions` | 12 | 30-day readmission rates |
 
 ### Expression layer
 
@@ -154,7 +155,7 @@ The generator reads image definitions from `stack.toml [images]`, dispatches to
 
 ### Workflows
 
-Seven workflows generated per backend:
+7 workflows generated per backend:
 
 | Workflow | Trigger | What it does |
 |----------|---------|--------------|
@@ -193,7 +194,7 @@ Daily automated pipeline (`pkg-supply-chain.yml`) that:
 
 ## Credential management
 
-All 19 service credentials derive from a single 256-bit root key via HKDF-SHA256. No passwords stored in `.env` — they regenerate deterministically from root key + commit SHA on each deploy.
+All 20 service credentials derive from a single 256-bit root key via HKDF-SHA256. No passwords stored in `.env` — they regenerate deterministically from root key + commit SHA on each deploy.
 
 ```
 ROOT_KEY + commit_sha -> HKDF-SHA256 -> all credentials -> .env + backends
@@ -256,18 +257,220 @@ Switch with `STACK_CONTEXT=aws` or edit `stack.toml [context] active`.
 
 ## Databricks
 
-Databricks Asset Bundle (`databricks.yml`) is auto-generated from the pipeline registry. 13 pipeline tasks with dependency ordering, 3 targets (dev/staging/prod), daily schedule.
+Databricks Asset Bundle (`databricks.yml`) is auto-generated from the pipeline registry. 14 pipeline tasks with dependency ordering, 3 targets (dev/staging/prod), daily schedule.
 
 ```bash
 uv run python dev/scripts/gen_config.py --dab-sql   # generate SQL + DDL (~4 min)
 databricks bundle deploy -t dev                       # deploy to Databricks
 ```
 
+### SDK integration
+
+The platform uses 18 Databricks SDK services via `UnityClient` (wrapping `WorkspaceClient`):
+
+| Category | Services | Module | Purpose |
+|----------|----------|--------|---------|
+| **Catalog CRUD** | `catalogs`, `schemas`, `tables`, `volumes` | `lake/unity.py` | Unity Catalog object management |
+| **Data loading** | `files`, `statement_execution` | `lake/sync.py` | Parquet upload + COPY INTO |
+| **Governance** | `grants` | `lake/governance.py` | Declarative schema permissions with drift audit |
+| **Secrets** | `secrets` | `lake/unity.py` | Scope management, env var sync |
+| **Constraints** | `table_constraints` | `lake/unity.py` | PK/FK on claims tables for lineage |
+| **Jobs** | `jobs` | `lake/jobs.py` | Create/run/monitor pipeline jobs from Python |
+| **Compute** | `warehouses` | `lake/unity.py` | Find/create/start/stop warehouses by name |
+| **Quality** | `quality_monitors` | `lake/quality.py` | Lakehouse Monitoring profiles |
+| **Audit** | `system_schemas` | `lake/unity.py` | Table lineage and audit log queries |
+
+### Governance
+
+`src/aco/lake/governance.py` provides declarative grant management:
+
+```python
+from aco.lake.governance import GovernancePolicy, GrantRule, apply_governance
+
+policy = GovernancePolicy(
+    catalog="aco",
+    rules=[
+        GrantRule(group="analysts", schemas=["readmissions"], privileges=["SELECT"]),
+        GrantRule(group="pipeline_svc", schemas=["*"], privileges=["ALL_PRIVILEGES"]),
+    ],
+    phi_schemas=["core", "claims_preprocessing", "cclf"],
+)
+
+# Preview changes
+apply_governance(client, policy, dry_run=True)
+
+# Detect drift between declared and actual grants
+from aco.lake.governance import audit_governance
+drifts = audit_governance(client, policy)
+```
+
+PHI schemas, group privilege mappings, and quality monitor config are declared in `stack.toml`:
+
+```toml
+[databricks.governance]
+phi_schemas = ["core", "claims_preprocessing", "cclf", "input_layer"]
+
+[databricks.governance.groups.analysts]
+schemas = ["readmissions", "quality_measures"]
+privileges = ["SELECT"]
+```
+
+### Job orchestration
+
+`src/aco/lake/jobs.py` translates `Pipeline` objects into Databricks Jobs:
+
+```python
+from aco.lake.jobs import JobManager
+
+mgr = JobManager(client, catalog="aco")
+job_id = mgr.create_pipeline_job("readmissions", schedule="0 0 6 * * ?")
+run_id = mgr.run_now(job_id)
+status = mgr.get_run_status(run_id)
+```
+
+### Secret sync
+
+Push environment variables to Databricks secret scopes per `stack.toml [databricks.secrets]`:
+
+```bash
+uv run python dev/scripts/sync_secrets.py --dry-run   # preview
+uv run python dev/scripts/sync_secrets.py              # push to Databricks
+uv run python dev/scripts/sync_secrets.py --list       # show current secrets
+```
+
+### Quality monitoring
+
+```python
+from aco.lake.quality import setup_monitors
+setup_monitors(client, catalog="aco", schemas=["core", "claims_preprocessing"])
+```
+
+## Testing
+
+### Architecture
+
+Testing follows a semantic coverage model (`src/sem/`) rather than tracking raw line numbers. The system parses every `.py` file into an AST, builds stable semantic nodes (functions, branches, exception handlers, loops), and attaches three signal layers:
+
+```
+source → AST → semantic nodes → attach (ruff, ty, coverage) → planner
+```
+
+| Layer | Module | What it does |
+|-------|--------|-------------|
+| **Parse** | `sem.parse` | Walk AST, emit `SemanticNode` per function, branch, except, loop, return, raise |
+| **Enrich** | `sem.enrich` | Run `ruff check` and `ty check`, map diagnostics to tightest-span node |
+| **Runtime** | `sem.runtime` | Map `coverage.py` JSON report onto nodes (hit/miss, per-test contexts) |
+| **Plan** | `sem.plan` | Score uncovered nodes by priority, suggest next test targets |
+| **State** | `sem.state` | Persist node status across runs, reset on source hash change |
+
+Each node has a stable identity derived from its module path, qualified symbol name, kind, and ordinal position — never from line numbers:
+
+```
+app.config::load_settings::branch_if[1]
+aco.express.pharmacy::pharmacy_claims::except_handler[0]
+```
+
+#### Priority heuristic
+
+```
+priority =
+    uncovered_branch × 5
+  + ty_diagnostic × 4
+  + inside_partially_tested_function × 3
+  + uncovered_exception_path × 3
+  + ruff_warning × 2
+  − prior_failures × 2
+  − already_covered × 5
+```
+
+#### Structural invariants
+
+`tests/test_ast_coverage.py` runs at collection time (pure AST, no imports) and enforces:
+
+1. Every `@nw.narwhalify` function has a docstring
+2. Every `Expr(...)` call supplies `name`, `fn`, `output`, and `after`
+3. Every public Pydantic model is importable
+4. Every `pipe/*.py` module exports a `Pipeline` with non-empty `.exprs`
+5. Implementation ratio stays above baseline (ratchet)
+6. `express/` and `pipe/` modules are symmetric
+
+### Pre-commit hooks
+
+Git hooks live in `dev/hooks/` (tracked) and are activated via:
+
+```bash
+git config core.hooksPath dev/hooks
+```
+
+The shell hook is a 3-line wrapper. All logic lives in `src/sem/hooks.py`:
+
+```bash
+#!/usr/bin/env bash
+exec uv run python -m sem.hooks
+```
+
+#### Smart test selection
+
+The hook classifies staged files and runs only what is relevant:
+
+| What changed | Tests run | Why |
+|-------------|-----------|-----|
+| `src/sem/*.py` | `tests/sem/` + `test_ast_coverage.py` | Module tests + structural invariants |
+| `src/aco/*.py` + `src/sem/*.py` | `tests/aco/` + `tests/sem/` + structural | Both module test dirs |
+| `tests/bib/test_sync.py` | `tests/bib/` | Changed test dir |
+| `pyproject.toml` or `conftest.py` | Full suite (12,320+ tests) | Infrastructure change |
+| `notebooks/pfs_calcs.py` | `marimo check` + notebook execution | Notebook validation only |
+| `README.md` only | Nothing | No testable changes |
+
+The mapping rule is: `src/<module>/` changes → `tests/<module>/` runs. Any `src/` change also triggers `test_ast_coverage.py` to verify structural invariants haven't regressed.
+
+#### Forcing full suite
+
+```bash
+GIT_PRE_COMMIT_FULL=1 git commit -m "message"
+```
+
+#### Hook steps (in order)
+
+1. **Venv recovery** — if `uv run python -c 'import sys'` fails, run `uv sync --dev`
+2. **Config regeneration** — if `stack.toml` or `gen_config.py` changed, regenerate CI workflows
+3. **Ruff lint + format** — only staged `.py` files
+4. **AST parse check** — verify staged source files have valid syntax
+5. **Pytest** — targeted or full suite based on what changed
+6. **Marimo check** — only if notebooks are staged
+7. **Notebook execution** — only staged notebooks in the safe-to-run list
+
+#### Post-commit
+
+`dev/hooks/post-commit` rebuilds the docs Docker image in the background after every commit so the documentation site stays current with docstring changes.
+
+### Running tests
+
+```bash
+uv run python -m pytest tests/                    # full suite
+uv run python -m pytest tests/sem/                 # one module
+uv run python -m pytest tests/ -m "not stub"       # skip stub inventory
+uv run python -m pytest tests/ -m stub             # only stub status
+uv run python -m pytest tests/test_ast_coverage.py # structural invariants only
+```
+
+### README generation
+
+`README.md` is generated from `README.md.j2` using live codebase data. Never edit `README.md` directly.
+
+```bash
+uv run python dev/scripts/gen_readme.py            # regenerate from template
+uv run python dev/scripts/gen_readme.py --check     # verify freshness (CI gate)
+uv run python dev/scripts/gen_readme.py --dry-run   # preview to stdout
+```
+
+Dynamic values pulled from: `compose.yml` (services), `pyproject.toml` (modules), pipeline registry (pipelines + steps), `pytest --collect-only` (test count), `api.auth.manifest` (credentials), `.gitea/workflows/` (workflow count).
+
 ## Project layout
 
 ```
 stack/
-├── compose.yml                  Docker Compose (22 services)
+├── compose.yml                  Docker Compose (25 services)
 ├── stack.toml                   Centralised configuration
 ├── pyproject.toml               Python project (uv, optional deps per module)
 ├── src/
@@ -280,8 +483,11 @@ stack/
 │   ├── cli/                      CLI entry point (typer)
 │   ├── cms/                      CMS public data tables
 │   ├── conf/                     Config loader, storage abstraction, table base
+│   ├── opps/                     Outpatient Prospective Payment System
+│   ├── perf/                     Pipeline telemetry (OpenTelemetry)
 │   ├── pfs/                      Physician Fee Schedule
-│   └── rex/                     REX fixed-width file processing
+│   ├── rex/                      REX fixed-width file processing
+│   └── sem/                      Semantic coverage orchestration
 ├── infra/                       Service configs and Dockerfiles
 │   ├── images/                  All Dockerfiles (api, notebooks, zotero, docs, mc)
 │   ├── traefik/                 Reverse proxy + loch CSS injection
@@ -300,10 +506,10 @@ stack/
 │   └── azure/                   ABFS, Azure SQL, Unity Catalog, Container Apps
 ├── dev/                         Dev tooling
 │   ├── scripts/                 Code generators, bootstrap, supply chain tools
-│   ├── hooks/                   Git pre-commit hook
+│   ├── hooks/                   Git hooks (tracked, core.hooksPath = dev/hooks)
 │   ├── seeds/                   Reference data (BCDA samples, CMS docs)
 │   └── pipelines/               CI-agnostic pipeline specs
-├── tests/                       11,955 tests at 99% coverage
+├── tests/                       12,320 tests at 99% coverage
 ├── notebooks/                   Marimo notebooks
 ├── docs/                        Docusaurus site
 ├── data/                        DuckDB, bib.sqlite, BCDA/CMS data, zotero (gitignored)

README.md.j2

@@ -0,0 +1,522 @@
+{# README.md.j2 — source of truth for README.md
+   Render with: uv run python dev/scripts/gen_readme.py
+   Check freshness: uv run python dev/scripts/gen_readme.py --check
+#}
+# Stack
+
+![coverage](assets/icons/coverage.svg)
+
+Healthcare analytics platform on self-hosted infrastructure. Replaces dbt SQL models with narwhals DataFrame-agnostic expression functions, backed by DuckDB locally and Iceberg/Trino in the lakehouse. {{ services.count }} services, {{ pipelines.count }} data pipelines, {{ tests.formatted }} tests at 99% coverage.
+
+## Quick start
+
+```bash
+# 1. Generate root key and seed .env
+ROOT_KEY=$(openssl rand -hex 32)
+cat > .env <<EOF
+DOMAIN=homelab.fhirworx.io
+HOST_IP=192.168.1.192
+EOF
+
+# 2. Start core services and bootstrap credentials
+docker compose up -d postgres rustfs traefik
+ROOT_KEY=$ROOT_KEY uv run python -m api.auth bootstrap $(git rev-parse HEAD)
+
+# 3. Start everything
+docker compose up -d
+
+# 4. Verify
+curl -s http://localhost:8000/health | python3 -m json.tool
+```
+
+## Installation
+
+The package supports skinny installs — install only the modules you need:
+
+```bash
+pip install stack[aco]         # ACO analytics only
+pip install stack[bib]         # bibliography only
+pip install stack[cli]         # CLI (pulls aco + api + bib)
+pip install stack[all]         # everything
+pip install stack[aco,aws]     # ACO analytics with AWS storage
+```
+
+{{ modules.count }} modules available as optional extras: {{ modules.names | map('backtick') | join(', ') }}. Cloud providers: {{ modules.cloud | map('backtick') | join(', ') }}. Aggregates: {{ modules.aggregates | map('backtick') | join(', ') }}.
+
+## Services
+
+All services route through Traefik at `*.homelab.fhirworx.io`.
+
+| Service | URL | Purpose |
+|---------|-----|---------|
+| API | `api` (internal) | FastAPI — health, pipelines, bib endpoints |
+| Notebooks | `notebooks.homelab.fhirworx.io` | Marimo notebooks (GPU-accelerated) |
+| Docs | `docs.homelab.fhirworx.io` | Docusaurus — API reference + CMS bibliography |
+| Gitea | `gitea.homelab.fhirworx.io` | Git server, container registry, LFS |
+| Zotero | `zotero.homelab.fhirworx.io` | Reference manager (KasmVNC desktop) |
+| Dashboard | `homelab.fhirworx.io` | Service launcher |
+| RustFS | `s3.homelab.fhirworx.io` | S3-compatible object storage |
+| Nessie | `nessie.homelab.fhirworx.io` | Git-like Iceberg catalog |
+| Polaris | `polaris.homelab.fhirworx.io` | Iceberg catalog with governance |
+| Trino | `trino.homelab.fhirworx.io` | Distributed SQL engine |
+| Grafana | `grafana.homelab.fhirworx.io` | Dashboards |
+| Prometheus | `prometheus.homelab.fhirworx.io` | Metrics |
+| Jaeger | `jaeger.homelab.fhirworx.io` | Distributed tracing |
+| Loki | `loki.homelab.fhirworx.io` | Log aggregation |
+
+## Analytics platform
+
+### Architecture
+
+```
+  CCLF/BCDA/CMS data
+        |
+        v
+  src/aco/express/     narwhals expressions (DataFrame-agnostic)
+        |
+        v
+  src/aco/pipe/        pipeline runner with schema validation
+        |
+        v
+  DuckDB (local)  --or--  Iceberg + Trino (lakehouse)  --or--  Databricks
+        |
+        v
+  src/api/             FastAPI (health, runs, bib)
+        |
+        v
+  notebooks/           Marimo interactive analysis
+```
+
+### Pipelines
+
+{{ pipelines.count }} registered pipelines with {{ pipelines.total_steps }} steps, validated by `run_pipeline(steps, load)` which enforces output schemas via pandera-style column checks.
+
+| Pipeline | Steps | Domain |
+|----------|------:|--------|
+{% for p in pipelines.list %}
+| `{{ p.name }}` | {{ p.steps }} | {{ p.doc }} |
+{% endfor %}
+
+### Expression layer
+
+`src/aco/express/` contains narwhals functions that work across polars, pandas, and cuDF. Each module mirrors a pipeline and exports pure transformation functions.
+
+### Configuration
+
+`stack.toml` is the single config file. `src/conf/` provides:
+
+- `cfg` — attribute access (`cfg.db.aco`)
+- `path("db.aco")` — absolute path resolution relative to repo root
+- `context()` — environment switching (local/lake/databricks/trino/aws/gcp/azure)
+- `secret("api.secret", "STACK_API_SECRET")` — env var with config fallback
+- `storage.get_filesystem()` — multi-cloud storage abstraction (S3/GCS/ABFS/local)
+
+Override context at runtime: `STACK_CONTEXT=lake` (or `aws`, `gcp`, `azure`).
+
+### Bibliography
+
+`src/bib/` manages a Zotero-backed citation store in `data/bib.sqlite`. Column-level provenance via `Tag.col(ref, desc)`. The docs site exports `library.json` for a searchable bibliography browser at `/library`.
+
+## CI/CD
+
+### Backend switching
+
+CI/CD workflows are auto-generated from `stack.toml` by `gen_config.py`. Three backends are supported:
+
+| Backend | Workflows directory | When to use |
+|---------|---------------------|-------------|
+| `gitea` | `.gitea/workflows/` | Self-hosted Gitea instance (default) |
+| `github` | `.github/workflows/` | GitHub repos / GitHub Actions |
+| `woodpecker` | `.woodpecker/` | Woodpecker CI server |
+
+Switch backends by editing `stack.toml`:
+
+```toml
+[ci]
+backend = "gitea"   # change to "github" or "woodpecker"
+```
+
+Then regenerate:
+
+```bash
+uv run python dev/scripts/gen_config.py          # generate new workflows
+uv run python dev/scripts/gen_config.py --check   # verify (used in CI)
+uv run python dev/scripts/gen_config.py --backend github  # one-off override
+```
+
+The generator reads image definitions from `stack.toml [images]`, dispatches to the active backend emitter (`dev/scripts/backends/{backend}.py`), writes workflow YAML, and auto-removes stale workflows from inactive backend directories.
+
+### Workflows
+
+{{ workflows.count }} workflows generated per backend:
+
+| Workflow | Trigger | What it does |
+|----------|---------|--------------|
+| `ci.yml` | Push/PR | Ruff lint + format, pytest (99% coverage), gen_config --check, skinny-install matrix |
+| `deploy.yml` | Push to main | Build 5 images, Trivy scan, vuln reporting |
+| `harden.yml` | Weekly cron / manual | Rebuild `--no-cache`, scan, auto-close/file vuln issues |
+| `rebuild-all.yml` | Manual | Full rebuild + scan + deploy |
+| `infra-ci.yml` | Path-filtered push | Hadolint Dockerfiles, dry-run builds |
+| `release.yml` | Tag (`v*`) | `uv build` + release |
+| `pkg-supply-chain.yml` | Daily / push | Package inventory, mirror sync, drift detection, vuln scan |
+
+### Secret mapping
+
+| Secret | Gitea | GitHub | Woodpecker |
+|--------|-------|--------|------------|
+| Registry auth | `REGISTRY_USER` + `REGISTRY_TOKEN` | `GITHUB_TOKEN` (built-in) | `registry_username` + `registry_password` |
+| Gitea API | `GITEA_TOKEN` | N/A | `gitea_token` |
+| Vuln reporting | `GITEA_TOKEN` | `GITHUB_TOKEN` | `gitea_token` |
+
+### Image tagging
+
+Images are tagged with the short commit SHA (8 chars). Compose resolves via `${COMMIT_SHA:-latest}` from `.env`. Custom images: `api`, `notebooks`, `zotero`, `docs`, `mc`.
+
+### Vulnerability management
+
+The `harden.yml` pipeline rebuilds all images with `--no-cache` to pick up OS patches, runs Trivy scans, and auto-files or auto-closes Gitea issues via `api.diag.vuln`.
+
+### Package supply chain
+
+Daily automated pipeline (`pkg-supply-chain.yml`) that:
+1. Scans Dockerfiles, pyproject.toml, and CI workflows for all package dependencies
+2. Syncs packages to the Gitea package registry (local mirror)
+3. Detects drift between manifest and mirror
+4. Runs Trivy vulnerability scans on dependencies
+5. Auto-creates Gitea issues for missing packages and CVEs
+
+## Credential management
+
+All {{ credentials.count }} service credentials derive from a single 256-bit root key via HKDF-SHA256. No passwords stored in `.env` — they regenerate deterministically from root key + commit SHA on each deploy.
+
+```
+ROOT_KEY + commit_sha -> HKDF-SHA256 -> all credentials -> .env + backends
+```
+
+| Tier | Salt | Rotates |
+|------|------|---------|
+| Bootstrap | `b"bootstrap"` | Only when root key changes |
+| Service | `commit_sha` | Every deploy |
+
+```bash
+# Bootstrap (first time)
+ROOT_KEY=$KEY uv run python -m api.auth bootstrap $(git rev-parse HEAD)
+
+# Rotate (every deploy, automatic in CI)
+ROOT_KEY=$KEY uv run python -m api.auth provision $COMMIT_SHA
+
+# Dry-run
+ROOT_KEY=$KEY uv run python -m api.auth derive $COMMIT_SHA --redact
+```
+
+## Network segmentation
+
+| Network | Type | Services |
+|---------|------|----------|
+| `gateway` | external | Traefik, all public-facing |
+| `storage` | internal | PostgreSQL, RustFS |
+| `data` | internal | Nessie, Trino, Polaris |
+| `observability` | internal | Grafana, Prometheus, Jaeger, Loki |
+| `ci` | external | Gitea, Act Runner, RustFS |
+
+## Design system
+
+Theme: **loch** (deep-navy). Traefik's `inject-loch` middleware rewrites HTML to inject `loch.css` and the favicon. Per-service CSS in `assets/css/`. Altair chart theme in `assets/nature.py` (Nature-conformant, accessible palette, IBM Plex Mono headings).
+
+## Data lakehouse
+
+| Component | Purpose |
+|-----------|---------|
+| Nessie | Git-like branching and time travel for Iceberg tables |
+| Polaris | Iceberg catalog with RBAC and multi-tenant governance |
+| Trino | Distributed SQL (`iceberg` catalog) |
+| RustFS | S3 storage backend (`s3://lakehouse/`) |
+
+Query via Trino CLI, JDBC, Web UI, PyIceberg, or DuckDB+PyArrow.
+
+### Cloud deployment
+
+The platform runs on self-hosted Docker Compose by default. Cloud provider guides in `cloud/`:
+
+| Provider | Storage | Catalog | Compute | Context |
+|----------|---------|---------|---------|---------|
+| Self-hosted | RustFS (S3) | Nessie / Polaris | Docker Compose | `local` / `lake` |
+| AWS | S3 | Glue | ECS / EKS | `aws` |
+| GCP | GCS | BigQuery | Cloud Run / GKE | `gcp` |
+| Azure | ABFS | Unity Catalog | Container Apps / AKS | `azure` |
+| Databricks | DBFS | Unity Catalog | Databricks Jobs | `databricks` |
+
+Switch with `STACK_CONTEXT=aws` or edit `stack.toml [context] active`.
+
+## Databricks
+
+Databricks Asset Bundle (`databricks.yml`) is auto-generated from the pipeline registry. {{ pipelines.count }} pipeline tasks with dependency ordering, 3 targets (dev/staging/prod), daily schedule.
+
+```bash
+uv run python dev/scripts/gen_config.py --dab-sql   # generate SQL + DDL (~4 min)
+databricks bundle deploy -t dev                       # deploy to Databricks
+```
+
+### SDK integration
+
+The platform uses 18 Databricks SDK services via `UnityClient` (wrapping `WorkspaceClient`):
+
+| Category | Services | Module | Purpose |
+|----------|----------|--------|---------|
+| **Catalog CRUD** | `catalogs`, `schemas`, `tables`, `volumes` | `lake/unity.py` | Unity Catalog object management |
+| **Data loading** | `files`, `statement_execution` | `lake/sync.py` | Parquet upload + COPY INTO |
+| **Governance** | `grants` | `lake/governance.py` | Declarative schema permissions with drift audit |
+| **Secrets** | `secrets` | `lake/unity.py` | Scope management, env var sync |
+| **Constraints** | `table_constraints` | `lake/unity.py` | PK/FK on claims tables for lineage |
+| **Jobs** | `jobs` | `lake/jobs.py` | Create/run/monitor pipeline jobs from Python |
+| **Compute** | `warehouses` | `lake/unity.py` | Find/create/start/stop warehouses by name |
+| **Quality** | `quality_monitors` | `lake/quality.py` | Lakehouse Monitoring profiles |
+| **Audit** | `system_schemas` | `lake/unity.py` | Table lineage and audit log queries |
+
+### Governance
+
+`src/aco/lake/governance.py` provides declarative grant management:
+
+```python
+from aco.lake.governance import GovernancePolicy, GrantRule, apply_governance
+
+policy = GovernancePolicy(
+    catalog="aco",
+    rules=[
+        GrantRule(group="analysts", schemas=["readmissions"], privileges=["SELECT"]),
+        GrantRule(group="pipeline_svc", schemas=["*"], privileges=["ALL_PRIVILEGES"]),
+    ],
+    phi_schemas=["core", "claims_preprocessing", "cclf"],
+)
+
+# Preview changes
+apply_governance(client, policy, dry_run=True)
+
+# Detect drift between declared and actual grants
+from aco.lake.governance import audit_governance
+drifts = audit_governance(client, policy)
+```
+
+PHI schemas, group privilege mappings, and quality monitor config are declared in `stack.toml`:
+
+```toml
+[databricks.governance]
+phi_schemas = ["core", "claims_preprocessing", "cclf", "input_layer"]
+
+[databricks.governance.groups.analysts]
+schemas = ["readmissions", "quality_measures"]
+privileges = ["SELECT"]
+```
+
+### Job orchestration
+
+`src/aco/lake/jobs.py` translates `Pipeline` objects into Databricks Jobs:
+
+```python
+from aco.lake.jobs import JobManager
+
+mgr = JobManager(client, catalog="aco")
+job_id = mgr.create_pipeline_job("readmissions", schedule="0 0 6 * * ?")
+run_id = mgr.run_now(job_id)
+status = mgr.get_run_status(run_id)
+```
+
+### Secret sync
+
+Push environment variables to Databricks secret scopes per `stack.toml [databricks.secrets]`:
+
+```bash
+uv run python dev/scripts/sync_secrets.py --dry-run   # preview
+uv run python dev/scripts/sync_secrets.py              # push to Databricks
+uv run python dev/scripts/sync_secrets.py --list       # show current secrets
+```
+
+### Quality monitoring
+
+```python
+from aco.lake.quality import setup_monitors
+setup_monitors(client, catalog="aco", schemas=["core", "claims_preprocessing"])
+```
+
+## Testing
+
+### Architecture
+
+Testing follows a semantic coverage model (`src/sem/`) rather than tracking raw line numbers. The system parses every `.py` file into an AST, builds stable semantic nodes (functions, branches, exception handlers, loops), and attaches three signal layers:
+
+```
+source → AST → semantic nodes → attach (ruff, ty, coverage) → planner
+```
+
+| Layer | Module | What it does |
+|-------|--------|-------------|
+| **Parse** | `sem.parse` | Walk AST, emit `SemanticNode` per function, branch, except, loop, return, raise |
+| **Enrich** | `sem.enrich` | Run `ruff check` and `ty check`, map diagnostics to tightest-span node |
+| **Runtime** | `sem.runtime` | Map `coverage.py` JSON report onto nodes (hit/miss, per-test contexts) |
+| **Plan** | `sem.plan` | Score uncovered nodes by priority, suggest next test targets |
+| **State** | `sem.state` | Persist node status across runs, reset on source hash change |
+
+Each node has a stable identity derived from its module path, qualified symbol name, kind, and ordinal position — never from line numbers:
+
+```
+app.config::load_settings::branch_if[1]
+aco.express.pharmacy::pharmacy_claims::except_handler[0]
+```
+
+#### Priority heuristic
+
+```
+priority =
+    uncovered_branch × 5
+  + ty_diagnostic × 4
+  + inside_partially_tested_function × 3
+  + uncovered_exception_path × 3
+  + ruff_warning × 2
+  − prior_failures × 2
+  − already_covered × 5
+```
+
+#### Structural invariants
+
+`tests/test_ast_coverage.py` runs at collection time (pure AST, no imports) and enforces:
+
+1. Every `@nw.narwhalify` function has a docstring
+2. Every `Expr(...)` call supplies `name`, `fn`, `output`, and `after`
+3. Every public Pydantic model is importable
+4. Every `pipe/*.py` module exports a `Pipeline` with non-empty `.exprs`
+5. Implementation ratio stays above baseline (ratchet)
+6. `express/` and `pipe/` modules are symmetric
+
+### Pre-commit hooks
+
+Git hooks live in `dev/hooks/` (tracked) and are activated via:
+
+```bash
+git config core.hooksPath dev/hooks
+```
+
+The shell hook is a 3-line wrapper. All logic lives in `src/sem/hooks.py`:
+
+```bash
+#!/usr/bin/env bash
+exec uv run python -m sem.hooks
+```
+
+#### Smart test selection
+
+The hook classifies staged files and runs only what is relevant:
+
+| What changed | Tests run | Why |
+|-------------|-----------|-----|
+| `src/sem/*.py` | `tests/sem/` + `test_ast_coverage.py` | Module tests + structural invariants |
+| `src/aco/*.py` + `src/sem/*.py` | `tests/aco/` + `tests/sem/` + structural | Both module test dirs |
+| `tests/bib/test_sync.py` | `tests/bib/` | Changed test dir |
+| `pyproject.toml` or `conftest.py` | Full suite ({{ tests.formatted }}+ tests) | Infrastructure change |
+| `notebooks/pfs_calcs.py` | `marimo check` + notebook execution | Notebook validation only |
+| `README.md` only | Nothing | No testable changes |
+
+The mapping rule is: `src/<module>/` changes → `tests/<module>/` runs. Any `src/` change also triggers `test_ast_coverage.py` to verify structural invariants haven't regressed.
+
+#### Forcing full suite
+
+```bash
+GIT_PRE_COMMIT_FULL=1 git commit -m "message"
+```
+
+#### Hook steps (in order)
+
+1. **Venv recovery** — if `uv run python -c 'import sys'` fails, run `uv sync --dev`
+2. **Config regeneration** — if `stack.toml` or `gen_config.py` changed, regenerate CI workflows
+3. **Ruff lint + format** — only staged `.py` files
+4. **AST parse check** — verify staged source files have valid syntax
+5. **Pytest** — targeted or full suite based on what changed
+6. **Marimo check** — only if notebooks are staged
+7. **Notebook execution** — only staged notebooks in the safe-to-run list
+
+#### Post-commit
+
+`dev/hooks/post-commit` rebuilds the docs Docker image in the background after every commit so the documentation site stays current with docstring changes.
+
+### Running tests
+
+```bash
+uv run python -m pytest tests/                    # full suite
+uv run python -m pytest tests/sem/                 # one module
+uv run python -m pytest tests/ -m "not stub"       # skip stub inventory
+uv run python -m pytest tests/ -m stub             # only stub status
+uv run python -m pytest tests/test_ast_coverage.py # structural invariants only
+```
+
+### README generation
+
+`README.md` is generated from `README.md.j2` using live codebase data. Never edit `README.md` directly.
+
+```bash
+uv run python dev/scripts/gen_readme.py            # regenerate from template
+uv run python dev/scripts/gen_readme.py --check     # verify freshness (CI gate)
+uv run python dev/scripts/gen_readme.py --dry-run   # preview to stdout
+```
+
+Dynamic values pulled from: `compose.yml` (services), `pyproject.toml` (modules), pipeline registry (pipelines + steps), `pytest --collect-only` (test count), `api.auth.manifest` (credentials), `.gitea/workflows/` (workflow count).
+
+## Project layout
+
+```
+stack/
+├── compose.yml                  Docker Compose ({{ services.count }} services)
+├── stack.toml                   Centralised configuration
+├── pyproject.toml               Python project (uv, optional deps per module)
+├── src/
+{% for m in src_modules %}
+│   {{ "├" if not loop.last else "└" }}── {{ m.name }}/{{ " " * (25 - m.name|length) }}{{ m.desc }}
+{% endfor %}
+├── infra/                       Service configs and Dockerfiles
+│   ├── images/                  All Dockerfiles (api, notebooks, zotero, docs, mc)
+│   ├── traefik/                 Reverse proxy + loch CSS injection
+│   ├── grafana/                 Dashboards and datasource provisioning
+│   ├── prometheus/              Metrics collection
+│   ├── loki/                    Log aggregation
+│   └── ...                      coredns, nginx, trino, polaris, rustfs, act-runner, gitea
+├── assets/                      Branding, styles, generated artifacts
+│   ├── css/                     Per-service CSS (dashboard, gitea, marimo, woodpecker)
+│   ├── icons/                   Favicons, logos, coverage badge
+│   └── nature.py                Nature-conformant chart palette
+├── cloud/                       Cloud provider deployment guides
+│   ├── self-hosted/             Docker Compose (default)
+│   ├── aws/                     S3, RDS, Glue, ECS
+│   ├── gcp/                     GCS, Cloud SQL, BigQuery, Cloud Run
+│   └── azure/                   ABFS, Azure SQL, Unity Catalog, Container Apps
+├── dev/                         Dev tooling
+│   ├── scripts/                 Code generators, bootstrap, supply chain tools
+│   ├── hooks/                   Git hooks (tracked, core.hooksPath = dev/hooks)
+│   ├── seeds/                   Reference data (BCDA samples, CMS docs)
+│   └── pipelines/               CI-agnostic pipeline specs
+├── tests/                       {{ tests.formatted }} tests at 99% coverage
+├── notebooks/                   Marimo notebooks
+├── docs/                        Docusaurus site
+├── data/                        DuckDB, bib.sqlite, BCDA/CMS data, zotero (gitignored)
+└── .env                         Derived credentials (not in git)
+```
+
+## Prerequisites
+
+- Docker (rootless mode)
+- NVIDIA GPU with container toolkit
+- Domain or `/etc/hosts` entries for `*.homelab.fhirworx.io`
+
+```bash
+# Fix for rootless NVIDIA containers
+sudo sed -i 's/#no-cgroups = false/no-cgroups = true/' /etc/nvidia-container-runtime/config.toml
+```
+
+## SSH access
+
+```
+# ~/.ssh/config
+Host gitea
+    HostName homelab.fhirworx.io
+    Port 2222
+    User git
+    IdentityFile ~/.ssh/gitea_ed25519
+    IdentitiesOnly yes
+```

dev/hooks/post-commit

@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+# Rebuild docs image and restart container in the background.
+# Runs after every commit so docs stay current with docstring
+# and bib.sqlite changes.
+
+(
+  cd "$(git rev-parse --show-toplevel)" || exit
+  docker build -q -t fhirworx/docs:latest -f docs/Dockerfile . \
+    && docker compose up -d docs
+) &>/dev/null &

dev/hooks/pre-commit

@@ -1,51 +1,4 @@
 #!/usr/bin/env bash
-set -e
+# Thin wrapper — all logic lives in src/sem/hooks.py (tracked).
-
+# Install: git config core.hooksPath dev/hooks
-# Auto-recover broken venv before running any checks
+exec uv run python -m sem.hooks
-if ! uv run python -c 'import sys' 2>/dev/null; then
-  echo "==> pre-commit: venv broken, running uv sync --dev"
-  uv sync --dev
-fi
-
-# Auto-regenerate derived config if stack.toml or generator changed
-CONF_CHANGED=$(git diff --cached --name-only -- stack.toml 'dev/scripts/gen_config.py' 'dev/scripts/backends/*.py')
-if [ -n "$CONF_CHANGED" ]; then
-  echo "==> pre-commit: regenerating config from stack.toml"
-  uv run python dev/scripts/gen_config.py
-  git add .woodpecker/*.yml .github/workflows/*.yml coredns/hosts coredns/Corefile 2>/dev/null || true
-fi
-
-echo "==> pre-commit: ruff check (staged)"
-# Only lint files that are staged for commit (avoid dirty-tree noise).
-STAGED=$(git diff --cached --name-only --diff-filter=d -- 'src/*.py' 'tests/*.py')
-if [ -n "$STAGED" ]; then
-  echo "$STAGED" | xargs uv run ruff check --quiet
-fi
-
-echo "==> pre-commit: ruff format --check (staged)"
-if [ -n "$STAGED" ]; then
-  echo "$STAGED" | xargs uv run ruff format --check --quiet
-fi
-
-echo "==> pre-commit: pytest"
-uv run python -m pytest tests/ --no-cov --tb=short -q
-
-echo "==> pre-commit: marimo check"
-uvx marimo check notebooks/*.py
-
-echo "==> pre-commit: notebook run check"
-# Only check notebooks that run standalone (no external services).
-# Notebooks needing containers (api, nessie, polaris) or optional
-# deps (vega_datasets, pyzotero) are excluded.
-for nb in \
-  notebooks/acodb_explorer.py \
-  notebooks/bib_explorer.py \
-  notebooks/cms_quality_measures.py \
-  notebooks/pfs_calcs.py \
-  notebooks/sql_generator.py \
-; do
-  echo "    running $nb"
-  uv run python "$nb" 2>&1 || { echo "FAILED: $nb"; exit 1; }
-done
-
-echo "==> pre-commit: all checks passed"

dev/scripts/gen_readme.py

@@ -0,0 +1,300 @@
+"""Generate README.md from README.md.j2 using live codebase data.
+
+Collects counts, tables, and metadata from the actual codebase:
+    - Service count and names from compose.yml
+    - Module list from pyproject.toml optional-dependencies
+    - Pipeline registry from src/aco/pipe/
+    - Test count via pytest --collect-only
+    - Credential count from api.auth.manifest
+    - Network list from compose.yml
+    - Workflow list from .gitea/workflows/
+
+Usage::
+
+    uv run python dev/scripts/gen_readme.py            # generate README.md
+    uv run python dev/scripts/gen_readme.py --check     # verify up-to-date
+    uv run python dev/scripts/gen_readme.py --dry-run   # print to stdout
+"""
+
+from __future__ import annotations
+
+import argparse
+import ast
+import subprocess
+import sys
+import tomllib
+from pathlib import Path
+
+import yaml
+from jinja2 import Environment, FileSystemLoader
+
+ROOT = Path(__file__).resolve().parent.parent.parent
+
+# ── Data collectors ──────────────────────────────────────────────────
+
+
+def _collect_services() -> dict:
+    """Parse compose.yml for service and network info."""
+    compose = yaml.safe_load((ROOT / "compose.yml").read_text())
+    services = sorted(compose.get("services", {}).keys())
+    networks = sorted(compose.get("networks", {}).keys())
+    return {"names": services, "count": len(services), "networks": networks}
+
+
+def _collect_modules() -> dict:
+    """Parse pyproject.toml for module list."""
+    with open(ROOT / "pyproject.toml", "rb") as f:
+        pyp = tomllib.load(f)
+
+    opt_deps = pyp["project"]["optional-dependencies"]
+    cloud = {"aws", "gcp", "azure"}
+    aggregates = {"all", "lake"}
+    modules = sorted(k for k in opt_deps if k not in cloud and k not in aggregates)
+
+    build_modules = (
+        pyp.get("tool", {})
+        .get("uv", {})
+        .get("build-backend", {})
+        .get("module-name", [])
+    )
+
+    return {
+        "names": modules,
+        "count": len(modules),
+        "cloud": sorted(cloud & set(opt_deps)),
+        "aggregates": sorted(aggregates & set(opt_deps)),
+        "build_modules": build_modules,
+    }
+
+
+def _collect_pipelines() -> dict:
+    """Import pipeline registry and collect metadata."""
+    sys.path.insert(0, str(ROOT / "src"))
+    from aco.pipe import registry
+
+    # Fallback descriptions for pipelines without module docstrings
+    fallback_docs = {
+        "ahrq_measures": "AHRQ quality indicators",
+        "cclf": "CCLF file parsing",
+        "claims_preprocessing": "Claim matching, encounter IDs",
+        "cms_quality_measures": "CMS ACO quality metrics",
+        "core": "Encounters, patients, providers",
+        "data_quality": "Validation and completeness",
+        "hcc_suspecting": "HCC risk adjustment",
+        "input_layer": "Raw CCLF/BCDA ingestion",
+        "main": "Full pipeline orchestration",
+        "opps": "Outpatient payment system",
+        "pharmacy": "Part D drug utilization",
+        "provider_attribution": "Provider-patient assignment",
+        "quality_measures": "Composite quality scores",
+        "readmissions": "30-day readmission rates",
+    }
+
+    pipelines = []
+    for name in sorted(registry.keys()):
+        p = registry[name]
+        doc = ""
+        # Try to get the pipeline module's docstring
+        mod_path = ROOT / "src" / "aco" / "pipe" / f"{name}.py"
+        if mod_path.exists():
+            try:
+                tree = ast.parse(mod_path.read_text())
+                raw = ast.get_docstring(tree) or ""
+                doc = raw.split("\n")[0].rstrip(".")
+            except SyntaxError:
+                pass
+        # Use fallback if docstring is empty
+        if not doc:
+            doc = fallback_docs.get(name, "")
+        pipelines.append(
+            {
+                "name": name,
+                "steps": len(p.exprs),
+                "doc": doc,
+            }
+        )
+
+    total_steps = sum(p["steps"] for p in pipelines)
+    return {
+        "list": pipelines,
+        "count": len(pipelines),
+        "total_steps": total_steps,
+    }
+
+
+def _collect_tests() -> dict:
+    """Run pytest --collect-only to get test count."""
+    result = subprocess.run(
+        ["uv", "run", "python", "-m", "pytest", "tests/", "--collect-only", "-q"],
+        capture_output=True,
+        text=True,
+        cwd=str(ROOT),
+        timeout=60,
+    )
+    # Last line: "12291 tests collected in 3.03s"
+    for line in result.stdout.strip().splitlines():
+        if "test" in line and "collected" in line:
+            count = int(line.split()[0])
+            return {"count": count, "formatted": f"{count:,}"}
+
+    return {"count": 0, "formatted": "?"}
+
+
+def _collect_credentials() -> dict:
+    """Count credentials from auth manifest."""
+    sys.path.insert(0, str(ROOT / "src"))
+    try:
+        from api.auth.manifest import CREDENTIALS
+
+        return {"count": len(CREDENTIALS)}
+    except ImportError:
+        return {"count": 0}
+
+
+def _collect_workflows() -> dict:
+    """List CI workflow files."""
+    workflow_dirs = [
+        ROOT / ".gitea" / "workflows",
+        ROOT / ".github" / "workflows",
+        ROOT / ".woodpecker",
+    ]
+    workflows = []
+    for d in workflow_dirs:
+        if d.is_dir():
+            for f in sorted(d.glob("*.yml")):
+                workflows.append({"name": f.name, "path": str(f.relative_to(ROOT))})
+    return {"list": workflows, "count": len(workflows)}
+
+
+def _collect_src_tree() -> list[dict]:
+    """Build the src/ module listing with descriptions."""
+    descriptions = {
+        "aco": "ACO analytics (express, pipe, table, lake, load)",
+        "api": "FastAPI server + auth + diag",
+        "bcda": "BCDA FHIR R4 client",
+        "bib": "Zotero bibliography store, tags, metadata",
+        "bls": "BLS data",
+        "ccw": "CCW data dictionary",
+        "cli": "CLI entry point (typer)",
+        "cms": "CMS public data tables",
+        "conf": "Config loader, storage abstraction, table base",
+        "opps": "Outpatient Prospective Payment System",
+        "perf": "Pipeline telemetry (OpenTelemetry)",
+        "pfs": "Physician Fee Schedule",
+        "rex": "REX fixed-width file processing",
+        "sem": "Semantic coverage orchestration",
+    }
+
+    src = ROOT / "src"
+    modules = []
+    for d in sorted(src.iterdir()):
+        if (
+            d.is_dir()
+            and not d.name.startswith(("_", "."))
+            and not d.name.endswith(".egg-info")
+        ):
+            modules.append(
+                {
+                    "name": d.name,
+                    "desc": descriptions.get(d.name, ""),
+                }
+            )
+    return modules
+
+
+def collect_all() -> dict:
+    """Collect all dynamic data for template rendering."""
+    services = _collect_services()
+    modules = _collect_modules()
+    pipelines = _collect_pipelines()
+    tests = _collect_tests()
+    credentials = _collect_credentials()
+    workflows = _collect_workflows()
+    src_modules = _collect_src_tree()
+
+    return {
+        "services": services,
+        "modules": modules,
+        "pipelines": pipelines,
+        "tests": tests,
+        "credentials": credentials,
+        "workflows": workflows,
+        "src_modules": src_modules,
+    }
+
+
+# ── Rendering ────────────────────────────────────────────────────────
+
+
+def render(data: dict) -> str:
+    """Render README.md.j2 with collected data."""
+    env = Environment(
+        loader=FileSystemLoader(str(ROOT)),
+        keep_trailing_newline=True,
+        trim_blocks=True,
+        lstrip_blocks=True,
+    )
+    env.filters["backtick"] = lambda s: f"`{s}`"
+    template = env.get_template("README.md.j2")
+    return template.render(**data)
+
+
+# ── CLI ──────────────────────────────────────────────────────────────
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument(
+        "--check",
+        action="store_true",
+        help="verify README.md matches template output (exit 1 if stale)",
+    )
+    parser.add_argument(
+        "--dry-run",
+        action="store_true",
+        help="print rendered output to stdout instead of writing",
+    )
+    args = parser.parse_args()
+
+    data = collect_all()
+    rendered = render(data)
+
+    if args.dry_run:
+        print(rendered, end="")
+        return 0
+
+    readme = ROOT / "README.md"
+
+    if args.check:
+        current = readme.read_text(encoding="utf-8")
+        if current == rendered:
+            print("README.md is up-to-date.")
+            return 0
+        else:
+            # Show first differing line
+            cur_lines = current.splitlines()
+            ren_lines = rendered.splitlines()
+            for i, (a, b) in enumerate(zip(cur_lines, ren_lines), 1):
+                if a != b:
+                    print(f"README.md is stale (first diff at line {i}):")
+                    print(f"  current:  {a[:100]}")
+                    print(f"  expected: {b[:100]}")
+                    break
+            else:
+                len_diff = len(cur_lines) - len(ren_lines)
+                print(f"README.md is stale (length differs by {len_diff} lines)")
+            print("\nRun: uv run python dev/scripts/gen_readme.py")
+            return 1
+
+    readme.write_text(rendered, encoding="utf-8")
+    print(
+        f"README.md generated ({data['services']['count']} services, "
+        f"{data['pipelines']['count']} pipelines, "
+        f"{data['tests']['formatted']} tests, "
+        f"{data['modules']['count']} modules)"
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

dev/scripts/sync_secrets.py

@@ -0,0 +1,71 @@
+"""Sync environment variables to Databricks secret scopes.
+
+Reads the ``[databricks.secrets]`` section from ``stack.toml``
+and pushes matching environment variables into a Databricks scope
+via the SDK.
+
+Usage::
+
+    uv run python dev/scripts/sync_secrets.py            # sync
+    uv run python dev/scripts/sync_secrets.py --dry-run   # preview
+    uv run python dev/scripts/sync_secrets.py --list      # show current secrets
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+
+from conf import cfg
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument(
+        "--dry-run", action="store_true", help="preview without changes"
+    )
+    parser.add_argument(
+        "--list", action="store_true", help="list current secrets in scope"
+    )
+    args = parser.parse_args()
+
+    secrets_cfg = getattr(cfg.databricks, "secrets", None)
+    if not secrets_cfg:
+        print("No [databricks.secrets] section in stack.toml")
+        return 1
+
+    scope = secrets_cfg.scope
+    mapping_list = secrets_cfg.mapping or []
+
+    # Build env→key mapping
+    mapping = {item["env"]: item["key"] for item in mapping_list}
+
+    if not mapping:
+        print(f"No secret mappings defined in [databricks.secrets]")
+        return 0
+
+    from aco.lake.unity import UnityClient
+
+    client = UnityClient.from_env()
+
+    if args.list:
+        try:
+            keys = client.list_secrets(scope)
+            print(f"Secrets in scope '{scope}':")
+            for k in keys:
+                print(f"  {k}")
+            if not keys:
+                print("  (empty)")
+        except Exception as e:
+            print(f"Error listing secrets: {e}")
+        return 0
+
+    actions = client.sync_secrets(scope, mapping, dry_run=args.dry_run)
+    for action in actions:
+        print(action)
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

notebooks/pfs_calcs.py

@@ -45,20 +45,26 @@ def _(con, mo):
     )
     _localities = con.execute(
         """
-        SELECT DISTINCT locality, locality_name
+        SELECT DISTINCT mac, locality, locality_name
         FROM pfs.gpci
-        ORDER BY locality
+        ORDER BY locality_name
         """
     ).fetchall()
 
+    _loc_options = {
+        f"{name} ({loc})": f"{mac}|{loc}"
+        for mac, loc, name in _localities
+    }
+    _first_key = next(iter(_loc_options))
+
     year_picker = mo.ui.dropdown(
         options={str(y): y for y in _years},
         value=str(_years[-1]),
         label="Year",
     )
     locality_picker = mo.ui.dropdown(
-        options={f"{loc} - {name}": loc for loc, name in _localities},
+        options=_loc_options,
-        value=f"{_localities[0][0]} - {_localities[0][1]}",
+        value=_first_key,
         label="Locality",
     )
     hcpcs_input = mo.ui.text(value="99213", label="HCPCS Code")
@@ -70,7 +76,7 @@ def _(con, mo):
 @app.cell(hide_code=True)
 def _(hcpcs_input, locality_picker, year_picker):
     _year = year_picker.value
-    _locality = locality_picker.value
+    _mac, _locality = locality_picker.value.split("|")
     _hcpcs = hcpcs_input.value.strip()
     return
 
@@ -90,13 +96,15 @@ def _(mo):
 
 @app.cell
 def _(hcpcs_input, locality_picker, q, year_picker):
+    _mac, _loc = locality_picker.value.split("|")
     carrier_result = q(f"""
-        SELECT year, locality, hcpcs, mod,
+        SELECT year, mac, locality, hcpcs, mod,
                non_fac_fee, fac_fee,
                non_fac_limiting_charge, fac_limiting_charge
         FROM pfs.carrier_locality
         WHERE year = {year_picker.value}
-          AND locality = '{locality_picker.value}'
+          AND mac = '{_mac}'
+          AND locality = '{_loc}'
           AND hcpcs = '{hcpcs_input.value.strip()}'
         ORDER BY mod
     """)
@@ -126,19 +134,19 @@ def _(hcpcs_input, locality_picker, pl, q, year_picker):
     from pfs.rules import RULES
 
     _year = int(year_picker.value)
-    _locality = locality_picker.value
+    _mac, _locality = locality_picker.value.split("|")
     _hcpcs = hcpcs_input.value.strip()
     _cf = RULES[_year].conversion_factor
 
     rvu_df = q(f"""
-        SELECT *, {_cf} as conv_factor, '{_locality}' as locality
+        SELECT *, {_cf} as conv_factor, '{_locality}' as locality, '{_mac}' as mac
         FROM pfs.rvu
         WHERE year = {_year} AND hcpcs = '{_hcpcs}'
     """)
 
     gpci_df = q(f"""
         SELECT * FROM pfs.gpci
-        WHERE year = {_year} AND locality = '{_locality}'
+        WHERE year = {_year} AND mac = '{_mac}' AND locality = '{_locality}'
     """)
 
     calc_nf = payment(rvu_df, gpci_df, facility=False).select(
@@ -172,6 +180,7 @@ def _(hcpcs_input, locality_picker, q, year_picker):
     from pfs.rules import RULES as _RULES
 
     _year = int(year_picker.value)
+    _mac, _loc = locality_picker.value.split("|")
     _cf = _RULES[_year].conversion_factor
 
     sql_result = q(f"""
@@ -179,6 +188,7 @@ def _(hcpcs_input, locality_picker, q, year_picker):
             r.hcpcs,
             r.mod,
             r.year,
+            g.mac,
             g.locality,
             g.locality_name,
             r.work_rvu,
@@ -204,7 +214,8 @@ def _(hcpcs_input, locality_picker, q, year_picker):
         JOIN pfs.gpci g ON r.year = g.year
         WHERE r.year = {_year}
           AND r.hcpcs = '{hcpcs_input.value.strip()}'
-          AND g.locality = '{locality_picker.value}'
+          AND g.mac = '{_mac}'
+          AND g.locality = '{_loc}'
         ORDER BY r.mod
     """)
     sql_result
@@ -227,6 +238,7 @@ def _(carrier_result, mo, pl, calc_result):
         _carrier_nf = _carrier_base.head(1).select("non_fac_fee").item()
         _calc_nf = _calc_base.head(1).select("non_fac_payment").item()
         _diff = round(abs(_carrier_nf - _calc_nf), 2)
+        _match = "EXACT MATCH" if _diff == 0 else f"${_diff:.2f} difference"
         _comparison = f"""
         ## Comparison
 
@@ -234,10 +246,11 @@ def _(carrier_result, mo, pl, calc_result):
         |--------|-----------------|
         | CMS Carrier File | **${_carrier_nf:.2f}** |
         | Calculated (RVU x GPCI x CF) | **${_calc_nf:.2f}** |
-        | Difference | **${_diff:.2f}** |
+        | Difference | **{_match}** |
 
-        Small differences are normal — CMS rounds intermediate values
+        The MAC-aware join (RVU x GPCI by MAC + locality) produces
-        and may apply adjustments not captured in the base formula.
+        exact parity with CMS carrier file rates.  Any difference
+        indicates a carrier-priced or status-indicator edge case.
         """
     else:
         _comparison = "*Select a valid year/locality/code combination to see comparison.*"

notebooks/skin_sub_pricing.py

@@ -0,0 +1,478 @@
+import marimo
+
+__generated_with = "0.21.1"
+app = marimo.App(width="medium")
+
+
+@app.cell(hide_code=True)
+def _():
+    import marimo as mo
+
+    return (mo,)
+
+
+@app.cell(hide_code=True)
+def _(mo):
+    mo.md("""
+    # Skin Substitute Pricing Over Time
+
+    Tracks skin substitute application code RVUs and locality-based
+    payment amounts (PFS) alongside quarterly ASP drug pricing, layered
+    over the OPPS payment method timeline (pass-through → high/low →
+    flat rate).
+
+    **Data sources:**
+    - `pfs.rvu` — Work, PE, MP RVUs for application codes 15271–15278
+    - `pfs.carrier_locality` — CMS-published locality fees (ground truth)
+    - `pfs.gpci` — Geographic practice cost indices by MAC/locality
+    - `skin_subs.asp_quarterly` — Quarterly ASP + 6% payment limits
+    - `opps.skin_sub_addendum_b` — OPPS status indicators and APC rates
+    """)
+    return
+
+
+@app.cell(hide_code=True)
+def _():
+    import altair as alt
+    import polars as pl
+    from conf import connect
+
+    con = connect.duckdb()
+
+    def q(sql):
+        return con.execute(sql).pl()
+
+    return alt, con, pl, q
+
+
+@app.cell(hide_code=True)
+def _(mo):
+    mo.md("""
+    ## 1. Application Code RVUs Over Time
+
+    CPT 15271–15278 are the procedure codes for applying skin
+    substitutes. Their RVUs determine the surgeon's payment
+    independent of the product cost.
+    """)
+    return
+
+
+@app.cell
+def _(alt, q):
+    rvu_ts = q("""
+        SELECT year, hcpcs,
+               CASE hcpcs
+                   WHEN '15271' THEN 'Trunk/limbs <100cm² (initial)'
+                   WHEN '15272' THEN 'Trunk/limbs <100cm² (add-on)'
+                   WHEN '15273' THEN 'Trunk/limbs ≥100cm² (initial)'
+                   WHEN '15274' THEN 'Trunk/limbs ≥100cm² (add-on)'
+                   WHEN '15275' THEN 'Face/hands/feet <100cm² (initial)'
+                   WHEN '15276' THEN 'Face/hands/feet <100cm² (add-on)'
+                   WHEN '15277' THEN 'Face/hands/feet ≥100cm² (initial)'
+                   WHEN '15278' THEN 'Face/hands/feet ≥100cm² (add-on)'
+               END as description,
+               work_rvu, non_fac_pe_rvu, fac_pe_rvu, mp_rvu,
+               work_rvu + non_fac_pe_rvu + mp_rvu as non_fac_total,
+               work_rvu + fac_pe_rvu + mp_rvu as fac_total
+        FROM pfs.rvu
+        WHERE hcpcs IN ('15271','15272','15273','15274',
+                        '15275','15276','15277','15278')
+        ORDER BY year, hcpcs
+    """)
+
+    rvu_chart = (
+        alt.Chart(rvu_ts.to_pandas())
+        .mark_line(point=True)
+        .encode(
+            x=alt.X("year:O", title="Year"),
+            y=alt.Y("non_fac_total:Q", title="Total Non-Facility RVUs"),
+            color=alt.Color("description:N", title="Code"),
+            tooltip=["year", "hcpcs", "description",
+                     "work_rvu", "non_fac_pe_rvu", "mp_rvu", "non_fac_total"],
+        )
+        .properties(title="Application Code RVUs (Non-Facility)", width=700, height=400)
+    )
+    rvu_chart
+    return
+
+
+@app.cell(hide_code=True)
+def _(con, mo):
+    _regions = con.execute("""
+        SELECT DISTINCT locality_name
+        FROM pfs.gpci WHERE year = 2025
+        ORDER BY locality_name
+    """).fetchall()
+
+    region_picker = mo.ui.dropdown(
+        options={r[0]: r[0] for r in _regions},
+        value="MANHATTAN",
+        label="Region",
+    )
+    mo.md(f"""
+    ## 2. Application Code Payment by Region
+
+    CMS-published carrier locality fees for skin sub application codes.
+    Select a region to see how payments vary geographically over time.
+
+    {region_picker}
+    """)
+    return (region_picker,)
+
+
+@app.cell(hide_code=True)
+def _(alt, q, region_picker):
+    _region = region_picker.value
+
+    carrier_ts = q(f"""
+        SELECT c.year, c.hcpcs,
+               CASE c.hcpcs
+                   WHEN '15271' THEN 'Trunk <100cm²'
+                   WHEN '15272' THEN 'Trunk <100cm² add-on'
+                   WHEN '15273' THEN 'Trunk ≥100cm²'
+                   WHEN '15274' THEN 'Trunk ≥100cm² add-on'
+                   WHEN '15275' THEN 'Face/hands <100cm²'
+                   WHEN '15276' THEN 'Face/hands <100cm² add-on'
+                   WHEN '15277' THEN 'Face/hands ≥100cm²'
+                   WHEN '15278' THEN 'Face/hands ≥100cm² add-on'
+               END as description,
+               c.non_fac_fee, c.fac_fee,
+               g.locality_name
+        FROM pfs.carrier_locality c
+        JOIN pfs.gpci g ON c.mac = g.mac AND c.locality = g.locality AND c.year = g.year
+        WHERE c.hcpcs IN ('15271','15272','15273','15274',
+                          '15275','15276','15277','15278')
+          AND g.locality_name = '{_region}'
+        ORDER BY c.year, c.hcpcs
+    """)
+
+    carrier_chart = (
+        alt.Chart(carrier_ts.to_pandas())
+        .mark_line(point=True)
+        .encode(
+            x=alt.X("year:O", title="Year"),
+            y=alt.Y("non_fac_fee:Q", title="Non-Facility Fee ($)"),
+            color=alt.Color("description:N", title="Code"),
+            tooltip=["year", "hcpcs", "description", "non_fac_fee",
+                     "fac_fee", "locality_name"],
+        )
+        .properties(
+            title=f"Application Code Fees — {_region}",
+            width=700, height=400,
+        )
+    )
+    carrier_chart
+    return
+
+
+@app.cell(hide_code=True)
+def _(mo):
+    mo.md("""
+    ## 3. Regional Comparison — 15271 (Primary Application)
+
+    Non-facility fee for 15271 (trunk/limbs, <100cm², initial
+    application) across selected high-volume regions.
+    """)
+    return
+
+
+@app.cell(hide_code=True)
+def _(alt, q):
+    regions = [
+        "MANHATTAN", "ALASKA*", "REST OF FLORIDA", "REST OF TEXAS",
+        "REST OF CALIFORNIA", "CHICAGO", "DETROIT", "ATLANTA",
+        "HAWAII, GUAM", "SOUTH CAROLINA",
+    ]
+    region_list = ", ".join(f"'{r}'" for r in regions)
+
+    regional_15271 = q(f"""
+        SELECT c.year, g.locality_name,
+               c.non_fac_fee
+        FROM pfs.carrier_locality c
+        JOIN pfs.gpci g ON c.mac = g.mac AND c.locality = g.locality AND c.year = g.year
+        WHERE c.hcpcs = '15271'
+          AND g.locality_name IN ({region_list})
+        ORDER BY c.year, g.locality_name
+    """)
+
+    regional_chart = (
+        alt.Chart(regional_15271.to_pandas())
+        .mark_line(point=True)
+        .encode(
+            x=alt.X("year:O", title="Year"),
+            y=alt.Y("non_fac_fee:Q", title="Non-Facility Fee ($)"),
+            color=alt.Color("locality_name:N", title="Region"),
+            tooltip=["year", "locality_name", "non_fac_fee"],
+        )
+        .properties(
+            title="15271 Non-Facility Fee — Regional Comparison",
+            width=700, height=400,
+        )
+    )
+    regional_chart
+    return
+
+
+@app.cell(hide_code=True)
+def _(mo):
+    mo.md("""
+    ## 4. ASP Drug Pricing Over Time
+
+    Quarterly ASP + 6% payment limits for skin substitute products.
+    The flat horizontal line at **$127.28** marks the CY2026 flat rate —
+    products above it lose revenue, products below it gain.
+    """)
+    return
+
+
+@app.cell(hide_code=True)
+def _(alt, pl, q):
+    # Top products by volume (most quarters of data)
+    top_products = q("""
+        SELECT hcpcs_code, short_description,
+               count(*) as quarters
+        FROM skin_subs.asp_quarterly
+        GROUP BY hcpcs_code, short_description
+        HAVING count(*) >= 8
+        ORDER BY quarters DESC
+        LIMIT 15
+    """)
+
+    top_codes = top_products.select("hcpcs_code").to_series().to_list()
+    code_list = ", ".join(f"'{c}'" for c in top_codes)
+
+    asp_ts = q(f"""
+        SELECT quarter, hcpcs_code, short_description,
+               payment_limit, asp_per_unit
+        FROM skin_subs.asp_quarterly
+        WHERE hcpcs_code IN ({code_list})
+        ORDER BY quarter, hcpcs_code
+    """)
+
+    flat_rate_rule = (
+        alt.Chart(pl.DataFrame({"y": [127.28]}).to_pandas())
+        .mark_rule(color="red", strokeDash=[4, 4], strokeWidth=2)
+        .encode(y="y:Q")
+    )
+
+    asp_lines = (
+        alt.Chart(asp_ts.to_pandas())
+        .mark_line()
+        .encode(
+            x=alt.X("quarter:O", title="Quarter",
+                     axis=alt.Axis(labelAngle=-45, labelFontSize=8)),
+            y=alt.Y("payment_limit:Q", title="Payment Limit (ASP + 6%) $",
+                     scale=alt.Scale(domainMax=800)),
+            color=alt.Color("short_description:N", title="Product"),
+            tooltip=["quarter", "hcpcs_code", "short_description",
+                     "payment_limit", "asp_per_unit"],
+        )
+    )
+
+    asp_chart = (
+        (asp_lines + flat_rate_rule)
+        .properties(
+            title="ASP Quarterly Payment Limits — Top 15 Products",
+            width=700, height=450,
+        )
+    )
+    asp_chart
+    return
+
+
+@app.cell(hide_code=True)
+def _(mo):
+    mo.md("""
+    ## 5. OPPS Payment Method Timeline
+
+    How CMS has paid for skin substitutes in the outpatient setting:
+
+    | Period | Method | Detail |
+    |--------|--------|--------|
+    | Pre-2023 | Pass-through | ASP + 6% per unit |
+    | 2023–2025 | High/low split | High-cost: pass-through; Low-cost: packaged into APC |
+    | 2026+ | Flat rate | $127.28/cm² for all products |
+
+    The table below shows OPPS status indicators and payment rates
+    for skin sub codes over time.
+    """)
+    return
+
+
+@app.cell(hide_code=True)
+def _(q):
+    opps_ts = q("""
+        SELECT year, hcpcs, short_description,
+               status_indicator, apc, payment_rate
+        FROM opps.skin_sub_addendum_b
+        WHERE payment_rate IS NOT NULL
+          AND payment_rate > 0
+        ORDER BY year, hcpcs
+    """)
+    opps_ts
+    return
+
+
+@app.cell(hide_code=True)
+def _(mo):
+    mo.md("""
+    ## 6. Total Episode Cost: Product + Application
+
+    For a typical 25cm² wound, the total Medicare payment is:
+
+    - **Product cost**: ASP + 6% payment limit × 25 units
+    - **Application fee**: 15271 non-facility fee (PFS, locality-specific)
+    - **Post-2026**: $127.28 × 25 = $3,182 flat + application fee
+
+    This chart shows the combined cost trend for selected products.
+    """)
+    return
+
+
+@app.cell(hide_code=True)
+def _(alt, pl, q, region_picker):
+    _region = region_picker.value
+
+    # Get 15271 fee for the selected region, all years
+    app_fee = q(f"""
+        SELECT c.year, c.non_fac_fee as application_fee
+        FROM pfs.carrier_locality c
+        JOIN pfs.gpci g ON c.mac = g.mac AND c.locality = g.locality AND c.year = g.year
+        WHERE c.hcpcs = '15271'
+          AND g.locality_name = '{_region}'
+    """)
+
+    # Get annual ASP for top products (use Q1 of each year)
+    episode_data = q("""
+        SELECT
+            CAST(substr(quarter, 1, 4) AS INTEGER) as year,
+            hcpcs_code, short_description,
+            payment_limit,
+            payment_limit * 25 as product_cost_25cm2
+        FROM skin_subs.asp_quarterly
+        WHERE substr(quarter, 6, 2) = 'Q1'
+          AND hcpcs_code IN ('Q4101','Q4186','Q4132','Q4116','Q4100')
+        ORDER BY year, hcpcs_code
+    """)
+
+    # Join to get total episode cost
+    combined = (
+        episode_data.join(app_fee, on="year", how="left")
+        .with_columns(
+            (pl.col("product_cost_25cm2") + pl.col("application_fee").fill_null(0))
+            .alias("total_episode_cost")
+        )
+    )
+
+    flat_line = (
+        alt.Chart(pl.DataFrame({"y": [127.28 * 25]}).to_pandas())
+        .mark_rule(color="red", strokeDash=[4, 4], strokeWidth=2)
+        .encode(y="y:Q")
+    )
+
+    episode_lines = (
+        alt.Chart(combined.to_pandas())
+        .mark_line(point=True)
+        .encode(
+            x=alt.X("year:O", title="Year"),
+            y=alt.Y("total_episode_cost:Q", title="Total Episode Cost ($)"),
+            color=alt.Color("short_description:N", title="Product"),
+            tooltip=["year", "hcpcs_code", "short_description",
+                     "product_cost_25cm2", "application_fee",
+                     "total_episode_cost"],
+        )
+    )
+
+    episode_chart = (
+        (episode_lines + flat_line)
+        .properties(
+            title=f"Total Episode Cost (25cm² wound) — {_region}",
+            width=700, height=400,
+        )
+    )
+    episode_chart
+    return
+
+
+@app.cell(hide_code=True)
+def _(mo):
+    mo.md("""
+    ## 7. Flat-Rate Impact: Winners and Losers
+
+    Products with ASP + 6% above $127.28 lose revenue under the 2026
+    flat rate; those below gain. This chart shows the latest quarter's
+    payment limit vs. the flat rate.
+    """)
+    return
+
+
+@app.cell(hide_code=True)
+def _(alt, q):
+    impact = q("""
+        SELECT hcpcs_code, short_description,
+               payment_limit,
+               127.28 as flat_rate,
+               payment_limit - 127.28 as delta,
+               CASE
+                   WHEN payment_limit > 127.28 THEN 'loses'
+                   WHEN payment_limit < 127.28 THEN 'gains'
+                   ELSE 'neutral'
+               END as impact
+        FROM skin_subs.asp_quarterly
+        WHERE quarter = (SELECT max(quarter) FROM skin_subs.asp_quarterly)
+        ORDER BY delta DESC
+    """)
+
+    impact_chart = (
+        alt.Chart(impact.to_pandas())
+        .mark_bar()
+        .encode(
+            x=alt.X("delta:Q", title="Payment Limit − $127.28 Flat Rate"),
+            y=alt.Y("short_description:N", title="", sort="-x",
+                     axis=alt.Axis(labelLimit=300)),
+            color=alt.Color("impact:N",
+                            scale=alt.Scale(
+                                domain=["loses", "gains", "neutral"],
+                                range=["#d62728", "#2ca02c", "#7f7f7f"],
+                            ),
+                            title="Impact"),
+            tooltip=["hcpcs_code", "short_description",
+                     "payment_limit", "flat_rate", "delta"],
+        )
+        .properties(title="CY2026 Flat-Rate Impact by Product", width=700)
+    )
+    impact_chart
+    return
+
+
+@app.cell(hide_code=True)
+def _(mo):
+    mo.md("""
+    ## 8. Summary: Application Code Fee Range by Year
+
+    Min, median, and max non-facility fee across all MAC/locality
+    combinations for application code 15271.
+    """)
+    return
+
+
+@app.cell(hide_code=True)
+def _(q):
+    fee_summary = q("""
+        SELECT c.year,
+               count(*) as localities,
+               round(min(c.non_fac_fee), 2) as min_fee,
+               round(percentile_cont(0.5) WITHIN GROUP (ORDER BY c.non_fac_fee), 2) as median_fee,
+               round(max(c.non_fac_fee), 2) as max_fee,
+               round(max(c.non_fac_fee) - min(c.non_fac_fee), 2) as fee_spread
+        FROM pfs.carrier_locality c
+        WHERE c.hcpcs = '15271'
+        GROUP BY c.year
+        ORDER BY c.year
+    """)
+    fee_summary
+    return
+
+
+if __name__ == "__main__":
+    app.run()

pyproject.toml

@@ -87,6 +87,9 @@ perf = [
     "opentelemetry-semantic-conventions>=0.46b0",
     "psutil>=5.9.0",
 ]
+sem = [
+    "coverage>=7.13.4",
+]
 lake = [
     "stack[aco]",
     "databricks-sdk>=0.85.0",
@@ -136,6 +139,7 @@ dev = [
     "pandas>=3.0.1",
     "obstore>=0.9.2",
     "s3fs>=2026.2.0",
+    "jinja2>=3.1.0",
 ]
 
 [build-system]
@@ -151,8 +155,11 @@ ignore = ["E501", "E741"]
 
 [tool.pytest.ini_options]
 testpaths = ["tests"]
+markers = [
+    "stub: marks tests that report stub vs implemented status (deselect with '-m \"not stub\"')",
+]
 
 [tool.uv.build-backend]
-module-name = ["aco", "api", "bcda", "bib", "bls", "ccw", "cli", "cms", "conf", "opps", "perf", "pfs", "rex"]
+module-name = ["aco", "api", "bcda", "bib", "bls", "ccw", "cli", "cms", "conf", "opps", "perf", "pfs", "rex", "sem"]
 namespace = true
 source-exclude = ["compose.yml","infra/**","data/**","notebooks/**","tuva/**","assets/**","docs/**","dev/**","bundle/**","cloud/**"]

src/aco/lake/__init__.py

@@ -53,6 +53,12 @@ from .engine import execute as execute
 from .transpile import transpile as transpile
 
 try:
+    from .governance import GovernancePolicy as GovernancePolicy
+    from .governance import GrantRule as GrantRule
+    from .governance import apply_governance as apply_governance
+    from .governance import audit_governance as audit_governance
+    from .jobs import JobManager as JobManager
+    from .quality import setup_monitors as setup_monitors
     from .sync import SyncReport as SyncReport
     from .sync import sync as sync
     from .unity import UnityCatalog as UnityCatalog

src/aco/lake/governance.py

@@ -0,0 +1,183 @@
+"""Declarative Unity Catalog grant management with drift detection.
+
+Defines a ``GovernancePolicy`` that maps groups to schema-level
+privileges, then applies or audits those grants via ``ws.grants``.
+
+Usage::
+
+    from aco.lake.governance import GovernancePolicy, apply_governance
+
+    policy = GovernancePolicy(
+        catalog="aco",
+        rules=[
+            GrantRule(group="analysts", schemas=["readmissions"], privileges=["SELECT"]),
+            GrantRule(group="pipeline_svc", schemas=["*"], privileges=["ALL_PRIVILEGES"]),
+        ],
+    )
+
+    client = UnityClient.from_env()
+    apply_governance(client, policy, dry_run=True)
+"""
+
+from __future__ import annotations
+
+from databricks.sdk.service.catalog import (
+    PermissionsChange,
+    Privilege,
+    SecurableType,
+)
+from pydantic import BaseModel
+
+
+class GrantRule(BaseModel):
+    """A single group-to-schemas privilege mapping."""
+
+    group: str
+    schemas: list[str]
+    privileges: list[str]
+
+
+class GovernancePolicy(BaseModel):
+    """Declarative specification of who gets what access to which schemas."""
+
+    catalog: str
+    rules: list[GrantRule] = []
+    phi_schemas: list[str] = []
+
+
+class GrantDrift(BaseModel):
+    """One permission that differs between declared policy and actual state."""
+
+    securable: str
+    principal: str
+    declared: list[str]
+    actual: list[str]
+    action: str  # "grant" | "revoke" | "missing"
+
+
+def _resolve_schemas(
+    rule: GrantRule,
+    all_schemas: list[str],
+) -> list[str]:
+    """Expand wildcard ``*`` to all schemas."""
+    if rule.schemas == ["*"]:
+        return all_schemas
+    return [s for s in rule.schemas if s in all_schemas]
+
+
+def apply_governance(
+    client,
+    policy: GovernancePolicy,
+    *,
+    dry_run: bool = True,
+) -> list[str]:
+    """Apply a governance policy to Unity Catalog schemas.
+
+    Parameters
+    ----------
+    client : UnityClient
+        Authenticated client with ``_ws`` attribute.
+    policy : GovernancePolicy
+        Declarative grant specification.
+    dry_run : bool
+        If True, report what would change without applying.
+
+    Returns
+    -------
+    list[str]
+        Summary of actions taken or planned.
+    """
+    ws = client._ws
+    all_schemas = [s.name for s in ws.schemas.list(catalog_name=policy.catalog)]
+    actions: list[str] = []
+
+    for rule in policy.rules:
+        schemas = _resolve_schemas(rule, all_schemas)
+        privs = [Privilege(p) for p in rule.privileges]
+
+        for schema in schemas:
+            full_name = f"{policy.catalog}.{schema}"
+            change = PermissionsChange(
+                add=privs,
+                principal=rule.group,
+            )
+            desc = (
+                f"GRANT {', '.join(rule.privileges)} "
+                f"ON SCHEMA {full_name} TO {rule.group}"
+            )
+
+            if dry_run:
+                actions.append(f"[dry-run] {desc}")
+            else:
+                ws.grants.update(
+                    securable_type=SecurableType.SCHEMA,
+                    full_name=full_name,
+                    changes=[change],
+                )
+                actions.append(desc)
+
+    return actions
+
+
+def audit_governance(
+    client,
+    policy: GovernancePolicy,
+) -> list[GrantDrift]:
+    """Compare declared policy against actual Unity Catalog grants.
+
+    Returns a list of ``GrantDrift`` objects describing mismatches.
+    """
+    ws = client._ws
+    all_schemas = [s.name for s in ws.schemas.list(catalog_name=policy.catalog)]
+    drifts: list[GrantDrift] = []
+
+    # Build declared map: (schema, group) → set of privileges
+    declared: dict[tuple[str, str], set[str]] = {}
+    for rule in policy.rules:
+        schemas = _resolve_schemas(rule, all_schemas)
+        for schema in schemas:
+            key = (schema, rule.group)
+            declared.setdefault(key, set()).update(rule.privileges)
+
+    # Check each declared (schema, group) against actual
+    for (schema, group), expected_privs in declared.items():
+        full_name = f"{policy.catalog}.{schema}"
+        try:
+            resp = ws.grants.get(
+                securable_type=SecurableType.SCHEMA,
+                full_name=full_name,
+            )
+        except Exception:
+            drifts.append(
+                GrantDrift(
+                    securable=full_name,
+                    principal=group,
+                    declared=sorted(expected_privs),
+                    actual=[],
+                    action="missing",
+                )
+            )
+            continue
+
+        # Find this group's actual privileges
+        actual_privs: set[str] = set()
+        for assignment in resp.privilege_assignments or []:
+            if assignment.principal == group:
+                for p in assignment.privileges or []:
+                    if p.privilege:
+                        actual_privs.add(p.privilege.value)
+
+        if expected_privs != actual_privs:
+            missing = expected_privs - actual_privs
+            action = "grant" if missing else "revoke"
+            drifts.append(
+                GrantDrift(
+                    securable=full_name,
+                    principal=group,
+                    declared=sorted(expected_privs),
+                    actual=sorted(actual_privs),
+                    action=action,
+                )
+            )
+
+    return drifts

src/aco/lake/jobs.py

@@ -0,0 +1,244 @@
+"""Programmatic Databricks job management from pipeline definitions.
+
+Translates ``Pipeline`` objects into Databricks Jobs with task
+dependencies, then submits and monitors runs via ``ws.jobs``.
+
+Usage::
+
+    from aco.lake.jobs import JobManager
+    from aco.lake.unity import UnityClient
+
+    client = UnityClient.from_env()
+    mgr = JobManager(client, catalog="aco", warehouse_name="stack-sql")
+
+    # Create a job from a pipeline definition
+    job_id = mgr.create_pipeline_job("readmissions")
+
+    # Trigger a run
+    run_id = mgr.run_now(job_id)
+
+    # Poll status
+    status = mgr.get_run_status(run_id)
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from databricks.sdk.service.jobs import (
+    CronSchedule,
+    JobCluster,
+    PauseStatus,
+    PythonWheelTask,
+    Task,
+    TaskDependency,
+)
+from pydantic import BaseModel
+
+
+class RunStatus(BaseModel):
+    """Simplified run status for polling."""
+
+    run_id: int
+    state: str
+    result_state: str | None = None
+    state_message: str = ""
+
+    @property
+    def is_terminal(self) -> bool:
+        return self.state in ("TERMINATED", "SKIPPED", "INTERNAL_ERROR")
+
+    @property
+    def succeeded(self) -> bool:
+        return self.result_state == "SUCCESS"
+
+
+class JobManager:
+    """Manage Databricks jobs backed by the pipeline registry.
+
+    Parameters
+    ----------
+    client : UnityClient
+        Authenticated client.
+    catalog : str
+        Unity Catalog name for table references.
+    package_name : str
+        Python package name for wheel tasks (default: "stack").
+    entry_point : str
+        CLI entry point name (default: "cli").
+    """
+
+    def __init__(
+        self,
+        client,
+        catalog: str = "aco",
+        package_name: str = "stack",
+        entry_point: str = "cli",
+    ) -> None:
+        self._ws = client._ws
+        self._catalog = catalog
+        self._package = package_name
+        self._entry = entry_point
+
+    def create_pipeline_job(
+        self,
+        pipeline_name: str,
+        *,
+        cluster_spec: dict[str, Any] | None = None,
+        schedule: str | None = None,
+        timezone: str = "America/New_York",
+    ) -> int:
+        """Create a Databricks Job from a pipeline definition.
+
+        Parameters
+        ----------
+        pipeline_name : str
+            Name of the pipeline in the registry.
+        cluster_spec : dict, optional
+            Cluster config (spark_version, node_type_id, num_workers).
+            Defaults to a small i3.xlarge cluster.
+        schedule : str, optional
+            Quartz cron expression. If None, job is manual-only.
+
+        Returns
+        -------
+        int
+            The created job ID.
+        """
+        from aco.pipe import registry
+
+        pipeline = registry[pipeline_name]
+        tasks = self._build_tasks(pipeline_name, pipeline)
+
+        cluster_def = cluster_spec or {
+            "spark_version": "15.4.x-scala2.12",
+            "node_type_id": "i3.xlarge",
+            "num_workers": 1,
+        }
+
+        job_clusters = [
+            JobCluster(
+                job_cluster_key="pipeline_cluster",
+                new_cluster=cluster_def,
+            )
+        ]
+
+        kwargs: dict[str, Any] = {
+            "name": f"stack_{pipeline_name}",
+            "tasks": tasks,
+            "job_clusters": job_clusters,
+        }
+
+        if schedule:
+            kwargs["schedule"] = CronSchedule(
+                quartz_cron_expression=schedule,
+                timezone_id=timezone,
+                pause_status=PauseStatus.UNPAUSED,
+            )
+
+        resp = self._ws.jobs.create(**kwargs)
+        return resp.job_id
+
+    def create_all_jobs(
+        self,
+        *,
+        cluster_spec: dict[str, Any] | None = None,
+        schedule: str | None = None,
+    ) -> dict[str, int]:
+        """Create jobs for all pipelines in the registry.
+
+        Returns mapping of pipeline_name → job_id.
+        """
+        from aco.pipe import registry
+
+        cluster_def = cluster_spec or {
+            "spark_version": "15.4.x-scala2.12",
+            "node_type_id": "i3.xlarge",
+            "num_workers": 1,
+        }
+
+        tasks: list[Task] = []
+        for name, pipeline in sorted(registry.items()):
+            tasks.extend(self._build_tasks(name, pipeline))
+
+        job_clusters = [
+            JobCluster(
+                job_cluster_key="pipeline_cluster",
+                new_cluster=cluster_def,
+            )
+        ]
+
+        kwargs: dict[str, Any] = {
+            "name": "stack_pipelines",
+            "tasks": tasks,
+            "job_clusters": job_clusters,
+        }
+        if schedule:
+            kwargs["schedule"] = CronSchedule(
+                quartz_cron_expression=schedule,
+                timezone_id="America/New_York",
+                pause_status=PauseStatus.UNPAUSED,
+            )
+
+        resp = self._ws.jobs.create(**kwargs)
+        return {"all": resp.job_id}
+
+    def run_now(self, job_id: int) -> int:
+        """Trigger an immediate run of a job. Returns the run ID."""
+        resp = self._ws.jobs.run_now(job_id=job_id)
+        return resp.run_id
+
+    def get_run_status(self, run_id: int) -> RunStatus:
+        """Get the current status of a run."""
+        run = self._ws.jobs.get_run(run_id=run_id)
+        state = run.state
+        return RunStatus(
+            run_id=run_id,
+            state=str(state.life_cycle_state) if state else "UNKNOWN",
+            result_state=str(state.result_state)
+            if state and state.result_state
+            else None,
+            state_message=state.state_message or "" if state else "",
+        )
+
+    def list_jobs(self, name_prefix: str = "stack_") -> list[dict[str, Any]]:
+        """List jobs matching a name prefix."""
+        results = []
+        for job in self._ws.jobs.list(name=name_prefix):
+            results.append(
+                {
+                    "job_id": job.job_id,
+                    "name": job.settings.name if job.settings else "",
+                }
+            )
+        return results
+
+    def delete_job(self, job_id: int) -> None:
+        """Delete a job by ID."""
+        self._ws.jobs.delete(job_id=job_id)
+
+    def _build_tasks(self, pipeline_name: str, pipeline) -> list[Task]:
+        """Convert a Pipeline into a list of Databricks Tasks."""
+        tasks: list[Task] = []
+
+        # Build dependencies from pipeline.upstream if available
+        upstream = getattr(pipeline, "upstream", []) or []
+
+        task = Task(
+            task_key=pipeline_name,
+            job_cluster_key="pipeline_cluster",
+            python_wheel_task=PythonWheelTask(
+                package_name=self._package,
+                entry_point=self._entry,
+                named_parameters={
+                    "command": "run",
+                    "pipeline": pipeline_name,
+                    "--target": "spark",
+                    "--catalog": self._catalog,
+                    "--save": "true",
+                },
+            ),
+            depends_on=[TaskDependency(task_key=dep) for dep in upstream] or None,
+        )
+        tasks.append(task)
+        return tasks

src/aco/lake/quality.py

@@ -0,0 +1,165 @@
+"""Lakehouse Monitoring for healthcare data quality.
+
+Sets up Unity Catalog quality monitors on key tables with
+healthcare-specific profiles: null rate checks on required fields,
+value range validation on dollar amounts, date ordering constraints,
+and rate bounds on quality measures.
+
+Usage::
+
+    from aco.lake.quality import setup_monitors, list_monitors
+
+    client = UnityClient.from_env()
+    results = setup_monitors(client, catalog="aco", schemas=["core", "claims_preprocessing"])
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from pydantic import BaseModel
+
+
+class MonitorProfile(BaseModel):
+    """Configuration for a quality monitor on a table."""
+
+    table: str
+    output_schema: str = "_monitoring"
+    schedule: str | None = None
+    slicing_exprs: list[str] = []
+    custom_metrics: list[dict[str, Any]] = []
+
+
+class MonitorResult(BaseModel):
+    """Summary of a monitor's latest state."""
+
+    table: str
+    status: str
+    refresh_id: str | None = None
+    dashboard_id: str | None = None
+
+
+def _default_profiles(catalog: str, schema: str) -> list[MonitorProfile]:
+    """Generate default monitoring profiles for healthcare schemas."""
+    return [
+        MonitorProfile(
+            table=f"{catalog}.{schema}",
+            output_schema=f"{catalog}._monitoring",
+        )
+    ]
+
+
+def setup_monitors(
+    client,
+    catalog: str,
+    schemas: list[str],
+    *,
+    output_schema: str = "_monitoring",
+    dry_run: bool = True,
+) -> list[str]:
+    """Create Lakehouse Monitoring profiles on tables in the given schemas.
+
+    Parameters
+    ----------
+    client : UnityClient
+        Authenticated client.
+    catalog : str
+        Unity Catalog name.
+    schemas : list[str]
+        Schema names to monitor.
+    output_schema : str
+        Where to write monitoring metrics (default: ``_monitoring``).
+    dry_run : bool
+        If True, report what would be created.
+
+    Returns
+    -------
+    list[str]
+        Summary of actions taken or planned.
+    """
+    ws = client._ws
+    actions: list[str] = []
+    output_full = f"{catalog}.{output_schema}"
+
+    # Ensure output schema exists
+    if not dry_run:
+        try:
+            ws.schemas.get(output_full)
+        except Exception:
+            ws.schemas.create(
+                name=output_schema,
+                catalog_name=catalog,
+                comment="Lakehouse Monitoring output tables",
+            )
+            actions.append(f"created schema {output_full}")
+
+    for schema in schemas:
+        try:
+            tables = list(ws.tables.list(catalog_name=catalog, schema_name=schema))
+        except Exception as e:
+            actions.append(f"ERROR listing {catalog}.{schema}: {e}")
+            continue
+
+        for table in tables:
+            full_name = table.full_name or f"{catalog}.{schema}.{table.name}"
+            desc = f"monitor {full_name} → {output_full}"
+
+            if dry_run:
+                actions.append(f"[dry-run] {desc}")
+                continue
+
+            try:
+                # Check if monitor already exists
+                ws.quality_monitors.get(table_name=full_name)
+                actions.append(f"[exists] {desc}")
+            except Exception:
+                try:
+                    ws.quality_monitors.create(
+                        table_name=full_name,
+                        output_schema_name=output_full,
+                        assets_dir=f"/Shared/monitoring/{catalog}",
+                    )
+                    actions.append(f"created {desc}")
+                except Exception as e:
+                    actions.append(f"ERROR {desc}: {e}")
+
+    return actions
+
+
+def list_monitors(client, catalog: str) -> list[MonitorResult]:
+    """List all quality monitors in a catalog."""
+    ws = client._ws
+    results: list[MonitorResult] = []
+
+    for schema_info in ws.schemas.list(catalog_name=catalog):
+        schema = schema_info.name
+        if not schema or schema.startswith("_"):
+            continue
+        try:
+            for table in ws.tables.list(catalog_name=catalog, schema_name=schema):
+                full_name = table.full_name or f"{catalog}.{schema}.{table.name}"
+                try:
+                    mon = ws.quality_monitors.get(table_name=full_name)
+                    results.append(
+                        MonitorResult(
+                            table=full_name,
+                            status=str(mon.status) if mon.status else "unknown",
+                            dashboard_id=mon.dashboard_id,
+                        )
+                    )
+                except Exception:
+                    pass
+        except Exception:
+            pass
+
+    return results
+
+
+def run_refresh(client, table_name: str) -> str | None:
+    """Trigger a monitor refresh. Returns the refresh ID."""
+    ws = client._ws
+    try:
+        resp = ws.quality_monitors.run_refresh(table_name=table_name)
+        return resp.refresh_id
+    except Exception:
+        return None

src/aco/lake/unity.py

@@ -399,6 +399,263 @@ class UnityClient:
         full = f"{catalog_name}.{schema_name}.{volume_name}"
         self._ws.volumes.delete(full)
 
+    # ── Grants ────────────────────────────────────────────────────
+
+    def list_grants(self, securable_type: str, full_name: str) -> list[dict[str, Any]]:
+        """List grants on a securable (CATALOG, SCHEMA, TABLE)."""
+        from databricks.sdk.service.catalog import SecurableType
+
+        resp = self._ws.grants.get(
+            securable_type=SecurableType(securable_type),
+            full_name=full_name,
+        )
+        results = []
+        for pa in resp.privilege_assignments or []:
+            privs = [p.privilege.value for p in (pa.privileges or []) if p.privilege]
+            results.append({"principal": pa.principal, "privileges": privs})
+        return results
+
+    def update_grants(
+        self,
+        securable_type: str,
+        full_name: str,
+        principal: str,
+        add: list[str] | None = None,
+        remove: list[str] | None = None,
+    ) -> None:
+        """Update grants on a securable."""
+        from databricks.sdk.service.catalog import (
+            PermissionsChange,
+            Privilege,
+        )
+        from databricks.sdk.service.catalog import (
+            SecurableType as ST,
+        )
+
+        changes = []
+        if add:
+            changes.append(
+                PermissionsChange(
+                    add=[Privilege(p) for p in add],
+                    principal=principal,
+                )
+            )
+        if remove:
+            changes.append(
+                PermissionsChange(
+                    remove=[Privilege(p) for p in remove],
+                    principal=principal,
+                )
+            )
+        self._ws.grants.update(
+            securable_type=ST(securable_type),
+            full_name=full_name,
+            changes=changes,
+        )
+
+    # ── Secrets ───────────────────────────────────────────────────
+
+    def ensure_scope(self, scope: str) -> None:
+        """Create a secret scope if it does not exist."""
+        existing = {s.name for s in self._ws.secrets.list_scopes()}
+        if scope not in existing:
+            self._ws.secrets.create_scope(scope=scope)
+
+    def put_secret(self, scope: str, key: str, value: str) -> None:
+        """Store a secret in a Databricks scope."""
+        self._ws.secrets.put_secret(scope=scope, key=key, string_value=value)
+
+    def get_secret(self, scope: str, key: str) -> str:
+        """Retrieve a secret value from a Databricks scope."""
+        resp = self._ws.secrets.get_secret(scope=scope, key=key)
+        return resp.value or ""
+
+    def list_secrets(self, scope: str) -> list[str]:
+        """List secret keys in a scope."""
+        return [s.key for s in self._ws.secrets.list_secrets(scope=scope) if s.key]
+
+    def sync_secrets(
+        self, scope: str, mapping: dict[str, str], *, dry_run: bool = True
+    ) -> list[str]:
+        """Push environment variables into a Databricks secret scope.
+
+        Parameters
+        ----------
+        scope : str
+            Target secret scope name.
+        mapping : dict[str, str]
+            Mapping of env var name → secret key name.
+        dry_run : bool
+            If True, report what would be synced.
+
+        Returns
+        -------
+        list[str]
+            Summary of actions.
+        """
+        actions: list[str] = []
+        if not dry_run:
+            self.ensure_scope(scope)
+
+        for env_var, secret_key in sorted(mapping.items()):
+            value = os.environ.get(env_var, "")
+            if not value:
+                actions.append(f"[skip] {env_var} → {scope}/{secret_key} (not set)")
+                continue
+            if dry_run:
+                actions.append(f"[dry-run] {env_var} → {scope}/{secret_key}")
+            else:
+                self.put_secret(scope, secret_key, value)
+                actions.append(f"synced {env_var} → {scope}/{secret_key}")
+        return actions
+
+    # ── Table Constraints ─────────────────────────────────────────
+
+    def set_primary_key(
+        self,
+        catalog_name: str,
+        schema_name: str,
+        table_name: str,
+        columns: list[str],
+        constraint_name: str | None = None,
+    ) -> None:
+        """Add a primary key constraint to a table."""
+        from databricks.sdk.service.catalog import (
+            PrimaryKeyConstraint,
+            TableConstraint,
+        )
+
+        full = f"{catalog_name}.{schema_name}.{table_name}"
+        name = constraint_name or f"pk_{table_name}"
+        self._ws.table_constraints.create(
+            full_name_arg=full,
+            constraint=TableConstraint(
+                primary_key_constraint=PrimaryKeyConstraint(
+                    name=name,
+                    child_columns=columns,
+                )
+            ),
+        )
+
+    def set_foreign_key(
+        self,
+        catalog_name: str,
+        schema_name: str,
+        table_name: str,
+        columns: list[str],
+        ref_catalog: str,
+        ref_schema: str,
+        ref_table: str,
+        ref_columns: list[str],
+        constraint_name: str | None = None,
+    ) -> None:
+        """Add a foreign key constraint between two tables."""
+        from databricks.sdk.service.catalog import (
+            ForeignKeyConstraint,
+            TableConstraint,
+        )
+
+        full = f"{catalog_name}.{schema_name}.{table_name}"
+        ref_full = f"{ref_catalog}.{ref_schema}.{ref_table}"
+        name = constraint_name or f"fk_{table_name}_{ref_table}"
+        self._ws.table_constraints.create(
+            full_name_arg=full,
+            constraint=TableConstraint(
+                foreign_key_constraint=ForeignKeyConstraint(
+                    name=name,
+                    child_columns=columns,
+                    parent_table=ref_full,
+                    parent_columns=ref_columns,
+                )
+            ),
+        )
+
+    # ── Warehouses ────────────────────────────────────────────────
+
+    def find_warehouse(self, name: str) -> str | None:
+        """Find a SQL warehouse by name. Returns its ID or None."""
+        for wh in self._ws.warehouses.list():
+            if wh.name == name:
+                return wh.id
+        return None
+
+    def ensure_warehouse(
+        self,
+        name: str,
+        size: str = "SMALL",
+        auto_stop_mins: int = 10,
+    ) -> str:
+        """Find or create a SQL warehouse. Returns its ID."""
+        existing = self.find_warehouse(name)
+        if existing:
+            return existing
+
+        resp = self._ws.warehouses.create_and_wait(
+            name=name,
+            cluster_size=size,
+            auto_stop_mins=auto_stop_mins,
+            enable_serverless_compute=True,
+        )
+        return resp.id
+
+    def start_warehouse(self, warehouse_id: str) -> None:
+        """Start a stopped warehouse."""
+        self._ws.warehouses.start(warehouse_id)
+
+    def stop_warehouse(self, warehouse_id: str) -> None:
+        """Stop a running warehouse."""
+        self._ws.warehouses.stop(warehouse_id)
+
+    # ── System Schemas ────────────────────────────────────────────
+
+    def enable_system_schema(self, metastore_id: str, schema_name: str) -> None:
+        """Enable a system schema (e.g. 'access', 'billing', 'lineage')."""
+        self._ws.system_schemas.enable(
+            metastore_id=metastore_id,
+            schema_name=schema_name,
+        )
+
+    def query_lineage(
+        self, catalog_name: str, warehouse_id: str, limit: int = 100
+    ) -> list[dict[str, Any]]:
+        """Query table lineage from system.access.table_lineage."""
+        sql = (
+            f"SELECT * FROM system.access.table_lineage "
+            f"WHERE target_table_catalog = '{catalog_name}' "
+            f"ORDER BY event_time DESC LIMIT {limit}"
+        )
+        resp = self._ws.statement_execution.execute_statement(
+            warehouse_id=warehouse_id,
+            statement=sql,
+            wait_timeout="30s",
+        )
+        if resp.result and resp.result.data_array:
+            columns = [c.name for c in (resp.manifest.schema.columns or [])]
+            return [dict(zip(columns, row)) for row in resp.result.data_array]
+        return []
+
+    def query_audit_log(
+        self, catalog_name: str, warehouse_id: str, days: int = 7
+    ) -> list[dict[str, Any]]:
+        """Query recent audit events from system.access.audit."""
+        sql = (
+            f"SELECT event_time, user_identity, action_name, "
+            f"request_params, response "
+            f"FROM system.access.audit "
+            f"WHERE event_date >= current_date() - INTERVAL {days} DAY "
+            f"AND request_params.full_name_arg LIKE '{catalog_name}.%' "
+            f"ORDER BY event_time DESC LIMIT 100"
+        )
+        resp = self._ws.statement_execution.execute_statement(
+            warehouse_id=warehouse_id,
+            statement=sql,
+            wait_timeout="30s",
+        )
+        if resp.result and resp.result.data_array:
+            columns = [c.name for c in (resp.manifest.schema.columns or [])]
+            return [dict(zip(columns, row)) for row in resp.result.data_array]
+        return []
+
 
 # ── Type Mapping ─────────────────────────────────────────────────────
 

src/pfs/calcs/payment.py

@@ -19,19 +19,25 @@ def payment(
 ) -> nw.DataFrame:
     """Calculate fee schedule payment amounts.
 
-    Joins RVU and GPCI data by locality and computes:
+    Joins RVU and GPCI data and computes:
 
     ``Payment = (Work_RVU × Work_GPCI + PE_RVU × PE_GPCI
                 + MP_RVU × MP_GPCI) × Conversion_Factor``
 
+    Joins on ``(mac, locality)`` — both columns are required.
+    Locality alone is not unique in the GPCI table (109 rows map to
+    47 locality codes).  Different MACs apply different GPCI values
+    to the same locality number, so joining without MAC produces
+    wrong results.
+
     Parameters
     ----------
     rvu : DataFrame
         RVU data with columns: ``hcpcs``, ``work_rvu``,
         ``non_fac_pe_rvu`` or ``fac_pe_rvu``, ``mp_rvu``,
-        ``conv_factor``, ``locality``.
+        ``conv_factor``, ``mac``, ``locality``.
     gpci : DataFrame
-        GPCI data with columns: ``locality``, ``work_gpci``,
+        GPCI data with columns: ``mac``, ``locality``, ``work_gpci``,
         ``pe_gpci``, ``mp_gpci``.
     facility : bool
         If True, use facility PE RVUs; otherwise non-facility.
@@ -43,7 +49,9 @@ def payment(
     """
     pe_col = "fac_pe_rvu" if facility else "non_fac_pe_rvu"
 
-    joined = rvu.join(gpci, on="locality", how="left")
+    # MAC + locality is the correct join — locality alone is not unique
+    # in the GPCI table and produces wrong results.
+    joined = rvu.join(gpci, on=["mac", "locality"], how="left")
 
     return joined.with_columns(
         (

src/sem/__init__.py

@@ -0,0 +1,92 @@
+"""Semantic coverage orchestration.
+
+AST-based coverage tracking that attaches Ruff diagnostics, ty type
+signals, and runtime coverage data to semantic nodes (functions, branches,
+exception handlers) rather than raw line numbers.
+
+Architecture
+------------
+
+::
+
+    source .py
+        → ast.parse
+            → SemanticNode[] (stable IDs, no line numbers)
+                → attach ruff diagnostics
+                → attach ty diagnostics
+                → attach coverage.py runtime data
+                    → rank by priority heuristic
+                        → next_targets()
+
+Layers
+------
+
+1. **parse** — Walk AST, emit a ``SemanticNode`` for every function,
+   method, class, branch (if/elif/else), match case, exception handler,
+   loop, return, raise, assert, and with block.
+
+2. **enrich** — Run ``ruff check --output-format=json`` and
+   ``ty check --output-format=json`` as subprocesses, map each diagnostic
+   to the tightest-span semantic node.
+
+3. **runtime** — Read a ``coverage.py`` JSON report and mark which nodes
+   were hit/missed.  Attach per-test execution contexts when available.
+
+4. **plan** — Score each uncovered node using a weighted heuristic
+   (branches > ty signals > partial functions > exception paths > ruff)
+   and return a ranked list of test targets.
+
+5. **state** — Persist node-level tracking across runs in a JSON file.
+   Nodes whose source hash changes are automatically reset.  Stale
+   entries (removed from the AST) are pruned on sync.
+
+6. **hooks** — Pre-commit hook orchestration.  Smart test selection
+   based on staged files: ``src/<mod>/`` changes → ``tests/<mod>/``
+   runs.  Falls back to full suite on infrastructure changes.
+
+Node identity
+-------------
+
+Every node has a stable ID built from its module path, qualified symbol
+name, kind, and ordinal position::
+
+    aco.express.pharmacy::pharmacy_claims::branch_if[0]
+    sem.parse::_Collector::visit_If::branch_else[0]
+
+Line numbers are stored as metadata in ``Span`` but never appear in the
+ID.  A SHA-1 hash of the source fragment detects edits — the state layer
+resets a node's history when its hash changes.
+
+Usage
+-----
+
+::
+
+    from sem import parse_module, attach_ruff, attach_coverage
+    from sem import rank_nodes, next_targets
+
+    nodes = parse_module(Path("src/aco/express/pharmacy.py"))
+    attach_ruff(nodes, Path("src/aco/express/pharmacy.py"))
+    attach_coverage(nodes, Path("coverage.json"), "src/aco/express/pharmacy.py")
+
+    for target in next_targets(nodes, limit=5):
+        print(target.node_id, target.priority, target.kind)
+"""
+
+from sem.enrich import attach_ruff, attach_ty
+from sem.nodes import NodeKind, SemanticNode
+from sem.parse import parse_module, parse_tree
+from sem.plan import next_targets, rank_nodes
+from sem.runtime import attach_coverage
+
+__all__ = [
+    "NodeKind",
+    "SemanticNode",
+    "parse_module",
+    "parse_tree",
+    "attach_ruff",
+    "attach_ty",
+    "attach_coverage",
+    "rank_nodes",
+    "next_targets",
+]

src/sem/enrich.py

@@ -0,0 +1,97 @@
+"""Layer 2 — Enrichment: attach Ruff and ty diagnostics to semantic nodes.
+
+Runs ``ruff check`` and ``ty check`` as subprocesses, parses their JSON
+output, and maps each diagnostic to the semantic node whose span
+contains the reported line.
+"""
+
+from __future__ import annotations
+
+import json
+import subprocess
+from pathlib import Path
+
+from sem.nodes import SemanticNode
+
+
+def _run_tool(cmd: list[str]) -> list[dict]:
+    """Run a linter/checker and return parsed JSON diagnostics."""
+    try:
+        result = subprocess.run(
+            cmd,
+            capture_output=True,
+            text=True,
+            timeout=120,
+        )
+    except FileNotFoundError:
+        return []
+    except subprocess.TimeoutExpired:
+        return []
+
+    raw = result.stdout.strip()
+    if not raw:
+        return []
+
+    try:
+        return json.loads(raw)
+    except json.JSONDecodeError:
+        return []
+
+
+def _find_node(nodes: list[SemanticNode], line: int) -> SemanticNode | None:
+    """Find the tightest (smallest span) node containing *line*."""
+    best: SemanticNode | None = None
+    best_size = float("inf")
+    for node in nodes:
+        if node.span.start_line <= line <= node.span.end_line:
+            size = node.span.end_line - node.span.start_line
+            if size < best_size:
+                best = node
+                best_size = size
+    return best
+
+
+def attach_ruff(
+    nodes: list[SemanticNode],
+    path: Path,
+    *,
+    ruff_bin: str = "ruff",
+) -> list[SemanticNode]:
+    """Run ``ruff check --output-format=json`` and attach codes to nodes.
+
+    Returns the same list (mutated in place) for chaining convenience.
+    """
+    diagnostics = _run_tool(
+        [ruff_bin, "check", "--output-format=json", "--no-fix", str(path)]
+    )
+    for diag in diagnostics:
+        line = diag.get("location", {}).get("row", 0)
+        code = diag.get("code", "")
+        if not line or not code:
+            continue
+        target = _find_node(nodes, line)
+        if target and code not in target.ruff_codes:
+            target.ruff_codes.append(code)
+    return nodes
+
+
+def attach_ty(
+    nodes: list[SemanticNode],
+    path: Path,
+    *,
+    ty_bin: str = "ty",
+) -> list[SemanticNode]:
+    """Run ``ty check --output-format=json`` and attach codes to nodes.
+
+    Returns the same list (mutated in place) for chaining convenience.
+    """
+    diagnostics = _run_tool([ty_bin, "check", "--output-format=json", str(path)])
+    for diag in diagnostics:
+        line = diag.get("location", {}).get("row", 0)
+        code = diag.get("code", diag.get("rule", ""))
+        if not line or not code:
+            continue
+        target = _find_node(nodes, line)
+        if target and code not in target.ty_codes:
+            target.ty_codes.append(code)
+    return nodes

src/sem/hooks.py

@@ -0,0 +1,357 @@
+"""Pre-commit hook logic — smart test selection via semantic analysis.
+
+This module implements the pre-commit orchestration for the stack repo.
+Instead of running all 12k+ tests on every commit, it analyses staged
+files to determine the minimal set of tests that cover the change.
+
+Architecture
+------------
+
+Git hooks live in ``dev/hooks/`` (tracked, version-controlled).
+``core.hooksPath`` is set to ``dev/hooks`` so git uses them directly —
+no manual copy to ``.git/hooks/`` needed.
+
+The shell hook is a 3-line wrapper::
+
+    #!/usr/bin/env bash
+    exec uv run python -m sem.hooks
+
+All intelligence is here in Python, where it can be tested, linted,
+and evolved alongside the codebase it protects.
+
+File classification
+-------------------
+
+Staged files are partitioned into categories:
+
+========== ============================================= =================
+Category   Pattern                                       Triggers
+========== ============================================= =================
+``src``    ``src/**/*.py``                                targeted pytest
+``tests``  ``tests/**/*.py``                              targeted pytest
+``infra``  ``conftest.py``, ``pyproject.toml``            full pytest
+``config`` ``stack.toml``, ``gen_config.py``, backends    config regen
+``notebooks`` ``notebooks/*.py``                          marimo + exec
+========== ============================================= =================
+
+Test selection rules
+--------------------
+
+1. ``src/aco/express/foo.py`` staged → run ``tests/aco/``
+2. ``src/sem/parse.py`` staged → run ``tests/sem/``
+3. Any ``src/`` change → also run ``tests/test_ast_coverage.py``
+   (structural invariants: docstrings, Expr contracts, import checks)
+4. ``tests/bib/test_sync.py`` staged → run ``tests/bib/``
+5. ``pyproject.toml`` or ``conftest.py`` → full suite
+6. Notebooks → ``marimo check`` + execute safe-list notebooks
+
+The mapping is always ``src/<module>/`` → ``tests/<module>/``.  Multiple
+modules can be targeted in the same commit (e.g. ``src/aco/`` +
+``src/sem/`` → ``tests/aco/`` + ``tests/sem/`` + structural).
+
+Steps (in execution order)
+--------------------------
+
+1. **Venv recovery** — auto-run ``uv sync --dev`` if interpreter broken
+2. **Config regen** — regenerate CI workflows if stack.toml changed
+3. **Ruff lint** — check only staged ``.py`` files
+4. **Ruff format** — verify formatting on staged files
+5. **AST parse** — validate syntax of staged src files
+6. **Pytest** — targeted or full suite
+7. **Marimo check** — validate staged notebooks
+8. **Notebook execution** — run staged notebooks in safe list
+
+Environment variables
+---------------------
+
+``GIT_PRE_COMMIT_FULL=1``
+    Force full test suite regardless of what changed.
+    Usage: ``GIT_PRE_COMMIT_FULL=1 git commit -m "message"``
+
+Post-commit
+-----------
+
+``dev/hooks/post-commit`` rebuilds the docs Docker image in the
+background after every commit so the documentation site stays current
+with docstring and bib.sqlite changes.
+
+Setup (once per clone)
+----------------------
+
+::
+
+    git config core.hooksPath dev/hooks
+"""
+
+from __future__ import annotations
+
+import ast
+import subprocess
+import sys
+from pathlib import Path
+
+
+def _staged_files() -> list[str]:
+    """Return list of staged file paths (excluding deletes)."""
+    result = subprocess.run(
+        ["git", "diff", "--cached", "--name-only", "--diff-filter=d"],
+        capture_output=True,
+        text=True,
+    )
+    return [f for f in result.stdout.strip().splitlines() if f]
+
+
+def _classify(staged: list[str]) -> dict[str, list[str]]:
+    """Partition staged files into categories."""
+    cats: dict[str, list[str]] = {
+        "src": [],
+        "tests": [],
+        "notebooks": [],
+        "infra": [],
+        "config": [],
+    }
+    infra_names = {"conftest.py", "pyproject.toml"}
+
+    for f in staged:
+        if f.startswith("src/") and f.endswith(".py"):
+            cats["src"].append(f)
+        elif f.startswith("tests/") and f.endswith(".py"):
+            cats["tests"].append(f)
+            if Path(f).name in infra_names:
+                cats["infra"].append(f)
+        elif f.startswith("notebooks/") and f.endswith(".py"):
+            cats["notebooks"].append(f)
+        elif Path(f).name in infra_names:
+            cats["infra"].append(f)
+        elif f in ("stack.toml", "dev/scripts/gen_config.py") or f.startswith(
+            "dev/scripts/backends/"
+        ):
+            cats["config"].append(f)
+    return cats
+
+
+def _changed_modules(src_files: list[str]) -> list[str]:
+    """Extract unique top-level module names from src/ paths.
+
+    ``src/aco/express/foo.py`` → ``aco``
+    ``src/sem/parse.py``       → ``sem``
+    """
+    mods: set[str] = set()
+    for f in src_files:
+        parts = f.split("/")
+        if len(parts) >= 2 and parts[0] == "src":
+            mods.add(parts[1])
+    return sorted(mods)
+
+
+def _changed_test_dirs(test_files: list[str]) -> list[str]:
+    """Extract unique test subdirectory names from test paths.
+
+    ``tests/aco/test_foo.py`` → ``tests/aco``
+    """
+    dirs: set[str] = set()
+    for f in test_files:
+        parts = f.split("/")
+        if len(parts) >= 3 and parts[0] == "tests":
+            dirs.add(f"tests/{parts[1]}")
+    return sorted(dirs)
+
+
+def _top_level_tests(test_files: list[str]) -> list[str]:
+    """Return test files directly under tests/ (not in subdirectories)."""
+    return [f for f in test_files if f.count("/") == 1]
+
+
+def check_syntax(src_files: list[str]) -> list[str]:
+    """Parse staged source files and return syntax errors."""
+    errors: list[str] = []
+    for f in src_files:
+        p = Path(f)
+        if not p.exists():
+            continue
+        try:
+            ast.parse(p.read_text(encoding="utf-8"), filename=f)
+        except SyntaxError as e:
+            errors.append(f"  {f}:{e.lineno}: {e.msg}")
+    return errors
+
+
+def compute_test_targets(
+    cats: dict[str, list[str]],
+    *,
+    force_full: bool = False,
+) -> tuple[list[str], str]:
+    """Determine which test paths to run.
+
+    Returns (test_paths, reason) where reason describes why this
+    scope was chosen.
+    """
+    if force_full:
+        return ["tests/"], "GIT_PRE_COMMIT_FULL=1"
+
+    if cats["infra"]:
+        return ["tests/"], "infrastructure files changed"
+
+    targets: list[str] = []
+    reasons: list[str] = []
+
+    # Map changed src/ modules to test directories
+    if cats["src"]:
+        for mod in _changed_modules(cats["src"]):
+            test_dir = Path("tests") / mod
+            if test_dir.is_dir():
+                targets.append(f"tests/{mod}/")
+                reasons.append(mod)
+
+        # Structural invariants on any src/ change
+        ast_cov = Path("tests/test_ast_coverage.py")
+        if ast_cov.exists():
+            targets.append(str(ast_cov))
+
+    # Include test dirs for directly-changed test files
+    if cats["tests"]:
+        for td in _changed_test_dirs(cats["tests"]):
+            if td + "/" not in targets and Path(td).is_dir():
+                targets.append(td + "/")
+                reasons.append(td.split("/")[-1])
+
+        for tf in _top_level_tests(cats["tests"]):
+            if tf not in targets:
+                targets.append(tf)
+
+    if not targets:
+        return [], "no testable changes"
+
+    # Deduplicate
+    seen: set[str] = set()
+    deduped: list[str] = []
+    for t in targets:
+        if t not in seen:
+            seen.add(t)
+            deduped.append(t)
+
+    return deduped, ", ".join(reasons) if reasons else "changed files"
+
+
+def run_step(label: str, cmd: list[str], **kwargs) -> int:
+    """Run a subprocess step, printing the label."""
+    print(f"==> pre-commit: {label}")
+    result = subprocess.run(cmd, **kwargs)
+    return result.returncode
+
+
+def main() -> int:
+    """Entry point for pre-commit hook orchestration."""
+    import os
+
+    staged = _staged_files()
+    if not staged:
+        print("==> pre-commit: nothing staged — skipped")
+        return 0
+
+    cats = _classify(staged)
+    force_full = os.environ.get("GIT_PRE_COMMIT_FULL", "0") == "1"
+
+    # ── Venv health ──────────────────────────────────────────────────
+    venv_ok = subprocess.run(
+        ["uv", "run", "python", "-c", "import sys"],
+        capture_output=True,
+    ).returncode
+    if venv_ok != 0:
+        run_step("venv broken, running uv sync --dev", ["uv", "sync", "--dev"])
+
+    # ── Config regeneration ──────────────────────────────────────────
+    if cats["config"]:
+        rc = run_step(
+            "regenerating config from stack.toml",
+            ["uv", "run", "python", "dev/scripts/gen_config.py"],
+        )
+        if rc != 0:
+            return rc
+        subprocess.run(
+            [
+                "git",
+                "add",
+                ".woodpecker/*.yml",
+                ".github/workflows/*.yml",
+                ".gitea/workflows/*.yml",
+                "coredns/hosts",
+                "coredns/Corefile",
+            ],
+            capture_output=True,
+        )
+
+    # ── Ruff lint + format (staged Python only) ──────────────────────
+    all_py = cats["src"] + cats["tests"]
+    if all_py:
+        rc = run_step(
+            "ruff check (staged)",
+            ["uv", "run", "ruff", "check", "--quiet"] + all_py,
+        )
+        if rc != 0:
+            return rc
+
+        rc = run_step(
+            "ruff format --check (staged)",
+            ["uv", "run", "ruff", "format", "--check", "--quiet"] + all_py,
+        )
+        if rc != 0:
+            return rc
+
+    # ── AST parse check (staged src) ─────────────────────────────────
+    if cats["src"]:
+        errors = check_syntax(cats["src"])
+        if errors:
+            print("==> pre-commit: syntax errors in staged files")
+            print("\n".join(errors))
+            return 1
+        print(f"==> pre-commit: sem parse check ({len(cats['src'])} files OK)")
+
+    # ── Pytest: smart selection ──────────────────────────────────────
+    targets, reason = compute_test_targets(cats, force_full=force_full)
+    if targets:
+        rc = run_step(
+            f"pytest ({reason})",
+            ["uv", "run", "python", "-m", "pytest"]
+            + targets
+            + ["--no-cov", "--tb=short", "-q"],
+        )
+        if rc != 0:
+            return rc
+    else:
+        print(f"==> pre-commit: pytest ({reason} — skipped)")
+
+    # ── Notebooks (only when staged) ─────────────────────────────────
+    if cats["notebooks"]:
+        rc = run_step(
+            "marimo check (staged notebooks)",
+            ["uvx", "marimo", "check"] + cats["notebooks"],
+        )
+        if rc != 0:
+            return rc
+
+        # Only execute notebooks in the safe-to-run list
+        safe = {
+            "acodb_explorer.py",
+            "bib_explorer.py",
+            "cms_quality_measures.py",
+            "pfs_calcs.py",
+            "sql_generator.py",
+        }
+        for nb in cats["notebooks"]:
+            if Path(nb).name in safe:
+                rc = run_step(
+                    f"notebook run: {nb}",
+                    ["uv", "run", "python", nb],
+                )
+                if rc != 0:
+                    return rc
+    else:
+        print("==> pre-commit: notebooks (no changes — skipped)")
+
+    print("==> pre-commit: all checks passed")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

src/sem/nodes.py

@@ -0,0 +1,119 @@
+"""Semantic node model.
+
+Each meaningful AST construct becomes a ``SemanticNode`` with a stable
+identity derived from its module path, qualified symbol name, node kind,
+and ordinal position — never from line numbers.
+"""
+
+from __future__ import annotations
+
+import enum
+import hashlib
+from dataclasses import dataclass, field
+
+
+class NodeKind(enum.Enum):
+    """Taxonomy of tracked AST constructs."""
+
+    FUNCTION = "function"
+    METHOD = "method"
+    CLASS = "class"
+    BRANCH_IF = "branch_if"
+    BRANCH_ELSE = "branch_else"
+    BRANCH_ELIF = "branch_elif"
+    MATCH_CASE = "match_case"
+    EXCEPT_HANDLER = "except_handler"
+    FOR_LOOP = "for_loop"
+    WHILE_LOOP = "while_loop"
+    WITH_BLOCK = "with_block"
+    RETURN = "return"
+    RAISE = "raise"
+    ASSERT = "assert"
+
+
+def _make_node_id(
+    module: str,
+    symbol_context: list[str],
+    kind: NodeKind,
+    ordinal: int,
+) -> str:
+    """Build a stable, human-readable node identity string.
+
+    Example: ``app.config::load_settings::branch_if[1]``
+    """
+    parts = [module]
+    if symbol_context:
+        parts.append("::".join(symbol_context))
+    parts.append(f"{kind.value}[{ordinal}]")
+    return "::".join(parts)
+
+
+@dataclass
+class Span:
+    """Byte-offset span within a source file (line numbers as metadata only)."""
+
+    start_line: int
+    end_line: int
+    start_col: int = 0
+    end_col: int = 0
+
+
+@dataclass
+class RuntimeInfo:
+    """Runtime coverage data attached after test execution."""
+
+    hit: bool = False
+    hit_count: int = 0
+    tests_seen: list[str] = field(default_factory=list)
+
+
+@dataclass
+class SemanticNode:
+    """A single tracked semantic unit in the codebase."""
+
+    node_id: str
+    kind: NodeKind
+    span: Span
+    module: str
+    symbol_context: list[str] = field(default_factory=list)
+    source_hash: str = ""
+
+    # Enrichment: Ruff diagnostics
+    ruff_codes: list[str] = field(default_factory=list)
+
+    # Enrichment: ty diagnostics
+    ty_codes: list[str] = field(default_factory=list)
+
+    # Runtime coverage
+    runtime: RuntimeInfo = field(default_factory=RuntimeInfo)
+
+    # Planner state
+    status: str = "unexplored"
+    priority: float = 0.0
+    attempts: int = 0
+    intent_tags: list[str] = field(default_factory=list)
+
+    @staticmethod
+    def build(
+        module: str,
+        symbol_context: list[str],
+        kind: NodeKind,
+        ordinal: int,
+        span: Span,
+        source_fragment: str = "",
+    ) -> SemanticNode:
+        """Construct a node with a stable ID and optional source hash."""
+        node_id = _make_node_id(module, symbol_context, kind, ordinal)
+        src_hash = ""
+        if source_fragment:
+            src_hash = hashlib.sha1(
+                source_fragment.encode(), usedforsecurity=False
+            ).hexdigest()[:12]
+        return SemanticNode(
+            node_id=node_id,
+            kind=kind,
+            span=span,
+            module=module,
+            symbol_context=list(symbol_context),
+            source_hash=src_hash,
+        )

src/sem/parse.py

@@ -0,0 +1,168 @@
+"""Layer 1 — Parse: build semantic nodes from Python AST.
+
+Walks a module's AST and emits a ``SemanticNode`` for every tracked
+construct: functions, methods, classes, branches, loops, exception
+handlers, returns, and raises.
+"""
+
+from __future__ import annotations
+
+import ast
+from pathlib import Path
+
+from sem.nodes import NodeKind, SemanticNode, Span
+
+
+def _span(node: ast.AST) -> Span:
+    return Span(
+        start_line=getattr(node, "lineno", 0),
+        end_line=getattr(node, "end_lineno", 0) or getattr(node, "lineno", 0),
+        start_col=getattr(node, "col_offset", 0),
+        end_col=getattr(node, "end_col_offset", 0),
+    )
+
+
+def _source_lines(source: str, span: Span) -> str:
+    lines = source.splitlines()
+    start = max(0, span.start_line - 1)
+    end = min(len(lines), span.end_line)
+    return "\n".join(lines[start:end])
+
+
+class _Collector(ast.NodeVisitor):
+    """Walk AST and collect semantic nodes with stable ordinal counters."""
+
+    def __init__(self, module: str, source: str) -> None:
+        self.module = module
+        self.source = source
+        self.nodes: list[SemanticNode] = []
+        self._symbol_stack: list[str] = []
+        self._class_depth: int = 0
+        self._counters: dict[str, int] = {}
+
+    def _next_ordinal(self, kind: NodeKind) -> int:
+        ctx = "::".join([self.module] + self._symbol_stack)
+        key = f"{ctx}::{kind.value}"
+        n = self._counters.get(key, 0)
+        self._counters[key] = n + 1
+        return n
+
+    def _emit(self, kind: NodeKind, node: ast.AST) -> None:
+        sp = _span(node)
+        ordinal = self._next_ordinal(kind)
+        frag = _source_lines(self.source, sp)
+        self.nodes.append(
+            SemanticNode.build(
+                module=self.module,
+                symbol_context=list(self._symbol_stack),
+                kind=kind,
+                ordinal=ordinal,
+                span=sp,
+                source_fragment=frag,
+            )
+        )
+
+    # -- functions / methods --------------------------------------------------
+
+    def _visit_func(self, node: ast.FunctionDef | ast.AsyncFunctionDef) -> None:
+        kind = NodeKind.METHOD if self._class_depth > 0 else NodeKind.FUNCTION
+        self._symbol_stack.append(node.name)
+        self._emit(kind, node)
+        self.generic_visit(node)
+        self._symbol_stack.pop()
+
+    def visit_FunctionDef(self, node: ast.FunctionDef) -> None:
+        self._visit_func(node)
+
+    def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> None:
+        self._visit_func(node)
+
+    # -- classes --------------------------------------------------------------
+
+    def visit_ClassDef(self, node: ast.ClassDef) -> None:
+        self._symbol_stack.append(node.name)
+        self._emit(NodeKind.CLASS, node)
+        self._class_depth += 1
+        self.generic_visit(node)
+        self._class_depth -= 1
+        self._symbol_stack.pop()
+
+    # -- control flow ---------------------------------------------------------
+
+    def visit_If(self, node: ast.If) -> None:
+        self._emit(NodeKind.BRANCH_IF, node)
+        if node.orelse:
+            for alt in node.orelse:
+                if isinstance(alt, ast.If):
+                    self._emit(NodeKind.BRANCH_ELIF, alt)
+                else:
+                    self._emit(NodeKind.BRANCH_ELSE, node)
+                    break
+        self.generic_visit(node)
+
+    def visit_Match(self, node: ast.Match) -> None:
+        for case in node.cases:
+            self._emit(NodeKind.MATCH_CASE, case)
+        self.generic_visit(node)
+
+    def visit_For(self, node: ast.For) -> None:
+        self._emit(NodeKind.FOR_LOOP, node)
+        self.generic_visit(node)
+
+    def visit_AsyncFor(self, node: ast.AsyncFor) -> None:
+        self._emit(NodeKind.FOR_LOOP, node)
+        self.generic_visit(node)
+
+    def visit_While(self, node: ast.While) -> None:
+        self._emit(NodeKind.WHILE_LOOP, node)
+        self.generic_visit(node)
+
+    def visit_With(self, node: ast.With) -> None:
+        self._emit(NodeKind.WITH_BLOCK, node)
+        self.generic_visit(node)
+
+    def visit_AsyncWith(self, node: ast.AsyncWith) -> None:
+        self._emit(NodeKind.WITH_BLOCK, node)
+        self.generic_visit(node)
+
+    # -- exception handling ---------------------------------------------------
+
+    def visit_ExceptHandler(self, node: ast.ExceptHandler) -> None:
+        self._emit(NodeKind.EXCEPT_HANDLER, node)
+        self.generic_visit(node)
+
+    # -- returns / raises -----------------------------------------------------
+
+    def visit_Return(self, node: ast.Return) -> None:
+        self._emit(NodeKind.RETURN, node)
+
+    def visit_Raise(self, node: ast.Raise) -> None:
+        self._emit(NodeKind.RAISE, node)
+
+    # -- assert ---------------------------------------------------------------
+
+    def visit_Assert(self, node: ast.Assert) -> None:
+        self._emit(NodeKind.ASSERT, node)
+
+
+def parse_module(path: Path, module: str | None = None) -> list[SemanticNode]:
+    """Parse a single Python file and return its semantic nodes.
+
+    Parameters
+    ----------
+    path
+        Path to the ``.py`` file.
+    module
+        Dotted module name.  If *None*, derived from the file stem.
+    """
+    source = path.read_text(encoding="utf-8")
+    tree = ast.parse(source, filename=str(path))
+    mod_name = module or path.stem
+    return parse_tree(tree, mod_name, source)
+
+
+def parse_tree(tree: ast.Module, module: str, source: str = "") -> list[SemanticNode]:
+    """Parse an already-built AST and return semantic nodes."""
+    collector = _Collector(module, source)
+    collector.visit(tree)
+    return collector.nodes

src/sem/plan.py

@@ -0,0 +1,108 @@
+"""Layer 4 — Planner: prioritise uncovered nodes and suggest test targets.
+
+Implements the priority heuristic from the design doc::
+
+    priority =
+        uncovered_branch * 5
+      + ty_related * 4
+      + ruff_warning * 2
+      + inside_partially_tested_function * 3
+      - prior_failures * 2
+      - likely_unreachable * 5
+"""
+
+from __future__ import annotations
+
+from sem.nodes import NodeKind, SemanticNode
+
+# Branch-like node kinds that carry higher weight when uncovered.
+_BRANCH_KINDS = {
+    NodeKind.BRANCH_IF,
+    NodeKind.BRANCH_ELSE,
+    NodeKind.BRANCH_ELIF,
+    NodeKind.MATCH_CASE,
+}
+
+_EXCEPTION_KINDS = {
+    NodeKind.EXCEPT_HANDLER,
+}
+
+
+def _is_branch(node: SemanticNode) -> bool:
+    return node.kind in _BRANCH_KINDS
+
+
+def _is_exception_path(node: SemanticNode) -> bool:
+    return node.kind in _EXCEPTION_KINDS
+
+
+def _function_coverage_ratio(
+    node: SemanticNode, all_nodes: list[SemanticNode]
+) -> float:
+    """Return fraction of nodes hit in the same function context."""
+    if not node.symbol_context:
+        return 0.0
+    fn_ctx = node.symbol_context[0] if node.symbol_context else ""
+    siblings = [
+        n
+        for n in all_nodes
+        if n.module == node.module
+        and n.symbol_context
+        and n.symbol_context[0] == fn_ctx
+    ]
+    if not siblings:
+        return 0.0
+    hit = sum(1 for s in siblings if s.runtime.hit)
+    return hit / len(siblings)
+
+
+def score_node(node: SemanticNode, all_nodes: list[SemanticNode]) -> float:
+    """Compute priority score for a single node."""
+    score = 0.0
+
+    # Uncovered branch
+    if not node.runtime.hit and _is_branch(node):
+        score += 5.0
+
+    # ty diagnostics present
+    if node.ty_codes:
+        score += 4.0
+
+    # Ruff warnings present
+    if node.ruff_codes:
+        score += 2.0
+
+    # Inside a partially tested function (some siblings hit, this one not)
+    fn_ratio = _function_coverage_ratio(node, all_nodes)
+    if 0.0 < fn_ratio < 1.0 and not node.runtime.hit:
+        score += 3.0
+
+    # Exception paths — valuable targets
+    if not node.runtime.hit and _is_exception_path(node):
+        score += 3.0
+
+    # Penalise prior failures
+    score -= node.attempts * 2.0
+
+    # Penalise already-covered nodes
+    if node.runtime.hit:
+        score -= 5.0
+
+    return score
+
+
+def rank_nodes(nodes: list[SemanticNode]) -> list[SemanticNode]:
+    """Score and sort nodes by descending priority.
+
+    Mutates ``node.priority`` in place, then returns nodes sorted
+    highest-priority-first.
+    """
+    for node in nodes:
+        node.priority = score_node(node, nodes)
+    return sorted(nodes, key=lambda n: n.priority, reverse=True)
+
+
+def next_targets(nodes: list[SemanticNode], *, limit: int = 10) -> list[SemanticNode]:
+    """Return the top-*limit* uncovered nodes worth testing next."""
+    ranked = rank_nodes(nodes)
+    return [n for n in ranked if not n.runtime.hit and n.priority > 0][:limit]

src/sem/runtime.py

@@ -0,0 +1,74 @@
+"""Layer 3 — Runtime: attach coverage data to semantic nodes.
+
+Reads a ``coverage.py`` JSON report and maps executed/missing lines
+back to semantic nodes, populating ``RuntimeInfo``.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+from sem.nodes import SemanticNode
+
+
+def _load_coverage_json(path: Path) -> dict:
+    """Load a coverage.py JSON report file."""
+    return json.loads(path.read_text(encoding="utf-8"))
+
+
+def attach_coverage(
+    nodes: list[SemanticNode],
+    coverage_json: Path,
+    source_file: str,
+) -> list[SemanticNode]:
+    """Map coverage data onto semantic nodes.
+
+    Parameters
+    ----------
+    nodes
+        Semantic nodes for a single module (from ``parse_module``).
+    coverage_json
+        Path to the ``coverage.json`` report (``coverage json -o file``).
+    source_file
+        The source file key as it appears in the coverage report
+        (e.g. ``src/aco/express/pharmacy.py``).
+
+    Returns
+    -------
+    list[SemanticNode]
+        Same list, mutated in place.
+    """
+    report = _load_coverage_json(coverage_json)
+    files = report.get("files", {})
+    file_data = files.get(source_file)
+    if file_data is None:
+        return nodes
+
+    executed_lines: set[int] = set(file_data.get("executed_lines", []))
+    missing_lines: set[int] = set(file_data.get("missing_lines", []))
+
+    for node in nodes:
+        node_lines = set(range(node.span.start_line, node.span.end_line + 1))
+        hit_lines = node_lines & executed_lines
+        miss_lines = node_lines & missing_lines
+
+        if hit_lines:
+            node.runtime.hit = True
+            node.runtime.hit_count = len(hit_lines)
+
+        if miss_lines and not hit_lines:
+            node.runtime.hit = False
+
+    # Attach per-test contexts if available (coverage.py contexts)
+    contexts = file_data.get("contexts", {})
+    if contexts:
+        for line_str, test_list in contexts.items():
+            line = int(line_str)
+            for node in nodes:
+                if node.span.start_line <= line <= node.span.end_line:
+                    for t in test_list:
+                        if t and t not in node.runtime.tests_seen:
+                            node.runtime.tests_seen.append(t)
+
+    return nodes

src/sem/state.py

@@ -0,0 +1,95 @@
+"""Rolling state persistence for semantic coverage.
+
+Stores node-level tracking state (status, attempts, notes) in a JSON
+file that survives across runs.  Node IDs + source hashes make the
+state drift-resistant — edits that change a node's content invalidate
+its history rather than silently reusing stale data.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+from sem.nodes import SemanticNode
+
+
+def _default_entry() -> dict:
+    return {
+        "status": "unexplored",
+        "attempts": 0,
+        "last_result": None,
+        "source_hash": "",
+        "notes": [],
+    }
+
+
+def load_state(path: Path) -> dict[str, dict]:
+    """Load persisted state from a JSON file."""
+    if not path.exists():
+        return {}
+    return json.loads(path.read_text(encoding="utf-8"))
+
+
+def save_state(state: dict[str, dict], path: Path) -> None:
+    """Persist state to a JSON file."""
+    path.parent.mkdir(parents=True, exist_ok=True)
+    path.write_text(json.dumps(state, indent=2) + "\n", encoding="utf-8")
+
+
+def sync_state(nodes: list[SemanticNode], state: dict[str, dict]) -> dict[str, dict]:
+    """Merge current nodes into persisted state.
+
+    - New nodes get a fresh entry.
+    - Nodes whose ``source_hash`` changed are reset (code was edited).
+    - Existing nodes with matching hashes keep their history.
+    - Stale entries (nodes no longer in the AST) are removed.
+    """
+    updated: dict[str, dict] = {}
+
+    for node in nodes:
+        existing = state.get(node.node_id)
+        if existing is None:
+            updated[node.node_id] = _default_entry() | {
+                "source_hash": node.source_hash,
+            }
+        elif existing.get("source_hash") != node.source_hash:
+            # Source changed — reset but keep notes
+            updated[node.node_id] = _default_entry() | {
+                "source_hash": node.source_hash,
+                "notes": existing.get("notes", []),
+            }
+        else:
+            updated[node.node_id] = existing
+
+    return updated
+
+
+def apply_state(nodes: list[SemanticNode], state: dict[str, dict]) -> None:
+    """Push persisted state back onto node objects."""
+    for node in nodes:
+        entry = state.get(node.node_id)
+        if entry:
+            node.status = entry.get("status", "unexplored")
+            node.attempts = entry.get("attempts", 0)
+
+
+def record_attempt(
+    state: dict[str, dict],
+    node_id: str,
+    result: str,
+    note: str | None = None,
+) -> None:
+    """Record a test attempt against a node."""
+    entry = state.get(node_id)
+    if entry is None:
+        entry = _default_entry()
+        state[node_id] = entry
+    entry["attempts"] = entry.get("attempts", 0) + 1
+    entry["last_result"] = result
+    if result == "covered":
+        entry["status"] = "covered"
+    elif result == "partial":
+        entry["status"] = "partial"
+    if note:
+        entry.setdefault("notes", []).append(note)

stack.toml

@@ -151,6 +151,33 @@ catalog = "aco_staging"
 mode = "production"
 catalog = "aco"
 
+[databricks.warehouse]
+name = "stack-sql-warehouse"
+size = "SMALL"
+auto_stop_mins = 10
+
+[databricks.secrets]
+scope = "stack"
+mapping = [
+    { env = "BCDA_CLIENT_ID", key = "bcda-client-id" },
+    { env = "BCDA_CLIENT_SECRET", key = "bcda-client-secret" },
+]
+
+[databricks.governance]
+phi_schemas = ["core", "claims_preprocessing", "cclf", "input_layer"]
+
+[databricks.governance.groups.analysts]
+schemas = ["readmissions", "quality_measures", "cms_quality_measures", "ahrq_measures"]
+privileges = ["SELECT"]
+
+[databricks.governance.groups.pipeline_svc]
+schemas = ["*"]
+privileges = ["ALL_PRIVILEGES"]
+
+[databricks.quality]
+monitored_schemas = ["core", "claims_preprocessing", "data_quality"]
+output_schema = "_monitoring"
+
 [default]
 target = "local" # duckdb | databricks | trino | lake
 

tests/aco/test_lake_governance.py

@@ -0,0 +1,167 @@
+"""Tests for aco.lake.governance — declarative grant management."""
+
+from __future__ import annotations
+
+from unittest.mock import MagicMock
+
+from aco.lake.governance import (
+    GovernancePolicy,
+    GrantRule,
+    _resolve_schemas,
+    apply_governance,
+    audit_governance,
+)
+
+
+class TestResolveSchemas:
+    def test_wildcard_expands(self):
+        rule = GrantRule(group="g", schemas=["*"], privileges=["SELECT"])
+        assert _resolve_schemas(rule, ["core", "claims"]) == ["core", "claims"]
+
+    def test_explicit_schemas(self):
+        rule = GrantRule(group="g", schemas=["core"], privileges=["SELECT"])
+        assert _resolve_schemas(rule, ["core", "claims"]) == ["core"]
+
+    def test_missing_schema_filtered(self):
+        rule = GrantRule(group="g", schemas=["missing"], privileges=["SELECT"])
+        assert _resolve_schemas(rule, ["core"]) == []
+
+
+class TestGovernancePolicyModel:
+    def test_basic_construction(self):
+        policy = GovernancePolicy(
+            catalog="aco",
+            rules=[
+                GrantRule(
+                    group="analysts",
+                    schemas=["readmissions"],
+                    privileges=["SELECT"],
+                ),
+            ],
+            phi_schemas=["core"],
+        )
+        assert policy.catalog == "aco"
+        assert len(policy.rules) == 1
+        assert policy.phi_schemas == ["core"]
+
+    def test_empty_rules(self):
+        policy = GovernancePolicy(catalog="aco")
+        assert policy.rules == []
+
+
+class TestApplyGovernance:
+    def test_dry_run_reports_actions(self):
+        client = MagicMock()
+        # Mock schemas list
+        schema_mock = MagicMock()
+        schema_mock.name = "readmissions"
+        client._ws.schemas.list.return_value = [schema_mock]
+
+        policy = GovernancePolicy(
+            catalog="aco",
+            rules=[
+                GrantRule(
+                    group="analysts",
+                    schemas=["readmissions"],
+                    privileges=["SELECT"],
+                ),
+            ],
+        )
+
+        actions = apply_governance(client, policy, dry_run=True)
+        assert len(actions) == 1
+        assert "[dry-run]" in actions[0]
+        assert "GRANT SELECT" in actions[0]
+        assert "analysts" in actions[0]
+        # Should NOT call ws.grants.update in dry_run
+        client._ws.grants.update.assert_not_called()
+
+    def test_apply_calls_sdk(self):
+        client = MagicMock()
+        schema_mock = MagicMock()
+        schema_mock.name = "core"
+        client._ws.schemas.list.return_value = [schema_mock]
+
+        policy = GovernancePolicy(
+            catalog="aco",
+            rules=[
+                GrantRule(
+                    group="pipeline_svc",
+                    schemas=["core"],
+                    privileges=["ALL_PRIVILEGES"],
+                ),
+            ],
+        )
+
+        actions = apply_governance(client, policy, dry_run=False)
+        assert len(actions) == 1
+        assert "GRANT" in actions[0]
+        client._ws.grants.update.assert_called_once()
+
+    def test_wildcard_expands_to_all(self):
+        client = MagicMock()
+        s1, s2 = MagicMock(), MagicMock()
+        s1.name, s2.name = "core", "claims"
+        client._ws.schemas.list.return_value = [s1, s2]
+
+        policy = GovernancePolicy(
+            catalog="aco",
+            rules=[
+                GrantRule(group="g", schemas=["*"], privileges=["SELECT"]),
+            ],
+        )
+
+        actions = apply_governance(client, policy, dry_run=True)
+        assert len(actions) == 2
+
+
+class TestAuditGovernance:
+    def test_detects_missing_grants(self):
+        client = MagicMock()
+        schema_mock = MagicMock()
+        schema_mock.name = "core"
+        client._ws.schemas.list.return_value = [schema_mock]
+
+        # No grants exist
+        resp = MagicMock()
+        resp.privilege_assignments = []
+        client._ws.grants.get.return_value = resp
+
+        policy = GovernancePolicy(
+            catalog="aco",
+            rules=[
+                GrantRule(group="analysts", schemas=["core"], privileges=["SELECT"]),
+            ],
+        )
+
+        drifts = audit_governance(client, policy)
+        assert len(drifts) == 1
+        assert drifts[0].principal == "analysts"
+        assert drifts[0].action == "grant"
+
+    def test_no_drift_when_matching(self):
+        client = MagicMock()
+        schema_mock = MagicMock()
+        schema_mock.name = "core"
+        client._ws.schemas.list.return_value = [schema_mock]
+
+        # Grants match exactly
+        priv = MagicMock()
+        priv.privilege = MagicMock()
+        priv.privilege.value = "SELECT"
+        assignment = MagicMock()
+        assignment.principal = "analysts"
+        assignment.privileges = [priv]
+        resp = MagicMock()
+        resp.privilege_assignments = [assignment]
+        client._ws.grants.get.return_value = resp
+
+        policy = GovernancePolicy(
+            catalog="aco",
+            rules=[
+                GrantRule(group="analysts", schemas=["core"], privileges=["SELECT"]),
+            ],
+        )
+
+        drifts = audit_governance(client, policy)
+        assert len(drifts) == 0

tests/aco/test_lake_jobs.py

@@ -0,0 +1,88 @@
+"""Tests for aco.lake.jobs — programmatic job management."""
+
+from __future__ import annotations
+
+from unittest.mock import MagicMock
+
+from aco.lake.jobs import JobManager, RunStatus
+
+
+class TestRunStatus:
+    def test_terminal_states(self):
+        assert RunStatus(run_id=1, state="TERMINATED").is_terminal
+        assert RunStatus(run_id=1, state="SKIPPED").is_terminal
+        assert not RunStatus(run_id=1, state="RUNNING").is_terminal
+
+    def test_success(self):
+        assert RunStatus(run_id=1, state="TERMINATED", result_state="SUCCESS").succeeded
+        assert not RunStatus(
+            run_id=1, state="TERMINATED", result_state="FAILED"
+        ).succeeded
+
+
+class TestJobManager:
+    def _make_manager(self):
+        client = MagicMock()
+        return JobManager(client, catalog="aco_dev"), client
+
+    def test_create_pipeline_job(self):
+        mgr, client = self._make_manager()
+        client._ws.jobs.create.return_value = MagicMock(job_id=42)
+
+        job_id = mgr.create_pipeline_job("readmissions")
+        assert job_id == 42
+        client._ws.jobs.create.assert_called_once()
+
+        # Verify task structure
+        call_kwargs = client._ws.jobs.create.call_args.kwargs
+        assert call_kwargs["name"] == "stack_readmissions"
+        assert len(call_kwargs["tasks"]) >= 1
+        task = call_kwargs["tasks"][0]
+        assert task.task_key == "readmissions"
+        assert task.python_wheel_task.package_name == "stack"
+
+    def test_create_with_schedule(self):
+        mgr, client = self._make_manager()
+        client._ws.jobs.create.return_value = MagicMock(job_id=99)
+
+        mgr.create_pipeline_job("core", schedule="0 0 6 * * ?")
+        call_kwargs = client._ws.jobs.create.call_args.kwargs
+        assert call_kwargs["schedule"] is not None
+
+    def test_run_now(self):
+        mgr, client = self._make_manager()
+        client._ws.jobs.run_now.return_value = MagicMock(run_id=123)
+
+        run_id = mgr.run_now(42)
+        assert run_id == 123
+        client._ws.jobs.run_now.assert_called_once_with(job_id=42)
+
+    def test_get_run_status(self):
+        mgr, client = self._make_manager()
+        state = MagicMock()
+        state.life_cycle_state = "RUNNING"
+        state.result_state = None
+        state.state_message = "In progress"
+        run = MagicMock()
+        run.state = state
+        client._ws.jobs.get_run.return_value = run
+
+        status = mgr.get_run_status(123)
+        assert status.state == "RUNNING"
+        assert not status.is_terminal
+
+    def test_list_jobs(self):
+        mgr, client = self._make_manager()
+        job = MagicMock()
+        job.job_id = 1
+        job.settings.name = "stack_core"
+        client._ws.jobs.list.return_value = [job]
+
+        jobs = mgr.list_jobs()
+        assert len(jobs) == 1
+        assert jobs[0]["name"] == "stack_core"
+
+    def test_delete_job(self):
+        mgr, client = self._make_manager()
+        mgr.delete_job(42)
+        client._ws.jobs.delete.assert_called_once_with(job_id=42)

tests/aco/test_lake_quality.py

@@ -0,0 +1,57 @@
+"""Tests for aco.lake.quality — Lakehouse Monitoring setup."""
+
+from __future__ import annotations
+
+from unittest.mock import MagicMock
+
+from aco.lake.quality import MonitorProfile, MonitorResult, setup_monitors
+
+
+class TestMonitorModels:
+    def test_profile_defaults(self):
+        p = MonitorProfile(table="aco.core")
+        assert p.output_schema == "_monitoring"
+        assert p.slicing_exprs == []
+
+    def test_result_model(self):
+        r = MonitorResult(table="aco.core.encounter", status="active")
+        assert r.table == "aco.core.encounter"
+        assert r.dashboard_id is None
+
+
+class TestSetupMonitors:
+    def test_dry_run(self):
+        client = MagicMock()
+        table = MagicMock()
+        table.full_name = "aco.core.encounter"
+        table.name = "encounter"
+        client._ws.tables.list.return_value = [table]
+
+        actions = setup_monitors(client, catalog="aco", schemas=["core"], dry_run=True)
+        assert len(actions) == 1
+        assert "[dry-run]" in actions[0]
+        assert "aco.core.encounter" in actions[0]
+        # Should not call create in dry_run
+        client._ws.quality_monitors.create.assert_not_called()
+
+    def test_creates_output_schema(self):
+        client = MagicMock()
+        client._ws.schemas.get.side_effect = Exception("not found")
+        client._ws.tables.list.return_value = []
+
+        setup_monitors(client, catalog="aco", schemas=["core"], dry_run=False)
+        client._ws.schemas.create.assert_called_once()
+
+    def test_skips_existing_monitor(self):
+        client = MagicMock()
+        client._ws.schemas.get.return_value = MagicMock()
+        table = MagicMock()
+        table.full_name = "aco.core.encounter"
+        table.name = "encounter"
+        client._ws.tables.list.return_value = [table]
+        # Monitor already exists
+        client._ws.quality_monitors.get.return_value = MagicMock()
+
+        actions = setup_monitors(client, catalog="aco", schemas=["core"], dry_run=False)
+        assert any("[exists]" in a for a in actions)
+        client._ws.quality_monitors.create.assert_not_called()

tests/conftest.py

@@ -526,6 +526,7 @@ def rvu_df() -> pl.DataFrame:
             "fac_pe_rvu": [0.41, 0.63, 0.95],
             "mp_rvu": [0.07, 0.11, 0.16],
             "conv_factor": [32.7442, 32.7442, 32.7442],
+            "mac": ["10212", "10212", "10212"],
             "locality": ["0201", "0201", "0201"],
         }
     )
@@ -536,6 +537,7 @@ def gpci_df() -> pl.DataFrame:
     """GPCI data for PFS payment calculation tests."""
     return pl.DataFrame(
         {
+            "mac": ["10212"],
             "locality": ["0201"],
             "work_gpci": [1.0],
             "pe_gpci": [0.998],

tests/pfs/test_payment.py

@@ -133,11 +133,13 @@ class TestPayment:
                 "fac_pe_rvu": [0.41],
                 "mp_rvu": [0.07],
                 "conv_factor": [32.7442],
+                "mac": ["10212"],
                 "locality": ["9999"],  # no match in gpci
             }
         )
         gpci = pl.DataFrame(
             {
+                "mac": ["10212"],
                 "locality": ["0201"],
                 "work_gpci": [1.0],
                 "pe_gpci": [0.998],
@@ -158,11 +160,13 @@ class TestPayment:
                 "fac_pe_rvu": [0.41, 0.41],
                 "mp_rvu": [0.07, 0.07],
                 "conv_factor": [32.7442, 32.7442],
+                "mac": ["10212", "10212"],
                 "locality": ["0201", "0101"],
             }
         )
         gpci = pl.DataFrame(
             {
+                "mac": ["10212", "10212"],
                 "locality": ["0201", "0101"],
                 "work_gpci": [1.0, 1.05],
                 "pe_gpci": [0.998, 1.10],
@@ -188,12 +192,14 @@ class TestPayment:
                 "fac_pe_rvu": [0.41],
                 "mp_rvu": [0.07],
                 "conv_factor": [32.7442],
+                "mac": ["10212"],
                 "locality": ["0201"],
             }
         )
         rvu_double = rvu_base.with_columns(pl.col("conv_factor") * 2)
         gpci = pl.DataFrame(
             {
+                "mac": ["10212"],
                 "locality": ["0201"],
                 "work_gpci": [1.0],
                 "pe_gpci": [0.998],
@@ -215,11 +221,13 @@ class TestPayment:
                 "fac_pe_rvu": [0.0],
                 "mp_rvu": [0.0],
                 "conv_factor": [32.7442],
+                "mac": ["10212"],
                 "locality": ["0201"],
             }
         )
         gpci = pl.DataFrame(
             {
+                "mac": ["10212"],
                 "locality": ["0201"],
                 "work_gpci": [1.0],
                 "pe_gpci": [0.998],
@@ -238,11 +246,13 @@ class TestPayment:
                 "fac_pe_rvu": pl.Series([], dtype=pl.Float64),
                 "mp_rvu": pl.Series([], dtype=pl.Float64),
                 "conv_factor": pl.Series([], dtype=pl.Float64),
+                "mac": pl.Series([], dtype=pl.Utf8),
                 "locality": pl.Series([], dtype=pl.Utf8),
             }
         )
         gpci = pl.DataFrame(
             {
+                "mac": ["10212"],
                 "locality": ["0201"],
                 "work_gpci": [1.0],
                 "pe_gpci": [0.998],

tests/sem/test_enrich.py

@@ -0,0 +1,38 @@
+"""Tests for sem.enrich — Ruff and ty diagnostic attachment."""
+
+from __future__ import annotations
+
+from sem.enrich import _find_node
+from sem.nodes import NodeKind, SemanticNode, Span
+
+
+def _make_node(start: int, end: int, node_id: str = "n") -> SemanticNode:
+    return SemanticNode(
+        node_id=node_id,
+        kind=NodeKind.FUNCTION,
+        span=Span(start_line=start, end_line=end),
+        module="test",
+    )
+
+
+class TestFindNode:
+    def test_exact_match(self):
+        nodes = [_make_node(1, 5, "a"), _make_node(10, 20, "b")]
+        assert _find_node(nodes, 3) is nodes[0]
+        assert _find_node(nodes, 15) is nodes[1]
+
+    def test_tightest_span(self):
+        outer = _make_node(1, 20, "outer")
+        inner = _make_node(5, 10, "inner")
+        assert _find_node([outer, inner], 7) is inner
+
+    def test_no_match(self):
+        nodes = [_make_node(1, 5)]
+        assert _find_node(nodes, 100) is None
+
+    def test_boundary_lines(self):
+        node = _make_node(10, 20, "n")
+        assert _find_node([node], 10) is node
+        assert _find_node([node], 20) is node
+        assert _find_node([node], 9) is None
+        assert _find_node([node], 21) is None

tests/sem/test_hooks.py

@@ -0,0 +1,137 @@
+"""Tests for sem.hooks — pre-commit smart test selection."""
+
+from __future__ import annotations
+
+from sem.hooks import (
+    _changed_modules,
+    _changed_test_dirs,
+    _classify,
+    _top_level_tests,
+    check_syntax,
+    compute_test_targets,
+)
+
+
+class TestClassify:
+    def test_src_files(self):
+        cats = _classify(["src/aco/express/foo.py", "src/sem/parse.py"])
+        assert cats["src"] == ["src/aco/express/foo.py", "src/sem/parse.py"]
+        assert cats["tests"] == []
+
+    def test_test_files(self):
+        cats = _classify(["tests/aco/test_foo.py"])
+        assert cats["tests"] == ["tests/aco/test_foo.py"]
+
+    def test_notebooks(self):
+        cats = _classify(["notebooks/pfs_calcs.py"])
+        assert cats["notebooks"] == ["notebooks/pfs_calcs.py"]
+
+    def test_infra_conftest(self):
+        cats = _classify(["tests/conftest.py"])
+        assert "tests/conftest.py" in cats["infra"]
+
+    def test_infra_pyproject(self):
+        cats = _classify(["pyproject.toml"])
+        assert "pyproject.toml" in cats["infra"]
+
+    def test_config_stack_toml(self):
+        cats = _classify(["stack.toml"])
+        assert cats["config"] == ["stack.toml"]
+
+    def test_non_python_ignored(self):
+        cats = _classify(["README.md", "compose.yml", "src/aco/data.json"])
+        assert all(not v for v in cats.values())
+
+
+class TestChangedModules:
+    def test_extracts_top_level(self):
+        assert _changed_modules(["src/aco/express/foo.py"]) == ["aco"]
+        assert _changed_modules(["src/sem/parse.py"]) == ["sem"]
+
+    def test_deduplicates(self):
+        result = _changed_modules(
+            [
+                "src/aco/express/foo.py",
+                "src/aco/pipe/bar.py",
+            ]
+        )
+        assert result == ["aco"]
+
+    def test_multiple_modules(self):
+        result = _changed_modules(
+            [
+                "src/aco/foo.py",
+                "src/sem/bar.py",
+                "src/bib/baz.py",
+            ]
+        )
+        assert result == ["aco", "bib", "sem"]
+
+
+class TestChangedTestDirs:
+    def test_extracts_dirs(self):
+        result = _changed_test_dirs(["tests/aco/test_foo.py"])
+        assert result == ["tests/aco"]
+
+    def test_deduplicates(self):
+        result = _changed_test_dirs(
+            [
+                "tests/aco/test_a.py",
+                "tests/aco/test_b.py",
+            ]
+        )
+        assert result == ["tests/aco"]
+
+
+class TestTopLevelTests:
+    def test_finds_top_level(self):
+        result = _top_level_tests(
+            [
+                "tests/test_ast_coverage.py",
+                "tests/aco/test_foo.py",
+            ]
+        )
+        assert result == ["tests/test_ast_coverage.py"]
+
+
+class TestCheckSyntax:
+    def test_valid_file(self, tmp_path):
+        f = tmp_path / "good.py"
+        f.write_text("x = 1\n")
+        assert check_syntax([str(f)]) == []
+
+    def test_invalid_file(self, tmp_path):
+        f = tmp_path / "bad.py"
+        f.write_text("def f(\n")
+        errors = check_syntax([str(f)])
+        assert len(errors) == 1
+        assert "bad.py" in errors[0]
+
+    def test_missing_file_skipped(self):
+        assert check_syntax(["nonexistent_file_xyz.py"]) == []
+
+
+class TestComputeTargets:
+    def test_full_on_infra_change(self):
+        cats = _classify(["pyproject.toml", "src/sem/parse.py"])
+        targets, reason = compute_test_targets(cats)
+        assert targets == ["tests/"]
+        assert "infrastructure" in reason
+
+    def test_full_on_force(self):
+        cats = _classify(["src/sem/parse.py"])
+        targets, reason = compute_test_targets(cats, force_full=True)
+        assert targets == ["tests/"]
+
+    def test_targeted_on_src_change(self):
+        cats = _classify(["src/sem/parse.py"])
+        targets, reason = compute_test_targets(cats)
+        # Should include tests/sem/ and test_ast_coverage.py
+        assert any("tests/sem/" in t for t in targets)
+        assert any("test_ast_coverage" in t for t in targets)
+
+    def test_empty_on_no_python(self):
+        cats = _classify(["README.md"])
+        targets, reason = compute_test_targets(cats)
+        assert targets == []
+        assert "no testable" in reason

tests/sem/test_parse.py

@@ -0,0 +1,170 @@
+"""Tests for sem.parse — AST to semantic nodes."""
+
+from __future__ import annotations
+
+import ast
+import textwrap
+
+from sem.nodes import NodeKind
+from sem.parse import parse_tree
+
+
+def _parse(source: str, module: str = "test_mod") -> list:
+    source = textwrap.dedent(source)
+    tree = ast.parse(source)
+    return parse_tree(tree, module, source)
+
+
+class TestFunctionNodes:
+    def test_top_level_function(self):
+        nodes = _parse("""
+            def greet(name):
+                return f"hi {name}"
+        """)
+        funcs = [n for n in nodes if n.kind == NodeKind.FUNCTION]
+        assert len(funcs) == 1
+        assert "greet" in funcs[0].node_id
+        assert funcs[0].module == "test_mod"
+
+    def test_method_inside_class(self):
+        nodes = _parse("""
+            class Foo:
+                def bar(self):
+                    pass
+        """)
+        classes = [n for n in nodes if n.kind == NodeKind.CLASS]
+        assert len(classes) == 1
+        # The method is nested inside the class context
+        methods = [n for n in nodes if n.kind in (NodeKind.FUNCTION, NodeKind.METHOD)]
+        assert len(methods) == 1
+        assert "Foo" in methods[0].node_id
+        assert "bar" in methods[0].node_id
+
+    def test_async_function(self):
+        nodes = _parse("""
+            async def fetch():
+                return 42
+        """)
+        funcs = [n for n in nodes if n.kind == NodeKind.FUNCTION]
+        assert len(funcs) == 1
+        assert "fetch" in funcs[0].node_id
+
+
+class TestBranchNodes:
+    def test_if_else(self):
+        nodes = _parse("""
+            def check(x):
+                if x > 0:
+                    return "pos"
+                else:
+                    return "neg"
+        """)
+        ifs = [n for n in nodes if n.kind == NodeKind.BRANCH_IF]
+        elses = [n for n in nodes if n.kind == NodeKind.BRANCH_ELSE]
+        assert len(ifs) == 1
+        assert len(elses) == 1
+
+    def test_elif(self):
+        nodes = _parse("""
+            def classify(x):
+                if x > 0:
+                    return "pos"
+                elif x == 0:
+                    return "zero"
+                else:
+                    return "neg"
+        """)
+        elifs = [n for n in nodes if n.kind == NodeKind.BRANCH_ELIF]
+        assert len(elifs) == 1
+
+    def test_match_case(self):
+        nodes = _parse("""
+            def route(cmd):
+                match cmd:
+                    case "start":
+                        pass
+                    case "stop":
+                        pass
+                    case _:
+                        pass
+        """)
+        cases = [n for n in nodes if n.kind == NodeKind.MATCH_CASE]
+        assert len(cases) == 3
+
+
+class TestExceptionNodes:
+    def test_except_handler(self):
+        nodes = _parse("""
+            def risky():
+                try:
+                    open("f")
+                except FileNotFoundError:
+                    pass
+                except ValueError:
+                    pass
+        """)
+        handlers = [n for n in nodes if n.kind == NodeKind.EXCEPT_HANDLER]
+        assert len(handlers) == 2
+
+
+class TestControlFlow:
+    def test_for_loop(self):
+        nodes = _parse("""
+            for i in range(10):
+                print(i)
+        """)
+        loops = [n for n in nodes if n.kind == NodeKind.FOR_LOOP]
+        assert len(loops) == 1
+
+    def test_while_loop(self):
+        nodes = _parse("""
+            while True:
+                break
+        """)
+        loops = [n for n in nodes if n.kind == NodeKind.WHILE_LOOP]
+        assert len(loops) == 1
+
+    def test_return_and_raise(self):
+        nodes = _parse("""
+            def f(x):
+                if x:
+                    return 1
+                raise ValueError("bad")
+        """)
+        returns = [n for n in nodes if n.kind == NodeKind.RETURN]
+        raises = [n for n in nodes if n.kind == NodeKind.RAISE]
+        assert len(returns) == 1
+        assert len(raises) == 1
+
+
+class TestNodeIdentity:
+    def test_node_id_is_stable(self):
+        """Same source should produce identical node IDs across parses."""
+        src = "def f():\n    return 1\n"
+        nodes_a = parse_tree(ast.parse(src), "mod", src)
+        nodes_b = parse_tree(ast.parse(src), "mod", src)
+        ids_a = [n.node_id for n in nodes_a]
+        ids_b = [n.node_id for n in nodes_b]
+        assert ids_a == ids_b
+
+    def test_source_hash_changes_with_content(self):
+        src_a = "def f():\n    return 1\n"
+        src_b = "def f():\n    return 2\n"
+        nodes_a = parse_tree(ast.parse(src_a), "mod", src_a)
+        nodes_b = parse_tree(ast.parse(src_b), "mod", src_b)
+        funcs_a = [n for n in nodes_a if n.kind == NodeKind.FUNCTION]
+        funcs_b = [n for n in nodes_b if n.kind == NodeKind.FUNCTION]
+        assert funcs_a[0].source_hash != funcs_b[0].source_hash
+
+    def test_node_id_no_line_numbers(self):
+        """Node IDs must not contain raw line numbers."""
+        nodes = _parse("""
+            def hello():
+                if True:
+                    return 1
+        """)
+        for node in nodes:
+            # Line numbers appear in span, not in node_id
+            assert node.span.start_line > 0
+            # node_id uses ordinal, not line number
+            assert ":L" not in node.node_id

tests/sem/test_plan.py

@@ -0,0 +1,76 @@
+"""Tests for sem.plan — prioritisation and target selection."""
+
+from __future__ import annotations
+
+from sem.nodes import NodeKind, RuntimeInfo, SemanticNode, Span
+from sem.plan import next_targets, rank_nodes, score_node
+
+
+def _node(
+    kind: NodeKind = NodeKind.BRANCH_IF,
+    hit: bool = False,
+    ruff: list[str] | None = None,
+    ty: list[str] | None = None,
+    attempts: int = 0,
+    module: str = "mod",
+    symbol_ctx: list[str] | None = None,
+) -> SemanticNode:
+    return SemanticNode(
+        node_id=f"{module}::{'::'.join(symbol_ctx or [])}::{kind.value}[0]",
+        kind=kind,
+        span=Span(start_line=1, end_line=5),
+        module=module,
+        symbol_context=symbol_ctx or [],
+        ruff_codes=ruff or [],
+        ty_codes=ty or [],
+        runtime=RuntimeInfo(hit=hit),
+        attempts=attempts,
+    )
+
+
+class TestScoring:
+    def test_uncovered_branch_scores_high(self):
+        n = _node(kind=NodeKind.BRANCH_IF, hit=False)
+        assert score_node(n, [n]) >= 5.0
+
+    def test_covered_node_penalised(self):
+        n = _node(hit=True)
+        assert score_node(n, [n]) < 0
+
+    def test_ty_codes_boost(self):
+        base = _node(kind=NodeKind.FUNCTION, hit=False)
+        with_ty = _node(kind=NodeKind.FUNCTION, hit=False, ty=["possibly-none"])
+        assert score_node(with_ty, [with_ty]) > score_node(base, [base])
+
+    def test_ruff_codes_boost(self):
+        base = _node(kind=NodeKind.FUNCTION, hit=False)
+        with_ruff = _node(kind=NodeKind.FUNCTION, hit=False, ruff=["B904"])
+        assert score_node(with_ruff, [with_ruff]) > score_node(base, [base])
+
+    def test_prior_failures_penalised(self):
+        fresh = _node(attempts=0)
+        stale = _node(attempts=3)
+        assert score_node(fresh, [fresh]) > score_node(stale, [stale])
+
+    def test_exception_handler_scores_high(self):
+        n = _node(kind=NodeKind.EXCEPT_HANDLER, hit=False)
+        assert score_node(n, [n]) >= 3.0
+
+
+class TestRanking:
+    def test_rank_order(self):
+        high = _node(kind=NodeKind.BRANCH_IF, hit=False, ty=["x"])
+        low = _node(kind=NodeKind.FUNCTION, hit=True)
+        ranked = rank_nodes([low, high])
+        assert ranked[0] is high
+
+    def test_next_targets_excludes_covered(self):
+        covered = _node(hit=True)
+        uncovered = _node(hit=False, kind=NodeKind.BRANCH_IF)
+        targets = next_targets([covered, uncovered], limit=5)
+        assert all(not t.runtime.hit for t in targets)
+
+    def test_next_targets_respects_limit(self):
+        nodes = [_node(hit=False, kind=NodeKind.BRANCH_IF) for _ in range(20)]
+        targets = next_targets(nodes, limit=3)
+        assert len(targets) <= 3

tests/sem/test_runtime.py

@@ -0,0 +1,78 @@
+"""Tests for sem.runtime — coverage data attachment."""
+
+from __future__ import annotations
+
+import json
+import tempfile
+from pathlib import Path
+
+from sem.nodes import NodeKind, SemanticNode, Span
+from sem.runtime import attach_coverage
+
+
+def _make_node(nid: str, start: int, end: int) -> SemanticNode:
+    return SemanticNode(
+        node_id=nid,
+        kind=NodeKind.BRANCH_IF,
+        span=Span(start_line=start, end_line=end),
+        module="pkg.mod",
+    )
+
+
+def _write_coverage(data: dict) -> Path:
+    p = Path(tempfile.mktemp(suffix=".json"))
+    p.write_text(json.dumps(data))
+    return p
+
+
+class TestAttachCoverage:
+    def test_marks_hit_nodes(self):
+        nodes = [_make_node("a", 1, 5), _make_node("b", 10, 15)]
+        cov = _write_coverage(
+            {
+                "files": {
+                    "src/pkg/mod.py": {
+                        "executed_lines": [1, 2, 3, 4, 5],
+                        "missing_lines": [10, 11, 12, 13, 14, 15],
+                    }
+                }
+            }
+        )
+        try:
+            attach_coverage(nodes, cov, "src/pkg/mod.py")
+            assert nodes[0].runtime.hit is True
+            assert nodes[0].runtime.hit_count == 5
+            assert nodes[1].runtime.hit is False
+        finally:
+            cov.unlink()
+
+    def test_missing_file_key_is_noop(self):
+        nodes = [_make_node("a", 1, 5)]
+        cov = _write_coverage({"files": {}})
+        try:
+            attach_coverage(nodes, cov, "nonexistent.py")
+            assert nodes[0].runtime.hit is False
+        finally:
+            cov.unlink()
+
+    def test_contexts_attached(self):
+        nodes = [_make_node("a", 1, 3)]
+        cov = _write_coverage(
+            {
+                "files": {
+                    "mod.py": {
+                        "executed_lines": [1, 2, 3],
+                        "missing_lines": [],
+                        "contexts": {
+                            "2": ["test_foo", "test_bar"],
+                        },
+                    }
+                }
+            }
+        )
+        try:
+            attach_coverage(nodes, cov, "mod.py")
+            assert "test_foo" in nodes[0].runtime.tests_seen
+            assert "test_bar" in nodes[0].runtime.tests_seen
+        finally:
+            cov.unlink()

tests/sem/test_state.py

@@ -0,0 +1,91 @@
+"""Tests for sem.state — rolling state persistence."""
+
+from __future__ import annotations
+
+from sem.nodes import NodeKind, SemanticNode, Span
+from sem.state import (
+    apply_state,
+    load_state,
+    record_attempt,
+    save_state,
+    sync_state,
+)
+
+
+def _node(nid: str, src_hash: str = "abc123") -> SemanticNode:
+    return SemanticNode(
+        node_id=nid,
+        kind=NodeKind.FUNCTION,
+        span=Span(start_line=1, end_line=5),
+        module="mod",
+        source_hash=src_hash,
+    )
+
+
+class TestPersistence:
+    def test_round_trip(self, tmp_path):
+        p = tmp_path / "state.json"
+        state = {"node_a": {"status": "covered", "attempts": 1}}
+        save_state(state, p)
+        loaded = load_state(p)
+        assert loaded["node_a"]["status"] == "covered"
+
+    def test_load_missing_file(self, tmp_path):
+        assert load_state(tmp_path / "nope.json") == {}
+
+
+class TestSync:
+    def test_new_node_gets_fresh_entry(self):
+        nodes = [_node("n1")]
+        result = sync_state(nodes, {})
+        assert "n1" in result
+        assert result["n1"]["status"] == "unexplored"
+
+    def test_unchanged_node_keeps_history(self):
+        nodes = [_node("n1", "abc")]
+        old = {"n1": {"status": "covered", "attempts": 3, "source_hash": "abc"}}
+        result = sync_state(nodes, old)
+        assert result["n1"]["status"] == "covered"
+        assert result["n1"]["attempts"] == 3
+
+    def test_changed_hash_resets_status(self):
+        nodes = [_node("n1", "new_hash")]
+        old = {"n1": {"status": "covered", "attempts": 3, "source_hash": "old_hash"}}
+        result = sync_state(nodes, old)
+        assert result["n1"]["status"] == "unexplored"
+        assert result["n1"]["attempts"] == 0
+
+    def test_stale_entries_removed(self):
+        nodes = [_node("n1")]
+        old = {"n1": {"source_hash": "abc123"}, "n_gone": {"source_hash": "x"}}
+        result = sync_state(nodes, old)
+        assert "n_gone" not in result
+
+
+class TestApplyState:
+    def test_pushes_state_to_nodes(self):
+        node = _node("n1")
+        state = {"n1": {"status": "partial", "attempts": 2}}
+        apply_state([node], state)
+        assert node.status == "partial"
+        assert node.attempts == 2
+
+
+class TestRecordAttempt:
+    def test_increments_attempts(self):
+        state = {"n1": {"status": "unexplored", "attempts": 0, "notes": []}}
+        record_attempt(state, "n1", "partial", "needs None input")
+        assert state["n1"]["attempts"] == 1
+        assert state["n1"]["last_result"] == "partial"
+        assert "needs None input" in state["n1"]["notes"]
+
+    def test_covered_sets_status(self):
+        state = {"n1": {"status": "unexplored", "attempts": 0, "notes": []}}
+        record_attempt(state, "n1", "covered")
+        assert state["n1"]["status"] == "covered"
+
+    def test_new_node_creates_entry(self):
+        state = {}
+        record_attempt(state, "new", "partial")
+        assert "new" in state
+        assert state["new"]["attempts"] == 1

uv.lock

@@ -3219,6 +3219,9 @@ rex = [
     { name = "pyarrow" },
     { name = "pydantic" },
 ]
+sem = [
+    { name = "coverage" },
+]
 
 [package.dev-dependencies]
 dev = [
@@ -3226,6 +3229,7 @@ dev = [
     { name = "coverage" },
     { name = "dbt-core" },
     { name = "dbt-duckdb" },
+    { name = "jinja2" },
     { name = "marimo" },
     { name = "obstore" },
     { name = "openpyxl" },
@@ -3244,6 +3248,7 @@ requires-dist = [
     { name = "adlfs", marker = "extra == 'azure'", specifier = ">=2024.1.0" },
     { name = "azure-storage-blob", marker = "extra == 'azure'", specifier = ">=12.23.0" },
     { name = "boto3", marker = "extra == 'aws'", specifier = ">=1.35.0" },
+    { name = "coverage", marker = "extra == 'sem'", specifier = ">=7.13.4" },
     { name = "cryptography", marker = "extra == 'api'", specifier = ">=46.0.5" },
     { name = "databricks-bundles", marker = "extra == 'lake'", specifier = ">=0.295.0" },
     { name = "databricks-sdk", marker = "extra == 'lake'", specifier = ">=0.85.0" },
@@ -3314,7 +3319,7 @@ requires-dist = [
     { name = "uvicorn", marker = "extra == 'cli'", specifier = ">=0.41.0" },
     { name = "xlrd", specifier = ">=2.0.2" },
 ]
-provides-extras = ["conf", "aco", "api", "bcda", "bib", "bls", "ccw", "cli", "cms", "opps", "pfs", "rex", "perf", "lake", "aws", "gcp", "azure", "all"]
+provides-extras = ["conf", "aco", "api", "bcda", "bib", "bls", "ccw", "cli", "cms", "opps", "pfs", "rex", "perf", "sem", "lake", "aws", "gcp", "azure", "all"]
 
 [package.metadata.requires-dev]
 dev = [
@@ -3322,6 +3327,7 @@ dev = [
     { name = "coverage", specifier = ">=7.13.4" },
     { name = "dbt-core", specifier = "==1.10.15" },
     { name = "dbt-duckdb", specifier = ">=1.10,<1.11" },
+    { name = "jinja2", specifier = ">=3.1.0" },
     { name = "marimo", specifier = ">=0.20.0" },
     { name = "obstore", specifier = ">=0.9.2" },
     { name = "openpyxl", specifier = ">=3.1.5" },