feat: sem — semantic coverage orchestration #289

Merged

kert merged 11 commits from feat/semantic-coverage into main 2026-03-27 02:16:22 +00:00

Conversation 0 Commits 11 Files Changed 37 +4585 -105

kert commented 2026-03-26 18:40:43 +00:00

Owner

Copy Link

Summary

• New sem skinny package implementing semantic coverage orchestration — tracks coverage as AST nodes (functions, branches, exception handlers) instead of raw line numbers
• 4-layer architecture: parse (AST → semantic nodes) → enrich (attach Ruff + ty diagnostics) → runtime (map coverage.py data) → plan (priority scoring + target selection)
• Stable node identities using module::symbol::kind[ordinal] + source hash — drift-resistant across edits
• Rolling state persistence with automatic invalidation when source changes
• Priority heuristic: uncovered branches (5), ty signals (4), partially-tested functions (3), exception paths (3), Ruff warnings (2), penalise prior failures and already-covered nodes

Test plan

• 39 tests covering all 4 layers + state persistence
• Ruff clean
• Full test suite passes (12271 tests)

## Summary - New `sem` skinny package implementing **semantic coverage orchestration** — tracks coverage as AST nodes (functions, branches, exception handlers) instead of raw line numbers - 4-layer architecture: **parse** (AST → semantic nodes) → **enrich** (attach Ruff + ty diagnostics) → **runtime** (map coverage.py data) → **plan** (priority scoring + target selection) - Stable node identities using `module::symbol::kind[ordinal]` + source hash — drift-resistant across edits - Rolling state persistence with automatic invalidation when source changes - Priority heuristic: uncovered branches (5), ty signals (4), partially-tested functions (3), exception paths (3), Ruff warnings (2), penalise prior failures and already-covered nodes ## Test plan - [x] 39 tests covering all 4 layers + state persistence - [x] Ruff clean - [x] Full test suite passes (12271 tests)

kert added 1 commit 2026-03-26 18:40:43 +00:00

feat: sem — semantic coverage orchestration (AST + Ruff + ty + coverage) ... 96b8ccf6df

4-layer architecture for tracking coverage as semantic AST nodes rather
than raw line numbers: parse → enrich → runtime → plan.  Stable node
identities, Ruff/ty diagnostic attachment, coverage mapping, priority
scoring heuristic, and drift-resistant rolling state persistence.

39 tests, ruff clean.

kert added 1 commit 2026-03-26 21:22:43 +00:00

feat: sem.hooks — smart pre-commit with tracked hook architecture ... 046fcb885b

Move pre-commit logic from shell into src/sem/hooks.py as a proper
Python module.  Shell hook (dev/hooks/pre-commit) is now a 3-line
wrapper.  git core.hooksPath points at dev/hooks/ so hooks are
version-controlled, not buried in .git/.

Smart test selection: only runs tests for changed src/ modules
(src/aco/ → tests/aco/), always includes structural invariants
(test_ast_coverage.py) on any src/ change, falls back to full
12k+ suite when infrastructure (conftest, pyproject.toml) changes.
Notebooks checked only when staged.

Force full suite: GIT_PRE_COMMIT_FULL=1 git commit

Also: register pytest 'stub' marker, move post-commit to tracked dir.

kert added 1 commit 2026-03-26 21:31:12 +00:00

docs: comprehensive documentation for sem module and hook architecture ... 89a7fa3055

- README: add Testing section with semantic coverage architecture,
  pre-commit smart selection rules, hook steps, forcing full suite,
  running tests.  Update module count (14), test count (12,291), and
  project layout to include sem, opps, perf.
- sem/__init__.py: full module docstring with architecture diagram,
  layer descriptions, node identity format, and usage example.
- sem/hooks.py: full docstring with file classification table, test
  selection rules, step order, env vars, post-commit, and setup.

kert added 1 commit 2026-03-26 21:48:08 +00:00

feat: jinja2 README templating — gen_readme.py renders from live codebase ... 880226cc24

README.md.j2 is the source of truth. dev/scripts/gen_readme.py collects
live counts from compose.yml, pyproject.toml, pipeline registry, pytest,
and auth manifest, then renders the template with jinja2.

Dynamic values:
  - Service count (25, from compose.yml)
  - Pipeline count + step counts (14 pipelines, 230 steps)
  - Test count (12,291, from pytest --collect-only)
  - Module list (14, from pyproject.toml optional-deps)
  - Credential count (20, from api.auth.manifest)
  - Workflow count (7, from .gitea/workflows/)
  - src/ tree (generated from filesystem)

Usage:
  uv run python dev/scripts/gen_readme.py            # generate
  uv run python dev/scripts/gen_readme.py --check     # verify freshness
  uv run python dev/scripts/gen_readme.py --dry-run   # preview to stdout

kert added 1 commit 2026-03-26 23:58:18 +00:00

feat: deep Databricks SDK integration — governance, jobs, quality, secrets ... 97852e818c

Expand Databricks SDK usage from 6 to 18 WorkspaceClient services.

Tier 1 — Governance & Compliance:
- governance.py: declarative GovernancePolicy with apply + audit drift
- UnityClient: grants CRUD (ws.grants), secret management (ws.secrets),
  table constraints PK/FK (ws.table_constraints)
- sync_secrets.py: push env vars to Databricks scopes per stack.toml

Tier 2 — Job Orchestration:
- jobs.py: JobManager translates Pipeline → Databricks Job with task
  dependencies, run_now, get_run_status, list/delete
- UnityClient: warehouse lookup by name (ws.warehouses), start/stop

Tier 3 — Data Quality & Monitoring:
- quality.py: setup_monitors creates Lakehouse Monitoring profiles,
  list_monitors, run_refresh (ws.quality_monitors)
- UnityClient: system schema access, table lineage queries, audit log
  queries (ws.system_schemas, ws.statement_execution)

Config: stack.toml gains [databricks.warehouse], [databricks.secrets],
[databricks.governance], [databricks.quality] sections.

23 new tests, all passing.

kert added 1 commit 2026-03-27 00:02:23 +00:00

docs: document Databricks SDK integration in README ... 62b1771b6a

Add SDK integration section to README.md.j2 with:
- 18-service table showing category, SDK service, module, purpose
- Governance example (declarative grants, drift audit, PHI schemas)
- Job orchestration example (create, run, poll)
- Secret sync usage
- Quality monitoring usage
- stack.toml config snippets

Regenerate README.md (12,320 tests, 25 services, 14 pipelines).

kert added 1 commit 2026-03-27 00:23:35 +00:00

docs: fill documentation gaps — MEMORY.md, secret sync, README generation ... 0e7085b17a

- MEMORY.md: update milestone count (32 milestones, 29 closed), fix pipe
  layer stats (14 pipelines, 230 steps), add key files (sem, lake, j2),
  add sem module and Databricks SDK sections with memory file pointers
- project_sem_module.md: new memory file for semantic coverage architecture
- project_databricks_sdk.md: new memory file for 18-service SDK integration
- README.md.j2: add README generation section with gen_readme.py usage,
  expand secret sync section with --list flag and description
- Regenerate README.md (12,320 tests)

kert added 1 commit 2026-03-27 00:58:16 +00:00

fix: PFS payment calculation — MAC-aware GPCI join for exact CMS parity ... 19bc98b7f4

The payment() function joined RVU and GPCI on locality alone, but
locality is not unique in the GPCI table — 109 rows map to only 47
distinct locality codes.  Different Medicare Administrative Contractors
(MACs) apply different GPCI values to the same locality number.

Joining on locality alone produced a fan-out (e.g. 19 GPCI rows matched
for locality 01) and the calculation silently used the first match,
giving correct results only for that one MAC.

Fix: join on (mac, locality) when both DataFrames have a mac column,
falling back to locality-only for backward compatibility.

Validated: 0 diff across 889,658 carrier_locality rows for CY2025.

Also: update pfs_calcs.py notebook to use MAC-aware locality picker
so the comparison shows exact match ($0.00 difference) for all
year/MAC/locality/HCPCS combinations.

kert added 1 commit 2026-03-27 01:02:22 +00:00

fix: remove locality-only fallback — mac is always required for payment() ... 7015674238

The payment() join on (mac, locality) is the only correct path.
Locality alone is not unique in GPCI and produces wrong results.
Remove the conditional fallback, require mac in both DataFrames.

Update all test fixtures and inline test DataFrames to include mac.

kert added 1 commit 2026-03-27 01:42:13 +00:00

feat: skin sub pricing notebook — RVU, locality fees, and ASP over time ... 941f78d017

New marimo notebook tracking skin substitute economics across payment
systems and years:

1. Application code RVUs (15271-15278) over time
2. Locality-based PFS fees by region (MAC-aware carrier join)
3. Regional comparison for 15271 across 10 high-volume areas
4. Quarterly ASP payment limits for top 15 products with $127.28
   flat-rate reference line
5. OPPS payment method timeline (pass-through → high/low → flat rate)
6. Total episode cost: product ASP + application fee for 25cm² wound
7. Flat-rate impact analysis: winners and losers under CY2026
8. Summary table: fee range spread across all localities by year

Data sources: pfs.rvu, pfs.carrier_locality, pfs.gpci,
skin_subs.asp_quarterly, opps.skin_sub_addendum_b

refs #246

kert added 1 commit 2026-03-27 02:14:55 +00:00

chore: marimo format pass on skin_sub_pricing notebook 4c1fd0c04b

kert merged commit 240cba5186 into main 2026-03-27 02:16:22 +00:00

kert referenced this issue from a commit 2026-03-27 02:16:22 +00:00

Merge pull request 'feat: sem — semantic coverage orchestration' (#289) from feat/semantic-coverage into main

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

api

REST API (FastAPI)

auth

Authentication and authorization

bib

Bibliography, Zotero, citations

ci

CI/CD pipelines and Woodpecker plugins

cli

CLI commands and UX

cloud

Cloud provider abstraction (AWS, GCP, Azure, self-hosted)

config

Configuration and wiring

data

Data loading, CCLF, BCDA, seeds

databricks

Databricks jobs and deployment

docs

Documentation and generated references

image:api

image:docs

image:mc

image:notebooks

image:zotero

infra

lakehouse

Nessie, Polaris, Trino, Iceberg

llm

llm module — local RAG over comments + Zotero corpus

notebooks

marimo notebook errors (auto-filed by nb-watcher/nb-integration)

packaging

Python packaging, optional deps, build system

pfs

pipeline

Pipeline execution and validation

pipeline:ci

pipeline:deploy

pipeline:harden

pipeline:infra-ci

pipeline:rebuild-all

pipeline:release

qpp

quality

Testing, coverage, error handling

step:build

step:deploy

step:hadolint

step:lint

step:publish

step:scan

step:test

step:validate

styles

Charts, figures, visual theming

tree

Project directory structure and file organization

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

kert renovate-bot

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#289