feat: sem — semantic coverage orchestration #289

Merged

kert merged 11 commits from feat/semantic-coverage into main 2026-03-27 02:16:22 +00:00

Conversation 0 Commits 11 Files Changed 37 +4585 -105

11 Commits

Author	SHA1	Message	Date
kert	4c1fd0c04b	chore: marimo format pass on skin_sub_pricing notebook	2026-03-26 22:14:49 -04:00
kert	941f78d017	feat: skin sub pricing notebook — RVU, locality fees, and ASP over time ... New marimo notebook tracking skin substitute economics across payment systems and years: 1. Application code RVUs (15271-15278) over time 2. Locality-based PFS fees by region (MAC-aware carrier join) 3. Regional comparison for 15271 across 10 high-volume areas 4. Quarterly ASP payment limits for top 15 products with $127.28 flat-rate reference line 5. OPPS payment method timeline (pass-through → high/low → flat rate) 6. Total episode cost: product ASP + application fee for 25cm² wound 7. Flat-rate impact analysis: winners and losers under CY2026 8. Summary table: fee range spread across all localities by year Data sources: pfs.rvu, pfs.carrier_locality, pfs.gpci, skin_subs.asp_quarterly, opps.skin_sub_addendum_b refs #246	2026-03-26 21:42:05 -04:00
kert	7015674238	fix: remove locality-only fallback — mac is always required for payment() ... The payment() join on (mac, locality) is the only correct path. Locality alone is not unique in GPCI and produces wrong results. Remove the conditional fallback, require mac in both DataFrames. Update all test fixtures and inline test DataFrames to include mac.	2026-03-26 21:00:47 -04:00
kert	19bc98b7f4	fix: PFS payment calculation — MAC-aware GPCI join for exact CMS parity ... The payment() function joined RVU and GPCI on locality alone, but locality is not unique in the GPCI table — 109 rows map to only 47 distinct locality codes. Different Medicare Administrative Contractors (MACs) apply different GPCI values to the same locality number. Joining on locality alone produced a fan-out (e.g. 19 GPCI rows matched for locality 01) and the calculation silently used the first match, giving correct results only for that one MAC. Fix: join on (mac, locality) when both DataFrames have a mac column, falling back to locality-only for backward compatibility. Validated: 0 diff across 889,658 carrier_locality rows for CY2025. Also: update pfs_calcs.py notebook to use MAC-aware locality picker so the comparison shows exact match ($0.00 difference) for all year/MAC/locality/HCPCS combinations.	2026-03-26 20:58:01 -04:00
kert	0e7085b17a	docs: fill documentation gaps — MEMORY.md, secret sync, README generation ... - MEMORY.md: update milestone count (32 milestones, 29 closed), fix pipe layer stats (14 pipelines, 230 steps), add key files (sem, lake, j2), add sem module and Databricks SDK sections with memory file pointers - project_sem_module.md: new memory file for semantic coverage architecture - project_databricks_sdk.md: new memory file for 18-service SDK integration - README.md.j2: add README generation section with gen_readme.py usage, expand secret sync section with --list flag and description - Regenerate README.md (12,320 tests)	2026-03-26 20:23:22 -04:00
kert	62b1771b6a	docs: document Databricks SDK integration in README ... Add SDK integration section to README.md.j2 with: - 18-service table showing category, SDK service, module, purpose - Governance example (declarative grants, drift audit, PHI schemas) - Job orchestration example (create, run, poll) - Secret sync usage - Quality monitoring usage - stack.toml config snippets Regenerate README.md (12,320 tests, 25 services, 14 pipelines).	2026-03-26 20:02:16 -04:00
kert	97852e818c	feat: deep Databricks SDK integration — governance, jobs, quality, secrets ... Expand Databricks SDK usage from 6 to 18 WorkspaceClient services. Tier 1 — Governance & Compliance: - governance.py: declarative GovernancePolicy with apply + audit drift - UnityClient: grants CRUD (ws.grants), secret management (ws.secrets), table constraints PK/FK (ws.table_constraints) - sync_secrets.py: push env vars to Databricks scopes per stack.toml Tier 2 — Job Orchestration: - jobs.py: JobManager translates Pipeline → Databricks Job with task dependencies, run_now, get_run_status, list/delete - UnityClient: warehouse lookup by name (ws.warehouses), start/stop Tier 3 — Data Quality & Monitoring: - quality.py: setup_monitors creates Lakehouse Monitoring profiles, list_monitors, run_refresh (ws.quality_monitors) - UnityClient: system schema access, table lineage queries, audit log queries (ws.system_schemas, ws.statement_execution) Config: stack.toml gains [databricks.warehouse], [databricks.secrets], [databricks.governance], [databricks.quality] sections. 23 new tests, all passing.	2026-03-26 19:57:51 -04:00
kert	880226cc24	feat: jinja2 README templating — gen_readme.py renders from live codebase ... README.md.j2 is the source of truth. dev/scripts/gen_readme.py collects live counts from compose.yml, pyproject.toml, pipeline registry, pytest, and auth manifest, then renders the template with jinja2. Dynamic values: - Service count (25, from compose.yml) - Pipeline count + step counts (14 pipelines, 230 steps) - Test count (12,291, from pytest --collect-only) - Module list (14, from pyproject.toml optional-deps) - Credential count (20, from api.auth.manifest) - Workflow count (7, from .gitea/workflows/) - src/ tree (generated from filesystem) Usage: uv run python dev/scripts/gen_readme.py # generate uv run python dev/scripts/gen_readme.py --check # verify freshness uv run python dev/scripts/gen_readme.py --dry-run # preview to stdout	2026-03-26 17:46:38 -04:00
kert	89a7fa3055	docs: comprehensive documentation for sem module and hook architecture ... - README: add Testing section with semantic coverage architecture, pre-commit smart selection rules, hook steps, forcing full suite, running tests. Update module count (14), test count (12,291), and project layout to include sem, opps, perf. - sem/__init__.py: full module docstring with architecture diagram, layer descriptions, node identity format, and usage example. - sem/hooks.py: full docstring with file classification table, test selection rules, step order, env vars, post-commit, and setup.	2026-03-26 17:31:01 -04:00
kert	046fcb885b	feat: sem.hooks — smart pre-commit with tracked hook architecture ... Move pre-commit logic from shell into src/sem/hooks.py as a proper Python module. Shell hook (dev/hooks/pre-commit) is now a 3-line wrapper. git core.hooksPath points at dev/hooks/ so hooks are version-controlled, not buried in .git/. Smart test selection: only runs tests for changed src/ modules (src/aco/ → tests/aco/), always includes structural invariants (test_ast_coverage.py) on any src/ change, falls back to full 12k+ suite when infrastructure (conftest, pyproject.toml) changes. Notebooks checked only when staged. Force full suite: GIT_PRE_COMMIT_FULL=1 git commit Also: register pytest 'stub' marker, move post-commit to tracked dir.	2026-03-26 17:21:11 -04:00
kert	96b8ccf6df	feat: sem — semantic coverage orchestration (AST + Ruff + ty + coverage) ... 4-layer architecture for tracking coverage as semantic AST nodes rather than raw line numbers: parse → enrich → runtime → plan. Stable node identities, Ruff/ty diagnostic attachment, coverage mapping, priority scoring heuristic, and drift-resistant rolling state persistence. 39 tests, ruff clean.	2026-03-26 14:37:54 -04:00