docs: comprehensive documentation for sem module and hook architecture

- README: add Testing section with semantic coverage architecture,
  pre-commit smart selection rules, hook steps, forcing full suite,
  running tests.  Update module count (14), test count (12,291), and
  project layout to include sem, opps, perf.
- sem/__init__.py: full module docstring with architecture diagram,
  layer descriptions, node identity format, and usage example.
- sem/hooks.py: full docstring with file classification table, test
  selection rules, step order, env vars, post-commit, and setup.

README.md

@@ -2,7 +2,7 @@
 
 ![coverage](assets/icons/coverage.svg)
 
-Healthcare analytics platform on self-hosted infrastructure. Replaces dbt SQL models with narwhals DataFrame-agnostic expression functions, backed by DuckDB locally and Iceberg/Trino in the lakehouse. 22 services, 13 data pipelines, 11,955 tests at 99% coverage.
+Healthcare analytics platform on self-hosted infrastructure. Replaces dbt SQL models with narwhals DataFrame-agnostic expression functions, backed by DuckDB locally and Iceberg/Trino in the lakehouse. 22 services, 13 data pipelines, 12,291 tests at 99% coverage.
 
 ## Quick start
 
@@ -37,7 +37,7 @@ pip install stack[all]         # everything
 pip install stack[aco,aws]     # ACO analytics with AWS storage
 ```
 
-11 modules available as optional extras: `conf`, `aco`, `api`, `bcda`, `bib`, `bls`, `ccw`, `cli`, `cms`, `pfs`, `rex`. Cloud providers: `aws`, `gcp`, `azure`. Aggregates: `all`, `lake`.
+14 modules available as optional extras: `conf`, `aco`, `api`, `bcda`, `bib`, `bls`, `ccw`, `cli`, `cms`, `opps`, `perf`, `pfs`, `rex`, `sem`. Cloud providers: `aws`, `gcp`, `azure`. Aggregates: `all`, `lake`.
 
 ## Services
 
@@ -263,6 +263,115 @@ uv run python dev/scripts/gen_config.py --dab-sql   # generate SQL + DDL (~4 min
 databricks bundle deploy -t dev                       # deploy to Databricks
 ```
 
+## Testing
+
+### Architecture
+
+Testing follows a semantic coverage model (`src/sem/`) rather than tracking raw line numbers. The system parses every `.py` file into an AST, builds stable semantic nodes (functions, branches, exception handlers, loops), and attaches three signal layers:
+
+```
+source → AST → semantic nodes → attach (ruff, ty, coverage) → planner
+```
+
+| Layer | Module | What it does |
+|-------|--------|-------------|
+| **Parse** | `sem.parse` | Walk AST, emit `SemanticNode` per function, branch, except, loop, return, raise |
+| **Enrich** | `sem.enrich` | Run `ruff check` and `ty check`, map diagnostics to tightest-span node |
+| **Runtime** | `sem.runtime` | Map `coverage.py` JSON report onto nodes (hit/miss, per-test contexts) |
+| **Plan** | `sem.plan` | Score uncovered nodes by priority, suggest next test targets |
+| **State** | `sem.state` | Persist node status across runs, reset on source hash change |
+
+Each node has a stable identity derived from its module path, qualified symbol name, kind, and ordinal position — never from line numbers:
+
+```
+app.config::load_settings::branch_if[1]
+aco.express.pharmacy::pharmacy_claims::except_handler[0]
+```
+
+#### Priority heuristic
+
+```
+priority =
+    uncovered_branch × 5
+  + ty_diagnostic × 4
+  + inside_partially_tested_function × 3
+  + uncovered_exception_path × 3
+  + ruff_warning × 2
+  − prior_failures × 2
+  − already_covered × 5
+```
+
+#### Structural invariants
+
+`tests/test_ast_coverage.py` runs at collection time (pure AST, no imports) and enforces:
+
+1. Every `@nw.narwhalify` function has a docstring
+2. Every `Expr(...)` call supplies `name`, `fn`, `output`, and `after`
+3. Every public Pydantic model is importable
+4. Every `pipe/*.py` module exports a `Pipeline` with non-empty `.exprs`
+5. Implementation ratio stays above baseline (ratchet)
+6. `express/` and `pipe/` modules are symmetric
+
+### Pre-commit hooks
+
+Git hooks live in `dev/hooks/` (tracked) and are activated via:
+
+```bash
+git config core.hooksPath dev/hooks
+```
+
+The shell hook is a 3-line wrapper. All logic lives in `src/sem/hooks.py`:
+
+```bash
+#!/usr/bin/env bash
+exec uv run python -m sem.hooks
+```
+
+#### Smart test selection
+
+The hook classifies staged files and runs only what is relevant:
+
+| What changed | Tests run | Why |
+|-------------|-----------|-----|
+| `src/sem/*.py` | `tests/sem/` + `test_ast_coverage.py` | Module tests + structural invariants |
+| `src/aco/*.py` + `src/sem/*.py` | `tests/aco/` + `tests/sem/` + structural | Both module test dirs |
+| `tests/bib/test_sync.py` | `tests/bib/` | Changed test dir |
+| `pyproject.toml` or `conftest.py` | Full suite (12k+ tests) | Infrastructure change |
+| `notebooks/pfs_calcs.py` | `marimo check` + notebook execution | Notebook validation only |
+| `README.md` only | Nothing | No testable changes |
+
+The mapping rule is: `src/<module>/` changes → `tests/<module>/` runs. Any `src/` change also triggers `test_ast_coverage.py` to verify structural invariants haven't regressed.
+
+#### Forcing full suite
+
+```bash
+GIT_PRE_COMMIT_FULL=1 git commit -m "message"
+```
+
+#### Hook steps (in order)
+
+1. **Venv recovery** — if `uv run python -c 'import sys'` fails, run `uv sync --dev`
+2. **Config regeneration** — if `stack.toml` or `gen_config.py` changed, regenerate CI workflows
+3. **Ruff lint + format** — only staged `.py` files
+4. **AST parse check** — verify staged source files have valid syntax
+5. **Pytest** — targeted or full suite based on what changed
+6. **Marimo check** — only if notebooks are staged
+7. **Notebook execution** — only staged notebooks in the safe-to-run list
+
+#### Post-commit
+
+`dev/hooks/post-commit` rebuilds the docs Docker image in the background after every commit so the documentation site stays current with docstring changes.
+
+### Running tests
+
+```bash
+uv run python -m pytest tests/                    # full suite
+uv run python -m pytest tests/sem/                 # one module
+uv run python -m pytest tests/ -m "not stub"       # skip stub inventory
+uv run python -m pytest tests/ -m stub             # only stub status
+uv run python -m pytest tests/test_ast_coverage.py # structural invariants only
+```
+
 ## Project layout
 
 ```
@@ -280,8 +389,11 @@ stack/
 │   ├── cli/                     CLI entry point (typer)
 │   ├── cms/                     CMS public data tables
 │   ├── conf/                    Config loader, storage abstraction, table base
+│   ├── opps/                    Outpatient Prospective Payment System
+│   ├── perf/                    Pipeline telemetry (OpenTelemetry)
 │   ├── pfs/                     Physician Fee Schedule
-│   └── rex/                     REX fixed-width file processing
+│   ├── rex/                     REX fixed-width file processing
+│   └── sem/                     Semantic coverage orchestration
 ├── infra/                       Service configs and Dockerfiles
 │   ├── images/                  All Dockerfiles (api, notebooks, zotero, docs, mc)
 │   ├── traefik/                 Reverse proxy + loch CSS injection
@@ -300,10 +412,10 @@ stack/
 │   └── azure/                   ABFS, Azure SQL, Unity Catalog, Container Apps
 ├── dev/                         Dev tooling
 │   ├── scripts/                 Code generators, bootstrap, supply chain tools
-│   ├── hooks/                   Git pre-commit hook
+│   ├── hooks/                   Git hooks (tracked, core.hooksPath = dev/hooks)
 │   ├── seeds/                   Reference data (BCDA samples, CMS docs)
 │   └── pipelines/               CI-agnostic pipeline specs
-├── tests/                       11,955 tests at 99% coverage
+├── tests/                       12,291 tests at 99% coverage
 ├── notebooks/                   Marimo notebooks
 ├── docs/                        Docusaurus site
 ├── data/                        DuckDB, bib.sqlite, BCDA/CMS data, zotero (gitignored)

src/sem/__init__.py

@@ -4,11 +4,73 @@ AST-based coverage tracking that attaches Ruff diagnostics, ty type
 signals, and runtime coverage data to semantic nodes (functions, branches,
 exception handlers) rather than raw line numbers.
 
-Four layers:
+Architecture
-    1. parse   — build AST index of semantic nodes
+------------
-    2. enrich  — attach Ruff + ty diagnostics
+
-    3. runtime — attach coverage / per-test execution data
+::
-    4. plan    — prioritise uncovered nodes, suggest test targets
+
+    source .py
+        → ast.parse
+            → SemanticNode[] (stable IDs, no line numbers)
+                → attach ruff diagnostics
+                → attach ty diagnostics
+                → attach coverage.py runtime data
+                    → rank by priority heuristic
+                        → next_targets()
+
+Layers
+------
+
+1. **parse** — Walk AST, emit a ``SemanticNode`` for every function,
+   method, class, branch (if/elif/else), match case, exception handler,
+   loop, return, raise, assert, and with block.
+
+2. **enrich** — Run ``ruff check --output-format=json`` and
+   ``ty check --output-format=json`` as subprocesses, map each diagnostic
+   to the tightest-span semantic node.
+
+3. **runtime** — Read a ``coverage.py`` JSON report and mark which nodes
+   were hit/missed.  Attach per-test execution contexts when available.
+
+4. **plan** — Score each uncovered node using a weighted heuristic
+   (branches > ty signals > partial functions > exception paths > ruff)
+   and return a ranked list of test targets.
+
+5. **state** — Persist node-level tracking across runs in a JSON file.
+   Nodes whose source hash changes are automatically reset.  Stale
+   entries (removed from the AST) are pruned on sync.
+
+6. **hooks** — Pre-commit hook orchestration.  Smart test selection
+   based on staged files: ``src/<mod>/`` changes → ``tests/<mod>/``
+   runs.  Falls back to full suite on infrastructure changes.
+
+Node identity
+-------------
+
+Every node has a stable ID built from its module path, qualified symbol
+name, kind, and ordinal position::
+
+    aco.express.pharmacy::pharmacy_claims::branch_if[0]
+    sem.parse::_Collector::visit_If::branch_else[0]
+
+Line numbers are stored as metadata in ``Span`` but never appear in the
+ID.  A SHA-1 hash of the source fragment detects edits — the state layer
+resets a node's history when its hash changes.
+
+Usage
+-----
+
+::
+
+    from sem import parse_module, attach_ruff, attach_coverage
+    from sem import rank_nodes, next_targets
+
+    nodes = parse_module(Path("src/aco/express/pharmacy.py"))
+    attach_ruff(nodes, Path("src/aco/express/pharmacy.py"))
+    attach_coverage(nodes, Path("coverage.json"), "src/aco/express/pharmacy.py")
+
+    for target in next_targets(nodes, limit=5):
+        print(target.node_id, target.priority, target.kind)
 """
 
 from sem.enrich import attach_ruff, attach_ty

src/sem/hooks.py

@@ -2,20 +2,85 @@
 
 This module implements the pre-commit orchestration for the stack repo.
 Instead of running all 12k+ tests on every commit, it analyses staged
-files to determine:
+files to determine the minimal set of tests that cover the change.
 
-1. Which ``src/`` modules changed → run matching ``tests/<module>/``
+Architecture
-2. Whether infrastructure changed (conftest, pyproject) → full suite
+------------
-3. Whether notebooks changed → marimo check + execution
-4. AST parse validation on every staged ``.py`` file
 
-The shell hook (``dev/hooks/pre-commit``) is a thin wrapper that calls
+Git hooks live in ``dev/hooks/`` (tracked, version-controlled).
-``uv run python -m sem.hooks``.
+``core.hooksPath`` is set to ``dev/hooks`` so git uses them directly —
+no manual copy to ``.git/hooks/`` needed.
+
+The shell hook is a 3-line wrapper::
+
+    #!/usr/bin/env bash
+    exec uv run python -m sem.hooks
+
+All intelligence is here in Python, where it can be tested, linted,
+and evolved alongside the codebase it protects.
+
+File classification
+-------------------
+
+Staged files are partitioned into categories:
+
+========== ============================================= =================
+Category   Pattern                                       Triggers
+========== ============================================= =================
+``src``    ``src/**/*.py``                                targeted pytest
+``tests``  ``tests/**/*.py``                              targeted pytest
+``infra``  ``conftest.py``, ``pyproject.toml``            full pytest
+``config`` ``stack.toml``, ``gen_config.py``, backends    config regen
+``notebooks`` ``notebooks/*.py``                          marimo + exec
+========== ============================================= =================
+
+Test selection rules
+--------------------
+
+1. ``src/aco/express/foo.py`` staged → run ``tests/aco/``
+2. ``src/sem/parse.py`` staged → run ``tests/sem/``
+3. Any ``src/`` change → also run ``tests/test_ast_coverage.py``
+   (structural invariants: docstrings, Expr contracts, import checks)
+4. ``tests/bib/test_sync.py`` staged → run ``tests/bib/``
+5. ``pyproject.toml`` or ``conftest.py`` → full suite
+6. Notebooks → ``marimo check`` + execute safe-list notebooks
+
+The mapping is always ``src/<module>/`` → ``tests/<module>/``.  Multiple
+modules can be targeted in the same commit (e.g. ``src/aco/`` +
+``src/sem/`` → ``tests/aco/`` + ``tests/sem/`` + structural).
+
+Steps (in execution order)
+--------------------------
+
+1. **Venv recovery** — auto-run ``uv sync --dev`` if interpreter broken
+2. **Config regen** — regenerate CI workflows if stack.toml changed
+3. **Ruff lint** — check only staged ``.py`` files
+4. **Ruff format** — verify formatting on staged files
+5. **AST parse** — validate syntax of staged src files
+6. **Pytest** — targeted or full suite
+7. **Marimo check** — validate staged notebooks
+8. **Notebook execution** — run staged notebooks in safe list
 
 Environment variables
 ---------------------
-GIT_PRE_COMMIT_FULL=1
+
+``GIT_PRE_COMMIT_FULL=1``
     Force full test suite regardless of what changed.
+    Usage: ``GIT_PRE_COMMIT_FULL=1 git commit -m "message"``
+
+Post-commit
+-----------
+
+``dev/hooks/post-commit`` rebuilds the docs Docker image in the
+background after every commit so the documentation site stays current
+with docstring and bib.sqlite changes.
+
+Setup (once per clone)
+----------------------
+
+::
+
+    git config core.hooksPath dev/hooks
 """
 
 from __future__ import annotations