feat: jinja2 README templating — gen_readme.py renders from live codebase
Some checks failed
CI / lint-test (push) Successful in 1m12s
CI / skinny-install (api) (push) Successful in 28s
CI / skinny-install (bcda) (push) Successful in 33s
CI / skinny-install (bib) (push) Successful in 28s
CI / skinny-install (aco) (push) Successful in 49s
CI / lint-test (pull_request) Successful in 1m19s
CI / skinny-install (ccw) (pull_request) Successful in 29s
CI / skinny-install (cli) (pull_request) Successful in 33s
CI / skinny-install (cms) (pull_request) Successful in 29s
CI / skinny-install (opps) (pull_request) Successful in 26s
CI / skinny-install (pfs) (pull_request) Successful in 30s
Infra CI / mc (push) Successful in 6s
Infra CI / api (push) Successful in 8s
Infra CI / notebooks (pull_request) Successful in 7s
Infra CI / zotero (pull_request) Successful in 6s
Infra CI / mc (pull_request) Successful in 6s
CI / skinny-install (bls) (push) Successful in 25s
CI / skinny-install (ccw) (push) Successful in 31s
CI / skinny-install (cli) (push) Successful in 29s
CI / skinny-install (cms) (push) Successful in 27s
CI / skinny-install (conf) (push) Successful in 30s
CI / skinny-install (opps) (push) Successful in 29s
CI / skinny-install (perf) (push) Successful in 35s
CI / skinny-install (pfs) (push) Successful in 32s
CI / skinny-install (rex) (push) Successful in 34s
CI / skinny-install (aco) (pull_request) Successful in 44s
CI / skinny-install (api) (pull_request) Successful in 30s
CI / skinny-install (bcda) (pull_request) Successful in 37s
CI / skinny-install (bib) (pull_request) Successful in 29s
CI / skinny-install (bls) (pull_request) Successful in 28s
CI / skinny-install (conf) (pull_request) Successful in 29s
CI / skinny-install (perf) (pull_request) Successful in 24s
Infra CI / notebooks (push) Successful in 11s
CI / skinny-install (rex) (pull_request) Successful in 32s
Infra CI / zotero (push) Successful in 6s
Infra CI / docs (push) Failing after 6s
Infra CI / docs (pull_request) Failing after 6s
Infra CI / api (pull_request) Successful in 7s
Package Supply Chain / pkg-supply-chain (push) Successful in 44s

README.md.j2 is the source of truth. dev/scripts/gen_readme.py collects
live counts from compose.yml, pyproject.toml, pipeline registry, pytest,
and auth manifest, then renders the template with jinja2.

Dynamic values:
  - Service count (25, from compose.yml)
  - Pipeline count + step counts (14 pipelines, 230 steps)
  - Test count (12,291, from pytest --collect-only)
  - Module list (14, from pyproject.toml optional-deps)
  - Credential count (20, from api.auth.manifest)
  - Workflow count (7, from .gitea/workflows/)
  - src/ tree (generated from filesystem)

Usage:
  uv run python dev/scripts/gen_readme.py            # generate
  uv run python dev/scripts/gen_readme.py --check     # verify freshness
  uv run python dev/scripts/gen_readme.py --dry-run   # preview to stdout
This commit is contained in:
kert
2026-03-26 17:46:38 -04:00
parent 89a7fa3055
commit 880226cc24
5 changed files with 770 additions and 37 deletions

View File

@@ -2,7 +2,7 @@
![coverage](assets/icons/coverage.svg) ![coverage](assets/icons/coverage.svg)
Healthcare analytics platform on self-hosted infrastructure. Replaces dbt SQL models with narwhals DataFrame-agnostic expression functions, backed by DuckDB locally and Iceberg/Trino in the lakehouse. 22 services, 13 data pipelines, 12,291 tests at 99% coverage. Healthcare analytics platform on self-hosted infrastructure. Replaces dbt SQL models with narwhals DataFrame-agnostic expression functions, backed by DuckDB locally and Iceberg/Trino in the lakehouse. 25 services, 14 data pipelines, 12,291 tests at 99% coverage.
## Quick start ## Quick start
@@ -37,7 +37,7 @@ pip install stack[all] # everything
pip install stack[aco,aws] # ACO analytics with AWS storage pip install stack[aco,aws] # ACO analytics with AWS storage
``` ```
14 modules available as optional extras: `conf`, `aco`, `api`, `bcda`, `bib`, `bls`, `ccw`, `cli`, `cms`, `opps`, `perf`, `pfs`, `rex`, `sem`. Cloud providers: `aws`, `gcp`, `azure`. Aggregates: `all`, `lake`. 14 modules available as optional extras: `aco`, `api`, `bcda`, `bib`, `bls`, `ccw`, `cli`, `cms`, `conf`, `opps`, `perf`, `pfs`, `rex`, `sem`. Cloud providers: `aws`, `azure`, `gcp`. Aggregates: `all`, `lake`.
## Services ## Services
@@ -85,23 +85,24 @@ All services route through Traefik at `*.homelab.fhirworx.io`.
### Pipelines ### Pipelines
13 registered pipelines with 189 steps, validated by `run_pipeline(steps, load)` which enforces output schemas via pandera-style column checks. 14 registered pipelines with 230 steps, validated by `run_pipeline(steps, load)` which enforces output schemas via pandera-style column checks.
| Pipeline | Domain | | Pipeline | Steps | Domain |
|----------|--------| |----------|------:|--------|
| `input_layer` | Raw CCLF/BCDA ingestion | | `ahrq_measures` | 39 | AHRQ quality indicators |
| `core` | Encounters, patients, providers | | `cclf` | 19 | CCLF connector pipeline -- raw CCLF files -> input_layer tables |
| `claims_preprocessing` | Claim matching, encounter IDs | | `claims_preprocessing` | 63 | Claim matching, encounter IDs |
| `readmissions` | 30-day readmission rates | | `cms_quality_measures` | 20 | CMS ACO quality metrics |
| `pharmacy` | Part D drug utilization | | `core` | 12 | Encounters, patients, providers |
| `ahrq_measures` | AHRQ quality indicators | | `data_quality` | 1 | Validation and completeness |
| `cms_quality_measures` | CMS ACO quality metrics | | `hcc_suspecting` | 6 | HCC risk adjustment |
| `hcc_suspecting` | HCC risk adjustment | | `input_layer` | 13 | Raw CCLF/BCDA ingestion |
| `provider_attribution` | Provider-patient assignment | | `main` | 14 | Full pipeline orchestration |
| `quality_measures` | Composite quality scores | | `opps` | 2 | Outpatient payment system |
| `data_quality` | Validation and completeness | | `pharmacy` | 4 | Part D drug utilization |
| `cclf` | CCLF file parsing | | `provider_attribution` | 10 | Provider-patient assignment |
| `main` | Full pipeline orchestration | | `quality_measures` | 15 | Composite quality scores |
| `readmissions` | 12 | 30-day readmission rates |
### Expression layer ### Expression layer
@@ -154,7 +155,7 @@ The generator reads image definitions from `stack.toml [images]`, dispatches to
### Workflows ### Workflows
Seven workflows generated per backend: 7 workflows generated per backend:
| Workflow | Trigger | What it does | | Workflow | Trigger | What it does |
|----------|---------|--------------| |----------|---------|--------------|
@@ -193,7 +194,7 @@ Daily automated pipeline (`pkg-supply-chain.yml`) that:
## Credential management ## Credential management
All 19 service credentials derive from a single 256-bit root key via HKDF-SHA256. No passwords stored in `.env` — they regenerate deterministically from root key + commit SHA on each deploy. All 20 service credentials derive from a single 256-bit root key via HKDF-SHA256. No passwords stored in `.env` — they regenerate deterministically from root key + commit SHA on each deploy.
``` ```
ROOT_KEY + commit_sha -> HKDF-SHA256 -> all credentials -> .env + backends ROOT_KEY + commit_sha -> HKDF-SHA256 -> all credentials -> .env + backends
@@ -256,7 +257,7 @@ Switch with `STACK_CONTEXT=aws` or edit `stack.toml [context] active`.
## Databricks ## Databricks
Databricks Asset Bundle (`databricks.yml`) is auto-generated from the pipeline registry. 13 pipeline tasks with dependency ordering, 3 targets (dev/staging/prod), daily schedule. Databricks Asset Bundle (`databricks.yml`) is auto-generated from the pipeline registry. 14 pipeline tasks with dependency ordering, 3 targets (dev/staging/prod), daily schedule.
```bash ```bash
uv run python dev/scripts/gen_config.py --dab-sql # generate SQL + DDL (~4 min) uv run python dev/scripts/gen_config.py --dab-sql # generate SQL + DDL (~4 min)
@@ -336,7 +337,7 @@ The hook classifies staged files and runs only what is relevant:
| `src/sem/*.py` | `tests/sem/` + `test_ast_coverage.py` | Module tests + structural invariants | | `src/sem/*.py` | `tests/sem/` + `test_ast_coverage.py` | Module tests + structural invariants |
| `src/aco/*.py` + `src/sem/*.py` | `tests/aco/` + `tests/sem/` + structural | Both module test dirs | | `src/aco/*.py` + `src/sem/*.py` | `tests/aco/` + `tests/sem/` + structural | Both module test dirs |
| `tests/bib/test_sync.py` | `tests/bib/` | Changed test dir | | `tests/bib/test_sync.py` | `tests/bib/` | Changed test dir |
| `pyproject.toml` or `conftest.py` | Full suite (12k+ tests) | Infrastructure change | | `pyproject.toml` or `conftest.py` | Full suite (12,291+ tests) | Infrastructure change |
| `notebooks/pfs_calcs.py` | `marimo check` + notebook execution | Notebook validation only | | `notebooks/pfs_calcs.py` | `marimo check` + notebook execution | Notebook validation only |
| `README.md` only | Nothing | No testable changes | | `README.md` only | Nothing | No testable changes |
@@ -376,24 +377,24 @@ uv run python -m pytest tests/test_ast_coverage.py # structural invariants only
``` ```
stack/ stack/
├── compose.yml Docker Compose (22 services) ├── compose.yml Docker Compose (25 services)
├── stack.toml Centralised configuration ├── stack.toml Centralised configuration
├── pyproject.toml Python project (uv, optional deps per module) ├── pyproject.toml Python project (uv, optional deps per module)
├── src/ ├── src/
│ ├── aco/ ACO analytics (express, pipe, table, lake, load) │ ├── aco/ ACO analytics (express, pipe, table, lake, load)
│ ├── api/ FastAPI server + auth + diag │ ├── api/ FastAPI server + auth + diag
│ ├── bcda/ BCDA FHIR R4 client │ ├── bcda/ BCDA FHIR R4 client
│ ├── bib/ Zotero bibliography store, tags, metadata │ ├── bib/ Zotero bibliography store, tags, metadata
│ ├── bls/ BLS data │ ├── bls/ BLS data
│ ├── ccw/ CCW data dictionary │ ├── ccw/ CCW data dictionary
│ ├── cli/ CLI entry point (typer) │ ├── cli/ CLI entry point (typer)
│ ├── cms/ CMS public data tables │ ├── cms/ CMS public data tables
│ ├── conf/ Config loader, storage abstraction, table base │ ├── conf/ Config loader, storage abstraction, table base
│ ├── opps/ Outpatient Prospective Payment System │ ├── opps/ Outpatient Prospective Payment System
│ ├── perf/ Pipeline telemetry (OpenTelemetry) │ ├── perf/ Pipeline telemetry (OpenTelemetry)
│ ├── pfs/ Physician Fee Schedule │ ├── pfs/ Physician Fee Schedule
│ ├── rex/ REX fixed-width file processing │ ├── rex/ REX fixed-width file processing
│ └── sem/ Semantic coverage orchestration │ └── sem/ Semantic coverage orchestration
├── infra/ Service configs and Dockerfiles ├── infra/ Service configs and Dockerfiles
│ ├── images/ All Dockerfiles (api, notebooks, zotero, docs, mc) │ ├── images/ All Dockerfiles (api, notebooks, zotero, docs, mc)
│ ├── traefik/ Reverse proxy + loch CSS injection │ ├── traefik/ Reverse proxy + loch CSS injection

429
README.md.j2 Normal file
View File

@@ -0,0 +1,429 @@
{# README.md.j2 — source of truth for README.md
Render with: uv run python dev/scripts/gen_readme.py
Check freshness: uv run python dev/scripts/gen_readme.py --check
#}
# Stack
![coverage](assets/icons/coverage.svg)
Healthcare analytics platform on self-hosted infrastructure. Replaces dbt SQL models with narwhals DataFrame-agnostic expression functions, backed by DuckDB locally and Iceberg/Trino in the lakehouse. {{ services.count }} services, {{ pipelines.count }} data pipelines, {{ tests.formatted }} tests at 99% coverage.
## Quick start
```bash
# 1. Generate root key and seed .env
ROOT_KEY=$(openssl rand -hex 32)
cat > .env <<EOF
DOMAIN=homelab.fhirworx.io
HOST_IP=192.168.1.192
EOF
# 2. Start core services and bootstrap credentials
docker compose up -d postgres rustfs traefik
ROOT_KEY=$ROOT_KEY uv run python -m api.auth bootstrap $(git rev-parse HEAD)
# 3. Start everything
docker compose up -d
# 4. Verify
curl -s http://localhost:8000/health | python3 -m json.tool
```
## Installation
The package supports skinny installs — install only the modules you need:
```bash
pip install stack[aco] # ACO analytics only
pip install stack[bib] # bibliography only
pip install stack[cli] # CLI (pulls aco + api + bib)
pip install stack[all] # everything
pip install stack[aco,aws] # ACO analytics with AWS storage
```
{{ modules.count }} modules available as optional extras: {{ modules.names | map('backtick') | join(', ') }}. Cloud providers: {{ modules.cloud | map('backtick') | join(', ') }}. Aggregates: {{ modules.aggregates | map('backtick') | join(', ') }}.
## Services
All services route through Traefik at `*.homelab.fhirworx.io`.
| Service | URL | Purpose |
|---------|-----|---------|
| API | `api` (internal) | FastAPI — health, pipelines, bib endpoints |
| Notebooks | `notebooks.homelab.fhirworx.io` | Marimo notebooks (GPU-accelerated) |
| Docs | `docs.homelab.fhirworx.io` | Docusaurus — API reference + CMS bibliography |
| Gitea | `gitea.homelab.fhirworx.io` | Git server, container registry, LFS |
| Zotero | `zotero.homelab.fhirworx.io` | Reference manager (KasmVNC desktop) |
| Dashboard | `homelab.fhirworx.io` | Service launcher |
| RustFS | `s3.homelab.fhirworx.io` | S3-compatible object storage |
| Nessie | `nessie.homelab.fhirworx.io` | Git-like Iceberg catalog |
| Polaris | `polaris.homelab.fhirworx.io` | Iceberg catalog with governance |
| Trino | `trino.homelab.fhirworx.io` | Distributed SQL engine |
| Grafana | `grafana.homelab.fhirworx.io` | Dashboards |
| Prometheus | `prometheus.homelab.fhirworx.io` | Metrics |
| Jaeger | `jaeger.homelab.fhirworx.io` | Distributed tracing |
| Loki | `loki.homelab.fhirworx.io` | Log aggregation |
## Analytics platform
### Architecture
```
CCLF/BCDA/CMS data
|
v
src/aco/express/ narwhals expressions (DataFrame-agnostic)
|
v
src/aco/pipe/ pipeline runner with schema validation
|
v
DuckDB (local) --or-- Iceberg + Trino (lakehouse) --or-- Databricks
|
v
src/api/ FastAPI (health, runs, bib)
|
v
notebooks/ Marimo interactive analysis
```
### Pipelines
{{ pipelines.count }} registered pipelines with {{ pipelines.total_steps }} steps, validated by `run_pipeline(steps, load)` which enforces output schemas via pandera-style column checks.
| Pipeline | Steps | Domain |
|----------|------:|--------|
{% for p in pipelines.list %}
| `{{ p.name }}` | {{ p.steps }} | {{ p.doc }} |
{% endfor %}
### Expression layer
`src/aco/express/` contains narwhals functions that work across polars, pandas, and cuDF. Each module mirrors a pipeline and exports pure transformation functions.
### Configuration
`stack.toml` is the single config file. `src/conf/` provides:
- `cfg` — attribute access (`cfg.db.aco`)
- `path("db.aco")` — absolute path resolution relative to repo root
- `context()` — environment switching (local/lake/databricks/trino/aws/gcp/azure)
- `secret("api.secret", "STACK_API_SECRET")` — env var with config fallback
- `storage.get_filesystem()` — multi-cloud storage abstraction (S3/GCS/ABFS/local)
Override context at runtime: `STACK_CONTEXT=lake` (or `aws`, `gcp`, `azure`).
### Bibliography
`src/bib/` manages a Zotero-backed citation store in `data/bib.sqlite`. Column-level provenance via `Tag.col(ref, desc)`. The docs site exports `library.json` for a searchable bibliography browser at `/library`.
## CI/CD
### Backend switching
CI/CD workflows are auto-generated from `stack.toml` by `gen_config.py`. Three backends are supported:
| Backend | Workflows directory | When to use |
|---------|---------------------|-------------|
| `gitea` | `.gitea/workflows/` | Self-hosted Gitea instance (default) |
| `github` | `.github/workflows/` | GitHub repos / GitHub Actions |
| `woodpecker` | `.woodpecker/` | Woodpecker CI server |
Switch backends by editing `stack.toml`:
```toml
[ci]
backend = "gitea" # change to "github" or "woodpecker"
```
Then regenerate:
```bash
uv run python dev/scripts/gen_config.py # generate new workflows
uv run python dev/scripts/gen_config.py --check # verify (used in CI)
uv run python dev/scripts/gen_config.py --backend github # one-off override
```
The generator reads image definitions from `stack.toml [images]`, dispatches to the active backend emitter (`dev/scripts/backends/{backend}.py`), writes workflow YAML, and auto-removes stale workflows from inactive backend directories.
### Workflows
{{ workflows.count }} workflows generated per backend:
| Workflow | Trigger | What it does |
|----------|---------|--------------|
| `ci.yml` | Push/PR | Ruff lint + format, pytest (99% coverage), gen_config --check, skinny-install matrix |
| `deploy.yml` | Push to main | Build 5 images, Trivy scan, vuln reporting |
| `harden.yml` | Weekly cron / manual | Rebuild `--no-cache`, scan, auto-close/file vuln issues |
| `rebuild-all.yml` | Manual | Full rebuild + scan + deploy |
| `infra-ci.yml` | Path-filtered push | Hadolint Dockerfiles, dry-run builds |
| `release.yml` | Tag (`v*`) | `uv build` + release |
| `pkg-supply-chain.yml` | Daily / push | Package inventory, mirror sync, drift detection, vuln scan |
### Secret mapping
| Secret | Gitea | GitHub | Woodpecker |
|--------|-------|--------|------------|
| Registry auth | `REGISTRY_USER` + `REGISTRY_TOKEN` | `GITHUB_TOKEN` (built-in) | `registry_username` + `registry_password` |
| Gitea API | `GITEA_TOKEN` | N/A | `gitea_token` |
| Vuln reporting | `GITEA_TOKEN` | `GITHUB_TOKEN` | `gitea_token` |
### Image tagging
Images are tagged with the short commit SHA (8 chars). Compose resolves via `${COMMIT_SHA:-latest}` from `.env`. Custom images: `api`, `notebooks`, `zotero`, `docs`, `mc`.
### Vulnerability management
The `harden.yml` pipeline rebuilds all images with `--no-cache` to pick up OS patches, runs Trivy scans, and auto-files or auto-closes Gitea issues via `api.diag.vuln`.
### Package supply chain
Daily automated pipeline (`pkg-supply-chain.yml`) that:
1. Scans Dockerfiles, pyproject.toml, and CI workflows for all package dependencies
2. Syncs packages to the Gitea package registry (local mirror)
3. Detects drift between manifest and mirror
4. Runs Trivy vulnerability scans on dependencies
5. Auto-creates Gitea issues for missing packages and CVEs
## Credential management
All {{ credentials.count }} service credentials derive from a single 256-bit root key via HKDF-SHA256. No passwords stored in `.env` — they regenerate deterministically from root key + commit SHA on each deploy.
```
ROOT_KEY + commit_sha -> HKDF-SHA256 -> all credentials -> .env + backends
```
| Tier | Salt | Rotates |
|------|------|---------|
| Bootstrap | `b"bootstrap"` | Only when root key changes |
| Service | `commit_sha` | Every deploy |
```bash
# Bootstrap (first time)
ROOT_KEY=$KEY uv run python -m api.auth bootstrap $(git rev-parse HEAD)
# Rotate (every deploy, automatic in CI)
ROOT_KEY=$KEY uv run python -m api.auth provision $COMMIT_SHA
# Dry-run
ROOT_KEY=$KEY uv run python -m api.auth derive $COMMIT_SHA --redact
```
## Network segmentation
| Network | Type | Services |
|---------|------|----------|
| `gateway` | external | Traefik, all public-facing |
| `storage` | internal | PostgreSQL, RustFS |
| `data` | internal | Nessie, Trino, Polaris |
| `observability` | internal | Grafana, Prometheus, Jaeger, Loki |
| `ci` | external | Gitea, Act Runner, RustFS |
## Design system
Theme: **loch** (deep-navy). Traefik's `inject-loch` middleware rewrites HTML to inject `loch.css` and the favicon. Per-service CSS in `assets/css/`. Altair chart theme in `assets/nature.py` (Nature-conformant, accessible palette, IBM Plex Mono headings).
## Data lakehouse
| Component | Purpose |
|-----------|---------|
| Nessie | Git-like branching and time travel for Iceberg tables |
| Polaris | Iceberg catalog with RBAC and multi-tenant governance |
| Trino | Distributed SQL (`iceberg` catalog) |
| RustFS | S3 storage backend (`s3://lakehouse/`) |
Query via Trino CLI, JDBC, Web UI, PyIceberg, or DuckDB+PyArrow.
### Cloud deployment
The platform runs on self-hosted Docker Compose by default. Cloud provider guides in `cloud/`:
| Provider | Storage | Catalog | Compute | Context |
|----------|---------|---------|---------|---------|
| Self-hosted | RustFS (S3) | Nessie / Polaris | Docker Compose | `local` / `lake` |
| AWS | S3 | Glue | ECS / EKS | `aws` |
| GCP | GCS | BigQuery | Cloud Run / GKE | `gcp` |
| Azure | ABFS | Unity Catalog | Container Apps / AKS | `azure` |
| Databricks | DBFS | Unity Catalog | Databricks Jobs | `databricks` |
Switch with `STACK_CONTEXT=aws` or edit `stack.toml [context] active`.
## Databricks
Databricks Asset Bundle (`databricks.yml`) is auto-generated from the pipeline registry. {{ pipelines.count }} pipeline tasks with dependency ordering, 3 targets (dev/staging/prod), daily schedule.
```bash
uv run python dev/scripts/gen_config.py --dab-sql # generate SQL + DDL (~4 min)
databricks bundle deploy -t dev # deploy to Databricks
```
## Testing
### Architecture
Testing follows a semantic coverage model (`src/sem/`) rather than tracking raw line numbers. The system parses every `.py` file into an AST, builds stable semantic nodes (functions, branches, exception handlers, loops), and attaches three signal layers:
```
source → AST → semantic nodes → attach (ruff, ty, coverage) → planner
```
| Layer | Module | What it does |
|-------|--------|-------------|
| **Parse** | `sem.parse` | Walk AST, emit `SemanticNode` per function, branch, except, loop, return, raise |
| **Enrich** | `sem.enrich` | Run `ruff check` and `ty check`, map diagnostics to tightest-span node |
| **Runtime** | `sem.runtime` | Map `coverage.py` JSON report onto nodes (hit/miss, per-test contexts) |
| **Plan** | `sem.plan` | Score uncovered nodes by priority, suggest next test targets |
| **State** | `sem.state` | Persist node status across runs, reset on source hash change |
Each node has a stable identity derived from its module path, qualified symbol name, kind, and ordinal position — never from line numbers:
```
app.config::load_settings::branch_if[1]
aco.express.pharmacy::pharmacy_claims::except_handler[0]
```
#### Priority heuristic
```
priority =
uncovered_branch × 5
+ ty_diagnostic × 4
+ inside_partially_tested_function × 3
+ uncovered_exception_path × 3
+ ruff_warning × 2
prior_failures × 2
already_covered × 5
```
#### Structural invariants
`tests/test_ast_coverage.py` runs at collection time (pure AST, no imports) and enforces:
1. Every `@nw.narwhalify` function has a docstring
2. Every `Expr(...)` call supplies `name`, `fn`, `output`, and `after`
3. Every public Pydantic model is importable
4. Every `pipe/*.py` module exports a `Pipeline` with non-empty `.exprs`
5. Implementation ratio stays above baseline (ratchet)
6. `express/` and `pipe/` modules are symmetric
### Pre-commit hooks
Git hooks live in `dev/hooks/` (tracked) and are activated via:
```bash
git config core.hooksPath dev/hooks
```
The shell hook is a 3-line wrapper. All logic lives in `src/sem/hooks.py`:
```bash
#!/usr/bin/env bash
exec uv run python -m sem.hooks
```
#### Smart test selection
The hook classifies staged files and runs only what is relevant:
| What changed | Tests run | Why |
|-------------|-----------|-----|
| `src/sem/*.py` | `tests/sem/` + `test_ast_coverage.py` | Module tests + structural invariants |
| `src/aco/*.py` + `src/sem/*.py` | `tests/aco/` + `tests/sem/` + structural | Both module test dirs |
| `tests/bib/test_sync.py` | `tests/bib/` | Changed test dir |
| `pyproject.toml` or `conftest.py` | Full suite ({{ tests.formatted }}+ tests) | Infrastructure change |
| `notebooks/pfs_calcs.py` | `marimo check` + notebook execution | Notebook validation only |
| `README.md` only | Nothing | No testable changes |
The mapping rule is: `src/<module>/` changes → `tests/<module>/` runs. Any `src/` change also triggers `test_ast_coverage.py` to verify structural invariants haven't regressed.
#### Forcing full suite
```bash
GIT_PRE_COMMIT_FULL=1 git commit -m "message"
```
#### Hook steps (in order)
1. **Venv recovery** — if `uv run python -c 'import sys'` fails, run `uv sync --dev`
2. **Config regeneration** — if `stack.toml` or `gen_config.py` changed, regenerate CI workflows
3. **Ruff lint + format** — only staged `.py` files
4. **AST parse check** — verify staged source files have valid syntax
5. **Pytest** — targeted or full suite based on what changed
6. **Marimo check** — only if notebooks are staged
7. **Notebook execution** — only staged notebooks in the safe-to-run list
#### Post-commit
`dev/hooks/post-commit` rebuilds the docs Docker image in the background after every commit so the documentation site stays current with docstring changes.
### Running tests
```bash
uv run python -m pytest tests/ # full suite
uv run python -m pytest tests/sem/ # one module
uv run python -m pytest tests/ -m "not stub" # skip stub inventory
uv run python -m pytest tests/ -m stub # only stub status
uv run python -m pytest tests/test_ast_coverage.py # structural invariants only
```
## Project layout
```
stack/
├── compose.yml Docker Compose ({{ services.count }} services)
├── stack.toml Centralised configuration
├── pyproject.toml Python project (uv, optional deps per module)
├── src/
{% for m in src_modules %}
│ {{ "├" if not loop.last else "└" }}── {{ m.name }}/{{ " " * (25 - m.name|length) }}{{ m.desc }}
{% endfor %}
├── infra/ Service configs and Dockerfiles
│ ├── images/ All Dockerfiles (api, notebooks, zotero, docs, mc)
│ ├── traefik/ Reverse proxy + loch CSS injection
│ ├── grafana/ Dashboards and datasource provisioning
│ ├── prometheus/ Metrics collection
│ ├── loki/ Log aggregation
│ └── ... coredns, nginx, trino, polaris, rustfs, act-runner, gitea
├── assets/ Branding, styles, generated artifacts
│ ├── css/ Per-service CSS (dashboard, gitea, marimo, woodpecker)
│ ├── icons/ Favicons, logos, coverage badge
│ └── nature.py Nature-conformant chart palette
├── cloud/ Cloud provider deployment guides
│ ├── self-hosted/ Docker Compose (default)
│ ├── aws/ S3, RDS, Glue, ECS
│ ├── gcp/ GCS, Cloud SQL, BigQuery, Cloud Run
│ └── azure/ ABFS, Azure SQL, Unity Catalog, Container Apps
├── dev/ Dev tooling
│ ├── scripts/ Code generators, bootstrap, supply chain tools
│ ├── hooks/ Git hooks (tracked, core.hooksPath = dev/hooks)
│ ├── seeds/ Reference data (BCDA samples, CMS docs)
│ └── pipelines/ CI-agnostic pipeline specs
├── tests/ {{ tests.formatted }} tests at 99% coverage
├── notebooks/ Marimo notebooks
├── docs/ Docusaurus site
├── data/ DuckDB, bib.sqlite, BCDA/CMS data, zotero (gitignored)
└── .env Derived credentials (not in git)
```
## Prerequisites
- Docker (rootless mode)
- NVIDIA GPU with container toolkit
- Domain or `/etc/hosts` entries for `*.homelab.fhirworx.io`
```bash
# Fix for rootless NVIDIA containers
sudo sed -i 's/#no-cgroups = false/no-cgroups = true/' /etc/nvidia-container-runtime/config.toml
```
## SSH access
```
# ~/.ssh/config
Host gitea
HostName homelab.fhirworx.io
Port 2222
User git
IdentityFile ~/.ssh/gitea_ed25519
IdentitiesOnly yes
```

300
dev/scripts/gen_readme.py Normal file
View File

@@ -0,0 +1,300 @@
"""Generate README.md from README.md.j2 using live codebase data.
Collects counts, tables, and metadata from the actual codebase:
- Service count and names from compose.yml
- Module list from pyproject.toml optional-dependencies
- Pipeline registry from src/aco/pipe/
- Test count via pytest --collect-only
- Credential count from api.auth.manifest
- Network list from compose.yml
- Workflow list from .gitea/workflows/
Usage::
uv run python dev/scripts/gen_readme.py # generate README.md
uv run python dev/scripts/gen_readme.py --check # verify up-to-date
uv run python dev/scripts/gen_readme.py --dry-run # print to stdout
"""
from __future__ import annotations
import argparse
import ast
import subprocess
import sys
import tomllib
from pathlib import Path
import yaml
from jinja2 import Environment, FileSystemLoader
ROOT = Path(__file__).resolve().parent.parent.parent
# ── Data collectors ──────────────────────────────────────────────────
def _collect_services() -> dict:
"""Parse compose.yml for service and network info."""
compose = yaml.safe_load((ROOT / "compose.yml").read_text())
services = sorted(compose.get("services", {}).keys())
networks = sorted(compose.get("networks", {}).keys())
return {"names": services, "count": len(services), "networks": networks}
def _collect_modules() -> dict:
"""Parse pyproject.toml for module list."""
with open(ROOT / "pyproject.toml", "rb") as f:
pyp = tomllib.load(f)
opt_deps = pyp["project"]["optional-dependencies"]
cloud = {"aws", "gcp", "azure"}
aggregates = {"all", "lake"}
modules = sorted(k for k in opt_deps if k not in cloud and k not in aggregates)
build_modules = (
pyp.get("tool", {})
.get("uv", {})
.get("build-backend", {})
.get("module-name", [])
)
return {
"names": modules,
"count": len(modules),
"cloud": sorted(cloud & set(opt_deps)),
"aggregates": sorted(aggregates & set(opt_deps)),
"build_modules": build_modules,
}
def _collect_pipelines() -> dict:
"""Import pipeline registry and collect metadata."""
sys.path.insert(0, str(ROOT / "src"))
from aco.pipe import registry
# Fallback descriptions for pipelines without module docstrings
fallback_docs = {
"ahrq_measures": "AHRQ quality indicators",
"cclf": "CCLF file parsing",
"claims_preprocessing": "Claim matching, encounter IDs",
"cms_quality_measures": "CMS ACO quality metrics",
"core": "Encounters, patients, providers",
"data_quality": "Validation and completeness",
"hcc_suspecting": "HCC risk adjustment",
"input_layer": "Raw CCLF/BCDA ingestion",
"main": "Full pipeline orchestration",
"opps": "Outpatient payment system",
"pharmacy": "Part D drug utilization",
"provider_attribution": "Provider-patient assignment",
"quality_measures": "Composite quality scores",
"readmissions": "30-day readmission rates",
}
pipelines = []
for name in sorted(registry.keys()):
p = registry[name]
doc = ""
# Try to get the pipeline module's docstring
mod_path = ROOT / "src" / "aco" / "pipe" / f"{name}.py"
if mod_path.exists():
try:
tree = ast.parse(mod_path.read_text())
raw = ast.get_docstring(tree) or ""
doc = raw.split("\n")[0].rstrip(".")
except SyntaxError:
pass
# Use fallback if docstring is empty
if not doc:
doc = fallback_docs.get(name, "")
pipelines.append(
{
"name": name,
"steps": len(p.exprs),
"doc": doc,
}
)
total_steps = sum(p["steps"] for p in pipelines)
return {
"list": pipelines,
"count": len(pipelines),
"total_steps": total_steps,
}
def _collect_tests() -> dict:
"""Run pytest --collect-only to get test count."""
result = subprocess.run(
["uv", "run", "python", "-m", "pytest", "tests/", "--collect-only", "-q"],
capture_output=True,
text=True,
cwd=str(ROOT),
timeout=60,
)
# Last line: "12291 tests collected in 3.03s"
for line in result.stdout.strip().splitlines():
if "test" in line and "collected" in line:
count = int(line.split()[0])
return {"count": count, "formatted": f"{count:,}"}
return {"count": 0, "formatted": "?"}
def _collect_credentials() -> dict:
"""Count credentials from auth manifest."""
sys.path.insert(0, str(ROOT / "src"))
try:
from api.auth.manifest import CREDENTIALS
return {"count": len(CREDENTIALS)}
except ImportError:
return {"count": 0}
def _collect_workflows() -> dict:
"""List CI workflow files."""
workflow_dirs = [
ROOT / ".gitea" / "workflows",
ROOT / ".github" / "workflows",
ROOT / ".woodpecker",
]
workflows = []
for d in workflow_dirs:
if d.is_dir():
for f in sorted(d.glob("*.yml")):
workflows.append({"name": f.name, "path": str(f.relative_to(ROOT))})
return {"list": workflows, "count": len(workflows)}
def _collect_src_tree() -> list[dict]:
"""Build the src/ module listing with descriptions."""
descriptions = {
"aco": "ACO analytics (express, pipe, table, lake, load)",
"api": "FastAPI server + auth + diag",
"bcda": "BCDA FHIR R4 client",
"bib": "Zotero bibliography store, tags, metadata",
"bls": "BLS data",
"ccw": "CCW data dictionary",
"cli": "CLI entry point (typer)",
"cms": "CMS public data tables",
"conf": "Config loader, storage abstraction, table base",
"opps": "Outpatient Prospective Payment System",
"perf": "Pipeline telemetry (OpenTelemetry)",
"pfs": "Physician Fee Schedule",
"rex": "REX fixed-width file processing",
"sem": "Semantic coverage orchestration",
}
src = ROOT / "src"
modules = []
for d in sorted(src.iterdir()):
if (
d.is_dir()
and not d.name.startswith(("_", "."))
and not d.name.endswith(".egg-info")
):
modules.append(
{
"name": d.name,
"desc": descriptions.get(d.name, ""),
}
)
return modules
def collect_all() -> dict:
"""Collect all dynamic data for template rendering."""
services = _collect_services()
modules = _collect_modules()
pipelines = _collect_pipelines()
tests = _collect_tests()
credentials = _collect_credentials()
workflows = _collect_workflows()
src_modules = _collect_src_tree()
return {
"services": services,
"modules": modules,
"pipelines": pipelines,
"tests": tests,
"credentials": credentials,
"workflows": workflows,
"src_modules": src_modules,
}
# ── Rendering ────────────────────────────────────────────────────────
def render(data: dict) -> str:
"""Render README.md.j2 with collected data."""
env = Environment(
loader=FileSystemLoader(str(ROOT)),
keep_trailing_newline=True,
trim_blocks=True,
lstrip_blocks=True,
)
env.filters["backtick"] = lambda s: f"`{s}`"
template = env.get_template("README.md.j2")
return template.render(**data)
# ── CLI ──────────────────────────────────────────────────────────────
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--check",
action="store_true",
help="verify README.md matches template output (exit 1 if stale)",
)
parser.add_argument(
"--dry-run",
action="store_true",
help="print rendered output to stdout instead of writing",
)
args = parser.parse_args()
data = collect_all()
rendered = render(data)
if args.dry_run:
print(rendered, end="")
return 0
readme = ROOT / "README.md"
if args.check:
current = readme.read_text(encoding="utf-8")
if current == rendered:
print("README.md is up-to-date.")
return 0
else:
# Show first differing line
cur_lines = current.splitlines()
ren_lines = rendered.splitlines()
for i, (a, b) in enumerate(zip(cur_lines, ren_lines), 1):
if a != b:
print(f"README.md is stale (first diff at line {i}):")
print(f" current: {a[:100]}")
print(f" expected: {b[:100]}")
break
else:
len_diff = len(cur_lines) - len(ren_lines)
print(f"README.md is stale (length differs by {len_diff} lines)")
print("\nRun: uv run python dev/scripts/gen_readme.py")
return 1
readme.write_text(rendered, encoding="utf-8")
print(
f"README.md generated ({data['services']['count']} services, "
f"{data['pipelines']['count']} pipelines, "
f"{data['tests']['formatted']} tests, "
f"{data['modules']['count']} modules)"
)
return 0
if __name__ == "__main__":
sys.exit(main())

View File

@@ -139,6 +139,7 @@ dev = [
"pandas>=3.0.1", "pandas>=3.0.1",
"obstore>=0.9.2", "obstore>=0.9.2",
"s3fs>=2026.2.0", "s3fs>=2026.2.0",
"jinja2>=3.1.0",
] ]
[build-system] [build-system]

2
uv.lock generated
View File

@@ -3229,6 +3229,7 @@ dev = [
{ name = "coverage" }, { name = "coverage" },
{ name = "dbt-core" }, { name = "dbt-core" },
{ name = "dbt-duckdb" }, { name = "dbt-duckdb" },
{ name = "jinja2" },
{ name = "marimo" }, { name = "marimo" },
{ name = "obstore" }, { name = "obstore" },
{ name = "openpyxl" }, { name = "openpyxl" },
@@ -3326,6 +3327,7 @@ dev = [
{ name = "coverage", specifier = ">=7.13.4" }, { name = "coverage", specifier = ">=7.13.4" },
{ name = "dbt-core", specifier = "==1.10.15" }, { name = "dbt-core", specifier = "==1.10.15" },
{ name = "dbt-duckdb", specifier = ">=1.10,<1.11" }, { name = "dbt-duckdb", specifier = ">=1.10,<1.11" },
{ name = "jinja2", specifier = ">=3.1.0" },
{ name = "marimo", specifier = ">=0.20.0" }, { name = "marimo", specifier = ">=0.20.0" },
{ name = "obstore", specifier = ">=0.9.2" }, { name = "obstore", specifier = ">=0.9.2" },
{ name = "openpyxl", specifier = ">=3.1.5" }, { name = "openpyxl", specifier = ">=3.1.5" },