stack/README.md.j2

{# README.md.j2 — source of truth for README.md
   Render with: uv run python dev/scripts/gen_readme.py
   Check freshness: uv run python dev/scripts/gen_readme.py --check
#}
# Stack

![coverage](assets/icons/coverage.svg)

Healthcare analytics platform on self-hosted infrastructure. Replaces dbt SQL models with narwhals DataFrame-agnostic expression functions, backed by DuckDB locally and Iceberg/Trino in the lakehouse. {{ services.count }} services, {{ pipelines.count }} data pipelines, {{ tests.formatted }} tests at 99% coverage.

## Quick start

```bash
# 1. Generate root key and seed .env
ROOT_KEY=$(openssl rand -hex 32)
cat > .env <<EOF
DOMAIN=fhirworx.io
HOST_IP=192.168.1.192
EOF

# 2. Start core services and bootstrap credentials
docker compose up -d postgres rustfs traefik
ROOT_KEY=$ROOT_KEY uv run python -m api.auth bootstrap $(git rev-parse HEAD)

# 3. Start everything
docker compose up -d

# 4. Verify
curl -s http://localhost:8000/health | python3 -m json.tool
```

## Installation

The package supports skinny installs — install only the modules you need:

```bash
pip install stack[aco]         # ACO analytics only
pip install stack[bib]         # bibliography only
pip install stack[cli]         # CLI (pulls aco + api + bib)
pip install stack[all]         # everything
pip install stack[aco,aws]     # ACO analytics with AWS storage
```

{{ modules.count }} modules available as optional extras: {{ modules.names | map('backtick') | join(', ') }}. Cloud providers: {{ modules.cloud | map('backtick') | join(', ') }}. Aggregates: {{ modules.aggregates | map('backtick') | join(', ') }}.

## Services

All services route through Traefik at `*.fhirworx.io`.

| Service | URL | Purpose |
|---------|-----|---------|
| API | `api` (internal) | FastAPI — health, pipelines, bib endpoints |
| Notebooks | `notebooks.fhirworx.io` | Marimo notebooks (GPU-accelerated) |
| LLM Chat | `llm.fhirworx.io` | SSO-guarded RAG chat over CMS comments (grounded, cited) |
| Docs | `docs.fhirworx.io` | Docusaurus — API reference + CMS bibliography |
| Gitea | `git.fhirworx.io` | Git server, container registry, LFS |
| Zotero | `zotero.fhirworx.io` | Reference manager (KasmVNC desktop) |
| Dashboard | `fhirworx.io` | Service launcher |
| RustFS | `s3.fhirworx.io` | S3-compatible object storage |
| Nessie | `nessie.fhirworx.io` | Git-like Iceberg catalog |
| Polaris | `polaris.fhirworx.io` | Iceberg catalog with governance |
| Trino | `trino.fhirworx.io` | Distributed SQL engine |
| Grafana | `grafana.fhirworx.io` | Dashboards |
| Prometheus | `prometheus.fhirworx.io` | Metrics |
| Tempo | `tempo.fhirworx.io` | Distributed tracing |
| Loki | `loki.fhirworx.io` | Log aggregation |

## Analytics platform

### Architecture

```
  CCLF/BCDA/CMS data
        |
        v
  src/aco/express/     narwhals expressions (DataFrame-agnostic)
        |
        v
  src/aco/pipe/        pipeline runner with schema validation
        |
        v
  DuckDB (local)  --or--  Iceberg + Trino (lakehouse)  --or--  Databricks
        |
        v
  src/api/             FastAPI (health, runs, bib)
        |
        v
  notebooks/           Marimo interactive analysis
```

### Pipelines

{{ pipelines.count }} registered pipelines with {{ pipelines.total_steps }} steps, validated by `run_pipeline(steps, load)` which enforces output schemas via pandera-style column checks.

| Pipeline | Steps | Domain |
|----------|------:|--------|
{% for p in pipelines.list %}
| `{{ p.name }}` | {{ p.steps }} | {{ p.doc }} |
{% endfor %}

### Expression layer

`src/aco/express/` contains narwhals functions that work across polars, pandas, and cuDF. Each module mirrors a pipeline and exports pure transformation functions.

### Configuration

`stack.toml` is the single config file. `src/conf/` provides:

- `cfg` — attribute access (`cfg.db.aco`)
- `path("db.aco")` — absolute path resolution relative to repo root
- `context()` — environment switching (local/lake/databricks/trino/aws/gcp/azure)
- `secret("api.secret", "STACK_API_SECRET")` — env var with config fallback
- `storage.get_filesystem()` — multi-cloud storage abstraction (S3/GCS/ABFS/local)

Override context at runtime: `STACK_CONTEXT=lake` (or `aws`, `gcp`, `azure`).

### Bibliography

`src/bib/` manages a Zotero-backed citation store in `data/bib.sqlite`. Column-level provenance via `Tag.col(ref, desc)`. The docs site exports `library.json` for a searchable bibliography browser at `/library`.

## CI/CD

### Backend switching

CI/CD workflows are auto-generated from `stack.toml` by `gen_config.py`. Two backends are supported:

| Backend | Workflows directory | When to use |
|---------|---------------------|-------------|
| `gitea` | `.gitea/workflows/` | Self-hosted Gitea instance (default) |
| `github` | `.github/workflows/` | GitHub repos / GitHub Actions |

Switch backends by editing `stack.toml`:

```toml
[ci]
backend = "gitea"   # change to "github"
```

Then regenerate:

```bash
uv run python dev/scripts/gen_config.py          # generate new workflows
uv run python dev/scripts/gen_config.py --check   # verify (used in CI)
uv run python dev/scripts/gen_config.py --backend github  # one-off override
```

The generator reads image definitions from `stack.toml [images]`, dispatches to the active backend emitter (`dev/scripts/backends/{backend}.py`), writes workflow YAML, and auto-removes stale workflows from inactive backend directories.

### Workflows

{{ workflows.count }} workflows generated per backend:

| Workflow | Trigger | What it does |
|----------|---------|--------------|
| `ci.yml` | Push/PR | Ruff lint + format, pytest (99% coverage), gen_config --check, skinny-install matrix |
| `deploy.yml` | Push to main | Build 5 images, Trivy scan, vuln reporting |
| `harden.yml` | Weekly cron / manual | Rebuild `--no-cache`, scan, auto-close/file vuln issues |
| `rebuild-all.yml` | Manual | Full rebuild + scan + deploy |
| `infra-ci.yml` | Path-filtered push | Hadolint Dockerfiles, dry-run builds |
| `release.yml` | Tag (`v*`) | `uv build` + release |
| `pkg-supply-chain.yml` | Daily / push | Package inventory, mirror sync, drift detection, vuln scan |
| `renovate.yml` | Cron / manual | Renovate dependency-update PRs |

### Secret mapping

| Secret | Gitea | GitHub |
|--------|-------|--------|
| Registry auth | `REGISTRY_USER` + `REGISTRY_TOKEN` | `GITHUB_TOKEN` (built-in) |
| Gitea API | `GITEA_TOKEN` | N/A |
| Vuln reporting | `GITEA_TOKEN` | `GITHUB_TOKEN` |

### Image tagging

Images are tagged with the short commit SHA (8 chars). Compose resolves via `${COMMIT_SHA:-latest}` from `.env`. Custom images: `api`, `notebooks`, `zotero`, `docs`, `mc`.

### Vulnerability management

The `harden.yml` pipeline rebuilds all images with `--no-cache` to pick up OS patches, runs Trivy scans, and auto-files or auto-closes Gitea issues via `api.diag.vuln`.

### Package supply chain

Daily automated pipeline (`pkg-supply-chain.yml`) that:
1. Scans Dockerfiles, pyproject.toml, and CI workflows for all package dependencies
2. Syncs packages to the Gitea package registry (local mirror)
3. Detects drift between manifest and mirror
4. Runs Trivy vulnerability scans on dependencies
5. Auto-creates Gitea issues for missing packages and CVEs

## Credential management

All {{ credentials.count }} service credentials derive from a single 256-bit root key via HKDF-SHA256. No passwords stored in `.env` — they regenerate deterministically from root key + commit SHA on each deploy.

```
ROOT_KEY + commit_sha -> HKDF-SHA256 -> all credentials -> .env + backends
```

| Tier | Salt | Rotates |
|------|------|---------|
| Bootstrap | `b"bootstrap"` | Only when root key changes |
| Service | `commit_sha` | Every deploy |

```bash
# Bootstrap (first time)
ROOT_KEY=$KEY uv run python -m api.auth bootstrap $(git rev-parse HEAD)

# Rotate (every deploy, automatic in CI)
ROOT_KEY=$KEY uv run python -m api.auth provision $COMMIT_SHA

# Dry-run
ROOT_KEY=$KEY uv run python -m api.auth derive $COMMIT_SHA --redact
```

## Network segmentation

| Network | Type | Services |
|---------|------|----------|
| `gateway` | external | Traefik, all public-facing |
| `storage` | internal | PostgreSQL, RustFS |
| `data` | internal | Nessie, Trino, Polaris |
| `observability` | internal | Grafana, Prometheus, Tempo, Loki |
| `ci` | external | Gitea, Act Runner, RustFS |

## Design system

Theme: **fhirworx** (federal editorial). Traefik's `inject-fhirworx` middleware rewrites HTML to inject `fhirworx.css` and the favicon. Per-service CSS in `assets/css/`. Altair chart theme in `assets/fhirworx.py` (federal palette, Source Serif 4 / Playfair Display / JetBrains Mono).

## Data lakehouse

| Component | Purpose |
|-----------|---------|
| Nessie | Git-like branching and time travel for Iceberg tables |
| Polaris | Iceberg catalog with RBAC and multi-tenant governance |
| Trino | Distributed SQL (`iceberg` catalog) |
| RustFS | S3 storage backend (`s3://lakehouse/`) |

Query via Trino CLI, JDBC, Web UI, PyIceberg, or DuckDB+PyArrow.

### Cloud deployment

The platform runs on self-hosted Docker Compose by default. Cloud provider guides in `cloud/`:

| Provider | Storage | Catalog | Compute | Context |
|----------|---------|---------|---------|---------|
| Self-hosted | RustFS (S3) | Nessie / Polaris | Docker Compose | `local` / `lake` |
| AWS | S3 | Glue | ECS / EKS | `aws` |
| GCP | GCS | BigQuery | Cloud Run / GKE | `gcp` |
| Azure | ABFS | Unity Catalog | Container Apps / AKS | `azure` |
| Databricks | DBFS | Unity Catalog | Databricks Jobs | `databricks` |

Switch with `STACK_CONTEXT=aws` or edit `stack.toml [context] active`.

## Databricks

Databricks Asset Bundle (`databricks.yml`) is auto-generated from the pipeline registry. {{ pipelines.count }} pipeline tasks with dependency ordering, 3 targets (dev/staging/prod), daily schedule.

```bash
uv run python dev/scripts/gen_config.py --dab-sql   # generate SQL + DDL (~4 min)
databricks bundle deploy -t dev                       # deploy to Databricks
```

### SDK integration

The platform uses 18 Databricks SDK services via `UnityClient` (wrapping `WorkspaceClient`):

| Category | Services | Module | Purpose |
|----------|----------|--------|---------|
| **Catalog CRUD** | `catalogs`, `schemas`, `tables`, `volumes` | `lake/unity.py` | Unity Catalog object management |
| **Data loading** | `files`, `statement_execution` | `lake/sync.py` | Parquet upload + COPY INTO |
| **Governance** | `grants` | `lake/governance.py` | Declarative schema permissions with drift audit |
| **Secrets** | `secrets` | `lake/unity.py` | Scope management, env var sync |
| **Constraints** | `table_constraints` | `lake/unity.py` | PK/FK on claims tables for lineage |
| **Jobs** | `jobs` | `lake/jobs.py` | Create/run/monitor pipeline jobs from Python |
| **Compute** | `warehouses` | `lake/unity.py` | Find/create/start/stop warehouses by name |
| **Quality** | `quality_monitors` | `lake/quality.py` | Lakehouse Monitoring profiles |
| **Audit** | `system_schemas` | `lake/unity.py` | Table lineage and audit log queries |

### Governance

`src/aco/lake/governance.py` provides declarative grant management:

```python
from aco.lake.governance import GovernancePolicy, GrantRule, apply_governance

policy = GovernancePolicy(
    catalog="aco",
    rules=[
        GrantRule(group="analysts", schemas=["readmissions"], privileges=["SELECT"]),
        GrantRule(group="pipeline_svc", schemas=["*"], privileges=["ALL_PRIVILEGES"]),
    ],
    phi_schemas=["core", "claims_preprocessing", "cclf"],
)

# Preview changes
apply_governance(client, policy, dry_run=True)

# Detect drift between declared and actual grants
from aco.lake.governance import audit_governance
drifts = audit_governance(client, policy)
```

PHI schemas, group privilege mappings, and quality monitor config are declared in `stack.toml`:

```toml
[databricks.governance]
phi_schemas = ["core", "claims_preprocessing", "cclf", "input_layer"]

[databricks.governance.groups.analysts]
schemas = ["readmissions", "quality_measures"]
privileges = ["SELECT"]
```

### Job orchestration

`src/aco/lake/jobs.py` translates `Pipeline` objects into Databricks Jobs:

```python
from aco.lake.jobs import JobManager

mgr = JobManager(client, catalog="aco")
job_id = mgr.create_pipeline_job("readmissions", schedule="0 0 6 * * ?")
run_id = mgr.run_now(job_id)
status = mgr.get_run_status(run_id)
```

### Secret sync

Push environment variables to Databricks secret scopes per `stack.toml [databricks.secrets]`:

```bash
uv run python dev/scripts/sync_secrets.py --dry-run   # preview
uv run python dev/scripts/sync_secrets.py              # push to Databricks
uv run python dev/scripts/sync_secrets.py --list       # show current secrets
```

### Quality monitoring

```python
from aco.lake.quality import setup_monitors
setup_monitors(client, catalog="aco", schemas=["core", "claims_preprocessing"])
```

## Testing

### Architecture

Testing follows a semantic coverage model (`src/sem/`) rather than tracking raw line numbers. The system parses every `.py` file into an AST, builds stable semantic nodes (functions, branches, exception handlers, loops), and attaches three signal layers:

```
source → AST → semantic nodes → attach (ruff, ty, coverage) → planner
```

| Layer | Module | What it does |
|-------|--------|-------------|
| **Parse** | `sem.parse` | Walk AST, emit `SemanticNode` per function, branch, except, loop, return, raise |
| **Enrich** | `sem.enrich` | Run `ruff check` and `ty check`, map diagnostics to tightest-span node |
| **Runtime** | `sem.runtime` | Map `coverage.py` JSON report onto nodes (hit/miss, per-test contexts) |
| **Plan** | `sem.plan` | Score uncovered nodes by priority, suggest next test targets |
| **State** | `sem.state` | Persist node status across runs, reset on source hash change |

Each node has a stable identity derived from its module path, qualified symbol name, kind, and ordinal position — never from line numbers:

```
app.config::load_settings::branch_if[1]
aco.express.pharmacy::pharmacy_claims::except_handler[0]
```

#### Priority heuristic

```
priority =
    uncovered_branch × 5
  + ty_diagnostic × 4
  + inside_partially_tested_function × 3
  + uncovered_exception_path × 3
  + ruff_warning × 2
  − prior_failures × 2
  − already_covered × 5
```

#### Structural invariants

`tests/test_ast_coverage.py` runs at collection time (pure AST, no imports) and enforces:

1. Every `@nw.narwhalify` function has a docstring
2. Every `Expr(...)` call supplies `name`, `fn`, `output`, and `after`
3. Every public Pydantic model is importable
4. Every `pipe/*.py` module exports a `Pipeline` with non-empty `.exprs`
5. Implementation ratio stays above baseline (ratchet)
6. `express/` and `pipe/` modules are symmetric

### Pre-commit hooks

Git hooks live in `dev/hooks/` (tracked) and are activated via:

```bash
git config core.hooksPath dev/hooks
```

The shell hook is a 3-line wrapper. All logic lives in `src/sem/hooks.py`:

```bash
#!/usr/bin/env bash
exec uv run python -m sem.hooks
```

#### Smart test selection

The hook classifies staged files and runs only what is relevant:

| What changed | Tests run | Why |
|-------------|-----------|-----|
| `src/sem/*.py` | `tests/sem/` + `test_ast_coverage.py` | Module tests + structural invariants |
| `src/aco/*.py` + `src/sem/*.py` | `tests/aco/` + `tests/sem/` + structural | Both module test dirs |
| `tests/bib/test_sync.py` | `tests/bib/` | Changed test dir |
| `pyproject.toml` or `conftest.py` | Full suite ({{ tests.formatted }}+ tests) | Infrastructure change |
| `notebooks/pfs_calcs.py` | `marimo check` + notebook execution | Notebook validation only |
| `README.md` only | Nothing | No testable changes |

The mapping rule is: `src/<module>/` changes → `tests/<module>/` runs. Any `src/` change also triggers `test_ast_coverage.py` to verify structural invariants haven't regressed.

#### Forcing full suite

```bash
GIT_PRE_COMMIT_FULL=1 git commit -m "message"
```

#### Hook steps (in order)

1. **Venv recovery** — if `uv run python -c 'import sys'` fails, run `uv sync --dev`
2. **Config regeneration** — if `stack.toml` or `gen_config.py` changed, regenerate CI workflows
3. **Ruff lint + format** — only staged `.py` files
4. **AST parse check** — verify staged source files have valid syntax
5. **Pytest** — targeted or full suite based on what changed
6. **Marimo check** — only if notebooks are staged
7. **Notebook execution** — only staged notebooks in the safe-to-run list

#### Post-commit

`dev/hooks/post-commit` rebuilds the docs Docker image in the background after every commit so the documentation site stays current with docstring changes.

### Running tests

```bash
uv run python -m pytest tests/                    # full suite
uv run python -m pytest tests/sem/                 # one module
uv run python -m pytest tests/ -m "not stub"       # skip stub inventory
uv run python -m pytest tests/ -m stub             # only stub status
uv run python -m pytest tests/test_ast_coverage.py # structural invariants only
```

### README generation

`README.md` is generated from `README.md.j2` using live codebase data. Never edit `README.md` directly.

```bash
uv run python dev/scripts/gen_readme.py            # regenerate from template
uv run python dev/scripts/gen_readme.py --check     # verify freshness (CI gate)
uv run python dev/scripts/gen_readme.py --dry-run   # preview to stdout
```

Dynamic values pulled from: `compose.yml` (services), `pyproject.toml` (modules), pipeline registry (pipelines + steps), `pytest --collect-only` (test count), `api.auth.manifest` (credentials), `.gitea/workflows/` (workflow count).

## Project layout

```
stack/
├── compose.yml                  Docker Compose ({{ services.count }} services)
├── stack.toml                   Centralised configuration
├── pyproject.toml               Python project (uv, optional deps per module)
├── src/
{% for m in src_modules %}
│   {{ "├" if not loop.last else "└" }}── {{ m.name }}/{{ " " * (25 - m.name|length) }}{{ m.desc }}
{% endfor %}
├── infra/                       Service configs and Dockerfiles
│   ├── images/                  All Dockerfiles (api, notebooks, zotero, docs, mc)
│   ├── traefik/                 Reverse proxy + fhirworx CSS injection
│   ├── grafana/                 Dashboards and datasource provisioning
│   ├── prometheus/              Metrics collection
│   ├── loki/                    Log aggregation
│   └── ...                      coredns, nginx, trino, polaris, rustfs, act-runner, gitea
├── assets/                      Branding, styles, generated artifacts
│   ├── css/                     Per-service CSS (dashboard, grafana, marimo)
│   ├── icons/                   Favicons, logos, coverage badge
│   └── fhirworx.py              fhirworx chart palette and Altair theme
├── cloud/                       Cloud provider deployment guides
│   ├── self-hosted/             Docker Compose (default)
│   ├── aws/                     S3, RDS, Glue, ECS
│   ├── gcp/                     GCS, Cloud SQL, BigQuery, Cloud Run
│   └── azure/                   ABFS, Azure SQL, Unity Catalog, Container Apps
├── dev/                         Dev tooling
│   ├── scripts/                 Code generators, bootstrap, supply chain tools
│   ├── hooks/                   Git hooks (tracked, core.hooksPath = dev/hooks)
│   ├── seeds/                   Reference data (BCDA samples, CMS docs)
│   └── pipelines/               CI-agnostic pipeline specs
├── tests/                       {{ tests.formatted }} tests at 99% coverage
├── notebooks/                   Marimo notebooks
├── docs/                        Docusaurus site
├── data/                        DuckDB, bib.sqlite, BCDA/CMS data, zotero (gitignored)
└── .env                         Derived credentials (not in git)
```

## Prerequisites

- Docker (rootless mode)
- NVIDIA GPU with container toolkit
- Domain or `/etc/hosts` entries for `*.fhirworx.io`

```bash
# Fix for rootless NVIDIA containers
sudo sed -i 's/#no-cgroups = false/no-cgroups = true/' /etc/nvidia-container-runtime/config.toml
```

## SSH access

```
# ~/.ssh/config
Host gitea
    HostName fhirworx.io
    Port 2222
    User git
    IdentityFile ~/.ssh/gitea_ed25519
    IdentitiesOnly yes
```