Stack

Healthcare analytics platform on self-hosted infrastructure. Replaces dbt SQL models with narwhals DataFrame-agnostic expression functions, backed by DuckDB locally and Iceberg/Trino in the lakehouse. 22 services, 13 data pipelines, 11,955 tests at 99% coverage.

Quick start

# 1. Generate root key and seed .env
ROOT_KEY=$(openssl rand -hex 32)
cat > .env <<EOF
DOMAIN=homelab.fhirworx.io
HOST_IP=192.168.1.192
EOF

# 2. Start core services and bootstrap credentials
docker compose up -d postgres rustfs traefik
ROOT_KEY=$ROOT_KEY uv run python -m api.auth bootstrap $(git rev-parse HEAD)

# 3. Start everything
docker compose up -d

# 4. Verify
curl -s http://localhost:8000/health | python3 -m json.tool

Installation

The package supports skinny installs — install only the modules you need:

pip install stack[aco]         # ACO analytics only
pip install stack[bib]         # bibliography only
pip install stack[cli]         # CLI (pulls aco + api + bib)
pip install stack[all]         # everything
pip install stack[aco,aws]     # ACO analytics with AWS storage

11 modules available as optional extras: conf, aco, api, bcda, bib, bls, ccw, cli, cms, pfs, rex. Cloud providers: aws, gcp, azure. Aggregates: all, lake.

Services

All services route through Traefik at *.homelab.fhirworx.io.

Service	URL	Purpose
API	api (internal)	FastAPI — health, pipelines, bib endpoints
Notebooks	notebooks.homelab.fhirworx.io	Marimo notebooks (GPU-accelerated)
Docs	docs.homelab.fhirworx.io	Docusaurus — API reference + CMS bibliography
Gitea	gitea.homelab.fhirworx.io	Git server, container registry, LFS
Zotero	zotero.homelab.fhirworx.io	Reference manager (KasmVNC desktop)
Dashboard	homelab.fhirworx.io	Service launcher
RustFS	s3.homelab.fhirworx.io	S3-compatible object storage
Nessie	nessie.homelab.fhirworx.io	Git-like Iceberg catalog
Polaris	polaris.homelab.fhirworx.io	Iceberg catalog with governance
Trino	trino.homelab.fhirworx.io	Distributed SQL engine
Grafana	grafana.homelab.fhirworx.io	Dashboards
Prometheus	prometheus.homelab.fhirworx.io	Metrics
Jaeger	jaeger.homelab.fhirworx.io	Distributed tracing
Loki	loki.homelab.fhirworx.io	Log aggregation

Analytics platform

Architecture

  CCLF/BCDA/CMS data
        |
        v
  src/aco/express/     narwhals expressions (DataFrame-agnostic)
        |
        v
  src/aco/pipe/        pipeline runner with schema validation
        |
        v
  DuckDB (local)  --or--  Iceberg + Trino (lakehouse)  --or--  Databricks
        |
        v
  src/api/             FastAPI (health, runs, bib)
        |
        v
  notebooks/           Marimo interactive analysis

Pipelines

13 registered pipelines with 189 steps, validated by run_pipeline(steps, load) which enforces output schemas via pandera-style column checks.

Pipeline	Domain
input_layer	Raw CCLF/BCDA ingestion
core	Encounters, patients, providers
claims_preprocessing	Claim matching, encounter IDs
readmissions	30-day readmission rates
pharmacy	Part D drug utilization
ahrq_measures	AHRQ quality indicators
cms_quality_measures	CMS ACO quality metrics
hcc_suspecting	HCC risk adjustment
provider_attribution	Provider-patient assignment
quality_measures	Composite quality scores
data_quality	Validation and completeness
cclf	CCLF file parsing
main	Full pipeline orchestration

Expression layer

src/aco/express/ contains narwhals functions that work across polars, pandas, and cuDF. Each module mirrors a pipeline and exports pure transformation functions.

Configuration

stack.toml is the single config file. src/conf/ provides:

• cfg — attribute access (cfg.db.aco)
• path("db.aco") — absolute path resolution relative to repo root
• context() — environment switching (local/lake/databricks/trino/aws/gcp/azure)
• secret("api.secret", "STACK_API_SECRET") — env var with config fallback
• storage.get_filesystem() — multi-cloud storage abstraction (S3/GCS/ABFS/local)

Override context at runtime: STACK_CONTEXT=lake (or aws, gcp, azure).

Bibliography

src/bib/ manages a Zotero-backed citation store in data/bib.sqlite. Column-level provenance via Tag.col(ref, desc). The docs site exports library.json for a searchable bibliography browser at /library.

CI/CD

Backend switching

CI/CD workflows are auto-generated from stack.toml by gen_config.py. Three backends are supported:

Backend	Workflows directory	When to use
gitea	.gitea/workflows/	Self-hosted Gitea instance (default)
github	.github/workflows/	GitHub repos / GitHub Actions
woodpecker	.woodpecker/	Woodpecker CI server

Switch backends by editing stack.toml:

[ci]
backend = "gitea"   # change to "github" or "woodpecker"

Then regenerate:

uv run python dev/scripts/gen_config.py          # generate new workflows
uv run python dev/scripts/gen_config.py --check   # verify (used in CI)
uv run python dev/scripts/gen_config.py --backend github  # one-off override

The generator reads image definitions from stack.toml [images], dispatches to the active backend emitter (dev/scripts/backends/{backend}.py), writes workflow YAML, and auto-removes stale workflows from inactive backend directories.

Workflows

Seven workflows generated per backend:

Workflow	Trigger	What it does
ci.yml	Push/PR	Ruff lint + format, pytest (99% coverage), gen_config --check, skinny-install matrix
deploy.yml	Push to main	Build 5 images, Trivy scan, vuln reporting
harden.yml	Weekly cron / manual	Rebuild --no-cache, scan, auto-close/file vuln issues
rebuild-all.yml	Manual	Full rebuild + scan + deploy
infra-ci.yml	Path-filtered push	Hadolint Dockerfiles, dry-run builds
release.yml	Tag (v*)	uv build + release
pkg-supply-chain.yml	Daily / push	Package inventory, mirror sync, drift detection, vuln scan

Secret mapping

Secret	Gitea	GitHub	Woodpecker
Registry auth	REGISTRY_USER + REGISTRY_TOKEN	GITHUB_TOKEN (built-in)	registry_username + registry_password
Gitea API	GITEA_TOKEN	N/A	gitea_token
Vuln reporting	GITEA_TOKEN	GITHUB_TOKEN	gitea_token

Image tagging

Images are tagged with the short commit SHA (8 chars). Compose resolves via ${COMMIT_SHA:-latest} from .env. Custom images: api, notebooks, zotero, docs, mc.

Vulnerability management

The harden.yml pipeline rebuilds all images with --no-cache to pick up OS patches, runs Trivy scans, and auto-files or auto-closes Gitea issues via api.diag.vuln.

Package supply chain

Daily automated pipeline (pkg-supply-chain.yml) that:

1. Scans Dockerfiles, pyproject.toml, and CI workflows for all package dependencies
2. Syncs packages to the Gitea package registry (local mirror)
3. Detects drift between manifest and mirror
4. Runs Trivy vulnerability scans on dependencies
5. Auto-creates Gitea issues for missing packages and CVEs

Credential management

All 19 service credentials derive from a single 256-bit root key via HKDF-SHA256. No passwords stored in .env — they regenerate deterministically from root key + commit SHA on each deploy.

ROOT_KEY + commit_sha -> HKDF-SHA256 -> all credentials -> .env + backends

Tier	Salt	Rotates
Bootstrap	b"bootstrap"	Only when root key changes
Service	commit_sha	Every deploy

# Bootstrap (first time)
ROOT_KEY=$KEY uv run python -m api.auth bootstrap $(git rev-parse HEAD)

# Rotate (every deploy, automatic in CI)
ROOT_KEY=$KEY uv run python -m api.auth provision $COMMIT_SHA

# Dry-run
ROOT_KEY=$KEY uv run python -m api.auth derive $COMMIT_SHA --redact

Network segmentation

Network	Type	Services
gateway	external	Traefik, all public-facing
storage	internal	PostgreSQL, RustFS
data	internal	Nessie, Trino, Polaris
observability	internal	Grafana, Prometheus, Jaeger, Loki
ci	external	Gitea, Act Runner, RustFS

Design system

Theme: loch (deep-navy). Traefik's inject-loch middleware rewrites HTML to inject loch.css and the favicon. Per-service CSS in assets/css/. Altair chart theme in assets/nature.py (Nature-conformant, accessible palette, IBM Plex Mono headings).

Data lakehouse

Component	Purpose
Nessie	Git-like branching and time travel for Iceberg tables
Polaris	Iceberg catalog with RBAC and multi-tenant governance
Trino	Distributed SQL (iceberg catalog)
RustFS	S3 storage backend (s3://lakehouse/)

Query via Trino CLI, JDBC, Web UI, PyIceberg, or DuckDB+PyArrow.

Cloud deployment

The platform runs on self-hosted Docker Compose by default. Cloud provider guides in cloud/:

Provider	Storage	Catalog	Compute	Context
Self-hosted	RustFS (S3)	Nessie / Polaris	Docker Compose	local / lake
AWS	S3	Glue	ECS / EKS	aws
GCP	GCS	BigQuery	Cloud Run / GKE	gcp
Azure	ABFS	Unity Catalog	Container Apps / AKS	azure
Databricks	DBFS	Unity Catalog	Databricks Jobs	databricks

Switch with STACK_CONTEXT=aws or edit stack.toml [context] active.

Databricks

Databricks Asset Bundle (databricks.yml) is auto-generated from the pipeline registry. 13 pipeline tasks with dependency ordering, 3 targets (dev/staging/prod), daily schedule.

uv run python dev/scripts/gen_config.py --dab-sql   # generate SQL + DDL (~4 min)
databricks bundle deploy -t dev                       # deploy to Databricks

Project layout

stack/
├── compose.yml                  Docker Compose (22 services)
├── stack.toml                   Centralised configuration
├── pyproject.toml               Python project (uv, optional deps per module)
├── src/
│   ├── aco/                     ACO analytics (express, pipe, table, lake, load)
│   ├── api/                     FastAPI server + auth + diag
│   ├── bcda/                    BCDA FHIR R4 client
│   ├── bib/                     Zotero bibliography store, tags, metadata
│   ├── bls/                     BLS data
│   ├── ccw/                     CCW data dictionary
│   ├── cli/                     CLI entry point (typer)
│   ├── cms/                     CMS public data tables
│   ├── conf/                    Config loader, storage abstraction, table base
│   ├── pfs/                     Physician Fee Schedule
│   └── rex/                     REX fixed-width file processing
├── infra/                       Service configs and Dockerfiles
│   ├── images/                  All Dockerfiles (api, notebooks, zotero, docs, mc)
│   ├── traefik/                 Reverse proxy + loch CSS injection
│   ├── grafana/                 Dashboards and datasource provisioning
│   ├── prometheus/              Metrics collection
│   ├── loki/                    Log aggregation
│   └── ...                      coredns, nginx, trino, polaris, rustfs, act-runner, gitea
├── assets/                      Branding, styles, generated artifacts
│   ├── css/                     Per-service CSS (dashboard, gitea, marimo, woodpecker)
│   ├── icons/                   Favicons, logos, coverage badge
│   └── nature.py                Nature-conformant chart palette
├── cloud/                       Cloud provider deployment guides
│   ├── self-hosted/             Docker Compose (default)
│   ├── aws/                     S3, RDS, Glue, ECS
│   ├── gcp/                     GCS, Cloud SQL, BigQuery, Cloud Run
│   └── azure/                   ABFS, Azure SQL, Unity Catalog, Container Apps
├── dev/                         Dev tooling
│   ├── scripts/                 Code generators, bootstrap, supply chain tools
│   ├── hooks/                   Git pre-commit hook
│   ├── seeds/                   Reference data (BCDA samples, CMS docs)
│   └── pipelines/               CI-agnostic pipeline specs
├── tests/                       11,955 tests at 99% coverage
├── notebooks/                   Marimo notebooks
├── docs/                        Docusaurus site
├── data/                        DuckDB, bib.sqlite, BCDA/CMS data, zotero (gitignored)
└── .env                         Derived credentials (not in git)

Prerequisites

• Docker (rootless mode)
• NVIDIA GPU with container toolkit
• Domain or /etc/hosts entries for *.homelab.fhirworx.io

# Fix for rootless NVIDIA containers
sudo sed -i 's/#no-cgroups = false/no-cgroups = true/' /etc/nvidia-container-runtime/config.toml

SSH access

# ~/.ssh/config
Host gitea
    HostName homelab.fhirworx.io
    Port 2222
    User git
    IdentityFile ~/.ssh/gitea_ed25519
    IdentitiesOnly yes