stack/README.md

Stack

Healthcare analytics platform on self-hosted infrastructure. Replaces dbt SQL models with narwhals DataFrame-agnostic expression functions, backed by DuckDB locally and Iceberg/Trino in the lakehouse. 32 services, 14 data pipelines, 14,010 tests at 99% coverage.

Quick start

# 1. Generate root key and seed .env
ROOT_KEY=$(openssl rand -hex 32)
cat > .env <<EOF
DOMAIN=fhirworx.io
HOST_IP=192.168.1.192
EOF

# 2. Start core services and bootstrap credentials
docker compose up -d postgres rustfs traefik
ROOT_KEY=$ROOT_KEY uv run python -m api.auth bootstrap $(git rev-parse HEAD)

# 3. Start everything
docker compose up -d

# 4. Verify
curl -s http://localhost:8000/health | python3 -m json.tool

Installation

The package supports skinny installs — install only the modules you need:

pip install stack[aco]         # ACO analytics only
pip install stack[bib]         # bibliography only
pip install stack[cli]         # CLI (pulls aco + api + bib)
pip install stack[all]         # everything
pip install stack[aco,aws]     # ACO analytics with AWS storage

18 modules available as optional extras: aco, api, bcda, bib, bls, ccw, cli, cms, conf, llm, mail, opps, perf, pfs, prisma, rec, rex, sem. Cloud providers: aws, azure, gcp. Aggregates: all, lake.

Services

All services route through Traefik at *.fhirworx.io.

Service	URL	Purpose
API	api (internal)	FastAPI — health, pipelines, bib endpoints
Notebooks	notebooks.fhirworx.io	Marimo notebooks (GPU-accelerated)
LLM Chat	llm.fhirworx.io	SSO-guarded RAG chat over CMS comments (grounded, cited)
Docs	docs.fhirworx.io	Docusaurus — API reference + CMS bibliography
Gitea	git.fhirworx.io	Git server, container registry, LFS
Zotero	zotero.fhirworx.io	Reference manager (KasmVNC desktop)
Dashboard	fhirworx.io	Service launcher
RustFS	s3.fhirworx.io	S3-compatible object storage
Nessie	nessie.fhirworx.io	Git-like Iceberg catalog
Polaris	polaris.fhirworx.io	Iceberg catalog with governance
Trino	trino.fhirworx.io	Distributed SQL engine
Grafana	grafana.fhirworx.io	Dashboards
Prometheus	prometheus.fhirworx.io	Metrics
Tempo	tempo.fhirworx.io	Distributed tracing
Loki	loki.fhirworx.io	Log aggregation

Analytics platform

Architecture

  CCLF/BCDA/CMS data
        |
        v
  src/aco/express/     narwhals expressions (DataFrame-agnostic)
        |
        v
  src/aco/pipe/        pipeline runner with schema validation
        |
        v
  DuckDB (local)  --or--  Iceberg + Trino (lakehouse)  --or--  Databricks
        |
        v
  src/api/             FastAPI (health, runs, bib)
        |
        v
  notebooks/           Marimo interactive analysis

Pipelines

14 registered pipelines with 230 steps, validated by run_pipeline(steps, load) which enforces output schemas via pandera-style column checks.

Pipeline	Steps	Domain
ahrq_measures	39	AHRQ quality indicators
cclf	19	CCLF connector pipeline -- raw CCLF files -> input_layer tables
claims_preprocessing	63	Claim matching, encounter IDs
cms_quality_measures	20	CMS ACO quality metrics
core	12	Encounters, patients, providers
data_quality	1	Validation and completeness
hcc_suspecting	6	HCC risk adjustment
input_layer	13	Raw CCLF/BCDA ingestion
main	14	Full pipeline orchestration
opps	2	Outpatient payment system
pharmacy	4	Part D drug utilization
provider_attribution	10	Provider-patient assignment
quality_measures	15	Composite quality scores
readmissions	12	30-day readmission rates

Expression layer

src/aco/express/ contains narwhals functions that work across polars, pandas, and cuDF. Each module mirrors a pipeline and exports pure transformation functions.

Configuration

stack.toml is the single config file. src/conf/ provides:

• cfg — attribute access (cfg.db.aco)
• path("db.aco") — absolute path resolution relative to repo root
• context() — environment switching (local/lake/databricks/trino/aws/gcp/azure)
• secret("api.secret", "STACK_API_SECRET") — env var with config fallback
• storage.get_filesystem() — multi-cloud storage abstraction (S3/GCS/ABFS/local)

Override context at runtime: STACK_CONTEXT=lake (or aws, gcp, azure).

Bibliography

src/bib/ manages a Zotero-backed citation store in data/bib.sqlite. Column-level provenance via Tag.col(ref, desc). The docs site exports library.json for a searchable bibliography browser at /library.

CI/CD

Backend switching

CI/CD workflows are auto-generated from stack.toml by gen_config.py. Two backends are supported:

Backend	Workflows directory	When to use
gitea	.gitea/workflows/	Self-hosted Gitea instance (default)
github	.github/workflows/	GitHub repos / GitHub Actions

Switch backends by editing stack.toml:

[ci]
backend = "gitea"   # change to "github"

Then regenerate:

uv run python dev/scripts/gen_config.py          # generate new workflows
uv run python dev/scripts/gen_config.py --check   # verify (used in CI)
uv run python dev/scripts/gen_config.py --backend github  # one-off override

The generator reads image definitions from stack.toml [images], dispatches to the active backend emitter (dev/scripts/backends/{backend}.py), writes workflow YAML, and auto-removes stale workflows from inactive backend directories.

Workflows

10 workflows generated per backend:

Workflow	Trigger	What it does
ci.yml	Push/PR	Ruff lint + format, pytest (99% coverage), gen_config --check, skinny-install matrix
deploy.yml	Push to main	Build 5 images, Trivy scan, vuln reporting
harden.yml	Weekly cron / manual	Rebuild --no-cache, scan, auto-close/file vuln issues
rebuild-all.yml	Manual	Full rebuild + scan + deploy
infra-ci.yml	Path-filtered push	Hadolint Dockerfiles, dry-run builds
release.yml	Tag (v*)	uv build + release
pkg-supply-chain.yml	Daily / push	Package inventory, mirror sync, drift detection, vuln scan
renovate.yml	Cron / manual	Renovate dependency-update PRs

Secret mapping

Secret	Gitea	GitHub
Registry auth	REGISTRY_USER + REGISTRY_TOKEN	GITHUB_TOKEN (built-in)
Gitea API	GITEA_TOKEN	N/A
Vuln reporting	GITEA_TOKEN	GITHUB_TOKEN

Image tagging

Images are tagged with the short commit SHA (8 chars). Compose resolves via ${COMMIT_SHA:-latest} from .env. Custom images: api, notebooks, zotero, docs, mc.

Vulnerability management

The harden.yml pipeline rebuilds all images with --no-cache to pick up OS patches, runs Trivy scans, and auto-files or auto-closes Gitea issues via api.diag.vuln.

Package supply chain

Daily automated pipeline (pkg-supply-chain.yml) that:

1. Scans Dockerfiles, pyproject.toml, and CI workflows for all package dependencies
2. Syncs packages to the Gitea package registry (local mirror)
3. Detects drift between manifest and mirror
4. Runs Trivy vulnerability scans on dependencies
5. Auto-creates Gitea issues for missing packages and CVEs

Credential management

All 17 service credentials derive from a single 256-bit root key via HKDF-SHA256. No passwords stored in .env — they regenerate deterministically from root key + commit SHA on each deploy.

ROOT_KEY + commit_sha -> HKDF-SHA256 -> all credentials -> .env + backends

Tier	Salt	Rotates
Bootstrap	b"bootstrap"	Only when root key changes
Service	commit_sha	Every deploy

# Bootstrap (first time)
ROOT_KEY=$KEY uv run python -m api.auth bootstrap $(git rev-parse HEAD)

# Rotate (every deploy, automatic in CI)
ROOT_KEY=$KEY uv run python -m api.auth provision $COMMIT_SHA

# Dry-run
ROOT_KEY=$KEY uv run python -m api.auth derive $COMMIT_SHA --redact

Network segmentation

Network	Type	Services
gateway	external	Traefik, all public-facing
storage	internal	PostgreSQL, RustFS
data	internal	Nessie, Trino, Polaris
observability	internal	Grafana, Prometheus, Tempo, Loki
ci	external	Gitea, Act Runner, RustFS

Design system

Theme: fhirworx (federal editorial). Traefik's inject-fhirworx middleware rewrites HTML to inject fhirworx.css and the favicon. Per-service CSS in assets/css/. Altair chart theme in assets/fhirworx.py (federal palette, Source Serif 4 / Playfair Display / JetBrains Mono).

Data lakehouse

Component	Purpose
Nessie	Git-like branching and time travel for Iceberg tables
Polaris	Iceberg catalog with RBAC and multi-tenant governance
Trino	Distributed SQL (iceberg catalog)
RustFS	S3 storage backend (s3://lakehouse/)

Query via Trino CLI, JDBC, Web UI, PyIceberg, or DuckDB+PyArrow.

Cloud deployment

The platform runs on self-hosted Docker Compose by default. Cloud provider guides in cloud/:

Provider	Storage	Catalog	Compute	Context
Self-hosted	RustFS (S3)	Nessie / Polaris	Docker Compose	local / lake
AWS	S3	Glue	ECS / EKS	aws
GCP	GCS	BigQuery	Cloud Run / GKE	gcp
Azure	ABFS	Unity Catalog	Container Apps / AKS	azure
Databricks	DBFS	Unity Catalog	Databricks Jobs	databricks

Switch with STACK_CONTEXT=aws or edit stack.toml [context] active.

Databricks

Databricks Asset Bundle (databricks.yml) is auto-generated from the pipeline registry. 14 pipeline tasks with dependency ordering, 3 targets (dev/staging/prod), daily schedule.

uv run python dev/scripts/gen_config.py --dab-sql   # generate SQL + DDL (~4 min)
databricks bundle deploy -t dev                       # deploy to Databricks

SDK integration

The platform uses 18 Databricks SDK services via UnityClient (wrapping WorkspaceClient):

Category	Services	Module	Purpose
Catalog CRUD	catalogs, schemas, tables, volumes	lake/unity.py	Unity Catalog object management
Data loading	files, statement_execution	lake/sync.py	Parquet upload + COPY INTO
Governance	grants	lake/governance.py	Declarative schema permissions with drift audit
Secrets	secrets	lake/unity.py	Scope management, env var sync
Constraints	table_constraints	lake/unity.py	PK/FK on claims tables for lineage
Jobs	jobs	lake/jobs.py	Create/run/monitor pipeline jobs from Python
Compute	warehouses	lake/unity.py	Find/create/start/stop warehouses by name
Quality	quality_monitors	lake/quality.py	Lakehouse Monitoring profiles
Audit	system_schemas	lake/unity.py	Table lineage and audit log queries

Governance

src/aco/lake/governance.py provides declarative grant management:

from aco.lake.governance import GovernancePolicy, GrantRule, apply_governance

policy = GovernancePolicy(
    catalog="aco",
    rules=[
        GrantRule(group="analysts", schemas=["readmissions"], privileges=["SELECT"]),
        GrantRule(group="pipeline_svc", schemas=["*"], privileges=["ALL_PRIVILEGES"]),
    ],
    phi_schemas=["core", "claims_preprocessing", "cclf"],
)

# Preview changes
apply_governance(client, policy, dry_run=True)

# Detect drift between declared and actual grants
from aco.lake.governance import audit_governance
drifts = audit_governance(client, policy)

PHI schemas, group privilege mappings, and quality monitor config are declared in stack.toml:

[databricks.governance]
phi_schemas = ["core", "claims_preprocessing", "cclf", "input_layer"]

[databricks.governance.groups.analysts]
schemas = ["readmissions", "quality_measures"]
privileges = ["SELECT"]

Job orchestration

src/aco/lake/jobs.py translates Pipeline objects into Databricks Jobs:

from aco.lake.jobs import JobManager

mgr = JobManager(client, catalog="aco")
job_id = mgr.create_pipeline_job("readmissions", schedule="0 0 6 * * ?")
run_id = mgr.run_now(job_id)
status = mgr.get_run_status(run_id)

Secret sync

Push environment variables to Databricks secret scopes per stack.toml [databricks.secrets]:

uv run python dev/scripts/sync_secrets.py --dry-run   # preview
uv run python dev/scripts/sync_secrets.py              # push to Databricks
uv run python dev/scripts/sync_secrets.py --list       # show current secrets

Quality monitoring

from aco.lake.quality import setup_monitors
setup_monitors(client, catalog="aco", schemas=["core", "claims_preprocessing"])

Testing

Architecture

Testing follows a semantic coverage model (src/sem/) rather than tracking raw line numbers. The system parses every .py file into an AST, builds stable semantic nodes (functions, branches, exception handlers, loops), and attaches three signal layers:

source → AST → semantic nodes → attach (ruff, ty, coverage) → planner

Layer	Module	What it does
Parse	sem.parse	Walk AST, emit SemanticNode per function, branch, except, loop, return, raise
Enrich	sem.enrich	Run ruff check and ty check, map diagnostics to tightest-span node
Runtime	sem.runtime	Map coverage.py JSON report onto nodes (hit/miss, per-test contexts)
Plan	sem.plan	Score uncovered nodes by priority, suggest next test targets
State	sem.state	Persist node status across runs, reset on source hash change

Each node has a stable identity derived from its module path, qualified symbol name, kind, and ordinal position — never from line numbers:

app.config::load_settings::branch_if[1]
aco.express.pharmacy::pharmacy_claims::except_handler[0]

Priority heuristic

priority =
    uncovered_branch × 5
  + ty_diagnostic × 4
  + inside_partially_tested_function × 3
  + uncovered_exception_path × 3
  + ruff_warning × 2
  − prior_failures × 2
  − already_covered × 5

Structural invariants

tests/test_ast_coverage.py runs at collection time (pure AST, no imports) and enforces:

1. Every @nw.narwhalify function has a docstring
2. Every Expr(...) call supplies name, fn, output, and after
3. Every public Pydantic model is importable
4. Every pipe/*.py module exports a Pipeline with non-empty .exprs
5. Implementation ratio stays above baseline (ratchet)
6. express/ and pipe/ modules are symmetric

Pre-commit hooks

Git hooks live in dev/hooks/ (tracked) and are activated via:

git config core.hooksPath dev/hooks

The shell hook is a 3-line wrapper. All logic lives in src/sem/hooks.py:

#!/usr/bin/env bash
exec uv run python -m sem.hooks

Smart test selection

The hook classifies staged files and runs only what is relevant:

What changed	Tests run	Why
src/sem/*.py	tests/sem/ + test_ast_coverage.py	Module tests + structural invariants
src/aco/*.py + src/sem/*.py	tests/aco/ + tests/sem/ + structural	Both module test dirs
tests/bib/test_sync.py	tests/bib/	Changed test dir
pyproject.toml or conftest.py	Full suite (14,010+ tests)	Infrastructure change
notebooks/pfs_calcs.py	marimo check + notebook execution	Notebook validation only
README.md only	Nothing	No testable changes

The mapping rule is: src/<module>/ changes → tests/<module>/ runs. Any src/ change also triggers test_ast_coverage.py to verify structural invariants haven't regressed.

Forcing full suite

GIT_PRE_COMMIT_FULL=1 git commit -m "message"

Hook steps (in order)

1. Venv recovery — if uv run python -c 'import sys' fails, run uv sync --dev
2. Config regeneration — if stack.toml or gen_config.py changed, regenerate CI workflows
3. Ruff lint + format — only staged .py files
4. AST parse check — verify staged source files have valid syntax
5. Pytest — targeted or full suite based on what changed
6. Marimo check — only if notebooks are staged
7. Notebook execution — only staged notebooks in the safe-to-run list

Post-commit

dev/hooks/post-commit rebuilds the docs Docker image in the background after every commit so the documentation site stays current with docstring changes.

Running tests

uv run python -m pytest tests/                    # full suite
uv run python -m pytest tests/sem/                 # one module
uv run python -m pytest tests/ -m "not stub"       # skip stub inventory
uv run python -m pytest tests/ -m stub             # only stub status
uv run python -m pytest tests/test_ast_coverage.py # structural invariants only

README generation

README.md is generated from README.md.j2 using live codebase data. Never edit README.md directly.

uv run python dev/scripts/gen_readme.py            # regenerate from template
uv run python dev/scripts/gen_readme.py --check     # verify freshness (CI gate)
uv run python dev/scripts/gen_readme.py --dry-run   # preview to stdout

Dynamic values pulled from: compose.yml (services), pyproject.toml (modules), pipeline registry (pipelines + steps), pytest --collect-only (test count), api.auth.manifest (credentials), .gitea/workflows/ (workflow count).

Project layout

stack/
├── compose.yml                  Docker Compose (32 services)
├── stack.toml                   Centralised configuration
├── pyproject.toml               Python project (uv, optional deps per module)
├── src/
│   ├── aco/                      ACO analytics (express, pipe, table, lake, load)
│   ├── api/                      FastAPI server + auth + diag
│   ├── bcda/                     BCDA FHIR R4 client
│   ├── bib/                      Zotero bibliography store, tags, metadata
│   ├── bls/                      BLS data
│   ├── ccw/                      CCW data dictionary
│   ├── cli/                      CLI entry point (typer)
│   ├── cms/                      CMS public data tables
│   ├── conf/                     Config loader, storage abstraction, table base
│   ├── llm/                      
│   ├── mail/                     
│   ├── opps/                     Outpatient Prospective Payment System
│   ├── perf/                     Pipeline telemetry (OpenTelemetry)
│   ├── pfs/                      Physician Fee Schedule
│   ├── prisma/                   
│   ├── rec/                      
│   ├── rex/                      REX fixed-width file processing
│   ├── sem/                      Semantic coverage orchestration
│   └── zot/                      
├── infra/                       Service configs and Dockerfiles
│   ├── images/                  All Dockerfiles (api, notebooks, zotero, docs, mc)
│   ├── traefik/                 Reverse proxy + fhirworx CSS injection
│   ├── grafana/                 Dashboards and datasource provisioning
│   ├── prometheus/              Metrics collection
│   ├── loki/                    Log aggregation
│   └── ...                      coredns, nginx, trino, polaris, rustfs, act-runner, gitea
├── assets/                      Branding, styles, generated artifacts
│   ├── css/                     Per-service CSS (dashboard, grafana, marimo)
│   ├── icons/                   Favicons, logos, coverage badge
│   └── fhirworx.py              fhirworx chart palette and Altair theme
├── cloud/                       Cloud provider deployment guides
│   ├── self-hosted/             Docker Compose (default)
│   ├── aws/                     S3, RDS, Glue, ECS
│   ├── gcp/                     GCS, Cloud SQL, BigQuery, Cloud Run
│   └── azure/                   ABFS, Azure SQL, Unity Catalog, Container Apps
├── dev/                         Dev tooling
│   ├── scripts/                 Code generators, bootstrap, supply chain tools
│   ├── hooks/                   Git hooks (tracked, core.hooksPath = dev/hooks)
│   ├── seeds/                   Reference data (BCDA samples, CMS docs)
│   └── pipelines/               CI-agnostic pipeline specs
├── tests/                       14,010 tests at 99% coverage
├── notebooks/                   Marimo notebooks
├── docs/                        Docusaurus site
├── data/                        DuckDB, bib.sqlite, BCDA/CMS data, zotero (gitignored)
└── .env                         Derived credentials (not in git)

Prerequisites

• Docker (rootless mode)
• NVIDIA GPU with container toolkit
• Domain or /etc/hosts entries for *.fhirworx.io

# Fix for rootless NVIDIA containers
sudo sed -i 's/#no-cgroups = false/no-cgroups = true/' /etc/nvidia-container-runtime/config.toml

SSH access

# ~/.ssh/config
Host gitea
    HostName fhirworx.io
    Port 2222
    User git
    IdentityFile ~/.ssh/gitea_ed25519
    IdentitiesOnly yes