rewrite README to reflect current platform state

Covers analytics pipelines, expression layer, CI/CD, credential management, vulnerability scanning, and updated project layout. Drops stale setup verbosity.
2026-03-23 19:18:02 -04:00
parent 22ebc46ec5
commit e703c88bae
1 changed files with 193 additions and 451 deletions
--- a/README.md
+++ b/README.md
@@ -1,470 +1,245 @@
-# Homelab Stack
+# Stack
 ![coverage](styles/coverage.svg)
-Self-hosted infrastructure stack with GPU support, S3-compatible storage, Git server with container registry, CI/CD, data lakehouse, observability, documentation, and research tools. All services share a unified deep-navy design system ("loch") injected via Traefik middleware.
+Healthcare analytics platform on self-hosted infrastructure. Replaces dbt SQL models with narwhals DataFrame-agnostic expression functions, backed by DuckDB locally and Iceberg/Trino in the lakehouse. 22 services, 13 data pipelines, 11,931 tests at 99% coverage.
-## Services
+## Quick start
 All services are accessible via Traefik reverse proxy at `*.homelab.fhirworx.io`.
 | Service | URL | Description |
 |---------|-----|-------------|
 | Dashboard | `homelab.fhirworx.io` | Homelab command center |
 | Traefik | `traefik.homelab.fhirworx.io` | Reverse proxy dashboard |
 | Gitea | `gitea.homelab.fhirworx.io` | Git server with LFS + container registry |
 | Woodpecker CI | `ci.homelab.fhirworx.io` | CI/CD pipelines |
 | RustFS Console | `s3console.homelab.fhirworx.io` | S3 storage management UI |
 | RustFS S3 API | `s3.homelab.fhirworx.io` | S3-compatible object storage |
 | Notebooks | `notebooks.homelab.fhirworx.io` | GPU-accelerated Marimo notebooks |
 | Zotero | `zotero.homelab.fhirworx.io` | Reference manager (KasmVNC) |
 | Docs | `docs.homelab.fhirworx.io` | API reference and CMS bibliography browser |
 | Nessie | `nessie.homelab.fhirworx.io` | Git-like data catalog for Iceberg |
 | Polaris | `polaris.homelab.fhirworx.io` | Apache Iceberg catalog with governance |
 | Trino | `trino.homelab.fhirworx.io` | Distributed SQL query engine |
 | Grafana | `grafana.homelab.fhirworx.io` | Observability dashboards |
 | Prometheus | `prometheus.homelab.fhirworx.io` | Metrics collection |
 | Jaeger | `jaeger.homelab.fhirworx.io` | Distributed tracing |
 | Loki | `loki.homelab.fhirworx.io` | Log aggregation |
 ## Architecture
 ### Network Segmentation
 The stack uses five isolated Docker networks. Services only join the networks they need.
 | Network | Type | Purpose |
 |---------|------|---------|
 | `gateway` | external | Traefik + public-facing services |
 | `storage` | internal | PostgreSQL + RustFS (S3) |
 | `data` | internal | Lakehouse (Nessie, Trino, Polaris) |
 | `observability` | internal | Grafana, Prometheus, Jaeger, Loki |
 | `ci` | external | Woodpecker + Gitea + RustFS |
 ```
                           ┌──────────────────────────┐
                           │     Traefik :80/:443      │
                           │  Reverse Proxy + Tracing  │
                           └────────────┬─────────────┘
                  gateway network       │
        ┌───────────┬──────────┬────────┼────────┬───────────┐
        ▼           ▼          ▼        ▼        ▼           ▼
   ┌────────┐ ┌─────────┐ ┌────────┐ ┌──────┐ ┌──────┐ ┌────────┐
   │ Gitea  │ │Woodpeck.│ │Notebook│ │Zotero│ │Dash- │ │Grafana │
   │ :3000  │ │  :8000  │ │ :2718  │ │:8080 │ │board │ │ :3000  │
   └───┬────┘ └────┬────┘ └────────┘ └──────┘ └──┬───┘ └───┬────┘
       │           │                              │         │
       │     ci network                     ┌─────┘  observability network
       │           │                        ▼   ┌───────┬───────┬───┘
       │           │                   ┌──────┐ ▼       ▼       ▼
       │           │                   │ Docs │ ┌──────┐ ┌───────┐ ┌──────┐
       │           │                   │ :80  │ │Jaeger│ │Promet.│ │ Loki │
       │           │                   └──────┘ │:16686│ │ :9090 │ │:3100 │
       │           │                            └──────┘ └───────┘ └──────┘
       │           │
       └─────┬─────┘        data network
             │         ┌────────┬──────────┐
      storage network  ▼        ▼          ▼
       ┌─────┼────┐ ┌──────┐ ┌──────┐ ┌───────┐
       ▼     ▼    │ │Nessie│ │Trino │ │Polaris│
  ┌────────┐ ┌────┴──┐ │:19120│ │:8080 │ │ :8181 │
  │Postgres│ │RustFS │ └──────┘ └──────┘ └───────┘
  │ :5432  │ │:9000/1│
  └────────┘ └───────┘
 ```
 ### Traefik Routing
 Routes are defined in `traefik/dynamic/services.yml` using Go templates with Sprig functions. The `$reef` dict is the single source of truth — each entry generates a router, service, and middleware chain:
 ```
 "gitea" (dict "port" "3000" "theme" true "mw" "secure-headers")
 ```
 - **`theme: true`** applies the `inject-loch` middleware (injects favicon + CSS via HTML rewrite)
 - **`theme: false`** skips injection (API-only services like Nessie, Polaris, Loki)
 - **`mw`** is a comma-separated list of additional middlewares (`secure-headers`, `local-only`, `infra-headers`)
 - **`subdomain`** overrides the FQDN prefix (e.g. `woodpecker-server` → `ci.homelab.fhirworx.io`)
 To add a new service, add one line to the `$reef` dict and a service block to `compose.yml`.
 ### Loch Design System
 All HTML-serving services get a unified deep-navy design system via the `inject-loch` Traefik middleware, which uses the [rewrite-body](https://github.com/packruler/rewrite-body) plugin to inject `<link>` tags before `</head>`:
 - **`fav32.png`** — 32x32 PNG favicon served from the dashboard container
 - **`loch.css`** — Global CSS overrides (colors, fonts, borders)
 - **`inject.css`** — Served as `loch.css`, hides service-specific logos and applies the palette
 Per-service CSS files handle service-specific overrides:
 | File | Purpose |
 |------|---------|
 | `styles/inject.css` | Global loch palette + logo hiding (served as `loch.css`) |
 | `styles/dashboard.css` | Dashboard tile grid layout |
 | `styles/gitea.css` | Gitea theme (registered as `loch` in Gitea UI settings) |
 | `styles/woodpecker.css` | Woodpecker custom CSS (via `WOODPECKER_CUSTOM_CSS_FILE`) |
 | `styles/marimo.css` | Marimo notebook theme |
 | `styles/fav32.png` | 32x32 PNG favicon (canonical, injected by Traefik) |
 | `styles/favicon.png` | 64x64 PNG favicon |
 | `styles/fav16.png` | 16x16 PNG favicon |
 | `styles/apple-touch-icon.png` | 180x180 Apple touch icon |
 | `styles/favicon.svg` | SVG favicon (legacy) |
 | `styles/logo.svg` | Navbar logo for Gitea |
 | `styles/logo.png` | 256x256 PNG logo |
 Grafana favicons are replaced via volume mounts over the default Grafana icons.
 ## Prerequisites
 - Docker with rootless mode
 - NVIDIA GPU with container toolkit
 - Domain pointing to your server (or `/etc/hosts` entries)
 Fix for rootless NVIDIA containers:
 ```bash
 sudo sed -i 's/#no-cgroups = false/no-cgroups = true/' /etc/nvidia-container-runtime/config.toml
 ```
 ## DNS Setup
 Add to `/etc/hosts` (replace IP with your server's LAN IP):
 ```
 192.168.1.192 homelab.fhirworx.io
 192.168.1.192 dashboard.homelab.fhirworx.io
 192.168.1.192 traefik.homelab.fhirworx.io
 192.168.1.192 gitea.homelab.fhirworx.io
 192.168.1.192 ci.homelab.fhirworx.io
 192.168.1.192 notebooks.homelab.fhirworx.io
 192.168.1.192 zotero.homelab.fhirworx.io
 192.168.1.192 docs.homelab.fhirworx.io
 192.168.1.192 nessie.homelab.fhirworx.io
 192.168.1.192 trino.homelab.fhirworx.io
 192.168.1.192 s3console.homelab.fhirworx.io
 192.168.1.192 s3.homelab.fhirworx.io
 192.168.1.192 grafana.homelab.fhirworx.io
 192.168.1.192 prometheus.homelab.fhirworx.io
 192.168.1.192 jaeger.homelab.fhirworx.io
 192.168.1.192 loki.homelab.fhirworx.io
 192.168.1.192 polaris.homelab.fhirworx.io
 ```
 ## Setup
 ### 1. Generate Root Key and Bootstrap Credentials
 All service credentials are derived from a single 256-bit root key via HKDF-SHA256. See [Credential Management](#credential-management) for details.
 ```bash
-# Generate root key
+# 1. Generate root key and seed .env
 ROOT_KEY=$(openssl rand -hex 32)
 # Add non-managed vars to .env first
 cat > .env <<EOF
 DOMAIN=homelab.fhirworx.io
 HOST_IP=192.168.1.192
-WOODPECKER_ADMIN=<admin_username>
+WOODPECKER_ADMIN=kert
 EOF
 ```
-### 2. Start Core Services and Bootstrap
+# 2. Start core services and bootstrap credentials
 ```bash
 docker compose up -d postgres rustfs traefik
 # Bootstrap: creates DB roles, databases, RustFS buckets/users/IAM,
 # derives all 21 credentials, rotates Grafana password, writes .env
 ROOT_KEY=$ROOT_KEY uv run python -m api.auth bootstrap $(git rev-parse HEAD)
 ```
-Bootstrap automatically handles:
+# 3. Start everything
 - PostgreSQL roles and databases (gitea, woodpecker, nessie, polaris)
 - RustFS buckets (`gitea`, `gitea-lfs`, `gitea-packages`, `lakehouse`)
 - RustFS IAM users and per-user policies from `rustfs/policies/`
 - Gitea admin password rotation and API token creation
 - Grafana admin password rotation (non-fatal if unreachable)
 - Woodpecker CI secret provisioning
 ### 3. Start All Services
 ```bash
 docker compose up -d
 # 4. Verify
 curl -s http://localhost:8000/health | python3 -m json.tool
 ```
-### 4. Configure Gitea
+## Services
-1. Complete initial setup at `gitea.homelab.fhirworx.io`
+All services route through Traefik at `*.homelab.fhirworx.io`.
 2. Create organization `homelab`
 3. Create repository `stack`
 4. Create OAuth2 application for Woodpecker:
   - Name: `Woodpecker CI`
   - Redirect URI: `http://ci.homelab.fhirworx.io/authorize`
   - Copy Client ID and Secret to `.env`
-### 5. Configure Woodpecker Root Key
+| Service | URL | Purpose |
 |---------|-----|---------|
 | API | `api` (internal) | FastAPI — health, pipelines, bib endpoints |
 | Notebooks | `notebooks.homelab.fhirworx.io` | Marimo notebooks (GPU-accelerated) |
 | Docs | `docs.homelab.fhirworx.io` | Docusaurus — API reference + CMS bibliography |
 | Gitea | `gitea.homelab.fhirworx.io` | Git server, container registry, LFS |
 | Woodpecker CI | `ci.homelab.fhirworx.io` | CI/CD pipelines |
 | Zotero | `zotero.homelab.fhirworx.io` | Reference manager (KasmVNC desktop) |
 | Dashboard | `homelab.fhirworx.io` | Service launcher |
 | RustFS | `s3.homelab.fhirworx.io` | S3-compatible object storage |
 | Nessie | `nessie.homelab.fhirworx.io` | Git-like Iceberg catalog |
 | Polaris | `polaris.homelab.fhirworx.io` | Iceberg catalog with governance |
 | Trino | `trino.homelab.fhirworx.io` | Distributed SQL engine |
 | Grafana | `grafana.homelab.fhirworx.io` | Dashboards |
 | Prometheus | `prometheus.homelab.fhirworx.io` | Metrics |
 | Jaeger | `jaeger.homelab.fhirworx.io` | Distributed tracing |
 | Loki | `loki.homelab.fhirworx.io` | Log aggregation |
-Add one secret in Woodpecker UI (`ci.homelab.fhirworx.io`):
+## Analytics platform
 - `root_key` — The hex-encoded root key (enables auto-rotation on every deploy)
 - `registry_user` — Gitea username
-All other Woodpecker secrets (`registry_pass`, `s3_access_key`, `s3_secret_key`, `gitea_token`, `gitea_url`) are provisioned automatically by `bootstrap` and rotated on each deploy by `provision`.
+### Architecture
-### 6. Verify Services
+```
  CCLF/BCDA/CMS data
        │
        ▼
  src/aco/express/     narwhals expressions (DataFrame-agnostic)
        │
        ▼
  src/aco/pipe/        pipeline runner with schema validation
        │
        ▼
  DuckDB (local)  ──or──  Iceberg + Trino (lakehouse)
        │
        ▼
  src/api/             FastAPI (health, runs, bib)
        │
        ▼
  notebooks/           Marimo interactive analysis
 ```
 ### Pipelines
 13 registered pipelines with 189 steps, validated by `run_pipeline(steps, load)` which enforces output schemas via pandera-style column checks.
 | Pipeline | Domain |
 |----------|--------|
 | `input_layer` | Raw CCLF/BCDA ingestion |
 | `core` | Encounters, patients, providers |
 | `claims_preprocessing` | Claim matching, encounter IDs |
 | `readmissions` | 30-day readmission rates |
 | `pharmacy` | Part D drug utilization |
 | `ahrq_measures` | AHRQ quality indicators |
 | `cms_quality_measures` | CMS ACO quality metrics |
 | `hcc_suspecting` | HCC risk adjustment |
 | `provider_attribution` | Provider-patient assignment |
 | `quality_measures` | Composite quality scores |
 | `data_quality` | Validation and completeness |
 | `cclf` | CCLF file parsing |
 | `main` | Full pipeline orchestration |
 ### Expression layer
 `src/aco/express/` contains narwhals functions that work across polars, pandas, and cuDF. Each module mirrors a pipeline and exports pure transformation functions.
 ### Configuration
 `stack.toml` is the single config file. `src/conf/` provides:
 - `cfg` — attribute access (`cfg.db.aco`)
 - `path("db.aco")` — absolute path resolution relative to repo root
 - `context()` — environment switching (local/lake/databricks/trino)
 - `secret("api.secret", "STACK_API_SECRET")` — env var with config fallback
 Override context at runtime: `STACK_CONTEXT=lake`.
 ### Bibliography
 `src/bib/` manages a Zotero-backed citation store in `data/bib.sqlite`. Column-level provenance via `Tag.col(ref, desc)`. The docs site exports `library.json` for a searchable bibliography browser at `/library`.
 ## CI/CD
 Six Woodpecker pipelines in `.woodpecker/`:
 | Pipeline | Trigger | What it does |
 |----------|---------|--------------|
 | `ci.yml` | Push | Ruff lint, pytest, marimo check |
 | `deploy.yml` | Push to main | Build images, Trivy scan, deploy, rotate credentials |
 | `harden.yml` | Manual/cron | Rebuild all images `--no-cache`, scan, auto-close vuln issues |
 | `rebuild-all.yml` | Manual | Full rebuild + scan + deploy |
 | `infra-ci.yml` | Push | Hadolint Dockerfiles, build validation |
 | `release.yml` | Tag | Release pipeline |
 ### Image tagging
 Images are tagged with the short commit SHA (8 chars):
 ```
 fhirworx/api:22ebc46e
 fhirworx/notebooks:a4a2dcb0
 fhirworx/docs:a4a2dcb0
 ```
 Compose resolves via `${COMMIT_SHA:-latest}` from `.env`. Custom images: `api`, `notebooks`, `zotero`, `docs`, `mc`.
 ### Vulnerability management
 The `harden.yml` pipeline rebuilds all images with `--no-cache` to pick up OS patches, runs Trivy scans, and auto-files or auto-closes Gitea issues via `api.diag.vuln`.
 ## Credential management
 All 19 service credentials derive from a single 256-bit root key via HKDF-SHA256. No passwords stored in `.env` — they regenerate deterministically from root key + commit SHA on each deploy.
 ```
 ROOT_KEY + commit_sha → HKDF-SHA256 → all credentials → .env + backends
 ```
 | Tier | Salt | Rotates |
 |------|------|---------|
 | Bootstrap | `b"bootstrap"` | Only when root key changes |
 | Service | `commit_sha` | Every deploy |
 ```bash
-# Check Traefik routes
+# Bootstrap (first time)
 curl -s http://localhost:8081/api/http/routers | jq -r '.[].name'
 # Check Nessie
 curl http://nessie.homelab.fhirworx.io/api/v2/config
 # Check Trino
 docker exec -it trino trino --execute "SHOW CATALOGS"
 # Check Prometheus targets
 curl -s http://prometheus.homelab.fhirworx.io/api/v1/targets | jq '.data.activeTargets[].labels.job'
 ```
 ## Observability
 | Component | Purpose |
 |-----------|---------|
 | **Prometheus** | Metrics collection and storage |
 | **Grafana** | Visualization and dashboards |
 | **Jaeger** | Distributed tracing (OTLP collector) |
 | **Loki** | Log aggregation |
 | **Promtail** | Log shipper (Docker container logs) |
 Services instrumented with OpenTelemetry tracing:
 | Service | Method |
 |---------|--------|
 | Traefik | Native OTLP export |
 | Nessie | Quarkus OTEL |
 | Polaris | Quarkus OTEL |
 | Trino | Native OTEL |
 All traces flow to Jaeger via OTLP gRPC (port 4317).
 ### Prometheus Targets
 | Job | Endpoint |
 |-----|----------|
 | prometheus | localhost:9090 |
 | traefik | traefik:8080/metrics |
 | jaeger | jaeger:14269 |
 | loki | loki:3100 |
 | nessie | nessie:19120/q/metrics |
 | polaris | polaris:8182/q/metrics |
 | trino | trino:8080/v1/status |
 ## Credential Management
 All service credentials are derived from a single 256-bit root key using HKDF-SHA256 (RFC 5869). No passwords are stored in `.env` — they are deterministically regenerated from the root key and a commit SHA on every deploy.
 ### How It Works
 ```
 ROOT_KEY (one Woodpecker secret) + commit_sha → HKDF-SHA256 → all service credentials → .env
 ```
 The derivation uses two tiers:
 | Tier | Salt | Rotates | Purpose |
 |------|------|---------|---------|
 | **Bootstrap** | `b"bootstrap"` | Only when root key changes | Superuser passwords, OAuth2 app |
 | **Service** | `commit_sha` | Every deploy | DB passwords, S3 keys, API tokens |
 ### Managed Credentials (19 derived, 2 skipped)
 | Variable | Tier | Format | Backend |
 |----------|------|--------|---------|
 | `POSTGRES_PASSWORD` | Bootstrap | password | env only |
 | `GITEA_DB_PASSWORD` | Service | password | PostgreSQL (role: git) |
 | `WOODPECKER_DB_PASSWORD` | Service | password | PostgreSQL (role: woodpecker) |
 | `NESSIE_DB_PASSWORD` | Service | password | PostgreSQL (role: nessie) |
 | `POLARIS_DB_PASSWORD` | Service | password | PostgreSQL (role: polaris) |
 | `RUSTFS_ACCESS_KEY` | Service | hex | env only |
 | `RUSTFS_SECRET_KEY` | Service | hex | env only |
 | `GITEA_S3_*`, `NESSIE_S3_*`, `POLARIS_S3_*` | Service | hex | env only (aliases of RUSTFS) |
 | `WOODPECKER_AGENT_SECRET` | Service | hex | env only |
 | `STACK_API_SECRET` | Service | hex | env only |
 | `POLARIS_ROOT_SECRET` | Service | password | env only |
 | `GF_ADMIN_PASSWORD` | Service | password | Grafana (admin API) |
 | `GITEA_ADMIN_PASSWORD` | Service | password | Gitea (change-password) |
 | `GITEA_TOKEN` | Service | — | Gitea (API token) |
 Not managed: `WOODPECKER_GITEA_CLIENT/SECRET` (created once via OAuth2 app), `DATABRICKS_TOKEN` (external).
 ### CLI Usage
 ```bash
 # First-time setup: create DB roles + databases, derive all creds, write .env
 ROOT_KEY=$KEY uv run python -m api.auth bootstrap $(git rev-parse HEAD)
-# Rotate: derive new service credentials, update backends, rewrite .env
+# Rotate (every deploy, automatic in CI)
 ROOT_KEY=$KEY uv run python -m api.auth provision $COMMIT_SHA
-# Dry-run: print derived values without provisioning
+# Dry-run
 ROOT_KEY=$KEY uv run python -m api.auth derive $COMMIT_SHA --redact
 ```
-### Auto-Rotation via CI
+## Network segmentation
-The `provision` step in `.woodpecker/deploy.yml` runs after every deploy, deriving fresh service-tier credentials from the commit SHA and rotating PostgreSQL passwords, Gitea tokens, Grafana admin password, and Woodpecker CI secrets automatically.
+| Network | Type | Services |
 |---------|------|----------|
 | `gateway` | external | Traefik, all public-facing |
 | `storage` | internal | PostgreSQL, RustFS |
 | `data` | internal | Nessie, Trino, Polaris |
 | `observability` | internal | Grafana, Prometheus, Jaeger, Loki |
 | `ci` | external | Woodpecker, Gitea, RustFS |
-### Troubleshooting
+## Design system
-**CI provision step fails with `No module named 'api'`**: The provision step needs `UV_PROJECT_ENVIRONMENT: .venv` and `UV_LINK_MODE: copy` in its environment block (matching the CI pipeline pattern). Without these, `uv run` creates a fresh venv in the CI container and doesn't find the installed project.
+Theme: **loch** (deep-navy). Traefik's `inject-loch` middleware rewrites HTML to inject `loch.css` and the favicon. Per-service CSS in `styles/`. Altair chart theme in `styles/nature.py` (Nature-conformant, accessible palette, IBM Plex Mono headings).
-**Grafana password rotation fails**: This is non-fatal — bootstrap/provision will log a warning but continue. Check that Grafana is running and reachable at its internal URL. On first bootstrap with default `admin`/`admin` credentials, rotation should succeed. On subsequent provisions, the old password is read from `.env` before overwrite.
+## Data lakehouse
 **RustFS bucket creation fails**: Ensure RustFS is running and the `RUSTFS_ACCESS_KEY`/`RUSTFS_SECRET_KEY` in `.env` match the root credentials configured in the RustFS container environment.
 ## Documentation Site
 The docs site at `docs.homelab.fhirworx.io` is a Docusaurus static site that auto-generates API reference from Python docstrings and provides a searchable CMS bibliography browser.
 ### Build Pipeline
 The Dockerfile uses three stages:
 1. **Python** (`uv` + `griffe`) — extracts docstrings from `src/` into markdown, exports `bib.sqlite` to `library.json`
 2. **Node** — runs `npm ci && npm run build` to produce static HTML
 3. **nginx:alpine** — serves the built site
 ### Auto-rebuild
 - **Local**: a `post-commit` git hook rebuilds the docs image and restarts the container in the background after every commit
 - **CI**: `.woodpecker/deploy.yml` builds, scans, and pushes the docs image on every push to main (no path filter — docstrings change with any `src/` edit)
 The CI build step mounts `data/bib.sqlite` from the host so the library browser has real data.
 ### Library Browser
 The `/library` page provides faceted search over the CMS bibliography (`bib.sqlite`). Data is exported to `library.json` at build time — no runtime database needed. Filters by collection, tag namespace (`module:`, `source:`, `year:`, `rule:`, `sup:`), and full-text search.
 ## Data Lakehouse
 | Component | Purpose |
 |-----------|---------|
-| **Nessie** | Git-like version control for data (branches, merges, time travel) |
+| Nessie | Git-like branching and time travel for Iceberg tables |
-| **Polaris** | Apache Iceberg catalog with governance, RBAC, and multi-tenant support |
+| Polaris | Iceberg catalog with RBAC and multi-tenant governance |
-| **Trino** | Distributed SQL query engine with JDBC/ODBC support |
+| Trino | Distributed SQL (`iceberg` catalog) |
-| **RustFS** | S3 storage backend (lakehouse bucket) |
+| RustFS | S3 storage backend (`s3://lakehouse/`) |
-### Nessie vs Polaris
+Query via Trino CLI, JDBC, Web UI, PyIceberg, or DuckDB+PyArrow.
-| Feature | Nessie | Polaris |
+## Project layout
 |---------|--------|---------|
 | Git-like branching | Yes | No |
 | Time travel | Yes | Yes |
 | RBAC/governance | Basic | Full |
 | Multi-catalog management | No | Yes |
 | Iceberg REST API | Yes | Yes |
 ### Query Access
 | Method | Connection |
 |--------|------------|
 | **Trino CLI** | `docker exec -it trino trino` |
 | **Trino JDBC** | `jdbc:trino://trino.homelab.fhirworx.io/iceberg` |
 | **Trino Web UI** | `trino.homelab.fhirworx.io` |
 | **PyIceberg** | Direct to Nessie REST API |
 | **DuckDB** | Read Iceberg via PyArrow |
 ### Example: Create a Table
 ```sql
 docker exec -it trino trino
 CREATE SCHEMA iceberg.analytics;
 CREATE TABLE iceberg.analytics.events (
    event_id VARCHAR,
    event_time TIMESTAMP(6) WITH TIME ZONE,
    user_id VARCHAR,
    event_type VARCHAR,
    properties MAP(VARCHAR, VARCHAR)
 )
 WITH (
    format = 'PARQUET',
    partitioning = ARRAY['day(event_time)']
 );
 INSERT INTO iceberg.analytics.events VALUES
    ('evt-001', CURRENT_TIMESTAMP, 'user-1', 'page_view', MAP(ARRAY['page'], ARRAY['/home']));
 SELECT * FROM iceberg.analytics.events;
 ```
 ## Directory Structure
 ```
 stack/
-├── compose.yml                  # Docker Compose (all services)
+├── compose.yml                  Docker Compose (22 services)
-├── .env                         # Environment variables (not in git)
+├── stack.toml                   Centralised configuration
-├── README.md
+├── pyproject.toml               Python project (uv)
-├── styles/                         # Centralized theme assets
+├── src/
-│   ├── inject.css               # Global loch CSS (served as loch.css)
+│   ├── aco/
-│   ├── dashboard.css            # Dashboard tile grid
+│   │   ├── express/             narwhals transformation functions (15 modules)
-│   ├── gitea.css                # Gitea loch theme
+│   │   ├── pipe/                pipeline runner + 13 pipeline modules
-│   ├── woodpecker.css           # Woodpecker custom CSS
+│   │   ├── table/               table schemas with column metadata
-│   ├── marimo.css               # Marimo notebook theme
+│   │   └── dag.py               pipeline DAG
-│   ├── fav32.png                # 32x32 PNG favicon (canonical, injected by Traefik)
+│   ├── api/
-│   ├── favicon.png              # 64x64 PNG favicon
+│   │   ├── routes/              health, pipelines, bib endpoints
-│   ├── fav16.png                # 16x16 PNG favicon
+│   │   ├── auth/                HKDF credential derivation + provisioning
-│   ├── apple-touch-icon.png     # 180x180 Apple touch icon
+│   │   └── diag/                Trivy vuln reporter
-│   ├── favicon.svg              # SVG favicon (legacy)
+│   ├── bib/                     Zotero store, tags, metadata
-│   ├── logo.svg                 # Navbar logo (Gitea)
+│   ├── conf/                    Config loader (cfg, path, context, secret)
-│   └── logo.png                 # 256x256 PNG logo
+│   ├── bcda/                    BCDA API client
-├── nginx/
+│   ├── cms/                     CMS data utilities
-│   ├── nginx.conf               # Dashboard nginx config (CORS for theme assets)
+│   ├── pfs/                     Physician Fee Schedule
-│   └── index.html               # Dashboard HTML
+│   └── rex/                     REX data processing
-├── traefik/
+├── notebooks/                   Marimo notebooks (12 notebooks)
-│   ├── traefik.yml              # Static config (entrypoints, plugins, OTEL)
+├── tests/                       11,931 tests at 99% coverage
-│   ├── dynamic/
+├── styles/                      Loch design system (CSS, favicons, nature.py)
-│   │   └── services.yml         # All routes, middlewares, services (Go template)
+├── docs/                        Docusaurus site (griffe → Node → nginx)
-│   └── plugins/                 # Local rewrite-body plugin
+├── .woodpecker/                 CI/CD pipelines (6 pipelines)
-├── gitea/
+├── api/Dockerfile               FastAPI service
-│   └── custom/                  # Gitea custom branding
+├── notebooks/Dockerfile         Marimo + CUDA 12.6 + cuDF
-├── grafana/
+├── zotero/Dockerfile            Zotero desktop (KasmVNC + GPU)
-│   └── provisioning/            # Datasources + dashboard config
+├── docs/Dockerfile              3-stage: griffe → pnpm build → nginx:alpine
-├── loki/
+├── traefik/                     Reverse proxy config + loch injection
-│   ├── loki-config.yml
+├── data/                        DuckDB, bib.sqlite, BCDA/CMS data
-│   └── promtail-config.yml
+└── .env                         Derived credentials (not in git)
 ├── prometheus/
 │   ├── prometheus.yml
 │   └── targets/services.yml
 ├── trino/
 │   └── etc/                     # Trino config + catalog properties
 ├── docs/                        # Documentation site (Docusaurus)
 │   ├── Dockerfile               # Multi-stage: griffe extract → Node build → nginx
 │   ├── docusaurus.config.js
 │   ├── scripts/
 │   │   ├── extract_docs.py      # griffe → markdown API reference
 │   │   └── export_library.py    # bib.Store → library.json
 │   ├── docs/                    # Hand-written + auto-generated markdown
 │   └── src/pages/library.tsx    # CMS bibliography browser
 ├── notebooks/                   # Marimo notebook files
 ├── zotero/                      # Zotero data + profiles
 ├── src/                         # Python source
 │   └── api/auth/                # HKDF credential derivation + provisioning
 ├── tests/                       # Test suite
 ├── .woodpecker/                 # CI/CD pipeline definitions
 │   ├── ci.yml                   # Tests + linting on push
 │   ├── deploy.yml               # Build, scan, push images + deploy on main
 │   ├── infra-ci.yml             # Dockerfile linting + build validation
 │   └── rebuild-all.yml          # Manual full rebuild of all images
 └── data/                        # Shared data directory
 ```
-## SSH Access
+## Prerequisites
-Configure Git SSH access in `~/.ssh/config`:
+- Docker (rootless mode)
 - NVIDIA GPU with container toolkit
 - Domain or `/etc/hosts` entries for `*.homelab.fhirworx.io`
 ```bash
 # Fix for rootless NVIDIA containers
 sudo sed -i 's/#no-cgroups = false/no-cgroups = true/' /etc/nvidia-container-runtime/config.toml
 ```
 ## SSH access
 ```
 # ~/.ssh/config
 Host gitea
    HostName homelab.fhirworx.io
    Port 2222
@@ -472,36 +247,3 @@ Host gitea
    IdentityFile ~/.ssh/gitea_ed25519
    IdentitiesOnly yes
 ```
 Generate and add SSH key:
 ```bash
 ssh-keygen -t ed25519 -C "user@homelab" -f ~/.ssh/gitea_ed25519 -N ""
 ssh-keyscan -p 2222 homelab.fhirworx.io >> ~/.ssh/known_hosts
 ```
 Add public key to Gitea: `gitea.homelab.fhirworx.io/user/settings/keys`
 ## Container Registry
 Custom images use the `fhirworx/<svc>:<sha>` naming convention locally. The registry prefix `localhost:3000/` is only added in CI push steps.
 ```bash
 # Login
 echo "<GITEA_TOKEN>" | docker login localhost:3000 -u <username> --password-stdin
 # Tag and push
 docker tag fhirworx/<svc>:latest localhost:3000/homelab/<svc>:latest
 docker push localhost:3000/homelab/<svc>:latest
 ```
 Custom images: `notebooks`, `zotero`, `docs`. Compose resolves `fhirworx/<svc>:${COMMIT_SHA:-latest}`.
 ## Ports Reference
 | Port | Service | Notes |
 |------|---------|-------|
 | 80 | Traefik HTTP | All subdomain routing |
 | 443 | Traefik HTTPS | TLS termination |
 | 8081 | Traefik API | Dashboard + API (localhost only) |
 | 2222 | Gitea SSH | Git over SSH |
 | 3478 | Zotero TURN | WebRTC for KasmVNC |