Homelab Stack

Self-hosted infrastructure stack with GPU support, S3-compatible storage, Git server with container registry, CI/CD, data lakehouse, observability, documentation, and research tools. All services share a unified deep-navy design system ("loch") injected via Traefik middleware.

Services

All services are accessible via Traefik reverse proxy at *.homelab.fhirworx.io.

Service	URL	Description
Dashboard	homelab.fhirworx.io	Homelab command center
Traefik	traefik.homelab.fhirworx.io	Reverse proxy dashboard
Gitea	gitea.homelab.fhirworx.io	Git server with LFS + container registry
Woodpecker CI	ci.homelab.fhirworx.io	CI/CD pipelines
RustFS Console	s3console.homelab.fhirworx.io	S3 storage management UI
RustFS S3 API	s3.homelab.fhirworx.io	S3-compatible object storage
Notebooks	notebooks.homelab.fhirworx.io	GPU-accelerated Marimo notebooks
Zotero	zotero.homelab.fhirworx.io	Reference manager (KasmVNC)
Docs	docs.homelab.fhirworx.io	API reference and CMS bibliography browser
Nessie	nessie.homelab.fhirworx.io	Git-like data catalog for Iceberg
Polaris	polaris.homelab.fhirworx.io	Apache Iceberg catalog with governance
Trino	trino.homelab.fhirworx.io	Distributed SQL query engine
Grafana	grafana.homelab.fhirworx.io	Observability dashboards
Prometheus	prometheus.homelab.fhirworx.io	Metrics collection
Jaeger	jaeger.homelab.fhirworx.io	Distributed tracing
Loki	loki.homelab.fhirworx.io	Log aggregation

Architecture

Network Segmentation

The stack uses five isolated Docker networks. Services only join the networks they need.

Network	Type	Purpose
gateway	external	Traefik + public-facing services
storage	internal	PostgreSQL + RustFS (S3)
data	internal	Lakehouse (Nessie, Trino, Polaris)
observability	internal	Grafana, Prometheus, Jaeger, Loki
ci	external	Woodpecker + Gitea + RustFS

                           ┌──────────────────────────┐
                           │     Traefik :80/:443      │
                           │  Reverse Proxy + Tracing  │
                           └────────────┬─────────────┘
                  gateway network       │
        ┌───────────┬──────────┬────────┼────────┬───────────┐
        ▼           ▼          ▼        ▼        ▼           ▼
   ┌────────┐ ┌─────────┐ ┌────────┐ ┌──────┐ ┌──────┐ ┌────────┐
   │ Gitea  │ │Woodpeck.│ │Notebook│ │Zotero│ │Dash- │ │Grafana │
   │ :3000  │ │  :8000  │ │ :2718  │ │:8080 │ │board │ │ :3000  │
   └───┬────┘ └────┬────┘ └────────┘ └──────┘ └──┬───┘ └───┬────┘
       │           │                              │         │
       │     ci network                     ┌─────┘  observability network
       │           │                        ▼   ┌───────┬───────┬───┘
       │           │                   ┌──────┐ ▼       ▼       ▼
       │           │                   │ Docs │ ┌──────┐ ┌───────┐ ┌──────┐
       │           │                   │ :80  │ │Jaeger│ │Promet.│ │ Loki │
       │           │                   └──────┘ │:16686│ │ :9090 │ │:3100 │
       │           │                            └──────┘ └───────┘ └──────┘
       │           │
       └─────┬─────┘        data network
             │         ┌────────┬──────────┐
      storage network  ▼        ▼          ▼
       ┌─────┼────┐ ┌──────┐ ┌──────┐ ┌───────┐
       ▼     ▼    │ │Nessie│ │Trino │ │Polaris│
  ┌────────┐ ┌────┴──┐ │:19120│ │:8080 │ │ :8181 │
  │Postgres│ │RustFS │ └──────┘ └──────┘ └───────┘
  │ :5432  │ │:9000/1│
  └────────┘ └───────┘

Traefik Routing

Routes are defined in traefik/dynamic/services.yml using Go templates with Sprig functions. The $reef dict is the single source of truth — each entry generates a router, service, and middleware chain:

"gitea" (dict "port" "3000" "theme" true "mw" "secure-headers")

• theme: true applies the inject-loch middleware (injects favicon + CSS via HTML rewrite)
• theme: false skips injection (API-only services like Nessie, Polaris, Loki)
• mw is a comma-separated list of additional middlewares (secure-headers, local-only, infra-headers)
• subdomain overrides the FQDN prefix (e.g. woodpecker-server → ci.homelab.fhirworx.io)

To add a new service, add one line to the $reef dict and a service block to compose.yml.

Loch Design System

All HTML-serving services get a unified deep-navy design system via the inject-loch Traefik middleware, which uses the rewrite-body plugin to inject <link> tags before </head>:

• fav32.png — 32x32 PNG favicon served from the dashboard container
• loch.css — Global CSS overrides (colors, fonts, borders)
• inject.css — Served as loch.css, hides service-specific logos and applies the palette

Per-service CSS files handle service-specific overrides:

File	Purpose
styles/inject.css	Global loch palette + logo hiding (served as loch.css)
styles/dashboard.css	Dashboard tile grid layout
styles/gitea.css	Gitea theme (registered as loch in Gitea UI settings)
styles/woodpecker.css	Woodpecker custom CSS (via WOODPECKER_CUSTOM_CSS_FILE)
styles/marimo.css	Marimo notebook theme
styles/fav32.png	32x32 PNG favicon (canonical, injected by Traefik)
styles/favicon.png	64x64 PNG favicon
styles/fav16.png	16x16 PNG favicon
styles/apple-touch-icon.png	180x180 Apple touch icon
styles/favicon.svg	SVG favicon (legacy)
styles/logo.svg	Navbar logo for Gitea
styles/logo.png	256x256 PNG logo

Grafana favicons are replaced via volume mounts over the default Grafana icons.

Prerequisites

• Docker with rootless mode
• NVIDIA GPU with container toolkit
• Domain pointing to your server (or /etc/hosts entries)

Fix for rootless NVIDIA containers:

sudo sed -i 's/#no-cgroups = false/no-cgroups = true/' /etc/nvidia-container-runtime/config.toml

DNS Setup

Add to /etc/hosts (replace IP with your server's LAN IP):

192.168.1.192 homelab.fhirworx.io
192.168.1.192 dashboard.homelab.fhirworx.io
192.168.1.192 traefik.homelab.fhirworx.io
192.168.1.192 gitea.homelab.fhirworx.io
192.168.1.192 ci.homelab.fhirworx.io
192.168.1.192 notebooks.homelab.fhirworx.io
192.168.1.192 zotero.homelab.fhirworx.io
192.168.1.192 docs.homelab.fhirworx.io
192.168.1.192 nessie.homelab.fhirworx.io
192.168.1.192 trino.homelab.fhirworx.io
192.168.1.192 s3console.homelab.fhirworx.io
192.168.1.192 s3.homelab.fhirworx.io
192.168.1.192 grafana.homelab.fhirworx.io
192.168.1.192 prometheus.homelab.fhirworx.io
192.168.1.192 jaeger.homelab.fhirworx.io
192.168.1.192 loki.homelab.fhirworx.io
192.168.1.192 polaris.homelab.fhirworx.io

Setup

1. Generate Root Key and Bootstrap Credentials

All service credentials are derived from a single 256-bit root key via HKDF-SHA256. See Credential Management for details.

# Generate root key
ROOT_KEY=$(openssl rand -hex 32)

# Add non-managed vars to .env first
cat > .env <<EOF
DOMAIN=homelab.fhirworx.io
HOST_IP=192.168.1.192
WOODPECKER_ADMIN=<admin_username>
EOF

2. Start Core Services and Bootstrap

docker compose up -d postgres rustfs traefik

# Bootstrap: creates DB roles, databases, RustFS buckets/users/IAM,
# derives all 21 credentials, rotates Grafana password, writes .env
ROOT_KEY=$ROOT_KEY uv run python -m api.auth bootstrap $(git rev-parse HEAD)

Bootstrap automatically handles:

• PostgreSQL roles and databases (gitea, woodpecker, nessie, polaris)
• RustFS buckets (gitea, gitea-lfs, gitea-packages, lakehouse)
• RustFS IAM users and per-user policies from rustfs/policies/
• Gitea admin password rotation and API token creation
• Grafana admin password rotation (non-fatal if unreachable)
• Woodpecker CI secret provisioning

3. Start All Services

docker compose up -d

4. Configure Gitea

1. Complete initial setup at gitea.homelab.fhirworx.io
2. Create organization homelab
3. Create repository stack
4. Create OAuth2 application for Woodpecker:
   • Name: Woodpecker CI
   • Redirect URI: http://ci.homelab.fhirworx.io/authorize
   • Copy Client ID and Secret to .env

5. Configure Woodpecker Root Key

Add one secret in Woodpecker UI (ci.homelab.fhirworx.io):

• root_key — The hex-encoded root key (enables auto-rotation on every deploy)
• registry_user — Gitea username

All other Woodpecker secrets (registry_pass, s3_access_key, s3_secret_key, gitea_token, gitea_url) are provisioned automatically by bootstrap and rotated on each deploy by provision.

6. Verify Services

# Check Traefik routes
curl -s http://localhost:8081/api/http/routers | jq -r '.[].name'

# Check Nessie
curl http://nessie.homelab.fhirworx.io/api/v2/config

# Check Trino
docker exec -it trino trino --execute "SHOW CATALOGS"

# Check Prometheus targets
curl -s http://prometheus.homelab.fhirworx.io/api/v1/targets | jq '.data.activeTargets[].labels.job'

Observability

Component	Purpose
Prometheus	Metrics collection and storage
Grafana	Visualization and dashboards
Jaeger	Distributed tracing (OTLP collector)
Loki	Log aggregation
Promtail	Log shipper (Docker container logs)

Services instrumented with OpenTelemetry tracing:

Service	Method
Traefik	Native OTLP export
Nessie	Quarkus OTEL
Polaris	Quarkus OTEL
Trino	Native OTEL

All traces flow to Jaeger via OTLP gRPC (port 4317).

Prometheus Targets

Job	Endpoint
prometheus	localhost:9090
traefik	traefik:8080/metrics
jaeger	jaeger:14269
loki	loki:3100
nessie	nessie:19120/q/metrics
polaris	polaris:8182/q/metrics
trino	trino:8080/v1/status

Credential Management

All service credentials are derived from a single 256-bit root key using HKDF-SHA256 (RFC 5869). No passwords are stored in .env — they are deterministically regenerated from the root key and a commit SHA on every deploy.

How It Works

ROOT_KEY (one Woodpecker secret) + commit_sha → HKDF-SHA256 → all service credentials → .env

The derivation uses two tiers:

Tier	Salt	Rotates	Purpose
Bootstrap	b"bootstrap"	Only when root key changes	Superuser passwords, OAuth2 app
Service	commit_sha	Every deploy	DB passwords, S3 keys, API tokens

Managed Credentials (19 derived, 2 skipped)

Variable	Tier	Format	Backend
POSTGRES_PASSWORD	Bootstrap	password	env only
GITEA_DB_PASSWORD	Service	password	PostgreSQL (role: git)
WOODPECKER_DB_PASSWORD	Service	password	PostgreSQL (role: woodpecker)
NESSIE_DB_PASSWORD	Service	password	PostgreSQL (role: nessie)
POLARIS_DB_PASSWORD	Service	password	PostgreSQL (role: polaris)
RUSTFS_ACCESS_KEY	Service	hex	env only
RUSTFS_SECRET_KEY	Service	hex	env only
GITEA_S3_*, NESSIE_S3_*, POLARIS_S3_*	Service	hex	env only (aliases of RUSTFS)
WOODPECKER_AGENT_SECRET	Service	hex	env only
STACK_API_SECRET	Service	hex	env only
POLARIS_ROOT_SECRET	Service	password	env only
GF_ADMIN_PASSWORD	Service	password	Grafana (admin API)
GITEA_ADMIN_PASSWORD	Service	password	Gitea (change-password)
GITEA_TOKEN	Service	—	Gitea (API token)

Not managed: WOODPECKER_GITEA_CLIENT/SECRET (created once via OAuth2 app), DATABRICKS_TOKEN (external).

CLI Usage

# First-time setup: create DB roles + databases, derive all creds, write .env
ROOT_KEY=$KEY uv run python -m api.auth bootstrap $(git rev-parse HEAD)

# Rotate: derive new service credentials, update backends, rewrite .env
ROOT_KEY=$KEY uv run python -m api.auth provision $COMMIT_SHA

# Dry-run: print derived values without provisioning
ROOT_KEY=$KEY uv run python -m api.auth derive $COMMIT_SHA --redact

Auto-Rotation via CI

The provision step in .woodpecker/deploy.yml runs after every deploy, deriving fresh service-tier credentials from the commit SHA and rotating PostgreSQL passwords, Gitea tokens, Grafana admin password, and Woodpecker CI secrets automatically.

Troubleshooting

CI provision step fails with No module named 'api': The provision step needs UV_PROJECT_ENVIRONMENT: .venv and UV_LINK_MODE: copy in its environment block (matching the CI pipeline pattern). Without these, uv run creates a fresh venv in the CI container and doesn't find the installed project.

Grafana password rotation fails: This is non-fatal — bootstrap/provision will log a warning but continue. Check that Grafana is running and reachable at its internal URL. On first bootstrap with default admin/admin credentials, rotation should succeed. On subsequent provisions, the old password is read from .env before overwrite.

RustFS bucket creation fails: Ensure RustFS is running and the RUSTFS_ACCESS_KEY/RUSTFS_SECRET_KEY in .env match the root credentials configured in the RustFS container environment.

Documentation Site

The docs site at docs.homelab.fhirworx.io is a Docusaurus static site that auto-generates API reference from Python docstrings and provides a searchable CMS bibliography browser.

Build Pipeline

The Dockerfile uses three stages:

1. Python (uv + griffe) — extracts docstrings from src/ into markdown, exports bib.sqlite to library.json
2. Node — runs npm ci && npm run build to produce static HTML
3. nginx:alpine — serves the built site

Auto-rebuild

• Local: a post-commit git hook rebuilds the docs image and restarts the container in the background after every commit
• CI: .woodpecker/deploy.yml builds, scans, and pushes the docs image on every push to main (no path filter — docstrings change with any src/ edit)

The CI build step mounts data/bib.sqlite from the host so the library browser has real data.

Library Browser

The /library page provides faceted search over the CMS bibliography (bib.sqlite). Data is exported to library.json at build time — no runtime database needed. Filters by collection, tag namespace (module:, source:, year:, rule:, sup:), and full-text search.

Data Lakehouse

Component	Purpose
Nessie	Git-like version control for data (branches, merges, time travel)
Polaris	Apache Iceberg catalog with governance, RBAC, and multi-tenant support
Trino	Distributed SQL query engine with JDBC/ODBC support
RustFS	S3 storage backend (lakehouse bucket)

Nessie vs Polaris

Feature	Nessie	Polaris
Git-like branching	Yes	No
Time travel	Yes	Yes
RBAC/governance	Basic	Full
Multi-catalog management	No	Yes
Iceberg REST API	Yes	Yes

Query Access

Method	Connection
Trino CLI	docker exec -it trino trino
Trino JDBC	jdbc:trino://trino.homelab.fhirworx.io/iceberg
Trino Web UI	trino.homelab.fhirworx.io
PyIceberg	Direct to Nessie REST API
DuckDB	Read Iceberg via PyArrow

Example: Create a Table

docker exec -it trino trino

CREATE SCHEMA iceberg.analytics;

CREATE TABLE iceberg.analytics.events (
    event_id VARCHAR,
    event_time TIMESTAMP(6) WITH TIME ZONE,
    user_id VARCHAR,
    event_type VARCHAR,
    properties MAP(VARCHAR, VARCHAR)
)
WITH (
    format = 'PARQUET',
    partitioning = ARRAY['day(event_time)']
);

INSERT INTO iceberg.analytics.events VALUES
    ('evt-001', CURRENT_TIMESTAMP, 'user-1', 'page_view', MAP(ARRAY['page'], ARRAY['/home']));

SELECT * FROM iceberg.analytics.events;

Directory Structure

stack/
├── compose.yml                  # Docker Compose (all services)
├── .env                         # Environment variables (not in git)
├── README.md
├── styles/                         # Centralized theme assets
│   ├── inject.css               # Global loch CSS (served as loch.css)
│   ├── dashboard.css            # Dashboard tile grid
│   ├── gitea.css                # Gitea loch theme
│   ├── woodpecker.css           # Woodpecker custom CSS
│   ├── marimo.css               # Marimo notebook theme
│   ├── fav32.png                # 32x32 PNG favicon (canonical, injected by Traefik)
│   ├── favicon.png              # 64x64 PNG favicon
│   ├── fav16.png                # 16x16 PNG favicon
│   ├── apple-touch-icon.png     # 180x180 Apple touch icon
│   ├── favicon.svg              # SVG favicon (legacy)
│   ├── logo.svg                 # Navbar logo (Gitea)
│   └── logo.png                 # 256x256 PNG logo
├── nginx/
│   ├── nginx.conf               # Dashboard nginx config (CORS for theme assets)
│   └── index.html               # Dashboard HTML
├── traefik/
│   ├── traefik.yml              # Static config (entrypoints, plugins, OTEL)
│   ├── dynamic/
│   │   └── services.yml         # All routes, middlewares, services (Go template)
│   └── plugins/                 # Local rewrite-body plugin
├── gitea/
│   └── custom/                  # Gitea custom branding
├── grafana/
│   └── provisioning/            # Datasources + dashboard config
├── loki/
│   ├── loki-config.yml
│   └── promtail-config.yml
├── prometheus/
│   ├── prometheus.yml
│   └── targets/services.yml
├── trino/
│   └── etc/                     # Trino config + catalog properties
├── docs/                        # Documentation site (Docusaurus)
│   ├── Dockerfile               # Multi-stage: griffe extract → Node build → nginx
│   ├── docusaurus.config.js
│   ├── scripts/
│   │   ├── extract_docs.py      # griffe → markdown API reference
│   │   └── export_library.py    # bib.Store → library.json
│   ├── docs/                    # Hand-written + auto-generated markdown
│   └── src/pages/library.tsx    # CMS bibliography browser
├── notebooks/                   # Marimo notebook files
├── zotero/                      # Zotero data + profiles
├── src/                         # Python source
│   └── api/auth/                # HKDF credential derivation + provisioning
├── tests/                       # Test suite
├── .woodpecker/                 # CI/CD pipeline definitions
│   ├── ci.yml                   # Tests + linting on push
│   ├── deploy.yml               # Build, scan, push images + deploy on main
│   ├── infra-ci.yml             # Dockerfile linting + build validation
│   └── rebuild-all.yml          # Manual full rebuild of all images
└── data/                        # Shared data directory

SSH Access

Configure Git SSH access in ~/.ssh/config:

Host gitea
    HostName homelab.fhirworx.io
    Port 2222
    User git
    IdentityFile ~/.ssh/gitea_ed25519
    IdentitiesOnly yes

Generate and add SSH key:

ssh-keygen -t ed25519 -C "user@homelab" -f ~/.ssh/gitea_ed25519 -N ""
ssh-keyscan -p 2222 homelab.fhirworx.io >> ~/.ssh/known_hosts

Add public key to Gitea: gitea.homelab.fhirworx.io/user/settings/keys

Container Registry

Custom images use the fhirworx/<svc>:<sha> naming convention locally. The registry prefix localhost:3000/ is only added in CI push steps.

# Login
echo "<GITEA_TOKEN>" | docker login localhost:3000 -u <username> --password-stdin

# Tag and push
docker tag fhirworx/<svc>:latest localhost:3000/homelab/<svc>:latest
docker push localhost:3000/homelab/<svc>:latest

Custom images: notebooks, zotero, docs. Compose resolves fhirworx/<svc>:${COMMIT_SHA:-latest}.

Ports Reference

Port	Service	Notes
80	Traefik HTTP	All subdomain routing
443	Traefik HTTPS	TLS termination
8081	Traefik API	Dashboard + API (localhost only)
2222	Gitea SSH	Git over SSH
3478	Zotero TURN	WebRTC for KasmVNC