homelab/stack
1
0
Fork 0
Code Issues 57 Pull Requests 1 Actions Packages Projects 1 Releases Wiki Activity

update README for current stack state
All checks were successful
ci/woodpecker/push/infra-ci Pipeline was successful
Details
ci/woodpecker/push/deploy Pipeline was successful
Details
ci/woodpecker/push/ci Pipeline was successful
Details

Browse Source 
Reflects domain change (homelab.fhirworx.io), network segmentation,
centralized CSS theming, Go template routing, and favicon injection.
This commit is contained in:
kert
2026-02-28 19:34:00 -05:00
parent d269695ad1
commit 89f8dec187
1 changed files with 201 additions and 294 deletions
Download Patch File Download Diff File Expand all files Collapse all files

495
README.md
View File

@@ -1,88 +1,117 @@
# Homelab Stack # Homelab Stack
Self-hosted infrastructure stack with GPU support, S3-compatible storage, Git server with container registry, CI/CD, data lakehouse, observability, and research tools. Self-hosted infrastructure stack with GPU support, S3-compatible storage, Git server with container registry, CI/CD, data lakehouse, observability, and research tools. All services share a unified retro "throwback" theme injected via Traefik middleware.
## Services ## Services
All services are accessible via Traefik reverse proxy on port 80. All services are accessible via Traefik reverse proxy at `*.homelab.fhirworx.io`.
| Service | URL | Description | | Service | URL | Description |
|---------|-----|-------------| |---------|-----|-------------|
| Dashboard | http://home.fhirworx.io | Homelab command center | | Dashboard | `homelab.fhirworx.io` | Homelab command center |
| Traefik | http://traefik.home.fhirworx.io | Reverse proxy dashboard | | Traefik | `traefik.homelab.fhirworx.io` | Reverse proxy dashboard |
| Gitea | http://gitea.home.fhirworx.io | Git server with LFS + container registry | | Gitea | `gitea.homelab.fhirworx.io` | Git server with LFS + container registry |
| Woodpecker CI | http://ci.home.fhirworx.io | CI/CD pipelines | | Woodpecker CI | `ci.homelab.fhirworx.io` | CI/CD pipelines |
| RustFS Console | http://minio.home.fhirworx.io | S3 storage management UI | | RustFS Console | `s3console.homelab.fhirworx.io` | S3 storage management UI |
| RustFS S3 API | http://s3.home.fhirworx.io | S3-compatible object storage | | RustFS S3 API | `s3.homelab.fhirworx.io` | S3-compatible object storage |
| Notebooks | http://notebooks.home.fhirworx.io | GPU-accelerated Marimo notebooks | | Notebooks | `notebooks.homelab.fhirworx.io` | GPU-accelerated Marimo notebooks |
| Zotero | http://zotero.home.fhirworx.io | Reference manager with VNC | | Zotero | `zotero.homelab.fhirworx.io` | Reference manager (KasmVNC) |
| Nessie | http://nessie.home.fhirworx.io | Git-like data catalog for Iceberg | | Nessie | `nessie.homelab.fhirworx.io` | Git-like data catalog for Iceberg |
| Polaris | http://polaris.home.fhirworx.io | Apache Iceberg catalog with governance | | Polaris | `polaris.homelab.fhirworx.io` | Apache Iceberg catalog with governance |
| Trino | http://trino.home.fhirworx.io | Distributed SQL query engine | | Trino | `trino.homelab.fhirworx.io` | Distributed SQL query engine |
| Grafana | http://grafana.home.fhirworx.io | Observability dashboards | | Grafana | `grafana.homelab.fhirworx.io` | Observability dashboards |
| Prometheus | http://prometheus.home.fhirworx.io | Metrics collection | | Prometheus | `prometheus.homelab.fhirworx.io` | Metrics collection |
| Jaeger | http://jaeger.home.fhirworx.io | Distributed tracing | | Jaeger | `jaeger.homelab.fhirworx.io` | Distributed tracing |
| Loki | http://loki.home.fhirworx.io | Log aggregation | | Loki | `loki.homelab.fhirworx.io` | Log aggregation |
| PostgreSQL | localhost:5432 | Database (Bitnami hardened) |
## Architecture ## Architecture
### Network Segmentation
The stack uses five isolated Docker networks. Services only join the networks they need.
| Network | Type | Purpose |
|---------|------|---------|
| `gateway` | external | Traefik + public-facing services |
| `storage` | internal | PostgreSQL + RustFS (S3) |
| `data` | internal | Lakehouse (Nessie, Trino, Polaris) |
| `observability` | internal | Grafana, Prometheus, Jaeger, Loki |
| `ci` | external | Woodpecker + Gitea + RustFS |
``` ```
┌─────────────────────────────────────────────────────────────────────────────┐ ┌──────────────────────────┐
│ intrastack network │ │ Traefik :80/:443 │
├─────────────────────────────────────────────────────────────────────────────┤ │ Reverse Proxy + Tracing │
│ │ └────────────┬─────────────┘
│ ┌─────────────────────────────────────────────────────────────────────┐ │ gateway network │
│ │ Traefik :80/:443 │ │ ┌───────────┬──────────┬────────┼────────┬───────────┐
│ │ Reverse Proxy + Load Balancer + OTEL Tracing │ │ ▼ ▼ ▼ ▼ ▼ ▼
│ └───────────────────────────────┬─────────────────────────────────────┘ │ ┌────────┐ ┌─────────┐ ┌────────┐ ┌──────┐ ┌──────┐ ┌────────┐
│ │ │ │ Gitea │ │Woodpeck.│ │Notebook│ │Zotero│ │Dash- │ │Grafana │
│ ┌─────────────────────────────┼─────────────────────────────┐ │ │ :3000 │ │ :8000 │ │ :2718 │ │:8080 │ │board │ │ :3000 │
│ │ │ │ │ └───┬────┘ └────┬────┘ └────────┘ └──────┘ └──────┘ └───┬────┘
│ ▼ ▼ ▼ │ │ │ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ ci network observability network
│ │ Gitea │ │Woodpecker│ │ Notebooks│ │ Zotero │ │ Dashboard│ │ │ │ ┌───────┬───────┬───┘
│ │ :3000 │ │ :8000 │ │ :2718 │ │ :8080 │ │ :80 │ │ │ │ ▼ ▼ ▼
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └──────────┘ └──────────┘ │ │ │ ┌──────┐ ┌───────┐ ┌──────┐
│ │ │ │ │ │ │ │Jaeger│ │Promet.│ │ Loki │
│ └─────────────┼─────────────┘ │ │ │ │:16686│ │ :9090 │ │:3100 │
│ ▼ │ │ │ └──────┘ └───────┘ └──────┘
│ ┌──────────────────────────────────────────────────────────────────────┐ │ │ │
│ │ PostgreSQL :5432 │ │ └─────┬─────┘ data network
│ │ (gitea, woodpecker, nessie) │ │ │ ┌────────┬──────────┐
│ └──────────────────────────────────────────────────────────────────────┘ │ storage network ▼ ▼ ▼
│ │ ┌─────┼────┐ ┌──────┐ ┌──────┐ ┌───────┐
│ ┌───────────────────────────────────────────────────────────────────────┐ │ ▼ ▼ │ │Nessie│ │Trino │ │Polaris│
│ │ Data Lakehouse │ │ ┌────────┐ ┌────┴──┐ │:19120│ │:8080 │ │ :8181 │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────────────────────────┐ │ │ │Postgres│ │RustFS │ └──────┘ └──────┘ └───────┘
│ │ │ Trino │────▶│ Nessie │────▶│ RustFS │ │ │ │ :5432 │ │:9000/1│
│ │ │ :8080 │ │ :19120 │ │ S3 :9000 | Console :9001 │ │ │ └────────┘ └───────┘
│ │ └──────────┘ └──────────┘ │ Buckets: gitea, lakehouse │ │ │
│ │ │ └──────────────────────────────┘ │ │
│ │ │ ┌──────────┐ ▲ │ │
│ │ └──────────▶│ Polaris │──────────────┘ │ │
│ │ │ :8181 │ Governance + RBAC │ │
│ │ └──────────┘ │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ Observability Stack │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │
│ │ │ Grafana │────▶│Prometheus│ │ Jaeger │◀────│ Loki │ │ │
│ │ │ :3000 │ │ :9090 │ │ :16686 │ │ :3100 │ │ │
│ │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ │
│ │ │ ▲ ▲ │ │
│ │ └──────── Dashboards ───────────────┴───────────────┘ │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
``` ```
### Traefik Routing
Routes are defined in `traefik/dynamic/services.yml` using Go templates with Sprig functions. The `$reef` dict is the single source of truth — each entry generates a router, service, and middleware chain:
```
"gitea" (dict "port" "3000" "theme" true "mw" "secure-headers")
```
- **`theme: true`** applies the `inject-throwback` middleware (injects favicon + CSS via HTML rewrite)
- **`theme: false`** skips injection (API-only services like Nessie, Polaris, Loki)
- **`mw`** is a comma-separated list of additional middlewares (`secure-headers`, `local-only`, `infra-headers`)
- **`subdomain`** overrides the FQDN prefix (e.g. `woodpecker-server` → `ci.homelab.fhirworx.io`)
To add a new service, add one line to the `$reef` dict and a service block to `compose.yml`.
### Throwback Theme
All HTML-serving services get a unified retro theme via the `inject-throwback` Traefik middleware, which uses the [rewrite-body](https://github.com/packruler/rewrite-body) plugin to inject `<link>` tags before `</head>`:
- **`favicon.svg`** — Pixel-art "H" favicon served from the dashboard container
- **`throwback.css`** — Global CSS overrides (colors, fonts, borders)
- **`inject.css`** — Served as `throwback.css`, hides service-specific logos and applies the palette
Per-service CSS files handle service-specific overrides:
| File | Purpose |
|------|---------|
| `css/inject.css` | Global throwback palette + logo hiding (served as `throwback.css`) |
| `css/dashboard.css` | Dashboard tile grid layout |
| `css/gitea.css` | Gitea theme (registered as `throwback` in Gitea UI settings) |
| `css/woodpecker.css` | Woodpecker custom CSS (via `WOODPECKER_CUSTOM_CSS_FILE`) |
| `css/marimo.css` | Marimo notebook theme |
| `css/favicon.svg` | Canonical SVG favicon |
| `css/logo.svg` | Navbar logo for Gitea |
Grafana favicons are replaced via volume mounts over the default Grafana icons.
## Prerequisites ## Prerequisites
- Docker with rootless mode - Docker with rootless mode
- NVIDIA GPU with container toolkit - NVIDIA GPU with container toolkit
- iptables for port 80 redirect (rootless Docker can't bind privileged ports) - Domain pointing to your server (or `/etc/hosts` entries)
Fix for rootless NVIDIA containers: Fix for rootless NVIDIA containers:
```bash ```bash
@@ -93,36 +122,23 @@ sudo sed -i 's/#no-cgroups = false/no-cgroups = true/' /etc/nvidia-container-run
Add to `/etc/hosts` (replace IP with your server's LAN IP): Add to `/etc/hosts` (replace IP with your server's LAN IP):
```bash
# Homelab Stack
192.168.1.192 home.fhirworx.io
192.168.1.192 dashboard.home.fhirworx.io
192.168.1.192 traefik.home.fhirworx.io
192.168.1.192 gitea.home.fhirworx.io
192.168.1.192 ci.home.fhirworx.io
192.168.1.192 notebooks.home.fhirworx.io
192.168.1.192 zotero.home.fhirworx.io
192.168.1.192 nessie.home.fhirworx.io
192.168.1.192 trino.home.fhirworx.io
192.168.1.192 minio.home.fhirworx.io
192.168.1.192 s3.home.fhirworx.io
192.168.1.192 grafana.home.fhirworx.io
192.168.1.192 prometheus.home.fhirworx.io
192.168.1.192 jaeger.home.fhirworx.io
192.168.1.192 loki.home.fhirworx.io
192.168.1.192 polaris.home.fhirworx.io
``` ```
192.168.1.192 homelab.fhirworx.io
## Port 80 Redirect 192.168.1.192 dashboard.homelab.fhirworx.io
192.168.1.192 traefik.homelab.fhirworx.io
Rootless Docker cannot bind to privileged ports. Use iptables to redirect port 80 to Traefik's port 8880: 192.168.1.192 gitea.homelab.fhirworx.io
192.168.1.192 ci.homelab.fhirworx.io
```bash 192.168.1.192 notebooks.homelab.fhirworx.io
sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8880 192.168.1.192 zotero.homelab.fhirworx.io
sudo iptables -t nat -A OUTPUT -p tcp --dport 80 -o lo -j REDIRECT --to-port 8880 192.168.1.192 nessie.homelab.fhirworx.io
192.168.1.192 trino.homelab.fhirworx.io
# Persist across reboots 192.168.1.192 s3console.homelab.fhirworx.io
sudo sh -c 'iptables-save > /etc/iptables/rules.v4' 192.168.1.192 s3.homelab.fhirworx.io
192.168.1.192 grafana.homelab.fhirworx.io
192.168.1.192 prometheus.homelab.fhirworx.io
192.168.1.192 jaeger.homelab.fhirworx.io
192.168.1.192 loki.homelab.fhirworx.io
192.168.1.192 polaris.homelab.fhirworx.io
``` ```
## Setup ## Setup
@@ -132,6 +148,9 @@ sudo sh -c 'iptables-save > /etc/iptables/rules.v4'
Create `.env` file: Create `.env` file:
```bash ```bash
DOMAIN=homelab.fhirworx.io
HOST_IP=192.168.1.192
POSTGRES_PASSWORD=<your_password> POSTGRES_PASSWORD=<your_password>
RUSTFS_ACCESS_KEY=<your_access_key> RUSTFS_ACCESS_KEY=<your_access_key>
RUSTFS_SECRET_KEY=<your_secret_key> RUSTFS_SECRET_KEY=<your_secret_key>
@@ -140,9 +159,11 @@ GITEA_S3_SECRET_KEY=<gitea_s3_password>
GITEA_DB_PASSWORD=<gitea_db_password> GITEA_DB_PASSWORD=<gitea_db_password>
GITEA_TOKEN=<gitea_api_token> GITEA_TOKEN=<gitea_api_token>
WOODPECKER_DB_PASSWORD=<woodpecker_db_password> WOODPECKER_DB_PASSWORD=<woodpecker_db_password>
WOODPECKER_ADMIN=<admin_username>
WOODPECKER_AGENT_SECRET=<generated_secret> WOODPECKER_AGENT_SECRET=<generated_secret>
WOODPECKER_GITEA_CLIENT=<oauth_client_id> WOODPECKER_GITEA_CLIENT=<oauth_client_id>
WOODPECKER_GITEA_SECRET=<oauth_client_secret> WOODPECKER_GITEA_SECRET=<oauth_client_secret>
GF_ADMIN_PASSWORD=<grafana_password>
# Nessie Data Lake # Nessie Data Lake
NESSIE_DB_PASSWORD=<nessie_db_password> NESSIE_DB_PASSWORD=<nessie_db_password>
@@ -169,10 +190,10 @@ docker compose up -d postgres rustfs traefik
### 3. Configure RustFS ### 3. Configure RustFS
1. Access console at http://minio.home.fhirworx.io 1. Access console at `s3console.homelab.fhirworx.io`
2. Create user `git` with S3 credentials matching `GITEA_S3_ACCESS_KEY` and `GITEA_S3_SECRET_KEY` 2. Create user `git` with S3 credentials matching `.env`
3. Create buckets: `gitea`, `gitea-lfs`, `gitea-packages` 3. Create buckets: `gitea`, `gitea-lfs`, `gitea-packages`
4. Create user `nessie` with S3 credentials matching `NESSIE_S3_ACCESS_KEY` and `NESSIE_S3_SECRET_KEY` 4. Create user `nessie` with S3 credentials matching `.env`
5. Create bucket: `lakehouse` 5. Create bucket: `lakehouse`
6. Apply IAM policies to users (see below) 6. Apply IAM policies to users (see below)
@@ -260,27 +281,10 @@ docker exec -e PGPASSWORD=<postgres_password> postgres psql -U postgres -c "CREA
# Nessie database # Nessie database
docker exec -e PGPASSWORD=<postgres_password> postgres psql -U postgres -c "CREATE USER nessie WITH PASSWORD '<nessie_db_password>';" docker exec -e PGPASSWORD=<postgres_password> postgres psql -U postgres -c "CREATE USER nessie WITH PASSWORD '<nessie_db_password>';"
docker exec -e PGPASSWORD=<postgres_password> postgres psql -U postgres -c "CREATE DATABASE nessie OWNER nessie;" docker exec -e PGPASSWORD=<postgres_password> postgres psql -U postgres -c "CREATE DATABASE nessie OWNER nessie;"
docker exec -e PGPASSWORD=<postgres_password> postgres psql -U postgres -c "GRANT ALL PRIVILEGES ON DATABASE nessie TO nessie;"
# Polaris database # Polaris database
docker exec -e PGPASSWORD=<postgres_password> postgres psql -U postgres -c "CREATE USER polaris WITH PASSWORD '<polaris_db_password>';" docker exec -e PGPASSWORD=<postgres_password> postgres psql -U postgres -c "CREATE USER polaris WITH PASSWORD '<polaris_db_password>';"
docker exec -e PGPASSWORD=<postgres_password> postgres psql -U postgres -c "CREATE DATABASE polaris OWNER polaris;" docker exec -e PGPASSWORD=<postgres_password> postgres psql -U postgres -c "CREATE DATABASE polaris OWNER polaris;"
docker exec -e PGPASSWORD=<postgres_password> postgres psql -U postgres -c "GRANT ALL PRIVILEGES ON DATABASE polaris TO polaris;"
```
### Bootstrap Polaris
After starting Polaris, bootstrap the realm:
```bash
docker run --rm --network stack_intrastack \
-e POLARIS_PERSISTENCE_TYPE=relational-jdbc \
-e QUARKUS_DATASOURCE_DB_KIND=postgresql \
-e QUARKUS_DATASOURCE_JDBC_URL=jdbc:postgresql://postgres:5432/polaris \
-e QUARKUS_DATASOURCE_USERNAME=polaris \
-e QUARKUS_DATASOURCE_PASSWORD=<polaris_db_password> \
apache/polaris-admin-tool:latest \
bootstrap -r POLARIS -c POLARIS,root,<polaris_root_secret> -p
``` ```
### 5. Start All Services ### 5. Start All Services
@@ -291,20 +295,20 @@ docker compose up -d
### 6. Configure Gitea ### 6. Configure Gitea
1. Complete initial setup at http://gitea.home.fhirworx.io 1. Complete initial setup at `gitea.homelab.fhirworx.io`
2. Create organization `homelab` 2. Create organization `homelab`
3. Create repository `stack` 3. Create repository `stack`
4. Generate API token for container registry access 4. Generate API token for container registry access
5. Create OAuth2 application for Woodpecker: 5. Create OAuth2 application for Woodpecker:
- Name: `Woodpecker CI` - Name: `Woodpecker CI`
- Redirect URI: `http://ci.home.fhirworx.io/authorize` - Redirect URI: `http://ci.homelab.fhirworx.io/authorize`
- Copy Client ID and Secret to `.env` - Copy Client ID and Secret to `.env`
### 7. Configure Woodpecker Secrets ### 7. Configure Woodpecker Secrets
Add secrets in Woodpecker UI (http://ci.home.fhirworx.io): Add secrets in Woodpecker UI (`ci.homelab.fhirworx.io`):
- `registry_user` - Gitea username - `registry_user` — Gitea username
- `registry_pass` - Gitea token - `registry_pass` — Gitea token
### 8. Verify Services ### 8. Verify Services
@@ -313,35 +317,29 @@ Add secrets in Woodpecker UI (http://ci.home.fhirworx.io):
curl -s http://localhost:8081/api/http/routers | jq -r '.[].name' curl -s http://localhost:8081/api/http/routers | jq -r '.[].name'
# Check Nessie # Check Nessie
curl http://nessie.home.fhirworx.io/api/v2/config curl http://nessie.homelab.fhirworx.io/api/v2/config
# Check Trino # Check Trino
docker exec -it trino trino --execute "SHOW CATALOGS" docker exec -it trino trino --execute "SHOW CATALOGS"
# Check Prometheus targets # Check Prometheus targets
curl -s http://prometheus.home.fhirworx.io/api/v1/targets | jq '.data.activeTargets[].labels.job' curl -s http://prometheus.homelab.fhirworx.io/api/v1/targets | jq '.data.activeTargets[].labels.job'
``` ```
## Observability ## Observability
The stack includes a complete observability solution: | Component | Purpose |
|-----------|---------|
| **Prometheus** | Metrics collection and storage |
| **Grafana** | Visualization and dashboards |
| **Jaeger** | Distributed tracing (OTLP collector) |
| **Loki** | Log aggregation |
| **Promtail** | Log shipper (Docker container logs) |
### Components Services instrumented with OpenTelemetry tracing:
| Component | Purpose | Metrics | | Service | Method |
|-----------|---------|---------| |---------|--------|
| **Prometheus** | Metrics collection and storage | Scrapes all services |
| **Grafana** | Visualization and dashboards | Homelab Overview, Traefik |
| **Jaeger** | Distributed tracing | OTLP collector |
| **Loki** | Log aggregation | Container logs via Promtail |
| **Promtail** | Log shipper | Scrapes Docker container logs |
### Tracing
Services instrumented with OpenTelemetry:
| Service | Tracing |
|---------|---------|
| Traefik | Native OTLP export | | Traefik | Native OTLP export |
| Nessie | Quarkus OTEL | | Nessie | Quarkus OTEL |
| Polaris | Quarkus OTEL | | Polaris | Quarkus OTEL |
@@ -349,12 +347,6 @@ Services instrumented with OpenTelemetry:
All traces flow to Jaeger via OTLP gRPC (port 4317). All traces flow to Jaeger via OTLP gRPC (port 4317).
### Dashboards
Pre-provisioned Grafana dashboards:
- **Homelab Overview** - Service health, logs, traces
- **Traefik** - Request rates, latencies, errors by router/service
### Prometheus Targets ### Prometheus Targets
| Job | Endpoint | | Job | Endpoint |
@@ -369,81 +361,40 @@ Pre-provisioned Grafana dashboards:
## Data Lakehouse ## Data Lakehouse
The stack includes a complete data lakehouse built on Apache Iceberg with two catalog options:
### Components
| Component | Purpose | | Component | Purpose |
|-----------|---------| |-----------|---------|
| **Nessie** | Git-like version control for data (branches, merges, time travel) | | **Nessie** | Git-like version control for data (branches, merges, time travel) |
| **Polaris** | Apache Iceberg catalog with governance, RBAC, and multi-tenant support | | **Polaris** | Apache Iceberg catalog with governance, RBAC, and multi-tenant support |
| **Apache Iceberg** | Open table format with ACID transactions, schema evolution |
| **Trino** | Distributed SQL query engine with JDBC/ODBC support | | **Trino** | Distributed SQL query engine with JDBC/ODBC support |
| **RustFS** | S3 storage backend (lakehouse, polaris buckets) | | **RustFS** | S3 storage backend (lakehouse bucket) |
| **PyIceberg** | Python client for Iceberg tables in notebooks |
### Nessie vs Polaris ### Nessie vs Polaris
| Feature | Nessie | Polaris | | Feature | Nessie | Polaris |
|---------|--------|---------| |---------|--------|---------|
| Git-like branching | ✓ | ✗ | | Git-like branching | Yes | No |
| Time travel | ✓ | ✓ | | Time travel | Yes | Yes |
| RBAC/governance | Basic | Full | | RBAC/governance | Basic | Full |
| Multi-catalog management | ✗ | ✓ | | Multi-catalog management | No | Yes |
| Iceberg REST API | ✓ | ✓ | | Iceberg REST API | Yes | Yes |
| Unity Catalog-like | ✗ | ✓ |
**Use Nessie** for data versioning, experimentation branches, and development workflows.
**Use Polaris** for production governance, access control, and multi-tenant catalog management.
### Polaris API
```bash
# Get access token
TOKEN=$(curl -s -X POST http://polaris.home.fhirworx.io/api/catalog/v1/oauth/tokens \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "grant_type=client_credentials&client_id=root&client_secret=<polaris_root_secret>" | jq -r '.access_token')
# List catalogs
curl -s http://polaris.home.fhirworx.io/api/management/v1/catalogs \
-H "Authorization: Bearer $TOKEN"
# Create a catalog
curl -s -X POST http://polaris.home.fhirworx.io/api/management/v1/catalogs \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "analytics",
"type": "INTERNAL",
"properties": {"default-base-location": "s3://polaris/analytics/"},
"storageConfigInfo": {
"storageType": "S3",
"allowedLocations": ["s3://polaris/"],
"s3": {"region": "us-east-1", "endpoint": "http://rustfs:9000", "pathStyleAccess": true}
}
}'
```
### Query Access ### Query Access
| Method | Use Case | Connection | | Method | Connection |
|--------|----------|------------| |--------|------------|
| **Trino CLI** | Ad-hoc SQL queries | `docker exec -it trino trino` | | **Trino CLI** | `docker exec -it trino trino` |
| **Trino JDBC** | External tools (DBeaver, etc.) | `jdbc:trino://trino.home.fhirworx.io/iceberg` | | **Trino JDBC** | `jdbc:trino://trino.homelab.fhirworx.io/iceberg` |
| **Trino Web UI** | Query monitoring | http://trino.home.fhirworx.io | | **Trino Web UI** | `trino.homelab.fhirworx.io` |
| **PyIceberg** | Python notebooks | Direct to Nessie REST API | | **PyIceberg** | Direct to Nessie REST API |
| **DuckDB** | Fast local queries | Read Iceberg via PyArrow | | **DuckDB** | Read Iceberg via PyArrow |
### Example: Create a Table ### Example: Create a Table
```sql ```sql
-- Connect to Trino
docker exec -it trino trino docker exec -it trino trino
-- Create a schema
CREATE SCHEMA iceberg.analytics; CREATE SCHEMA iceberg.analytics;
-- Create a partitioned table
CREATE TABLE iceberg.analytics.events ( CREATE TABLE iceberg.analytics.events (
event_id VARCHAR, event_id VARCHAR,
event_time TIMESTAMP(6) WITH TIME ZONE, event_time TIMESTAMP(6) WITH TIME ZONE,
@@ -456,101 +407,62 @@ WITH (
partitioning = ARRAY['day(event_time)'] partitioning = ARRAY['day(event_time)']
); );
-- Insert data
INSERT INTO iceberg.analytics.events VALUES INSERT INTO iceberg.analytics.events VALUES
('evt-001', CURRENT_TIMESTAMP, 'user-1', 'page_view', MAP(ARRAY['page'], ARRAY['/home'])); ('evt-001', CURRENT_TIMESTAMP, 'user-1', 'page_view', MAP(ARRAY['page'], ARRAY['/home']));
-- Query data
SELECT * FROM iceberg.analytics.events; SELECT * FROM iceberg.analytics.events;
``` ```
## Traefik Reverse Proxy
All HTTP services are routed through Traefik using file-based configuration (not Docker labels).
### Configuration
Routes are defined in `traefik/dynamic/services.yml`. To add a new service:
```yaml
http:
routers:
myservice:
rule: "Host(`myservice.home.fhirworx.io`)"
service: myservice
entryPoints:
- web
services:
myservice:
loadBalancer:
servers:
- url: "http://myservice:8080"
```
### Features
- **OTEL Tracing** - All requests traced to Jaeger
- **Prometheus Metrics** - Request rates, latencies by router/service
- **File Provider** - Routes defined in YAML, hot-reloaded
- **Dashboard** - http://traefik.home.fhirworx.io
### TCP/UDP Routing
Traefik can also route TCP and UDP traffic. See `traefik/dynamic/` for examples.
## Directory Structure ## Directory Structure
``` ```
stack/ stack/
├── compose.yml # Docker Compose configuration ├── compose.yml # Docker Compose (all services)
├── .env # Environment variables (not in git) ├── .env # Environment variables (not in git)
├── .woodpecker.yml # CI/CD pipeline ├── README.md
├── README.md # This file ├── css/ # Centralized theme assets
├── data/ # Shared data directory │ ├── inject.css # Global throwback CSS (served as throwback.css)
│ ├── dashboard.css # Dashboard tile grid
│ ├── gitea.css # Gitea throwback theme
│ ├── woodpecker.css # Woodpecker custom CSS
│ ├── marimo.css # Marimo notebook theme
│ ├── favicon.svg # Pixel-art "H" favicon (canonical)
│ ├── favicon.png # 64x64 PNG favicon
│ ├── fav32.png # 32x32 PNG favicon
│ ├── fav16.png # 16x16 PNG favicon
│ ├── apple-touch-icon.png # 180x180 Apple touch icon
│ ├── logo.svg # Navbar logo (Gitea)
│ └── logo.png # PNG logo (256x256)
├── nginx/
│ ├── nginx.conf # Dashboard nginx config (CORS for theme assets)
│ └── index.html # Dashboard HTML
├── traefik/
│ ├── traefik.yml # Static config (entrypoints, plugins, OTEL)
│ ├── dynamic/
│ │ └── services.yml # All routes, middlewares, services (Go template)
│ └── plugins/ # Local rewrite-body plugin
├── gitea/ ├── gitea/
│ └── custom/ │ └── custom/ # Gitea custom branding
│ └── public/assets/css/
│ └── theme-throwback.css
├── grafana/ ├── grafana/
│ ├── provisioning/ │ └── provisioning/ # Datasources + dashboard config
│ │ ├── dashboards/
│ │ │ └── dashboards.yml
│ │ └── datasources/
│ │ └── datasources.yml
│ └── dashboards/
│ ├── homelab-overview.json
│ └── traefik.json
├── loki/ ├── loki/
│ ├── loki-config.yml │ ├── loki-config.yml
│ └── promtail-config.yml │ └── promtail-config.yml
├── nginx/
│ ├── nginx.conf # Dashboard nginx config
│ └── index.html # Dashboard HTML
├── notebooks/
│ ├── Dockerfile
│ ├── pyproject.toml
│ ├── .marimo.toml
│ └── retro-arcade.css
├── prometheus/ ├── prometheus/
│ └── prometheus.yml │ ├── prometheus.yml
├── traefik/ │ └── targets/services.yml
│ ├── traefik.yml # Static configuration
│ └── dynamic/
│ ├── middlewares.yml # Rate limiting, security headers
│ └── services.yml # All HTTP routes
├── trino/ ├── trino/
│ └── etc/ │ └── etc/ # Trino config + catalog properties
│ ├── config.properties ├── notebooks/ # Marimo notebook files
│ ├── jvm.config ├── zotero/ # Zotero data + profiles
│ ├── node.properties ├── src/ # Python source (narwhals expressions, pipes)
│ └── catalog/ ├── tests/ # Test suite
│ └── iceberg.properties ├── .woodpecker/ # CI/CD pipeline definitions
├── woodpecker/ │ ├── ci.yml
│ └── custom.css │ ├── deploy.yml
└── zotero/ │ ├── infra-ci.yml
├── Dockerfile │ └── rebuild-all.yml
└── data/ └── data/ # Shared data directory
``` ```
## SSH Access ## SSH Access
@@ -559,7 +471,7 @@ Configure Git SSH access in `~/.ssh/config`:
``` ```
Host gitea Host gitea
HostName gitea.home.fhirworx.io HostName homelab.fhirworx.io
Port 2222 Port 2222
User git User git
IdentityFile ~/.ssh/gitea_ed25519 IdentityFile ~/.ssh/gitea_ed25519
@@ -569,33 +481,28 @@ Host gitea
Generate and add SSH key: Generate and add SSH key:
```bash ```bash
ssh-keygen -t ed25519 -C "user@homelab" -f ~/.ssh/gitea_ed25519 -N "" ssh-keygen -t ed25519 -C "user@homelab" -f ~/.ssh/gitea_ed25519 -N ""
ssh-keyscan -p 2222 gitea.home.fhirworx.io >> ~/.ssh/known_hosts ssh-keyscan -p 2222 homelab.fhirworx.io >> ~/.ssh/known_hosts
``` ```
Add public key to Gitea: http://gitea.home.fhirworx.io/user/settings/keys Add public key to Gitea: `gitea.homelab.fhirworx.io/user/settings/keys`
## Container Registry ## Container Registry
Login:
```bash ```bash
echo "<GITEA_TOKEN>" | docker login gitea.home.fhirworx.io -u <username> --password-stdin # Login
``` echo "<GITEA_TOKEN>" | docker login gitea.homelab.fhirworx.io -u <username> --password-stdin
Tag and push: # Tag and push
```bash docker tag <image>:latest gitea.homelab.fhirworx.io/homelab/<image>:latest
docker tag <image>:latest gitea.home.fhirworx.io/homelab/<image>:latest docker push gitea.homelab.fhirworx.io/homelab/<image>:latest
docker push gitea.home.fhirworx.io/homelab/<image>:latest
``` ```
## Ports Reference ## Ports Reference
| Port | Service | Notes | | Port | Service | Notes |
|------|---------|-------| |------|---------|-------|
| 80 | Traefik HTTP | iptables redirect to 8880 | | 80 | Traefik HTTP | All subdomain routing |
| 443 | Traefik HTTPS | iptables redirect to 8443 | | 443 | Traefik HTTPS | TLS termination |
| 2222 | Gitea SSH | Direct access | | 8081 | Traefik API | Dashboard + API (localhost only) |
| 4317 | Jaeger OTLP gRPC | Direct access for external traces | | 2222 | Gitea SSH | Git over SSH |
| 4318 | Jaeger OTLP HTTP | Direct access for external traces | | 3478 | Zotero TURN | WebRTC for KasmVNC |
| 5432 | PostgreSQL | Direct access |
| 9000 | RustFS S3 API | Also via s3.home.fhirworx.io |
| 9001 | RustFS Console | Also via minio.home.fhirworx.io |