add HKDF credential derivation and auto-rotation

Single root key derives all 18 service credentials via HKDF-SHA256. Bootstrap/service tiers rotate on key change vs every commit. Provisioners update PostgreSQL roles and Gitea tokens automatically. CI pipeline runs `api.auth provision` after each deploy.
2026-02-28 22:45:14 -05:00
parent 5261032a9b
commit 2c0a69a4d8
13 changed files with 1107 additions and 60 deletions
--- a/.woodpecker/deploy.yml
+++ b/.woodpecker/deploy.yml
@@ -1,6 +1,7 @@
 # ── Deploy ───────────────────────────────────────────────────────
 # Builds the Python package, pushes it to Gitea's PyPI registry,
-# then builds, scans, and pushes container images.
+# builds, scans, and pushes container images, then updates the
+# host working copy and restarts all changed services.
 # Only runs on pushes to main (i.e. after PR merge).

 when:
@@ -146,3 +147,46 @@ steps:
      - scan-zotero
    when:
      - path: "zotero/**"
+
+  # ── Deploy: pull latest code + restart services ────────────────
+  # Runs AFTER all image builds/pushes complete.  Updates the host
+  # repo to the commit that triggered this pipeline, pulls any new
+  # registry images, and restarts changed services.
+  - name: deploy
+    image: docker:cli
+    volumes:
+      - /run/user/1000/docker.sock:/var/run/docker.sock
+      - /home/kert/stack:/home/kert/stack
+    commands:
+      # Install git so we can update the host working copy
+      - apk add --no-cache git
+      - cd /home/kert/stack
+      # Fetch the exact commit via HTTP (public repo, intrastack)
+      # and hard-reset the deploy dir to match it.  Untracked files
+      # (duckdb, zotero data, etc.) are untouched.
+      - git fetch http://gitea:3000/homelab/stack.git main
+      - git reset --hard FETCH_HEAD
+      # Pull new images from registry, restart changed services
+      - docker compose pull --ignore-buildable
+      - docker compose up -d --remove-orphans
+    depends_on:
+      - publish-package
+      - push-notebooks
+      - upload-notebooks-scan
+      - push-zotero
+      - upload-zotero-scan
+
+  # ── Provision: derive credentials and rotate backends ────────
+  - name: provision
+    image: ghcr.io/astral-sh/uv:python3.13-bookworm-slim
+    volumes:
+      - /run/user/1000/docker.sock:/var/run/docker.sock
+      - /home/kert/stack:/home/kert/stack
+    environment:
+      ROOT_KEY:
+        from_secret: root_key
+    commands:
+      - cd /home/kert/stack
+      - uv run python -m api.auth provision ${CI_COMMIT_SHA}
+    depends_on:
+      - deploy
--- a/README.md
+++ b/README.md
@@ -143,49 +143,29 @@ Add to `/etc/hosts` (replace IP with your server's LAN IP):

 ## Setup

-### 1. Environment Variables
+### 1. Generate Root Key and Bootstrap Credentials

-Create `.env` file:
+All service credentials are derived from a single 256-bit root key via HKDF-SHA256. See [Credential Management](#credential-management) for details.

 ```bash
+# Generate root key
+ROOT_KEY=$(openssl rand -hex 32)
+
+# Add non-managed vars to .env first
+cat > .env <<EOF
 DOMAIN=homelab.fhirworx.io
 HOST_IP=192.168.1.192
-
-POSTGRES_PASSWORD=<your_password>
-RUSTFS_ACCESS_KEY=<your_access_key>
-RUSTFS_SECRET_KEY=<your_secret_key>
-GITEA_S3_ACCESS_KEY=<gitea_s3_user>
-GITEA_S3_SECRET_KEY=<gitea_s3_password>
-GITEA_DB_PASSWORD=<gitea_db_password>
-GITEA_TOKEN=<gitea_api_token>
-WOODPECKER_DB_PASSWORD=<woodpecker_db_password>
 WOODPECKER_ADMIN=<admin_username>
-WOODPECKER_AGENT_SECRET=<generated_secret>
-WOODPECKER_GITEA_CLIENT=<oauth_client_id>
-WOODPECKER_GITEA_SECRET=<oauth_client_secret>
-GF_ADMIN_PASSWORD=<grafana_password>
-
-# Nessie Data Lake
-NESSIE_DB_PASSWORD=<nessie_db_password>
-NESSIE_S3_ACCESS_KEY=<nessie_s3_user>
-NESSIE_S3_SECRET_KEY=<nessie_s3_password>
-
-# Polaris Catalog
-POLARIS_DB_PASSWORD=<polaris_db_password>
-POLARIS_ROOT_SECRET=<polaris_root_secret>
-POLARIS_S3_ACCESS_KEY=<polaris_s3_user>
-POLARIS_S3_SECRET_KEY=<polaris_s3_password>
+EOF
 ```

-Generate agent secret:
-```bash
-openssl rand -hex 32
-```
-
-### 2. Start Core Services
+### 2. Start Core Services and Bootstrap

 ```bash
 docker compose up -d postgres rustfs traefik
+
+# Bootstrap: creates DB roles, databases, derives all credentials, writes .env
+ROOT_KEY=$ROOT_KEY uv run python -m api.auth bootstrap $(git rev-parse HEAD)
 ```

 ### 3. Configure RustFS
@@ -265,35 +245,13 @@ docker compose up -d postgres rustfs traefik
 }
 ```

-### 4. Configure PostgreSQL
-
-Create databases and users:
-
-```bash
-# Gitea database
-docker exec -e PGPASSWORD=<postgres_password> postgres psql -U postgres -c "CREATE DATABASE gitea;"
-docker exec -e PGPASSWORD=<postgres_password> postgres psql -U postgres -c "CREATE USER git WITH PASSWORD '<gitea_db_password>'; ALTER DATABASE gitea OWNER TO git;"
-
-# Woodpecker database
-docker exec -e PGPASSWORD=<postgres_password> postgres psql -U postgres -c "CREATE DATABASE woodpecker;"
-docker exec -e PGPASSWORD=<postgres_password> postgres psql -U postgres -c "CREATE USER woodpecker WITH PASSWORD '<woodpecker_db_password>'; ALTER DATABASE woodpecker OWNER TO woodpecker;"
-
-# Nessie database
-docker exec -e PGPASSWORD=<postgres_password> postgres psql -U postgres -c "CREATE USER nessie WITH PASSWORD '<nessie_db_password>';"
-docker exec -e PGPASSWORD=<postgres_password> postgres psql -U postgres -c "CREATE DATABASE nessie OWNER nessie;"
-
-# Polaris database
-docker exec -e PGPASSWORD=<postgres_password> postgres psql -U postgres -c "CREATE USER polaris WITH PASSWORD '<polaris_db_password>';"
-docker exec -e PGPASSWORD=<postgres_password> postgres psql -U postgres -c "CREATE DATABASE polaris OWNER polaris;"
-```
-
-### 5. Start All Services
+### 4. Start All Services

 ```bash
 docker compose up -d
 ```

-### 6. Configure Gitea
+### 5. Configure Gitea

 1. Complete initial setup at `gitea.homelab.fhirworx.io`
 2. Create organization `homelab`
@@ -304,13 +262,16 @@ docker compose up -d
   - Redirect URI: `http://ci.homelab.fhirworx.io/authorize`
   - Copy Client ID and Secret to `.env`

-### 7. Configure Woodpecker Secrets
+### 6. Configure Woodpecker Secrets

 Add secrets in Woodpecker UI (`ci.homelab.fhirworx.io`):
+- `root_key` — The hex-encoded root key (enables auto-rotation on every deploy)
 - `registry_user` — Gitea username
 - `registry_pass` — Gitea token
+- `s3_access_key` — RustFS access key
+- `s3_secret_key` — RustFS secret key

-### 8. Verify Services
+### 7. Verify Services

 ```bash
 # Check Traefik routes
@@ -359,6 +320,60 @@ All traces flow to Jaeger via OTLP gRPC (port 4317).
 | polaris | polaris:8182/q/metrics |
 | trino | trino:8080/v1/status |

+## Credential Management
+
+All service credentials are derived from a single 256-bit root key using HKDF-SHA256 (RFC 5869). No passwords are stored in `.env` — they are deterministically regenerated from the root key and a commit SHA on every deploy.
+
+### How It Works
+
+```
+ROOT_KEY (one Woodpecker secret) + commit_sha → HKDF-SHA256 → all service credentials → .env
+```
+
+The derivation uses two tiers:
+
+| Tier | Salt | Rotates | Purpose |
+|------|------|---------|---------|
+| **Bootstrap** | `b"bootstrap"` | Only when root key changes | Superuser passwords, OAuth2 app |
+| **Service** | `commit_sha` | Every deploy | DB passwords, S3 keys, API tokens |
+
+### Managed Credentials (18 derived, 2 skipped)
+
+| Variable | Tier | Format | Backend |
+|----------|------|--------|---------|
+| `POSTGRES_PASSWORD` | Bootstrap | password | env only |
+| `GITEA_DB_PASSWORD` | Service | password | PostgreSQL (role: git) |
+| `WOODPECKER_DB_PASSWORD` | Service | password | PostgreSQL (role: woodpecker) |
+| `NESSIE_DB_PASSWORD` | Service | password | PostgreSQL (role: nessie) |
+| `POLARIS_DB_PASSWORD` | Service | password | PostgreSQL (role: polaris) |
+| `RUSTFS_ACCESS_KEY` | Service | hex | env only |
+| `RUSTFS_SECRET_KEY` | Service | hex | env only |
+| `GITEA_S3_*`, `NESSIE_S3_*`, `POLARIS_S3_*` | Service | hex | env only (aliases of RUSTFS) |
+| `WOODPECKER_AGENT_SECRET` | Service | hex | env only |
+| `POLARIS_ROOT_SECRET` | Service | password | env only |
+| `GF_ADMIN_PASSWORD` | Service | password | env only |
+| `GITEA_ADMIN_PASSWORD` | Service | password | Gitea (change-password) |
+| `GITEA_TOKEN` | Service | — | Gitea (API token) |
+
+Not managed: `WOODPECKER_GITEA_CLIENT/SECRET` (created once via OAuth2 app), `DATABRICKS_TOKEN` (external).
+
+### CLI Usage
+
+```bash
+# First-time setup: create DB roles + databases, derive all creds, write .env
+ROOT_KEY=$KEY uv run python -m api.auth bootstrap $(git rev-parse HEAD)
+
+# Rotate: derive new service credentials, update backends, rewrite .env
+ROOT_KEY=$KEY uv run python -m api.auth provision $COMMIT_SHA
+
+# Dry-run: print derived values without provisioning
+ROOT_KEY=$KEY uv run python -m api.auth derive $COMMIT_SHA --redact
+```
+
+### Auto-Rotation via CI
+
+The `provision` step in `.woodpecker/deploy.yml` runs after every deploy, deriving fresh service-tier credentials from the commit SHA and rotating PostgreSQL passwords and Gitea tokens automatically.
+
 ## Data Lakehouse

 | Component | Purpose |
@@ -455,7 +470,8 @@ stack/
 │   └── etc/                     # Trino config + catalog properties
 ├── notebooks/                   # Marimo notebook files
 ├── zotero/                      # Zotero data + profiles
-├── src/                         # Python source (narwhals expressions, pipes)
+├── src/                         # Python source
+│   └── api/auth/                # HKDF credential derivation + provisioning
 ├── tests/                       # Test suite
 ├── .woodpecker/                 # CI/CD pipeline definitions
 │   ├── ci.yml
--- a/src/api/auth/init.py
+++ b/src/api/auth/init.py
@@ -0,0 +1,20 @@
+"""HKDF credential derivation and auto-rotation."""
+
+from api.auth.derive import derive, derive_hex, derive_password
+from api.auth.manifest import CREDENTIALS, MANAGED_VARS, Credential, Format, Tier
+from api.auth.provision import bootstrap, derive_all, provision, write_env
+
+__all__ = [
+    "CREDENTIALS",
+    "Credential",
+    "Format",
+    "MANAGED_VARS",
+    "Tier",
+    "bootstrap",
+    "derive",
+    "derive_all",
+    "derive_hex",
+    "derive_password",
+    "provision",
+    "write_env",
+]
--- a/src/api/auth/main.py
+++ b/src/api/auth/main.py
@@ -0,0 +1,76 @@
+"""CLI for credential derivation and provisioning.
+
+Usage:
+    uv run python -m api.auth bootstrap <commit_sha>
+    uv run python -m api.auth provision <commit_sha>
+    uv run python -m api.auth derive <commit_sha> [--redact]
+"""
+
+from __future__ import annotations
+
+import os
+import sys
+from pathlib import Path
+
+
+def main() -> int:
+    args = sys.argv[1:]
+    if len(args) < 2 or args[0] not in ("bootstrap", "provision", "derive"):
+        print(__doc__.strip(), file=sys.stderr)
+        return 1
+
+    command = args[0]
+    commit_sha = args[1]
+    redact = "--redact" in args
+
+    root_hex = os.environ.get("ROOT_KEY", "")
+    if not root_hex:
+        print("ERROR: ROOT_KEY environment variable not set", file=sys.stderr)
+        return 1
+
+    try:
+        root_key = bytes.fromhex(root_hex)
+    except ValueError:
+        print("ERROR: ROOT_KEY must be hex-encoded", file=sys.stderr)
+        return 1
+
+    if len(root_key) < 16:
+        print("ERROR: ROOT_KEY too short (need >= 128 bits)", file=sys.stderr)
+        return 1
+
+    env_path = Path(".env")
+
+    if command == "derive":
+        from api.auth.provision import derive_all
+
+        values = derive_all(root_key, commit_sha)
+        for k, v in sorted(values.items()):
+            display = v[:4] + "..." if redact else v
+            print(f"{k}={display}")
+        return 0
+
+    if command == "provision":
+        from api.auth.provision import provision
+
+        provision(root_key, commit_sha, env_path)
+        print(f"Provisioned {len(derive_all_count(root_key, commit_sha))} credentials")
+        return 0
+
+    if command == "bootstrap":
+        from api.auth.provision import bootstrap
+
+        bootstrap(root_key, commit_sha, env_path)
+        print("Bootstrap complete")
+        return 0
+
+    return 1
+
+
+def derive_all_count(root_key: bytes, commit_sha: str) -> dict[str, str]:
+    from api.auth.provision import derive_all
+
+    return derive_all(root_key, commit_sha)
+
+
+if __name__ == "__main__":
+    sys.exit(main())
--- a/src/api/auth/derive.py
+++ b/src/api/auth/derive.py
@@ -0,0 +1,54 @@
+"""HKDF-SHA256 credential derivation using only stdlib."""
+
+from __future__ import annotations
+
+import base64
+import hashlib
+import hmac
+import math
+
+
+def _hkdf_extract(salt: bytes, ikm: bytes) -> bytes:
+    """HKDF-Extract (RFC 5869 Section 2.2)."""
+    return hmac.new(salt, ikm, hashlib.sha256).digest()
+
+
+def _hkdf_expand(prk: bytes, info: bytes, length: int) -> bytes:
+    """HKDF-Expand (RFC 5869 Section 2.3)."""
+    hash_len = 32  # SHA-256
+    n = math.ceil(length / hash_len)
+    okm = b""
+    t = b""
+    for i in range(1, n + 1):
+        t = hmac.new(prk, t + info + bytes([i]), hashlib.sha256).digest()
+        okm += t
+    return okm[:length]
+
+
+def derive(root_key: bytes, salt: bytes, info: bytes, length: int = 32) -> bytes:
+    """Derive keying material via HKDF-SHA256.
+
+    Parameters
+    ----------
+    root_key : bytes
+        Input keying material (256-bit recommended).
+    salt : bytes
+        Non-secret salt (tier identifier or commit SHA).
+    info : bytes
+        Context string (credential scope).
+    length : int
+        Output length in bytes (max 255 * 32).
+    """
+    prk = _hkdf_extract(salt, root_key)
+    return _hkdf_expand(prk, info, length)
+
+
+def derive_hex(root_key: bytes, salt: bytes, info: bytes, length: int = 32) -> str:
+    """Derive a hex-encoded credential string."""
+    return derive(root_key, salt, info, length).hex()
+
+
+def derive_password(root_key: bytes, salt: bytes, info: bytes, length: int = 24) -> str:
+    """Derive a URL-safe base64 password (no padding)."""
+    raw = derive(root_key, salt, info, length)
+    return base64.urlsafe_b64encode(raw).decode().rstrip("=")
--- a/src/api/auth/manifest.py
+++ b/src/api/auth/manifest.py
@@ -0,0 +1,199 @@
+"""Declarative credential registry."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from enum import Enum, auto
+
+
+class Tier(Enum):
+    BOOTSTRAP = auto()
+    SERVICE = auto()
+
+
+class Format(Enum):
+    HEX = auto()
+    PASSWORD = auto()
+
+
+class Provisioner(Enum):
+    ENV_ONLY = auto()
+    POSTGRES = auto()
+    GITEA = auto()
+    SKIP = auto()
+
+
+@dataclass(frozen=True, slots=True)
+class Credential:
+    env_var: str
+    scope: str
+    tier: Tier
+    fmt: Format
+    provisioner: Provisioner
+    alias_of: str | None = None
+
+
+CREDENTIALS: tuple[Credential, ...] = (
+    # ── PostgreSQL ───────────────────────────────────────────
+    Credential(
+        "POSTGRES_PASSWORD",
+        "postgres/superuser",
+        Tier.BOOTSTRAP,
+        Format.PASSWORD,
+        Provisioner.ENV_ONLY,
+    ),
+    Credential(
+        "GITEA_DB_PASSWORD",
+        "postgres/gitea",
+        Tier.SERVICE,
+        Format.PASSWORD,
+        Provisioner.POSTGRES,
+    ),
+    Credential(
+        "WOODPECKER_DB_PASSWORD",
+        "postgres/woodpecker",
+        Tier.SERVICE,
+        Format.PASSWORD,
+        Provisioner.POSTGRES,
+    ),
+    Credential(
+        "NESSIE_DB_PASSWORD",
+        "postgres/nessie",
+        Tier.SERVICE,
+        Format.PASSWORD,
+        Provisioner.POSTGRES,
+    ),
+    Credential(
+        "POLARIS_DB_PASSWORD",
+        "postgres/polaris",
+        Tier.SERVICE,
+        Format.PASSWORD,
+        Provisioner.POSTGRES,
+    ),
+    # ── RustFS S3 ────────────────────────────────────────────
+    Credential(
+        "RUSTFS_ACCESS_KEY",
+        "rustfs/access",
+        Tier.SERVICE,
+        Format.HEX,
+        Provisioner.ENV_ONLY,
+    ),
+    Credential(
+        "RUSTFS_SECRET_KEY",
+        "rustfs/secret",
+        Tier.SERVICE,
+        Format.HEX,
+        Provisioner.ENV_ONLY,
+    ),
+    Credential(
+        "GITEA_S3_ACCESS_KEY",
+        "rustfs/access",
+        Tier.SERVICE,
+        Format.HEX,
+        Provisioner.ENV_ONLY,
+        alias_of="RUSTFS_ACCESS_KEY",
+    ),
+    Credential(
+        "GITEA_S3_SECRET_KEY",
+        "rustfs/secret",
+        Tier.SERVICE,
+        Format.HEX,
+        Provisioner.ENV_ONLY,
+        alias_of="RUSTFS_SECRET_KEY",
+    ),
+    Credential(
+        "NESSIE_S3_ACCESS_KEY",
+        "rustfs/access",
+        Tier.SERVICE,
+        Format.HEX,
+        Provisioner.ENV_ONLY,
+        alias_of="RUSTFS_ACCESS_KEY",
+    ),
+    Credential(
+        "NESSIE_S3_SECRET_KEY",
+        "rustfs/secret",
+        Tier.SERVICE,
+        Format.HEX,
+        Provisioner.ENV_ONLY,
+        alias_of="RUSTFS_SECRET_KEY",
+    ),
+    Credential(
+        "POLARIS_S3_ACCESS_KEY",
+        "rustfs/access",
+        Tier.SERVICE,
+        Format.HEX,
+        Provisioner.ENV_ONLY,
+        alias_of="RUSTFS_ACCESS_KEY",
+    ),
+    Credential(
+        "POLARIS_S3_SECRET_KEY",
+        "rustfs/secret",
+        Tier.SERVICE,
+        Format.HEX,
+        Provisioner.ENV_ONLY,
+        alias_of="RUSTFS_SECRET_KEY",
+    ),
+    # ── Woodpecker ───────────────────────────────────────────
+    Credential(
+        "WOODPECKER_AGENT_SECRET",
+        "woodpecker/agent",
+        Tier.SERVICE,
+        Format.HEX,
+        Provisioner.ENV_ONLY,
+    ),
+    Credential(
+        "WOODPECKER_GITEA_CLIENT",
+        "woodpecker/oauth-client",
+        Tier.BOOTSTRAP,
+        Format.HEX,
+        Provisioner.SKIP,
+    ),
+    Credential(
+        "WOODPECKER_GITEA_SECRET",
+        "woodpecker/oauth-secret",
+        Tier.BOOTSTRAP,
+        Format.HEX,
+        Provisioner.SKIP,
+    ),
+    # ── Others ───────────────────────────────────────────────
+    Credential(
+        "POLARIS_ROOT_SECRET",
+        "polaris/root",
+        Tier.SERVICE,
+        Format.PASSWORD,
+        Provisioner.ENV_ONLY,
+    ),
+    Credential(
+        "GF_ADMIN_PASSWORD",
+        "grafana/admin",
+        Tier.SERVICE,
+        Format.PASSWORD,
+        Provisioner.ENV_ONLY,
+    ),
+    Credential(
+        "GITEA_ADMIN_PASSWORD",
+        "gitea/admin",
+        Tier.SERVICE,
+        Format.PASSWORD,
+        Provisioner.GITEA,
+    ),
+    Credential(
+        "GITEA_TOKEN", "gitea/token", Tier.SERVICE, Format.PASSWORD, Provisioner.GITEA
+    ),
+)
+
+MANAGED_VARS: frozenset[str] = frozenset(c.env_var for c in CREDENTIALS)
+
+POSTGRES_ROLES: dict[str, str] = {
+    "git": "GITEA_DB_PASSWORD",
+    "woodpecker": "WOODPECKER_DB_PASSWORD",
+    "nessie": "NESSIE_DB_PASSWORD",
+    "polaris": "POLARIS_DB_PASSWORD",
+}
+
+POSTGRES_DATABASES: dict[str, str] = {
+    "git": "gitea",
+    "woodpecker": "woodpecker",
+    "nessie": "nessie",
+    "polaris": "polaris",
+}
--- a/src/api/auth/provision.py
+++ b/src/api/auth/provision.py
@@ -0,0 +1,242 @@
+"""Credential derivation, provisioning, and .env generation."""
+
+from __future__ import annotations
+
+import base64
+import subprocess
+import time
+from pathlib import Path
+
+from api.auth.derive import derive_hex, derive_password
+from api.auth.manifest import (
+    CREDENTIALS,
+    POSTGRES_DATABASES,
+    POSTGRES_ROLES,
+    Credential,
+    Format,
+    Provisioner,
+    Tier,
+)
+
+BOOTSTRAP_SALT = b"bootstrap"
+
+
+def _derive_one(cred: Credential, root_key: bytes, salt: bytes) -> str:
+    info = cred.scope.encode()
+    if cred.fmt is Format.HEX:
+        return derive_hex(root_key, salt, info)
+    return derive_password(root_key, salt, info)
+
+
+def derive_all(root_key: bytes, commit_sha: str) -> dict[str, str]:
+    """Derive all credentials from root key and commit SHA."""
+    service_salt = commit_sha.encode()
+    values: dict[str, str] = {}
+    for cred in CREDENTIALS:
+        if cred.provisioner is Provisioner.SKIP:
+            continue
+        salt = BOOTSTRAP_SALT if cred.tier is Tier.BOOTSTRAP else service_salt
+        values[cred.env_var] = _derive_one(cred, root_key, salt)
+    return values
+
+
+def write_env(values: dict[str, str], path: Path) -> None:
+    """Merge derived values into .env, preserving unmanaged keys."""
+    existing: dict[str, str] = {}
+    if path.exists():
+        for line in path.read_text().splitlines():
+            line = line.strip()
+            if not line or line.startswith("#"):
+                continue
+            if "=" in line:
+                k, _, v = line.partition("=")
+                existing[k.strip()] = v.strip()
+
+    existing.update(values)
+    lines = [f"{k}={v}" for k, v in sorted(existing.items())]
+    path.write_text("\n".join(lines) + "\n")
+
+
+def provision_postgres(values: dict[str, str], *, container: str = "postgres") -> None:
+    """Rotate passwords for all managed PostgreSQL roles."""
+    for role, env_var in POSTGRES_ROLES.items():
+        pw = values[env_var]
+        subprocess.run(
+            [
+                "docker",
+                "exec",
+                container,
+                "psql",
+                "-U",
+                "postgres",
+                "-c",
+                f"ALTER ROLE {role} PASSWORD '{pw}'",
+            ],
+            check=True,
+            capture_output=True,
+        )
+
+
+def bootstrap_postgres(values: dict[str, str], *, container: str = "postgres") -> None:
+    """Idempotent first-time setup: create roles and databases."""
+    superuser_pw = values["POSTGRES_PASSWORD"]
+    subprocess.run(
+        [
+            "docker",
+            "exec",
+            container,
+            "psql",
+            "-U",
+            "postgres",
+            "-c",
+            f"ALTER ROLE postgres PASSWORD '{superuser_pw}'",
+        ],
+        check=True,
+        capture_output=True,
+    )
+    for role, env_var in POSTGRES_ROLES.items():
+        pw = values[env_var]
+        db = POSTGRES_DATABASES[role]
+        sql = (
+            f"DO $$ BEGIN "
+            f"IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname='{role}') "
+            f"THEN CREATE ROLE {role} LOGIN PASSWORD '{pw}'; END IF; END $$; "
+            f"SELECT 'CREATE DATABASE {db} OWNER {role}' "
+            f"WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname='{db}');"
+        )
+        subprocess.run(
+            [
+                "docker",
+                "exec",
+                container,
+                "psql",
+                "-U",
+                "postgres",
+                "-c",
+                sql,
+            ],
+            check=True,
+            capture_output=True,
+        )
+        # Create database if it doesn't exist (psql doesn't support
+        # IF NOT EXISTS for CREATE DATABASE inside DO blocks).
+        subprocess.run(
+            [
+                "docker",
+                "exec",
+                container,
+                "psql",
+                "-U",
+                "postgres",
+                "-c",
+                f"CREATE DATABASE {db} OWNER {role}",
+            ],
+            capture_output=True,
+        )
+
+
+def provision_gitea(
+    values: dict[str, str],
+    *,
+    container: str = "gitea",
+    admin_user: str = "kert",
+) -> str:
+    """Rotate Gitea admin password and create a fresh API token."""
+    pw = values["GITEA_ADMIN_PASSWORD"]
+
+    # Change admin password
+    subprocess.run(
+        [
+            "docker",
+            "exec",
+            container,
+            "gitea",
+            "admin",
+            "user",
+            "change-password",
+            "-u",
+            admin_user,
+            "-p",
+            pw,
+        ],
+        check=True,
+        capture_output=True,
+    )
+
+    # Delete stale deploy tokens then create a fresh one, via API
+    auth_header = _basic_header(admin_user, pw)
+    client = _make_gitea_client(container, auth_header)
+
+    # List and delete stale deploy-* tokens
+    tokens = client.get(
+        f"/users/{admin_user}/tokens",
+        headers={"Authorization": auth_header},
+    ).json()
+    for tok in tokens:
+        if tok.get("name", "").startswith("deploy-"):
+            client.delete(
+                f"/users/{admin_user}/tokens/{tok['id']}",
+                headers={"Authorization": auth_header},
+            )
+
+    # Create fresh token
+    token_name = f"deploy-{int(time.time())}"
+    resp = client.post(
+        f"/users/{admin_user}/tokens",
+        json={"name": token_name, "scopes": ["all"]},
+        headers={"Authorization": auth_header},
+    ).json()
+    return resp["sha1"]
+
+
+def _basic_header(user: str, password: str) -> str:
+    encoded = base64.b64encode(f"{user}:{password}".encode()).decode()
+    return f"Basic {encoded}"
+
+
+def _make_gitea_client(container: str, auth_header: str):
+    from api.clients.gitea import GiteaClient
+
+    return GiteaClient(
+        "unused",
+        base_url=f"http://{container}:3000/api/v1",
+    )
+
+
+def provision(
+    root_key: bytes,
+    commit_sha: str,
+    env_path: Path,
+    *,
+    skip_backends: bool = False,
+) -> dict[str, str]:
+    """Main entry: derive, provision backends, write .env."""
+    values = derive_all(root_key, commit_sha)
+
+    if not skip_backends:
+        provision_postgres(values)
+        token = provision_gitea(values)
+        values["GITEA_TOKEN"] = token
+
+    write_env(values, env_path)
+    return values
+
+
+def bootstrap(
+    root_key: bytes,
+    commit_sha: str,
+    env_path: Path,
+    *,
+    skip_backends: bool = False,
+) -> dict[str, str]:
+    """First-time setup: create DBs, roles, then provision."""
+    values = derive_all(root_key, commit_sha)
+
+    if not skip_backends:
+        bootstrap_postgres(values)
+        provision_postgres(values)
+        token = provision_gitea(values)
+        values["GITEA_TOKEN"] = token
+
+    write_env(values, env_path)
+    return values
--- a/src/api/clients/gitea/client.py
+++ b/src/api/clients/gitea/client.py
@@ -60,3 +60,27 @@ class GiteaClient(Client):

    def create_webhook(self, owner: str, repo: str, body: dict) -> dict:
        return self.post(f"/repos/{owner}/{repo}/hooks", json=body).json()
+
+    # ── Admin ──────────────────────────────────────────────
+
+    def change_admin_password(self, username: str, password: str) -> dict:
+        return self.patch(
+            f"/admin/users/{username}",
+            json={"password": password, "must_change_password": False},
+        ).json()
+
+    # ── Tokens ─────────────────────────────────────────────
+
+    def list_tokens(self, username: str) -> list[dict]:
+        return self.get(f"/users/{username}/tokens").json()
+
+    def create_token(
+        self, username: str, name: str, scopes: list[str] | None = None
+    ) -> dict:
+        body: dict = {"name": name}
+        if scopes is not None:
+            body["scopes"] = scopes
+        return self.post(f"/users/{username}/tokens", json=body).json()
+
+    def delete_token(self, username: str, token_id: int) -> None:
+        self.delete(f"/users/{username}/tokens/{token_id}")
--- a/src/api/clients/woodpecker/client.py
+++ b/src/api/clients/woodpecker/client.py
@@ -56,6 +56,23 @@ class WoodpeckerClient(Client):
    def create_secret(self, repo_id: int, body: dict) -> dict:
        return self.post(f"/repos/{repo_id}/secrets", json=body).json()

+    def update_secret(self, repo_id: int, name: str, body: dict) -> dict:
+        return self.patch(f"/repos/{repo_id}/secrets/{name}", json=body).json()
+
+    def delete_secret(self, repo_id: int, name: str) -> None:
+        self.delete(f"/repos/{repo_id}/secrets/{name}")
+
+    # ── Global secrets ─────────────────────────────────────
+
+    def list_global_secrets(self) -> list[dict]:
+        return self.get("/secrets").json()
+
+    def create_global_secret(self, body: dict) -> dict:
+        return self.post("/secrets", json=body).json()
+
+    def update_global_secret(self, name: str, body: dict) -> dict:
+        return self.patch(f"/secrets/{name}", json=body).json()
+
    # ── Server ───────────────────────────────────────────────

    def version(self) -> dict:
--- a/tests/api/test_derive.py
+++ b/tests/api/test_derive.py
@@ -0,0 +1,92 @@
+"""Tests for HKDF credential derivation."""
+
+from __future__ import annotations
+
+from api.auth.derive import derive, derive_hex, derive_password
+
+ROOT = bytes.fromhex("0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b")
+SALT = b"test-salt"
+INFO = b"test-info"
+
+
+class TestDerive:
+    def test_deterministic(self):
+        a = derive(ROOT, SALT, INFO)
+        b = derive(ROOT, SALT, INFO)
+        assert a == b
+
+    def test_output_length(self):
+        assert len(derive(ROOT, SALT, INFO, 16)) == 16
+        assert len(derive(ROOT, SALT, INFO, 32)) == 32
+        assert len(derive(ROOT, SALT, INFO, 64)) == 64
+
+    def test_different_salt_different_output(self):
+        a = derive(ROOT, b"salt-a", INFO)
+        b = derive(ROOT, b"salt-b", INFO)
+        assert a != b
+
+    def test_different_info_different_output(self):
+        a = derive(ROOT, SALT, b"info-a")
+        b = derive(ROOT, SALT, b"info-b")
+        assert a != b
+
+    def test_different_key_different_output(self):
+        key2 = bytes(32)
+        a = derive(ROOT, SALT, INFO)
+        b = derive(key2, SALT, INFO)
+        assert a != b
+
+
+class TestDeriveHex:
+    def test_hex_format(self):
+        h = derive_hex(ROOT, SALT, INFO)
+        assert len(h) == 64
+        bytes.fromhex(h)  # must not raise
+
+    def test_hex_deterministic(self):
+        assert derive_hex(ROOT, SALT, INFO) == derive_hex(ROOT, SALT, INFO)
+
+    def test_hex_matches_raw(self):
+        assert derive_hex(ROOT, SALT, INFO) == derive(ROOT, SALT, INFO).hex()
+
+    def test_hex_custom_length(self):
+        h = derive_hex(ROOT, SALT, INFO, length=16)
+        assert len(h) == 32
+
+
+class TestDerivePassword:
+    def test_password_url_safe(self):
+        pw = derive_password(ROOT, SALT, INFO)
+        assert "+" not in pw
+        assert "/" not in pw
+        assert "=" not in pw
+
+    def test_password_deterministic(self):
+        a = derive_password(ROOT, SALT, INFO)
+        b = derive_password(ROOT, SALT, INFO)
+        assert a == b
+
+    def test_password_length(self):
+        pw = derive_password(ROOT, SALT, INFO, length=24)
+        assert len(pw) == 32  # 24 bytes -> 32 base64url chars
+
+    def test_password_different_from_hex(self):
+        pw = derive_password(ROOT, SALT, INFO)
+        h = derive_hex(ROOT, SALT, INFO)
+        assert pw != h
+
+
+class TestRFCVector:
+    """Verify against RFC 5869 Test Case 1."""
+
+    def test_rfc5869_case1(self):
+        ikm = bytes.fromhex("0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b")
+        salt = bytes.fromhex("000102030405060708090a0b0c")
+        info = bytes.fromhex("f0f1f2f3f4f5f6f7f8f9")
+        expected = bytes.fromhex(
+            "3cb25f25faacd57a90434f64d0362f2a"
+            "2d2d0a90cf1a5a4c5db02d56ecc4c5bf"
+            "34007208d5b887185865"
+        )
+        result = derive(ikm, salt, info, length=42)
+        assert result == expected
--- a/tests/api/test_gitea.py
+++ b/tests/api/test_gitea.py
@@ -75,3 +75,35 @@ class TestGiteaRoutes:
        req = cap.requests[0]
        assert req.method == "GET"
        assert req.url.path == "/api/v1/repos/o/r/hooks"
+
+    def test_change_admin_password(self, capture_transport):
+        cap = capture_transport
+        c = GiteaClient("t", _transport=cap.transport({}))
+        c.change_admin_password("kert", "newpw")
+        req = cap.requests[0]
+        assert req.method == "PATCH"
+        assert req.url.path == "/api/v1/admin/users/kert"
+
+    def test_list_tokens(self, capture_transport):
+        cap = capture_transport
+        c = GiteaClient("t", _transport=cap.transport([]))
+        c.list_tokens("kert")
+        req = cap.requests[0]
+        assert req.method == "GET"
+        assert req.url.path == "/api/v1/users/kert/tokens"
+
+    def test_create_token(self, capture_transport):
+        cap = capture_transport
+        c = GiteaClient("t", _transport=cap.transport({"sha1": "abc"}))
+        c.create_token("kert", "deploy", scopes=["all"])
+        req = cap.requests[0]
+        assert req.method == "POST"
+        assert req.url.path == "/api/v1/users/kert/tokens"
+
+    def test_delete_token(self, capture_transport):
+        cap = capture_transport
+        c = GiteaClient("t", _transport=cap.transport(None, status=204))
+        c.delete_token("kert", 42)
+        req = cap.requests[0]
+        assert req.method == "DELETE"
+        assert req.url.path == "/api/v1/users/kert/tokens/42"
--- a/tests/api/test_provision.py
+++ b/tests/api/test_provision.py
@@ -0,0 +1,191 @@
+"""Tests for credential provisioning."""
+
+from __future__ import annotations
+
+from unittest.mock import patch
+
+from api.auth.manifest import (
+    CREDENTIALS,
+    MANAGED_VARS,
+    POSTGRES_DATABASES,
+    POSTGRES_ROLES,
+    Format,
+    Provisioner,
+    Tier,
+)
+from api.auth.provision import derive_all, write_env
+
+ROOT = bytes.fromhex("deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef")
+COMMIT = "abc1234"
+
+
+class TestManifestInvariants:
+    def test_no_duplicate_env_vars(self):
+        env_vars = [c.env_var for c in CREDENTIALS]
+        assert len(env_vars) == len(set(env_vars))
+
+    def test_all_managed_vars_in_credentials(self):
+        assert MANAGED_VARS == frozenset(c.env_var for c in CREDENTIALS)
+
+    def test_postgres_roles_have_credentials(self):
+        for env_var in POSTGRES_ROLES.values():
+            assert env_var in MANAGED_VARS
+
+    def test_postgres_roles_have_databases(self):
+        for role in POSTGRES_ROLES:
+            assert role in POSTGRES_DATABASES
+
+    def test_aliases_share_scope(self):
+        by_var = {c.env_var: c for c in CREDENTIALS}
+        for cred in CREDENTIALS:
+            if cred.alias_of:
+                parent = by_var[cred.alias_of]
+                assert cred.scope == parent.scope
+                assert cred.fmt == parent.fmt
+
+    def test_credential_count(self):
+        assert len(CREDENTIALS) == 20
+
+
+class TestDeriveAll:
+    def test_returns_dict(self):
+        values = derive_all(ROOT, COMMIT)
+        assert isinstance(values, dict)
+
+    def test_skips_skip_provisioner(self):
+        values = derive_all(ROOT, COMMIT)
+        skip_vars = {
+            c.env_var for c in CREDENTIALS if c.provisioner is Provisioner.SKIP
+        }
+        for var in skip_vars:
+            assert var not in values
+
+    def test_hex_format_values(self):
+        values = derive_all(ROOT, COMMIT)
+        for cred in CREDENTIALS:
+            if cred.provisioner is Provisioner.SKIP:
+                continue
+            if cred.fmt is Format.HEX:
+                v = values[cred.env_var]
+                assert len(v) == 64
+                bytes.fromhex(v)
+
+    def test_password_format_values(self):
+        values = derive_all(ROOT, COMMIT)
+        for cred in CREDENTIALS:
+            if cred.provisioner is Provisioner.SKIP:
+                continue
+            if cred.fmt is Format.PASSWORD:
+                v = values[cred.env_var]
+                assert "+" not in v
+                assert "/" not in v
+
+    def test_deterministic(self):
+        a = derive_all(ROOT, COMMIT)
+        b = derive_all(ROOT, COMMIT)
+        assert a == b
+
+    def test_different_commit_different_service_values(self):
+        a = derive_all(ROOT, "commit-a")
+        b = derive_all(ROOT, "commit-b")
+        # Bootstrap values should be the same
+        for cred in CREDENTIALS:
+            if cred.tier is Tier.BOOTSTRAP and cred.provisioner is not Provisioner.SKIP:
+                assert a[cred.env_var] == b[cred.env_var]
+        # At least one service value should differ
+        service_vars = [
+            c.env_var
+            for c in CREDENTIALS
+            if c.tier is Tier.SERVICE
+            and c.provisioner is not Provisioner.SKIP
+            and c.alias_of is None
+        ]
+        assert any(a[v] != b[v] for v in service_vars)
+
+    def test_aliases_match_parent(self):
+        values = derive_all(ROOT, COMMIT)
+        for cred in CREDENTIALS:
+            if cred.alias_of and cred.provisioner is not Provisioner.SKIP:
+                assert values[cred.env_var] == values[cred.alias_of]
+
+
+class TestWriteEnv:
+    def test_creates_new_file(self, tmp_path):
+        p = tmp_path / ".env"
+        write_env({"A": "1", "B": "2"}, p)
+        lines = p.read_text().splitlines()
+        assert lines == ["A=1", "B=2"]
+
+    def test_preserves_unmanaged_keys(self, tmp_path):
+        p = tmp_path / ".env"
+        p.write_text("DATABRICKS_TOKEN=ext\nOLD_KEY=val\n")
+        write_env({"NEW_KEY": "new"}, p)
+        content = p.read_text()
+        assert "DATABRICKS_TOKEN=ext" in content
+        assert "NEW_KEY=new" in content
+        assert "OLD_KEY=val" in content
+
+    def test_overwrites_managed_keys(self, tmp_path):
+        p = tmp_path / ".env"
+        p.write_text("MY_KEY=old\n")
+        write_env({"MY_KEY": "new"}, p)
+        lines = p.read_text().splitlines()
+        assert "MY_KEY=new" in lines
+        assert "MY_KEY=old" not in lines
+
+    def test_sorted_output(self, tmp_path):
+        p = tmp_path / ".env"
+        write_env({"Z": "3", "A": "1", "M": "2"}, p)
+        lines = p.read_text().splitlines()
+        keys = [l.split("=")[0] for l in lines]
+        assert keys == sorted(keys)
+
+    def test_skips_comments_and_blanks(self, tmp_path):
+        p = tmp_path / ".env"
+        p.write_text("# comment\n\nKEY=val\n")
+        write_env({"NEW": "v"}, p)
+        content = p.read_text()
+        assert "KEY=val" in content
+        assert "NEW=v" in content
+        # Comments are not preserved (by design)
+        assert "# comment" not in content
+
+
+class TestProvisionPostgres:
+    def test_calls_docker_exec(self):
+        values = derive_all(ROOT, COMMIT)
+        with patch("api.auth.provision.subprocess.run") as mock_run:
+            from api.auth.provision import provision_postgres
+
+            provision_postgres(values, container="test-pg")
+
+        assert mock_run.call_count == len(POSTGRES_ROLES)
+        for call in mock_run.call_args_list:
+            cmd = call[0][0]
+            assert cmd[:3] == ["docker", "exec", "test-pg"]
+            assert "ALTER ROLE" in cmd[-1]
+
+
+class TestProvisionGitea:
+    def test_calls_change_password(self):
+        values = derive_all(ROOT, COMMIT)
+        mock_client = type("C", (), {})()
+        mock_client.get = lambda *a, **kw: type("R", (), {"json": lambda self: []})()
+        mock_client.post = lambda *a, **kw: type(
+            "R", (), {"json": lambda self: {"sha1": "tok123"}}
+        )()
+        with (
+            patch("api.auth.provision.subprocess.run") as mock_run,
+            patch(
+                "api.auth.provision._make_gitea_client",
+                return_value=mock_client,
+            ),
+        ):
+            from api.auth.provision import provision_gitea
+
+            token = provision_gitea(values, container="test-gitea")
+
+        assert token == "tok123"
+        assert mock_run.call_count == 1
+        cmd = mock_run.call_args[0][0]
+        assert "change-password" in cmd
--- a/tests/api/test_woodpecker.py
+++ b/tests/api/test_woodpecker.py
@@ -76,3 +76,43 @@ class TestWoodpeckerRoutes:
        req = cap.requests[0]
        assert req.method == "GET"
        assert req.url.path == "/api/repos/10/secrets"
+
+    def test_update_secret(self, capture_transport):
+        cap = capture_transport
+        c = WoodpeckerClient("t", _transport=cap.transport({}))
+        c.update_secret(10, "my_secret", {"value": "new"})
+        req = cap.requests[0]
+        assert req.method == "PATCH"
+        assert req.url.path == "/api/repos/10/secrets/my_secret"
+
+    def test_delete_secret(self, capture_transport):
+        cap = capture_transport
+        c = WoodpeckerClient("t", _transport=cap.transport(None, status=204))
+        c.delete_secret(10, "old_secret")
+        req = cap.requests[0]
+        assert req.method == "DELETE"
+        assert req.url.path == "/api/repos/10/secrets/old_secret"
+
+    def test_list_global_secrets(self, capture_transport):
+        cap = capture_transport
+        c = WoodpeckerClient("t", _transport=cap.transport([]))
+        c.list_global_secrets()
+        req = cap.requests[0]
+        assert req.method == "GET"
+        assert req.url.path == "/api/secrets"
+
+    def test_create_global_secret(self, capture_transport):
+        cap = capture_transport
+        c = WoodpeckerClient("t", _transport=cap.transport({}))
+        c.create_global_secret({"name": "s", "value": "v"})
+        req = cap.requests[0]
+        assert req.method == "POST"
+        assert req.url.path == "/api/secrets"
+
+    def test_update_global_secret(self, capture_transport):
+        cap = capture_transport
+        c = WoodpeckerClient("t", _transport=cap.transport({}))
+        c.update_global_secret("root_key", {"value": "new"})
+        req = cap.requests[0]
+        assert req.method == "PATCH"
+        assert req.url.path == "/api/secrets/root_key"