two-phase deploy: rotate backends FIRST, then restart services

New api.auth.deploy module implements the correct rotation order:
1. derive_all() → new passwords
2. provision_postgres() → ALTER ROLE (backend first)
3. provision_gitea() → rotate token via API
4. write_env() → write .env with matching passwords
5. docker compose up -d → services read correct passwords
6. provision_woodpecker() → sync CI secrets
7. verify_all() → health check postgres/gitea/rustfs
8. rollback to .env.bak if health checks fail

Merged deploy + provision into single Woodpecker step to eliminate
the race between git reset and .env writes.

CLI: uv run python -m api.auth deploy <sha>
CLI: uv run python -m api.auth verify <sha>

13 tests covering health checks, two-phase rotation, rollback,
and partial failure scenarios.

fix #119 fix #120 fix #121 fix #122

.woodpecker/deploy.yml

@@ -245,15 +245,26 @@ steps:
       - scan-docs
       - scan-api
 
-  # ── Deploy: pull latest code + restart services ────────────────
+  # ── Deploy + Provision (two-phase) ─────────────────────────────
+  # Phase 1: rotate backend passwords (ALTER ROLE, Gitea API)
+  # Phase 2: write .env + docker compose up (services read matching passwords)
+  # Phase 3: sync Woodpecker secrets + health check + rollback on failure
   - name: deploy
-    image: docker:cli
+    image: ghcr.io/astral-sh/uv:python3.13-bookworm-slim
     volumes:
       - /run/user/1000/docker.sock:/var/run/docker.sock
       - /home/kert/stack:/home/kert/stack
+    environment:
+      UV_PYTHON_PREFERENCE: only-system
+      UV_LINK_MODE: copy
+      UV_PROJECT_ENVIRONMENT: .venv
+      ROOT_KEY:
+        from_secret: root_key
     commands:
-      - apk add --no-cache git
+      # Install docker CLI for compose + docker exec
+      - apt-get update -qq && apt-get install -y -qq docker.io git >/dev/null 2>&1
       - cd /home/kert/stack
+      # Pull latest code, preserve .env
       - cp .env .env.bak 2>/dev/null || true
       - git fetch http://gitea:3000/homelab/stack.git main
       - git reset --hard FETCH_HEAD
@@ -265,8 +276,9 @@ steps:
           docker pull $FQDN/homelab/$SVC:$TAG &&
           docker tag  $FQDN/homelab/$SVC:$TAG fhirworx/$SVC:$TAG;
         done
-      - sed -i "s/^COMMIT_SHA=.*/COMMIT_SHA=$TAG/" .env 2>/dev/null || echo "COMMIT_SHA=$TAG" >> .env
+      # Two-phase provision + restart + health check
-      - docker compose up -d --remove-orphans
+      - uv sync --no-dev
+      - uv run python -m api.auth deploy ${CI_COMMIT_SHA}
     depends_on:
       - publish-package
       - scan-notebooks
@@ -275,32 +287,6 @@ steps:
       - scan-api
       - build-push-mc
 
-  # ── Provision: derive credentials and rotate backends ────────
-  # Runs after deploy but also after builds — decoupled so that
-  # scan/upload failures don't prevent credential rotation.
-  - name: provision
-    image: ghcr.io/astral-sh/uv:python3.13-bookworm-slim
-    failure: ignore
-    volumes:
-      - /run/user/1000/docker.sock:/var/run/docker.sock
-      - /home/kert/stack:/home/kert/stack
-    environment:
-      UV_PYTHON_PREFERENCE: only-system
-      UV_LINK_MODE: copy
-      UV_PROJECT_ENVIRONMENT: .venv
-      ROOT_KEY:
-        from_secret: root_key
-    commands:
-      - cd /home/kert/stack
-      - uv sync --no-dev
-      - uv run python -m api.auth provision ${CI_COMMIT_SHA}
-    depends_on:
-      - build-push-notebooks
-      - build-push-zotero
-      - build-push-docs
-      - build-push-api
-      - build-push-mc
-
   # ── Failure reporter ─────────────────────────────────────────
   - name: report-failure
     image: ghcr.io/astral-sh/uv:python3.13-bookworm-slim

src/api/auth/__main__.py

@@ -3,6 +3,8 @@
 Usage:
     uv run python -m api.auth bootstrap <commit_sha>
     uv run python -m api.auth provision <commit_sha>
+    uv run python -m api.auth deploy <commit_sha>
+    uv run python -m api.auth verify <commit_sha>
     uv run python -m api.auth derive <commit_sha> [--redact]
 """
 
@@ -18,7 +20,8 @@ def main() -> int:
     logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
 
     args = sys.argv[1:]
-    if len(args) < 2 or args[0] not in ("bootstrap", "provision", "derive"):
+    commands = ("bootstrap", "provision", "derive", "deploy", "verify")
+    if len(args) < 2 or args[0] not in commands:
         print(__doc__.strip(), file=sys.stderr)
         return 1
 
@@ -64,6 +67,30 @@ def main() -> int:
             return 1 if not result.postgres and not result.gitea else 0
         return 0
 
+    if command == "deploy":
+        from api.auth.deploy import deploy as do_deploy
+
+        result = do_deploy(root_key, commit_sha, env_path)
+        if result.errors:
+            for backend, err in result.errors:
+                print(f"  FAILED: {backend} — {err}", file=sys.stderr)
+            return 1 if not result.env_written else 0
+        print("Deploy + provision complete")
+        return 0
+
+    if command == "verify":
+        from api.auth.deploy import verify_all
+        from api.auth.provision import derive_all
+
+        values = derive_all(root_key, commit_sha)
+        errors = verify_all(values)
+        if errors:
+            for err in errors:
+                print(f"  FAIL: {err}", file=sys.stderr)
+            return 1
+        print("All credential checks passed")
+        return 0
+
     if command == "bootstrap":
         from api.auth.provision import bootstrap
 

src/api/auth/deploy.py

@@ -0,0 +1,236 @@
+"""Two-phase deploy: rotate backends FIRST, then restart services.
+
+Called from the Woodpecker deploy step to ensure credential consistency:
+1. Derive new credentials from ROOT_KEY + commit SHA
+2. Rotate PostgreSQL passwords (ALTER ROLE)
+3. Rotate Gitea admin password + create fresh token
+4. Write .env with all new credentials
+5. Restart services (docker compose up -d)
+6. Sync Woodpecker secrets
+7. Verify all backends are healthy
+8. Rollback on failure
+
+Usage:
+    uv run python -m api.auth deploy <commit_sha>
+"""
+
+from __future__ import annotations
+
+import logging
+import shutil
+import subprocess
+import time
+from pathlib import Path
+
+import httpx
+
+from api.auth.provision import (
+    ProvisionResult,
+    _retry,
+    derive_all,
+    provision_gitea,
+    provision_postgres,
+    provision_woodpecker,
+    write_env,
+)
+
+log = logging.getLogger(__name__)
+
+
+def verify_postgres(
+    values: dict[str, str], *, container: str = "postgres"
+) -> list[str]:
+    """Verify PostgreSQL connections for all managed roles."""
+    from api.auth.manifest import POSTGRES_DATABASES, POSTGRES_ROLES
+
+    errors = []
+    for role, env_var in POSTGRES_ROLES.items():
+        pw = values[env_var]
+        db = POSTGRES_DATABASES[role]
+        result = subprocess.run(
+            [
+                "docker",
+                "exec",
+                "-e",
+                f"PGPASSWORD={pw}",
+                container,
+                "psql",
+                "-U",
+                role,
+                "-d",
+                db,
+                "-c",
+                "SELECT 1",
+            ],
+            capture_output=True,
+            text=True,
+        )
+        if result.returncode != 0:
+            errors.append(f"postgres/{role}: {result.stderr.strip()}")
+    return errors
+
+
+def verify_gitea(
+    values: dict[str, str], *, base_url: str = "http://gitea:3000/api/v1"
+) -> list[str]:
+    """Verify Gitea API is reachable with the current token."""
+    errors = []
+    token = values.get("GITEA_TOKEN", "")
+    if not token:
+        errors.append("gitea: no GITEA_TOKEN")
+        return errors
+
+    try:
+        resp = httpx.get(
+            f"{base_url}/user",
+            headers={"Authorization": f"token {token}"},
+            timeout=10,
+        )
+        if resp.status_code != 200:
+            errors.append(f"gitea: HTTP {resp.status_code}")
+    except Exception as e:
+        errors.append(f"gitea: {e}")
+    return errors
+
+
+def verify_rustfs(*, endpoint: str = "http://rustfs:9000") -> list[str]:
+    """Verify RustFS is healthy."""
+    errors = []
+    try:
+        resp = httpx.get(f"{endpoint}/minio/health/live", timeout=10)
+        if resp.status_code != 200:
+            errors.append(f"rustfs: HTTP {resp.status_code}")
+    except Exception as e:
+        errors.append(f"rustfs: {e}")
+    return errors
+
+
+def verify_all(values: dict[str, str]) -> list[str]:
+    """Run all health checks. Returns list of errors (empty = healthy)."""
+    errors = []
+    errors.extend(verify_postgres(values))
+    errors.extend(verify_gitea(values))
+    errors.extend(verify_rustfs())
+    return errors
+
+
+def deploy(
+    root_key: bytes,
+    commit_sha: str,
+    env_path: Path,
+    *,
+    compose_dir: Path | None = None,
+) -> ProvisionResult:
+    """Two-phase deploy: rotate backends, then restart services.
+
+    Phase 1: Rotate backend credentials (ALTER ROLE, Gitea API)
+    Phase 2: Write .env + docker compose up -d
+    Phase 3: Sync Woodpecker secrets + health check
+    """
+    result = ProvisionResult()
+    cwd = compose_dir or env_path.parent
+
+    # ── Derive ────────────────────────────────────────────────
+    values = derive_all(root_key, commit_sha)
+    log.info("Derived %d credentials for commit %s", len(values), commit_sha[:8])
+
+    # ── Phase 1: Rotate backends FIRST ────────────────────────
+    # PostgreSQL
+    try:
+        _retry(lambda: provision_postgres(values), "postgres")
+        result.postgres = True
+        log.info("Phase 1: PostgreSQL passwords rotated")
+    except Exception as e:
+        result.errors.append(("postgres", e))
+        log.error("Phase 1: PostgreSQL rotation failed: %s", e)
+
+    # Gitea
+    try:
+        token = _retry(lambda: provision_gitea(values), "gitea")
+        values["GITEA_TOKEN"] = token
+        result.gitea = True
+        log.info("Phase 1: Gitea token rotated")
+    except Exception as e:
+        result.errors.append(("gitea", e))
+        log.error("Phase 1: Gitea rotation failed: %s", e)
+
+    # ── Phase 2: Write .env + restart services ────────────────
+    # Back up current .env
+    env_bak = env_path.with_suffix(".env.bak")
+    if env_path.exists():
+        shutil.copy2(env_path, env_bak)
+
+    write_env(values, env_path)
+    result.env_written = True
+    log.info("Phase 2: Wrote .env")
+
+    # Set COMMIT_SHA for compose image tags
+    _set_env_var(env_path, "COMMIT_SHA", f"sha-{commit_sha[:8]}")
+
+    # Restart services
+    try:
+        subprocess.run(
+            ["docker", "compose", "up", "-d", "--remove-orphans"],
+            cwd=str(cwd),
+            check=True,
+            capture_output=True,
+            text=True,
+        )
+        log.info("Phase 2: Services restarted")
+    except subprocess.CalledProcessError as e:
+        result.errors.append(("compose-up", e))
+        log.error("Phase 2: compose up failed: %s", e.stderr[:500])
+
+    # ── Phase 3: Sync secrets + health check ──────────────────
+    # Wait for services to be ready
+    time.sleep(10)
+
+    # Woodpecker secrets
+    try:
+        _retry(lambda: provision_woodpecker(values), "woodpecker")
+        result.woodpecker = True
+        log.info("Phase 3: Woodpecker secrets synced")
+    except Exception as e:
+        result.errors.append(("woodpecker", e))
+        log.error("Phase 3: Woodpecker sync failed: %s", e)
+
+    # Health check
+    health_errors = verify_all(values)
+    if health_errors:
+        for err in health_errors:
+            log.error("Health check FAIL: %s", err)
+            result.errors.append(("health", Exception(err)))
+
+        # Rollback
+        if env_bak.exists():
+            log.warning("Rolling back to %s", env_bak)
+            shutil.copy2(env_bak, env_path)
+            try:
+                subprocess.run(
+                    ["docker", "compose", "up", "-d", "--remove-orphans"],
+                    cwd=str(cwd),
+                    check=True,
+                    capture_output=True,
+                    text=True,
+                )
+                log.info("Rollback: services restarted with old credentials")
+            except subprocess.CalledProcessError:
+                log.error("Rollback: compose up failed!")
+    else:
+        log.info("Phase 3: All health checks passed")
+
+    return result
+
+
+def _set_env_var(env_path: Path, key: str, value: str) -> None:
+    """Set a single env var in the .env file."""
+    lines = env_path.read_text().splitlines()
+    found = False
+    for i, line in enumerate(lines):
+        if line.startswith(f"{key}="):
+            lines[i] = f"{key}={value}"
+            found = True
+            break
+    if not found:
+        lines.append(f"{key}={value}")
+    env_path.write_text("\n".join(lines) + "\n")

tests/api/test_deploy.py

@@ -0,0 +1,170 @@
+"""Tests for api.auth.deploy — two-phase credential rotation."""
+
+from __future__ import annotations
+
+import subprocess
+from pathlib import Path
+from unittest.mock import MagicMock, patch
+
+from api.auth.deploy import (
+    _set_env_var,
+    deploy,
+    verify_all,
+    verify_gitea,
+    verify_postgres,
+    verify_rustfs,
+)
+from api.auth.provision import derive_all
+
+ROOT = bytes.fromhex("deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef")
+COMMIT = "abc1234"
+
+
+class TestVerifyPostgres:
+    def test_returns_empty_on_success(self):
+        values = derive_all(ROOT, COMMIT)
+        with patch("api.auth.deploy.subprocess.run") as mock:
+            mock.return_value = subprocess.CompletedProcess([], 0)
+            errors = verify_postgres(values, container="pg")
+        assert errors == []
+
+    def test_returns_errors_on_failure(self):
+        values = derive_all(ROOT, COMMIT)
+        with patch("api.auth.deploy.subprocess.run") as mock:
+            mock.return_value = subprocess.CompletedProcess([], 1, stderr="auth failed")
+            errors = verify_postgres(values, container="pg")
+        assert len(errors) > 0
+        assert "auth failed" in errors[0]
+
+
+class TestVerifyGitea:
+    def test_returns_empty_on_success(self):
+        values = {"GITEA_TOKEN": "tok"}
+        mock_resp = MagicMock(status_code=200)
+        with patch("api.auth.deploy.httpx.get", return_value=mock_resp):
+            errors = verify_gitea(values)
+        assert errors == []
+
+    def test_returns_error_on_401(self):
+        values = {"GITEA_TOKEN": "bad"}
+        mock_resp = MagicMock(status_code=401)
+        with patch("api.auth.deploy.httpx.get", return_value=mock_resp):
+            errors = verify_gitea(values)
+        assert len(errors) == 1
+        assert "401" in errors[0]
+
+    def test_returns_error_on_no_token(self):
+        errors = verify_gitea({})
+        assert len(errors) == 1
+        assert "no GITEA_TOKEN" in errors[0]
+
+
+class TestVerifyRustfs:
+    def test_returns_empty_on_success(self):
+        mock_resp = MagicMock(status_code=200)
+        with patch("api.auth.deploy.httpx.get", return_value=mock_resp):
+            errors = verify_rustfs()
+        assert errors == []
+
+    def test_returns_error_on_failure(self):
+        with patch("api.auth.deploy.httpx.get", side_effect=Exception("down")):
+            errors = verify_rustfs()
+        assert len(errors) == 1
+
+
+class TestVerifyAll:
+    def test_aggregates_errors(self):
+        values = derive_all(ROOT, COMMIT)
+        values["GITEA_TOKEN"] = "tok"
+        with (
+            patch("api.auth.deploy.verify_postgres", return_value=["pg: fail"]),
+            patch("api.auth.deploy.verify_gitea", return_value=[]),
+            patch("api.auth.deploy.verify_rustfs", return_value=["rustfs: fail"]),
+        ):
+            errors = verify_all(values)
+        assert len(errors) == 2
+
+
+class TestSetEnvVar:
+    def test_updates_existing(self, tmp_path: Path):
+        p = tmp_path / ".env"
+        p.write_text("A=1\nB=2\n")
+        _set_env_var(p, "B", "99")
+        assert "B=99" in p.read_text()
+        assert "B=2" not in p.read_text()
+
+    def test_appends_new(self, tmp_path: Path):
+        p = tmp_path / ".env"
+        p.write_text("A=1\n")
+        _set_env_var(p, "B", "2")
+        content = p.read_text()
+        assert "A=1" in content
+        assert "B=2" in content
+
+
+class TestDeploy:
+    def test_two_phase_rotation(self, tmp_path: Path):
+        env = tmp_path / ".env"
+        env.write_text("OLD_KEY=keep\n")
+
+        with (
+            patch("api.auth.deploy.provision_postgres"),
+            patch("api.auth.deploy.provision_gitea", return_value="new-tok"),
+            patch("api.auth.deploy.provision_woodpecker"),
+            patch("api.auth.deploy.subprocess.run"),
+            patch("api.auth.deploy.verify_all", return_value=[]),
+            patch("api.auth.deploy.time.sleep"),
+        ):
+            result = deploy(ROOT, COMMIT, env, compose_dir=tmp_path)
+
+        assert result.ok
+        assert result.postgres
+        assert result.gitea
+        assert result.woodpecker
+        assert result.env_written
+        content = env.read_text()
+        assert "GITEA_TOKEN=new-tok" in content
+        assert "OLD_KEY=keep" in content
+
+    def test_rollback_on_health_failure(self, tmp_path: Path):
+        env = tmp_path / ".env"
+        env.write_text("SAFE=original\n")
+
+        with (
+            patch("api.auth.deploy.provision_postgres"),
+            patch("api.auth.deploy.provision_gitea", return_value="tok"),
+            patch("api.auth.deploy.provision_woodpecker"),
+            patch("api.auth.deploy.subprocess.run"),
+            patch(
+                "api.auth.deploy.verify_all",
+                return_value=["pg: auth failed"],
+            ),
+            patch("api.auth.deploy.time.sleep"),
+        ):
+            result = deploy(ROOT, COMMIT, env, compose_dir=tmp_path)
+
+        assert not result.ok
+        # .env should be rolled back to original
+        content = env.read_text()
+        assert "SAFE=original" in content
+
+    def test_partial_backend_failure_still_deploys(self, tmp_path: Path):
+        env = tmp_path / ".env"
+        env.write_text("")
+
+        with (
+            patch(
+                "api.auth.deploy.provision_postgres",
+                side_effect=Exception("pg down"),
+            ),
+            patch("api.auth.deploy.provision_gitea", return_value="tok"),
+            patch("api.auth.deploy.provision_woodpecker"),
+            patch("api.auth.deploy.subprocess.run"),
+            patch("api.auth.deploy.verify_all", return_value=[]),
+            patch("api.auth.deploy.time.sleep"),
+        ):
+            result = deploy(ROOT, COMMIT, env, compose_dir=tmp_path)
+
+        assert not result.postgres
+        assert result.gitea
+        assert result.env_written