[P16] RCA: Provision never ran — credential rotation broken from inception #118

New Issue

kert · 2026-03-23T13:22:07Z

kert commented

2026-03-23 13:22:07 +00:00

Root Cause Analysis

Summary

Provision has NEVER successfully executed in any CI pipeline. It was added in pipeline #97 and has been skipped in every pipeline since (#97–#123+). When we finally made it run, it caused a stack-wide outage because .env passwords didn't match PostgreSQL.

5 Whys

Why did the stack go down? Gitea couldn't connect to PostgreSQL — password mismatch.
Why password mismatch? .env has HKDF-derived passwords, but PostgreSQL roles still had original bootstrap passwords.
Why weren't PostgreSQL passwords synced? provision_postgres() never ran.
Why did provision never run? It depended on deploy, which depended on buildx, which was broken for 25+ pipelines.
Why did nobody notice? report-failure was also skipped. No health checks.

Compounding Factors

deploy does git reset --hard → overwrites .env if provision wrote it first
deploy does docker compose up -d → restarts services with .env passwords that don't match backends
No credential consistency check between .env and actual backend state
RustFS IAM corrupted by concurrent restarts

Structural Fix: Two-Phase Rotation

1. derive_all() → ALTER ROLE passwords in PostgreSQL FIRST
2. provision_gitea() → rotate token via API
3. write .env with new passwords
4. docker compose up -d (services read matching passwords)
5. provision_woodpecker() → sync secrets
6. health check all backends
7. rollback .env.bak if health checks fail

Key insight: change the backend FIRST, then update .env, then restart services. Never restart services with passwords that haven't been applied to the backend yet.

## Root Cause Analysis ### Summary Provision has **NEVER** successfully executed in any CI pipeline. It was added in pipeline #97 and has been `skipped` in every pipeline since (#97–#123+). When we finally made it run, it caused a stack-wide outage because `.env` passwords didn't match PostgreSQL. ### 5 Whys 1. **Why did the stack go down?** Gitea couldn't connect to PostgreSQL — password mismatch. 2. **Why password mismatch?** `.env` has HKDF-derived passwords, but PostgreSQL roles still had original bootstrap passwords. 3. **Why weren't PostgreSQL passwords synced?** `provision_postgres()` never ran. 4. **Why did provision never run?** It depended on `deploy`, which depended on buildx, which was broken for 25+ pipelines. 5. **Why did nobody notice?** `report-failure` was also skipped. No health checks. ### Compounding Factors - `deploy` does `git reset --hard` → overwrites `.env` if provision wrote it first - `deploy` does `docker compose up -d` → restarts services with `.env` passwords that don't match backends - No credential consistency check between `.env` and actual backend state - RustFS IAM corrupted by concurrent restarts ### Structural Fix: Two-Phase Rotation ``` 1. derive_all() → ALTER ROLE passwords in PostgreSQL FIRST 2. provision_gitea() → rotate token via API 3. write .env with new passwords 4. docker compose up -d (services read matching passwords) 5. provision_woodpecker() → sync secrets 6. health check all backends 7. rollback .env.bak if health checks fail ``` Key insight: **change the backend FIRST, then update .env, then restart services.** Never restart services with passwords that haven't been applied to the backend yet.

kert added this to the P16: Provision Hardening — resilient credential rotation milestone 2026-03-23 13:22:07 +00:00

kert added the ci label 2026-03-23 13:22:07 +00:00

kert referenced this issue from a commit

2026-03-23 17:17:57 +00:00

disable auto-deploy: manual only until rootless Docker issues resolved

kert closed this issue

2026-03-23 17:56:50 +00:00

Sign in to join this conversation.