[P16] Add retry logic and graceful degradation to provision #111

New Issue

kert · 2026-03-22T17:12:53Z

kert commented

2026-03-22 17:12:53 +00:00

Problem

provision() calls provision_postgres() then provision_gitea() with no error handling. If postgres fails, gitea is never attempted. If gitea fails, .env is never written. A single transient network error kills the entire rotation.

Fix

Wrap each backend provisioner in try/except
Log errors but continue to next backend
Still write .env with whatever succeeded (partial rotation is better than none)
Add retry with backoff for transient errors (connection refused, timeout)
Return a summary of what succeeded/failed
Exit non-zero only if ALL backends fail

def provision(root_key, commit_sha, env_path):
    values = derive_all(root_key, commit_sha)
    errors = []
    try:
        provision_postgres(values)
    except Exception as e:
        errors.append(('postgres', e))
    try:
        token = provision_gitea(values)
        values['GITEA_TOKEN'] = token
    except Exception as e:
        errors.append(('gitea', e))
    write_env(values, env_path)  # always write
    return values, errors

Acceptance criteria

Postgres failure doesn't prevent Gitea rotation
Gitea failure doesn't prevent .env write
Transient errors retried 3x with backoff
Exit code 0 if any backend succeeds, 1 if all fail
Provision step logs clear summary of results

## Problem `provision()` calls `provision_postgres()` then `provision_gitea()` with no error handling. If postgres fails, gitea is never attempted. If gitea fails, `.env` is never written. A single transient network error kills the entire rotation. ## Fix 1. Wrap each backend provisioner in try/except 2. Log errors but continue to next backend 3. Still write `.env` with whatever succeeded (partial rotation is better than none) 4. Add retry with backoff for transient errors (connection refused, timeout) 5. Return a summary of what succeeded/failed 6. Exit non-zero only if ALL backends fail ```python def provision(root_key, commit_sha, env_path): values = derive_all(root_key, commit_sha) errors = [] try: provision_postgres(values) except Exception as e: errors.append(('postgres', e)) try: token = provision_gitea(values) values['GITEA_TOKEN'] = token except Exception as e: errors.append(('gitea', e)) write_env(values, env_path) # always write return values, errors ``` ## Acceptance criteria - [ ] Postgres failure doesn't prevent Gitea rotation - [ ] Gitea failure doesn't prevent .env write - [ ] Transient errors retried 3x with backoff - [ ] Exit code 0 if any backend succeeds, 1 if all fail - [ ] Provision step logs clear summary of results

kert added this to the P16: Provision Hardening — resilient credential rotation milestone 2026-03-22 17:12:53 +00:00

kert added the ci label 2026-03-22 17:12:53 +00:00

kert referenced this issue from a commit

2026-03-22 17:32:24 +00:00

harden provision: API-only Gitea, safe SQL, retry, Woodpecker sync

kert closed this issue

2026-03-22 17:32:24 +00:00

Sign in to join this conversation.