ci: test job flakes intermittently (transient failures, HEAD is green) #515
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: homelab/stack#515
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The
ci.ymltestjob (pytest -x --cov=src --cov-fail-under=99 -q -n auto) fails intermittently — the same suite passes on superset commits.Observed (run / commit / test-job result):
3c835e9fail; #558cc6fecapass; #561c7ea28dpass3760541fail; #567b21b1aefailc1c51d6pass; #57190651ffpass ← current HEAD greenc1c51d6/90651ffcontain a superset of3760541/b21b1aechanges, so those failures were not code regressions. Other unrelated commits (3edde5a,a94d3d4) show the same flakiness.Impact: false-positive CI-failure issues (auto-filer noise); erodes trust in the gate.
Likely causes to investigate: pytest-xdist worker crash/OOM/timeout under
-n autoon the runner; a nondeterministic test; coverage-combine races (parallel=true).Options: capture per-worker logs on failure; add a single automatic rerun for the test job; pin worker count; quarantine any test found nondeterministic.
Fixed in
9b58e2dfb2. The ci test job now preinstalls the duckdb sqlite extension before pytest, so the per-workerINSTALL sqlitecalls in tests/zot/test_duck.py become no-ops instead of racing the extension-file rename in ~/.duckdb ('Could not remove file ... sqlite_scanner'). The flake hit runs 584, 591, and 600; first post-fix test run is green.