perf: pytest suite takes 20+ min — optimize to <5 min #388
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: homelab/stack#388
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem
The full pytest suite (13K+ tests) takes 20-25 min single-process. This blocks CI on every push and makes the pre-commit hook painful.
Root Causes
1. Zotero DB creation (~10s each, called ~200 times)
zot.schema.create_db()builds the full Zotero schema from scratch for every test that needs a DB. With ~200 Zotero-dependent tests, that's ~33 min of just DB creation.Fix: Session-scoped
zotero_dbfixture (already added in conftest.py) — creates once, copies per test viashutil.copy2(~0.01s vs ~10s). Migrate all tests to usezotero_dbfixture instead of callingcreate_db()directly.2. No test parallelism in CI
The runner has 32 cores but pytest runs single-process.
pytest-xdist(-n auto) parallelizes but breaks--covcoverage combining.Fix: Use
pytest-cov's--cov-combinewith xdist, or split into fast parallel test run + separate single-process coverage check:3.
pytest-testmonnot usedpytest-testmontracks which source lines each test touches and only re-runs affected tests on change. With typical commits touching 2-3 modules, this could cut from 13K tests to ~500.Fix:
pip install pytest-testmon+--testmonflag. Use in pre-commit hook (already has smart selection insem/hooks.py). CI keeps full suite for coverage gate.4. Redundant test execution
test_sync_full.py+test_sync_deeper.py+test_sync_exercise.pyall testingbib.sync)assert callable(func)) add collection overhead without coverage valueFix: Audit and consolidate test files. Remove pure-import tests where deeper tests exist.
5. Slow fixtures
prisma.project.init()tests create full Zotero DBs + anchor items (~15s each)bib.store.Store()tests create SQLite DBs from scratchFix: Session-scoped fixtures for expensive setup. Shared mock factories.
6. Coverage overhead
--covadds ~30% overhead to every test. The coverage report generation (--cov-report=term-missing) adds another ~10s for 48K lines.Fix: Only run coverage in CI, not in pre-commit. Use
--cov-report=(empty) to skip report in CI, check threshold only.Implementation Plan
Phase 1 (quick wins, ~2 min saved):
create_db()calls tozotero_dbfixture--cov-report=to CI (skip report, just check threshold)Phase 2 (parallelism, ~10 min saved):
pytest-xdistwithpytest-covcombiningPhase 3 (smart selection, pre-commit only):
pytest-testmonintosem/hooks.py.testmondatabetween runsPhase 4 (fixture optimization):
Target
Related: the
sem.hooksmodule already implements smart test selection based on staged files → module mapping. Phase 3 (pytest-testmon) builds on this —sem/hooks.py:compute_test_targets()already knows which test dirs to run for each changed module. Testmon adds line-level precision on top of that.Phase 1 partial — pre-commit hook xdist landed
Commit (pending):
sem.hooksnow passes-n autoto its targeted pytest invocation. The hook already uses--no-covso xdist's coverage-combining incompatibility doesn't apply here.Measured speedup on cli/zot/rex subset (610 tests):
-n autoon 32-core runner: 3:03 (183s)For the full-suite case (e.g. when
pyproject.tomlis staged → 13K+ tests), the speedup should be similar or better — more tests = more parallelism. That puts the worst-case hook from ~47min → ~15min. Not yet at the <5min target but a meaningful improvement.Also migrated 2 of ~10
create_db()callsites to the existing session-scopedzotero_dbfixture (test_duck.py, test_extract.py). The remaining migrations (prisma/, test_final_coverage_gaps.py) are diminishing returns now that xdist parallelizes the create_db calls across workers — deferred unless we want to chase the <5min target harder.Still open:
-n autoto CI workflows (needs the cov-combine workaround the issue mentions)test_sync_*.pyfilesWill chase those when they're the binding constraint.
Phase 2 — CI xdist + coverage combining shipped (
16d657a)The earlier xdist attempt was reverted in
847713dbecause pytest-cov combined results unreliably under-n auto. Root cause: missing[tool.coverage.run] parallel=truein pyproject.toml. Without it, each worker overwrites.coverageatomically and the report drops data from all but one worker.Fix:
pyproject.toml: added[tool.coverage.run]withparallel=true,source=['src'],concurrency=['multiprocessing','thread'].dev/scripts/backends/{gitea,github}.py: emit-n autoon the CI pytest line. Also persisted theif: contains(needs.*.result, 'failure')gate on the deploy report step (was applied to the file in5f826f6but not the emitter, so regen would strip it)..gitea/workflows/ci.yml: regenerated.End-to-end speedup measurement (today's pre-commit hook on the full 13,799-test suite, since the hook now uses xdist too):
-n auto: 9:56 (this commit's hook)Better than the 3.1× we measured earlier on a subset — more tests = more parallelism opportunities.
Done for #388 in this session:
49f004a)16d657a)create_db()callsites to session-scopedzotero_dbfixture (49f004a)Still open (lower priority, defer until binding):
pytest-testmonfor incremental selection on pre-commit (could cut typical hook to ~30s)test_sync_*.pyfilescreate_db()callsites in prisma/ testsClosing the issue for now — the binding constraint (full-suite hook taking 47 min) is solved.