perf(ci): add -n auto to CI pytest + fix coverage combining (#388)
Some checks failed
Deploy / docs (push) Has been skipped
CI / lint (push) Successful in 34s
Deploy / notebooks (push) Has been skipped
Deploy / zotero (push) Has been skipped
Deploy / api (push) Successful in 48s
Deploy / mc (push) Has been skipped
Infra CI / notebooks (push) Failing after 20s
Infra CI / zotero (push) Successful in 12s
Infra CI / docs (push) Successful in 12s
Infra CI / api (push) Successful in 18s
Infra CI / mc (push) Successful in 12s
Package Supply Chain / pkg-supply-chain (push) Failing after 49s
Deploy / report (push) Successful in 18s
CI / test (push) Failing after 11m9s
Some checks failed
Deploy / docs (push) Has been skipped
CI / lint (push) Successful in 34s
Deploy / notebooks (push) Has been skipped
Deploy / zotero (push) Has been skipped
Deploy / api (push) Successful in 48s
Deploy / mc (push) Has been skipped
Infra CI / notebooks (push) Failing after 20s
Infra CI / zotero (push) Successful in 12s
Infra CI / docs (push) Successful in 12s
Infra CI / api (push) Successful in 18s
Infra CI / mc (push) Successful in 12s
Package Supply Chain / pkg-supply-chain (push) Failing after 49s
Deploy / report (push) Successful in 18s
CI / test (push) Failing after 11m9s
The earlier xdist attempt was reverted in847713dbecause pytest-cov combined results unreliably under -n auto. Root cause: missing [tool.coverage.run] parallel=true in pyproject.toml. Without it, each worker overwrites .coverage atomically and the report drops data from all but one worker. Fix: - pyproject.toml: add [tool.coverage.run] with parallel=true, source=["src"], concurrency=["multiprocessing","thread"]. Verified locally: 56-test xdist run produces a clean combined coverage report at 99% (vs flaky/missing data before). - dev/scripts/backends/gitea.py + github.py: add -n auto to the emitted pytest command. Also persist the deploy.yml report-step failure gate (was applied to the file in5f826f6but the emitter itself didn't have it, so regen would strip it). - .gitea/workflows/ci.yml: regenerated to pick up -n auto. The 32-core runner should now run CI pytest in roughly 1/Nth the prior time (where N is bounded by serialization on slow tests). Phase 1 of #388 (pre-commit hook xdist) shipped in49f004a. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -47,7 +47,11 @@ jobs:
|
||||
run: uv sync --dev
|
||||
|
||||
- name: Pytest
|
||||
run: uv run pytest tests/ -x --cov=src --cov-report=term-missing --cov-fail-under=99 -q
|
||||
# -n auto parallelizes across runner cores. Coverage combining
|
||||
# is configured via [tool.coverage.run] parallel=true in
|
||||
# pyproject.toml — without that the per-worker .coverage.* files
|
||||
# don't merge reliably (see #388 for the prior revert).
|
||||
run: uv run pytest tests/ -x --cov=src --cov-report=term-missing --cov-fail-under=99 -q -n auto
|
||||
|
||||
- name: File failure issue
|
||||
if: failure()
|
||||
|
||||
@@ -178,7 +178,11 @@ jobs:
|
||||
run: uv sync --dev
|
||||
|
||||
- name: Pytest
|
||||
run: uv run pytest tests/ -x --cov=src --cov-report=term-missing --cov-fail-under={coverage_threshold} -q
|
||||
# -n auto parallelizes across runner cores. Coverage combining
|
||||
# is configured via [tool.coverage.run] parallel=true in
|
||||
# pyproject.toml — without that the per-worker .coverage.* files
|
||||
# don't merge reliably (see #388 for the prior revert).
|
||||
run: uv run pytest tests/ -x --cov=src --cov-report=term-missing --cov-fail-under={coverage_threshold} -q -n auto
|
||||
|
||||
{_failure_step("CI", "test")}
|
||||
"""
|
||||
@@ -266,6 +270,10 @@ jobs:
|
||||
{_setup_uv_step(uv_version)}
|
||||
|
||||
- name: Report
|
||||
# Only file an issue if at least one upstream build job failed.
|
||||
# Without this gate api.diag.ci runs unconditionally and files
|
||||
# a fake "failed" issue on every successful Deploy run.
|
||||
if: contains(needs.*.result, 'failure')
|
||||
env:
|
||||
GITEA_TOKEN: ${{{{ secrets.DEPLOY_TOKEN }}}}
|
||||
run: |
|
||||
|
||||
@@ -159,7 +159,9 @@ jobs:
|
||||
run: uv run ruff format --check src/ tests/
|
||||
|
||||
- name: Pytest
|
||||
run: uv run pytest tests/ --cov=src --cov-report=term-missing --cov-fail-under={coverage_threshold} -q
|
||||
# -n auto parallelizes across runner cores; pytest-cov combines
|
||||
# results via [tool.coverage.run] parallel=true in pyproject.toml.
|
||||
run: uv run pytest tests/ --cov=src --cov-report=term-missing --cov-fail-under={coverage_threshold} -q -n auto
|
||||
|
||||
- name: Validate generated config
|
||||
run: uv run python dev/scripts/gen_config.py --check
|
||||
|
||||
@@ -187,6 +187,14 @@ markers = [
|
||||
"stub: marks tests that report stub vs implemented status (deselect with '-m \"not stub\"')",
|
||||
]
|
||||
|
||||
[tool.coverage.run]
|
||||
# Required for reliable pytest-cov + pytest-xdist combining: each worker
|
||||
# writes .coverage.<host>.<pid>, then pytest-cov auto-combines at exit.
|
||||
# Without this, parallel test runs randomly drop coverage data.
|
||||
parallel = true
|
||||
source = ["src"]
|
||||
concurrency = ["multiprocessing", "thread"]
|
||||
|
||||
[tool.uv.build-backend]
|
||||
module-name = ["aco", "api", "bcda", "bib", "bls", "ccw", "cli", "cms", "conf", "mail", "opps", "perf", "pfs", "prisma", "rec", "rex", "sem"]
|
||||
namespace = true
|
||||
|
||||
Reference in New Issue
Block a user