37a970ba02
Two minors from the final fix wave:
- dev/scripts/backends/gitea.py::_gen_llm_golden: the docker cp of the
report back to the runner and the cat of it now both end in
`|| true` — under `set -euo pipefail`, a golden run that crashed
before ever writing /tmp/golden/report.json used to abort the step
right there, before it could reach the explicit
`exit "${golden_rc:-0}"`. Also switched the runner invocation to
`uv run --no-sync --project /app`, matching the pattern the other
generators already use for a container that shouldn't try to
resolve/sync dependencies at request time. Regenerated
.gitea/workflows/llm-golden.yml; gen_config.py --check passes.
- pyproject.toml: dev/scripts/llm_golden.py (`import yaml`) runs
inside the llm container in that same workflow, but pyyaml was only
ever a dev-group dependency — added to the `llm` extra so the
container actually has it.
63 lines
2.3 KiB
YAML
63 lines
2.3 KiB
YAML
# DO NOT EDIT — generated by gen_config.py from stack.toml
|
|
# Re-generate: uv run python dev/scripts/gen_config.py
|
|
|
|
name: LLM Golden
|
|
|
|
on:
|
|
workflow_dispatch:
|
|
schedule:
|
|
- cron: "40 3 * * *"
|
|
|
|
jobs:
|
|
llm-golden:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: Checkout
|
|
uses: https://github.com/actions/checkout@v4
|
|
|
|
- name: Set up uv
|
|
run: curl -LsSf https://astral.sh/uv/install.sh | sh
|
|
env:
|
|
UV_INSTALL_DIR: /usr/local/bin
|
|
|
|
- name: Run golden longitudinal evaluation against the live chat
|
|
# golden_rc is captured (not left to `set -e`) so a failing golden
|
|
# run still docker-cps the report back and cats it — llm_golden.py
|
|
# itself exits 0 once --file-issues has filed the regressions
|
|
# (nb_issue_filer is the failure channel here), but this still
|
|
# guards the case where it exits nonzero before reaching that
|
|
# point (e.g. a crash while loading the golden set).
|
|
env:
|
|
GITEA_TOKEN: ${{ secrets.DEPLOY_TOKEN }}
|
|
run: |
|
|
set -euo pipefail
|
|
docker exec llm mkdir -p /tmp/golden
|
|
docker cp dev/scripts/llm_golden.py llm:/tmp/golden/
|
|
docker cp dev/scripts/nb_issue_filer.py llm:/tmp/golden/
|
|
docker cp tests/llm/golden_lineage.yaml llm:/tmp/golden/
|
|
docker exec \
|
|
-e GITEA_TOKEN \
|
|
-e GITEA_API_BASE=http://git:3000/api/v1 \
|
|
-e NB_ISSUE_LABEL=llm \
|
|
llm \
|
|
uv run --no-sync --project /app python /tmp/golden/llm_golden.py run \
|
|
--url http://localhost:8000 \
|
|
--set /tmp/golden/golden_lineage.yaml \
|
|
--report /tmp/golden/report.json \
|
|
--file-issues --source nightly-llm-golden || golden_rc=$?
|
|
docker cp llm:/tmp/golden/report.json llm-golden-report.json || true
|
|
cat llm-golden-report.json || true
|
|
exit "${golden_rc:-0}"
|
|
|
|
- name: File failure issue
|
|
if: failure()
|
|
env:
|
|
GITEA_TOKEN: ${{ secrets.DEPLOY_TOKEN }}
|
|
run: |
|
|
uv sync --no-dev --quiet 2>/dev/null || true
|
|
uv run python -m api.diag.ci \
|
|
--workflow "LLM Golden" --job "llm-golden" \
|
|
--run "${{ github.run_number }}" \
|
|
--sha "${{ github.sha }}" \
|
|
--ref "${{ github.ref }}" || true
|