bc3bab8a42
All checks were successful
CI / lint (push) Successful in 47s
CI / notebooks-smoke (push) Successful in 1m33s
Deploy / notebooks (push) Has been skipped
Deploy / zotero (push) Has been skipped
Deploy / docs (push) Has been skipped
Deploy / api (push) Has been skipped
Deploy / llm (push) Has been skipped
Deploy / mc (push) Successful in 18s
Infra CI / notebooks (push) Successful in 1m12s
Infra CI / zotero (push) Successful in 23s
Infra CI / docs (push) Successful in 1m37s
Infra CI / api (push) Successful in 1m6s
Infra CI / llm (push) Successful in 47s
Infra CI / mc (push) Successful in 14s
Deploy / report (push) Successful in 14s
CI / test (push) Successful in 17m50s
Infra CI's mc job has failed on every push since docker.io/minio/mc
started returning 'pull access denied, repository does not exist' — and
nobody heard, because the generated File-failure-issue step ran
`uv run` in a job that never installs uv ("uv: command not found",
swallowed by the trailing || true). Pin quay.io/minio/mc to a release
tag; the filer installs uv when it is missing.
68 lines
2.6 KiB
YAML
68 lines
2.6 KiB
YAML
# DO NOT EDIT — generated by gen_config.py from stack.toml
|
|
# Re-generate: uv run python dev/scripts/gen_config.py
|
|
|
|
name: LLM Golden
|
|
|
|
on:
|
|
workflow_dispatch:
|
|
schedule:
|
|
- cron: "40 3 * * *"
|
|
|
|
jobs:
|
|
llm-golden:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: Checkout
|
|
uses: https://github.com/actions/checkout@v4
|
|
|
|
- name: Set up uv
|
|
run: curl -LsSf https://astral.sh/uv/install.sh | sh
|
|
env:
|
|
UV_INSTALL_DIR: /usr/local/bin
|
|
|
|
- name: Run golden longitudinal evaluation against the live chat
|
|
# golden_rc is captured (not left to `set -e`) so a failing golden
|
|
# run still docker-cps the report back and cats it — llm_golden.py
|
|
# itself exits 0 once --file-issues has filed the regressions
|
|
# (nb_issue_filer is the failure channel here), but this still
|
|
# guards the case where it exits nonzero before reaching that
|
|
# point (e.g. a crash while loading the golden set).
|
|
env:
|
|
GITEA_TOKEN: ${{ secrets.DEPLOY_TOKEN }}
|
|
run: |
|
|
set -euo pipefail
|
|
docker exec llm mkdir -p /tmp/golden
|
|
docker cp dev/scripts/llm_golden.py llm:/tmp/golden/
|
|
docker cp dev/scripts/nb_issue_filer.py llm:/tmp/golden/
|
|
docker cp tests/llm/golden_lineage.yaml llm:/tmp/golden/
|
|
docker exec \
|
|
-e GITEA_TOKEN \
|
|
-e GITEA_API_BASE=http://git:3000/api/v1 \
|
|
-e NB_ISSUE_LABEL=llm \
|
|
llm \
|
|
uv run --no-sync --project /app python /tmp/golden/llm_golden.py run \
|
|
--url http://localhost:8000 \
|
|
--set /tmp/golden/golden_lineage.yaml \
|
|
--report /tmp/golden/report.json \
|
|
--file-issues --source nightly-llm-golden || golden_rc=$?
|
|
docker cp llm:/tmp/golden/report.json llm-golden-report.json || true
|
|
cat llm-golden-report.json || true
|
|
exit "${golden_rc:-0}"
|
|
|
|
- name: File failure issue
|
|
if: failure()
|
|
env:
|
|
GITEA_TOKEN: ${{ secrets.DEPLOY_TOKEN }}
|
|
run: |
|
|
# Jobs that never install uv (image builds, infra checks) still
|
|
# need it here — otherwise the filer dies with "uv: command not
|
|
# found" and the failure is never filed.
|
|
command -v uv >/dev/null 2>&1 || curl -LsSf https://astral.sh/uv/install.sh | sh
|
|
export PATH="$HOME/.local/bin:$PATH"
|
|
uv sync --no-dev --quiet 2>/dev/null || true
|
|
uv run python -m api.diag.ci \
|
|
--workflow "LLM Golden" --job "llm-golden" \
|
|
--run "${{ github.run_number }}" \
|
|
--sha "${{ github.sha }}" \
|
|
--ref "${{ github.ref }}" || true
|