stack/docs/superpowers/specs/2026-07-09-notebook-quality-gates-design.md

Notebook quality gates: integration tests, frontend smoke, error auto-filing

Date: 2026-07-09 · Status: approved (kert, in-session)

Why

Two production incidents shipped undetected in one week:

1. pkg-supply-chain failed silently for months (stale container name masked as fake HTTP 500s) — fixed in eba1b77.
2. The marimo editor shipped broken ("d is not a constructor" on every notebook open) because apply-overlay.sh deleted upstream's pnpm lockfile, floating the whole frontend dependency graph — fixed in 464f553.

Nothing exercised (a) notebook execution or (b) the built frontend bundle. The container healthcheck and home page rendered fine throughout incident 2. The existing api.diag.ci failure filer creates one issue per failed run and never closes them (23 duplicates accumulated for one workflow).

Components

1. dev/scripts/nb_integration.py — headless notebook runner

Wraps marimo export session (executes notebooks, writes JSON snapshots to __marimo__/session/<nb>.py.json). The script:

• runs the exporter with --continue-on-error --force-overwrite over a notebook set,
• parses each snapshot for outputs[].type == "error" entries (ename/evalue, format verified against marimo 0.23.13),
• writes one consolidated JSON report (data/nb-integration-report.json, CI artifact) mapping notebook → status/errors,
• exits 1 if any notebook not in the allowlist has errors.

Config notebooks/nb-tests.toml:

[ci_smoke]                    # data-independent set, runs in CI per push
notebooks = ["sample.py", "_template.py", ...]

[expected_failures]           # known-bad, reported but non-fatal
"acodb_explorer.py" = "duckdb single-writer lock until #508-514"

Modes:

• --set ci-smoke (CI): runs only [ci_smoke] notebooks in the checkout.
• --set all (nightly host run): the scheduled workflow docker execs into the production notebooks container and runs the full mounted notebooks dir against real data. MARIMO_OUTPUT_MAX_BYTES etc. inherited from service env. Errors route through the filer (component 3) instead of failing loudly.

2. dev/scripts/nb_fe_smoke.py — frontend smoke gate

Productionized version of the playwright probe that reproduced incident 2: loads / and /?file=<nb> headlessly, collects console/pageerror events, fails on any error-level event, asserts the editor actually rendered (cell content present). Runs via the pinned mcr.microsoft.com/playwright/python image so CI needs no browser setup.

Wired in two places:

• Gate: infra-ci.yml notebooks job, right after the load_only image build — boots a throwaway container from the just-built image on the CI docker network, probes it, tears it down. A bundle that crashes the editor can no longer ship.
• Live-service check: the nightly workflow probes the running service. (deploy.yml only builds/pushes images — container recreation is a manual docker compose up -d — so there is no in-workflow "post-deploy" moment to hook.)

3. dev/scripts/nb_issue_filer.py + nb-watcher sidecar — dedup/auto-close filer

Filer library + CLI. Signature = sha1(notebook|ename|normalized evalue) (paths/line numbers/addresses collapsed). Behavior per signature:

• no open issue → create one ([nb] <notebook>: <ename>: <evalue> title, body contains nb-sig:<hash> marker + traceback/console excerpt + source),
• open issue exists → comment only if the last comment is older than a cooldown (default 6 h), bumping an occurrence count,
• signature not seen for 24 h (watcher) or covered notebook goes green (integration run) → close with a comment.

Issues are found by searching open issues for the nb-sig: marker (Gitea issue search covers bodies). Labels resolved name→ID as in file_concurrency_issues.py.

nb-watcher is a compose sidecar (pattern: cleanup) on a small python image with the docker socket (ro) and ./notebooks mounted:

• tails docker logs notebooks for Traceback/ERROR blocks (server-side, kernel stderr),
• polls notebooks/__marimo__/session/*.json mtimes and parses new/changed snapshots for error outputs (live cell errors),
• feeds both into the filer with the same signature scheme.

State (last-seen per signature, log cursor) in data/nb-watcher-state.json.

4. Build reproducibility hardening

notebooks.Dockerfile: replace corepack prepare pnpm@latest with activation of the version pinned in marimo's own packageManager field — the last floating input to the frontend build after the lockfile fix.

CI wiring (via gen_config)

Workflows are generated from stack.toml by dev/scripts/backends/gitea.py; all changes go through generators so gen_config.py --check (already in the lint job) enforces them:

• _gen_ci: add notebooks-smoke job (uv sync, nb_integration.py --set ci-smoke) gated on notebook-related paths.
• _gen_infra_ci: notebooks job gains the FE smoke step after image build.
• new _gen_notebooks_integration: notebooks-integration.yml, nightly schedule (03:30, before the 06:00 pkg-supply-chain), runs full set via docker exec into the prod container, then FE smoke against the live service; failures go through the filer (no api.diag.ci duplicates).

Testing

• Unit tests (tests/dev/): snapshot parsing (fixture from a real 0.23.13 snapshot), signature normalization/stability, filer dedup decisions (mocked Gitea API), config parsing.
• Self-verification: ci-smoke set green locally; FE smoke red against the broken image's bundle behavior (regression-verified against prod before the fix), green against current; watcher dry-run against live service.

Out of scope

• Fixing the duckdb single-writer lock itself (#508–514).
• Migrating api.diag.ci to the dedup filer (follow-up; new system must prove itself first).
• GitHub backend parity for the new workflow generators (backend = gitea).