stack/docs/superpowers/specs/2026-07-09-notebook-quality-gates-design.md

# Notebook quality gates: integration tests, frontend smoke, error auto-filing

**Date:** 2026-07-09 · **Status:** approved (kert, in-session)

## Why

Two production incidents shipped undetected in one week:

1. `pkg-supply-chain` failed silently for months (stale container name masked
   as fake HTTP 500s) — fixed in `eba1b77`.
2. The marimo editor shipped broken ("`d is not a constructor`" on every
   notebook open) because `apply-overlay.sh` deleted upstream's pnpm lockfile,
   floating the whole frontend dependency graph — fixed in `464f553`.

Nothing exercised (a) notebook execution or (b) the built frontend bundle.
The container healthcheck and home page rendered fine throughout incident 2.
The existing `api.diag.ci` failure filer creates one issue per failed run and
never closes them (23 duplicates accumulated for one workflow).

## Components

### 1. `dev/scripts/nb_integration.py` — headless notebook runner

Wraps `marimo export session` (executes notebooks, writes JSON snapshots to
`__marimo__/session/<nb>.py.json`). The script:

- runs the exporter with `--continue-on-error --force-overwrite` over a
  notebook set,
- parses each snapshot for `outputs[].type == "error"` entries
  (`ename`/`evalue`, format verified against marimo 0.23.13),
- writes one consolidated JSON report (`data/nb-integration-report.json`,
  CI artifact) mapping notebook → status/errors,
- exits 1 if any notebook not in the allowlist has errors.

Config `notebooks/nb-tests.toml`:

```toml
[ci_smoke]                    # data-independent set, runs in CI per push
notebooks = ["sample.py", "_template.py", ...]

[expected_failures]           # known-bad, reported but non-fatal
"acodb_explorer.py" = "duckdb single-writer lock until #508-514"
```

Modes:
- `--set ci-smoke` (CI): runs only `[ci_smoke]` notebooks in the checkout.
- `--set all` (nightly host run): the scheduled workflow `docker exec`s into
  the production notebooks container and runs the full mounted notebooks dir
  against real data. MARIMO_OUTPUT_MAX_BYTES etc. inherited from service env.
  Errors route through the filer (component 3) instead of failing loudly.

### 2. `dev/scripts/nb_fe_smoke.py` — frontend smoke gate

Productionized version of the playwright probe that reproduced incident 2:
loads `/` and `/?file=<nb>` headlessly, collects `console`/`pageerror`
events, fails on any error-level event, asserts the editor actually rendered
(cell content present). Runs via the pinned
`mcr.microsoft.com/playwright/python` image so CI needs no browser setup.

Wired in two places:
- **Gate:** `infra-ci.yml` notebooks job, right after the `load_only` image
  build — boots a throwaway container from the just-built image on the CI
  docker network, probes it, tears it down. A bundle that crashes the editor
  can no longer ship.
- **Live-service check:** the nightly workflow probes the running service.
  (`deploy.yml` only builds/pushes images — container recreation is a manual
  `docker compose up -d` — so there is no in-workflow "post-deploy" moment
  to hook.)

### 3. `dev/scripts/nb_issue_filer.py` + `nb-watcher` sidecar — dedup/auto-close filer

Filer library + CLI. Signature = `sha1(notebook|ename|normalized evalue)`
(paths/line numbers/addresses collapsed). Behavior per signature:

- no open issue → create one (`[nb] <notebook>: <ename>: <evalue>` title,
  body contains `nb-sig:<hash>` marker + traceback/console excerpt + source),
- open issue exists → comment only if the last comment is older than a
  cooldown (default 6 h), bumping an occurrence count,
- signature not seen for 24 h (watcher) or covered notebook goes green
  (integration run) → close with a comment.

Issues are found by searching open issues for the `nb-sig:` marker (Gitea
issue search covers bodies). Labels resolved name→ID as in
`file_concurrency_issues.py`.

`nb-watcher` is a compose sidecar (pattern: `cleanup`) on a small python
image with the docker socket (ro) and `./notebooks` mounted:

- tails `docker logs notebooks` for `Traceback`/`ERROR` blocks (server-side,
  kernel stderr),
- polls `notebooks/__marimo__/session/*.json` mtimes and parses new/changed
  snapshots for error outputs (live cell errors),
- feeds both into the filer with the same signature scheme.

State (last-seen per signature, log cursor) in `data/nb-watcher-state.json`.

### 4. Build reproducibility hardening

`notebooks.Dockerfile`: replace `corepack prepare pnpm@latest` with
activation of the version pinned in marimo's own `packageManager` field —
the last floating input to the frontend build after the lockfile fix.

## CI wiring (via gen_config)

Workflows are generated from `stack.toml` by `dev/scripts/backends/gitea.py`;
all changes go through generators so `gen_config.py --check` (already in the
lint job) enforces them:

- `_gen_ci`: add `notebooks-smoke` job (uv sync, `nb_integration.py --set
  ci-smoke`) gated on notebook-related paths.
- `_gen_infra_ci`: notebooks job gains the FE smoke step after image build.
- new `_gen_notebooks_integration`: `notebooks-integration.yml`, nightly
  schedule (03:30, before the 06:00 pkg-supply-chain), runs full set via
  `docker exec` into the prod container, then FE smoke against the live
  service; failures go through the filer (no `api.diag.ci` duplicates).

## Testing

- Unit tests (`tests/dev/`): snapshot parsing (fixture from a real 0.23.13
  snapshot), signature normalization/stability, filer dedup decisions
  (mocked Gitea API), config parsing.
- Self-verification: ci-smoke set green locally; FE smoke red against the
  broken image's bundle behavior (regression-verified against prod before the
  fix), green against current; watcher dry-run against live service.

## Out of scope

- Fixing the duckdb single-writer lock itself (#508–514).
- Migrating `api.diag.ci` to the dedup filer (follow-up; new system must
  prove itself first).
- GitHub backend parity for the new workflow generators (backend = gitea).