docs: notebook quality gates design spec (integration tests, FE smoke, dedup filer)

docs/superpowers/specs/2026-07-09-notebook-quality-gates-design.md

@@ -0,0 +1,129 @@
+# Notebook quality gates: integration tests, frontend smoke, error auto-filing
+
+**Date:** 2026-07-09 · **Status:** approved (kert, in-session)
+
+## Why
+
+Two production incidents shipped undetected in one week:
+
+1. `pkg-supply-chain` failed silently for months (stale container name masked
+   as fake HTTP 500s) — fixed in `eba1b77`.
+2. The marimo editor shipped broken ("`d is not a constructor`" on every
+   notebook open) because `apply-overlay.sh` deleted upstream's pnpm lockfile,
+   floating the whole frontend dependency graph — fixed in `464f553`.
+
+Nothing exercised (a) notebook execution or (b) the built frontend bundle.
+The container healthcheck and home page rendered fine throughout incident 2.
+The existing `api.diag.ci` failure filer creates one issue per failed run and
+never closes them (23 duplicates accumulated for one workflow).
+
+## Components
+
+### 1. `dev/scripts/nb_integration.py` — headless notebook runner
+
+Wraps `marimo export session` (executes notebooks, writes JSON snapshots to
+`__marimo__/session/<nb>.py.json`). The script:
+
+- runs the exporter with `--continue-on-error --force-overwrite` over a
+  notebook set,
+- parses each snapshot for `outputs[].type == "error"` entries
+  (`ename`/`evalue`, format verified against marimo 0.23.13),
+- writes one consolidated JSON report (`data/nb-integration-report.json`,
+  CI artifact) mapping notebook → status/errors,
+- exits 1 if any notebook not in the allowlist has errors.
+
+Config `notebooks/nb-tests.toml`:
+
+```toml
+[ci_smoke]                    # data-independent set, runs in CI per push
+notebooks = ["sample.py", "_template.py", ...]
+
+[expected_failures]           # known-bad, reported but non-fatal
+"acodb_explorer.py" = "duckdb single-writer lock until #508-514"
+```
+
+Modes:
+- `--set ci-smoke` (CI): runs only `[ci_smoke]` notebooks in the checkout.
+- `--set all` (nightly host run): the scheduled workflow `docker exec`s into
+  the production notebooks container and runs the full mounted notebooks dir
+  against real data. MARIMO_OUTPUT_MAX_BYTES etc. inherited from service env.
+  Errors route through the filer (component 3) instead of failing loudly.
+
+### 2. `dev/scripts/nb_fe_smoke.py` — frontend smoke gate
+
+Productionized version of the playwright probe that reproduced incident 2:
+loads `/` and `/?file=<nb>` headlessly, collects `console`/`pageerror`
+events, fails on any error-level event, asserts the editor actually rendered
+(cell content present). Runs via the pinned
+`mcr.microsoft.com/playwright/python` image so CI needs no browser setup.
+
+Wired in two places:
+- **Gate:** `infra-ci.yml` notebooks job, right after the `load_only` image
+  build — boots a throwaway container from the just-built image on the CI
+  docker network, probes it, tears it down. A bundle that crashes the editor
+  can no longer ship.
+- **Post-deploy check:** `deploy.yml` notebooks job probes the live service
+  after recreate.
+
+### 3. `dev/scripts/nb_issue_filer.py` + `nb-watcher` sidecar — dedup/auto-close filer
+
+Filer library + CLI. Signature = `sha1(notebook|ename|normalized evalue)`
+(paths/line numbers/addresses collapsed). Behavior per signature:
+
+- no open issue → create one (`[nb] <notebook>: <ename>: <evalue>` title,
+  body contains `nb-sig:<hash>` marker + traceback/console excerpt + source),
+- open issue exists → comment only if the last comment is older than a
+  cooldown (default 6 h), bumping an occurrence count,
+- signature not seen for 24 h (watcher) or covered notebook goes green
+  (integration run) → close with a comment.
+
+Issues are found by searching open issues for the `nb-sig:` marker (Gitea
+issue search covers bodies). Labels resolved name→ID as in
+`file_concurrency_issues.py`.
+
+`nb-watcher` is a compose sidecar (pattern: `cleanup`) on a small python
+image with the docker socket (ro) and `./notebooks` mounted:
+
+- tails `docker logs notebooks` for `Traceback`/`ERROR` blocks (server-side,
+  kernel stderr),
+- polls `notebooks/__marimo__/session/*.json` mtimes and parses new/changed
+  snapshots for error outputs (live cell errors),
+- feeds both into the filer with the same signature scheme.
+
+State (last-seen per signature, log cursor) in `data/nb-watcher-state.json`.
+
+### 4. Build reproducibility hardening
+
+`notebooks.Dockerfile`: replace `corepack prepare pnpm@latest` with
+activation of the version pinned in marimo's own `packageManager` field —
+the last floating input to the frontend build after the lockfile fix.
+
+## CI wiring (via gen_config)
+
+Workflows are generated from `stack.toml` by `dev/scripts/backends/gitea.py`;
+all changes go through generators so `gen_config.py --check` (already in the
+lint job) enforces them:
+
+- `_gen_ci`: add `notebooks-smoke` job (uv sync, `nb_integration.py --set
+  ci-smoke`) gated on notebook-related paths.
+- `_gen_infra_ci`: notebooks job gains the FE smoke step after image build.
+- new `_gen_notebooks_integration`: `notebooks-integration.yml`, nightly
+  schedule (03:30, before the 06:00 pkg-supply-chain), runs full set via
+  `docker exec` into the prod container, then FE smoke against the live
+  service; failures go through the filer (no `api.diag.ci` duplicates).
+
+## Testing
+
+- Unit tests (`tests/dev/`): snapshot parsing (fixture from a real 0.23.13
+  snapshot), signature normalization/stability, filer dedup decisions
+  (mocked Gitea API), config parsing.
+- Self-verification: ci-smoke set green locally; FE smoke red against the
+  broken image's bundle behavior (regression-verified against prod before the
+  fix), green against current; watcher dry-run against live service.
+
+## Out of scope
+
+- Fixing the duckdb single-writer lock itself (#508–514).
+- Migrating `api.diag.ci` to the dedup filer (follow-up; new system must
+  prove itself first).
+- GitHub backend parity for the new workflow generators (backend = gitea).