Files
stack/tests/dev/test_nb_issue_filer.py
kert 187e77615d
Some checks failed
CI / lint (push) Successful in 32s
CI / notebooks-smoke (push) Failing after 1m39s
Deploy / notebooks (push) Successful in 6m26s
Deploy / zotero (push) Has been skipped
Deploy / docs (push) Has been skipped
Deploy / api (push) Has been skipped
Deploy / mc (push) Has been skipped
Infra CI / notebooks (push) Successful in 48s
Infra CI / zotero (push) Successful in 27s
Infra CI / docs (push) Successful in 14s
Infra CI / api (push) Successful in 24s
Infra CI / mc (push) Successful in 13s
CI / test (push) Failing after 16m28s
Deploy / report (push) Successful in 16s
feat(notebooks): quality gates — headless integration test, FE smoke gate, dedup issue auto-filer
Three gates so notebook breakage can't ship or linger silently again
(spec: docs/superpowers/specs/2026-07-09-notebook-quality-gates-design.md):

1. nb_integration.py: runs notebooks headless via 'marimo export session',
   parses the JSON snapshots for cell errors, emits a report, exits 1 on
   unexpected failures. New ci.yml notebooks-smoke job runs the
   data-independent [ci_smoke] set (infra/marimo/nb-tests.toml) on every
   push; new nightly notebooks-integration.yml runs the full set inside
   the prod container against real data, filing failures as issues.

2. nb_fe_smoke.py: headless-browser gate that loads the editor and fails
   on any console/page error — the test that would have blocked the
   'd is not a constructor' bundle. Wired into infra-ci.yml after the
   notebooks image build (all traffic over the docker socket; -v bind
   mounts silently arrive empty in CI). Verified: exit 0 on the fixed
   image and live prod, exit 1 on a synthetic crashing page.

3. nb_issue_filer.py + nb-watcher compose sidecar: one issue per error
   signature (notebook + ename + normalized message), rate-limited
   recurrence comments, auto-close after 24h quiet — the watcher is the
   single closing authority. Tails container logs via the docker socket
   (stdlib unix-socket HTTP, stream demux) and parses live session
   snapshots. First production tick filed 9 real deduplicated issues
   (#546-#554: a real skin_subs_explorer bug, missing pyzotero/trino,
   nessie/api connectivity) under the new 'notebooks' label.

Also: notebooks.Dockerfile now lets corepack honor marimo's pinned
packageManager instead of 'pnpm@latest' — the last floating input to the
frontend build after the lockfile fix.

Tests: 6 new unit-test groups (snapshot parsing, signature stability,
dedup decisions); full suite green including notebook-layout policy
(config placed in infra/marimo/, not notebooks/).
2026-07-10 10:14:46 -04:00

124 lines
4.1 KiB
Python

"""Tests for dev/scripts/nb_issue_filer.py — signature stability and
dedup/auto-close decisions.
The filer exists because api.diag.ci files one issue per failure and never
closes them (23 duplicates accumulated for pkg-supply-chain alone). The
invariants under test: identical failures collapse to one signature across
runs, volatile details (paths, line numbers, addresses, counts) don't
change the signature, and the decide/close logic never files a duplicate
for an already-open signature.
"""
from __future__ import annotations
import importlib.util
import sys
from pathlib import Path
_SCRIPT = Path(__file__).resolve().parents[2] / "dev" / "scripts" / "nb_issue_filer.py"
_spec = importlib.util.spec_from_file_location("_nb_issue_filer", _SCRIPT)
assert _spec and _spec.loader
filer = importlib.util.module_from_spec(_spec)
sys.modules["_nb_issue_filer"] = filer
_spec.loader.exec_module(filer)
# ── signature ────────────────────────────────────────────────────
def test_signature_stable_for_identical_error():
a = filer.signature("acodb_explorer.py", "exception", "division by zero")
b = filer.signature("acodb_explorer.py", "exception", "division by zero")
assert a == b
def test_signature_differs_across_notebooks_and_errors():
base = filer.signature("a.py", "exception", "boom")
assert filer.signature("b.py", "exception", "boom") != base
assert filer.signature("a.py", "interruption", "boom") != base
assert filer.signature("a.py", "exception", "other") != base
def test_signature_ignores_volatile_details():
a = filer.signature(
"nb.py",
"exception",
'IO Error: could not open "/tmp/marimo_2487488/cell_MJUe.py" at 0x7f3a2c1b'
" (attempt 3 of 5)",
)
b = filer.signature(
"nb.py",
"exception",
'IO Error: could not open "/tmp/marimo_9911223/cell_MJUe.py" at 0x559e00aa'
" (attempt 4 of 5)",
)
assert a == b
def test_signature_is_short_hex():
sig = filer.signature("nb.py", "exception", "x")
assert len(sig) == 12
int(sig, 16) # parses as hex
# ── normalize ────────────────────────────────────────────────────
def test_normalize_collapses_paths_hex_and_numbers():
s = filer.normalize_evalue(
"failed /home/kert/notebooks/x.py line 42 addr 0xDEADBEEF took 3.14s"
)
assert "0x" not in s.lower() or "0xN" in s
assert "/home/kert" not in s
assert "42" not in s
assert "3.14" not in s
def test_normalize_keeps_error_identity():
s = filer.normalize_evalue("Could not remove file: No such file or directory")
assert "no such file or directory" in s.lower()
# ── decide ───────────────────────────────────────────────────────
def test_decide_creates_when_no_open_issue():
action = filer.decide(existing=None, last_comment_age_s=None, cooldown_s=3600)
assert action == "create"
def test_decide_comments_after_cooldown():
action = filer.decide(
existing={"number": 7}, last_comment_age_s=7200, cooldown_s=3600
)
assert action == "comment"
def test_decide_skips_within_cooldown():
action = filer.decide(
existing={"number": 7}, last_comment_age_s=60, cooldown_s=3600
)
assert action == "skip"
# ── issue body marker round-trip ─────────────────────────────────
def test_marker_embeds_and_extracts_signature():
sig = filer.signature("nb.py", "exception", "boom")
body = filer.build_body(
notebook="nb.py",
ename="exception",
evalue="boom",
source="nightly-integration",
detail="Traceback ...",
sig=sig,
)
assert filer.extract_sig(body) == sig
assert "nb.py" in body
assert "nightly-integration" in body
def test_extract_sig_none_when_absent():
assert filer.extract_sig("no marker here") is None