stack/.claude/plans/hazy-mixing-spark.md

Plan: Full Test Coverage Across All Namespaces

Context

5,555 tests at 92% line coverage, but several namespaces have zero or minimal test files. Goal: every namespace with real code gets dedicated tests.

Current state:

Namespace	Source files	Test files	Gap
ccw	314	0	41 table models + 308 doc modules, all generated
cms	161	1	~120 generated table models (helpers already tested)
bcda	12	1	client.py, store.py, log.py at 0% coverage
aco	73	16	table/ (1063 classes), dag.py, rex/ untested
bib	10	4	client.py, spider.py, translate.py untested
pfs	12	2	pipe.py untested
rex	7	3	format.py, pipe.py untested
bls	2	0	All empty files — skip

Files to Create (14 files)

Phase 1: BCDA (highest value — complex untested logic)

tests/bcda/test_log.py — JsonlHandler emit, extras, exc_info, setup() idempotency (~6 tests)

tests/bcda/conftest.py — MockTransport fixtures for auth + counting responses

tests/bcda/test_client.py — Full client coverage via MockTransport (~20 tests)

• Auth: token extraction, expiry timing, refresh on 401
• Retry: 5xx backoff, 429 rate-limit, connection error, max retries
• Export flow: start_export (202+Content-Location), poll loop, download (plain + gzip)
• Edge cases: JobExpiredError on 410, metadata (no auth)
• Patch time.sleep to zero via retry_interval=0.0 and poll_interval=0.0

tests/bcda/test_store.py — Store with fsspec.filesystem("memory") (~12 tests)

• Parquet I/O: read missing → empty, append + read back
• _current_state dedup: keeps last row per key
• Job state: append, get, list, filter by status, KeyError
• File state: append, get, dedup by URL

Phase 2: Parametrized table tests (bulk coverage, fast)

tests/ccw/__init__.py — empty package marker

tests/ccw/test_tables.py — Discover all SQLTable subclasses in ccw.table (~200 parametrized tests)

• __schema__ == "ccw", __tablename__ set, fields non-empty, model() with no args succeeds
• Also test ccw.docs modules: name, label, type, length, source attributes present

tests/cms/test_tables.py — Same pattern for cms.table.__all__ (~480 parametrized)

• Iterate __all__, validate __schema__/__tablename__ set, fields present, instantiable

tests/aco/test_tables.py — Same pattern for aco.table submodules (~300 parametrized)

• Validate qualified_name() returns schema.tablename, column_names() non-empty

Phase 3: Remaining modules

tests/aco/test_dag.py — Graph builder + renderers (~12 tests)

• build_graph: nodes from exprs, edges from inputs, external node detection
• Graph.schemas(), Graph.filter()
• to_dot: valid DOT with subgraph clusters
• to_mermaid: starts with "graph LR", has nodes + edges
• to_html: contains <script>, has Cytoscape elements JSON

tests/bib/test_client.py — Trivial (~3 tests)

• connect(":memory:") returns Store, COLLECTIONS has expected keys

tests/bib/test_spider.py — Pure regex functions (~12 tests)

• classify_url: each URL pattern returns correct name, unknown → None
• Reference extraction regex patterns (CFR, FR, section)

tests/bib/test_translate.py — Translator parsing (~15 tests)

• Mock _fetch to return canned HTML/JSON
• Each translator: URL parsing, field extraction, tag assignment

tests/pfs/test_pipe.py — Column maps + structural (~5 tests)

• _RVU_COLUMNS/_GPCI_COLUMNS have expected keys, no duplicate targets

tests/rex/test_format.py — Stub coverage (~5 tests)

• Readers raise NotImplementedError, Format model instantiates

What We Skip

• bls — all files are empty (0 lines of code)
• aco/lake/sync.py — requires DuckDB + Databricks context, integration-level
• aco/lake/unity.py — wraps Databricks SDK, can't mock meaningfully
• bib/sync.py, bib/ingest.py — require Zotero 61-table EAV schema, integration-level
• rex/express.py — all NotImplementedError stubs, already tracked by test_ast_coverage.py

Key Patterns to Reuse

• httpx.MockTransport + _counting_transport() from tests/api/test_base.py
• SQLTable base class at src/aco/table/base.py (has qualified_name(), column_names())
• Pipeline + Expr from src/aco/pipe/base.py for dag tests
• fsspec.filesystem("memory") for bcda/store tests

Implementation Order

1. tests/bcda/test_log.py (simplest, builds confidence)
2. tests/bcda/conftest.py + tests/bcda/test_client.py (high-value)
3. tests/bcda/test_store.py (high-value)
4. tests/ccw/ + tests/cms/test_tables.py + tests/aco/test_tables.py (bulk parametrized)
5. tests/aco/test_dag.py (pure functions)
6. tests/bib/test_client.py + test_spider.py + test_translate.py
7. tests/pfs/test_pipe.py + tests/rex/test_format.py

Verification

uv run ruff check tests/
uv run pytest tests/ -v --tb=short
uv run pytest --cov=aco --cov=api --cov=bcda --cov=bib --cov=ccw --cov=cms --cov=pfs --cov=rex --cov-report=term-missing tests/

Estimated: ~1,100 new tests, bringing total to ~6,650+.