stack/.claude/plans/2026-04-23-comments-extraction-plan.md

Comments Extraction Pipeline Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Build stack comments {extract, extract-ocr, stats} to convert downloaded reg.gov attachments (~18.5K PDFs + ~9.6K DOCX) into per-comment combined.md files with YAML frontmatter, plus a DuckDB view over the index.

Architecture: New library at src/rex/comments/ with extract/combine/view modules. New CLI at src/cli/comments.py registered in cli/__init__.py. Filesystem-only resume tracking — combined.md exists ⇒ done. Per-comment work parallelized via ThreadPoolExecutor (PyMuPDF releases the GIL during decoding). OCR (extract-ocr) ships as a stub raising NotImplementedError — phase 2.

Tech Stack: PyMuPDF (pymupdf>=1.24, AGPL-3.0), python-docx>=1.1, existing typer CLI + duckdb, existing bib.connect() for inline-body lookup.

Spec: .claude/specs/2026-04-23-comments-extraction-design.md (commit 333aafa)

Tracks: #253

---

Task 1: Add dependencies

Files:

• Modify: pyproject.toml

• Step 1: Add pymupdf and python-docx to dependencies

In pyproject.toml, find the dependencies = [...] array (the project-level one, around line 30-100) and add two lines (alphabetical):

    "pymupdf>=1.24",     # AGPL-3.0 — flag if shipping outside internal use
    "python-docx>=1.1",

• Step 2: Sync the lockfile

Run: uv sync --quiet Expected: completes without error, uv.lock updates.

• Step 3: Verify imports work

Run: uv run python -c "import fitz, docx; print(fitz.__doc__[:40], docx.__version__)" Expected: prints PyMuPDF doc string + python-docx version, no ImportError.

• Step 4: Commit

git add pyproject.toml uv.lock
git commit -m "deps: add pymupdf + python-docx for comment extraction (refs #253)"

---

Task 2: Skeleton module + public API stubs

Files:

• Create: src/rex/comments/__init__.py

• Create: src/rex/comments/extract.py

• Test: tests/rex/comments/__init__.py (empty)

• Test: tests/rex/comments/test_init.py

• Step 1: Write the failing test

Create tests/rex/comments/__init__.py as an empty file.

Create tests/rex/comments/test_init.py:

"""Public API surface tests for rex.comments."""

from __future__ import annotations


def test_public_api_exports():
    from rex import comments

    assert hasattr(comments, "extract_attachment")
    assert hasattr(comments, "extract_comment")


def test_extract_attachment_returns_namedtuple_like():
    from rex.comments import ExtractResult, extract_attachment

    # Signature check only — implementation comes in Task 3.
    assert callable(extract_attachment)
    assert ExtractResult.__annotations__ == {
        "text": str,
        "status": str,
        "chars": int,
    }

• Step 2: Run test to verify it fails

Run: uv run pytest tests/rex/comments/test_init.py -v Expected: FAIL with ModuleNotFoundError: No module named 'rex.comments'.

• Step 3: Create the module skeleton

Create src/rex/comments/extract.py:

"""Per-attachment text extraction (PDF, DOCX, plain text).

Pulls text out of files downloaded by `bib backfill-comments`.
PyMuPDF for PDFs (releases the GIL during decoding so threads work),
python-docx for DOCX, plain-read fallback for txt/html.
"""

from __future__ import annotations

from dataclasses import dataclass
from pathlib import Path


@dataclass(frozen=True, slots=True)
class ExtractResult:
    """Result of extracting one attachment."""

    text: str
    status: str  # ok | ocr_needed | failed | unsupported
    chars: int


def extract_attachment(path: Path) -> ExtractResult:
    """Extract text from a single attachment file.

    Implementation lands in Task 3.
    """
    raise NotImplementedError

Create src/rex/comments/__init__.py:

"""rex.comments — extract and aggregate CMS rulemaking comment text.

Public API:
    extract_attachment(path) -> ExtractResult — single file
    extract_comment(comment_dir) -> Path | None — write combined.md for a dir
"""

from __future__ import annotations

from rex.comments.extract import ExtractResult, extract_attachment

__all__ = ["ExtractResult", "extract_attachment", "extract_comment"]


def extract_comment(comment_dir):  # type: ignore[no-untyped-def]
    """Write combined.md for one comment dir. Implementation in Task 5."""
    raise NotImplementedError

• Step 4: Run test to verify it passes

Run: uv run pytest tests/rex/comments/test_init.py -v Expected: 2 PASS.

• Step 5: Commit

git add src/rex/comments/__init__.py src/rex/comments/extract.py \
        tests/rex/comments/__init__.py tests/rex/comments/test_init.py
git commit -m "feat(comments): module skeleton + ExtractResult dataclass (refs #253)"

---

Task 3: PDF extraction (PyMuPDF)

Files:

• Modify: src/rex/comments/extract.py

• Test: tests/rex/comments/test_extract_pdf.py

• Step 1: Write failing tests

Create tests/rex/comments/test_extract_pdf.py:

"""PDF extraction via PyMuPDF."""

from __future__ import annotations

from pathlib import Path

import fitz  # pymupdf

from rex.comments import extract_attachment


def _make_pdf(path: Path, text: str = "Hello world. This is page text.") -> Path:
    doc = fitz.open()
    page = doc.new_page()
    page.insert_text((50, 72), text)
    doc.save(str(path))
    doc.close()
    return path


def test_extract_pdf_with_text(tmp_path: Path):
    pdf = _make_pdf(tmp_path / "a.pdf", "Position statement: oppose the rule.")
    result = extract_attachment(pdf)
    assert result.status == "ok"
    assert "Position statement" in result.text
    assert result.chars == len(result.text)
    assert result.chars > 50


def test_extract_pdf_image_only_marked_ocr_needed(tmp_path: Path):
    # Empty page — no text content, simulates a scanned image.
    doc = fitz.open()
    doc.new_page()
    pdf = tmp_path / "scan.pdf"
    doc.save(str(pdf))
    doc.close()

    result = extract_attachment(pdf)
    assert result.status == "ocr_needed"
    assert result.chars <= 50


def test_extract_pdf_corrupted_marked_failed(tmp_path: Path):
    bad = tmp_path / "bad.pdf"
    bad.write_bytes(b"not a real pdf")

    result = extract_attachment(bad)
    assert result.status == "failed"
    assert result.text == ""
    assert result.chars == 0

• Step 2: Run tests to verify they fail

Run: uv run pytest tests/rex/comments/test_extract_pdf.py -v Expected: 3 FAIL with NotImplementedError.

• Step 3: Implement PDF extraction

Replace the extract_attachment function body in src/rex/comments/extract.py:

import logging

import fitz  # pymupdf

log = logging.getLogger(__name__)


_SUPPORTED = {".pdf", ".docx", ".txt", ".html", ".htm"}
_OCR_THRESHOLD = 50  # below this many chars, treat as scanned image


def extract_attachment(path: Path) -> ExtractResult:
    """Extract text from one attachment.

    Status values:
      ok          — text extracted, > 50 chars
      ocr_needed  — file parsed but ≤ 50 chars (likely scanned image PDF)
      failed      — exception during parsing
      unsupported — file extension we don't handle
    """
    suffix = path.suffix.lower()
    if suffix not in _SUPPORTED:
        return ExtractResult(text="", status="unsupported", chars=0)
    if suffix == ".pdf":
        return _extract_pdf(path)
    # .docx and plain text land in later tasks; default to unsupported here so
    # the test in Task 3 isolates PDF behaviour.
    return ExtractResult(text="", status="unsupported", chars=0)


def _extract_pdf(path: Path) -> ExtractResult:
    try:
        with fitz.open(path) as doc:
            text = "\n\n".join(page.get_text() for page in doc)
    except Exception as e:  # noqa: BLE001 — pymupdf raises a few different types
        log.warning("pdf extract failed for %s: %s", path, e)
        return ExtractResult(text="", status="failed", chars=0)
    chars = len(text)
    status = "ok" if chars > _OCR_THRESHOLD else "ocr_needed"
    return ExtractResult(text=text, status=status, chars=chars)

The full file should now read:

"""Per-attachment text extraction (PDF, DOCX, plain text).

Pulls text out of files downloaded by `bib backfill-comments`.
PyMuPDF for PDFs (releases the GIL during decoding so threads work),
python-docx for DOCX, plain-read fallback for txt/html.
"""

from __future__ import annotations

import logging
from dataclasses import dataclass
from pathlib import Path

import fitz  # pymupdf

log = logging.getLogger(__name__)

_SUPPORTED = {".pdf", ".docx", ".txt", ".html", ".htm"}
_OCR_THRESHOLD = 50


@dataclass(frozen=True, slots=True)
class ExtractResult:
    """Result of extracting one attachment."""

    text: str
    status: str  # ok | ocr_needed | failed | unsupported
    chars: int


def extract_attachment(path: Path) -> ExtractResult:
    """Extract text from one attachment.

    Status values:
      ok          — text extracted, > 50 chars
      ocr_needed  — file parsed but ≤ 50 chars (likely scanned image PDF)
      failed      — exception during parsing
      unsupported — file extension we don't handle
    """
    suffix = path.suffix.lower()
    if suffix not in _SUPPORTED:
        return ExtractResult(text="", status="unsupported", chars=0)
    if suffix == ".pdf":
        return _extract_pdf(path)
    return ExtractResult(text="", status="unsupported", chars=0)


def _extract_pdf(path: Path) -> ExtractResult:
    try:
        with fitz.open(path) as doc:
            text = "\n\n".join(page.get_text() for page in doc)
    except Exception as e:  # noqa: BLE001
        log.warning("pdf extract failed for %s: %s", path, e)
        return ExtractResult(text="", status="failed", chars=0)
    chars = len(text)
    status = "ok" if chars > _OCR_THRESHOLD else "ocr_needed"
    return ExtractResult(text=text, status=status, chars=chars)

• Step 4: Run tests to verify they pass

Run: uv run pytest tests/rex/comments/test_extract_pdf.py -v Expected: 3 PASS.

• Step 5: Commit

git add src/rex/comments/extract.py tests/rex/comments/test_extract_pdf.py
git commit -m "feat(comments): PDF extraction via pymupdf with status taxonomy (refs #253)"

---

Task 4: DOCX + plain-text extraction

Files:

• Modify: src/rex/comments/extract.py

• Test: tests/rex/comments/test_extract_docx.py

• Test: tests/rex/comments/test_extract_other.py

• Step 1: Write failing tests

Create tests/rex/comments/test_extract_docx.py:

"""DOCX extraction via python-docx."""

from __future__ import annotations

from pathlib import Path

import docx

from rex.comments import extract_attachment


def _make_docx(path: Path, paragraphs: list[str]) -> Path:
    d = docx.Document()
    for p in paragraphs:
        d.add_paragraph(p)
    d.save(str(path))
    return path


def test_extract_docx_with_text(tmp_path: Path):
    p = _make_docx(
        tmp_path / "a.docx",
        ["First paragraph of comment letter.", "Second paragraph with details."],
    )
    result = extract_attachment(p)
    assert result.status == "ok"
    assert "First paragraph" in result.text
    assert "Second paragraph" in result.text


def test_extract_docx_empty_marked_ocr_needed(tmp_path: Path):
    p = _make_docx(tmp_path / "empty.docx", [])
    result = extract_attachment(p)
    # No paragraphs ⇒ 0 chars ⇒ ocr_needed (we treat empty docx the same as
    # an image-only pdf — body lives somewhere we can't read it).
    assert result.status == "ocr_needed"
    assert result.chars <= 50


def test_extract_docx_corrupted_marked_failed(tmp_path: Path):
    bad = tmp_path / "bad.docx"
    bad.write_bytes(b"this is not a docx zip")
    result = extract_attachment(bad)
    assert result.status == "failed"

Create tests/rex/comments/test_extract_other.py:

"""Plain-text fallback + unsupported extensions."""

from __future__ import annotations

from pathlib import Path

from rex.comments import extract_attachment


def test_extract_txt(tmp_path: Path):
    p = tmp_path / "note.txt"
    p.write_text("This is a plain text comment longer than fifty characters total.")
    result = extract_attachment(p)
    assert result.status == "ok"
    assert "plain text comment" in result.text


def test_extract_html_strips_tags(tmp_path: Path):
    p = tmp_path / "page.html"
    p.write_text(
        "<html><body><p>This is a comment letter</p>"
        "<p>with two paragraphs of substance.</p></body></html>"
    )
    result = extract_attachment(p)
    assert result.status == "ok"
    assert "This is a comment letter" in result.text
    assert "<p>" not in result.text


def test_extract_unknown_extension(tmp_path: Path):
    p = tmp_path / "weird.xyz"
    p.write_bytes(b"some bytes")
    result = extract_attachment(p)
    assert result.status == "unsupported"
    assert result.chars == 0

• Step 2: Run tests to verify they fail

Run: uv run pytest tests/rex/comments/test_extract_docx.py tests/rex/comments/test_extract_other.py -v Expected: docx tests fail (status unsupported instead of ok/ocr_needed/failed); txt test fails (still unsupported); html test fails; unknown-extension test should already PASS.

• Step 3: Implement DOCX, txt, html

In src/rex/comments/extract.py, replace the body of extract_attachment so it dispatches to all three handlers, and add the new helper functions. Add import re to imports. The new dispatch and helpers:

def extract_attachment(path: Path) -> ExtractResult:
    """Extract text from one attachment. See module docstring for status values."""
    suffix = path.suffix.lower()
    if suffix == ".pdf":
        return _extract_pdf(path)
    if suffix == ".docx":
        return _extract_docx(path)
    if suffix in (".txt", ".html", ".htm"):
        return _extract_text_like(path, suffix)
    return ExtractResult(text="", status="unsupported", chars=0)


def _extract_docx(path: Path) -> ExtractResult:
    try:
        import docx  # local import — only loaded when needed

        d = docx.Document(str(path))
        text = "\n\n".join(p.text for p in d.paragraphs if p.text)
    except Exception as e:  # noqa: BLE001
        log.warning("docx extract failed for %s: %s", path, e)
        return ExtractResult(text="", status="failed", chars=0)
    chars = len(text)
    status = "ok" if chars > _OCR_THRESHOLD else "ocr_needed"
    return ExtractResult(text=text, status=status, chars=chars)


def _extract_text_like(path: Path, suffix: str) -> ExtractResult:
    try:
        raw = path.read_text(encoding="utf-8", errors="ignore")
    except OSError as e:
        log.warning("text-like extract failed for %s: %s", path, e)
        return ExtractResult(text="", status="failed", chars=0)
    text = _strip_html(raw) if suffix in (".html", ".htm") else raw
    chars = len(text)
    status = "ok" if chars > _OCR_THRESHOLD else "ocr_needed"
    return ExtractResult(text=text, status=status, chars=chars)


def _strip_html(s: str) -> str:
    """Cheap HTML→text — break on block tags, drop the rest."""
    s = re.sub(r"<(script|style)[^>]*>.*?</\1>", "", s, flags=re.S | re.I)
    s = re.sub(r"<(p|div|br|li|h[1-6])[^>]*>", "\n", s, flags=re.I)
    s = re.sub(r"<[^>]+>", "", s)
    return re.sub(r"\n\s*\n+", "\n\n", s).strip()

Remove the now-unused _SUPPORTED constant (extension dispatch is explicit).

• Step 4: Run tests

Run: uv run pytest tests/rex/comments/ -v Expected: all tests in test_extract_pdf, test_extract_docx, test_extract_other PASS (8 total in this batch).

• Step 5: Commit

git add src/rex/comments/extract.py tests/rex/comments/test_extract_docx.py tests/rex/comments/test_extract_other.py
git commit -m "feat(comments): DOCX + txt/html extraction (refs #253)"

---

Task 5: Per-comment combine — write combined.md with frontmatter

Files:

• Create: src/rex/comments/combine.py

• Modify: src/rex/comments/__init__.py

• Test: tests/rex/comments/test_combine.py

• Step 1: Write failing test

Create tests/rex/comments/test_combine.py:

"""Per-comment combine — write combined.md with YAML frontmatter."""

from __future__ import annotations

from pathlib import Path

import fitz

from rex.comments import extract_comment
from rex.comments.combine import parse_combined


def _make_pdf(path: Path, text: str) -> None:
    doc = fitz.open()
    doc.new_page().insert_text((50, 72), text)
    doc.save(str(path))
    doc.close()


def _setup_comment_dir(tmp_path: Path, cid: str = "CMS-2024-0001-0042") -> Path:
    """Layout matches .state/comments/{docket}/{cid}/."""
    docket = "-".join(cid.split("-")[:3])
    d = tmp_path / docket / cid
    d.mkdir(parents=True)
    _make_pdf(d / "attachment_1.pdf", "Position: oppose the proposed rule.")
    return d


def test_extract_comment_writes_combined_md(tmp_path: Path):
    d = _setup_comment_dir(tmp_path)
    out = extract_comment(d, inline_body="Inline body text from items.abstract.")
    assert out == d / "combined.md"
    assert out.is_file()


def test_combined_md_has_frontmatter_and_body(tmp_path: Path):
    d = _setup_comment_dir(tmp_path)
    extract_comment(d, inline_body="Inline body.")
    text = (d / "combined.md").read_text()
    fm, body = parse_combined(text)
    assert fm["comment_id"] == "CMS-2024-0001-0042"
    assert fm["docket_id"] == "CMS-2024-0001"
    assert fm["chars"] > 0
    assert len(fm["attachments"]) == 1
    assert fm["attachments"][0]["file"] == "attachment_1.pdf"
    assert fm["attachments"][0]["status"] == "ok"
    assert "## Inline comment" in body
    assert "Inline body." in body
    assert "## attachment_1.pdf" in body
    assert "Position: oppose" in body


def test_extract_comment_is_idempotent(tmp_path: Path):
    d = _setup_comment_dir(tmp_path)
    first = extract_comment(d, inline_body="x")
    mtime1 = first.stat().st_mtime
    second = extract_comment(d, inline_body="x")
    assert second == first
    assert second.stat().st_mtime == mtime1  # no rewrite on second call


def test_extract_comment_force_rewrites(tmp_path: Path):
    d = _setup_comment_dir(tmp_path)
    extract_comment(d, inline_body="first")
    extract_comment(d, inline_body="second", force=True)
    body = (d / "combined.md").read_text()
    assert "second" in body
    assert "first" not in body


def test_extract_comment_no_attachments(tmp_path: Path):
    d = tmp_path / "CMS-2024-0001" / "CMS-2024-0001-0099"
    d.mkdir(parents=True)
    out = extract_comment(d, inline_body="Inline only.")
    fm, body = parse_combined(out.read_text())
    assert fm["attachments"] == []
    assert "Inline only." in body


def test_extract_comment_handles_failed_attachment(tmp_path: Path):
    d = tmp_path / "CMS-2024-0001" / "CMS-2024-0001-0100"
    d.mkdir(parents=True)
    (d / "attachment_1.pdf").write_bytes(b"not a real pdf")
    out = extract_comment(d, inline_body="x")
    fm, _body = parse_combined(out.read_text())
    assert fm["attachments"][0]["status"] == "failed"

• Step 2: Run test to verify it fails

Run: uv run pytest tests/rex/comments/test_combine.py -v Expected: FAIL — extract_comment raises NotImplementedError and parse_combined doesn't exist yet.

• Step 3: Implement combine.py

Create src/rex/comments/combine.py:

"""Aggregate inline comment + attachment text into combined.md.

One comment dir → one combined.md with YAML frontmatter (status of each
attachment, char counts) and a markdown body (inline body + per-attachment
sections). Resumable: existing combined.md is a no-op unless force=True.
"""

from __future__ import annotations

from datetime import datetime, timezone
from pathlib import Path
from typing import Any

import yaml

from rex.comments.extract import ExtractResult, extract_attachment

_FILENAME = "combined.md"


def extract_comment(
    comment_dir: Path,
    *,
    inline_body: str = "",
    force: bool = False,
) -> Path:
    """Extract every attachment in *comment_dir* and write *combined.md*.

    Returns the path to combined.md. If it already exists and ``force``
    is False, returns it without doing any work (resume rule).

    *comment_dir* must be ``.../{docket_id}/{comment_id}/``.
    """
    out = comment_dir / _FILENAME
    if out.is_file() and not force:
        return out

    docket_id = comment_dir.parent.name
    comment_id = comment_dir.name

    attachments_meta: list[dict[str, Any]] = []
    body_sections: list[str] = []

    for path in sorted(_attachment_paths(comment_dir)):
        result = extract_attachment(path)
        attachments_meta.append(
            {"file": path.name, "status": result.status, "chars": result.chars}
        )
        body_sections.append(_render_section(path.name, result))

    inline_section = (
        f"## Inline comment\n\n{inline_body.strip()}\n" if inline_body.strip() else ""
    )
    body_text = inline_section + "\n".join(body_sections)
    total_chars = len(inline_body) + sum(a["chars"] for a in attachments_meta)

    frontmatter = {
        "comment_id": comment_id,
        "docket_id": docket_id,
        "extracted_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        "chars": total_chars,
        "attachments": attachments_meta,
    }

    out.write_text(_emit(frontmatter, body_text))
    return out


def parse_combined(text: str) -> tuple[dict[str, Any], str]:
    """Split a combined.md into (frontmatter dict, body markdown)."""
    if not text.startswith("---\n"):
        raise ValueError("missing frontmatter")
    end = text.find("\n---\n", 4)
    if end < 0:
        raise ValueError("unterminated frontmatter")
    fm = yaml.safe_load(text[4:end]) or {}
    body = text[end + len("\n---\n") :].lstrip("\n")
    return fm, body


# ── internals ─────────────────────────────────────────────────────


def _attachment_paths(comment_dir: Path) -> list[Path]:
    return [
        p
        for p in comment_dir.iterdir()
        if p.is_file() and p.name != _FILENAME and not p.name.startswith(".")
    ]


def _render_section(filename: str, result: ExtractResult) -> str:
    if result.status == "ok":
        body = result.text.strip()
    elif result.status == "ocr_needed":
        body = f"_(ocr_needed — {result.chars} chars extracted)_"
    elif result.status == "failed":
        body = "_(failed — extraction error, see logs)_"
    else:  # unsupported
        body = "_(unsupported file type)_"
    return f"## {filename}\n\n{body}\n"


def _emit(frontmatter: dict[str, Any], body: str) -> str:
    yaml_text = yaml.safe_dump(frontmatter, sort_keys=False, default_flow_style=False)
    return f"---\n{yaml_text}---\n\n{body}"

• Step 4: Wire into the package public API

Replace src/rex/comments/__init__.py:

"""rex.comments — extract and aggregate CMS rulemaking comment text.

Public API:
    extract_attachment(path) -> ExtractResult — single file
    extract_comment(comment_dir, *, inline_body, force) -> Path — write combined.md
"""

from __future__ import annotations

from rex.comments.combine import extract_comment, parse_combined
from rex.comments.extract import ExtractResult, extract_attachment

__all__ = [
    "ExtractResult",
    "extract_attachment",
    "extract_comment",
    "parse_combined",
]

• Step 5: Run tests to verify pass

Run: uv run pytest tests/rex/comments/ -v Expected: all tests across test_init, test_extract_pdf, test_extract_docx, test_extract_other, test_combine PASS (~16 total).

• Step 6: Commit

git add src/rex/comments/combine.py src/rex/comments/__init__.py tests/rex/comments/test_combine.py
git commit -m "feat(comments): combined.md writer with yaml frontmatter (refs #253)"

---

Task 6: Walker — iterate dirs, parallelize, look up inline body from bib

Files:

• Create: src/rex/comments/walker.py

• Test: tests/rex/comments/test_walker.py

• Step 1: Write failing test

Create tests/rex/comments/test_walker.py:

"""Walker — iterate .state/comments/, look up inline body, parallelize."""

from __future__ import annotations

from pathlib import Path

import fitz

from rex.comments.walker import walk_and_extract


def _make_pdf(path: Path, text: str = "x" * 200) -> None:
    doc = fitz.open()
    doc.new_page().insert_text((50, 72), text)
    doc.save(str(path))
    doc.close()


def _setup_two_comments(root: Path) -> tuple[Path, Path]:
    a = root / "CMS-2024-0001" / "CMS-2024-0001-0001"
    b = root / "CMS-2024-0001" / "CMS-2024-0001-0002"
    a.mkdir(parents=True)
    b.mkdir(parents=True)
    _make_pdf(a / "attachment_1.pdf", "Comment A body content.")
    _make_pdf(b / "attachment_1.pdf", "Comment B body content.")
    return a, b


def _stub_inline_body(_cid: str) -> str:
    return f"inline body for stub"


def test_walk_and_extract_writes_all_combined(tmp_path: Path):
    a, b = _setup_two_comments(tmp_path)

    stats = walk_and_extract(
        tmp_path, inline_body_lookup=_stub_inline_body, workers=2
    )

    assert (a / "combined.md").is_file()
    assert (b / "combined.md").is_file()
    assert stats["written"] == 2
    assert stats["skipped"] == 0


def test_walk_and_extract_skips_existing(tmp_path: Path):
    a, _b = _setup_two_comments(tmp_path)
    (a / "combined.md").write_text("---\ncomment_id: x\n---\n\npre-existing\n")

    stats = walk_and_extract(
        tmp_path, inline_body_lookup=_stub_inline_body, workers=1
    )

    assert stats["written"] == 1  # only the second one
    assert stats["skipped"] == 1
    assert "pre-existing" in (a / "combined.md").read_text()  # untouched


def test_walk_and_extract_filters_by_docket(tmp_path: Path):
    _setup_two_comments(tmp_path)
    other = tmp_path / "CMS-2099-9999" / "CMS-2099-9999-0001"
    other.mkdir(parents=True)
    _make_pdf(other / "attachment_1.pdf", "other docket content")

    stats = walk_and_extract(
        tmp_path,
        inline_body_lookup=_stub_inline_body,
        docket="CMS-2099-9999",
        workers=1,
    )

    assert stats["written"] == 1
    assert (other / "combined.md").is_file()


def test_walk_and_extract_respects_limit(tmp_path: Path):
    _setup_two_comments(tmp_path)
    stats = walk_and_extract(
        tmp_path, inline_body_lookup=_stub_inline_body, limit=1, workers=1
    )
    assert stats["written"] == 1

• Step 2: Run tests to verify they fail

Run: uv run pytest tests/rex/comments/test_walker.py -v Expected: FAIL with ModuleNotFoundError: No module named 'rex.comments.walker'.

• Step 3: Implement walker

Create src/rex/comments/walker.py:

"""Walk .state/comments/ and run extract_comment over every dir.

Parallelized with ThreadPoolExecutor. PyMuPDF releases the GIL during
PDF decoding so threads (not processes) are the right primitive — keeps
shared state simple and avoids fork overhead per attachment.
"""

from __future__ import annotations

import logging
from collections.abc import Callable
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path

from rex.comments.combine import extract_comment

log = logging.getLogger(__name__)

_COMBINED = "combined.md"


def walk_and_extract(
    root: Path,
    *,
    inline_body_lookup: Callable[[str], str],
    docket: str | None = None,
    limit: int | None = None,
    workers: int = 8,
    force: bool = False,
) -> dict[str, int]:
    """Process every comment dir under *root*.

    *root* should be the dir containing per-docket subdirs (typically
    ``.state/comments/``). *inline_body_lookup* is called with each
    ``comment_id`` and returns the inline body text from bib.sqlite (or
    "" if missing).

    Returns counts: ``{"written": N, "skipped": N, "failed": N}``.
    """
    dirs = list(_iter_comment_dirs(root, docket=docket, limit=limit, force=force))
    stats = {"written": 0, "skipped": 0, "failed": 0}

    if not dirs:
        return stats

    with ThreadPoolExecutor(max_workers=workers) as pool:
        futures = {
            pool.submit(_one, d, inline_body_lookup, force): d for d in dirs
        }
        for fut in as_completed(futures):
            outcome = fut.result()
            stats[outcome] += 1
    return stats


def _iter_comment_dirs(
    root: Path,
    *,
    docket: str | None,
    limit: int | None,
    force: bool,
):
    """Yield comment dirs (skip ones with combined.md unless force=True)."""
    n = 0
    for docket_dir in sorted(root.iterdir()):
        if not docket_dir.is_dir():
            continue
        if docket and docket_dir.name != docket:
            continue
        for cdir in sorted(docket_dir.iterdir()):
            if not cdir.is_dir():
                continue
            if (cdir / _COMBINED).is_file() and not force:
                continue
            yield cdir
            n += 1
            if limit and n >= limit:
                return


def _one(
    comment_dir: Path,
    inline_body_lookup: Callable[[str], str],
    force: bool,
) -> str:
    """Process one comment dir. Returns "written" / "skipped" / "failed"."""
    if (comment_dir / _COMBINED).is_file() and not force:
        return "skipped"
    try:
        body = inline_body_lookup(comment_dir.name)
    except Exception as e:  # noqa: BLE001
        log.warning("inline body lookup failed for %s: %s", comment_dir.name, e)
        body = ""
    try:
        extract_comment(comment_dir, inline_body=body, force=force)
    except Exception as e:  # noqa: BLE001
        log.warning("extract_comment failed for %s: %s", comment_dir, e)
        return "failed"
    return "written"

Note: the "skipped" branch in _one is a defensive double-check (the iterator already filtered) — keeps the function safe to call directly in tests.

Update the _iter_comment_dirs call in tests for the "skips existing" test: that test should report skipped=1 because the iterator filters out the pre-existing dir. Adjust the assertion if the iterator-only count differs from the expected behaviour.

Wait — the iterator already skips dirs with combined.md, so the second test would see dirs=[b only] and the result would be {written: 1, skipped: 0}. The test expects skipped: 1. Fix: update walk_and_extract to also count skipped dirs from the iterator. Replace _iter_comment_dirs and the call:

def walk_and_extract(
    root: Path,
    *,
    inline_body_lookup: Callable[[str], str],
    docket: str | None = None,
    limit: int | None = None,
    workers: int = 8,
    force: bool = False,
) -> dict[str, int]:
    """Process every comment dir under *root*. See module docstring."""
    candidate_dirs, skipped = _collect_dirs(
        root, docket=docket, limit=limit, force=force
    )
    stats = {"written": 0, "skipped": skipped, "failed": 0}

    if not candidate_dirs:
        return stats

    with ThreadPoolExecutor(max_workers=workers) as pool:
        futures = {
            pool.submit(_one, d, inline_body_lookup, force): d
            for d in candidate_dirs
        }
        for fut in as_completed(futures):
            stats[fut.result()] += 1
    return stats


def _collect_dirs(
    root: Path,
    *,
    docket: str | None,
    limit: int | None,
    force: bool,
) -> tuple[list[Path], int]:
    """Return (dirs to process, count of dirs skipped due to existing combined.md)."""
    todo: list[Path] = []
    skipped = 0
    for docket_dir in sorted(root.iterdir()):
        if not docket_dir.is_dir():
            continue
        if docket and docket_dir.name != docket:
            continue
        for cdir in sorted(docket_dir.iterdir()):
            if not cdir.is_dir():
                continue
            if (cdir / _COMBINED).is_file() and not force:
                skipped += 1
                continue
            todo.append(cdir)
            if limit and len(todo) >= limit:
                return todo, skipped
    return todo, skipped

Use this version (not the _iter_comment_dirs generator above). Remove the unused _iter_comment_dirs helper.

• Step 4: Run tests

Run: uv run pytest tests/rex/comments/test_walker.py -v Expected: all 4 PASS.

• Step 5: Commit

git add src/rex/comments/walker.py tests/rex/comments/test_walker.py
git commit -m "feat(comments): parallel walker for .state/comments/ (refs #253)"

---

Task 7: DuckDB view + index rebuild

Files:

• Create: src/rex/comments/view.py

• Test: tests/rex/comments/test_view.py

• Step 1: Write failing test

Create tests/rex/comments/test_view.py:

"""DuckDB view over combined.md index."""

from __future__ import annotations

from pathlib import Path

import duckdb

from rex.comments import extract_comment
from rex.comments.view import body_path, rebuild_index, register


def _seed_two(tmp_path: Path) -> None:
    import fitz

    for cid_suffix, text in (("0001", "Long body " * 20), ("0002", "x")):
        cdir = tmp_path / "CMS-2024-0001" / f"CMS-2024-0001-{cid_suffix}"
        cdir.mkdir(parents=True)
        doc = fitz.open()
        doc.new_page().insert_text((50, 72), text)
        doc.save(str(cdir / "attachment_1.pdf"))
        doc.close()
        extract_comment(cdir, inline_body="inline body")


def test_rebuild_index_writes_csv(tmp_path: Path):
    _seed_two(tmp_path)
    csv = rebuild_index(tmp_path)
    assert csv == tmp_path / "_index.csv"
    rows = csv.read_text().splitlines()
    assert rows[0].startswith("comment_id,docket_id,")
    assert len(rows) == 3  # header + 2 comments


def test_register_view_queryable(tmp_path: Path):
    _seed_two(tmp_path)
    rebuild_index(tmp_path)

    con = duckdb.connect(":memory:")
    register(con, root=tmp_path)
    rows = con.execute(
        "SELECT comment_id, n_attachments, n_ok FROM comments_index ORDER BY comment_id"
    ).fetchall()
    assert len(rows) == 2
    assert rows[0][0] == "CMS-2024-0001-0001"
    assert rows[0][1] == 1  # n_attachments
    assert rows[0][2] == 1  # n_ok
    # Second row: short text → ocr_needed
    assert rows[1][2] == 0


def test_body_path():
    p = body_path(
        Path("/x/.state/comments"),
        comment_id="CMS-2024-0001-0099",
        docket_id="CMS-2024-0001",
    )
    assert p == Path("/x/.state/comments/CMS-2024-0001/CMS-2024-0001-0099/combined.md")

• Step 2: Run test to verify it fails

Run: uv run pytest tests/rex/comments/test_view.py -v Expected: FAIL — rex.comments.view doesn't exist.

• Step 3: Implement view module

Create src/rex/comments/view.py:

"""DuckDB view over the combined.md index.

We don't load body text into DuckDB — bodies stay on disk. The view
exposes per-comment metadata (status counts, chars) for filtering and
aggregation. Downstream consumers read body text from disk via
``body_path()``.
"""

from __future__ import annotations

import csv
from pathlib import Path

import duckdb

from rex.comments.combine import parse_combined

_INDEX = "_index.csv"
_VIEW = "comments_index"
_COLUMNS = (
    "comment_id",
    "docket_id",
    "extracted_at",
    "chars",
    "n_attachments",
    "n_ok",
    "n_ocr_needed",
    "n_failed",
)


def rebuild_index(root: Path) -> Path:
    """Walk *root* and write _index.csv. Returns the path written."""
    out = root / _INDEX
    rows = list(_iter_index_rows(root))
    with out.open("w", newline="") as f:
        w = csv.writer(f)
        w.writerow(_COLUMNS)
        w.writerows(rows)
    return out


def register(con: duckdb.DuckDBPyConnection, *, root: Path) -> None:
    """Create the ``comments_index`` view in *con* over *root*/_index.csv."""
    csv_path = root / _INDEX
    if not csv_path.is_file():
        rebuild_index(root)
    con.execute(
        f"CREATE OR REPLACE VIEW {_VIEW} AS "
        f"SELECT * FROM read_csv('{csv_path}', header=true, auto_detect=true)"
    )


def body_path(root: Path, *, comment_id: str, docket_id: str) -> Path:
    """Filesystem path of the combined.md for one comment."""
    return root / docket_id / comment_id / "combined.md"


# ── internals ─────────────────────────────────────────────────────


def _iter_index_rows(root: Path):
    for docket_dir in sorted(root.iterdir()):
        if not docket_dir.is_dir():
            continue
        for cdir in sorted(docket_dir.iterdir()):
            md = cdir / "combined.md"
            if not md.is_file():
                continue
            try:
                fm, _body = parse_combined(md.read_text())
            except (ValueError, OSError):
                continue
            atts = fm.get("attachments") or []
            yield (
                fm.get("comment_id", ""),
                fm.get("docket_id", ""),
                fm.get("extracted_at", ""),
                fm.get("chars", 0),
                len(atts),
                sum(1 for a in atts if a.get("status") == "ok"),
                sum(1 for a in atts if a.get("status") == "ocr_needed"),
                sum(1 for a in atts if a.get("status") == "failed"),
            )

• Step 4: Run tests

Run: uv run pytest tests/rex/comments/test_view.py -v Expected: 3 PASS.

• Step 5: Commit

git add src/rex/comments/view.py tests/rex/comments/test_view.py
git commit -m "feat(comments): duckdb view over combined.md index (refs #253)"

---

Task 8: CLI — stack comments {extract, extract-ocr, stats}

Files:

• Create: src/cli/comments.py

• Modify: src/cli/__init__.py

• Test: tests/cli/test_comments.py

• Step 1: Write failing test

Create tests/cli/test_comments.py:

"""CLI: stack comments {extract, extract-ocr, stats}."""

from __future__ import annotations

from pathlib import Path

import fitz
from typer.testing import CliRunner

from cli.comments import app

runner = CliRunner()


def _seed(root: Path) -> Path:
    cdir = root / "CMS-2024-0001" / "CMS-2024-0001-0001"
    cdir.mkdir(parents=True)
    doc = fitz.open()
    doc.new_page().insert_text((50, 72), "Long body content " * 10)
    doc.save(str(cdir / "attachment_1.pdf"))
    doc.close()
    return cdir


def test_extract_writes_combined(tmp_path: Path):
    cdir = _seed(tmp_path)
    result = runner.invoke(
        app,
        [
            "extract",
            "--root",
            str(tmp_path),
            "--no-bib",  # skip bib.sqlite lookup in tests
            "--workers",
            "1",
        ],
    )
    assert result.exit_code == 0, result.output
    assert (cdir / "combined.md").is_file()
    assert "written" in result.output.lower()


def test_extract_ocr_is_stub(tmp_path: Path):
    result = runner.invoke(app, ["extract-ocr", "--root", str(tmp_path)])
    assert result.exit_code != 0
    assert "phase 2" in result.output.lower() or "not implemented" in result.output.lower()


def test_stats_after_extract(tmp_path: Path):
    _seed(tmp_path)
    runner.invoke(
        app,
        ["extract", "--root", str(tmp_path), "--no-bib", "--workers", "1"],
    )
    result = runner.invoke(app, ["stats", "--root", str(tmp_path)])
    assert result.exit_code == 0, result.output
    assert "1" in result.output  # at least one comment counted

• Step 2: Run test to verify it fails

Run: uv run pytest tests/cli/test_comments.py -v Expected: FAIL — cli.comments doesn't exist.

• Step 3: Implement the CLI

Create src/cli/comments.py:

"""stack comments — text extraction + analysis for CMS rulemaking comments.

Operates on per-comment dirs created by `bib backfill-comments`:
    .state/comments/{docket_id}/{comment_id}/{attachment_*, combined.md}

Phase 1 (this file):
    extract       — write combined.md per comment (PDF/DOCX → text)
    extract-ocr   — phase-2 stub (raises until tesseract integration lands)
    stats         — count comments by extraction status
"""

from __future__ import annotations

from pathlib import Path

import typer

app = typer.Typer(no_args_is_help=True)

_DEFAULT_ROOT = Path(".state/comments")


def _bib_lookup_factory(use_bib: bool):
    """Return a callable comment_id -> inline body. Empty if --no-bib."""
    if not use_bib:
        return lambda _cid: ""

    from bib import connect

    store = connect()
    con = store._con()  # noqa: SLF001

    def lookup(comment_id: str) -> str:
        row = con.execute(
            "SELECT abstract FROM items WHERE key = ?", (comment_id,)
        ).fetchone()
        return (row[0] if row and row[0] else "") or ""

    return lookup


@app.command()
def extract(
    root: Path = typer.Option(_DEFAULT_ROOT, "--root", help="Comments root dir."),
    docket: str = typer.Option(
        None, "--docket", help="Limit to one docket (e.g. CMS-2023-0121)."
    ),
    limit: int = typer.Option(
        0, "--limit", help="Cap dirs processed this run. 0 = no cap."
    ),
    workers: int = typer.Option(
        8, "--workers", help="Concurrent extraction threads."
    ),
    force: bool = typer.Option(
        False, "--force", help="Re-extract dirs that already have combined.md."
    ),
    use_bib: bool = typer.Option(
        True,
        "--bib/--no-bib",
        help="Look up inline body from bib.sqlite items.abstract.",
    ),
) -> None:
    """Walk comment dirs and write combined.md (PDF/DOCX → text)."""
    from rex.comments.walker import walk_and_extract

    if not root.is_dir():
        typer.echo(f"root not found: {root}", err=True)
        raise typer.Exit(1)

    lookup = _bib_lookup_factory(use_bib)
    stats = walk_and_extract(
        root,
        inline_body_lookup=lookup,
        docket=docket,
        limit=limit or None,
        workers=workers,
        force=force,
    )
    for k, v in stats.items():
        typer.echo(f"  {k}: {v}")


@app.command(name="extract-ocr")
def extract_ocr(
    root: Path = typer.Option(_DEFAULT_ROOT, "--root"),  # noqa: ARG001
) -> None:
    """OCR pass for ocr_needed attachments. Phase 2 — not yet implemented."""
    typer.echo(
        "extract-ocr is phase 2 — not yet implemented. "
        "Tracks: see #253 follow-up. "
        "Today, ocr_needed attachments stay flagged in combined.md frontmatter.",
        err=True,
    )
    raise typer.Exit(2)


@app.command()
def stats(
    root: Path = typer.Option(_DEFAULT_ROOT, "--root", help="Comments root dir."),
) -> None:
    """Rebuild the index and print per-status counts."""
    import duckdb

    from rex.comments.view import rebuild_index, register

    if not root.is_dir():
        typer.echo(f"root not found: {root}", err=True)
        raise typer.Exit(1)

    rebuild_index(root)
    con = duckdb.connect(":memory:")
    register(con, root=root)
    rows = con.execute(
        "SELECT COUNT(*) AS comments, "
        "SUM(n_attachments) AS attachments, "
        "SUM(n_ok) AS ok, "
        "SUM(n_ocr_needed) AS ocr_needed, "
        "SUM(n_failed) AS failed "
        "FROM comments_index"
    ).fetchone()
    cols = ["comments", "attachments", "ok", "ocr_needed", "failed"]
    for k, v in zip(cols, rows, strict=True):
        typer.echo(f"  {k}: {v or 0}")

• Step 4: Register the subcommand

In src/cli/__init__.py, find the block of from cli.X import app as X_app imports (alphabetical) and add:

from cli.comments import app as comments_app

Then find the matching app.add_typer(...) block and add:

app.add_typer(comments_app, name="comments", help="CMS rulemaking comment text extraction & analysis.")

• Step 5: Run tests

Run: uv run pytest tests/cli/test_comments.py -v Expected: 3 PASS.

Run: uv run stack comments --help Expected: shows extract, extract-ocr, stats subcommands without error.

• Step 6: Commit

git add src/cli/comments.py src/cli/__init__.py tests/cli/test_comments.py
git commit -m "feat(comments): stack comments CLI {extract,extract-ocr,stats} (refs #253)"

---

Task 9: Integration smoke test on a real comment dir

Files:

• Create: tests/rex/comments/test_integration.py

• Step 1: Write the integration test

Create tests/rex/comments/test_integration.py:

"""Smoke test against a real .state/comments/ dir if available.

Skipped on CI where state isn't present. The point is to exercise the
PyMuPDF code path on real CMS comment PDFs (formatting, headers,
multi-page) before running the full backfill.
"""

from __future__ import annotations

import shutil
from pathlib import Path

import pytest

from rex.comments import extract_comment, parse_combined

_LIVE = Path(".state/comments")


def _pick_real_comment_dir() -> Path | None:
    """Find one .state/comments/{docket}/{cid}/ dir with at least one PDF."""
    if not _LIVE.is_dir():
        return None
    for docket in sorted(_LIVE.iterdir()):
        if not docket.is_dir() or not docket.name.startswith("CMS-"):
            continue
        for cdir in sorted(docket.iterdir()):
            if not cdir.is_dir():
                continue
            if any(p.suffix.lower() == ".pdf" for p in cdir.iterdir()):
                return cdir
    return None


def test_real_comment_extracts(tmp_path: Path):
    src = _pick_real_comment_dir()
    if src is None:
        pytest.skip(".state/comments/ not present")

    # Copy to tmp so we don't write combined.md into the live tree.
    dst_docket = tmp_path / src.parent.name
    dst_docket.mkdir(parents=True)
    dst = dst_docket / src.name
    shutil.copytree(src, dst)
    # Strip any pre-existing combined.md so we re-run extraction.
    pre = dst / "combined.md"
    if pre.exists():
        pre.unlink()

    out = extract_comment(dst, inline_body="(no inline body in this fixture)")
    fm, body = parse_combined(out.read_text())
    assert fm["comment_id"] == src.name
    assert fm["docket_id"] == src.parent.name
    assert fm["attachments"], "expected at least one attachment"
    # At least one of the attachments should be ok or ocr_needed (not failed).
    statuses = {a["status"] for a in fm["attachments"]}
    assert statuses & {"ok", "ocr_needed"}, f"all failed: {statuses}"
    assert "## Inline comment" in body

• Step 2: Run the test

Run: uv run pytest tests/rex/comments/test_integration.py -v Expected: PASS (or SKIP if .state/comments/ is missing in this checkout).

• Step 3: Commit

git add tests/rex/comments/test_integration.py
git commit -m "test(comments): smoke test against a real .state/comments/ dir (refs #253)"

---

Task 10: End-to-end run on one docket + post issue update

Files: none (operational task, then write a short note)

• Step 1: Run extraction on the smallest docket

Pick the smallest live docket (rough size: du -sh .state/comments/CMS-* | sort -h | head -3). Run:

uv run stack comments extract --docket CMS-2017-0092 --workers 8

Expected: completes, prints written: N skipped: 0 failed: 0 (or near-zero failed). Note any errors that surface — they're our list of edge cases to look at.

• Step 2: Run stats

uv run stack comments stats

Expected: prints per-status totals. ok should dominate; ocr_needed is a small fraction; failed should be near zero.

• Step 3: Spot check one combined.md

ls .state/comments/CMS-2017-0092/$(ls .state/comments/CMS-2017-0092 | head -1)/combined.md \
  && head -40 .state/comments/CMS-2017-0092/$(ls .state/comments/CMS-2017-0092 | head -1)/combined.md

Expected: frontmatter is well-formed YAML, body has ## Inline comment + ## attachment_* sections.

• Step 4: Post a status update on Gitea issue #253

Using the same pattern as earlier in the session (docker exec into git container, POST to /api/v1/repos/homelab/stack/issues/253/comments):

# Build the comment body (substitute real numbers from `stats` output)
cat > /tmp/i253_done.json <<'EOF'
{"body": "Phase 1 of the extraction pipeline landed.\n\n**Live counts** (from `stack comments stats`):\n- comments: <N>\n- attachments: <N>\n- ok: <N> (~<%>%)\n- ocr_needed: <N> (~<%>%)\n- failed: <N>\n\nReady to feed #254/#255. OCR (`extract-ocr`) is a phase-2 stub — the `ocr_needed` count above is the work it would do."}
EOF
docker cp /tmp/i253_done.json git:/tmp/i253.json
docker exec -e T="$GITEA_TOKEN" git sh -c '
  wget -q --header="Authorization: token $T" --header="Content-Type: application/json" \
       --post-file=/tmp/i253.json -O- \
       "http://localhost:3000/api/v1/repos/homelab/stack/issues/253/comments" | head -c 200
'

Expected: response includes "html_url" for the new comment. Open the link to confirm rendering.

• Step 5: Final commit (if anything was tweaked during the live run)

If the run surfaced bugs you fixed, commit those fixes individually with fix(comments): ... messages, not bundled here.

---

Self-Review

Spec coverage:

Spec section	Covered by
Module layout (extract.py, combine.py, view.py, cli/comments.py)	Tasks 2, 3, 4, 5, 7, 8
Skip phase-2 ocr.py file	Task 8 (stub command, no module file) ✅
Filesystem markdown + DuckDB view	Tasks 5, 7
PyMuPDF for PDFs	Task 3
python-docx for DOCX	Task 4
Lazy OCR (stub now, phase-2 later)	Task 8
Filesystem-only resume tracking (combined.md exists ⇒ done)	Tasks 5, 6
New stack comments CLI namespace	Task 8
YAML frontmatter format	Task 5
body_path() helper for downstream	Task 7
Per-attachment exception handling (mark failed, continue)	Tasks 3, 4
Idempotency tested	Task 5
Synthetic + real fixtures	Tasks 3-5 (synthetic), Task 9 (real)
Smoke run + issue update	Task 10

No gaps.

Placeholder check: All steps have concrete code. No "TBD" or "handle errors" steps.

Type consistency:

• ExtractResult(text, status, chars) — used identically in Tasks 3, 4, 5, 7
• extract_comment(comment_dir, *, inline_body, force) -> Path — Tasks 5, 6
• walk_and_extract(...) -> dict[str, int] keys {written, skipped, failed} — Tasks 6, 8
• register(con, *, root) and rebuild_index(root) — Tasks 7, 8
• body_path(root, *, comment_id, docket_id) -> Path — Task 7

All consistent.

One issue caught and fixed inline: Task 6's first walker draft used a generator that pre-skipped existing-combined.md dirs, which broke the test expecting skipped: 1 to be reported. Replaced with _collect_dirs that returns (todo, skipped_count) and updated the call site.