homelab/stack
1
0
Fork 0
Code Issues 20 Pull Requests Actions Packages Projects 1 Releases Wiki Activity
Files
81797a54b532886cd5c7fc29a2b42f2a038ae962
stack/docs/superpowers/plans/2026-08-17-fr-jump-links-p40.md
kert 0f96fdba12
All checks were successful
CI / lint (push) Successful in 35s
Details
CI / notebooks-smoke (push) Successful in 1m28s
Details
Deploy / notebooks (push) Has been skipped
Details
Deploy / zotero (push) Has been skipped
Details
Deploy / docs (push) Has been skipped
Details
Deploy / api (push) Has been skipped
Details
Deploy / llm (push) Has been skipped
Details
Deploy / mc (push) Has been skipped
Details
Infra CI / notebooks (push) Successful in 1m9s
Details
Infra CI / zotero (push) Successful in 22s
Details
Infra CI / docs (push) Successful in 1m13s
Details
Infra CI / api (push) Successful in 1m14s
Details
Infra CI / llm (push) Successful in 46s
Details
Infra CI / mc (push) Successful in 14s
Details
Deploy / report (push) Successful in 12s
Details
CI / test (push) Successful in 11m49s
Details
docs(spec): P40 build outcomes; plan checked off (refs #634-#638)
2026-08-17 14:45:36 -04:00

12 KiB
Raw Blame History

P40: FR Jump Links Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: A context-aware bib.frlink utility that grabs each rule's federalregister.gov paragraph/page anchor map into bib and places deep jump links for any FR citation — notebooks, :pincite:, Zotero child links, CLI.

Architecture: One new module src/bib/frlink.py (grabber + resolver + placement) over three new bib/schema.sql tables (fr_anchors, fr_anchor_docs, fr_links), with thin hooks in bib/pincite.py (FR locator grammar + jump_url), bib/sync.py (_sync_fr_links → Zotero linkMode=3 children), and src/cli/bib.py (fr-grab, fr-jump). Rollout: backfill all FR-linked rules, then live-link the CY2027 notebook's citations and sync the placed links to Zotero.

Tech Stack: httpx (FR API + body HTML), sqlite3 via bib.Store, typer CLI, pytest; notebook gate via nb_integration.py in the prod notebooks container.

Spec: docs/superpowers/specs/2026-08-17-fr-jump-links-design.md (committed; design user-approved in-session). Tracker (real links):

Global Constraints

  • Conventional commits, (closes #N)/(refs #N); no Co-Authored-By trailer.
  • TDD per task; uv run pytest tests/bib/test_frlink.py -q (new file) plus touched existing test files must be green before each commit.
  • parse_anchors and all resolvers are pure functions over stored data — network only in grab()/backfill.
  • Backfill is polite to the FR API: sequential fetches, small sleep, skip already-grabbed (no --force in the rollout run).
  • Zotero writes only through the existing sync path (container held); never bulk-place anchors.
  • Verified anchors (CY2027, doc 2026-14327): 5,079 p-N, 716 page-N, data-page on every paragraph; body HTML cached at the P40 scratchpad (fr_body.html) for offline smoke tests.

Task 1: Schema + parse_anchors + grab + fr-grab CLI (#634)

Files:

  • Modify: src/bib/schema.sql (three tables + indexes), src/cli/bib.py (fr-grab)
  • Create: src/bib/frlink.py
  • Test: tests/bib/test_frlink.py (new)

Interfaces:

  • Produces:

    • Anchor dataclass: p_id: int, page: int, ordinal: int, text: str
    • parse_anchors(html: str) -> list[Anchor] — pure; walks id="p-N" with data-page, assigns 1-based ordinal per page, tag-strips + html.unescapes + whitespace-collapses text
    • grab(store, item_key, *, force=False, fetch=None) -> dict — resolves document_number (item extra_json, fallback URL parse), fetches the doc API for body_html_url/start_page/end_page/volume unless already in fr_anchor_docs, replaces the item's fr_anchors partition + fr_anchor_docs row; returns counts. fetch injectable for tests.
    • backfill(store, *, force=False, sleep=1.0) -> list[dict] over item_type='rule' items with federalregister.gov URLs.
    • Schema: as specced — fr_anchors UNIQUE(item_key, p_id) + idx (item_key, page); fr_anchor_docs PK item_key; fr_links UNIQUE(item_key, url).

  • Step 1 (failing tests): tests/bib/test_frlink.py with a synthetic body-HTML fixture:

FIXTURE_HTML = """
<html><body>
<div id="page-100"></div>
<p id="p-1" data-page="100">First para on 100 with &sect;&thinsp;414.1425 text.</p>
<p id="p-2" data-page="100">Second para <em>with markup</em> on 100.</p>
<div id="page-101"></div>
<p id="p-3" data-page="101">Only para on 101.</p>
</body></html>
"""

def test_parse_anchors_pages_ordinals_text():
    a = frlink.parse_anchors(FIXTURE_HTML)
    assert [(x.p_id, x.page, x.ordinal) for x in a] == [(1, 100, 1), (2, 100, 2), (3, 101, 1)]
    assert "§ 414.1425" in a[0].text.replace(" ", " ") or "§" in a[0].text
    assert a[1].text == "Second para with markup on 100."

def test_grab_persists_and_replaces(tmp_path):
    # Store with a rule item carrying extra_json document_number/volume;
    # grab(fetch=lambda ...) injecting FIXTURE_HTML + fake doc meta;
    # assert fr_anchors rows == 3, fr_anchor_docs row present;
    # re-grab with a 2-anchor fixture → rows == 2 (partition replaced).

(write both fully, using Store(":memory:") + a Rule item with extra_json fields matching the real shape: {"document_number": "2026-14327", "fr_volume": "91", "fr_page": "43842"})

  • Step 2: Run: uv run pytest tests/bib/test_frlink.py -q → FAIL (module missing).
  • Step 3: Implement src/bib/frlink.py (_P_RE = re.compile(r'id="p-(\d+)"[^>]*data-page="(\d+)"'); text = substring to the next </p>, tag-strip + unescape + " ".join(split())), schema additions, and the fr-grab typer command calling grab/backfill.
  • Step 4: uv run pytest tests/bib/test_frlink.py tests/bib/test_store.py -q → PASS (store tests prove schema addition breaks nothing).
  • Step 5: Offline real-file smoke: run parse_anchors on the cached CY2027 fr_body.html → assert 5079 paragraphs, 716 distinct pages (one-off command, not a committed test — the 4.2 MB file stays out of the repo).
  • Step 6: Commit: feat(bib): fr_anchors schema + FR body-HTML grabber + fr-grab CLI (closes #634)

Task 2: Resolver + transmutations + md_link (#635)

Files:

  • Modify: src/bib/frlink.py
  • Test: tests/bib/test_frlink.py

Interfaces:

  • Produces:

    • JumpLink dataclass: url, item_key, page, p_id (None for page links), ordinal (None), snippet ("")
    • resolve(ref, *, store, item_key="") -> JumpLink — ref grammar precedence: raw anchor ^p-\d+$ → FR cite ^(\d+)\s+FR\s+(\d+)(?:[,\s]+(?:¶|para\.?|p\.)\s*(\d+))?$ (page, optional ordinal) → otherwise quote (≥15 chars after normalization; else ValueError). FR cites resolve the rule by fr_volume + start_page ≤ page ≤ end_page over fr_anchor_docs; 0 or >1 candidates → ValueError naming them (unless item_key given).
    • page_of(store, item_key, p_id) -> int, paragraphs_of(ref, *, store) -> list[Anchor]
    • md_link(ref, *, store, item_key="", text="") -> str
    • URL forms: {html_url}#page-{page} / {html_url}#p-{p_id}

  • Step 1 (failing tests): extend the fixture store helper to populate fr_anchor_docs (html_url https://example.test/doc, volume 91, pages 100–101) + the 3 fixture anchors; add tests: page ref, page+ordinal ("91 FR 100 ¶2" → #p-2), raw p-3, quote ("Only para on 101" → #p-3), quote-multi-hit raises, unknown page raises, page_of/paragraphs_of round-trip, md_link default text equals the ref.

  • Step 2: Run → new tests FAIL.

  • Step 3: Implement; quote normalization shared with parse_anchors storage form.

  • Step 4: uv run pytest tests/bib/test_frlink.py -q → PASS.

  • Step 5: Commit: feat(bib): context-aware FR jump resolver — page/ordinal/anchor/quote + transmutations (closes #635)

Task 3: :pincite: FR locators + jump_url (#636)

Files:

  • Modify: src/bib/pincite.py (_classify_locator + Pincite.jump_url + graph edge URL)
  • Test: tests/bib/test_pincite.py (extend)

Interfaces:

  • Consumes: frlink.resolve (lazy import inside jump_url to avoid import cycles).

  • Produces: locator_type "fr_page" (91 FR 44218), "fr_para" (91 FR 44218 ¶3), "fr_anchor" (p-3600); Pincite.jump_url(store) -> str ("" when unresolvable — never raises); graph edges carry metadata["jump_url"] when non-empty.

  • Step 1 (failing tests): classification of the three forms (existing forms p.14/§3.2 keep their current types — assert unchanged); jump_url resolves against the Task-2 fixture store; absent anchor map → "".

  • Step 2: Run → FAIL. Step 3: Implement. Step 4: uv run pytest tests/bib/test_pincite.py tests/bib/test_frlink.py -q → PASS.

  • Step 5: Commit: feat(bib): :pincite: FR locators resolve to jump URLs (closes #636)

Task 4: place() + Zotero linkMode=3 children + fr-jump CLI (#637)

Files:

  • Modify: src/bib/frlink.py (place), src/bib/sync.py (_sync_fr_links, called from the push loop next to _sync_attachments), src/cli/bib.py (fr-jump)
  • Test: tests/bib/test_frlink.py, tests/bib/test_sync.py (extend, _make_zotero_db fixture)

Interfaces:

  • Produces: place(store, ref, *, item_key="", label="") -> JumpLink (upsert fr_links; default label = "¶ p-N — {ref}" or the page ref); _sync_fr_links(db, store, bib_item, zot_parent_id) -> int creating child attachments via db.add_attachment(parent, link_mode=3, content_type="text/html") + db.set_fields(att_id, {"url": url, "title": label}), idempotent by existing children's url itemData; fr-jump REF [--key K] [--md] [--place] [--label L] CLI.

  • Step 1 (failing tests): place writes one fr_links row, double-place no-ops; sync test: bib item + placed link → push creates exactly one linkMode=3 child with url+title itemData; second push creates none.

  • Step 2: Run → FAIL. Step 3: Implement. Step 4: uv run pytest tests/bib/test_frlink.py tests/bib/test_sync.py -q → PASS.

  • Step 5: Commit: feat(bib): place FR jump links as Zotero child link attachments + fr-jump CLI (closes #637)

Task 5: Rollout (#638)

Files:

  • Modify: notebooks/cy2027_pfs_proposed_rule.py (live-link FR cites). Data: bib.sqlite, zotero.sqlite.

  • Step 1: uv run stack bib fr-grab --all — backfill; report per-rule paragraph/page counts; expect ~30 rules, CY2027 = 5079/716. Failures on odd old rules: log + continue, note on the issue.

  • Step 2: Notebook: in the APM narrative cell and Section-5 sources, replace the dead 91 FR 44218-44219, 44286 style cites with frlink.md_link(...) calls (import in the notebook's setup cell; reads bib.sqlite via conf.connect.bib() — available in the notebooks container through the mounted data/). In-cell fallback: if the anchor map is missing, degrade to plain text (notebook must not break offline).

  • Step 3: Gate: single-notebook nb_integration.py run in the prod container → pass=1.

  • Step 4: Place the cited paragraphs (91 FR 44218 QP-alignment ¶, the 414.1450(b)(1) revival cite) via stack bib fr-jump ... --place, then uv run stack bib sync-zotero --tag sup:2027_PFS_NPRM; verify linkMode=3 children on 5ITGVDJV in zotero.sqlite.

  • Step 5: Commit notebook: feat(notebooks): live FR jump links in CY2027 NPRM citations (closes #638)

Task 6: Close the loop

  • Step 1: Push; CI green on HEAD (poll with skipped treated as terminal — see gitea-tracker-ops memory).
  • Step 2: Issues #634–#638 closed (auto-close footers) with verification comments where data-only; close milestone P40.
  • Step 3: Append P40 build outcomes to the spec; commit docs.
  • Step 4: Memory: new fr_jump_links.md (anchor-map mechanics, data-page fact, resolver grammar, don't-bulk-place rule).

Self-Review

  • Coverage: #634→T1, #635→T2, #636→T3, #637→T4, #638→T5, closure→T6; spec sections all mapped (grammar, transmutations, placement, CLI, rollout).
  • Placeholders: none — regexes, fixtures, URL forms, and CLI shapes are spelled out; Step-1 test stubs state exact assertions.
  • Type consistency: Anchor(p_id, page, ordinal, text) used by T1/T2/T3 alike; JumpLink fields consistent across resolve/place/md_link; fr_links UNIQUE(item_key,url) matches place-idempotency tests.
Reference in New Issue View Git Blame Copy Permalink