0f96fdba12
All checks were successful
CI / lint (push) Successful in 35s
CI / notebooks-smoke (push) Successful in 1m28s
Deploy / notebooks (push) Has been skipped
Deploy / zotero (push) Has been skipped
Deploy / docs (push) Has been skipped
Deploy / api (push) Has been skipped
Deploy / llm (push) Has been skipped
Deploy / mc (push) Has been skipped
Infra CI / notebooks (push) Successful in 1m9s
Infra CI / zotero (push) Successful in 22s
Infra CI / docs (push) Successful in 1m13s
Infra CI / api (push) Successful in 1m14s
Infra CI / llm (push) Successful in 46s
Infra CI / mc (push) Successful in 14s
Deploy / report (push) Successful in 12s
CI / test (push) Successful in 11m49s
148 lines
12 KiB
Markdown
148 lines
12 KiB
Markdown
# P40: FR Jump Links Implementation Plan
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||
|
||
**Goal:** A context-aware `bib.frlink` utility that grabs each rule's federalregister.gov paragraph/page anchor map into bib and places deep jump links for any FR citation — notebooks, `:pincite:`, Zotero child links, CLI.
|
||
|
||
**Architecture:** One new module `src/bib/frlink.py` (grabber + resolver + placement) over three new `bib/schema.sql` tables (`fr_anchors`, `fr_anchor_docs`, `fr_links`), with thin hooks in `bib/pincite.py` (FR locator grammar + `jump_url`), `bib/sync.py` (`_sync_fr_links` → Zotero linkMode=3 children), and `src/cli/bib.py` (`fr-grab`, `fr-jump`). Rollout: backfill all FR-linked rules, then live-link the CY2027 notebook's citations and sync the placed links to Zotero.
|
||
|
||
**Tech Stack:** httpx (FR API + body HTML), sqlite3 via `bib.Store`, typer CLI, pytest; notebook gate via `nb_integration.py` in the prod notebooks container.
|
||
|
||
**Spec:** `docs/superpowers/specs/2026-08-17-fr-jump-links-design.md` (committed; design user-approved in-session). Tracker (real links):
|
||
- Milestone [P40](https://git.fhirworx.io/homelab/stack/milestone/41)
|
||
- [#634 — schema + grabber + fr-grab/backfill](https://git.fhirworx.io/homelab/stack/issues/634)
|
||
- [#635 — context-aware resolver + transmutations](https://git.fhirworx.io/homelab/stack/issues/635)
|
||
- [#636 — :pincite: FR locators + jump_url](https://git.fhirworx.io/homelab/stack/issues/636)
|
||
- [#637 — place() + Zotero child link attachments](https://git.fhirworx.io/homelab/stack/issues/637)
|
||
- [#638 — rollout: backfill + notebook + Zotero](https://git.fhirworx.io/homelab/stack/issues/638)
|
||
|
||
## Global Constraints
|
||
|
||
- Conventional commits, `(closes #N)`/`(refs #N)`; **no Co-Authored-By trailer**.
|
||
- TDD per task; `uv run pytest tests/bib/test_frlink.py -q` (new file) plus touched existing test files must be green before each commit.
|
||
- `parse_anchors` and all resolvers are **pure functions over stored data** — network only in `grab()`/backfill.
|
||
- Backfill is polite to the FR API: sequential fetches, small sleep, skip already-grabbed (no `--force` in the rollout run).
|
||
- Zotero writes only through the existing sync path (container held); never bulk-place anchors.
|
||
- Verified anchors (CY2027, doc 2026-14327): 5,079 `p-N`, 716 `page-N`, `data-page` on every paragraph; body HTML cached at the P40 scratchpad (`fr_body.html`) for offline smoke tests.
|
||
|
||
---
|
||
|
||
### Task 1: Schema + `parse_anchors` + `grab` + `fr-grab` CLI ([#634](https://git.fhirworx.io/homelab/stack/issues/634))
|
||
|
||
**Files:**
|
||
- Modify: `src/bib/schema.sql` (three tables + indexes), `src/cli/bib.py` (fr-grab)
|
||
- Create: `src/bib/frlink.py`
|
||
- Test: `tests/bib/test_frlink.py` (new)
|
||
|
||
**Interfaces:**
|
||
- Produces:
|
||
- `Anchor` dataclass: `p_id: int, page: int, ordinal: int, text: str`
|
||
- `parse_anchors(html: str) -> list[Anchor]` — pure; walks `id="p-N"` with `data-page`, assigns 1-based ordinal per page, tag-strips + `html.unescape`s + whitespace-collapses text
|
||
- `grab(store, item_key, *, force=False, fetch=None) -> dict` — resolves `document_number` (item `extra_json`, fallback URL parse), fetches the doc API for `body_html_url`/`start_page`/`end_page`/`volume` unless already in `fr_anchor_docs`, replaces the item's `fr_anchors` partition + `fr_anchor_docs` row; returns counts. `fetch` injectable for tests.
|
||
- `backfill(store, *, force=False, sleep=1.0) -> list[dict]` over `item_type='rule'` items with federalregister.gov URLs.
|
||
- Schema: as specced — `fr_anchors` UNIQUE(item_key, p_id) + idx (item_key, page); `fr_anchor_docs` PK item_key; `fr_links` UNIQUE(item_key, url).
|
||
|
||
- [x] **Step 1 (failing tests):** `tests/bib/test_frlink.py` with a synthetic body-HTML fixture:
|
||
|
||
```python
|
||
FIXTURE_HTML = """
|
||
<html><body>
|
||
<div id="page-100"></div>
|
||
<p id="p-1" data-page="100">First para on 100 with § 414.1425 text.</p>
|
||
<p id="p-2" data-page="100">Second para <em>with markup</em> on 100.</p>
|
||
<div id="page-101"></div>
|
||
<p id="p-3" data-page="101">Only para on 101.</p>
|
||
</body></html>
|
||
"""
|
||
|
||
def test_parse_anchors_pages_ordinals_text():
|
||
a = frlink.parse_anchors(FIXTURE_HTML)
|
||
assert [(x.p_id, x.page, x.ordinal) for x in a] == [(1, 100, 1), (2, 100, 2), (3, 101, 1)]
|
||
assert "§ 414.1425" in a[0].text.replace(" ", " ") or "§" in a[0].text
|
||
assert a[1].text == "Second para with markup on 100."
|
||
|
||
def test_grab_persists_and_replaces(tmp_path):
|
||
# Store with a rule item carrying extra_json document_number/volume;
|
||
# grab(fetch=lambda ...) injecting FIXTURE_HTML + fake doc meta;
|
||
# assert fr_anchors rows == 3, fr_anchor_docs row present;
|
||
# re-grab with a 2-anchor fixture → rows == 2 (partition replaced).
|
||
```
|
||
(write both fully, using `Store(":memory:")` + a `Rule` item with `extra_json` fields matching the real shape: `{"document_number": "2026-14327", "fr_volume": "91", "fr_page": "43842"}`)
|
||
|
||
- [x] **Step 2:** Run: `uv run pytest tests/bib/test_frlink.py -q` → FAIL (module missing).
|
||
- [x] **Step 3:** Implement `src/bib/frlink.py` (`_P_RE = re.compile(r'id="p-(\d+)"[^>]*data-page="(\d+)"')`; text = substring to the next `</p>`, tag-strip + unescape + `" ".join(split())`), schema additions, and the `fr-grab` typer command calling `grab`/`backfill`.
|
||
- [x] **Step 4:** `uv run pytest tests/bib/test_frlink.py tests/bib/test_store.py -q` → PASS (store tests prove schema addition breaks nothing).
|
||
- [x] **Step 5:** Offline real-file smoke: run `parse_anchors` on the cached CY2027 `fr_body.html` → assert 5079 paragraphs, 716 distinct pages (one-off command, not a committed test — the 4.2 MB file stays out of the repo).
|
||
- [x] **Step 6:** Commit: `feat(bib): fr_anchors schema + FR body-HTML grabber + fr-grab CLI (closes #634)`
|
||
|
||
### Task 2: Resolver + transmutations + `md_link` ([#635](https://git.fhirworx.io/homelab/stack/issues/635))
|
||
|
||
**Files:**
|
||
- Modify: `src/bib/frlink.py`
|
||
- Test: `tests/bib/test_frlink.py`
|
||
|
||
**Interfaces:**
|
||
- Produces:
|
||
- `JumpLink` dataclass: `url, item_key, page, p_id (None for page links), ordinal (None), snippet ("")`
|
||
- `resolve(ref, *, store, item_key="") -> JumpLink` — ref grammar precedence: raw anchor `^p-\d+$` → FR cite `^(\d+)\s+FR\s+(\d+)(?:[,\s]+(?:¶|para\.?|p\.)\s*(\d+))?$` (page, optional ordinal) → otherwise quote (≥15 chars after normalization; else ValueError). FR cites resolve the rule by `fr_volume` + `start_page ≤ page ≤ end_page` over `fr_anchor_docs`; 0 or >1 candidates → ValueError naming them (unless `item_key` given).
|
||
- `page_of(store, item_key, p_id) -> int`, `paragraphs_of(ref, *, store) -> list[Anchor]`
|
||
- `md_link(ref, *, store, item_key="", text="") -> str`
|
||
- URL forms: `{html_url}#page-{page}` / `{html_url}#p-{p_id}`
|
||
|
||
- [x] **Step 1 (failing tests):** extend the fixture store helper to populate `fr_anchor_docs` (html_url `https://example.test/doc`, volume 91, pages 100–101) + the 3 fixture anchors; add tests: page ref, page+ordinal (`"91 FR 100 ¶2"` → `#p-2`), raw `p-3`, quote (`"Only para on 101"` → `#p-3`), quote-multi-hit raises, unknown page raises, `page_of`/`paragraphs_of` round-trip, `md_link` default text equals the ref.
|
||
- [x] **Step 2:** Run → new tests FAIL.
|
||
- [x] **Step 3:** Implement; quote normalization shared with `parse_anchors` storage form.
|
||
- [x] **Step 4:** `uv run pytest tests/bib/test_frlink.py -q` → PASS.
|
||
- [x] **Step 5:** Commit: `feat(bib): context-aware FR jump resolver — page/ordinal/anchor/quote + transmutations (closes #635)`
|
||
|
||
### Task 3: `:pincite:` FR locators + `jump_url` ([#636](https://git.fhirworx.io/homelab/stack/issues/636))
|
||
|
||
**Files:**
|
||
- Modify: `src/bib/pincite.py` (`_classify_locator` + `Pincite.jump_url` + graph edge URL)
|
||
- Test: `tests/bib/test_pincite.py` (extend)
|
||
|
||
**Interfaces:**
|
||
- Consumes: `frlink.resolve` (lazy import inside `jump_url` to avoid import cycles).
|
||
- Produces: locator_type `"fr_page"` (`91 FR 44218`), `"fr_para"` (`91 FR 44218 ¶3`), `"fr_anchor"` (`p-3600`); `Pincite.jump_url(store) -> str` ("" when unresolvable — never raises); graph edges carry `metadata["jump_url"]` when non-empty.
|
||
|
||
- [x] **Step 1 (failing tests):** classification of the three forms (existing forms `p.14`/`§3.2` keep their current types — assert unchanged); `jump_url` resolves against the Task-2 fixture store; absent anchor map → `""`.
|
||
- [x] **Step 2:** Run → FAIL. **Step 3:** Implement. **Step 4:** `uv run pytest tests/bib/test_pincite.py tests/bib/test_frlink.py -q` → PASS.
|
||
- [x] **Step 5:** Commit: `feat(bib): :pincite: FR locators resolve to jump URLs (closes #636)`
|
||
|
||
### Task 4: `place()` + Zotero linkMode=3 children + `fr-jump` CLI ([#637](https://git.fhirworx.io/homelab/stack/issues/637))
|
||
|
||
**Files:**
|
||
- Modify: `src/bib/frlink.py` (`place`), `src/bib/sync.py` (`_sync_fr_links`, called from the push loop next to `_sync_attachments`), `src/cli/bib.py` (`fr-jump`)
|
||
- Test: `tests/bib/test_frlink.py`, `tests/bib/test_sync.py` (extend, `_make_zotero_db` fixture)
|
||
|
||
**Interfaces:**
|
||
- Produces: `place(store, ref, *, item_key="", label="") -> JumpLink` (upsert `fr_links`; default label = `"¶ p-N — {ref}"` or the page ref); `_sync_fr_links(db, store, bib_item, zot_parent_id) -> int` creating child attachments via `db.add_attachment(parent, link_mode=3, content_type="text/html")` + `db.set_fields(att_id, {"url": url, "title": label})`, idempotent by existing children's `url` itemData; `fr-jump REF [--key K] [--md] [--place] [--label L]` CLI.
|
||
|
||
- [x] **Step 1 (failing tests):** `place` writes one `fr_links` row, double-place no-ops; sync test: bib item + placed link → push creates exactly one linkMode=3 child with url+title itemData; second push creates none.
|
||
- [x] **Step 2:** Run → FAIL. **Step 3:** Implement. **Step 4:** `uv run pytest tests/bib/test_frlink.py tests/bib/test_sync.py -q` → PASS.
|
||
- [x] **Step 5:** Commit: `feat(bib): place FR jump links as Zotero child link attachments + fr-jump CLI (closes #637)`
|
||
|
||
### Task 5: Rollout ([#638](https://git.fhirworx.io/homelab/stack/issues/638))
|
||
|
||
**Files:**
|
||
- Modify: `notebooks/cy2027_pfs_proposed_rule.py` (live-link FR cites). Data: `bib.sqlite`, `zotero.sqlite`.
|
||
|
||
- [x] **Step 1:** `uv run stack bib fr-grab --all` — backfill; report per-rule paragraph/page counts; expect ~30 rules, CY2027 = 5079/716. Failures on odd old rules: log + continue, note on the issue.
|
||
- [x] **Step 2:** Notebook: in the APM narrative cell and Section-5 sources, replace the dead `91 FR 44218-44219, 44286` style cites with `frlink.md_link(...)` calls (import in the notebook's setup cell; reads bib.sqlite via `conf.connect.bib()` — available in the notebooks container through the mounted `data/`). In-cell fallback: if the anchor map is missing, degrade to plain text (notebook must not break offline).
|
||
- [x] **Step 3:** Gate: single-notebook `nb_integration.py` run in the prod container → `pass=1`.
|
||
- [x] **Step 4:** Place the cited paragraphs (`91 FR 44218` QP-alignment ¶, the 414.1450(b)(1) revival cite) via `stack bib fr-jump ... --place`, then `uv run stack bib sync-zotero --tag sup:2027_PFS_NPRM`; verify linkMode=3 children on `5ITGVDJV` in zotero.sqlite.
|
||
- [x] **Step 5:** Commit notebook: `feat(notebooks): live FR jump links in CY2027 NPRM citations (closes #638)`
|
||
|
||
### Task 6: Close the loop
|
||
|
||
- [x] **Step 1:** Push; CI green on HEAD (poll with `skipped` treated as terminal — see gitea-tracker-ops memory).
|
||
- [x] **Step 2:** Issues #634–#638 closed (auto-close footers) with verification comments where data-only; close [milestone P40](https://git.fhirworx.io/homelab/stack/milestone/41).
|
||
- [x] **Step 3:** Append P40 build outcomes to the spec; commit docs.
|
||
- [x] **Step 4:** Memory: new `fr_jump_links.md` (anchor-map mechanics, `data-page` fact, resolver grammar, don't-bulk-place rule).
|
||
|
||
## Self-Review
|
||
|
||
- Coverage: #634→T1, #635→T2, #636→T3, #637→T4, #638→T5, closure→T6; spec sections all mapped (grammar, transmutations, placement, CLI, rollout).
|
||
- Placeholders: none — regexes, fixtures, URL forms, and CLI shapes are spelled out; Step-1 test stubs state exact assertions.
|
||
- Type consistency: `Anchor(p_id, page, ordinal, text)` used by T1/T2/T3 alike; `JumpLink` fields consistent across resolve/place/md_link; `fr_links` UNIQUE(item_key,url) matches place-idempotency tests.
|