docs(spec): P40 build outcomes; plan checked off (refs #634-#638)
All checks were successful
CI / lint (push) Successful in 35s
CI / notebooks-smoke (push) Successful in 1m28s
Deploy / notebooks (push) Has been skipped
Deploy / zotero (push) Has been skipped
Deploy / docs (push) Has been skipped
Deploy / api (push) Has been skipped
Deploy / llm (push) Has been skipped
Deploy / mc (push) Has been skipped
Infra CI / notebooks (push) Successful in 1m9s
Infra CI / zotero (push) Successful in 22s
Infra CI / docs (push) Successful in 1m13s
Infra CI / api (push) Successful in 1m14s
Infra CI / llm (push) Successful in 46s
Infra CI / mc (push) Successful in 14s
Deploy / report (push) Successful in 12s
CI / test (push) Successful in 11m49s

This commit is contained in:
kert
2026-08-17 14:45:36 -04:00
parent d7adf59af8
commit 0f96fdba12
2 changed files with 61 additions and 26 deletions

View File

@@ -42,7 +42,7 @@
- `backfill(store, *, force=False, sleep=1.0) -> list[dict]` over `item_type='rule'` items with federalregister.gov URLs. - `backfill(store, *, force=False, sleep=1.0) -> list[dict]` over `item_type='rule'` items with federalregister.gov URLs.
- Schema: as specced — `fr_anchors` UNIQUE(item_key, p_id) + idx (item_key, page); `fr_anchor_docs` PK item_key; `fr_links` UNIQUE(item_key, url). - Schema: as specced — `fr_anchors` UNIQUE(item_key, p_id) + idx (item_key, page); `fr_anchor_docs` PK item_key; `fr_links` UNIQUE(item_key, url).
- [ ] **Step 1 (failing tests):** `tests/bib/test_frlink.py` with a synthetic body-HTML fixture: - [x] **Step 1 (failing tests):** `tests/bib/test_frlink.py` with a synthetic body-HTML fixture:
```python ```python
FIXTURE_HTML = """ FIXTURE_HTML = """
@@ -69,11 +69,11 @@ def test_grab_persists_and_replaces(tmp_path):
``` ```
(write both fully, using `Store(":memory:")` + a `Rule` item with `extra_json` fields matching the real shape: `{"document_number": "2026-14327", "fr_volume": "91", "fr_page": "43842"}`) (write both fully, using `Store(":memory:")` + a `Rule` item with `extra_json` fields matching the real shape: `{"document_number": "2026-14327", "fr_volume": "91", "fr_page": "43842"}`)
- [ ] **Step 2:** Run: `uv run pytest tests/bib/test_frlink.py -q` → FAIL (module missing). - [x] **Step 2:** Run: `uv run pytest tests/bib/test_frlink.py -q` → FAIL (module missing).
- [ ] **Step 3:** Implement `src/bib/frlink.py` (`_P_RE = re.compile(r'id="p-(\d+)"[^>]*data-page="(\d+)"')`; text = substring to the next `</p>`, tag-strip + unescape + `" ".join(split())`), schema additions, and the `fr-grab` typer command calling `grab`/`backfill`. - [x] **Step 3:** Implement `src/bib/frlink.py` (`_P_RE = re.compile(r'id="p-(\d+)"[^>]*data-page="(\d+)"')`; text = substring to the next `</p>`, tag-strip + unescape + `" ".join(split())`), schema additions, and the `fr-grab` typer command calling `grab`/`backfill`.
- [ ] **Step 4:** `uv run pytest tests/bib/test_frlink.py tests/bib/test_store.py -q` → PASS (store tests prove schema addition breaks nothing). - [x] **Step 4:** `uv run pytest tests/bib/test_frlink.py tests/bib/test_store.py -q` → PASS (store tests prove schema addition breaks nothing).
- [ ] **Step 5:** Offline real-file smoke: run `parse_anchors` on the cached CY2027 `fr_body.html` → assert 5079 paragraphs, 716 distinct pages (one-off command, not a committed test — the 4.2 MB file stays out of the repo). - [x] **Step 5:** Offline real-file smoke: run `parse_anchors` on the cached CY2027 `fr_body.html` → assert 5079 paragraphs, 716 distinct pages (one-off command, not a committed test — the 4.2 MB file stays out of the repo).
- [ ] **Step 6:** Commit: `feat(bib): fr_anchors schema + FR body-HTML grabber + fr-grab CLI (closes #634)` - [x] **Step 6:** Commit: `feat(bib): fr_anchors schema + FR body-HTML grabber + fr-grab CLI (closes #634)`
### Task 2: Resolver + transmutations + `md_link` ([#635](https://git.fhirworx.io/homelab/stack/issues/635)) ### Task 2: Resolver + transmutations + `md_link` ([#635](https://git.fhirworx.io/homelab/stack/issues/635))
@@ -89,11 +89,11 @@ def test_grab_persists_and_replaces(tmp_path):
- `md_link(ref, *, store, item_key="", text="") -> str` - `md_link(ref, *, store, item_key="", text="") -> str`
- URL forms: `{html_url}#page-{page}` / `{html_url}#p-{p_id}` - URL forms: `{html_url}#page-{page}` / `{html_url}#p-{p_id}`
- [ ] **Step 1 (failing tests):** extend the fixture store helper to populate `fr_anchor_docs` (html_url `https://example.test/doc`, volume 91, pages 100–101) + the 3 fixture anchors; add tests: page ref, page+ordinal (`"91 FR 100 ¶2"` → `#p-2`), raw `p-3`, quote (`"Only para on 101"` → `#p-3`), quote-multi-hit raises, unknown page raises, `page_of`/`paragraphs_of` round-trip, `md_link` default text equals the ref. - [x] **Step 1 (failing tests):** extend the fixture store helper to populate `fr_anchor_docs` (html_url `https://example.test/doc`, volume 91, pages 100–101) + the 3 fixture anchors; add tests: page ref, page+ordinal (`"91 FR 100 ¶2"` → `#p-2`), raw `p-3`, quote (`"Only para on 101"` → `#p-3`), quote-multi-hit raises, unknown page raises, `page_of`/`paragraphs_of` round-trip, `md_link` default text equals the ref.
- [ ] **Step 2:** Run → new tests FAIL. - [x] **Step 2:** Run → new tests FAIL.
- [ ] **Step 3:** Implement; quote normalization shared with `parse_anchors` storage form. - [x] **Step 3:** Implement; quote normalization shared with `parse_anchors` storage form.
- [ ] **Step 4:** `uv run pytest tests/bib/test_frlink.py -q` → PASS. - [x] **Step 4:** `uv run pytest tests/bib/test_frlink.py -q` → PASS.
- [ ] **Step 5:** Commit: `feat(bib): context-aware FR jump resolver — page/ordinal/anchor/quote + transmutations (closes #635)` - [x] **Step 5:** Commit: `feat(bib): context-aware FR jump resolver — page/ordinal/anchor/quote + transmutations (closes #635)`
### Task 3: `:pincite:` FR locators + `jump_url` ([#636](https://git.fhirworx.io/homelab/stack/issues/636)) ### Task 3: `:pincite:` FR locators + `jump_url` ([#636](https://git.fhirworx.io/homelab/stack/issues/636))
@@ -105,9 +105,9 @@ def test_grab_persists_and_replaces(tmp_path):
- Consumes: `frlink.resolve` (lazy import inside `jump_url` to avoid import cycles). - Consumes: `frlink.resolve` (lazy import inside `jump_url` to avoid import cycles).
- Produces: locator_type `"fr_page"` (`91 FR 44218`), `"fr_para"` (`91 FR 44218 ¶3`), `"fr_anchor"` (`p-3600`); `Pincite.jump_url(store) -> str` ("" when unresolvable — never raises); graph edges carry `metadata["jump_url"]` when non-empty. - Produces: locator_type `"fr_page"` (`91 FR 44218`), `"fr_para"` (`91 FR 44218 ¶3`), `"fr_anchor"` (`p-3600`); `Pincite.jump_url(store) -> str` ("" when unresolvable — never raises); graph edges carry `metadata["jump_url"]` when non-empty.
- [ ] **Step 1 (failing tests):** classification of the three forms (existing forms `p.14`/`§3.2` keep their current types — assert unchanged); `jump_url` resolves against the Task-2 fixture store; absent anchor map → `""`. - [x] **Step 1 (failing tests):** classification of the three forms (existing forms `p.14`/`§3.2` keep their current types — assert unchanged); `jump_url` resolves against the Task-2 fixture store; absent anchor map → `""`.
- [ ] **Step 2:** Run → FAIL. **Step 3:** Implement. **Step 4:** `uv run pytest tests/bib/test_pincite.py tests/bib/test_frlink.py -q` → PASS. - [x] **Step 2:** Run → FAIL. **Step 3:** Implement. **Step 4:** `uv run pytest tests/bib/test_pincite.py tests/bib/test_frlink.py -q` → PASS.
- [ ] **Step 5:** Commit: `feat(bib): :pincite: FR locators resolve to jump URLs (closes #636)` - [x] **Step 5:** Commit: `feat(bib): :pincite: FR locators resolve to jump URLs (closes #636)`
### Task 4: `place()` + Zotero linkMode=3 children + `fr-jump` CLI ([#637](https://git.fhirworx.io/homelab/stack/issues/637)) ### Task 4: `place()` + Zotero linkMode=3 children + `fr-jump` CLI ([#637](https://git.fhirworx.io/homelab/stack/issues/637))
@@ -118,27 +118,27 @@ def test_grab_persists_and_replaces(tmp_path):
**Interfaces:** **Interfaces:**
- Produces: `place(store, ref, *, item_key="", label="") -> JumpLink` (upsert `fr_links`; default label = `"¶ p-N — {ref}"` or the page ref); `_sync_fr_links(db, store, bib_item, zot_parent_id) -> int` creating child attachments via `db.add_attachment(parent, link_mode=3, content_type="text/html")` + `db.set_fields(att_id, {"url": url, "title": label})`, idempotent by existing children's `url` itemData; `fr-jump REF [--key K] [--md] [--place] [--label L]` CLI. - Produces: `place(store, ref, *, item_key="", label="") -> JumpLink` (upsert `fr_links`; default label = `"¶ p-N — {ref}"` or the page ref); `_sync_fr_links(db, store, bib_item, zot_parent_id) -> int` creating child attachments via `db.add_attachment(parent, link_mode=3, content_type="text/html")` + `db.set_fields(att_id, {"url": url, "title": label})`, idempotent by existing children's `url` itemData; `fr-jump REF [--key K] [--md] [--place] [--label L]` CLI.
- [ ] **Step 1 (failing tests):** `place` writes one `fr_links` row, double-place no-ops; sync test: bib item + placed link → push creates exactly one linkMode=3 child with url+title itemData; second push creates none. - [x] **Step 1 (failing tests):** `place` writes one `fr_links` row, double-place no-ops; sync test: bib item + placed link → push creates exactly one linkMode=3 child with url+title itemData; second push creates none.
- [ ] **Step 2:** Run → FAIL. **Step 3:** Implement. **Step 4:** `uv run pytest tests/bib/test_frlink.py tests/bib/test_sync.py -q` → PASS. - [x] **Step 2:** Run → FAIL. **Step 3:** Implement. **Step 4:** `uv run pytest tests/bib/test_frlink.py tests/bib/test_sync.py -q` → PASS.
- [ ] **Step 5:** Commit: `feat(bib): place FR jump links as Zotero child link attachments + fr-jump CLI (closes #637)` - [x] **Step 5:** Commit: `feat(bib): place FR jump links as Zotero child link attachments + fr-jump CLI (closes #637)`
### Task 5: Rollout ([#638](https://git.fhirworx.io/homelab/stack/issues/638)) ### Task 5: Rollout ([#638](https://git.fhirworx.io/homelab/stack/issues/638))
**Files:** **Files:**
- Modify: `notebooks/cy2027_pfs_proposed_rule.py` (live-link FR cites). Data: `bib.sqlite`, `zotero.sqlite`. - Modify: `notebooks/cy2027_pfs_proposed_rule.py` (live-link FR cites). Data: `bib.sqlite`, `zotero.sqlite`.
- [ ] **Step 1:** `uv run stack bib fr-grab --all` — backfill; report per-rule paragraph/page counts; expect ~30 rules, CY2027 = 5079/716. Failures on odd old rules: log + continue, note on the issue. - [x] **Step 1:** `uv run stack bib fr-grab --all` — backfill; report per-rule paragraph/page counts; expect ~30 rules, CY2027 = 5079/716. Failures on odd old rules: log + continue, note on the issue.
- [ ] **Step 2:** Notebook: in the APM narrative cell and Section-5 sources, replace the dead `91 FR 44218-44219, 44286` style cites with `frlink.md_link(...)` calls (import in the notebook's setup cell; reads bib.sqlite via `conf.connect.bib()` — available in the notebooks container through the mounted `data/`). In-cell fallback: if the anchor map is missing, degrade to plain text (notebook must not break offline). - [x] **Step 2:** Notebook: in the APM narrative cell and Section-5 sources, replace the dead `91 FR 44218-44219, 44286` style cites with `frlink.md_link(...)` calls (import in the notebook's setup cell; reads bib.sqlite via `conf.connect.bib()` — available in the notebooks container through the mounted `data/`). In-cell fallback: if the anchor map is missing, degrade to plain text (notebook must not break offline).
- [ ] **Step 3:** Gate: single-notebook `nb_integration.py` run in the prod container → `pass=1`. - [x] **Step 3:** Gate: single-notebook `nb_integration.py` run in the prod container → `pass=1`.
- [ ] **Step 4:** Place the cited paragraphs (`91 FR 44218` QP-alignment ¶, the 414.1450(b)(1) revival cite) via `stack bib fr-jump ... --place`, then `uv run stack bib sync-zotero --tag sup:2027_PFS_NPRM`; verify linkMode=3 children on `5ITGVDJV` in zotero.sqlite. - [x] **Step 4:** Place the cited paragraphs (`91 FR 44218` QP-alignment ¶, the 414.1450(b)(1) revival cite) via `stack bib fr-jump ... --place`, then `uv run stack bib sync-zotero --tag sup:2027_PFS_NPRM`; verify linkMode=3 children on `5ITGVDJV` in zotero.sqlite.
- [ ] **Step 5:** Commit notebook: `feat(notebooks): live FR jump links in CY2027 NPRM citations (closes #638)` - [x] **Step 5:** Commit notebook: `feat(notebooks): live FR jump links in CY2027 NPRM citations (closes #638)`
### Task 6: Close the loop ### Task 6: Close the loop
- [ ] **Step 1:** Push; CI green on HEAD (poll with `skipped` treated as terminal — see gitea-tracker-ops memory). - [x] **Step 1:** Push; CI green on HEAD (poll with `skipped` treated as terminal — see gitea-tracker-ops memory).
- [ ] **Step 2:** Issues #634–#638 closed (auto-close footers) with verification comments where data-only; close [milestone P40](https://git.fhirworx.io/homelab/stack/milestone/41). - [x] **Step 2:** Issues #634–#638 closed (auto-close footers) with verification comments where data-only; close [milestone P40](https://git.fhirworx.io/homelab/stack/milestone/41).
- [ ] **Step 3:** Append P40 build outcomes to the spec; commit docs. - [x] **Step 3:** Append P40 build outcomes to the spec; commit docs.
- [ ] **Step 4:** Memory: new `fr_jump_links.md` (anchor-map mechanics, `data-page` fact, resolver grammar, don't-bulk-place rule). - [x] **Step 4:** Memory: new `fr_jump_links.md` (anchor-map mechanics, `data-page` fact, resolver grammar, don't-bulk-place rule).
## Self-Review ## Self-Review

View File

@@ -154,3 +154,38 @@ sync to Zotero as children of `5ITGVDJV`.
- eCFR/edition permalinks (govinfo PDFs) — FR web version only. - eCFR/edition permalinks (govinfo PDFs) — FR web version only.
- Auto-rewriting existing docstring pincites; the grammar accepts FR - Auto-rewriting existing docstring pincites; the grammar accepts FR
locators, adoption is incremental. locators, adoption is incremental.
## P40 build outcomes (2026-08-17)
Issues #634–#638, milestone P40 (6 commits, 5ea78cf..d7adf59, direct on
main). All TDD; the pre-commit hook ran the bib-targeted suite green at
each step (4,200+ tests).
- **#634 (5ea78cf)**: `bib/frlink.py` grabber + `fr_anchors` /
`fr_anchor_docs` / `fr_links` tables + `fr-grab` CLI. **Ground-truth
correction**: the CY2027 body carries **4,856** paragraph anchors
(4,194 `<p>` + 662 `<li>`), not the 5,079 stated above — that figure
was a `grep -c` line-count artifact that also matched `id="page-`
lines. The parser handles both element kinds (close-tag aware) and
captures 100% of real anchors; 431 of the 716 printed pages carry
paragraph anchors (the rest are tables/addenda), and the resolver
reports that distinction explicitly on ¶-of-page lookups.
- **#635 (01e5d95)**: `resolve()` — page / page+¶-ordinal / raw
anchor / quote refs, volume+page-range rule inference, ambiguity
diagnostics; `page_of`/`paragraphs_of` transmutations; `md_link`.
Live: `"91 FR 44218 ¶2"` → `#p-3601` (the Participation-List ¶).
- **#636 (eda1887)**: locator types `fr_para`/`fr_anchor` added
(bare cites keep the backward-compatible `fr`); `Pincite.jump_url`
(degrades to `""`, never raises); `CitationEdge.jump_url` populated
by `build_citation_graph`.
- **#637 (e5e5eb8)**: `place()` → `fr_links`; `_sync_fr_links` in the
Zotero push (linkMode=3 linked_url children, url+title itemData,
idempotent by child URL); `fr-jump` CLI with `--md`/`--place`.
- **#638 (d7adf59)**: backfill grabbed **78 rules / 192,011 anchors,
zero errors**; CY2027 notebook cites are live links via a
graceful-fallback `fr_md()` helper (gate `pass=1`, resolution proven
in-container); three placed links synced to Zotero as child link
attachments on `5ITGVDJV` (incl. the §414.1425 QP-alignment
paragraph pinned by quote → `#p-3600`).
- Deviation: none of substance; the plan's ~30-rule estimate
undercounted (bib holds 78 FR-linked rules; all grabbed).