docs(spec): P40 build outcomes; plan checked off (refs #634-#638)
All checks were successful
CI / lint (push) Successful in 35s
CI / notebooks-smoke (push) Successful in 1m28s
Deploy / notebooks (push) Has been skipped
Deploy / zotero (push) Has been skipped
Deploy / docs (push) Has been skipped
Deploy / api (push) Has been skipped
Deploy / llm (push) Has been skipped
Deploy / mc (push) Has been skipped
Infra CI / notebooks (push) Successful in 1m9s
Infra CI / zotero (push) Successful in 22s
Infra CI / docs (push) Successful in 1m13s
Infra CI / api (push) Successful in 1m14s
Infra CI / llm (push) Successful in 46s
Infra CI / mc (push) Successful in 14s
Deploy / report (push) Successful in 12s
CI / test (push) Successful in 11m49s
All checks were successful
CI / lint (push) Successful in 35s
CI / notebooks-smoke (push) Successful in 1m28s
Deploy / notebooks (push) Has been skipped
Deploy / zotero (push) Has been skipped
Deploy / docs (push) Has been skipped
Deploy / api (push) Has been skipped
Deploy / llm (push) Has been skipped
Deploy / mc (push) Has been skipped
Infra CI / notebooks (push) Successful in 1m9s
Infra CI / zotero (push) Successful in 22s
Infra CI / docs (push) Successful in 1m13s
Infra CI / api (push) Successful in 1m14s
Infra CI / llm (push) Successful in 46s
Infra CI / mc (push) Successful in 14s
Deploy / report (push) Successful in 12s
CI / test (push) Successful in 11m49s
This commit is contained in:
@@ -42,7 +42,7 @@
|
|||||||
- `backfill(store, *, force=False, sleep=1.0) -> list[dict]` over `item_type='rule'` items with federalregister.gov URLs.
|
- `backfill(store, *, force=False, sleep=1.0) -> list[dict]` over `item_type='rule'` items with federalregister.gov URLs.
|
||||||
- Schema: as specced — `fr_anchors` UNIQUE(item_key, p_id) + idx (item_key, page); `fr_anchor_docs` PK item_key; `fr_links` UNIQUE(item_key, url).
|
- Schema: as specced — `fr_anchors` UNIQUE(item_key, p_id) + idx (item_key, page); `fr_anchor_docs` PK item_key; `fr_links` UNIQUE(item_key, url).
|
||||||
|
|
||||||
- [ ] **Step 1 (failing tests):** `tests/bib/test_frlink.py` with a synthetic body-HTML fixture:
|
- [x] **Step 1 (failing tests):** `tests/bib/test_frlink.py` with a synthetic body-HTML fixture:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
FIXTURE_HTML = """
|
FIXTURE_HTML = """
|
||||||
@@ -69,11 +69,11 @@ def test_grab_persists_and_replaces(tmp_path):
|
|||||||
```
|
```
|
||||||
(write both fully, using `Store(":memory:")` + a `Rule` item with `extra_json` fields matching the real shape: `{"document_number": "2026-14327", "fr_volume": "91", "fr_page": "43842"}`)
|
(write both fully, using `Store(":memory:")` + a `Rule` item with `extra_json` fields matching the real shape: `{"document_number": "2026-14327", "fr_volume": "91", "fr_page": "43842"}`)
|
||||||
|
|
||||||
- [ ] **Step 2:** Run: `uv run pytest tests/bib/test_frlink.py -q` → FAIL (module missing).
|
- [x] **Step 2:** Run: `uv run pytest tests/bib/test_frlink.py -q` → FAIL (module missing).
|
||||||
- [ ] **Step 3:** Implement `src/bib/frlink.py` (`_P_RE = re.compile(r'id="p-(\d+)"[^>]*data-page="(\d+)"')`; text = substring to the next `</p>`, tag-strip + unescape + `" ".join(split())`), schema additions, and the `fr-grab` typer command calling `grab`/`backfill`.
|
- [x] **Step 3:** Implement `src/bib/frlink.py` (`_P_RE = re.compile(r'id="p-(\d+)"[^>]*data-page="(\d+)"')`; text = substring to the next `</p>`, tag-strip + unescape + `" ".join(split())`), schema additions, and the `fr-grab` typer command calling `grab`/`backfill`.
|
||||||
- [ ] **Step 4:** `uv run pytest tests/bib/test_frlink.py tests/bib/test_store.py -q` → PASS (store tests prove schema addition breaks nothing).
|
- [x] **Step 4:** `uv run pytest tests/bib/test_frlink.py tests/bib/test_store.py -q` → PASS (store tests prove schema addition breaks nothing).
|
||||||
- [ ] **Step 5:** Offline real-file smoke: run `parse_anchors` on the cached CY2027 `fr_body.html` → assert 5079 paragraphs, 716 distinct pages (one-off command, not a committed test — the 4.2 MB file stays out of the repo).
|
- [x] **Step 5:** Offline real-file smoke: run `parse_anchors` on the cached CY2027 `fr_body.html` → assert 5079 paragraphs, 716 distinct pages (one-off command, not a committed test — the 4.2 MB file stays out of the repo).
|
||||||
- [ ] **Step 6:** Commit: `feat(bib): fr_anchors schema + FR body-HTML grabber + fr-grab CLI (closes #634)`
|
- [x] **Step 6:** Commit: `feat(bib): fr_anchors schema + FR body-HTML grabber + fr-grab CLI (closes #634)`
|
||||||
|
|
||||||
### Task 2: Resolver + transmutations + `md_link` ([#635](https://git.fhirworx.io/homelab/stack/issues/635))
|
### Task 2: Resolver + transmutations + `md_link` ([#635](https://git.fhirworx.io/homelab/stack/issues/635))
|
||||||
|
|
||||||
@@ -89,11 +89,11 @@ def test_grab_persists_and_replaces(tmp_path):
|
|||||||
- `md_link(ref, *, store, item_key="", text="") -> str`
|
- `md_link(ref, *, store, item_key="", text="") -> str`
|
||||||
- URL forms: `{html_url}#page-{page}` / `{html_url}#p-{p_id}`
|
- URL forms: `{html_url}#page-{page}` / `{html_url}#p-{p_id}`
|
||||||
|
|
||||||
- [ ] **Step 1 (failing tests):** extend the fixture store helper to populate `fr_anchor_docs` (html_url `https://example.test/doc`, volume 91, pages 100–101) + the 3 fixture anchors; add tests: page ref, page+ordinal (`"91 FR 100 ¶2"` → `#p-2`), raw `p-3`, quote (`"Only para on 101"` → `#p-3`), quote-multi-hit raises, unknown page raises, `page_of`/`paragraphs_of` round-trip, `md_link` default text equals the ref.
|
- [x] **Step 1 (failing tests):** extend the fixture store helper to populate `fr_anchor_docs` (html_url `https://example.test/doc`, volume 91, pages 100–101) + the 3 fixture anchors; add tests: page ref, page+ordinal (`"91 FR 100 ¶2"` → `#p-2`), raw `p-3`, quote (`"Only para on 101"` → `#p-3`), quote-multi-hit raises, unknown page raises, `page_of`/`paragraphs_of` round-trip, `md_link` default text equals the ref.
|
||||||
- [ ] **Step 2:** Run → new tests FAIL.
|
- [x] **Step 2:** Run → new tests FAIL.
|
||||||
- [ ] **Step 3:** Implement; quote normalization shared with `parse_anchors` storage form.
|
- [x] **Step 3:** Implement; quote normalization shared with `parse_anchors` storage form.
|
||||||
- [ ] **Step 4:** `uv run pytest tests/bib/test_frlink.py -q` → PASS.
|
- [x] **Step 4:** `uv run pytest tests/bib/test_frlink.py -q` → PASS.
|
||||||
- [ ] **Step 5:** Commit: `feat(bib): context-aware FR jump resolver — page/ordinal/anchor/quote + transmutations (closes #635)`
|
- [x] **Step 5:** Commit: `feat(bib): context-aware FR jump resolver — page/ordinal/anchor/quote + transmutations (closes #635)`
|
||||||
|
|
||||||
### Task 3: `:pincite:` FR locators + `jump_url` ([#636](https://git.fhirworx.io/homelab/stack/issues/636))
|
### Task 3: `:pincite:` FR locators + `jump_url` ([#636](https://git.fhirworx.io/homelab/stack/issues/636))
|
||||||
|
|
||||||
@@ -105,9 +105,9 @@ def test_grab_persists_and_replaces(tmp_path):
|
|||||||
- Consumes: `frlink.resolve` (lazy import inside `jump_url` to avoid import cycles).
|
- Consumes: `frlink.resolve` (lazy import inside `jump_url` to avoid import cycles).
|
||||||
- Produces: locator_type `"fr_page"` (`91 FR 44218`), `"fr_para"` (`91 FR 44218 ¶3`), `"fr_anchor"` (`p-3600`); `Pincite.jump_url(store) -> str` ("" when unresolvable — never raises); graph edges carry `metadata["jump_url"]` when non-empty.
|
- Produces: locator_type `"fr_page"` (`91 FR 44218`), `"fr_para"` (`91 FR 44218 ¶3`), `"fr_anchor"` (`p-3600`); `Pincite.jump_url(store) -> str` ("" when unresolvable — never raises); graph edges carry `metadata["jump_url"]` when non-empty.
|
||||||
|
|
||||||
- [ ] **Step 1 (failing tests):** classification of the three forms (existing forms `p.14`/`§3.2` keep their current types — assert unchanged); `jump_url` resolves against the Task-2 fixture store; absent anchor map → `""`.
|
- [x] **Step 1 (failing tests):** classification of the three forms (existing forms `p.14`/`§3.2` keep their current types — assert unchanged); `jump_url` resolves against the Task-2 fixture store; absent anchor map → `""`.
|
||||||
- [ ] **Step 2:** Run → FAIL. **Step 3:** Implement. **Step 4:** `uv run pytest tests/bib/test_pincite.py tests/bib/test_frlink.py -q` → PASS.
|
- [x] **Step 2:** Run → FAIL. **Step 3:** Implement. **Step 4:** `uv run pytest tests/bib/test_pincite.py tests/bib/test_frlink.py -q` → PASS.
|
||||||
- [ ] **Step 5:** Commit: `feat(bib): :pincite: FR locators resolve to jump URLs (closes #636)`
|
- [x] **Step 5:** Commit: `feat(bib): :pincite: FR locators resolve to jump URLs (closes #636)`
|
||||||
|
|
||||||
### Task 4: `place()` + Zotero linkMode=3 children + `fr-jump` CLI ([#637](https://git.fhirworx.io/homelab/stack/issues/637))
|
### Task 4: `place()` + Zotero linkMode=3 children + `fr-jump` CLI ([#637](https://git.fhirworx.io/homelab/stack/issues/637))
|
||||||
|
|
||||||
@@ -118,27 +118,27 @@ def test_grab_persists_and_replaces(tmp_path):
|
|||||||
**Interfaces:**
|
**Interfaces:**
|
||||||
- Produces: `place(store, ref, *, item_key="", label="") -> JumpLink` (upsert `fr_links`; default label = `"¶ p-N — {ref}"` or the page ref); `_sync_fr_links(db, store, bib_item, zot_parent_id) -> int` creating child attachments via `db.add_attachment(parent, link_mode=3, content_type="text/html")` + `db.set_fields(att_id, {"url": url, "title": label})`, idempotent by existing children's `url` itemData; `fr-jump REF [--key K] [--md] [--place] [--label L]` CLI.
|
- Produces: `place(store, ref, *, item_key="", label="") -> JumpLink` (upsert `fr_links`; default label = `"¶ p-N — {ref}"` or the page ref); `_sync_fr_links(db, store, bib_item, zot_parent_id) -> int` creating child attachments via `db.add_attachment(parent, link_mode=3, content_type="text/html")` + `db.set_fields(att_id, {"url": url, "title": label})`, idempotent by existing children's `url` itemData; `fr-jump REF [--key K] [--md] [--place] [--label L]` CLI.
|
||||||
|
|
||||||
- [ ] **Step 1 (failing tests):** `place` writes one `fr_links` row, double-place no-ops; sync test: bib item + placed link → push creates exactly one linkMode=3 child with url+title itemData; second push creates none.
|
- [x] **Step 1 (failing tests):** `place` writes one `fr_links` row, double-place no-ops; sync test: bib item + placed link → push creates exactly one linkMode=3 child with url+title itemData; second push creates none.
|
||||||
- [ ] **Step 2:** Run → FAIL. **Step 3:** Implement. **Step 4:** `uv run pytest tests/bib/test_frlink.py tests/bib/test_sync.py -q` → PASS.
|
- [x] **Step 2:** Run → FAIL. **Step 3:** Implement. **Step 4:** `uv run pytest tests/bib/test_frlink.py tests/bib/test_sync.py -q` → PASS.
|
||||||
- [ ] **Step 5:** Commit: `feat(bib): place FR jump links as Zotero child link attachments + fr-jump CLI (closes #637)`
|
- [x] **Step 5:** Commit: `feat(bib): place FR jump links as Zotero child link attachments + fr-jump CLI (closes #637)`
|
||||||
|
|
||||||
### Task 5: Rollout ([#638](https://git.fhirworx.io/homelab/stack/issues/638))
|
### Task 5: Rollout ([#638](https://git.fhirworx.io/homelab/stack/issues/638))
|
||||||
|
|
||||||
**Files:**
|
**Files:**
|
||||||
- Modify: `notebooks/cy2027_pfs_proposed_rule.py` (live-link FR cites). Data: `bib.sqlite`, `zotero.sqlite`.
|
- Modify: `notebooks/cy2027_pfs_proposed_rule.py` (live-link FR cites). Data: `bib.sqlite`, `zotero.sqlite`.
|
||||||
|
|
||||||
- [ ] **Step 1:** `uv run stack bib fr-grab --all` — backfill; report per-rule paragraph/page counts; expect ~30 rules, CY2027 = 5079/716. Failures on odd old rules: log + continue, note on the issue.
|
- [x] **Step 1:** `uv run stack bib fr-grab --all` — backfill; report per-rule paragraph/page counts; expect ~30 rules, CY2027 = 5079/716. Failures on odd old rules: log + continue, note on the issue.
|
||||||
- [ ] **Step 2:** Notebook: in the APM narrative cell and Section-5 sources, replace the dead `91 FR 44218-44219, 44286` style cites with `frlink.md_link(...)` calls (import in the notebook's setup cell; reads bib.sqlite via `conf.connect.bib()` — available in the notebooks container through the mounted `data/`). In-cell fallback: if the anchor map is missing, degrade to plain text (notebook must not break offline).
|
- [x] **Step 2:** Notebook: in the APM narrative cell and Section-5 sources, replace the dead `91 FR 44218-44219, 44286` style cites with `frlink.md_link(...)` calls (import in the notebook's setup cell; reads bib.sqlite via `conf.connect.bib()` — available in the notebooks container through the mounted `data/`). In-cell fallback: if the anchor map is missing, degrade to plain text (notebook must not break offline).
|
||||||
- [ ] **Step 3:** Gate: single-notebook `nb_integration.py` run in the prod container → `pass=1`.
|
- [x] **Step 3:** Gate: single-notebook `nb_integration.py` run in the prod container → `pass=1`.
|
||||||
- [ ] **Step 4:** Place the cited paragraphs (`91 FR 44218` QP-alignment ¶, the 414.1450(b)(1) revival cite) via `stack bib fr-jump ... --place`, then `uv run stack bib sync-zotero --tag sup:2027_PFS_NPRM`; verify linkMode=3 children on `5ITGVDJV` in zotero.sqlite.
|
- [x] **Step 4:** Place the cited paragraphs (`91 FR 44218` QP-alignment ¶, the 414.1450(b)(1) revival cite) via `stack bib fr-jump ... --place`, then `uv run stack bib sync-zotero --tag sup:2027_PFS_NPRM`; verify linkMode=3 children on `5ITGVDJV` in zotero.sqlite.
|
||||||
- [ ] **Step 5:** Commit notebook: `feat(notebooks): live FR jump links in CY2027 NPRM citations (closes #638)`
|
- [x] **Step 5:** Commit notebook: `feat(notebooks): live FR jump links in CY2027 NPRM citations (closes #638)`
|
||||||
|
|
||||||
### Task 6: Close the loop
|
### Task 6: Close the loop
|
||||||
|
|
||||||
- [ ] **Step 1:** Push; CI green on HEAD (poll with `skipped` treated as terminal — see gitea-tracker-ops memory).
|
- [x] **Step 1:** Push; CI green on HEAD (poll with `skipped` treated as terminal — see gitea-tracker-ops memory).
|
||||||
- [ ] **Step 2:** Issues #634–#638 closed (auto-close footers) with verification comments where data-only; close [milestone P40](https://git.fhirworx.io/homelab/stack/milestone/41).
|
- [x] **Step 2:** Issues #634–#638 closed (auto-close footers) with verification comments where data-only; close [milestone P40](https://git.fhirworx.io/homelab/stack/milestone/41).
|
||||||
- [ ] **Step 3:** Append P40 build outcomes to the spec; commit docs.
|
- [x] **Step 3:** Append P40 build outcomes to the spec; commit docs.
|
||||||
- [ ] **Step 4:** Memory: new `fr_jump_links.md` (anchor-map mechanics, `data-page` fact, resolver grammar, don't-bulk-place rule).
|
- [x] **Step 4:** Memory: new `fr_jump_links.md` (anchor-map mechanics, `data-page` fact, resolver grammar, don't-bulk-place rule).
|
||||||
|
|
||||||
## Self-Review
|
## Self-Review
|
||||||
|
|
||||||
|
|||||||
@@ -154,3 +154,38 @@ sync to Zotero as children of `5ITGVDJV`.
|
|||||||
- eCFR/edition permalinks (govinfo PDFs) — FR web version only.
|
- eCFR/edition permalinks (govinfo PDFs) — FR web version only.
|
||||||
- Auto-rewriting existing docstring pincites; the grammar accepts FR
|
- Auto-rewriting existing docstring pincites; the grammar accepts FR
|
||||||
locators, adoption is incremental.
|
locators, adoption is incremental.
|
||||||
|
|
||||||
|
## P40 build outcomes (2026-08-17)
|
||||||
|
|
||||||
|
Issues #634–#638, milestone P40 (6 commits, 5ea78cf..d7adf59, direct on
|
||||||
|
main). All TDD; the pre-commit hook ran the bib-targeted suite green at
|
||||||
|
each step (4,200+ tests).
|
||||||
|
|
||||||
|
- **#634 (5ea78cf)**: `bib/frlink.py` grabber + `fr_anchors` /
|
||||||
|
`fr_anchor_docs` / `fr_links` tables + `fr-grab` CLI. **Ground-truth
|
||||||
|
correction**: the CY2027 body carries **4,856** paragraph anchors
|
||||||
|
(4,194 `<p>` + 662 `<li>`), not the 5,079 stated above — that figure
|
||||||
|
was a `grep -c` line-count artifact that also matched `id="page-`
|
||||||
|
lines. The parser handles both element kinds (close-tag aware) and
|
||||||
|
captures 100% of real anchors; 431 of the 716 printed pages carry
|
||||||
|
paragraph anchors (the rest are tables/addenda), and the resolver
|
||||||
|
reports that distinction explicitly on ¶-of-page lookups.
|
||||||
|
- **#635 (01e5d95)**: `resolve()` — page / page+¶-ordinal / raw
|
||||||
|
anchor / quote refs, volume+page-range rule inference, ambiguity
|
||||||
|
diagnostics; `page_of`/`paragraphs_of` transmutations; `md_link`.
|
||||||
|
Live: `"91 FR 44218 ¶2"` → `#p-3601` (the Participation-List ¶).
|
||||||
|
- **#636 (eda1887)**: locator types `fr_para`/`fr_anchor` added
|
||||||
|
(bare cites keep the backward-compatible `fr`); `Pincite.jump_url`
|
||||||
|
(degrades to `""`, never raises); `CitationEdge.jump_url` populated
|
||||||
|
by `build_citation_graph`.
|
||||||
|
- **#637 (e5e5eb8)**: `place()` → `fr_links`; `_sync_fr_links` in the
|
||||||
|
Zotero push (linkMode=3 linked_url children, url+title itemData,
|
||||||
|
idempotent by child URL); `fr-jump` CLI with `--md`/`--place`.
|
||||||
|
- **#638 (d7adf59)**: backfill grabbed **78 rules / 192,011 anchors,
|
||||||
|
zero errors**; CY2027 notebook cites are live links via a
|
||||||
|
graceful-fallback `fr_md()` helper (gate `pass=1`, resolution proven
|
||||||
|
in-container); three placed links synced to Zotero as child link
|
||||||
|
attachments on `5ITGVDJV` (incl. the §414.1425 QP-alignment
|
||||||
|
paragraph pinned by quote → `#p-3600`).
|
||||||
|
- Deviation: none of substance; the plan's ~30-rule estimate
|
||||||
|
undercounted (bib holds 78 FR-linked rules; all grabbed).
|
||||||
|
|||||||
Reference in New Issue
Block a user