homelab/stack
1
0
Fork 0
Code Issues 70 Pull Requests 1 Actions Packages Projects 1 Releases Wiki Activity

bib: fr_anchors schema + grabber — persist FR web anchor maps per rule (fr-grab CLI + backfill) #634

New Issue
Closed
opened 2026-08-17 18:18:07 +00:00 by kert · 0 comments
kert commented 2026-08-17 18:18:07 +00:00
Owner
Copy Link

New src/bib/frlink.py grabber + three tables in bib/schema.sql (auto-applied on Store init):

  • fr_anchors(item_key, p_id, page, ordinal, text) — one row per id="p-N" paragraph anchor from the FR body_html_url, with printed page from the explicit data-page attribute and 1-based ordinal-on-page; text entity-decoded/tag-stripped for quote search. UNIQUE(item_key, p_id).
  • fr_anchor_docs(item_key PK, document_number, html_url, body_html_url, start_page, end_page, fr_volume, fetched_at, sha256, n_paragraphs, n_pages)
  • fr_links(item_key, p_id, page, label, url, created_at) UNIQUE(item_key, url) — placed links (consumed by #637)

grab(store, item_key, force=False): resolve document_number from the rule item's extra_json (fallback: parse the FR URL), fetch body_html_url via the FR API, parse_anchors(html) pure function, replace that item's partition. stack bib fr-grab [--all|--key K] [--force] backfills every item_type='rule' with a federalregister.gov URL (~30 rules).

Verified ground truth: CMS-1848-P body HTML = 4.2 MB, 5,079 p-N anchors, 716 page-N anchors, data-page on every paragraph.

Tests: synthetic-HTML parser fixture (entities, ordinals), partition-replace idempotency, real-file smoke against the downloaded CY2027 body.

Spec: docs/superpowers/specs/2026-08-17-fr-jump-links-design.md.

New `src/bib/frlink.py` grabber + three tables in `bib/schema.sql` (auto-applied on Store init): - `fr_anchors(item_key, p_id, page, ordinal, text)` — one row per `id="p-N"` paragraph anchor from the FR `body_html_url`, with printed page from the explicit `data-page` attribute and 1-based ordinal-on-page; text entity-decoded/tag-stripped for quote search. UNIQUE(item_key, p_id). - `fr_anchor_docs(item_key PK, document_number, html_url, body_html_url, start_page, end_page, fr_volume, fetched_at, sha256, n_paragraphs, n_pages)` - `fr_links(item_key, p_id, page, label, url, created_at)` UNIQUE(item_key, url) — placed links (consumed by #637) `grab(store, item_key, force=False)`: resolve document_number from the rule item's extra_json (fallback: parse the FR URL), fetch body_html_url via the FR API, `parse_anchors(html)` pure function, replace that item's partition. `stack bib fr-grab [--all|--key K] [--force]` backfills every item_type='rule' with a federalregister.gov URL (~30 rules). Verified ground truth: CMS-1848-P body HTML = 4.2 MB, 5,079 `p-N` anchors, 716 `page-N` anchors, `data-page` on every paragraph. Tests: synthetic-HTML parser fixture (entities, ordinals), partition-replace idempotency, real-file smoke against the downloaded CY2027 body. Spec: docs/superpowers/specs/2026-08-17-fr-jump-links-design.md.
kert added this to the P40: FR Jump Links — page/paragraph deep links for every FR citation milestone 2026-08-17 18:18:07 +00:00
kert added the bibdatacli labels 2026-08-17 18:18:07 +00:00
kert closed this issue 2026-08-17 18:44:53 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
api
REST API (FastAPI)
 auth
Authentication and authorization
 bib
Bibliography, Zotero, citations
 ci
CI/CD pipelines and Woodpecker plugins
 cli
CLI commands and UX
 cloud
Cloud provider abstraction (AWS, GCP, Azure, self-hosted)
 config
Configuration and wiring
 data
Data loading, CCLF, BCDA, seeds
 databricks
Databricks jobs and deployment
 docs
Documentation and generated references
 image:api
image:docs
image:mc
image:notebooks
image:zotero
infra
lakehouse
Nessie, Polaris, Trino, Iceberg
 llm
llm module — local RAG over comments + Zotero corpus
 notebooks
marimo notebook errors (auto-filed by nb-watcher/nb-integration)
 packaging
Python packaging, optional deps, build system
 pfs
pipeline
Pipeline execution and validation
 pipeline:ci
pipeline:deploy
pipeline:harden
pipeline:infra-ci
pipeline:rebuild-all
pipeline:release
qpp
quality
Testing, coverage, error handling
 step:build
step:deploy
step:hadolint
step:lint
step:publish
step:scan
step:test
step:validate
styles
Charts, figures, visual theming
 tree
Project directory structure and file organization
No Label bib cli data
Milestone
No items
No Milestone P40: FR Jump Links — page/paragraph deep links for every FR citation
Projects
Clear projects
No project
Assignees
Clear assignees
kert renovate-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: homelab/stack#634
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?