bib/llm: code + family anchors across the whole corpus — item tags and chunk metadata on rules, comments, IOM, eCFR, MLN #688

New Issue

kert · 2026-09-09T14:30:04Z

kert commented

2026-09-09 14:30:04 +00:00

Spec docs/superpowers/specs/2026-09-09-code-family-longitudinal-design.md (§Decisions 4; bib/tag.py, llm/chunk.py, llm/index.py).

Gap measured 2026-09-09. Only the rules pgvector collection carries codes metadata (29,595 of 67,340 chunks); comments (918,484 chunks) and corpus (235,140 chunks) carry none. No bib item has a code: or family: tag; the 177 IOM chapters (MCPM ch. 12 SPJB4FX2, MBPM ch. 15 YQFGB6JS = 988 corpus chunks), the 699 eCFR sections (§ 410.78 SAJSGG28, § 410.26 Q78GP4Z9, § 414.22 T7PXFV4S) and MLN909188 (PV8APQ4A, CCM/PCM/APCM booklet) are reachable only by semantic search.

What. Two anchor layers, one per consumer:

bib/Zotero item tags code:<hcpcs> and family:<key> (new namespaces in bib/tag.py), applied by a tagger that walks fr_anchors.text for rules, chunk text for comments, attachment text for IOM/MLN, section text for eCFR. Nightly zotero-sync carries them under the P38 tag-merge rules (sup: tags stay load-bearing).
pgvector chunk metadata codes (existing), families, elements (type=value slugs when the chunk mentions a vocabulary value) on all three collections; one-time stack llm index --collection all --force (rules re-index ran at ≈5–6 rules/min on the 3-GPU pool in P34; comments will take days — run per docket, sealed dockets first).

Done when code_cited_sources (P48) can pull comment and IOM chunks for a code, e.g. 1,507 comment items whose chunks mention 99490/CCM appear with codes ∋ 99490, and stack bib tags --ns code lists counts per code per source type.

Spec docs/superpowers/specs/2026-09-09-code-family-longitudinal-design.md (§Decisions 4; `bib/tag.py`, `llm/chunk.py`, `llm/index.py`). **Gap measured 2026-09-09.** Only the `rules` pgvector collection carries `codes` metadata (29,595 of 67,340 chunks); `comments` (918,484 chunks) and `corpus` (235,140 chunks) carry none. No bib item has a `code:` or `family:` tag; the 177 IOM chapters (MCPM ch. 12 `SPJB4FX2`, MBPM ch. 15 `YQFGB6JS` = 988 corpus chunks), the 699 eCFR sections (§ 410.78 `SAJSGG28`, § 410.26 `Q78GP4Z9`, § 414.22 `T7PXFV4S`) and MLN909188 (`PV8APQ4A`, CCM/PCM/APCM booklet) are reachable only by semantic search. **What.** Two anchor layers, one per consumer: - bib/Zotero item tags `code:<hcpcs>` and `family:<key>` (new namespaces in `bib/tag.py`), applied by a tagger that walks `fr_anchors.text` for rules, chunk text for comments, attachment text for IOM/MLN, section text for eCFR. Nightly zotero-sync carries them under the P38 tag-merge rules (`sup:` tags stay load-bearing). - pgvector chunk metadata `codes` (existing), `families`, `elements` (`type=value` slugs when the chunk mentions a vocabulary value) on all three collections; one-time `stack llm index --collection all --force` (rules re-index ran at ≈5–6 rules/min on the 3-GPU pool in P34; comments will take days — run per docket, sealed dockets first). **Done when** `code_cited_sources` (P48) can pull comment and IOM chunks for a code, e.g. 1,507 comment items whose chunks mention 99490/CCM appear with `codes ∋ 99490`, and `stack bib tags --ns code` lists counts per code per source type.

kert added this to the P49: Code families as first-class objects — element model, lineage anchors, longitudinal chat milestone 2026-09-09 14:30:04 +00:00

kert added the llm bib labels 2026-09-09 14:30:04 +00:00

kert referenced this issue from a commit

2026-09-10 01:50:02 +00:00

feat(pfs,llm): anchor metadata — families and elements stamped beside codes; GIN indexes (refs #688)

kert referenced this issue from a commit

2026-09-10 01:50:02 +00:00

perf(pfs): code→family index for anchor_metadata (refs #688)

kert referenced this issue from a commit

2026-09-10 01:50:02 +00:00

feat(llm): stack llm restamp — metadata-only anchor backfill on pgvector (refs #688)

kert referenced this issue from a commit

2026-09-10 01:50:02 +00:00

fix(pfs): code→family index keeps every family so indexed and scan anchors agree (refs #688)

kert referenced this issue from a commit

2026-09-10 01:50:02 +00:00

feat(llm): corpus indexer skips items tagged llm:skip (refs #688)

kert referenced this issue from a commit

2026-09-10 01:50:02 +00:00

merge: P49 slice 2a + anchors 1–3 — CPT manual organizing principles (parser, cpt tables, CPT-named families, lineage/extraction sources), chunk anchors + restamp, explainer notebook (refs #684 #685 #686 #687 #688)

kert commented

2026-09-10 02:47:54 +00:00

Anchors progress (branch worktree-p49-anchors-2, unmerged): stack bib code-tags (7eeee59 + fix c3628d8) tags bib items from chunk anchors and FR paragraphs — code:<HCPCS> / family:<KEY> namespaces, additive and idempotent, wide items (>200 distinct codes across all sources) get family: tags only. Live: 67,198 items, 486,519 tags (22,111 distinct code:, 786 family:); the six CPT edition items carry their family tags. Fix round merged the per-source mappings before the cap and added Store.add_tags (one commit per item; the first run took 34 min on per-tag commits). Next: code/family-cited sources across collections (Task 5).

Anchors progress (branch worktree-p49-anchors-2, unmerged): `stack bib code-tags` (7eeee59 + fix c3628d8) tags bib items from chunk anchors and FR paragraphs — `code:<HCPCS>` / `family:<KEY>` namespaces, additive and idempotent, wide items (>200 distinct codes across all sources) get `family:` tags only. Live: 67,198 items, 486,519 tags (22,111 distinct `code:`, 786 `family:`); the six CPT edition items carry their family tags. Fix round merged the per-source mappings before the cap and added `Store.add_tags` (one commit per item; the first run took 34 min on per-tag commits). Next: code/family-cited sources across collections (Task 5).

kert referenced this issue from a commit

2026-09-10 05:29:41 +00:00

feat(bib): code:/family: item tags from chunk anchors and FR paragraphs — stack bib code-tags (refs #688)

kert referenced this issue from a commit

2026-09-10 05:29:41 +00:00

fix(bib): merge every source before capping code-tags; batch tag commits per item (refs #688)

kert referenced this issue from a commit

2026-09-10 05:29:41 +00:00

feat(llm): code-cited and family-cited sources from rules, comments and corpus (refs #688)

kert referenced this issue from a commit

2026-09-10 05:29:41 +00:00

fix(llm): family-cited window is one chunk per item, newest items first (refs #688)

kert referenced this issue from a commit

2026-09-10 05:29:42 +00:00

test(notebooks): section-5 tolerance test asserts the rendered tag table (refs #688)

kert referenced this issue from a commit

2026-09-10 05:29:42 +00:00

merge: P49 anchors slice — code:/family: item tags, cross-collection cited sources, IOM/CFR guidance crosswalk, public-reaction series, notebook sections 5–7 (refs #688 #689 #690)

kert referenced this issue

2026-09-10 05:30:11 +00:00

P49 anchors follow-ups: CPT manual as a chat source, IOM page locating, MLN resolver, shared code-or-family SQL predicate #705

kert commented

2026-09-10 05:30:33 +00:00

Merged to main as 4616d36 (branch worktree-p49-anchors-2, Tasks 4–8 of the anchors plan; suite 14,944 passed). Residue parked by rulings is tracked in #705 (CPT manual as a chat source, IOM page locating, MLN resolver, shared SQL predicate, key case).

Merged to main as 4616d36 (branch worktree-p49-anchors-2, Tasks 4–8 of the anchors plan; suite 14,944 passed). Residue parked by rulings is tracked in #705 (CPT manual as a chat source, IOM page locating, MLN resolver, shared SQL predicate, key case).

kert closed this issue

2026-09-10 05:30:33 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#688