homelab/stack
1
0
Fork 0
Code Issues 70 Pull Requests 1 Actions Packages Projects 1 Releases Wiki Activity

comments: post-close re-farm + re-index of CMS-2026-2377 #615

New Issue
Closed
opened 2026-08-13 20:36:16 +00:00 by kert · 13 comments
kert commented 2026-08-13 20:36:16 +00:00
Owner
Copy Link

Delta D (ops, scheduled). After the DATES close date (transcribed in the registry issue): re-run the same resumable commands to capture the full docket; post final counts. BLOCKS milestone close.

Spec: docs/superpowers/specs/2026-08-13-cy2027-pfs-proposed-rule-design.md (P37)

Delta D (ops, scheduled). After the DATES close date (transcribed in the registry issue): re-run the same resumable commands to capture the full docket; post final counts. BLOCKS milestone close. Spec: [`docs/superpowers/specs/2026-08-13-cy2027-pfs-proposed-rule-design.md`](https://git.fhirworx.io/homelab/stack/src/branch/main/docs/superpowers/specs/2026-08-13-cy2027-pfs-proposed-rule-design.md) (P37)
kert added this to the P37: CY2027 PFS Proposed Rule — capture, ingest, APM analysis milestone 2026-08-13 20:36:16 +00:00
kert added the datallm labels 2026-08-13 20:36:16 +00:00
kert commented 2026-08-13 23:48:22 +00:00
Author
Owner
Copy Link

Implementation plan: docs/superpowers/plans/2026-08-13-cy2027-pfs-proposed-rule-p37.md — Task 9. Spec: docs/superpowers/specs/2026-08-13-cy2027-pfs-proposed-rule-design.md.

Implementation plan: [`docs/superpowers/plans/2026-08-13-cy2027-pfs-proposed-rule-p37.md`](https://git.fhirworx.io/homelab/stack/src/branch/main/docs/superpowers/plans/2026-08-13-cy2027-pfs-proposed-rule-p37.md) — Task 9. Spec: [`docs/superpowers/specs/2026-08-13-cy2027-pfs-proposed-rule-design.md`](https://git.fhirworx.io/homelab/stack/src/branch/main/docs/superpowers/specs/2026-08-13-cy2027-pfs-proposed-rule-design.md).
kert commented 2026-08-14 00:40:05 +00:00
Author
Owner
Copy Link

Scheduled. Comment period closes 2026-09-14 (verified in the DATES block of the captured NPRM text, 91 FR 43842). Host crontab fires dev/scripts/refarm_cms_2026_2377.sh on 2026-09-15 03:17 — resumable fetch → backfill → extract/OCR → pgvector index, then posts final counts here automatically. Human close-out after: verify counts, close this issue, close milestone P37, and remove the crontab line (it's pinned to Sept 15 and would refire annually).

Scheduled. Comment period closes **2026-09-14** (verified in the DATES block of the captured NPRM text, 91 FR 43842). Host crontab fires [`dev/scripts/refarm_cms_2026_2377.sh`](https://git.fhirworx.io/homelab/stack/src/branch/p37-cy2027-rule/dev/scripts/refarm_cms_2026_2377.sh) on **2026-09-15 03:17** — resumable fetch → backfill → extract/OCR → pgvector index, then posts final counts here automatically. Human close-out after: verify counts, close this issue, close milestone P37, and remove the crontab line (it's pinned to Sept 15 and would refire annually).
kert commented 2026-08-14 15:10:52 +00:00
Author
Owner
Copy Link

P37 merged (a75b4c7); this issue + milestone stay open for the 2026-09-15 scheduled re-farm. Initial pull captured 2,727 comments.

P37 merged (a75b4c7); this issue + milestone stay open for the 2026-09-15 scheduled re-farm. Initial pull captured 2,727 comments.
kert commented 2026-08-14 19:48:00 +00:00
Author
Owner
Copy Link

Ops correction ahead of the September re-farm, in a03801f:

  1. bib backfill-comments gained --docket (src/cli/bib.py:341 → src/bib/regulations_gov.py:407, filtering on the store's reg-docket:<id> tag). The unscoped walk covers ~167k un-enriched stubs at reg.gov's ~970/hr cap — a week-long crawl that ordered the fresh CMS-2026-2377 comments (highest item ids) last, which had silently blocked this docket's extract/index chain. The scoped run enriches 2,727 comments in ~3h.
  2. dev/scripts/refarm_cms_2026_2377.sh fixed twice: backfill now scoped with --docket CMS-2026-2377, and the comments extract-ocr step is removed — it is a phase-2 stub that always exits 2 (#253) and even lacks a --docket option, so every cron run would have logged a spurious failed step.
  3. The corrected chain (backfill → extract → index comments → index rule text 5ITGVDJV) is running now against the 2,727 farmed comments; final counts will be posted here when it completes.

Gotcha for the record: docket:<id> is NOT the docket tag — the live convention is reg-docket:<id> (docket: exists only as an empty-valued wart, per src/llm/source.py:11).

Ops correction ahead of the September re-farm, in [a03801f](https://git.fhirworx.io/homelab/stack/commit/a03801f): 1. **`bib backfill-comments` gained `--docket`** ([src/cli/bib.py:341](https://git.fhirworx.io/homelab/stack/src/branch/main/src/cli/bib.py#L341) → [src/bib/regulations_gov.py:407](https://git.fhirworx.io/homelab/stack/src/branch/main/src/bib/regulations_gov.py#L407), filtering on the store's `reg-docket:<id>` tag). The unscoped walk covers ~167k un-enriched stubs at reg.gov's ~970/hr cap — a week-long crawl that ordered the fresh CMS-2026-2377 comments (highest item ids) *last*, which had silently blocked this docket's extract/index chain. The scoped run enriches 2,727 comments in ~3h. 2. **[dev/scripts/refarm_cms_2026_2377.sh](https://git.fhirworx.io/homelab/stack/src/branch/main/dev/scripts/refarm_cms_2026_2377.sh) fixed twice**: backfill now scoped with `--docket CMS-2026-2377`, and the `comments extract-ocr` step is removed — it is a phase-2 stub that always exits 2 (#253) and even lacks a `--docket` option, so every cron run would have logged a spurious failed step. 3. The corrected chain (backfill → extract → index comments → index rule text 5ITGVDJV) is running now against the 2,727 farmed comments; final counts will be posted here when it completes. Gotcha for the record: `docket:<id>` is NOT the docket tag — the live convention is `reg-docket:<id>` (`docket:` exists only as an empty-valued wart, per [src/llm/source.py:11](https://git.fhirworx.io/homelab/stack/src/branch/main/src/llm/source.py#L11)).
kert commented 2026-09-03 14:58:37 +00:00
Author
Owner
Copy Link

Re-index kicked off 2026-09-03 on the 3-GPU pool: rules --force (80 rules from fr_anchors, exact paragraph provenance), then comments --force newest-posted-first (stamps date/kind/attachment-page metadata — legacy chunks have no date so recency ranking needs the force pass; picks up the ~30k comments posted since 2026-08-14 incl. CMS-2026-2377 through 2026-08-19), then corpus. Logs in .state/llm/index-*.log. Will post counts when done.

Re-index kicked off 2026-09-03 on the 3-GPU pool: `rules --force` (80 rules from fr_anchors, exact paragraph provenance), then `comments --force` newest-posted-first (stamps date/kind/attachment-page metadata — legacy chunks have no date so recency ranking needs the force pass; picks up the ~30k comments posted since 2026-08-14 incl. CMS-2026-2377 through 2026-08-19), then `corpus`. Logs in .state/llm/index-*.log. Will post counts when done.
kert commented 2026-09-04 14:08:31 +00:00
Author
Owner
Copy Link

Re-index chain finished 2026-09-04T14:08:31Z (rules --force → comments --force newest-first → corpus, 3-GPU pool).

index_state: comments: 196733 items (195531 re-stamped since 2026-09-03); corpus: 30 items (30 re-stamped since 2026-09-03); rules: 80 items (80 re-stamped since 2026-09-03)
chunks: comments=884137 rules=67340 corpus=71533
runner output: rules: rules: indexed=80 skipped=0 chunks=67340 | comments: ConnectError: [Errno -2] Name or service not known | corpus: ReadTimeout: timed out

Recency ranking now applies to every re-stamped comment; run stack llm index --collection all (after set -a; . ./.env; set +a) for incremental refreshes.

Re-index chain finished 2026-09-04T14:08:31Z (rules --force → comments --force newest-first → corpus, 3-GPU pool). index_state: comments: 196733 items (195531 re-stamped since 2026-09-03); corpus: 30 items (30 re-stamped since 2026-09-03); rules: 80 items (80 re-stamped since 2026-09-03) chunks: comments=884137 rules=67340 corpus=71533 runner output: rules: rules: indexed=80 skipped=0 chunks=67340 | comments: ConnectError: [Errno -2] Name or service not known | corpus: ReadTimeout: timed out Recency ranking now applies to every re-stamped comment; run `stack llm index --collection all` (after `set -a; . ./.env; set +a`) for incremental refreshes.
kert commented 2026-09-08 12:33:45 +00:00
Author
Owner
Copy Link

Kicked off the re-farm script by hand 2026-09-08 08:32 ET (pre-close, resumable) because regulations.gov already shows 18,647 comments on CMS-2026-2377 vs 3,533 local (last pull 2026-08-19); at ~970/hr backfill that is ~16 h of enrichment, so pulling now leaves the scheduled 2026-09-15 cron run only the final days to catch up. The script posts its own counts here when done. Log: .state/comments/refarm-CMS-2026-2377.log. Plan to re-run once more a week or two after the 09-14 close for late postings before closing this issue.

Kicked off the re-farm script by hand 2026-09-08 08:32 ET (pre-close, resumable) because regulations.gov already shows 18,647 comments on CMS-2026-2377 vs 3,533 local (last pull 2026-08-19); at ~970/hr backfill that is ~16 h of enrichment, so pulling now leaves the scheduled 2026-09-15 cron run only the final days to catch up. The script posts its own counts here when done. Log: .state/comments/refarm-CMS-2026-2377.log. Plan to re-run once more a week or two after the 09-14 close for late postings before closing this issue.
kert commented 2026-09-08 16:09:38 +00:00
Author
Owner
Copy Link

Automated post-close re-farm ran 2026-09-08: comments=18647 enriched=1040. Log: .state/comments/refarm-CMS-2026-2377.log. Review counts, spot-check indexing, then close this issue and milestone P37.

If complete: close #615, close milestone P37, and REMOVE the crontab line (re-fires annually).

Automated post-close re-farm ran 2026-09-08: comments=18647 enriched=1040. Log: .state/comments/refarm-CMS-2026-2377.log. Review counts, spot-check indexing, then close this issue and milestone P37. If complete: close #615, close milestone P37, and REMOVE the crontab line (re-fires annually).
kert commented 2026-09-08 16:19:18 +00:00
Author
Owner
Copy Link

Correction to the automated note above (enriched=1040 is wrong, and it is our own fault). The chain did enrich everything: mirror backfill created+enriched 15,118 comments (0 errors), extract wrote 757 dirs, index embedded 15,114 comments (30,565 chunks) — 18,647 distinct comments are now in pgvector. But the api fetch step that followed (old Store.upsert) re-listed the whole docket from the list endpoint, which carries no body, and overwrote 17,607 enriched abstracts with empty strings. Every row still carries enriched:ok, so the API backfill then skipped them. My aborted first launch this morning did the same to 1,115 comments in CMS-2017-0092. Other dockets are untouched (≤2 empties each, pre-existing).

Recovery is scheduled as part of the P47 seal-and-skip work (branch p47-seal-skip): upsert becomes data-preserving (empty incoming values never replace stored ones), attach_file becomes idempotent, then the affected rows get their enriched:ok cleared and are re-enriched from the mirror (~15 min). The pgvector index still holds the pre-wipe text. Counts will be re-posted after recovery. Do NOT close this issue on the automated note.

**Correction to the automated note above (enriched=1040 is wrong, and it is our own fault).** The chain did enrich everything: mirror backfill created+enriched 15,118 comments (0 errors), extract wrote 757 dirs, index embedded 15,114 comments (30,565 chunks) — 18,647 distinct comments are now in pgvector. But the *api fetch* step that followed (old `Store.upsert`) re-listed the whole docket from the list endpoint, which carries no body, and overwrote 17,607 enriched abstracts with empty strings. Every row still carries `enriched:ok`, so the API backfill then skipped them. My aborted first launch this morning did the same to 1,115 comments in CMS-2017-0092. Other dockets are untouched (≤2 empties each, pre-existing). Recovery is scheduled as part of the P47 seal-and-skip work (branch `p47-seal-skip`): upsert becomes data-preserving (empty incoming values never replace stored ones), attach_file becomes idempotent, then the affected rows get their `enriched:ok` cleared and are re-enriched from the mirror (~15 min). The pgvector index still holds the pre-wipe text. Counts will be re-posted after recovery. Do NOT close this issue on the automated note.
kert commented 2026-09-09 01:45:40 +00:00
Author
Owner
Copy Link

P47 rolled out (main @ e331fc7, merge db8193c). Every stage now skips complete dockets by default; --force is the only way to redo sealed work.

Recovery of this morning's clobber (see the correction above):

  • CMS-2026-2377: 20,385 / 20,385 comments with bodies (17,604 restored from the Mirrulations mirror; 1,738 + 7 new since the morning listing), 18,427 re-embedded (29,293 chunks).
  • CMS-2017-0092: 1,618 / 1,619 restored via the reg.gov API (the mirror lacks that docket's old comments; the 1 remaining is genuinely empty), 1,618 re-embedded (15,852 chunks).

Cleanup: 33,379 duplicate attachment groups → 33,538 rows and 33,458 files removed, 12.05 GB freed; unique index on (item, filename) now enforced.

State: 9 historical dockets sealed (stack comments dockets), CMS-2026-2377 open with a pull watermark; two 0-comment 2026 dockets (1156, 2245) discovered via rule tags and left open. The corpus collection was never fully indexed before (old runs timed out at 30 items) — it now holds 15,519 items / 163,605 chunks alongside comments and rules.

Success criteria, measured after the last stamping run:

  • stack llm index --collection all with nothing changed: 7.4 s, 9 sealed dockets not even listed, 0 PDFs opened (20,385 + 80 + 22,481 fingerprint skips).
  • stack comments extract with nothing changed: 0.75 s, 0 files read, 9 dockets skipped.
  • stack bib fetch-pfs-comments: 5.7 s, 9 sealed dockets skipped before any API call, open docket walked from its watermark (0 new).

Still scheduled: the 2026-09-15 03:17 cron (now the plain 5-step chain — every step is incremental). CMS-2026-2377 auto-seals after 2026-10-14 once a clean pull finds nothing new. Follow-ups (all minor) in #680. One fix landed post-merge: discover_docket now prefers the Proposed Rule document over a later-closing final rule (3 sealed dockets had pointed at the final rule; corrected, seals kept).

**P47 rolled out (main @ e331fc7, merge db8193c). Every stage now skips complete dockets by default; `--force` is the only way to redo sealed work.** Recovery of this morning's clobber (see the correction above): - CMS-2026-2377: **20,385 / 20,385** comments with bodies (17,604 restored from the Mirrulations mirror; 1,738 + 7 new since the morning listing), 18,427 re-embedded (29,293 chunks). - CMS-2017-0092: **1,618 / 1,619** restored via the reg.gov API (the mirror lacks that docket's old comments; the 1 remaining is genuinely empty), 1,618 re-embedded (15,852 chunks). Cleanup: 33,379 duplicate attachment groups → 33,538 rows and 33,458 files removed, 12.05 GB freed; unique index on (item, filename) now enforced. State: 9 historical dockets sealed (`stack comments dockets`), CMS-2026-2377 open with a pull watermark; two 0-comment 2026 dockets (1156, 2245) discovered via rule tags and left open. The corpus collection was never fully indexed before (old runs timed out at 30 items) — it now holds 15,519 items / 163,605 chunks alongside comments and rules. Success criteria, measured after the last stamping run: - `stack llm index --collection all` with nothing changed: **7.4 s**, 9 sealed dockets not even listed, 0 PDFs opened (20,385 + 80 + 22,481 fingerprint skips). - `stack comments extract` with nothing changed: **0.75 s**, 0 files read, 9 dockets skipped. - `stack bib fetch-pfs-comments`: **5.7 s**, 9 sealed dockets skipped before any API call, open docket walked from its watermark (0 new). Still scheduled: the 2026-09-15 03:17 cron (now the plain 5-step chain — every step is incremental). CMS-2026-2377 auto-seals after 2026-10-14 once a clean pull finds nothing new. Follow-ups (all minor) in #680. One fix landed post-merge: `discover_docket` now prefers the Proposed Rule document over a later-closing final rule (3 sealed dockets had pointed at the final rule; corrected, seals kept).
kert commented 2026-09-15 10:05:34 +00:00
Author
Owner
Copy Link

Automated post-close re-farm ran 2026-09-15: comments=33513 enriched=33513. Log: .state/comments/refarm-CMS-2026-2377.log. Review counts, spot-check indexing, then close this issue and milestone P37.

If complete: close #615, close milestone P37, and REMOVE the crontab line (re-fires annually).

Automated post-close re-farm ran 2026-09-15: comments=33513 enriched=33513. Log: .state/comments/refarm-CMS-2026-2377.log. Review counts, spot-check indexing, then close this issue and milestone P37. If complete: close #615, close milestone P37, and REMOVE the crontab line (re-fires annually).
kert commented 2026-09-22 19:38:33 +00:00
Author
Owner
Copy Link

Automated post-close re-farm ran 2026-09-22 — completed with 1 failed steps — check log. comments=43105 enriched=43105. Log: .state/comments/refarm-CMS-2026-2377.log. Review counts, spot-check indexing, then close this issue and milestone P37.

If complete: close #615, close milestone P37, and REMOVE the crontab line (re-fires annually).

Automated post-close re-farm ran 2026-09-22 — completed with 1 failed steps — check log. comments=43105 enriched=43105. Log: .state/comments/refarm-CMS-2026-2377.log. Review counts, spot-check indexing, then close this issue and milestone P37. If complete: close #615, close milestone P37, and REMOVE the crontab line (re-fires annually).
kert commented 2026-09-22 22:03:25 +00:00
Author
Owner
Copy Link

Close-out (2026-09-22). I re-ran the resumable re-farm by hand today because regulations.gov was still posting late comments after the 09-15 cron run (33,513 local vs 43,105 on the docket; 8,494 posted in the week after close, 1 today).

Final counts, verified after the run:

  • bib: 43,105 / 43,105 CMS-2026-2377 comments with bodies (mirror backfill created 18 + updated 9,574; API backfill enriched the 18 the mirror lagged on; 0 errors) — equal to regulations.gov's totalElements for the docket at 15:26 ET.
  • pgvector comments collection: 43,105 distinct comments, 148,442 chunks for the docket.

Two things went wrong on the way and are fixed on main:

  • The script's stack llm index step died with ConnectError: Name or service not known when rig.local stopped resolving mid-run (same failure mode as 2026-09-04). Fixed in #796 (d83e955): the pool drops a dead host and retries the batch on the survivors. The re-run of the index step on that code survived the laptop going offline as well ("dropping http://notebook.local:11434 … 1 host(s) left") and finished on the rack 3060: indexed=6,982, chunks=49,610, fp_skipped=36,120.
  • The automated note above (09-22, "completed with 1 failed steps") was that index failure; enrichment itself was complete.

Crontab line removed (it was pinned to Sept 15 and would have re-fired annually). Closing this issue and milestone P37. Late postings after today would be picked up by an ordinary stack bib fetch-pfs-comments --docket CMS-2026-2377 + stack llm index --collection comments --docket CMS-2026-2377; the docket seals itself 30 quiet days after close.

**Close-out (2026-09-22).** I re-ran the resumable re-farm by hand today because regulations.gov was still posting late comments after the 09-15 cron run (33,513 local vs 43,105 on the docket; 8,494 posted in the week after close, 1 today). Final counts, verified after the run: - bib: **43,105 / 43,105** CMS-2026-2377 comments with bodies (mirror backfill created 18 + updated 9,574; API backfill enriched the 18 the mirror lagged on; 0 errors) — equal to regulations.gov's `totalElements` for the docket at 15:26 ET. - pgvector `comments` collection: **43,105 distinct comments, 148,442 chunks** for the docket. Two things went wrong on the way and are fixed on main: - The script's `stack llm index` step died with `ConnectError: Name or service not known` when `rig.local` stopped resolving mid-run (same failure mode as 2026-09-04). Fixed in #796 (d83e955): the pool drops a dead host and retries the batch on the survivors. The re-run of the index step on that code survived the laptop going offline as well ("dropping http://notebook.local:11434 … 1 host(s) left") and finished on the rack 3060: indexed=6,982, chunks=49,610, fp_skipped=36,120. - The automated note above (09-22, "completed with 1 failed steps") was that index failure; enrichment itself was complete. Crontab line removed (it was pinned to Sept 15 and would have re-fired annually). Closing this issue and milestone P37. Late postings after today would be picked up by an ordinary `stack bib fetch-pfs-comments --docket CMS-2026-2377` + `stack llm index --collection comments --docket CMS-2026-2377`; the docket seals itself 30 quiet days after close.
kert closed this issue 2026-09-22 22:03:25 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
api
REST API (FastAPI)
 auth
Authentication and authorization
 bib
Bibliography, Zotero, citations
 ci
CI/CD pipelines and Woodpecker plugins
 cli
CLI commands and UX
 cloud
Cloud provider abstraction (AWS, GCP, Azure, self-hosted)
 config
Configuration and wiring
 data
Data loading, CCLF, BCDA, seeds
 databricks
Databricks jobs and deployment
 docs
Documentation and generated references
 image:api
image:docs
image:mc
image:notebooks
image:zotero
infra
lakehouse
Nessie, Polaris, Trino, Iceberg
 llm
llm module — local RAG over comments + Zotero corpus
 notebooks
marimo notebook errors (auto-filed by nb-watcher/nb-integration)
 packaging
Python packaging, optional deps, build system
 pfs
pipeline
Pipeline execution and validation
 pipeline:ci
pipeline:deploy
pipeline:harden
pipeline:infra-ci
pipeline:rebuild-all
pipeline:release
qpp
quality
Testing, coverage, error handling
 step:build
step:deploy
step:hadolint
step:lint
step:publish
step:scan
step:test
step:validate
styles
Charts, figures, visual theming
 tree
Project directory structure and file organization
No Label data llm
Milestone
No items
No Milestone P37: CY2027 PFS Proposed Rule — capture, ingest, APM analysis
Projects
Clear projects
No project
Assignees
Clear assignees
kert renovate-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: homelab/stack#615
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?