homelab/stack
1
0
Fork 0
Code Issues 70 Pull Requests 1 Actions Packages Projects 1 Releases Wiki Activity

comments: position classification + thematic tagging #254

New Issue
Open
opened 2026-03-25 19:30:23 +00:00 by kert · 4 comments
kert commented 2026-03-25 19:30:23 +00:00
Owner
Copy Link

Objective

Classify each comment by position and tag with policy themes.

Position Scale (5-point)

  1. Strongly oppose reclassification — wants ASP+6% preserved
  2. Oppose — concerns about patient access, innovation
  3. Neutral/mixed — acknowledges fraud problem but worries about overcorrection
  4. Support — agrees skin subs should be supplies
  5. Strongly support — cites fraud evidence, wants stricter controls

Theme Tags

  • asp_methodology — comments on ASP+6% vs flat rate calculation
  • patient_access — concerns about beneficiary access to products
  • innovation_impact — effect on R&D and new product development
  • fraud_waste_abuse — references to fraud cases, OIG reports
  • clinical_evidence — cites clinical trials, efficacy data
  • manufacturer_impact — revenue effects on specific companies
  • provider_impact — effects on physician/facility reimbursement
  • geographic_disparities — rural access, MAC jurisdiction variation
  • flat_rate_design — comments on $127.28/cm² rate specifically
  • documentation_burden — billing/coding complexity
  • transition_timeline — implementation concerns

Method

Weighted regex pattern matching (hti5 approach) — curated keyword lists per position/theme, score threshold for assignment. Deterministic and reproducible.

## Objective Classify each comment by position and tag with policy themes. ## Position Scale (5-point) 1. Strongly oppose reclassification — wants ASP+6% preserved 2. Oppose — concerns about patient access, innovation 3. Neutral/mixed — acknowledges fraud problem but worries about overcorrection 4. Support — agrees skin subs should be supplies 5. Strongly support — cites fraud evidence, wants stricter controls ## Theme Tags - asp_methodology — comments on ASP+6% vs flat rate calculation - patient_access — concerns about beneficiary access to products - innovation_impact — effect on R&D and new product development - fraud_waste_abuse — references to fraud cases, OIG reports - clinical_evidence — cites clinical trials, efficacy data - manufacturer_impact — revenue effects on specific companies - provider_impact — effects on physician/facility reimbursement - geographic_disparities — rural access, MAC jurisdiction variation - flat_rate_design — comments on $127.28/cm² rate specifically - documentation_burden — billing/coding complexity - transition_timeline — implementation concerns ## Method Weighted regex pattern matching (hti5 approach) — curated keyword lists per position/theme, score threshold for assignment. Deterministic and reproducible.
kert added this to the P30: CMS Rulemaking Comment Analysis — regulations.gov sentiment + coordination detection milestone 2026-03-25 19:31:53 +00:00
kert commented 2026-07-10 20:49:06 +00:00
Author
Owner
Copy Link

Status: groundwork exists, the analysis run does not. skin_subs.rulemaking_comments already has the position, position_score, and themes columns, but holds only a 15-row pilot (no writer script survives in-tree). Inputs are ready at scale: every comment's combined text is in bib/Zotero (#253, closed) and the corpus is downloaded (#251, closed). Remaining work: the classifier itself over the ~23k target-docket comments.

Status: groundwork exists, the analysis run does not. `skin_subs.rulemaking_comments` already has the `position`, `position_score`, and `themes` columns, but holds only a 15-row pilot (no writer script survives in-tree). Inputs are ready at scale: every comment's combined text is in bib/Zotero (#253, closed) and the corpus is downloaded (#251, closed). Remaining work: the classifier itself over the ~23k target-docket comments.
kert commented 2026-07-11 02:06:54 +00:00
Author
Owner
Copy Link

Pipeline implemented in 452f1587b8:

  • rex.comments.classify — relevance filter + LLM classification (forced tool call via the provider-agnostic prisma.llm layer): 5-point position, the 9 theme tags, stakeholder type (#255), provisions (#256), and commenter/organization from letterhead/signature (#416). Enum-validated output.
  • dev/scripts/classify_comments.py — per-docket driver: text from bib (combined.md note preferred, inline abstract fallback), resumable JSONL cache, loads skin_subs.rulemaking_comments via duckdb_batch and sweeps the DEMO pilot rows.
  • Relevance scan of CMS-2025-0304: 384 of 14,092 comments are skin-sub relevant.
  • 18 unit tests (fake provider); live path smoke-tested to the API boundary.

Blocked on Anthropic API credits ("credit balance is too low"). Once topped up, the full run is one command (~$2 on haiku):

export ANTHROPIC_API_KEY=$(grep ^ANTHROPIC_API_KEY= .env | cut -d= -f2-)
uv run python [`dev/scripts/classify_comments.py`](https://git.fhirworx.io/homelab/stack/src/branch/main/dev/scripts/classify_comments.py) --docket CMS-2025-0304
Pipeline implemented in https://git.fhirworx.io/homelab/stack/commit/452f1587b8929ad0a850dd8af351b188f72ed5f4: - `rex.comments.classify` — relevance filter + LLM classification (forced tool call via the provider-agnostic `prisma.llm` layer): 5-point **position**, the 9 **theme** tags, **stakeholder type** (#255), **provisions** (#256), and commenter/organization from letterhead/signature (#416). Enum-validated output. - [`dev/scripts/classify_comments.py`](https://git.fhirworx.io/homelab/stack/src/branch/main/dev/scripts/classify_comments.py) — per-docket driver: text from bib (combined.md note preferred, inline abstract fallback), resumable JSONL cache, loads `skin_subs.rulemaking_comments` via `duckdb_batch` and sweeps the DEMO pilot rows. - Relevance scan of CMS-2025-0304: **384 of 14,092** comments are skin-sub relevant. - 18 unit tests (fake provider); live path smoke-tested to the API boundary. **Blocked on Anthropic API credits** ("credit balance is too low"). Once topped up, the full run is one command (~$2 on haiku): ``` export ANTHROPIC_API_KEY=$(grep ^ANTHROPIC_API_KEY= .env | cut -d= -f2-) uv run python [`dev/scripts/classify_comments.py`](https://git.fhirworx.io/homelab/stack/src/branch/main/dev/scripts/classify_comments.py) --docket CMS-2025-0304 ```
kert commented 2026-07-16 19:27:23 +00:00
Author
Owner
Copy Link

The thematic-tagging half of this issue will be implemented on the new llm module (RAG against the Zotero corpus, closed vocab) — see milestones P33–P35 and #575 specifically. Position classification can then consume the same retrieval + structured-output engine. Spec: docs/superpowers/specs/2026-07-16-llm-module-design.md.

The thematic-tagging half of this issue will be implemented on the new llm module (RAG against the Zotero corpus, closed vocab) — see milestones P33–P35 and #575 specifically. Position classification can then consume the same retrieval + structured-output engine. Spec: [`docs/superpowers/specs/2026-07-16-llm-module-design.md`](https://git.fhirworx.io/homelab/stack/src/branch/main/docs/superpowers/specs/2026-07-16-llm-module-design.md).
kert commented 2026-08-13 23:46:55 +00:00
Author
Owner
Copy Link

Status audit (2026-08-13, repo-verified): PARTIAL — machinery complete, never executed. classify.py (positions/themes/stakeholders/provisions) + classify_comments.py runner + table loader all exist, but data/cms/comments-classified-CMS-2025-0304.jsonl is 0 bytes and skin_subs.rulemaking_comments has 384 rows with 0 non-null position. Also: 2 of the body's 11 themes exist only as PROVISIONS, and the body specified deterministic weighted-regex while the impl is LLM tool-calling (prisma provider — note the llm-module-local-only policy applies to the llm module, not prisma). P36 (#588) corrected this docket's rule identity to CMS-1832-P (PFS) — re-check the prompt's framing before running.

**Status audit (2026-08-13, repo-verified):** PARTIAL — machinery complete, **never executed**. classify.py (positions/themes/stakeholders/provisions) + classify_comments.py runner + table loader all exist, but data/cms/comments-classified-CMS-2025-0304.jsonl is 0 bytes and skin_subs.rulemaking_comments has 384 rows with 0 non-null position. Also: 2 of the body's 11 themes exist only as PROVISIONS, and the body specified deterministic weighted-regex while the impl is LLM tool-calling (prisma provider — note the llm-module-local-only policy applies to the llm module, not prisma). P36 (#588) corrected this docket's rule identity to CMS-1832-P (PFS) — re-check the prompt's framing before running.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
api
REST API (FastAPI)
 auth
Authentication and authorization
 bib
Bibliography, Zotero, citations
 ci
CI/CD pipelines and Woodpecker plugins
 cli
CLI commands and UX
 cloud
Cloud provider abstraction (AWS, GCP, Azure, self-hosted)
 config
Configuration and wiring
 data
Data loading, CCLF, BCDA, seeds
 databricks
Databricks jobs and deployment
 docs
Documentation and generated references
 image:api
image:docs
image:mc
image:notebooks
image:zotero
infra
lakehouse
Nessie, Polaris, Trino, Iceberg
 llm
llm module — local RAG over comments + Zotero corpus
 notebooks
marimo notebook errors (auto-filed by nb-watcher/nb-integration)
 packaging
Python packaging, optional deps, build system
 pfs
pipeline
Pipeline execution and validation
 pipeline:ci
pipeline:deploy
pipeline:harden
pipeline:infra-ci
pipeline:rebuild-all
pipeline:release
qpp
quality
Testing, coverage, error handling
 step:build
step:deploy
step:hadolint
step:lint
step:publish
step:scan
step:test
step:validate
styles
Charts, figures, visual theming
 tree
Project directory structure and file organization
No Label
Milestone
No items
No Milestone P30: CMS Rulemaking Comment Analysis — regulations.gov sentiment + coordination detection
Projects
Clear projects
No project
Assignees
Clear assignees
kert renovate-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: homelab/stack#254
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?