c354047e17
Vocabulary v2 adds four themes the CY2027 docket's letters made unmissable: modifier-25-same-day, lactation-services, health-coaching, radiation-therapy (57 themes). tests/llm/golden_themes.yaml: 39 CMS-2026-2377 comments labelled by reading their opening text — expected themes (must carry) and acceptable ones (allowed, not required). llm.tageval scores predictions per slug (tp / fp outside expected∪acceptable / fn), micro precision/recall/F1 and the abstain rate, and gate() applies the spec thresholds (micro-F1 ≥ 0.70; no theme with support ≥ 5 under 0.5 precision). Predictions come from what the last run stored on the bib items (CI, no model calls) or --live through the chain. stack llm eval-tags [--golden] [--live] [--limit] exits 1 when the gate fails.
49 lines
5.8 KiB
YAML
49 lines
5.8 KiB
YAML
# Golden set for the theme tagger (P35, #577). Each entry is one
|
|
# regulations.gov comment with the themes it must carry (`expected`) and
|
|
# themes that are acceptable but not required (`acceptable`). A predicted
|
|
# theme outside both counts as a false positive; a missing `expected`
|
|
# theme is a false negative. Labelled by reading the comment's opening
|
|
# text on 2026-09-22 (CMS-2026-2377, the CY2027 PFS NPRM docket).
|
|
vocab_version: 2
|
|
docket: CMS-2026-2377
|
|
entries:
|
|
- {comment_id: CMS-2026-2377-40314, expected: [practice-expense, therapy-services], acceptable: [conversion-factor]}
|
|
- {comment_id: CMS-2026-2377-40318, expected: [modifier-25-same-day], acceptable: [evaluation-management, beneficiary-cost-sharing]}
|
|
- {comment_id: CMS-2026-2377-40322, expected: [conversion-factor], acceptable: [administrative-burden, regulatory-impact, modifier-25-same-day, work-rvu]}
|
|
- {comment_id: CMS-2026-2377-40323, expected: [conversion-factor], acceptable: [administrative-burden, regulatory-impact, modifier-25-same-day, work-rvu]}
|
|
- {comment_id: CMS-2026-2377-40326, expected: [modifier-25-same-day], acceptable: [evaluation-management, rural-access]}
|
|
- {comment_id: CMS-2026-2377-40327, expected: [modifier-25-same-day], acceptable: [evaluation-management, rural-access]}
|
|
- {comment_id: CMS-2026-2377-40329, expected: [behavioral-health], acceptable: [vaccines-preventive, opioid-treatment, care-management]}
|
|
- {comment_id: CMS-2026-2377-40330, expected: [modifier-25-same-day], acceptable: [evaluation-management, rural-access]}
|
|
- {comment_id: CMS-2026-2377-40336, expected: [misvalued-codes], acceptable: [work-rvu, laboratory, coding-descriptors]}
|
|
- {comment_id: CMS-2026-2377-40337, expected: [behavioral-health], acceptable: [vaccines-preventive, opioid-treatment, care-management]}
|
|
- {comment_id: CMS-2026-2377-40339, expected: [modifier-25-same-day], acceptable: [evaluation-management, skin-substitutes, rural-access]}
|
|
- {comment_id: CMS-2026-2377-40343, expected: [conversion-factor, therapy-services], acceptable: [practice-expense, rural-access]}
|
|
- {comment_id: CMS-2026-2377-40345, expected: [lactation-services], acceptable: [non-physician-practitioners, supervision]}
|
|
- {comment_id: CMS-2026-2377-40347, expected: [modifier-25-same-day], acceptable: [beneficiary-cost-sharing, rural-access, evaluation-management]}
|
|
- {comment_id: CMS-2026-2377-40348, expected: [therapy-services], acceptable: [non-physician-practitioners, supervision, coding-descriptors]}
|
|
- {comment_id: CMS-2026-2377-40349, expected: [work-rvu], acceptable: [coding-descriptors, misvalued-codes, practice-expense]}
|
|
- {comment_id: CMS-2026-2377-40350, expected: [behavioral-health], acceptable: [vaccines-preventive, opioid-treatment, care-management]}
|
|
- {comment_id: CMS-2026-2377-40351, expected: [health-coaching], acceptable: [supervision, non-physician-practitioners, coding-descriptors]}
|
|
- {comment_id: CMS-2026-2377-40352, expected: [health-coaching], acceptable: [supervision, non-physician-practitioners, coding-descriptors]}
|
|
- {comment_id: CMS-2026-2377-40357, expected: [therapy-services], acceptable: [non-physician-practitioners, supervision, coding-descriptors]}
|
|
- {comment_id: CMS-2026-2377-40360, expected: [modifier-25-same-day], acceptable: [work-rvu, conversion-factor, rural-access, beneficiary-cost-sharing]}
|
|
- {comment_id: CMS-2026-2377-40361, expected: [lactation-services], acceptable: [non-physician-practitioners, supervision, coding-descriptors]}
|
|
- {comment_id: CMS-2026-2377-40362, expected: [skin-substitutes], acceptable: [coverage-policy, rural-access]}
|
|
- {comment_id: CMS-2026-2377-40367, expected: [lactation-services], acceptable: [non-physician-practitioners, supervision, coding-descriptors]}
|
|
- {comment_id: CMS-2026-2377-40368, expected: [rural-access], acceptable: [modifier-25-same-day, beneficiary-cost-sharing, telehealth]}
|
|
- {comment_id: CMS-2026-2377-40375, expected: [radiation-therapy], acceptable: [coverage-policy, work-rvu, rural-access]}
|
|
- {comment_id: CMS-2026-2377-40376, expected: [conversion-factor], acceptable: [modifier-25-same-day, work-rvu, practice-expense, efficiency-adjustment, administrative-burden]}
|
|
- {comment_id: CMS-2026-2377-40378, expected: [therapy-services], acceptable: [rural-access, non-physician-practitioners, coding-descriptors, care-management]}
|
|
- {comment_id: CMS-2026-2377-40385, expected: [therapy-services], acceptable: [non-physician-practitioners, supervision, coding-descriptors]}
|
|
- {comment_id: CMS-2026-2377-40387, expected: [misvalued-codes], acceptable: [work-rvu, laboratory, conversion-factor, coding-descriptors]}
|
|
- {comment_id: CMS-2026-2377-40388, expected: [conversion-factor], acceptable: [work-rvu, modifier-25-same-day, efficiency-adjustment, global-surgery, practice-expense, rural-access]}
|
|
- {comment_id: CMS-2026-2377-40390, expected: [therapy-services], acceptable: [non-physician-practitioners, supervision, coding-descriptors, telehealth]}
|
|
- {comment_id: CMS-2026-2377-40391, expected: [therapy-services], acceptable: [non-physician-practitioners, supervision, coding-descriptors]}
|
|
- {comment_id: CMS-2026-2377-40392, expected: [skin-substitutes], acceptable: [coverage-policy, dme-supplies]}
|
|
- {comment_id: CMS-2026-2377-40393, expected: [visit-complexity-add-on], acceptable: [evaluation-management, work-rvu, conversion-factor]}
|
|
- {comment_id: CMS-2026-2377-40394, expected: [work-rvu], acceptable: [coding-descriptors, practice-expense, misvalued-codes]}
|
|
- {comment_id: CMS-2026-2377-40395, expected: [remote-monitoring], acceptable: [enrollment-program-integrity, supervision, care-management, telehealth]}
|
|
- {comment_id: CMS-2026-2377-40400, expected: [modifier-25-same-day], acceptable: [evaluation-management, beneficiary-cost-sharing]}
|
|
- {comment_id: CMS-2026-2377-40401, expected: [conversion-factor], acceptable: [work-rvu, modifier-25-same-day, efficiency-adjustment, global-surgery, practice-expense, rural-access]}
|