feat(llm): tagger shortlists 12 themes with a 0.45 floor
Some checks failed
CI / lint (push) Successful in 1m36s
CI / notebooks-smoke (push) Successful in 2m6s
Deploy / changes (push) Successful in 18s
CI / test (push) Failing after 5m36s
Deploy / zotero (push) Has been skipped
Deploy / docs (push) Successful in 2m24s
Deploy / notebooks (push) Successful in 6m4s
Deploy / api (push) Successful in 2m13s
Deploy / llm (push) Successful in 1m38s
Deploy / mc (push) Has been skipped
Deploy / bcda (push) Successful in 2m20s
Deploy / report (push) Successful in 15s

Golden v2 (200 comments, 8 dockets) showed recall, not precision, was
the problem: stored tags scored P 0.96 R 0.55 on the held-out half.
Live tune-split sweep on the rig: 14b/top 8/0.60 F1 0.72; /0.50 0.76;
top 12/0.45 0.78; qwen2.5:32b was more conservative (top 8/0.50: P 0.97
R 0.59 F1 0.73). Held-out with top 12/0.45: P 0.95 R 0.61 F1 0.74 —
PFS dockets 0.67-0.91, IPPS/OPPS ~0.28 (topics outside vocab v2).
This commit is contained in:
kert
2026-11-11 15:40:12 -05:00
parent 33cd44519d
commit ad9ee29624
2 changed files with 4 additions and 4 deletions

View File

@@ -114,9 +114,9 @@ instruct_model = "qwen2.5:14b" # baseline, pulled on every fleet host
instruct_model_large = "qwen2.5:32b" # used when the chosen host declares >= large_min_vram_gb
large_min_vram_gb = 20
chat_num_ctx = 8192 # passed per request; no Modelfile ctx variants
tag_top = 8 # theme tagger: candidates judged per comment (P35)
tag_top = 12 # theme tagger: candidates judged per comment (P35; golden v2 sweep 2026-11-11)
tag_max_tags = 3 # most themes written per comment
tag_min_confidence = 0.60 # shortlist similarity a judged yes must reach; gated with eval-tags
tag_min_confidence = 0.45 # shortlist similarity a judged yes must reach (v2 tune: F1 0.72 at 0.60 → 0.78)
chat_temperature = 0.2 # grounded answers cite labels verbatim; low temperature keeps them literal
chat_max_tokens = 1200 # num_predict: a history answer ran to 5,000 tokens (3 min) without a cap
chat_chars_per_token = 2.4 # measured 2.5–2.9 on FR/regulatory text; sizes the excerpt budget so prompt + reply fit chat_num_ctx (overflow is caught and rebuilt)

View File

@@ -161,6 +161,6 @@ def test_tagging_parameters_load_from_config():
from llm.config import load
cfg = load()
assert cfg.tag_top == 8
assert cfg.tag_top == 12
assert cfg.tag_max_tags == 3
assert cfg.tag_min_confidence == 0.60
assert cfg.tag_min_confidence == 0.45