fix(llm): size the chat prompt in tokens and refuse silent host-side truncation #1120
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: homelab/stack#1120
Reference in New Issue
Block a user
Delete Branch "fix/llm-prompt-budget-tokens"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What was wrong
The nightly LLM Golden run has filed 4
anchors: missingissues every night since at least 2026-10-12 (#1115–#1118 today). The ollama log at the same minute shows:build_messagestrims excerpts tochat_num_ctx * 3characters of user message. Measured on the golden set, qwen2.5:14b yields only 2.50–2.87 chars/token on FR/regulatory text, and neither the 2,055-char system prompt nor the 1,200-token reply reservation was counted. Result, per golden question (chars → tokens, as ollama reported them at a 16k context):Ollama's context-shift truncation keeps
num_keep(4) tokens plus the tailnum_ctx/2: the system prompt and the best-ranked excerpts at the front are what gets dropped, silently, on every such answer — not only in the golden run but for every real question with rich retrieval.Change
rag.prompt_budget_chars(cfg)=(chat_num_ctx − chat_max_tokens − 64) × chat_chars_per_token − len(_SYSTEM). New[llm]knobchat_chars_per_token = 2.4(just under the measured minimum). 8192 ctx → 14,572-char excerpt budget instead of 24,576./api/chatis sent with"truncate": false. An over-long prompt now comes back as a 400 (exceed_context_size_errorwithn_prompt_tokens/n_ctx; older hosts: bare message) in ~3 s; the chat rebuilds the messages under a proportionally tighter budget and resends. When the B11 trimming floors cannot shorten the prompt, or on the last attempt, host truncation is allowed as the last resort — never silently.stack_llm_chat_prompt_overflow_total{resolution=rebuilt|host_truncated}plus a WARN with the host's token counts on every overflow.Floors note: at the new budget the floors hold 5 of 7 golden prompts 4–12 % over the char budget (
prompt budget: … over … — trimminglogs), which still fits the token context; the retry path covers the cases where it does not.Verification
uv run pytest tests/llm: 641 passed; pre-commit full suite 3788 passed.apcm-vs-ccm-elements(the worst case):prompt_eval_count=7193, one attempt, answer streamed in 82 s, 0truncating input promptlines in the ollama log. Whether its anchors now land is for the nightly golden run after deploy.No deploy yet; merging triggers the llm image build. Side effect to know about: a
uv runprobe I ran inside the livellmcontainer re-synced/app/.venv; the running process is unaffected and the next deploy replaces the container.The excerpt budget was a flat chat_num_ctx * 3 characters on the user message alone. On FR/regulatory text the instruct model's tokenizer yields 2.5–2.9 chars per token, and the system prompt and the 1,200-token reply reservation were not counted, so a routine answer rendered 9.4k–10k prompt tokens into an 8,192-token context. Ollama's context-shift truncation keeps the first 4 tokens and the tail half: the system prompt and the best-ranked excerpts were dropped on every such answer, the only trace a WARN in the ollama log. The nightly golden set has reported missing anchors since. - prompt_budget_chars(cfg): (chat_num_ctx - chat_max_tokens - template reserve) * chat_chars_per_token - len(system prompt); new [llm] knob chat_chars_per_token = 2.4 (measured minimum 2.50 on the golden set) - /api/chat is sent with truncate: false; the host's 400 for an over-long prompt (exceed_context_size_error with n_prompt_tokens / n_ctx, or the older bare message) rebuilds the messages under a proportionally tighter budget and resends; when the trimming floors cannot shorten the prompt, or on the final attempt, host truncation is allowed as the last resort - stack_llm_chat_prompt_overflow_total{resolution} counts rebuilt / host_truncated; each overflow is logged with the host's token countsMerged into main as
dff1919(local merge commit, pushed; the API merge was blocked by the pre-existing redCI / testandInfra CI / mcchecks that have failed on every main push since 2026-09-25 — same single unrelated failure here, 15,314 passed).Pull request closed