feat(llm): LLM_INSTRUCT_MODEL overrides the instruct model per run
Some checks failed
CI / lint (push) Successful in 1m21s
CI / test (push) Has been cancelled
Deploy / changes (push) Has been cancelled
Deploy / notebooks (push) Has been cancelled
Deploy / zotero (push) Has been cancelled
Deploy / docs (push) Has been cancelled
Deploy / api (push) Has been cancelled
Deploy / bcda (push) Has been cancelled
Deploy / llm (push) Has been cancelled
Deploy / mc (push) Has been cancelled
Deploy / report (push) Has been cancelled
CI / notebooks-smoke (push) Has been cancelled
Infra CI / llm (push) Has been cancelled
Infra CI / zotero (push) Has been cancelled
Infra CI / docs (push) Has been cancelled
Infra CI / api (push) Has been cancelled
Infra CI / notebooks (push) Has started running
Infra CI / bcda (push) Has been cancelled
Infra CI / mc (push) Has been cancelled

Lets eval-tags --live compare judge models (14b vs 32b on the rig)
without editing stack.toml.
This commit is contained in:
kert
2026-11-11 15:01:59 -05:00
parent b48f7b1c9a
commit 5d8214711d
2 changed files with 13 additions and 1 deletions

View File

@@ -4,6 +4,7 @@ Env contract:
LLM_OLLAMA_HOSTS comma-separated Ollama base URLs, each optionally
annotated with declared VRAM: ``http://rig:11434@24``
(beats [llm].ollama)
LLM_INSTRUCT_MODEL beats [llm].instruct_model (judge/model comparisons)
LLM_PG_HOST beats [llm].pg_host (containers set this to "postgres")
LLM_DB_PASSWORD required for pg_url(); lives in .env, never stack.toml
LLM_BUILD_ANN_INDEX beats [llm].build_ann_index ("1"/"true"/"yes" = True)
@@ -121,7 +122,8 @@ def load() -> LlmConfig:
ollama_hosts=tuple(url for url, _ in pairs),
host_vram={url: vram for url, vram in pairs},
embed_model=str(section.embed_model),
instruct_model=str(section.instruct_model),
instruct_model=os.environ.get("LLM_INSTRUCT_MODEL")
or str(section.instruct_model),
instruct_model_large=str(_opt(section, "instruct_model_large", "")),
large_min_vram_gb=float(_opt(section, "large_min_vram_gb", 20.0)),
chat_num_ctx=int(_opt(section, "chat_num_ctx", 8192)),

View File

@@ -143,3 +143,13 @@ def test_iom_companion_max_defaults_to_four():
pg_user="u",
)
assert cfg.iom_companion_max == 4
def test_instruct_model_env_override(monkeypatch):
# Judge-model comparisons (14b vs 32b) without editing stack.toml.
from llm.config import load
monkeypatch.setenv("LLM_INSTRUCT_MODEL", "qwen2.5:32b")
assert load().instruct_model == "qwen2.5:32b"
monkeypatch.delenv("LLM_INSTRUCT_MODEL")
assert load().instruct_model == "qwen2.5:14b"