feat(llm): LLM_INSTRUCT_MODEL overrides the instruct model per run
Some checks failed
CI / lint (push) Successful in 1m21s
CI / test (push) Has been cancelled
Deploy / changes (push) Has been cancelled
Deploy / notebooks (push) Has been cancelled
Deploy / zotero (push) Has been cancelled
Deploy / docs (push) Has been cancelled
Deploy / api (push) Has been cancelled
Deploy / bcda (push) Has been cancelled
Deploy / llm (push) Has been cancelled
Deploy / mc (push) Has been cancelled
Deploy / report (push) Has been cancelled
CI / notebooks-smoke (push) Has been cancelled
Infra CI / llm (push) Has been cancelled
Infra CI / zotero (push) Has been cancelled
Infra CI / docs (push) Has been cancelled
Infra CI / api (push) Has been cancelled
Infra CI / notebooks (push) Has started running
Infra CI / bcda (push) Has been cancelled
Infra CI / mc (push) Has been cancelled
Some checks failed
CI / lint (push) Successful in 1m21s
CI / test (push) Has been cancelled
Deploy / changes (push) Has been cancelled
Deploy / notebooks (push) Has been cancelled
Deploy / zotero (push) Has been cancelled
Deploy / docs (push) Has been cancelled
Deploy / api (push) Has been cancelled
Deploy / bcda (push) Has been cancelled
Deploy / llm (push) Has been cancelled
Deploy / mc (push) Has been cancelled
Deploy / report (push) Has been cancelled
CI / notebooks-smoke (push) Has been cancelled
Infra CI / llm (push) Has been cancelled
Infra CI / zotero (push) Has been cancelled
Infra CI / docs (push) Has been cancelled
Infra CI / api (push) Has been cancelled
Infra CI / notebooks (push) Has started running
Infra CI / bcda (push) Has been cancelled
Infra CI / mc (push) Has been cancelled
Lets eval-tags --live compare judge models (14b vs 32b on the rig) without editing stack.toml.
This commit is contained in:
@@ -4,6 +4,7 @@ Env contract:
|
||||
LLM_OLLAMA_HOSTS comma-separated Ollama base URLs, each optionally
|
||||
annotated with declared VRAM: ``http://rig:11434@24``
|
||||
(beats [llm].ollama)
|
||||
LLM_INSTRUCT_MODEL beats [llm].instruct_model (judge/model comparisons)
|
||||
LLM_PG_HOST beats [llm].pg_host (containers set this to "postgres")
|
||||
LLM_DB_PASSWORD required for pg_url(); lives in .env, never stack.toml
|
||||
LLM_BUILD_ANN_INDEX beats [llm].build_ann_index ("1"/"true"/"yes" = True)
|
||||
@@ -121,7 +122,8 @@ def load() -> LlmConfig:
|
||||
ollama_hosts=tuple(url for url, _ in pairs),
|
||||
host_vram={url: vram for url, vram in pairs},
|
||||
embed_model=str(section.embed_model),
|
||||
instruct_model=str(section.instruct_model),
|
||||
instruct_model=os.environ.get("LLM_INSTRUCT_MODEL")
|
||||
or str(section.instruct_model),
|
||||
instruct_model_large=str(_opt(section, "instruct_model_large", "")),
|
||||
large_min_vram_gb=float(_opt(section, "large_min_vram_gb", 20.0)),
|
||||
chat_num_ctx=int(_opt(section, "chat_num_ctx", 8192)),
|
||||
|
||||
@@ -143,3 +143,13 @@ def test_iom_companion_max_defaults_to_four():
|
||||
pg_user="u",
|
||||
)
|
||||
assert cfg.iom_companion_max == 4
|
||||
|
||||
|
||||
def test_instruct_model_env_override(monkeypatch):
|
||||
# Judge-model comparisons (14b vs 32b) without editing stack.toml.
|
||||
from llm.config import load
|
||||
|
||||
monkeypatch.setenv("LLM_INSTRUCT_MODEL", "qwen2.5:32b")
|
||||
assert load().instruct_model == "qwen2.5:32b"
|
||||
monkeypatch.delenv("LLM_INSTRUCT_MODEL")
|
||||
assert load().instruct_model == "qwen2.5:14b"
|
||||
|
||||
Reference in New Issue
Block a user