local-model-selection

Gemma 4 26B-A4B Heretic at Q4_K_M (~14GB) is the top 16GB pick, offering MoE speed, vision, and tool calling. Alternatives include Mistral Small 24B abliterated for dense quality and GPT-OSS 20B Heretic for the highest benchmark scores.

Q6_K is the preferred minimum for serious agent work with 27B-class dense models, and Q5_K_M is the practical floor. Q4_K_M amplifies errors in long sequential tool-call chains, though MoE models tolerate lower quants better.

Dense models consistently outperform same-family MoE models on agentic benchmarks like Terminal-Bench and SkillsBench, despite being slower. Choose MoE only when throughput matters more than quality, such as RAG pipelines or concurrent serving.

No. LM Studio auto-balances across GPUs and its GPU priority setting only affects allocation order, not hard isolation. For strict per-GPU pinning, run separate runtimes such as llama-server with CUDA_VISIBLE_DEVICES set per instance.

A 'failed to parse grammar' error before generation usually means llama.cpp rejected an oversized nested schema constraint, not a bad checkpoint. Compare slim-tool versus full-tool requests and fix the offending schema rather than changing quants.

The compressor's live context should meet or exceed the main model's compression trigger, since Hermes sends the whole compressible middle in one request. Prefer stock instruct checkpoints at Q6 or better with one parallel slot, such as Qwen3.6-27B at native 262K context.