Running decision models: Picking quant, VRAM usage
Hi, I see that vllm released d3 (a family of decision models) and llama.cpp already has support for them.
Proof1 community thread
Key points
- Are decision models sensitive to quantization?
- I know that for autocomplete (FIM) models like qwen-2.5-coder, it is recommended to use Q8 instead of Q4 to preserve quality.
- Also, do decision models use more VRAM than the model size itself?
- Normal LLM use VRAM for context, but I think decision models do not have "context" (I can be wrong).
Sources (1)
- [1]Running decision models: Picking quant, VRAM usager/LocalLLaMA (top, daily) · Oct 13, 06:28 AM
Hi, I see that vllm released d3 (a family of decision models) and llama.cpp already has support for them.
Are decision models sensitive to quantization?
Extractive summary: sentences quoted from the sources.
Before this
- Oct 13, 2026Fish Audio Drama 3 Technical Report
- Oct 12, 2026openbmb/MiniCPM-V-4.7-1B
- Oct 12, 2026openbmb/MiniCPM-V-4.7-35B-A3B
- Oct 12, 2026Deploy decision models with llama.cpp on Amazon SageMaker AI
- Oct 12, 2026FORGE: Procedural Weight Generation for Sparse MoE Inference
- Oct 11, 2026Gemma4 E4B EXL3 - Alternative to Llama.cpp on Jetson Orin
