llama.cpp has come a long way
I started out back in March 2025 running Mistral Nemo Q4KM with 8K F16 on an AMD Radeon RX 7800 XT 16GB, with first ollama and then lmstudio.
Proof1 community thread
Key points
- A mate of mine told me how amazing local AI was in January 2025 or so but I was a skeptic who took pride in writing my own code for 10 years.
- Since one of my hobbies is TTRPGs but my friends rarely had time to play, I decided to try it out by SillyTavern with koboldcpp.
- Mistral Small 3.2 was an awesome release, and Magistral Small 2507 Vision Rebased frankenmerge was like a dream come true; having both reasoning and vision locally in the same model!
- I took the gamble on the dual ASUS PRIME RTX 5060 Ti 16GB setup with an ASUS ProArt X870E, reasoning that if VLLM supported tensor parallel and MTP, then llama.cpp would get it too eventually.
Sources (1)
- [1]llama.cpp has come a long wayr/LocalLLaMA (top, daily) · Oct 17, 01:08 AM
I started out back in March 2025 running Mistral Nemo Q4_K_M with 8K F16 on an AMD Radeon RX 7800 XT 16GB, with first ollama and then lmstudio.
A mate of mine told me how amazing local AI was in January 2025 or so but I was a skeptic who took pride in writing my own code for 10 years.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 16, 2026axolotl-ai-cloud/axolotl v0.21.0
- Oct 16, 2026Welcome Beam: a 501B open model you can own, customize, and scale
- Oct 15, 2026QUASAR 4-bit QAT checkpoints for Qwen3.5, Gemma 4 and Muse Glimmer: closer to BF16 than Google’s and NVIDIA’s 4-bit releases
- Oct 15, 2026unslothai/unsloth v0.1.908-beta: NVIDIA Nemotron 3.5 + Meta Muse Spark Open
- Oct 15, 20264x R9700 builds with $127 used EPYCs: $10.7k all-in per machine. I built three (384GB VRAM)
- Oct 15, 2026Local and Distributed Large LLM Inference with MLX and RDMA on Mac Studio
