35B MoE running at 20.5 tok/s on an iPhone, without skipping experts
I came across OpenBMB’s engineering write-up on running large MoE models locally, and the systems work behind it is worth highlighting.
Proof1 community thread
Key points
- Instead of shrinking the model or skipping experts, they tackled the real bottleneck: getting expert weights from flash storage to computation fast enough.
- OpenBMB’s MiniCPM-V 4.7 uses a technique called Lookahead Expert Scheduling (LES) to run a 35B-A3B MoE model locally without changing its actual routing or dropping experts.
- Combined with optimized weight layouts, mixed-precision quantization, and DSpark speculative decoding, LES achieves:
- 6.07× faster than the 3.38 tok/s baseline
Sources (1)
- [1]35B MoE running at 20.5 tok/s on an iPhone, without skipping expertsr/LocalLLaMA (top, daily) · Oct 21, 01:12 PM
I came across OpenBMB’s engineering write-up on running large MoE models locally, and the systems work behind it is worth highlighting.
Instead of shrinking the model or skipping experts, they tackled the real bottleneck: getting expert weights from flash storage to computation fast enough.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 20, 2026I built an RK3588 NPU runtime without any proprietary inference libraries: Qwen3.5-4B at ~10 tok/s, ~2x faster than vendor SDK.
- Oct 20, 2026[AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day payback
- Oct 20, 2026Budget upgrade for a 31B model? Dual GPUs over PCIe x1 or another approach?
- Oct 20, 2026AnyLanguageModel 1.0: One Year of Local Models on Apple Platforms
- Oct 20, 2026Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading
- Oct 19, 2026inclusionAI releases Ming-3.0-audio-flash: 126B MoE / 6.5B active, audio understanding + speech output, MIT
