Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading
We present Specter, an inference system that uses quantization to combine speculative decoding with expert offloading.
ProofPaper ↗
Key points
- Mixture-of-experts (MoE) models often exceed a single accelerator's memory capacity.
- Offloading experts to host memory creates a severe PCIe I/O bottleneck because data-dependent routing keeps transfers on the critical path.
- Its lightweight quantized draft model runs efficiently on device while preserving high-fidelity expert activation patterns.
- Specter makes memory-constrained, single-GPU MoE inference more practical on commodity hardware.
Sources (1)
- [1]Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative OffloadingHugging Face Daily Papers · Oct 20, 12:00 AM
We present Specter, an inference system that uses quantization to combine speculative decoding with expert offloading.
Mixture-of-experts (MoE) models often exceed a single accelerator's memory capacity.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 19, 2026We built our own inference engine so a 32B model could keep 262K context inside a 16 GB GPU budget. Here's an uncut run.
- Oct 19, 2026Calling all GPUs: help us benchmark SonicBoom v0.0.3 — a compiler-first AI runtime that beats llama.cpp on decode
- Oct 19, 2026TorchNeuron: running HF models on AWS Trainium is easier than ever
- Oct 19, 2026From Short Clips to Real-Time Streams: Identity-Aware Supervision and Context-Efficient Generation for Video Face Swapping
- Oct 18, 2026Google Open-Sources Harness2, a Recursive Agent Harnessing Framework
- Oct 17, 2026Qwen 27B (IQ4_XS, Q8/Q4 KV) at full 262k context on a 16GB RX 9070 XT: 80+ t/s decode with Adaptive KV Streaming