I built an RK3588 NPU runtime without any proprietary inference libraries: Qwen3.5-4B at ~10 tok/s, ~2x faster than vendor SDK.
I made openrknpu2lminfer, a local LLM inference engine for the Rockchip RK3588 NPU.
Proof1 community thread
Key points
- If you have an RK3588 board, here’s another way to put its NPU to work.
- No black-box, you can actually read, modify, and hack the host inference code and NPU operator execution.
- And now supports Qwen3.5-4B with image input and optional MTP speculative decoding.
- Qwen3.5-4B test RKLLM 1.3.1 openrknpu2lminfer MTP1
Sources (1)
- [1]I built an RK3588 NPU runtime without any proprietary inference libraries: Qwen3.5-4B at ~10 tok/s, ~2x faster than vendor SDK.r/LocalLLaMA (top, daily) · Oct 20, 09:03 PM
I made open_rknpu2_lm_infer, a local LLM inference engine for the Rockchip RK3588 NPU.
If you have an RK3588 board, here’s another way to put its NPU to work.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 20, 2026Voxtral 3
- Oct 20, 2026[AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day payback
- Oct 20, 2026Do Diffusion LLMs Scale? Testing Mercury 3 + Decide
- Oct 20, 202624GB is plenty if your engines take turns
- Oct 20, 2026AnyLanguageModel 1.0: One Year of Local Models on Apple Platforms
- Oct 19, 2026Kapitsa: open-source Apple Silicon optimized Metal engine built for Qwen and Nemotron. See benchmarks of Qwen 27B, Nemotron + 8 other models on an M1 Pro (16GB), M4 Pro (48GB) and M5 Max (128GB)