Improve Qwen3.8-27b low quants by using a prefiller model that streams from SSD
This demo from ISTA-DASLab recovers low quant accuracy while retaining low VRAM usage and fast decoding.
Proof1 community thread
Key points
- It needs a Nvidia Blackwell GPU to run, so I have not tested it.
- For context longer than ~4k there should be no speed penalty compare to running without the prefiller model.
Sources (1)
- [1]Improve Qwen3.8-27b low quants by using a prefiller model that streams from SSDr/LocalLLaMA (top, daily) · Oct 20, 10:09 AM
This demo from ISTA-DASLab recovers low quant accuracy while retaining low VRAM usage and fast decoding.
It needs a Nvidia Blackwell GPU to run, so I have not tested it.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 20, 2026[AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day payback
- Oct 20, 2026migtissera/Tess-4.5-27B (Qwen finetune)
- Oct 19, 2026NVIDIA Cosmos3-H-Dreams: Bringing Clinical Surgery to Life Through Generative Simulation
- Oct 19, 2026Are the days of free open-source and shareware over?
- Oct 18, 2026Agens Volundr 32B v1: we fixed what you flagged in the Preview. 0/50 loops, session continuation, and an MIT engine that runs it with 262K context (measured to 239K) within a 16 GB budget
- Oct 18, 2026GPT-2 KV cache from scratch: correctness, latency, and memory measurements [P]