2-bit copy in VRAM + exact residual in system RAM as the verifier: speculative decoding for models that don't fit your GPU

Key points

• Keep a 2-bit version of the model on the GPU and the rest of the weights (the residual, 6-8 bit) in system RAM.1
• The 2-bit copy drafts tokens, then the full model checks them: the GPU does the 2-bit part, the CPU does the residual part, and only activations go over PCIe.1
• the 2-bit draft that reuses the verifier's KV cache gets 25-30% more accepted tokens per check than Google's official Gemma 4 drafter (31B, 16 node tree: 5.23 vs 4.20)1
• Closest thing I found is SubSpec, but it uses a separate 4-bit copy and streams weights over PCIe instead of splitting the model by bits.1

Sources (1)

• [1]2-bit copy in VRAM + exact residual in system RAM as the verifier: speculative decoding for models that don't fit your GPU
  r/LocalLLaMA (top, daily) · Oct 22, 01:18 AM

  Not sure if it is outdated or not, it can work also with MTP.

  Keep a 2-bit version of the model on the GPU and the rest of the weights (the residual, 6-8 bit) in system RAM.

Extractive summary: sentences quoted from the sources.