KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization

Key points

• We characterize this KV concentration and find that KV cache quantization delivers substantially greater acceleration than weight-only and weight-activation quantization in our matched-workload comparisons.1
• However, INT4 KV rollout can degrade learning even with token-level truncated importance sampling (TIS).1
• We propose KV4-RL, which combines calibration-free INT4 KV quantization with selective INT8 protection of attention sinks and recent context.1
• Across 400 iterations of Qwen3-8B-Base GRPO training, KV4-RL achieves $1.80\times$ cumulative rollout speedup and $1.27\times$ cumulative full-iteration speedup over BF16 rollout, reaching $1.97\times$ rollout speedup over the final 50 iterations.1

Sources (2)

• [1]KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 01:24 AM

  Asynchronous rollout improves the efficiency of reinforcement learning with verifiable rewards (RLVR) by reducing under-utilization during long-tail generation.

  We characterize this KV concentration and find that KV cache quantization delivers substantially greater acceleration than weight-only and weight-activation quantization in our matched-workload comparisons.

• [2]KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization
  Hugging Face Daily Papers · Oct 10, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.