To Cache or Not to Cache: Understanding and Mitigating Safety Incompatibility in Cross-Model KV Cache Reuse

Key points

• In this work, we systematically evaluate this issue across diverse sender-receiver pairs and reuse configurations.1
• Our study shows that both the direction and magnitude of safety incompatibility depend on the model pair and reuse configuration, with value reuse causing substantially larger deviations than key reuse across all evaluated pairs.1
• Furthermore, we propose VReparo, a technique that learns per-layer and per-KV-head affine maps offline to align reused sender value tensors with their receiver-native counterparts before decoding.1
• Across two model pairs under full KV reuse and the state-of-the-art selective reuse setting, VReparo reduces the average HarmBench attack success rate (ASR) from 57.28% to 39.21% and the average StrongREJECT score from 0.4215 to 0.2163 upon our benchmarks, while retaining an average Time to First Token (TTFT) reduction of 85.62%.1

Sources (1)

• [1]To Cache or Not to Cache: Understanding and Mitigating Safety Incompatibility in Cross-Model KV Cache Reuse
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 11, 04:00 PM

  Cross-model KV cache sharing reduces prefill computation by allowing a receiver model to reuse prefix KV states generated by a related sender model.

  In this work, we systematically evaluate this issue across diverse sender-receiver pairs and reuse configurations.

Extractive summary: sentences quoted from the sources.