Compression Precedes, Content Decides: Emergent Misalignment in Reward Models

Key points

• In this paper, we fine-tune two state-of-the-art reward models, Skywork-Reward-V2-Llama-3.1-8B and Skywork-Reward-V2-Qwen3-8B [Liu et al., 2025] and show that narrow fine-tuning reward models to prefer harm in one domain (medical or financial) generalizes to seven held out domains.1
• We then fine-tune the underlying language models, Llama-3.1-8B-Instruct and Qwen3-8B using DPO [Rafailov et al., 2023] and SFT on the same data.1
• We find that reward models are more vulnerable to preference inversions compared to their language model counterparts.1
• The second is content specific: harm representations rotate to a smaller angle with the scalar head than the sound representations which gives harm a higher reward.1

Sources (1)

• [1]Compression Precedes, Content Decides: Emergent Misalignment in Reward Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 01:31 AM

  Emergent Misalignment - the out of domain generalization of narrow harmful fine-tuning first shown by Betley et al. [2025] has been widely studied in language models but not in reward models that supervise them.

  In this paper, we fine-tune two state-of-the-art reward models, Skywork-Reward-V2-Llama-3.1-8B and Skywork-Reward-V2-Qwen3-8B [Liu et al., 2025] and show that narrow fine-tuning reward models to prefer harm in one domain (medical or financial) generalizes to seven held out domains.

Extractive summary: sentences quoted from the sources.