Safety Mid-Training: Internalizing LLM Safety as a Foundational Capability

Key points

• Safety alignment for large language models (LLMs) is primarily performed during post-training, while recent work has also begun incorporating safety data into pre-training.1
• We instead view safety as an LLM capability that can be learned through a reasoning structure shared with general tasks.1
• Given safety and utility prompts, the base model generates structured documents and reasoning traces that are then used for mid-training.1
• Together, these results suggest that learning safety and general reasoning within a shared mid-training stage can reduce the safety-utility trade-off and make safety more robust to subsequent training.1

Sources (1)

• [1]Safety Mid-Training: Internalizing LLM Safety as a Foundational Capability
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 05:49 PM

  We introduce safety mid-training, a stage between pre-training and post-training that jointly develops safety and general reasoning through this shared structure.

  Safety alignment for large language models (LLMs) is primarily performed during post-training, while recent work has also begun incorporating safety data into pre-training.

Extractive summary: sentences quoted from the sources.