Training User Simulators with Latent Intrinsic Feedback

Key points

• User simulators are gaining attention as environments to train LLM assistants that can better cope with the nuances of multi-turn conversations with real users.1
• Prior work on user simulation has emphasized generating realistic user utterances, leaving unexplored their potential as a source of direct feedback on the assistant's responses.1
• We train LLM-based simulators using interaction logs between real users and assistants and demonstrate that intrinsic feedback improves simulation realism and critiquing behaviors versus prompted simulator baselines, while also providing turn-level evaluations that existing fine-tuned simulators lack.1
• Lastly, our experiments reveal that simulators with intrinsic feedback provide effective reward signals for multi-turn RL training, matching setups that rely on external judge LLMs for rewards while cutting RL compute by more than half.1

Sources (1)

• [1]Training User Simulators with Latent Intrinsic Feedback
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 02:00 PM

  In this work, we introduce user simulators with intrinsic feedback, namely, models equipped with latent evaluation states: internal thoughts, reactions, and scalar ratings that evaluate the assistant from a first-person user perspective at each conversation turn.

  User simulators are gaining attention as environments to train LLM assistants that can better cope with the nuances of multi-turn conversations with real users.

Extractive summary: sentences quoted from the sources.