When Should AI Chime In? Benchmarking Timely Contribution of Proactive Agents in Multi-Party Chat

Key points

• Proactive agents can support group collaboration by surfacing useful information without waiting to be asked, but even relevant contributions can be disruptive if poorly timed.1
• In this work, we isolate the timing question: given a fixed candidate contribution such as a clarification or relevant fact, when should an AI agent chime in?1
• To collect high-quality judgments, we design a novel interface in which 2-4 people chat synchronously, author candidate AI contribution messages, place their own and others' contributions into the chat history, and then deliberate as a group about the best placement for each message.1
• Evaluating 15 models across 28 configurations, we find that while models can place contributions well when given the entire conversation history, they systematically chime in too early when asked to decide live as the conversation unfolds: best-turn accuracy falls from 57.8% to 19.0%, with nearly half of placements occurring before every participant's acceptable window.1

Sources (1)

• [1]When Should AI Chime In? Benchmarking Timely Contribution of Proactive Agents in Multi-Party Chat
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 06:54 AM

  We introduce PITCH (Proactive Intervention Timing in CHat), to our knowledge the first benchmark with large-scale human judgments of proactive agent timing collected from participants in the same unscripted multi-party conversations they took part in.

  Proactive agents can support group collaboration by surfacing useful information without waiting to be asked, but even relevant contributions can be disruptive if poorly timed.

Extractive summary: sentences quoted from the sources.