Heartbeat Thinking: Speculative Reasoning and Tool Execution While Listening in Realtime Voice Agents

Key points

• Tool-using voice agents often leave users waiting after speech ends, even when actionable information arrives earlier in the request.1
• Speculative approaches begin work during speech, but chunk-bounded or interrupt-on-update generation can truncate a tool call before its arguments are complete.1
• Across SpeechFCEval, FDBench v3, and AudioCRAG, Heartbeat improves aggregate speculative tool coverage or retrieval reuse over a SHANKS-style windowed baseline, with benchmark-dependent effects on final task quality.1
• The results show that heartbeat thinking paradigm preserving ongoing computation during speech can turn listening time into reusable preparation for faster and more capable voice-agent responses.1

Sources (1)

• [1]Heartbeat Thinking: Speculative Reasoning and Tool Execution While Listening in Realtime Voice Agents
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 11:59 AM

  We introduce Heartbeat Thinking, which handles updated input through new Thinker invocations while allowing ongoing invocations and dispatched tools to finish.

  Tool-using voice agents often leave users waiting after speech ends, even when actionable information arrives earlier in the request.

Extractive summary: sentences quoted from the sources.