Plan or Retrieve First? A Scale Study of Tool Discovery Architectures

Key points

• The dominant approach, which we term Retrieve-then-Plan (RtP), embeds the user query and retrieves similar tools via approximate nearest-neighbor search.1
• We study an alternative Plan-then-Retrieve (PtR) paradigm in which the LLM first generates Abstract Tool Stubs (hypothesized function signatures describing needed capabilities) which then serve as retrieval queries against the registry.1
• Evaluating on ToolBench (46,984 unique API endpoints, 50 categories) across single-tool (G1), intra-category multi-tool (G2), and cross-category multi-tool (G3) query subsets, we find a clear complexity-dependent crossover: RtP outperforms PtR on single-tool queries while PtR outperforms RtP on multi-tool queries.1
• We further analyze token overhead and show that PtR’s extra discovery call roughly doubles inference cost, which pays off on multi-tool queries but hurts on single-tool ones, motivating complexity-adaptive routing as a practical direction—one that can be driven by a zero-cost signal already present in the pipeline (RtP’s top-1 retrieval score).1

Sources (2)

• [1]Plan or Retrieve First? A Scale Study of Tool Discovery Architectures
  Apple Machine Learning Research · Oct 15, 12:00 AM

  Tool-augmented LLM agents discover relevant APIs from large registries before planning execution, yet finite context windows prohibit injecting all tool definitions into a single prompt.

  The dominant approach, which we term Retrieve-then-Plan (RtP), embeds the user query and retrieves similar tools via approximate nearest-neighbor search.

• [2]Plan or Retrieve First? A Scale Study of Tool Discovery Architectures
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 07:03 PM

Extractive summary: sentences quoted from the sources.