COMPASS: Routing via Capability-Oriented Model Profiling with Adaptive Search, Selection, and Tool Use

Key points

• LLM routing aims to select a capable model at low cost, but specialized routers can require updates as models and tasks change.1
• Our evaluations on LLMRouterBench and DeepSWE show that agentic routing can improve accuracy and lower model-call cost relative to specialized and trained routing baselines, without routing-specific training.1
• On LLMRouterBench, COMPASS reaches 69.85% accuracy at $139.57, compared with RLCascadeRouter's 69.14% at $180.22.1
• COMPASS also supports routing across benchmarks and adapts to an expanded model pool by reading new evaluation records, without changing the router's weights or prompt.1

Sources (1)

• [1]COMPASS: Routing via Capability-Oriented Model Profiling with Adaptive Search, Selection, and Tool Use
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 08:13 AM

  We propose COMPASS, an agentic router in which an LLM follows a routing recipe, gathers performance and cost evidence through tools, and optionally inspects candidate answers before selecting a model.

  LLM routing aims to select a capable model at low cost, but specialized routers can require updates as models and tasks change.

Extractive summary: sentences quoted from the sources.