Who Pays for Efficient Inference? Confidence-Based Small-to-Large LLM Routing Across Tasks and Languages

Key points

• Its success, however, depends on identifying examples that benefit from escalation, and the reliability of confidence across multilingual tasks remains unclear.1
• We present a systematic evaluation of confidence-based routing across three model families, four languages, and three task types, covering 72 model, dataset, language, and scale configurations and 42,288 predictions.1
• We compare global confidence routing with random routing, equal-quota confidence routing, and an oracle upper bound.1
• These results show that confidence routing is task-dependent and can allocate computation unevenly across languages.1

Sources (1)

• [1]Who Pays for Efficient Inference? Confidence-Based Small-to-Large LLM Routing Across Tasks and Languages
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 10:23 AM

  Small-to-large language model routing can reduce inference cost by escalating difficult inputs while retaining easier ones on a smaller model.

  Its success, however, depends on identifying examples that benefit from escalation, and the reliability of confidence across multilingual tasks remains unclear.

Extractive summary: sentences quoted from the sources.