Articulation Upper Bounds for Language Models

Key points

• Language captures only a subset of the perceptions, intuitions and intents that form meaning between speakers.1
• This requires novel measurement tools: (1) optimal messages: we introduce a prompt optimization algorithm that beats existing algorithms by up to 14% on benchmarks and accurately estimates prompt upper bounds.1
• (2) A transmission measure: the sender is asked to label text; the receiver then tries to recover the sender's prompt, and recovery is measured; we find this correlates strongly with human label error.1
• Interesting, these same concepts are not all challenging for humans: humans have moderate-to-high levels of agreement in 76% of concepts that LLMs struggle with; articulation limits, we find, are executor relative.1

Sources (1)

• [1]Articulation Upper Bounds for Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 07:47 AM

  With these measures, we use observational scaling laws to study how transmissibility of a concept, conveyed with optimal messages, increases as transmitter and receiver capacity grows: does it asymptote below or above a threshold?

  Language captures only a subset of the perceptions, intuitions and intents that form meaning between speakers.

Extractive summary: sentences quoted from the sources.