Bellman Language Models: Solving the Recurring Decisions of LLM Agents

Key points

• LLMs generally don't treat these decisions as an optimization problem and instead re-derive them from their context/prompt on every turn.1
• We show that treating decision making within reasonable bounds as an optimization problem, solving it once, and reusing the solved decisions can often make an agent faster, cheaper and more reliable.1
• Our approach, which we call Discretize, Then Decide, starts from data such as the workflow's own logs or, when none exists, runs of the LLM agent itself.1
• We call an LLM run this way a Bellman language model (BLM), after Richard Bellman's equation for the best move in each state.1

Sources (1)

• [1]Bellman Language Models: Solving the Recurring Decisions of LLM Agents
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 09:19 PM

  A general observation in agentic workflows, especially bounded ones, is that the decision space of the LLM, when we separate a decision from generation, starts approaching saturation: the agent meets the same states again and again, and after enough conversations it stops meeting new ones.

  LLMs generally don't treat these decisions as an optimization problem and instead re-derive them from their context/prompt on every turn.

Extractive summary: sentences quoted from the sources.