The Self-Improvement Problem

Key points

• With the rise of large language models, agentic systems are becoming increasingly autonomous, as they write code, call tools, revise prompts, and propose modifications to their own components.1
• The gate must be a terminating, computable procedure, while the property we ultimately care about, namely whether an update is a genuine semantic improvement, depends on potentially unbounded program behavior.1
• We formalize this mismatch by modeling agents as programs and defining improvement in terms of a semantic score that need not be computable.1
• We show that, in the fully general setting, no terminating procedure can simultaneously guarantee reliability (never accepting a non-improvement) and non-conservatism (never missing an available improvement): self-referential constructions can force any such gate or searcher to fail.1

Sources (1)

• [1]The Self-Improvement Problem
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 10:27 AM

  This autonomy raises a fundamental question for safe self-improvement: can an automated gate like a reward model or an automated reviewer reliably decide whether a proposed update should be accepted?

  With the rise of large language models, agentic systems are becoming increasingly autonomous, as they write code, call tools, revise prompts, and propose modifications to their own components.

Extractive summary: sentences quoted from the sources.