Elicitation or Expansion? Demystifying Gains from Agent Harness Optimization

Key points

• We conduct a systematic analysis to study the effect of gains from harness optimization.1
• Building on pass@k analyses in the reinforcement learning (RL) literature that compare models before and after RL training, we compare optimized harnesses with their pre-optimization counterparts.1
• Specifically, we assess changes in reachable task coverage, compare harness optimization with model scaling and increased inference-time compute, and examine how gains vary with task difficulty.1
• Across static prediction and dynamic tasks, we find that pass@1 advantages of harness over stronger interventions can reverse at larger attempt budgets and aggregate gains conceal shortfalls on hard tasks, underscoring the need for a fine-grained diagnosis.1

Sources (1)

• [1]Elicitation or Expansion? Demystifying Gains from Agent Harness Optimization
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 09:11 AM

  Recent studies in harness optimization have shown substantial performance gains in language model agents without updating model weights.

  We conduct a systematic analysis to study the effect of gains from harness optimization.

Extractive summary: sentences quoted from the sources.