EvolvingBrain: An Adaptive Skill System for Multi-modal Agents Self-Evolution in Open Worlds

Key points

• Multi-modal agents have made significant progress in interacting with open worlds.1
• Stratified Skill Self-Evolution: extracts Procedural Patterns and Problematic Patterns from model-specific trajectories, summarizes experience by generating skills, and evolves through multiple interactions with the open world, ultimately forming a Stratified Skill Architecture consisting of Domain, Plan, and Action/Trap Skills, which provides the agent with a multi-granularity knowledge repository and a novel evolution approach.1
• State-Aware Adaptive Execution: summarizes long-horizon interactions into Structured Abstractive Memory and uses it to guide the State-Aware Skill Fusion Strategy, enabling the agent to utilize knowledge according to its current state.1
• Experiments on VAB-Minecraft and DiscoveryWorld demonstrate that EvolvingBrain improves success rates on open-world tasks and exhibits generalization capabilities.1

Sources (1)

• [1]EvolvingBrain: An Adaptive Skill System for Multi-modal Agents Self-Evolution in Open Worlds
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 06:30 PM

  Therefore, we propose EvolvingBrain, which uses an adaptive skill system as the brain for agent self-evolution, consisting of two following components.

  Multi-modal agents have made significant progress in interacting with open worlds.

Extractive summary: sentences quoted from the sources.