Pretraining and Test-Time Training Budgets Matter for ARC: A Controlled Study of VARC and TRM
The Abstraction and Reasoning Corpus (ARC) is increasingly used to evaluate whether neural networks can infer task-specific transformations from few demonstrations.
ProofPaper ↗
Key points
- Recent ARC models differ in architecture, but also in broader training and evaluation protocols, including pretraining data, data augmentation, test-time setup, and optimization budgets, making conclusions about architecture benefits debatable.
- In this work, we study how the performance of the vision-inspired VARC and recursion-based TRM models is affected by pretraining and test-time training (TTT) budgets under a common inductive protocol on ARC-AGI-1.
- We cross five pretraining budgets with four TTT budgets for each test task.
- We observe that increasing TTT sample presentations improves the standard ARC pass@2 metric in all settings for both models.
Sources (1)
- [1]Pretraining and Test-Time Training Budgets Matter for ARC: A Controlled Study of VARC and TRMarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 02:33 PM
The Abstraction and Reasoning Corpus (ARC) is increasingly used to evaluate whether neural networks can infer task-specific transformations from few demonstrations.
Recent ARC models differ in architecture, but also in broader training and evaluation protocols, including pretraining data, data augmentation, test-time setup, and optimization budgets, making conclusions about architecture benefits debatable.
Extractive summary: sentences quoted from the sources.