PRECOCIAL: Pretraining with Code Curriculums via Iterative Search for Accelerated Learning

Key points

• Fully synthetic pretraining data generated by symbolic programs can bootstrap downstream training on naturalistic tasks, such as language modeling, but the generating programs have so far been designed by hand.1
• Because every score requires training, PRECOCIAL keeps a single network for the entire search, continuing its pretraining for each accepted program instead of training a new network per candidate, making it significantly more efficient.1
• On four language and code datasets, at a fixed fine-tuning budget, optimized programs reach lower perplexity than hand-designed curricula, published procedural generators, random program mutation and the program the LLM writes in the first iteration of the search, and reach the same perplexity as the hand-designed curricula with 9-33% fewer fine-tuning steps.1
• We also observe that an optimized program generator from one domain transfers to datasets in other domains.1

Sources (1)

• [1]PRECOCIAL: Pretraining with Code Curriculums via Iterative Search for Accelerated Learning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 02:53 AM

  We introduce PRECOCIAL, a method for automatically optimizing programs that generate synthetic pretraining data.

  Fully synthetic pretraining data generated by symbolic programs can bootstrap downstream training on naturalistic tasks, such as language modeling, but the generating programs have so far been designed by hand.

Extractive summary: sentences quoted from the sources.