recAMPLIFY: Large-Scale Pretraining of Recursive Protein Language Models

Key points

• Protein language models (pLMs) are powerful tools for protein annotation and mutation-effect prediction, but progress has largely depended on scaling model size and data, making them expensive and prone to memorization.1
• Test-time scaling offers a complementary path, but methods developed for autoregressive LLMs do not transfer directly: masked encoders lack a generation loop, and amino acids do not verbalize in a reasoning language.1
• We instantiate this recipe as recAMPLIFY, a pLM with 155M unique parameters trained on the same 2T tokens as AMPLIFY-350M.1
• Mechanistic analyses show that residual gating controls recurrence stability beyond the trained depth and that halt depth tracks biological properties.1

Sources (1)

• [1]recAMPLIFY: Large-Scale Pretraining of Recursive Protein Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 05:12 PM

  Because optimal depth for recAMPLIFY varies across tasks and assays, we then introduce Gated Mixture Recurrence (GMR), a framework that decodes each protein from a probability-weighted mixture of hidden states visited during recursion.

  Protein language models (pLMs) are powerful tools for protein annotation and mutation-effect prediction, but progress has largely depended on scaling model size and data, making them expensive and prone to memorization.

Extractive summary: sentences quoted from the sources.