The Effectiveness and Durability of Pretraining Data Filtering in Protein Language Models
We find that filtering viral proteins from the pretraining data meaningfully degrades masked language modelling, mutation effect prediction and long-range contact prediction on viral proteins, while largely preserving performance on non-viral proteins.
ProofPaper ↗
Key points
- Pretraining filtering is one of the most common safeguards for open-weight language models.
- Here we study its effectiveness, durability and scalability using the ESM-2 family of protein language models.
- As models increase in size, we find that stringent filtering becomes more effective.
- This work establishes pretraining filtering as moderately effective at selectively degrading performance on viral proteins, but not durable against fine-tuning with modest computational resources and access to relevant data.
Sources (1)
- [1]The Effectiveness and Durability of Pretraining Data Filtering in Protein Language ModelsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 04:08 PM
We find that filtering viral proteins from the pretraining data meaningfully degrades masked language modelling, mutation effect prediction and long-range contact prediction on viral proteins, while largely preserving performance on non-viral proteins.
Pretraining filtering is one of the most common safeguards for open-weight language models.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 19, 2026Communication-Efficient Federated Domain Generalization in Edge Networks via Dynamic Prototype Regulation
- Oct 19, 2026TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token Pruning
- Oct 19, 2026TorchNeuron: running HF models on AWS Trainium is easier than ever
- Oct 19, 2026Scaling Parameter-Efficient Fine-Tuning for Brain Encoding
- Oct 16, 2026Hugging Face ❤️ TorchTPU: TPUs, Natively
- Oct 16, 2026[Research] 2,788 tokens/s on a single RTX 5090 with a diffusion LLM — Introducing ORL (fully open source)