How Far Can Foundation Models Be Distilled for Efficient Jet Tagging?

Key points

• Recent models build in more physics knowledge or rely on large-scale pretraining, but the experiments place strict limits on their inference cost.1
• On a jet tagging benchmark, we distil six teachers that differ in architecture, symmetry and pretraining, including two foundation models, into five small students.1
• With 20 thousand parameters, a distilled L-GATr-slim outperforms every published jet tagger trained from scratch at a fraction of the inference cost of every teacher.1
• We release the code and the weights of all trained models.1

Sources (1)

• [1]How Far Can Foundation Models Be Distilled for Efficient Jet Tagging?
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 09:00 AM

  Jet tagging is one of the most important applications of machine learning at the Large Hadron Collider.

  Recent models build in more physics knowledge or rely on large-scale pretraining, but the experiments place strict limits on their inference cost.

Extractive summary: sentences quoted from the sources.