FLMNet: Self-Ensemble Distillation from Masks to Seeds for Single-Tower CTR Prediction

Key points

• Several recent CTR models, such as FinalNet and QNN, report higher accuracy when a network also learns from its own averaged predictions, for example by running the same network under different dropout patterns and training each run toward the average.1
• Why this self-distillation helps, and whether its random sampling is needed, remains unclear.1
• This work shows that as the number of dropout patterns grows without bound, the objective reduces exactly to the label loss of the logit-averaged prediction plus a penalty on how much the dropout copies of the network disagree.1
• Networks trained from different seeds disagree in the same form, and distilling several of them into one network recovers part of their ensemble gain.1

Sources (1)

• [1]FLMNet: Self-Ensemble Distillation from Masks to Seeds for Single-Tower CTR Prediction
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 10:27 AM

  Click-through rate (CTR) prediction underpins recommender systems and online advertising, where high-dimensional categorical features interact in combinatorially many ways.

  Several recent CTR models, such as FinalNet and QNN, report higher accuracy when a network also learns from its own averaged predictions, for example by running the same network under different dropout patterns and training each run toward the average.

Extractive summary: sentences quoted from the sources.