Understanding and Mitigating Training Instability in Large-Scale Recommender Systems: A Cascade Perspective
Training large-scale recommender systems on dynamic industrial platforms, such as Short-Form Video (SFV) services with billions of daily interactions, faces critical stability challenges.
ProofPaper ↗
Key points
- Rapid distribution shifts in user data and increasing model complexity frequently trigger a divergence cascade, typically initiating with gradient explosion, propagating to weight explosion, and culminating in loss explosion.
- This paper provides a systematic analysis of this G-W-L cascade within an industrial SFV recommender context.
- Our contribution provides a structured framework for understanding and tackling training divergence, offering practical insights for deploying robust, continuously updated recommenders in large-scale, non-stationary settings.
- We empirically validate the effectiveness of these strategies on a major industrial SFV recommendation system.
Sources (1)
- [1]Understanding and Mitigating Training Instability in Large-Scale Recommender Systems: A Cascade PerspectivearXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 11:37 PM
Training large-scale recommender systems on dynamic industrial platforms, such as Short-Form Video (SFV) services with billions of daily interactions, faces critical stability challenges.
Rapid distribution shifts in user data and increasing model complexity frequently trigger a divergence cascade, typically initiating with gradient explosion, propagating to weight explosion, and culminating in loss explosion.
Extractive summary: sentences quoted from the sources.