Megatron-LM

Timeline

1. Oct 13, 2026 · Research paper · 2 sources
   FlashMask-3: Efficient and Expressive Mask-Aware Distributed Attention
   Long-context training of large language and multimodal models couples structured attention masks (document, prefix-LM, share-question, sliding-window, block-sparse) with multi-node context parallelism, making the mask a dominant bottleneck.
2. Oct 12, 2026 · Research paper · 2 sources
   Alibaba Qwen Introduces QwenGyre: Elastic Reinforcement Learning RL for XLong-Horizon Agents
   Alibaba’s Qwen team has introduced QwenGyre, an elastic reinforcement learning RL framework for extreme-long-horizon agent training.
3. Oct 12, 2026 · Research paper · 1 source
   A Production-Grade, High-Efficiency Agentic RL Framework for Frontier LLMs with Validated Recipes
   To address these challenges, we present LoongSage, an agent-native, fully asynchronous RL framework with a lightweight core of roughly 14K lines of code.
4. Oct 7, 2026 · Research paper · 1 source
   Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling
   Mixture-of-Experts (MoE) layers replace the feed-forward block of a Transformer with E expert networks, and each token is routed to k of these experts.
5. Oct 6, 2026 · Benchmark result · 1 source
   Scale Bitwise-Deterministic Pretraining with NVIDIA Megatron Core
   Bitwise determinism makes large-scale pretraining easier to debug, validate, and resume reproducibly.

Often appears with