FlashAttention

Timeline

1. Oct 16, 2026 · Opinion / analysis · 1 source
   [P] my rust framework trains nanoGPT with the exact same bits as pytorch. worth continuing? [P]
   the thing I'm kind of proud of: if you point import torch at it and run karpathy's nanoGPT unchanged, you get the exact same numbers as pytorch. same initial weights from manualseed, same loss, same gradients, same weights, bit for bit, for all 50 steps I tested, even with dropout on. not "close enough", literally identical.
2. Oct 16, 2026 · Open-source release · 1 source
   axolotl-ai-cloud/axolotl v0.21.0
   This release is headlined by Reflection's Beam, a 501B-parameter MoE you can QLoRA-tune on a single node.
3. Oct 16, 2026 · Research paper · 1 source
   Prism Attention: Proposal-Refined Index Sharing Mechanism for Efficient LLMs Inference
   Long-context decoding in large language models (LLMs) is increasingly bottlenecked by the memory and computation required to attend over an ever-growing Key-Value (KV) cache at each autoregressive step.
4. Oct 15, 2026 · Research paper · 2 sources
   Velox: Spatial Memory Makes World Models Efficient
   To address this bottleneck, we introduce Velox, a training-free attention router for region-level context selection using frustum slab correspondences from streaming geometry.
5. Oct 15, 2026 · Research paper · 1 source
   FlashAttention with Statistics: Low-Overhead Attention Features for Compact Hallucination Detectors
   Large language models can produce fluent answers that the supplied context does not support.
6. Oct 15, 2026 · Research paper · 1 source
   Sparse Attention for Video Generation Acceleration via Growing Sparsity and Reduced Search
   Based on these findings, we propose Growing Sparsity and Reduced Search (GSRS), a training-free framework that retains more computation early and progressively reduces it later, while restricting subsequent searches to an early candidate set to reduce search overhead.
7. Oct 13, 2026 · Research paper · 1 source
   Visual Tokens on Demand: Visual Budgeting for Scalable and Efficient Multi-Image Reasoning in Multimodal LLMs
   Multi-image reasoning introduces additional challenges: redundancy may occur across images, task-relevant evidence may be sparse or distributed, and aggressive per-image compression can remove details needed for comparison, counting, temporal reasoning, retrieval, or evidence grounding.
8. Oct 13, 2026 · Research paper · 2 sources
   FlashMask-3: Efficient and Expressive Mask-Aware Distributed Attention
   Long-context training of large language and multimodal models couples structured attention masks (document, prefix-LM, share-question, sliding-window, block-sparse) with multi-node context parallelism, making the mask a dominant bottleneck.
9. Oct 13, 2026 · Research paper · 1 source
   Foundation Models for Event Prediction Across Heterogeneous Event Logs
   We address this challenge with the Foundation Event Predictor (FEP), a simple and scalable in-context architecture that performs next-event prediction through learned, event-level nonparametric inference.
10. Oct 12, 2026 · Research paper · 1 source
    GPU Goldmines: Tunable WebGPU Kernels for Edge LLM Inference
    To address this, we present GPU Goldmines, a framework for specifying, executing, and tuning WebGPU kernels.
11. Oct 12, 2026 · Research paper · 1 source
    A Scaled-Up Empirical Study of Syntactic Language Models
    Syntactic language models (SLMs) that generate sentences along with their syntactic tree structures have shown promise in both syntactic generalization and downstream tasks.
12. Oct 12, 2026 · Opinion / analysis · 1 source
    strix-llama new branch: prompt processing +21–54% over Vulkan and up to 2.7× over stock ROCm kernel at 128k, same outputs quality
    I've been tuning llama.cpp on my Strix Halo box (Radeon 8060S, 128 GB) and the numbers are good enough to share.
13. Oct 11, 2026 · Opinion / analysis · 1 source
    Taming the AI Hardware Cambrian Explosion — Abdul Dakkak, Modular
    Abdul Dakkak, chief scientist at Modular, explains why today's AI software stack is a mess, with vLLM, SGLang, TensorRT-LLM and llama.cpp each patching around different hardware, and how Modular rebuilt the stack from the ground up.
14. Oct 11, 2026 · Research paper · 1 source
    Channel Match: Calibrating Shape Latents for Few-Step 3D Generation
    In this work, we introduce a few-step distillation framework for native 3D generation, enabled by continuous self-consistency training with a custom JVP FlashAttention kernel and an analytic velocity regression objective for sparse structured 3D latents.
15. Oct 8, 2026 · Open-source release · 1 source
    huggingface/trl v1.15.0
    SFT, DPO, KTO, GRPO, RLOO and Distillation now score tokens with a fused LM head: a Triton kernel projects the hidden states through the LM head in tiles and reduces to per-token log-probs and entropy directly, so the [batch, seq, vocab] logits tensor is never built.
16. Oct 6, 2026 · Open-source release · 1 source
    huggingface/transformers v5.19.0: Release v5.19.0
    EmbeddingGemma 2 is a multimodal embedding model from Google built on the Gemma 4 architecture.
17. Oct 6, 2026 · Research paper · 1 source
    Cleave: Scaling Tensor Program Optimization via Decoupled Algebraic Search and Operator Scheduling
    We propose Cleave, an ML compiler built on symbolic decoupling: Cleave discovers transformations by performing superoptimization on a graph with symbolic shapes, and then schedules each resulting graph on concrete shapes.
18. Aug 10, 2026 · Open-source release · 1 source
    vllm-project/vllm v0.27.0
    Kimi K3 support with a full stack landing in one release: core model files and kernels (#50089, #50000), Python (#50093) and Rust (#50104) frontends, AttnRes kernels (#50090), DeepGEMM support (#50458), compressed-tensors quantized checkpoints (#50500), DSpark AR fusion (#50242), and an option to shard the shared expert instead of replicating it (#50656).
19. Aug 10, 2026 · Open-source release · 1 source
    huggingface/transformers v5.15.0: Release: v5.15.0
    Muse Glimmer, released today, is Meta’s new multimodal model, especially designed for agentic use cases.
20. Jul 15, 2026 · Open-source release · 1 source
    huggingface/transformers v5.14.0: Release v5.14.0
    Inkling is a general-purpose multimodal model that accepts text, image and audio inputs and
21. Jul 11, 2026 · Open-source release · 1 source
    vllm-project/vllm v0.25.0
    Building on quantized-model support from the previous release, MRv2 is now the standard execution path, with new support for EVS (#46535), realtime embeddings (#46762), prefix caching for Mamba hybrid models (#42406), multimodal-prefix bidirectional attention (#46942), and dynamic speculative decoding compatible with full CUDA graphs (#45953).
22. Jun 29, 2026 · Open-source release · 1 source
    vllm-project/vllm v0.24.0
    MiniMax-M3: Added support for the new MiniMax-M3 model (#45381), with a fast follow-on of BF16/FP8 indexer via MSA (#45892), MXFP4 support (#45896), FP8 sparse GQA (#45744), and extensive AMD/ROCm tuning — mxfp8 MoE/linear on gfx950 (#45725), fp8perchannel for bf16 weights on MI300X (#45854), FP8 KV-cache fix (#45720), and packed-modules mapping (#45794).
23. Apr 27, 2026 · Open-source release · 1 source
    vllm-project/vllm v0.20.0
    CUDA 13.0 default: Default CUDA wheel on PyPI and vllm/vllm-openai:v0.20.0 image switched to CUDA 13.0; architecture lists and build-args cleaned up (#39878), and CUDA bumped to 13.0.2 to match PyTorch 2.11.0 (#40669).
24. Apr 6, 2026 · Open-source release · 1 source
    sgl-project/sglang v0.5.10
    Piecewise CUDA Graph Enabled by Default: Piecewise CUDA graph capture is now the default execution mode, reducing memory overhead and improving throughput for models with complex control flow patterns: #16331
25. Mar 28, 2026 · Open-source release · 1 source
    sgl-project/sglang v0.5.10rc0
    Piecewise CUDA Graph Enabled by Default: Piecewise CUDA graph capture is now the default execution mode, reducing memory overhead and improving throughput for models with complex control flow patterns: #16331
26. Mar 23, 2026 · Open-source release · 1 source
    pytorch/pytorch v2.11.0: PyTorch 2.11.0 Release
    <strong>FlexAttention</strong> now has a <strong>FlashAttention-4</strong> backend on <strong>Hopper</strong> and <strong>Blackwell</strong> GPUs
27. Jan 23, 2026 · Open-source release · 1 source
    sgl-project/sglang v0.5.8
    Qwen3-VL-Embedding & Qwen3-VL-Reranker model support: #16635, #16403
28. Oct 17, 2024 · Open-source release · 1 source
    pytorch/pytorch v2.5.0: PyTorch 2.5.0 Release, SDPA CuDNN backend, Flex Attention
    We are excited to announce the release of PyTorch® 2.5!

Often appears with