Open GPU kernel engineering series: naive GEMM to FlashAttention, H100 and Blackwell (44 sessions, code public) [P]
I'm building a free, code-first series on GPU kernel engineering and just published the intro and roadmap video.
Proof1 community thread
Key points
- The core idea behind the series: for most ML workloads, kernel performance is a data-movement problem before it's a compute problem.
- A kernel's position relative to the ridge point tells you whether to optimize memory access or compute.
- Foundations: CPU parallelism, GPU architecture, roofline, first CUDA kernels
- Hopper and Blackwell: TMA, WGMMA, CUTLASS, FP8/quantization, NVFP4, multi-GPU basics, FlashAttention 4, DeepSeek's kernels, AI-written kernels, capstone project
Sources (1)
- [1]Open GPU kernel engineering series: naive GEMM to FlashAttention, H100 and Blackwell (44 sessions, code public) [P]r/MachineLearning (top, daily) · Oct 19, 06:15 PM
I'm building a free, code-first series on GPU kernel engineering and just published the intro and roadmap video.
The core idea behind the series: for most ML workloads, kernel performance is a data-movement problem before it's a compute problem.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 18, 202672 hours, four big open weights drops. The only one that fits in 24GB came from NVIDIA, of all companies.
- Oct 18, 2026vllm-project/vllm v0.32.0
- Oct 16, 2026[Research] 2,788 tokens/s on a single RTX 5090 with a diffusion LLM — Introducing ORL (fully open source)
- Oct 16, 2026FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper
- Oct 15, 2026QUASAR 4-bit QAT checkpoints for Qwen3.5, Gemma 4 and Muse Glimmer: closer to BF16 than Google’s and NVIDIA’s 4-bit releases
- Oct 15, 2026NVIDIA Nemotron 3.5 Super VL Powers Specialized Agents with Multimodal Reasoning