Coordination-Free Workload Regularization for Variable-Cost Distributed Training
We present Batch Cost Lookahead (BCL), a coordination-free data-loading policy that buffers a bounded rank-local window and repartitions it into fixed-cardinality packs with balanced estimated cost.
ProofPaper ↗
Key points
- Synchronous data-parallel training usually assigns every rank the same number of records, even when record costs vary by orders of magnitude.
- For variable-length models, this converts input variation into idle time because each step waits for its most expensive rank-local batch.
- Compiled H200 measurements on an internal sequence workload show speedups of 1.197 to 1.343 times over packed execution from one to four nodes.
- We also identify a memory-feasibility boundary: a p99 cap for high-variance or OOM-prone data bounds outliers before BCL balances the rest.
Sources (1)
- [1]Coordination-Free Workload Regularization for Variable-Cost Distributed TrainingarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 03:22 AM
We present Batch Cost Lookahead (BCL), a coordination-free data-loading policy that buffers a bounded rank-local window and repartitions it into fixed-cardinality packs with balanced estimated cost.
Synchronous data-parallel training usually assigns every rank the same number of records, even when record costs vary by orders of magnitude.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 15, 2026Local and Distributed Large LLM Inference with MLX and RDMA on Mac Studio
- Oct 14, 2026CausalEvo: Causal Skill Self-Evolution for Health LLM Agents
- Oct 14, 2026Cross-subject decoding of imagined speech from scalp EEG: a leakage-controlled evaluation on three public corpora
- Oct 14, 2026Learning Warm Starts for the Simplex Method from Measured Solver Cost
- Oct 14, 2026Selection-Safe Conformal Early Exit: Distribution-Free Coverage under Adaptive Depth
- Oct 14, 2026mistralai/Timestral-209M