ResearchResearch paperEfficiency & Inference · Large Language Models1 source · Oct 11, 2026

Fewer Bits, More Tries: Recovering Capability Under Memory Constraints

Large language models are generally more capable than smaller ones, but their memory requirements can make local deployment difficult.

We ask whether part of this capability gap can instead be recovered by running a smaller or more compressed model multiple times and selecting among its outputs using public or model-generated tests.
Single-user local inference can leave hardware underutilized, allowing parallel candidate generation to trade additional test-time computation for accuracy at modest latency cost.
Across four code-generation benchmarks, verified retry recovers substantial accuracy lost to model size or compression.
A 7B Qwen model with eight parallel attempts essentially matches a 32B model on MBPP+ and exceeds it on the other three benchmarks, while using about $4\times$ less model memory, running up to $6\times$ faster, and using less energy, as measured on Nvidia GPU.

[1]Fewer Bits, More Tries: Recovering Capability Under Memory Constraints
arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 11, 05:52 AM
Large language models are generally more capable than smaller ones, but their memory requirements can make local deployment difficult.
We ask whether part of this capability gap can instead be recovered by running a smaller or more compressed model multiple times and selecting among its outputs using public or model-generated tests.

Extractive summary: sentences quoted from the sources.

Before this