Answering While You Type: Optimizing Time-to-Answer with Streaming Generation

Key points

• Work on efficient large-language-model inference has concentrated almost entirely on reducing the cost of decoding once a request is complete.1
• The quantity a user experiences, however, is the interval between submitting a request and receiving a usable response, and that interval can be shortened without making decoding itself faster.1
• While a request is being composed it is already partially specified, yet conventional deployments begin no computation until submission.1
• We evaluate the regime on autoregressive and diffusion language models, characterise the conditions under which it reduces user-visible latency, and analyse the mechanism responsible.1

Sources (1)

• [1]Answering While You Type: Optimizing Time-to-Answer with Streaming Generation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 12:46 PM

  We propose and study streaming generation, a decoding regime in which the model begins producing a response from an incomplete request and revises it as the remaining input arrives, so that only a small residue of the answer remains to be computed once the request is submitted.

  Work on efficient large-language-model inference has concentrated almost entirely on reducing the cost of decoding once a request is complete.

Extractive summary: sentences quoted from the sources.