Prefix-Tuning for Arbitrary Output Sequences on Pretrained Transformers

Key points

• Despite its practical success, the theoretical understanding of the expressiveness of prefixes remains incomplete.1
• In particular, the following question remains open: Given a pretrained autoregressive Transformer, for any input-output sequence pair $(\mathtt{x},\mathtt{y})$, does there always exist a prefix $\mathcal{S}$ such that prepending $\mathcal{S}$ to $\mathtt{x}$ induces the model to generate the target output $\mathtt{y}$?1
• In this work, we prove that under the assumption that $\mathtt{y}$ is in the range of the pretrained Transformer, such a prefix $\mathcal{S}$ exists and can be explicitly computed by solving multiple linear systems.1
• Extensive experiments on GPT-2 and Qwen3-4B demonstrate that our algorithm produces prefixes with strong control over model outputs.1

Sources (1)

• [1]Prefix-Tuning for Arbitrary Output Sequences on Pretrained Transformers
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 06:49 PM

  Prefix-tuning is a parameter-efficient fine-tuning method for large language models (LLMs) that steers model outputs by prepending task-specific prefix representations to the input sequence.

  Despite its practical success, the theoretical understanding of the expressiveness of prefixes remains incomplete.

Extractive summary: sentences quoted from the sources.