Out of Support, Not Out of Reach: In-Context Learning with Unsupported Instructions and Composed Tasks
In-Context Learning (ICL) enables pretrained LLMs to adapt to downstream tasks from a small set of input-output demonstrations, without parameter updates.
ProofPaper ↗
Key points
- Yet existing theory primarily explains ICL as identifying a latent task already represented in pretraining, leaving unclear whether ICL can succeed when either the instruction or the target task falls outside the pretraining support.
- For target tasks represented in pretraining, we show that demonstrations can identify the task and reduce sensitivity to pretrain-supported prompt templates as their number increases.
- For an unsupported instruction, we show that under a fixed model-induced reference extension, demonstration evidence can dominate an $N$-independent instruction shift while preserving the exponential identification rate.
- Applying the supported-task analysis step by step yields a standard accumulated-error bound and clarifies how invoking familiar subtasks along an intermediate-state trajectory can avoid directly selecting the unsupported complete task.
Sources (1)
- [1]Out of Support, Not Out of Reach: In-Context Learning with Unsupported Instructions and Composed TasksarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 02:27 PM
In-Context Learning (ICL) enables pretrained LLMs to adapt to downstream tasks from a small set of input-output demonstrations, without parameter updates.
Yet existing theory primarily explains ICL as identifying a latent task already represented in pretraining, leaving unclear whether ICL can succeed when either the instruction or the target task falls outside the pretraining support.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 15, 2026SAFETYEDIT: Sparse Internal Targets with Prompt-Wide Control for LLM Jailbreak Backdoors
- Oct 15, 2026Don't Retrain, Remix: Adaptation of Web Agents using Mixture Hypernetworks
- Oct 15, 2026Towards Continual Learning: Mitigating Forgetting With Subspace Tuning
- Oct 15, 2026Decomposing Brain-LLM Alignment with Interpretable Semantic Representations
- Oct 15, 2026Incentivizing Tool-Use Reinforcement Learning for Post-Arrival Emergency Care Prediction
- Oct 15, 2026From Observation to Intervention: Distilling Self-Generated In-Context Demonstrations via Executable Causal Intervention