ResearchResearch paperLarge Language Models · Multimodal Models1 source · Oct 15, 2026

Retrofitting Recurrent Vision-Language Models from Pretrained Backbones

Recent work shows that depth recurrence can be retrofitted into pretrained language models.

In this work, we adapt this paradigm for visual understanding, using depth recurrence to reduce the parameter footprint of a pretrained vision-language model (VLM).
As the pruning impacts the original model performance, we propose a three-stage recovery training: first, we leverage a language-only pretraining to recover the performance of the language backbone, followed by an alignment of the vision-language projector, and a full fine-tuning of the model via visual instruction tuning.
We evaluate the proposed approach on the TinyLLaVA-1.4B backbone without introducing any new vision-language data during retrofitting.
Our experiments demonstrate that the retrofitted recurrent model outperforms the unpruned baseline trained with the same recovery data while storing 23.3% fewer language parameters.

[1]Retrofitting Recurrent Vision-Language Models from Pretrained Backbones
arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 03:34 PM
Recent work shows that depth recurrence can be retrofitted into pretrained language models.
In this work, we adapt this paradigm for visual understanding, using depth recurrence to reduce the parameter footprint of a pretrained vision-language model (VLM).

Extractive summary: sentences quoted from the sources.

Before this