DVTR-KD: Distilling Vision-Language Models via Intermediate Visual Token Reconstruction

Key points

• Multimodal large language models (MLLMs) have achieved strong performance on diverse vision-language tasks, but their computational demands limit deployment under constrained budgets, particularly on edge devices.1
• In addition to reducing model size, visual token compression provides a complementary route to efficiency, but introduces mismatched visual token counts that complicate direct token-wise knowledge transfer.1
• We reconstruct visual states after the final LLM layer and at the input to an intermediate layer identified as important for visual-information readout.1
• Evaluations on diverse benchmarks demonstrate that DVTR-KD improves the performance of token-compressed students while retaining the inference efficiency afforded by visual token compression.1

Sources (1)

• [1]DVTR-KD: Distilling Vision-Language Models via Intermediate Visual Token Reconstruction
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 07:18 AM

  To mitigate this, we propose DVTR-KD, a knowledge distillation framework based on Dense Visual Token Reconstruction, which reconstructs the teacher's dense visual tokens from the student's compressed tokens.

  Multimodal large language models (MLLMs) have achieved strong performance on diverse vision-language tasks, but their computational demands limit deployment under constrained budgets, particularly on edge devices.

Extractive summary: sentences quoted from the sources.