Revisiting Natural Language Prompts from Large Language Models for Open-Vocabulary Video Recognition
Vision-language models (VLMs) like CLIP have shown great success in image tasks, however, applying CLIP to open-vocabulary video recognition poses challenges.
ProofPaper ↗
Key points
- Due to the limited text information, directly fine-tuning CLIP on video recognition datasets may lead to overfitting and hinder its generalizability.
- In this paper, we propose P2CLIP (Prompt-guided CLIP), which aims to bridge this gap by exploring the impact of prompts across three aspects: category, video, and frame.
- To address this issue, we design Generic-Discriminative Prompt using LLMs to obtain discriminative category embeddings with rich semantic information.
- Then we propose a Knowledge-Guided Temporal Aggregation mechanism, which leverages the expert knowledge derived from LLMs to guide the temporal modeling of videos and encourage a better construction of the joint visual-textual feature space.
Sources (1)
- [1]Revisiting Natural Language Prompts from Large Language Models for Open-Vocabulary Video RecognitionarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 12:15 PM
Vision-language models (VLMs) like CLIP have shown great success in image tasks, however, applying CLIP to open-vocabulary video recognition poses challenges.
Due to the limited text information, directly fine-tuning CLIP on video recognition datasets may lead to overfitting and hinder its generalizability.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 12, 2026Rethinking LLM-Guided Visual Question Answering: Does Deeper Knowledge Integration Really Help?
- Oct 10, 2026Zeroth-Order Fine-Tuning of LLMs in Learnable Low-Rank Subspaces
- Oct 10, 2026SocraticRepair: Planning-Guided Socratic Tutoring for Program Debugging and Misconception Repair
- Oct 10, 2026Toward System One Clinical Intelligence: Parameter-Efficient Fine-Tuning and Counterfactual Explainability of Decision Foundation Models
- Oct 10, 2026Training-Free BioNEL-RAG: Advancing Biomedical Entity Linking through Retrieval-Augmented Generation and Large Language Model
- Oct 10, 2026Vela: User Behavior as Supervision Sparks LLM Intelligence