Learning Visual Speech Representations via Cross-Modal Distillation and Joint Face-Lip Modeling

Key points

• Lipreading remains challenging due to the inherent ambiguity of visual speech cues and their sensitivity to head pose and video resolution.1
• FLARE leverages semantically rich Whisper representations as the acoustic teacher signal, converting them into discrete distillation targets through a random-projection quantizer and pretraining the model with a masked prediction objective.1
• To capture complementary visual information, FLARE employs a dual-stream architecture that jointly processes lip region-of-interest (ROI) and full-face inputs, where the lip stream captures finegrained articulatory motion and the face stream encodes broader facial dynamics.1
• Further analyses demonstrate that joint face-lip modeling improves robustness under degraded video resolution and large head pose variation, while yielding more phoneme-discriminative visual representations.1

Sources (1)

• [1]Learning Visual Speech Representations via Cross-Modal Distillation and Joint Face-Lip Modeling
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 04:18 AM

  In this work, we propose FLARE, a framework for robust visual speech representation learning via cross-modal distillation and joint face-lip modeling.

  Lipreading remains challenging due to the inherent ambiguity of visual speech cues and their sensitivity to head pose and video resolution.

Extractive summary: sentences quoted from the sources.