From Spectrograms to Shallow Speech SSL: Audio-Frontend Design for Generator-Disjoint and Low-Latency Audio--Visual Deepfake Detection

Key points

• We study audio-frontend design in a fixed MRDF (Modality-Regularization-based DeepFake)-based pipeline with AV-HuBERT visual processing and an MSCT (multi-scale cross-modal transformer) fusion encoder.1
• We compare spectrogram-style log-filterbank features with frozen wav2vec 2.0 and WavLM representations, dual-SSL fusion, executed depth, and shallow-layer aggregation.1
• Shallow execution reduces measured inference latency relative to full-depth wav2vec 2.0 across the tested input durations; first-second evaluation reaches $86.85%$ macro AUROC.1
• Layer-weight analysis shows greater emphasis on early speech layers, while removing the selector's visual input has little effect on detection performance.1

Sources (1)

• [1]From Spectrograms to Shallow Speech SSL: Audio-Frontend Design for Generator-Disjoint and Low-Latency Audio--Visual Deepfake Detection
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 07:07 PM

  Video-mediated conversational systems rely on synchronized speech and facial video, making audio--visual input integrity important for reliable perception.

  We study audio-frontend design in a fixed MRDF (Modality-Regularization-based DeepFake)-based pipeline with AV-HuBERT visual processing and an MSCT (multi-scale cross-modal transformer) fusion encoder.

Extractive summary: sentences quoted from the sources.