AlignAudio: A Unified Dual-Alignment Framework for Noise-Robust Speech-Conditioned Generation

Key points

• Speech-conditioned generation maps speech signals to target audio outputs.1
• Prior end-to-end speech-conditioned generative models achieve high fidelity in clean acoustic conditions, but their performance degrades substantially under environmental noise.1
• Experiments on speech-to-audio (STA) and voice conversion (VC) tasks demonstrate that AlignAudio maintains performance comparable to baselines on clean speech while significantly outperforming them across various noisy conditions.1
• Furthermore, the proposed dual-alignment strategy is applicable to both single- and multi-stream speech conditioning and can be integrated into different flow matching-based generation frameworks and post-training settings.1

Sources (1)

• [1]AlignAudio: A Unified Dual-Alignment Framework for Noise-Robust Speech-Conditioned Generation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 08:15 AM

  To address this, we propose AlignAudio, a noise-robust generative framework based on dual alignment: (i) aligning the representations of clean and noisy speech to preserve semantic cues, and (ii) enforcing generation-level consistency within a flow matching paradigm to improve conditioning fidelity.

  Speech-conditioned generation maps speech signals to target audio outputs.

Extractive summary: sentences quoted from the sources.