Decoding Naturalistic Speech from fMRI with Temporal Modeling and Multimodal Alignment

Key points

• Reconstructing what a person perceives from their brain activity is a stringent test of whether neural recordings carry meaningful information.1
• Naturalistic speech offers a rich testbed for this challenge: it unfolds over multiple timescales, but fMRI's noisy, temporally smeared measurements make this structure hard to recover.1
• These embeddings combine complementary text, audio, and speech representations of the same stimulus segment, enabling its retrieval directly from brain activity.1
• Controlled ablations establish the contributions of temporal modeling and complementary stimulus representations.1

Sources (1)

• [1]Decoding Naturalistic Speech from fMRI with Temporal Modeling and Multimodal Alignment
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 03:17 PM

  We introduce NOEMA, a temporal convolutional decoder that aligns windows of consecutive fMRI measurements with multimodal stimulus embeddings.

  Reconstructing what a person perceives from their brain activity is a stringent test of whether neural recordings carry meaningful information.

Extractive summary: sentences quoted from the sources.