Learning Multimodal Sensor Fusion with JEPA Representations for Breathing Activity and Pose Recognition
A shared fusion model uses modality queries, fixed sensor slots, and explicit availability to classify breathing activity and pose from all seven non-empty sensor subsets.
ProofPaper ↗
Key points
- Predictive video representations do not by themselves specify how a multimodal system should handle missing sensors.
- We transfer frozen V-JEPA2 features to a set of heterogeneous sensors comprised by RGB video, depth video, and radar time series.
- The contribution is a controlled evaluation of subset supervision, pooling capacity, and representation transfer on 39 OMuSense-23 subjects with five subject-group folds and three seeds.
- Radar probes support transfer beyond pooled wavelet pixels, 64-token inference approximately preserves mean activity performance, and the tested Consensus-JEPA loss adds no measured benefit against its exact control.
Sources (1)
- [1]Learning Multimodal Sensor Fusion with JEPA Representations for Breathing Activity and Pose RecognitionarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 11, 08:13 PM
A shared fusion model uses modality queries, fixed sensor slots, and explicit availability to classify breathing activity and pose from all seven non-empty sensor subsets.
Predictive video representations do not by themselves specify how a multimodal system should handle missing sensors.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 11, 2026Vision-Predictor-Action: Towards Efficient Embodied AI via Neuro-Symbolic JEPA Predictors
- Oct 11, 2026World Modeling through Spectral Alignment
- Oct 11, 2026LeHami: Symplectic Latent Dynamics for World Models
- Oct 10, 2026MORE: Learning Motion-Oriented Registers for Driving World Models and Planning
- Oct 9, 2026Sparse-TD-JEPA: Improving Zero-Shot Reinforcement Learning With Sparse Latent Task Representations
- Oct 9, 2026RSI-Anything: Unlocking Recursive Self-Improvement with World Models