Data-Efficient ConvNeXt Pretraining for High-Resolution Sentinel-1 GRD SAR Representation Learning
We propose SparK-SAR, a sparse convolutional masked pretraining framework built on SparK and ConvNeXt, designed to exploit convolutional inductive biases for S1GRD imagery under constrained compute.
ProofPaper ↗
Key points
- Labeled Sentinel-1 Ground Range Detected (S1GRD) datasets remain scarce, limiting supervised deep learning for land-surface remote sensing, while self-supervised pretraining for this domain has been dominated by Vision Transformer (ViT)-based masked image modeling requiring substantial data and compute.
- On four Copernicus-Bench S1GRD tasks, our finetuned 27.8M-parameter ConvNeXt-RS encoder improves over a non-pretrained baseline by +4.7, +5.0, +5.8, and +1.4 pp on EuroSAT-S1, BigEarthNet-S1, DFC2020-S1, and Flood-S1 respectively, and outperforms the Copernicus-FM ViT (139M parameters, frozen) on three of four tasks at roughly 5x fewer parameters.
- A pretraining data-scaling study reveals task-dependent data efficiency: classification saturates with as little as 10-25% of the SSL4EO-S12 v1.1 corpus, while dense prediction continues to improve significantly up to 25-50%.
- The study also includes a CKA representation analysis and a geographic de-collision experiment.
Sources (1)
- [1]Data-Efficient ConvNeXt Pretraining for High-Resolution Sentinel-1 GRD SAR Representation LearningarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 02:40 PM
We propose SparK-SAR, a sparse convolutional masked pretraining framework built on SparK and ConvNeXt, designed to exploit convolutional inductive biases for S1GRD imagery under constrained compute.
Labeled Sentinel-1 Ground Range Detected (S1GRD) datasets remain scarce, limiting supervised deep learning for land-surface remote sensing, while self-supervised pretraining for this domain has been dominated by Vision Transformer (ViT)-based masked image modeling requiring substantial data and compute.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 15, 2026Code, Test, and Deploy on One Machine with NVIDIA RTX Spark
- Oct 15, 2026Spurious Attention Heads in Vision Transformers: Mechanistic Identification of Annotation-Driven Bias in Medical Chest X-Ray Analysis
- Oct 14, 2026mistralai/Timestral-209M
- Oct 14, 2026The Design Space of Tri-Modal Masked Diffusion Models
- Oct 14, 2026The COUNT: Counting N-grams Unlocks Scaling on Host Memory and Training-Free Adaptation
- Oct 13, 2026How to Rewrite Apache Spark UDFs for GPU Acceleration with AI Agents