RSAJEV: Training-Free Abstaining Prototype Memory for Eliminating Silent Hallucination in JEV-Based Remote Sensing Classification

Key points

• We retain the slot representation and replace the readout with RSAJEV, a training-free abstaining prototype memory that performs retrieval-based prototype matching: K reference images per class are cached as prototypes via their slot representations under the same prompt, and classification becomes nearest-neighbor retrieval over the prototypes, refined by a shrinkage-Mahalanobis geometric correction.1
• This readout is mechanistically isomorphic to retrieval-augmented generation (RAG) but operates on discriminative decisions; it is the first retrieval-based readout for MLLM slot representations.1
• The retrieval readout reaches 89.43% on the UCM cross-dataset gate with 16 references per class and zero gradient updates, and 96.90% on AID.1
• Its ID-ness scores attain 0.91-0.99 AUROC on open-set evaluation over two datasets, exceeding symbolic confidence (0.38-0.96) in all four open-set settings, restoring a calibratable abstention channel and supporting class addition/removal at zero extra inference cost via offline rescoring of the cached matrix.1

Sources (1)

• [1]RSAJEV: Training-Free Abstaining Prototype Memory for Eliminating Silent Hallucination in JEV-Based Remote Sensing Classification
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 11, 04:27 PM

  Scene classification of remote sensing imagery with multimodal large language models (MLLMs) is formulated as a single-forward-pass decision problem: candidate class names are lettered options in the prompt, a registered answer slot is conditioned on the prompt, and classification is read out by scoring letter tokens with the language-modeling head.

  We retain the slot representation and replace the readout with RSAJEV, a training-free abstaining prototype memory that performs retrieval-based prototype matching: K reference images per class are cached as prototypes via their slot representations under the same prompt, and classification becomes nearest-neighbor retrieval over the prototypes, refined by a shrinkage-Mahalanobis geometric correct

Extractive summary: sentences quoted from the sources.