FedRAPP: Federated Test-Time Prompt Tuning for Vision-Language Models with Reliability-Aware Prompt Personalization

Key points

• However, existing federated prompt learning methods are designed primarily for supervised settings with labeled client data and assumed relatively stable training distributions.1
• In this work, we explore Source-Free Federated Test-Time Prompt Tuning (SF-FTTPT), where distributed clients collaboratively adapt soft prompts on unlabeled test streams without access to source data.1
• To address this challenge, we propose FedRAPP, a personalized aggregation framework that combines reliability-aware client assessment, server-side exponential moving-average prompt memory, and similarity-guided peer aggregation.1
• Extensive experiments across nine CLIP benchmark datasets and ImageNet variants demonstrate that FedRAPP consistently outperforms standard federated aggregation baselines under heterogeneous client distributions while maintaining the efficiency advantages of prompt-only adaptation.1

Sources (1)

• [1]FedRAPP: Federated Test-Time Prompt Tuning for Vision-Language Models with Reliability-Aware Prompt Personalization
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 04:29 PM

  Federated prompt learning enables vision-language models like CLIP to collaboratively adapt lightweight soft prompts across distributed clients without sharing raw data.

  However, existing federated prompt learning methods are designed primarily for supervised settings with labeled client data and assumed relatively stable training distributions.

Extractive summary: sentences quoted from the sources.