Fed-DAPT: Federated Dynamic Alignment and Prompt Tuning for Efficient Personalized Vision Transformers

Federated learning (FL) enables multiple clients to collaboratively train a model without exchanging raw data, making it attractive for privacy-sensitive edge computing where edge devices are constrained by bandwidth, energy, and computation. Personalized FL (PFL) further addresses client heterogeneity by learning personalized models; however, deploying large-scale vision transformers (ViTs) in PFL remains challenging under non-IID data due to prohibitive communication and computation costs. To tackle these challenges, we propose Fed-DAPT, a lightweight framework that decouples communication cost from backbone size by freezing the vision–language backbone and communicating only global prompt parameters. Fed-DAPT further integrates variational adaptive inference (VAI) into early-exit transformer blocks on easy samples and an unbalanced optimal transport (UOT) module to align heterogeneous feature distributions across clients for improved robustness. Experiments on CIFAR-10, CIFAR-100, and ImageNet-100 show that Fed-DAPT consistently outperforms strong personalized FL baselines while substantially reducing both communication and computation overhead. In particular, Fed-DAPT achieves up to 87.84%, 58.86%, and 76.72% Top-1 accuracy, improving by up to 3.48 percentage points over the strongest baseline, while reducing per-round communication by over 99% and lowering inference cost by approximately 40% in GFLOPs.