Federated Learning (FL) presents a promising approach to privacy-preserving, decentralized learning, enabling the collaborative training of models across distributed devices without requiring the sharing of data. However, when applied to Multimodal Federated Learning (MFL), existing federated learning (FL) frameworks face significant challenges due to modality imbalance, where contributions from weaker modalities can hinder the learning process and degrade overall model performance. To address these challenges, we propose FedKoE, a framework that integrates a Mixture-of-Experts (MoE) architecture with Knowledge Distillation (KD) to regulate and enhance the contributions of individual modalities dynamically. Specifically, we integrate a Softmax Gating function to quantify the importance of each modality during training, and use knowledge distillation to transfer predictive knowledge from stronger modalities to weaker ones. We conduct extensive experiments on three benchmark datasets (CREMA-D, KU-HAR, and UCI-HAR) to demonstrate the effectiveness of FedKoE in achieving enhanced performance across modalities under diverse multimodal scenarios, including heterogeneous data and missing modalities. The results demonstrate that our framework consistently enhances the performance of weaker modalities, resulting in a more balanced and robust multimodal learning process. Notably, FedKoE exhibits strong resilience to scenarios where certain modalities are missing or underrepresented, highlighting its generalization capabilities across heterogeneous clients.
