The Heterogeneity and Privacy in Cross-Silo Federated Learning

Author

Xiangjian Hou, Mohamed bin Zayed University of Artificial IntelligenceFollow

Date of Award

4-30-2024

Document Type

Thesis

Degree Name

Master of Science in Computer Vision

Department

Computer Vision

First Advisor

Dr. Mohammad Yaqub

Second Advisor

Dr. Samuel Horvath

Abstract

In addressing the twin challenges of data heterogeneity and privacy in Federated Learning (FL), this thesis introduces a dual-faceted approach that not only pioneers advancements in FL methodologies but also contributes significantly to the discourse on privacy preserving techniques within the domain. Our research is divided into two primary foci: improving FL performance amid data heterogeneity through a novel Federated Learning with Partially Personalized (FedPP) method and optimizing privacy-preserving mechanisms to balance efficacy and confidentiality in FL implementations. The first segment of our investigation unveils the FedPP method, an innovative approach designed to mitigate the limitations imposed by the heterogeneity of the data in FL. By integrating the Learning Rate Adjustment (LoRA) technique, FedPP not only surpasses the performance benchmarks of centralized training models, but also sets a new standard for personalized federated learning. This methodological advancement underscores the potential of partial personalization in bridging the performance gap often witnessed in conventional FL settings, thus enhancing model accuracy and efficiency across diverse data distributions. At the same time, our exploration extends into the realm of differential privacy (DP), where we posit the existence of an optimal interplay between the number of local updates and communication rounds, a crucial balance to maximize convergence performance within a predefined privacy budget. Through rigorous theoretical analysis, we delineate the optimal configurations for local steps and communication rounds that significantly improve the convergence bounds of the DP-enhanced ScaffNew algorithm, particularly in the landscape of strongly convex optimization problems. Our theoretical propositions are further corroborated by empirical evidence, establishing a direct correlation between these optimal configurations and various critical parameters, including the DP privacy budget.

Comments

Thesis submitted to the Deanship of Graduate and Postdoctoral Studies

In partial fulfilment of the requirements for the M.Sc degree in Computer Vision

Advisors: Mohammad Yaqub, Samuel Horvath

Online access available for MBZUAI patrons

Recommended Citation

X. Hou, "The Heterogeneity and Privacy in Cross-Silo Federated Learning,", Apr 2024.