Byzantine Tolerant Gradient Aggregation for Cross-Silo Federated Learning

Author

Joseph Benjamin, Mohamed bin Zayed University of Artificial IntelligenceFollow

Date of Award

4-30-2024

Document Type

Thesis

Degree Name

Master of Science in Computer Vision

Department

Computer Vision

First Advisor

Dr. Karthik Nandakumar

Second Advisor

Dr. Mohammad Yaqub

Abstract

Model poisoning is an extensively studied threat model in the federated learning paradigm. Though plenty of Byzantine robust aggregation methods show theoretical significance and empirical effectiveness, there exist quintessential issues with existing approaches when adapted for practical application. (i)~With heavy heterogeneity, median/outlier estimation methods become sub-optimal or fail to converge even in the absence of malicious actors i.e.~Byzantines. (ii)~Variation reduction approaches like clipping with bucketing strategies handle heterogeneity with provable convergence, but these inevitably lead to the mixing of harmful updates because of their formulation under harsher settings, leading to poisoning. Also, there are practical difficulties in determining the appropriate hyperparameters, such as assuming the number of Byzantines or setting clipping radius, which are functions of data heterogeneity, model parameter complexity, and optimization techniques. To overcome these limitations, we propose FedRISE aggregation, which uses sparsified gradients. The proposed method involves sign-voting to determine the individual gradients's optimal sign. We incorporate a reputation score for weighing each client's votes during sign selection. Our experiments show that existing robust aggregators fail under severe attacks, while FedRISE demonstrates better robustness under different attacks. In addition, we also introduce a data heterogeneity measurement method that is compliant with the differential privacy principles, which is helpful for a preliminary evaluation of heterogeneity among clients.

Comments

Thesis submitted to the Deanship of Graduate and Postdoctoral Studies

In partial fulfilment of the requirements for the M.Sc degree in Computer Vision

Advisors: Karthik Nandakumar, Mohammad Yaqub

with 2 years embargo period

Recommended Citation

J. Benjamin, "Byzantine Tolerant Gradient Aggregation for Cross-Silo Federated Learning,", Apr 2024.