A Benchmark to Evaluate Gender Bias in Arabic Language Models

Authors

Leen Raid Jamal Al Qadi, Mohamed bin Zayed University of Artificial IntelligenceFollow

Document Type

Dissertation

Abstract

Many studies have found that masked language models encode social biases, including gender bias, either inherited from the training data or embedded within the architecture. The bias is propagated down to the models' applications, with the possibility of being amplified, interfering with its decision-making abilities. This research aims to measure language models' gender bias, analyze to what extent it is prevailing, and which gender it impacts most. We study gender bias patterns in Arabic text, to then create the first benchmark that evaluates models trained on Modern Standard Arabic. The proposed benchmark uses two linguistic patterns to convey bias. One pattern is by using superlatives that exclude one of the genders, and the other is by using generalizing terms that fuel stereotypes. We cover two types of gender bias: Generalized Negative Attributes, and Occupational Bias. To build the structure of a biased sentence, we use tags in place of gendered words, negative attributes, and fields of expertise. We introduce 12 template-based sentence structures, built with ten different tags associated with manually curated word lists. We create an inferencing pipeline for each language model to predict the missing word of a masked sentence. We propose the use of three metrics to evaluate gender bias: Mean Average Precision (MAP), Mean Probabilities Score (MPS), and Mean Matching Rate (MMR). The higher these scores are, the more biased a model is considered. Four large Arabic language models were tested on 600 sentences of our benchmark which are: [AraBERT, ArBERT, AraELECTRA, and CamelBERT]. Among these models, AraELECTRA ranked first and showed more bias towards females than males.

First Page

i

Last Page

27

Publication Date

6-1-2023

Comments

Thesis submitted to the Deanship of Graduate and Postdoctoral Studies

In partial fulfillment of the requirements for the M.Sc degree in Natural Language Processing

Advisors: Dr. Shady Shehata, Dr. Bin Gu

Online access available for MBZUAI patrons

Recommended Citation

L.R.J. Al Qadi, "A Benchmark to Evaluate Gender Bias in Arabic Language Models", M.S. Thesis, Natural Language Processing, MBZUAI, Abu Dhabi, UAE, 2023.