M4: Multi-Generator, Multi-Domain, and Multi-Lingual Black-Box Machine-Generated Text Detection

Authors

Yuxia Wang, Mohamed Bin Zayed University of Artificial IntelligenceFollow
Jonibek Mansurov, Mohamed Bin Zayed University of Artificial IntelligenceFollow
Petar Ivanov, Mohamed Bin Zayed University of Artificial IntelligenceFollow
Jinyan Su, Mohamed Bin Zayed University of Artificial IntelligenceFollow
Artem Shelmanov, Mohamed Bin Zayed University of Artificial IntelligenceFollow
Akim Tsvigun, Mohamed Bin Zayed University of Artificial Intelligence
Chenxi Whitehouse, Mohamed Bin Zayed University of Artificial IntelligenceFollow
Osama Mohammed Afzal, Mohamed Bin Zayed University of Artificial IntelligenceFollow

Document Type

Conference Proceeding

Publication Title

EACL 2024 - 18th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference

Abstract

Large language models (LLMs) have demonstrated remarkable capability to generate fluent responses to a wide variety of user queries. However, this has also raised concerns about the potential misuse of such texts in journalism, education, and academia. In this study, we strive to create automated systems that can detect machine-generated texts and pinpoint potential misuse. We first introduce a large-scale benchmark M4, which is a multi-generator, multi-domain, and multi-lingual corpus for machine-generated text detection. Through an extensive empirical study of this dataset, we show that it is challenging for detectors to generalize well on instances from unseen domains or LLMs. In such cases, detectors tend to misclassify machine-generated text as human-written. These results show that the problem is far from solved and that there is a lot of room for improvement. We believe that our dataset will enable future research towards more robust approaches to this pressing societal problem. The dataset is available at https://github.com/mbzuai-nlp/M4.

First Page

1369

Last Page

1407

Publication Date

1-1-2024

Recommended Citation

Y. Wang et al., "M4: Multi-Generator, Multi-Domain, and Multi-Lingual Black-Box Machine-Generated Text Detection," EACL 2024 - 18th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference, vol. 1, pp. 1369 - 1407, Jan 2024.