NusaCrowd: Open Source Initiative for Indonesian NLP Resources

Authors

Samuel Cahyawijaya, Hong Kong University of Science and Technology
Holy Lovenia, Hong Kong University of Science and Technology
Alham Fikri Aji, Mohamed bin Zayed University of Artificial IntelligenceFollow
Genta Indra Winata, Bloomberg
Bryan Wilie, Hong Kong University of Science and Technology
Fajri Koto, INACL
Rahmad Mahendra, INACL
Christian Wibisono, Institut Teknologi Bandung

Document Type

Conference Proceeding

Publication Title

Proceedings of the Annual Meeting of the Association for Computational Linguistics

Abstract

We present NusaCrowd, a collaborative initiative to collect and unify existing resources for Indonesian languages, including opening access to previously non-public resources. Through this initiative, we have brought together 137 datasets and 118 standardized data loaders. The quality of the datasets has been assessed manually and automatically, and their value is demonstrated through multiple experiments. NusaCrowd's data collection enables the creation of the first zero-shot benchmarks for natural language understanding and generation in Indonesian and the local languages of Indonesia. Furthermore, NusaCrowd brings the creation of the first multilingual automatic speech recognition benchmark in Indonesian and the local languages of Indonesia. Our work strives to advance natural language processing (NLP) research for languages that are underrepresented despite being widely spoken.

First Page

13745

Last Page

13819

Publication Date

1-1-2023

Recommended Citation

S. Cahyawijaya et al., "NusaCrowd: Open Source Initiative for Indonesian NLP Resources," Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 13745 - 13819, Jan 2023.