On the Risk of Misinformation Pollution with Large Language Models
Document Type
Conference Proceeding
Publication Title
Findings of the Association for Computational Linguistics: EMNLP 2023
Abstract
We investigate the potential misuse of modern Large Language Models (LLMs) for generating credible-sounding misinformation and its subsequent impact on information-intensive applications, particularly Open-Domain Question Answering (ODQA) systems. We establish a threat model and simulate potential misuse scenarios, both unintentional and intentional, to assess the extent to which LLMs can be utilized to produce misinformation. Our study reveals that LLMs can act as effective misinformation generators, leading to a significant degradation (up to 87%) in the performance of ODQA systems. Moreover, we uncover disparities in the attributes associated with persuading humans and machines, presenting an obstacle to current human-centric approaches to combat misinformation. To mitigate the harm caused by LLM-generated misinformation, we propose three defense strategies: misinformation detection, vigilant prompting, and reader ensemble. These approaches have demonstrated promising results, albeit with certain associated costs. Lastly, we discuss the practicality of utilizing LLMs as automatic misinformation generators and provide relevant resources and code to facilitate future research in this area.
First Page
1389
Last Page
1403
Publication Date
1-1-2023
Recommended Citation
Y. Pan et al., "On the Risk of Misinformation Pollution with Large Language Models," Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1389 - 1403, Jan 2023.