Exploring the Integration of Large Language Models in Peer Review: Challenges, Opportunities, and Implications

Date of Award

4-30-2024

Document Type

Thesis

Degree Name

Master of Science in Natural Language Processing

Department

Natural Language Processing

First Advisor

Prof. Preslav Nakov

Second Advisor

Prof. Qirong Ho

Abstract

The increasing accessibility of large language models (LLMs) has raised concerns about their potential impact on the peer review process in scientific research. This study explores the challenges and opportunities associated with integrating LLMs into the peer review ecosystem. We establish a framework to examine two scenarios: (1) reviewers using AI for generating entire reviews, and (2) reviewers collaborating with AI in a mixed-authorship setting. Our refined problem formulation focuses on three primary tasks: identifying exclusively AI-generated reviews, detecting transition points in mixed reviews, and assessing the novelty of scientific work. We contribute to this domain by prompting open and closed source LLMs to generate data replicating the defined settings and conducting an in-depth analysis of automatic detectors for identifying machine-generated text. Furthermore, we innovatively repurpose an LLM to serve as an annotator for extracting novelty assessments from existing reviews. The annotated data obtained through this process will be leveraged as distant supervision to train an automated novelty assessment module in future work. Our findings demonstrate that LLMs exhibit moderate reliability in accomplishing this task, highlighting their potential for supporting the development of advanced review analysis tools. This study provides valuable insights and observations to guide future research endeavors in this domain, ultimately aiming to enhance the efficiency and effectiveness of peer review while upholding the principles of scientific inquiry and discourse.

Comments

Thesis submitted to the Deanship of Graduate and Postdoctoral Studies

In partial fulfilment of the requirements for the M.Sc degree in Science in Natural Language Processing

Advisors: Preslav Nakov, Qirong Ho

Online access available for MBZUAI patrons

Link to Full Text

Share

COinS