Machine-learning tool outperforms existing AI detectors in identifying chemistry papers written by the chatbot ChatGPT.
A recent study published in Cell Reports Physical Science reveals that a specialized machine-learning tool can accurately identify chemistry papers written using the chatbot ChatGPT. This tool, which surpassed the performance of existing AI detectors, could prove invaluable in helping academic publishers detect papers created by AI text generators. The study’s co-author, Heather Desaire, emphasizes the importance of tailoring software to specific types of writing, suggesting that this approach could enhance the development of AI detectors.
The Elements of Style:
Desaire and her colleagues initially introduced their ChatGPT detector in June, focusing on Perspective articles from the journal Science. By utilizing machine learning, the detector analyzes 20 writing style features, including sentence length variation, word frequency, and punctuation usage, to determine whether a piece of text was written by an academic scientist or ChatGPT. The results showcased the ability to achieve a high level of accuracy using a small set of features.
How ChatGPT and AI Tools Could Impact Scientific Publishing:
In the latest study, the detector was trained on the introductory sections of papers from ten chemistry journals published by the American Chemical Society (ACS). The researchers selected the section as it is relatively easy for ChatGPT to write when provided with access to background literature. The tool was trained on 100 human-written introductions and then tasked with generating 200 introductions in ACS journal style using ChatGPT-3.5. The tool achieved 100% accuracy in identifying ChatGPT-3.5-written sections based on titles and 98% accuracy when based on abstracts. The tool also performed well with ChatGPT-4, the latest version of the chatbot. In comparison, existing AI detectors, such as ZeroGPT and the text-classifier tool produced by OpenAI, performed significantly worse.
Wider Issues:
Debora Weber-Wulff, a computer scientist specializing in academic plagiarism, commends the authors’ use of stylometrics to detect AI-generated writing, highlighting the novelty of this approach. However, she emphasizes that the use of ChatGPT in academia is driven by other issues, such as the pressure on researchers to produce papers quickly and the devaluation of the writing process. AI-detection tools alone cannot address these underlying problems and should not be viewed as a panacea for social issues within academia.
The development of a machine-learning tool capable of accurately detecting AI-generated science papers marks a significant advancement in the field of text analysis. By tailoring software to specific types of writing, researchers have achieved higher levels of accuracy in identifying AI-generated content. However, it is crucial to recognize that AI-detection tools cannot solve the broader challenges faced by academia, such as the pressure to publish quickly. Nevertheless, this research opens up new possibilities for improving the detection of AI-generated content in scientific publishing.
Bob Thompson is our in-house Home and Garden, Energy, and Gaming news writer. Bob is keenly aware of the need to recycle. Bob has written for many online publications over the course of his writing career, before joining our team here at DailyTechFeed.
