Detecting LLM-generated academic writing poses a challenge, as existing detectors are ineffective, particularly for polished text. However, CheckGPT, a novel LLM-content detector, offers a solution by providing accurate and interpretable classification of LLM-generated text in academic writing. It achieves a high classification accuracy of 98-99% for discipline-specific detectors and demonstrates transferability across domains.
Even experienced faculty members and researchers find it highly challenging to identify GPT-generated abstracts. In this regard, CheckGPT outperforms both open-source and commercial GPT detectors in classifying GPT-generated academic writing. This is significant because the usage of ChatGPT in academic writing has been on the rise, and CheckGPT successfully detects GPT-generated content.
When analyzing LLM-generated texts, word-level interpretations fall short due to the complexity of linguistic and semantic features captured by LLMs. Instead, sentence-level analysis provides more comprehensive insights. Language comprehension relies on context, nuance, and syntactic structures, making sentence-level analysis crucial.
By employing Shapley Values, sentence-level analysis reveals distinguishable and unique patterns for GPT-generated texts. Supporting sentences for human texts and GPT texts are located differently in an abstract, further highlighting the distinguishable nature of GPT-generated texts. Additionally, GPT's writing style often involves starting the abstract with a declarative statement to emphasize the paper's focus. Furthermore, a conclusive statement in the last sentence of the abstract is commonly observed, summarizing the findings or contributions of the paper, which aligns with regular ChatGPT conversations.
These interpretation experiments underscore the necessity of employing sophisticated and automated tools, such as deep learning techniques, to effectively detect complex linguistic and presentation patterns in GPT-generated texts.
Reference: https://arxiv.org/abs/2306.05524