Quantitative Analysis of AI-Generated Texts in Academic Research: A Study of AI Presence in arXiv Submissions using AI Detection Tool

Introduction

Since ChatGPT's release, artificial intelligence has significantly impacted developments in natural language understanding (NLU) and natural language generation (NLG) [1]. Academic journals are among the many sectors that have felt the effects of ChatGPT's influence. Academic discourse and approaches increasingly include AI as it advances. An increase in AI development affects the technical research articles on arXiv, and this study examines this trend in detail. Natural language generation (NLG) models developed more recently have significantly improved the control, variety, and quality of text generated by machines. Phishing, disinformation, fraudulent product reviews, academic dishonesty, and toxic spam all exploit NLG models' ability to generate novel, manipulable, human-like text at high speeds and efficiencies [1-4]. Generative models like ChatGPT have recently attracted much attention due to their ability to produce material resembling human writing, images, and more. ChatGPT, an OpenAI-developed variation of the widely used GPT-3 language model, can be trained to generate conversational text, translate text, and even create new languages [5]. Despite generative models like ChatGPT having come a long way in producing natural language, there is still no straightforward method to distinguish machine-written text from human-written content. This is true even though ChatGPT and other generative models. When it comes to content moderation and other similar applications, this is crucial for detecting and removing harmful information and automated spam [6].

Figure 1 demonstrates how arXiv distributes different papers submitted after 2019 to 2023. The vertical axis represents the number of documents in each category, with other colors or bars indicating categories. This graph shows academic research interests and trends during that time. Popularity and/or academic commitment may be suggested by categories with greater activity. The proportional representation of the submissions in various types of work also points to new intellectual orientations. Taking these data into account alongside previous patterns and external conditions could provide a clue as to the evolution of academic research and its new directions in specific research areas [7].

Figure 1: Papers Submitted by Categories on arXiv After 2019 [7]

The data indicates that the top three categories significantly increased in the number of published papers from 2019 to 2023. The biggest increase was in computer science (200.42%), then physics (44.68%) and then mathematics (22.04%). Artificial intelligence writing tools like ChatGPT may have been responsible for the rapid development of academic publications in certain fields, which could be linked to these exponential growths. The theory aligns with the broader trend of new technology in the publishing of academic articles, which means that AI could be a game-changer as far as quality of research articles in these areas are concerned.

                                                Table 1: Influence of AI Text Generation Tools on arXiv Submissions

Generative models like ChatGPT have recently attracted much attention due to their ability to produce material resembling human writing, images, and more. You can train ChatGPT, an OpenAI-developed variation of the widely used GPT-3 language model, to generate conversational text, translate text, and even create new languages [8]. There still needs to be a straightforward way to tell machine-written text from human-written content, even though generative models like ChatGPT have come a long way in producing language that sounds natural. This is true even though ChatGPT and other generative models. When it comes to content moderation and other similar applications, this is crucial for detecting and removing harmful information and automated spam [9].

The purpose of this study is to quantitatively examine the originality.ai AI-generated text identification tool using a dataset that the researchers have created themselves arXiv submissions. To achieve this, the researchers will scour arXiv.org for literature covering three distinct fields. This study is different from the other studies published in that way; it uses a wide range of text sizes, type and formatting. The next step is to try the instrument out and record the outcomes. Below are the main points that were identified in the summary of the study.

• Collecting articles of three different fields arXiv submissions and form a dataset to use for analysis.

• Reporting performance of originality.ai for detection of AI content arXiv submissions. Here is how this article is organized: While Section 2 provides a brief overview of the pertinent literature, Section 4 delves into the results and observations. Lastly, the conclusions of the study and directions for future research are addressed in the conclusion section of the research.

Literature Review

Evolution of AI in Research

AI has made a huge impact on the research landscape. It was first used for simple tasks, but it has developed into more complex tasks thanks to the use of algorithms with increased complexity and by the increased computing power [10]. AI now can analyze hugely large amounts of data, detect patterns, and even create new ideas [11]. The benefits of AI for data analysis are not the only impact that has been demonstrated in important deep learning studies [12,13]. In many research fields, AI can now be applied to speed up research and find new discoveries and innovations [14,15].

AI in Scientific Publishing

The applications of AI in scientific publishing have changed drastically. Now it is becoming an important tool in the writing process and platforms such as ChatGPT and Grammarly are supporting researchers in writing and editing [16]. For instance, AI helps from the first draft of writing to make it more structured. Over the years AI writing has become more accurate and reliable for scientific publications [17].

About the Study on AI in arXiv

This study examines how AI influences both the quantity and quality of papers on arXiv. This study includes use of Originality. AI’s AI detection tool to check how many arXiv papers are likely to be written by AI. This study helps us understand how AI is changing academic research and shows the need for strong methods to make sure these papers are original and trustworthy.

Materials and Methods

Originality.AI's detection model is trained on one million pieces of textual data labeled 'human-generated' or 'AI-generated.' During testing, the model was evaluated on documents generated by various artificial intelligence models, including GPT-3, GPT-J, and GPT-NEO (20 thousand data points each). And the result is that the model successfully identified 94.06% of the text created by GPT-3, 94.14% of text written by GPT-J, and 95.64% of text generated by GPT-Neo. The results show that the more powerful the models like GPT-J/3, the harder it is for the model to recognize that the human or AI is writing [18,19]. After training, the model takes text as input and determines whether it is likely to have been generated by AI or not.

Figure 2: Implementation Framework for Analysis

Originality.AI's AI Detection Tool is an advanced tool that can distinguish AI-generated content from human-written text. It uses a special version of the BERT model that’s good at finding AI written content [20]. Originality.AI's AI Detection Tool can accurately identify AI-written text with more than 98% accuracy. This high accuracy comes from thorough testing and updates to keep up with new AI writing models. The tool has been carefully developed and improved to address the increasing use of AI in writing content [21,22].

Figure 3: Confusion Matrix on a GPT-4 Human Dataset Test [21]

The analysis involved using Originality.AI powered AI content detection tool to identify AI-generated content from human content. This tool uses advanced AI algorithms to identify potential AI writing and offers a quantitative measure of the extent to which these papers may have been written with AI. In addition, a validation process was carried out to evaluate the accuracy of the tool, minimising false positives [21].

Dataset Collection

In this study, we collected information from 13,000 research papers from the arXiv repository. This collection was selected as it spans a variety of academic fields, which helps to grasp the influence of AI on academic writing. The papers we read were selected based on the date they were published, so that we could observe the impact of AI on writing since the advent of ChatGPT.

Selection Criteria

We started with a set of 60,000 papers scraped from arXiv, a preprint repository for papers on many research areas [23]. The selection criteria consisted of selecting papers according to their relevance to the study's focus of the impact of AI on academic writing. Out of this larger set of papers, a sub-set of 13,000 papers were further culled based on different fields and criteria described in our research.

Size and Characteristics

The final dataset is composed of 13.000 papers from arXiv, which represents a wide and complete set of academic research papers. The papers address a variety of disciplines and offer a wide range of research topics and approaches. The size of the data set provides statistical significance for analysis and an opportunity for trends across courses to be explored.

Figure 4: Total Papers Downloaded from arXiv.org

Data Preprocessing

Several steps were taken to preprocess the data, ensuring the quality and relevance of the papers. This involved tidying the data, ensuring that no blank or irrelevant documents existed and that the formatting of the documents was standardized for easier working with, and splitting the papers into categories that were relevant to the study and specific to certain areas of the research. To analyze the data accurately, it was necessary to use the Originality, which required preprocessing.AI tool.

Results and Discussions

Using Originality.AI's detection tool, we scored the papers for AI content and noticed a clear increase in AI-written papers. This method helps us see how AI tools are being used more in academic writing.

Figure 5: Percentage of Papers Likely Generated by AI After 2019, Scored by Originality.AI Detection Tool

It became clear from the graph how much use is made of AI in technical writing in the past. The number of AI written papers increased from 3.61% to 6.22% in one year after the launch of ChatGPT in November 2022, as evidenced by a noticeable increase. The trend is evident as more papers are being published since the launch with a high score from AI, as highlighted. This indicates that the use of AI is transforming academic writing. This increase in the upward trend brought up another question on the impact of AI on writing in various categories. A detailed analysis of AI score has been undertaken to understand the impact of AI across Computer Science, Mathematics and physics.

Figure 6: Percentage of Papers that are AI-Generated After 2019 by Category with Respect to AI Score

Our detailed study highlights that AI's role in academic fields, especially in computer science, is quite significant. Starting from the year 2019, there is a gradual increase from 3.17% to 3.31% by the launch of GPT-3. The launch of GPT-3 has a gradual impact, with a slight increase to 4.38% until the launch of ChatGPT. However, after the launch of ChatGPT a significant impact can be observed. The sharp rise reaches 7.37% until the end of the year 2023. This trend supports the idea that using AI for writing is impacting various academic fields, with the effect being especially strong in computer science. The trend is not significant for Physics and Mathematics. The variation in these trends could be influenced by the limitation of AI detection tools, especially in the fields with constant use of numbers and equations.

Conclusion

As AI becomes more common in research papers, there is growing concern about the impact it may have on the originality and integrity of academic papers. With the help of AI, more papers are being written, and it is crucial to consider how to ensure these papers remain honest and free from unintended biases or inaccuracies. Additionally, there's a risk that AI could make research less diverse and limit creative thinking, as it might lead to similar styles and ideas being repeated. This necessitates attention to and guidelines on how AI can be used to augment, and not diminish, the quality and diversity of academic work. In the following study, we examine how Originlity.AI generated content can be identified using AI's AI detection tool. It emphasizes how effective this tool is in preserving content authenticity and clarifies the importance of research originality and honesty in the use of this tool. In the future, the ability to detect AI-written content will be essential for maintaining honesty and authenticity of academic work. As AI grows, tools like Originality.AI will be key to making sure research stays authentic and trustworthy.

Availability of Data and Materials

Data will be provided on request. It is also publicly available.

References

1. Baki, S., Verma, R., Mukherjee, A., & Gnawali, O. (2017, April). Scaling and effectiveness of email masquerade attacks: Exploiting natural language generation. In Proceedings of the 2017 ACM on Asia conference on computer and communications security (pp. 469-482).
2. Shu, K., Wang, S., Lee, D., & Liu, H. (2020). Mining disinformation and fake news: Concepts, methods, and recent advancements. In Disinformation, misinformation, and fake news in social media: Emerging research challenges and opportunities (pp. 1-19). Cham: Springer International Publishing.
3. Stiff, H., & Johansson, F. (2022). Detecting computer-generated disinformation. International Journal of Data Science and Analytics, 13(4), 363-383.
4. Dehouche, N. (2021). Plagiarism in the age of massive Generative Pre-trained Transformers (GPT-3). Ethics in Science and Environmental Politics, 21, 17-23.
5. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8), 9.
6. Fortuna, P., & Nunes, S. (2018). A survey on automatic detection of hate speech in text. Acm Computing Surveys (Csur), 51(4), 1-30.
7. “Monthly Submissions.” Accessed: Feb. 09, 2024. arXiv [Online].
8. Kashnitsky, Y., Herrmannova, D., De Waard, A., Tsatsaronis, G., Fennell, C. C., & Labbé, C. (2022, October). Overview of the DAGPap22 shared task on detecting automatically generated scientific papers. In Proceedings of the ThirdWorkshop on Scholarly Document Processing (pp. 210-213).
9. Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltýnek, T., Guerrero-Dib, J., Popoola, O., ... & Waddington,L. (2023). Testing of detection tools for AI-generated text.International Journal for Educational Integrity, 19(1), 1-39.
10. Burdisso, S. G., Errecalde, M. L., & Montes y Gómez, M. (2021). Using text classification to estimate the depression level of reddit users. Journal of Computer Science & Technology, 21.
11. Wu, Y., Guan, S., & Wang, G. (2023). Drug effect deep learner based on graphical convolutional network. In Machine learning and deep learning in computational toxicology (pp. 83-140). Cham: Springer International Publishing.
12. Akram, A., Rashid, J., Jaffar, M. A., Faheem, M., & Amin,R. U. (2023). Segmentation and classification of skin lesions using hybrid deep learning method in the Internet of Medical Things. Skin Research and Technology, 29(11), e13524.
13. Akram, A., Rashid, J., Jaffar, A., Hajjej, F., Iqbal, W., & Sarwar, N. (2024). Weber law based approach for multi-class image forgery detection. Computers, Materials, & Continua, 78(1), 145.
14. Akram, A., Ramzan, S., Rasool, A., Jaffar, A., Furqan, U., & Javed, W. (2022). Image splicing detection using discriminative robust local binary pattern and support vector machine. World Journal of Engineering, 19(4), 459-466.
15. Akram, A., Khan, I., Rashid, J., Saddique, M., Idrees, M., Ghadi, Y., & Algarni, A. (2024). Enhanced steganalysis for color images using curvelet features and support vector machine. Computers, Materials, & Continua, 78(1), 1311.
16. Grimaldi, G., & Ehrler, B. (2023). AI et al.: Machines are about to change scientific publishing forever. ACS energy letters, 8(1), 878-880.
17. Mohammed, O., Sahib, T. M., Akhtom, D. A., Hayder, I. M., Salisu, S., & Shahid, M. (2023). ChatGPT Evaluation: Can It Replace Grammarly and Quillbot Tools?. British Journal of Applied Linguistics, 3(2), 34-46.
18. Chaka, C. (2023). Detecting AI content in responses generated by ChatGPT, YouChat, and Chatsonic: The case of five AI content detection tools. Journal of Applied Learning & Teaching, 6(2), 94-104.
19. “How Does AI Content Detection Work? – Originality.AI.” Accessed: Feb. 09, 2024.
20. Müller, M., Salathé, M., & Kummervold, P. E. (2023). Covid-twitter-bert: A natural language processing model to analyse covid-19 content on twitter. Frontiers in artificial intelligence, 6, 1023281.
21. “AI Content Detector Accuracy Review + Open Source Dataset and Research Tool – Originality.AI.” Accessed: Feb. 09, 2024. [Online].
22. Akram, A. (2023). An Empirical Study of AI-Generated Text Detection Tools. Adv Mach Lear Art Inte, 4(2), 44-55.
23. “arXiv Dataset.” Accessed: Feb. 09, 2024. [Online].