Takis Konstantopoulos, University of Liverpool, is joined again by George Kesidis, Pennsylvania State University, for their second column in a series on the impact of AI on academia. They write:
In our article in the April/May 2024 issue, we touched upon how generative AI impacts on higher education in arguably absurd ways. But what about research? AI is a tool that can be of use to the scientific researcher. Large Language Models (LLMs) like ChatGPT can quickly distill information available online and respond with good grammar and syntax, produce images, and produce code which performs specific functions. Even computer-assisted theorem proving (dating back to the 1950s) can be attempted by AI, but certainty or rigor is not guaranteed. But at present can AI create interesting theorems? Even if, in the future, it could, is it clear that such an ability may replace the modern (human) scholar, as many believe?
It’s important to understand the limits and potential failings of any tool as it evolves, despite the hype, and AI is no exception. Indeed, it has been demonstrated how an AI may confidently produce incorrect responses [1], or “hallucinations” for LLMs [2], and demonstrably not “understand” certain words it produces. AI, by its nature, immediately poses serious ethical issues and invites fraud. It is difficult to consider the role of AI without considering these issues. For example, generative AIs have recently been used to “co-author” papers and to peer review them. In this article, we focus on such ethical issues, but to do so we first need to discuss the (degradation of) ethical standards of research in the years prior to the emergence of ChatGPT.
Some aspects of modern research ethics.
Problems such as data fabrication [3] and falsification [4, 5] are far from new. “Publish or perish” and research by “least publishable units” have been practiced for some time. The threat of such corruption is presently acknowledged [6, 7, 8]. To ostensibly address this, universities and research organizations now require mandatory research ethics training*, research corruption is endemic today nevertheless [9].
Just as the industrialization (or “democratization”) of higher education has led to the dramatic increases in student throughput, the industrialization of scientific research has dramatically increased the rate of production** of research articles [10]. With such mechanization, fueled by intense research funding, comes bureaucracy and bean counting, where those workers who produce more beans, as measured by metrics, receive greater reward. Not long ago, research metrics were nascent and the range of salaries among academics was narrow compared to the present. Metrics do translate into real money even though many believe they’re just supposed to measure scholarly quality.
Boosting metrics (e.g., H-index) has become a raison d’être serving not only those who seek higher salaries but also academic administrators who do not wish to “waste” time to try to understand an iota of the employees’ research. Consequently, the peer review process is being compromised (as alleged in [11, 12]); authorship is being boosted through paper mills [13, 14]; co-authorship and co-citation cartels are being formed [15, 16]. Quid pro quos may also play-out in promotion within scholarly organizations and in grant proposal panels. H-index may be reported for priority work-visa applications in the US.
Enter generative AI
Like other technologies, generative AI can support legitimate research and development when guided by human expertise and creativity. However, AI also facilitates cheating. With generative AI, dubious research articles can be easily produced and sold through paper mills. Even before AI, some highly decorated and very well paid researchers maintained a publication rate of a few dozens of papers per year; with AI, this rate could easily double or triple. Given that universities no longer judge the content of the paper but merely quote a number (metric) associated with it, AI-assisted or generated articles may quickly become highly profitable.
Recent reports indicate that ChatGPT-3 has been listed as a co-author on some research articles [17] and even served as a peer reviewer [18]—something we’ve experienced personally. This highlights how AI is being used to partly or fully replace both authors and referees. The shift in universities, as noted earlier, to metrics-based assessments only worsens the impact of AI-assisted fraud.
Some may argue that ChatGPT and similar tools—whether for text, images, videos, web searches, computations, or graphics—are as deserving of authorship as some human co-authors today. This raises a critical question: has the standard of research co-authorship become so diluted that such tools now meet the criteria? Imagine a generative AI achieving an H-index of 100, becoming a fellow of a learned society, securing millions in government funding as a principal investigator, and being awarded a chaired professorship at a major university. Would such absurdity prompt genuine reforms in research practices and evaluation? Unfortunately, we fear it would not. Long before the AI era, Adler, Ewing, and Taylor [19] warned us that research evaluation cannot be reduced to simple metrics. Now, we face new challenges that could inflict unprecedented damage.
Some academics, particularly those inclined towards modern-style administration, argue that using metrics and AI in research evaluation is a step towards fairness, advocating for the complete elimination of traditional informed judgment. However, this shift to bean counting can be seen as driven by laziness and a disdain for genuine critical thinking. AI creates an ideal environment for those who have gained power without true scholarly effort. While alarmists claim AI will surpass humans, this is presently unfounded; however, its impact on research, and research ethics, is already evident. Without intervention, not from university administrators, but from active and honest scholars, we risk academia entering a darker phase than the one Peter Fleming described [20].
References
[1] D.J. Miller, Z. Xiang and G. Kesidis. Adversarial Learning and Robust AI. Cambridge University Press, Sept. 2023.
[2] J. Raghuram, G. Kesidis and D.J. Miller. A Study of Backdoors in Instruction Fine-tuned Language Models. https://arxiv.org/abs/2406.07778
[3] The Cyril Burt Affair. http://intelltheory.com/intelli/the-cyril-burt-affair
[4] D. Huff. How to Lie with Statistics. W.W. Norton & Co., 1954.
[5] Z. Hitzig and J. Stegenga. The Problem of New Evidence: P-Hacking and Pre-Analysis Plans. Diametros 17(66): 10–33, 2020. https://doi.org/10.33392/diam.1587
[6] Committee on Publication Ethics (COPE). https://publicationethics.org
[7] NSF. Responsible and Ethical Conduct of Research. https://new.nsf.gov/policies/responsible-research-conduct
[8] US Dept. of Health and Human Services. The Office of Research Integrity. https://ori.hhs.gov
[9] J. Brainard. Fake scientific papers are alarmingly common but new tools show promise in tackling growing symptom of academia’s “publish or perish” culture. Science, 9 May 2023. https://www.science.org/content/article/fake-scientific-papers-are-alarmingly-common
[10] The number of papers over time. https://www.researchgate.net/figure/The-number-of-papers-over-time-The-total-number-of-papers-has-surged-exponentially-over_fig1_333487946
[11] Evidence Puts Doubts on the IEEE/ACM’s Investigation. https://huixiangvoice.medium.com/evidence-put-doubts-on-the-ieee-acms-investigation-991a6d50802a
[12] Public Announcement of the Results of the Joint Investigative Committee (JIC); Investigation into Significant Allegations of Professional and Publications Related Misconduct. https://www.sigarch.org/wp-content/uploads/2021/02/JIC-Public-Announcement-Feb-8-2021.pdf
[13] K. Krämer. Publishers grapple with an invisible foe as huge organised fraud hits scientific journals. Chemistry World, 25 May 2021. https://www.chemistryworld.com/news/publishers-grapple-with-an-invisible-foe-as-huge-organised-fraud-hits-scientific-journals/4013652.article
[14] R. Richardson. Engineering the world’s highest cited cat, Larry. July 18, 2024. https://reeserichardson.blog/2024/07/18/engineering-the-worlds-highest-cited-cat-larry/
[15] W. Knight. The AI research paper was real. The “co-author” wasn’t. Wired, 21 Feb. 2021. https://www.wired.com/story/ai-research-paper-real-coauthor-not/
[16] E. Catanzaro. Citation cartels help some mathematicians–and their universities–climb the rankings. Science, 30 Jan. 2024. https://www.science.org/content/article/citation-cartels-help-some-mathematicians-and-their-universities-climb-rankings
[17] C. Stokel-Walker. ChatGPT listed as author on research papers: Many scientists disapprove. Nature, 18 Jan. 2023. https://www.nature.com/articles/d41586-023-00107-z
[18] D.S. Chawla. Is ChatGPT corrupting peer review? Telltale words hint at AI use. Nature, 10 April 2024. https://www.nature.com/articles/d41586-024-01051-2
[19] R. Adler, J. Ewing and P. Taylor. Citation statistics. Statistical Science 24(1), 1–14, 2009. https://arxiv.org/pdf/0910.3529
[20] P. Fleming. Dark Academia: How Universities Die. Pluto Press, May 2021.