You’re refereeing a paper and you come to suspect it was substantially and/or illicily written by an LLM. What should you do? My suggestion: Keep in mind the possibility that you are mistaken as well as the possibility that the AI use falls within the journal’s rules and was disclosed during the submission process. Pause your work refereeing the piece, as your suspicions of illicit AI use may unduly negatively influence your assessment of the submission. Gather a few examples of passages that strike you as AI-written, briefly explaining why. (Do not feed the manuscript into an AI-detector, for two reasons: first, doing so may violate the norms of confidentiality referees are expected to abide by, and second, such detectors appear to currently be of questionable accuracy.) Write to the journal’s editor or managing editor expressing your concerns and sharing the evidence you’ve gathered. With luck, the editor will look into the matter and either agree with you and relieve you of refereeing the piece, or the editor will convince you that you are mistaken and you can proceed to finish refereeing the paper. But what if you aren’t so lucky? One associate professor of philosophy wrote in with the following: I recently accepted a referee invitation for a good generalist journal. I was completely convinced after about 20 minutes with the submission that it was mostly (if not entirely) written using AI tools of some sort. The paper was in my areas of research. The editor disagreed (or at least did not share my level of confidence), and so I was left with the awkward task of writing (brief) comments for the “author” to go along with my rejection. For obvious reasons, this seemed to me to be an obvious waste of everyone’s time. Nothing like this has happened to me before, and I referee fairly regularly.  The professor is curious about the frequency with which reviewers are finding themselves in disagreement with journal editors and/or other reviewers on particular cases. Has this happened to you? What did you end up doing? How should referees and editors navigate disagreement on this?

The post If You Think You’re Refereeing an AI-Written Submission, What Should You Do? first appeared on Daily Nous.