Academia is in turmoil regarding partially or wholly AI-written research. AI-written work is being submitted to journals, but it isn’t clear how much. Publisher, journal, university, and department policies are emerging and fluctuating. Technology develops, rendering recently enacted policies ambiguous, vague, or subject to unintended loopholes. With everything in flux, it’s impossible for norms to develop. There is no general sense of whether and what kind and what degree of AI use by authors is acceptable. Different policies across different institutions means that what may be problematic AI use in one place is acceptable AI use in another, and vice versa. Into this chaos steps forward the vigilantes. The vigilantes have always been there, but now they are newly armed and emboldened. In the everchanging and never ending race between the technology of deployment and the technology of detection, the latter seems to have a temporary lead in the form of Pangram. You can feed a text into Pangram and it will tell you its analysis of whether it is AI generated, and if so, what percentage of it is AI generated. Pangram claims to have a remarkably high detection and low false-positive rates, and cites studies by third parties to support these claims. Yet I have seen plenty of posts by friends and colleagues across social media showing false positives with some of their own work. So user beware. That said, even if Pangram’s claims about its accuracy are correct, it won’t always be helpful to simply know that a text is “AI generated.” This is because there are two further problems. The first problem is ambiguity about what “N% AI generated” means. If someone asks Claude to write them a novel article about a philosophical problem, not even giving it a thesis to work with but asking it to come up with one itself, and that’s the extent of the human’s contribution, the resultant paper will get a “100% AI generated” evaluation. And if someone spends a year researching, writing, workshopping, and revising a paper, and then asks an AI to improve the writing style throughout the paper, the resultant paper will get a “100% AI generated” evaluation, too. Despite their same scores, though, these seem like cases that it might..

The post Careful with those Accusations first appeared on Daily Nous.