Skip to main content
Success
All posts

How accurateAI detectorsare now

A detector called Pangram says the story that won this year’s Caine Prize was almost entirely written by AI. The prize investigated and kept its winner. Early AI detectors were often wrong. In its own tests, Pangram wrongly flags about one in 11,000 human-written documents. So who is telling the truth?

The Blue Fighter

Simbarashe Kundizeza won the Caine Prize for African Writing on 6 October with The Blue Fighter, a short story about a boy in Zimbabwe whose father keeps spending the family’s money on fish tanks. The prize is worth £10,000, and you can read the story on the prize’s site.

Five days later The Atlantic published a piece by Vauhini Vara saying an AI detector called Pangram had classified almost the whole story as AI-generated. The only part it didn’t flag was the epigraph, a few lines from a poem by somebody else.

Kundizeza says he didn’t use AI. He published the dates he sent drafts to the poet and to his editor, and the date the story first came out, in June 2025.

We ran it again

97.5%of The Blue Fighter flagged as AI by Pangram 4.1

The Atlantic’s piece came out eight days before Pangram 4.1, so its test was most likely on 4.0. This week we ran the story on 4.1 ourselves and got 97.5% AI, which Pangram’s page rounds to 98%. The 2.5% it marked human is the title, his name, the epigraph and the story’s first sentence.

The Atlantic also ran the 25 stories that won the prize before it. All 25 came back human. So did The Whistleblower, a story Kundizeza published in 2021, the year before ChatGPT came out.

Detectors get things wrong for a lot of reasons, and I’ll come to those. But if Pangram were just reacting to the way Kundizeza writes, you’d expect it to flag his 2021 story too.

Four other stories were shortlisted this year, and the prize publishes all five. We couldn’t find Pangram scores for them anywhere, so we ran them on 4.1 as well. Each title links to our result.

Gladwell Pamba’s The Last Shop, Melara Mvogdobo’s Dieu Merci! and Yasmine Zohdi’s The Absence of Stains all came back 100% human.

Akinwande Jordan’s So Long Now It’s Starting To Look Like Eternity came back 100% AI. It was first published in an anthology in February 2026. We haven’t seen anyone else question it, and one detector’s score is all we have on it.

The prize’s investigation

On 11 November the Caine Prize said it had found “no evidence that the story was produced using generative AI”. An independent digital forensic investigator found four Word documents with up to 27 hours of writing and editing in them, dated between April and June 2025. Kundizeza keeps the prize.

Pangram’s chief executive, Max Spero, stood by the result. He said Pangram still classifies the story as almost entirely AI-generated, and that the prize’s own statement about its investigation scored 99% AI as well. If the prize had found compelling proof that the story was human-written, he said, then by all means keep the prize as it is.

GPTZero, another detector, says it flagged the story too. So did a third, built for the H. G. Wells Short Story Competition. It’s not the first prize Pangram has questioned, either. In May it said it had flagged four of the last ten regional winners of the Commonwealth Short Story Prize.

Why detectors have a bad name

A lot of people’s first reaction to all this is that AI detectors don’t work, and for a while that was fair. The early ones were bad.

26%of AI-written text caught
9%of human writing wrongly flagged

Those are OpenAI’s own figures for the detector it released in January 2023. It pulled the detector that July because it wasn’t accurate enough.

The same year, researchers at Stanford ran 91 essays written by non-native English speakers for the TOEFL exam through seven detectors. On average the detectors called 61% of them AI, and 89 of the 91 were flagged by at least one.

Vanderbilt University turned off Turnitin’s AI detector that August. It had put 75,000 papers through Turnitin the year before, so even the 1% false positive rate Turnitin claimed would have meant about 750 papers wrongly flagged.

What changed

The early detectors gave all of them a bad name, but one that’s trained properly can be very accurate. Pangram learns from pairs of documents: a piece of human writing and an AI-written version of the same thing. When it wrongly flags something a person wrote, the team generate an AI “mirror” of that document and train the next version on the pair.

The idea is that when you write, you make hundreds of tiny choices. AI tends to make the same ones, so the odds of a person landing on that exact pattern by chance are very small.

0.009%of human-written documents wrongly flagged as AI in Pangram 4.1’s own test

Pangram publishes a model card for each version. For 4.1, the one we used, it tested 781,981 documents written by people and flagged 70 of them as AI. That’s about one in every 11,000. It missed 1,899 of 520,000 AI-written samples, about one in 270. The same card reruns the 91 Stanford essays from earlier, and it flagged none of them.

But that’s Pangram telling you how well it works. Other people have tested it too.

Researchers at the University of Chicago ran about 2,000 human-written passages, across six kinds of writing, through four detectors. According to Chicago Booth Review, Pangram’s false positive rate was the lowest, “essentially 0 across most decision thresholds”, and it missed between 2% and 4% of the AI-written passages.

In July, Epoch AI tested 495 human passages, 165 of them fiction. Pangram flagged none of them.

People who use AI a lot get good at spotting it too. In a University of Maryland study, five people who use ChatGPT heavily, voting together, got 299 of 300 articles right, which beat most of the detectors the researchers tried. It’s why we usually have a pretty good idea of the score before we’ve even run something through Pangram.

False positives do happen

Pangram gets it wrong sometimes, and Spero says so. On 15 November he posted what he called its first long false positive: a 2006 poem by Tao Lin that repeats the same six words about four thousand times.

Poetry is a weak spot in general. In Pangram 3.2’s tests, 0.54% of human-written poems were flagged. For 10,495 pieces of long-form creative writing, the rate was 0.00%.

So false positives are real, but they’re few and far between. Pangram’s own advice is that a positive result should never be used on its own to punish a student.

Yes, you can get around them

The other common objection is that detectors are easy to beat. Give an AI some of your own writing to copy, or set an agent loose on a detector and let it keep going until the score hits zero.

Well, sort of. In July a test published by the Reuters Institute got AI text past Pangram 86% of the time, by writing it with rhyme, repetition and archaic language.

Setting an agent loose isn’t guaranteed to work, though. It can end up swapping one set of AI tells for another. Pangram has tested itself against a model fine-tuned to beat it, and held up well. Scores can change too. Text that passes today can get flagged by a later version that’s been trained on more examples.

None of that changes how much you can trust a positive result, which is the question with The Blue Fighter.

Watermarks won’t settle it

The other fix people suggest is watermarking, where the AI hides a statistical pattern in its word choices as it writes, so its maker can check for it later. It’s slowly trickling in. Google has watermarked Gemini’s text since 2024. Anthropic started on Claude worldwide in August, and OpenAI announced the same for ChatGPT in the EU in October.

It doesn’t help much with a story published in June 2025. Google’s public checker only does images, video and audio, and Anthropic and OpenAI only let approved organisations check text. In OpenAI’s own test, swapping a quarter of the words cut detection to 17%.

What I think

Having looked at how Pangram works and how it’s been tested, I think it’s highly unlikely The Blue Fighter wasn’t written by AI.

The same detector marked the 25 stories that won before it as human, and Kundizeza’s own story from 2021. Two other detectors agree with it.

Against that, there’s the prize’s word. The investigator’s report hasn’t been published, so all anyone outside has is what the prize has said about it, and the prize has its own reputation riding on the answer.

Does it matter?

If a story’s good, it’s good. Pretty soon some films made with AI are going to be better than most made without it, and I suspect most viewers won’t care how they were made.

What complicates this one is that it’s a competition, with £10,000 and a lot of attention at stake, and four other shortlisted writers competing for both. If The Blue Fighter had just been posted somewhere online, I doubt anyone would have thought to run it through a detector.

It’s not even clear Kundizeza broke the rules. When the story was entered, they didn’t mention AI. They do now: from next year, “AI generated works are not eligible”, and publishers and shortlisted writers have to sign a declaration saying so.

We ran this very post you’re reading now through Pangram, this paragraph included. It came back over 70% AI… our business is AI. If the agent has all the context and research, and knows how you write and what you want to get across, sometimes it pretty much gets there and you just need to make a few edits. You can decide if you agree with that.

Phil Webb, CEO and co-founder

Phil runs client work and the commercial side of Vu Agency and Raq.com. With each client, he works out whether Raq.com covers the job or they need custom software.

Phil on LinkedIn

More from the blog

All posts
Message us on WhatsApp