A brief discussion with colleagues about AI detectors made me decide to experiment by putting an AI detector to the test to see if it could spot AI text.

The discussion was after a recorded podcast with Stephen Herrera, Stephanie Clerkin and David Horrigan, where we discussed AI detectors. One of the AI detectors mentioned was ZeroGPT, which touts itself as “the most Advanced and Reliable Chat GPT, GPT4 & AI Content Detector”.

Since I have a few writing samples to choose from (🤣), I thought I would try a couple of examples to see how ZeroGPT did in detecting whether content was AI or not. I conducted three examples in all: one with content totally written by me, and two that were written by AI.

Test #1: Content written by me.

For this test, I chose this blog post I wrote a couple of years ago comparing the rise and fall of the BlackBerry to the emergence of LLMs and GenAI in eDiscovery. I loaded in the plain text of the title and almost the entire blog post – down to (but not including) the paragraph beginning with “P.S.”, which seemed like a large clue that this was written by a person. What were the results?

As you can see by the screen shot above, ZeroGPT identified the text as human written. Good job! However, it didn’t identify all the text as human written: it said that 5.76% of the content was generated by AI GPT and it flagged a couple of paragraphs (including the one highlighted above) as AI-generated. They were not AI generated.

Test #2: Content written by NotebookLM from Google

For this test, I chose the Briefing Document portion of this blog post about DeepSeek that I wrote back in January. This was a post where I loaded 15 articles into NotebookLM and had it generate a briefing document. The text I loaded for this one was the paragraph starting with “DeepSeek: AI’s Sputnik Moment” (but not the following text in parenthesis) down through the “Conclusion” paragraph. As I noted in the post, this text is all AI-generated. What were the results?

Eh, could be better! As you can see by the screen shot above, ZeroGPT also identified this text as human written. Not so good a job! It did flag the first paragraph, the Conclusion and one other section as AI GPT generated, but only found 17.17% of the total text to be AI-generated. Whoops!

Test #3: Content written by ChatGPT

Since ZeroGPT calls out its detection capabilities with ChatGPT, I thought I would test with an example from there. I had recently asked ChatGPT 4o about the validation protocol in the In re Broiler Chicken Antitrust Litigation case from 2018 that had been established by Special Master Maura R. Grossman in that case (sorry, Maura, ChatGPT failed to acknowledge you as the creator of the protocol), and it gave me about 500 words on the topic (text of the ChatGPT output I loaded into ZeroGPT available here). So, this text is also all AI-generated. What were the results?

Well…better, but still not great. It flagged the first paragraph and selected sentences, hedged its determination as “Most Likely Human written, may include parts generated by AI/GPT”, but still only found 27.69% of the total text to be AI-generated. Once again, most of the text was not flagged as AI-generated, even though it was.

This was, admittedly, not a scientific case of putting an AI detector to the test, but it does hopefully illustrate how poor AI detectors are at accurately differentiating human generated text from AI generated text. ZeroGPT is just one example – I suspect you’d get similar results with any of the other AI detectors out there. The tools just aren’t there yet.

So, what do you think? Have any of you been putting an AI detector to the test? If so, what did you find? Please share any comments you might have or if you’d like to know more about a particular topic.

Disclaimer: The views represented herein are exclusively the views of the authors and speakers themselves, and do not necessarily represent the views held by my employer, my partners or my clients. eDiscovery Today is made available solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Today should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.