Ralph Losey’s latest post illustrates some of the pitfalls with AI-generated content as his initial AI top cases of 2022 doesn’t include any cases from 2022!

Whoops!

Ralph’s post (AI Analysis of the Top Five Cases in 2022 Shows Major Flaws in Use of Open AI’s GPT Software for Legal Research and Analysis) discusses how he asked “my AI helper, Open AI’s GPT-3” to “identify the top five cases for 2022”. But he noted, “the AI performed very poorly and made me look bad. I published the AI’s report last night after cleaning up the words only, and not actually reading each case (it cited to WestLaw and I have Lexis and, good grief, its a Holiday and I’m only human). I gave the robot too much slack.”

Ralph goes on to note that “An astute reader, Maura” (who I’m assuming is Maura Grossman?), “noticed right away” that his initial AI top cases of 2022 “were not all in 2022. You could easily tell that from the Westlaw citations the AI gave. I checked further this morning, as I should have done last night. I then realized that the AI ‘emperor had no clothes,’ the AI was totally off and I had been fooled. As soon as I received Maura’s helpful comment, I took my blog offline.”

Ralph stated: “I blame a concussion {he recently had an auto accident and is still recovering – get well soon, Ralph!}, and my over-trust of the new, much hyped GPT 3.5. I will be more skeptical henceforth. When it comes to legal research Text-Davinci-003, the Open AI tool that I used, is definitely not ready for primetime.”

Ralph did note that he “did not use Chat GTP 3.5 for any of these queries because that database ends in 2021”, so he used the beta playground instead which does go through 2022. Here is what it initially selected, per Ralph:

1. Toth v. Johnson & Johnson, 1:20-cv-00591, 2020 WL 5444439 (C.D. Cal. Sept. 11, 2020). (Note: The AI claims the opinion was written by Judge Banikian. I can find no such judge!)
2. Google LLC v. Oracle America Inc., 5:20-cv-03624, 2021 U.S. Dist. LEXIS 66448 (N.D. Cal. Apr. 9, 2021)
3. Schulman v. Smith & Wesson Corp., 0:20-cv-60780, 2020 U.S. Dist. LEXIS 148034 (S.D. Fla. Aug. 14, 2020)
4. Oracle America Inc. v. Google LLC, 786 F.3d 1179 (N.D. Cal. 2014)
5. Microsoft Corporation v. Quinnipiac University, 3:20-cv-01705, 2020 WL 6079805 (N.D. Cal. Oct. 14, 2020)

See the problem?

Ralph proceeds to go through his iterations to attempt to get a good AI top cases of 2022 for eDiscovery, with full legal citations and a brief explanation of why each case is important. What’s notable about the iterations is that: 1) the AI didn’t come up with the same 5 cases each time, and 2) the “reason” that the AI came up with as to why each case is important is virtually identical and not really reflective of an in-depth knowledge of eDiscovery considerations (at least IMHO).

As Ralph notes in his closing: “GPT software is very interesting and shows a lot of potential. But be warned, it should not be used at this time without a great deal of skepticism and close human supervision. Hybrid, human and computer, working together – IMO that is the answer to for the proper use of AI for the foreseeable future. I am still very pro AI, but for now at least, the human needs to keep a close eye on the robot.”

Great post, and kudos to Ralph for discussing his process and acknowledging his initial error! Happens to all of us!

A couple of years ago, I covered what predecessor GPT-3 thought about “Legaltech” (at least back then). My favorite quote from the algorithm was this one:

“The one hope for the legaltech industry would be that if you, the lawyer, are no longer relevant, there is always the judge. They have a life-tenured job and can’t be fired. If the industry can get enough AI and other technologies to make you obsolete, then the judge is still needed. So, the industry may still have a future in lobbying to keep judges.”

That one still cracks me up! 😀

So, what do you think? Are you surprised that the initial AI top cases of 2022 for eDiscovery didn’t include any cases from 2022?  Please share any comments you might have or if you’d like to know more about a particular topic.

Disclaimer: The views represented herein are exclusively the views of the author, and do not necessarily represent the views held by my employer, my partners or my clients. eDiscovery Today is made available solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Today should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.