This may be the first time I’ve ever linked to a comment in a post, but, in it, Craig Ball got ChatGPT to double down on fake cases.
The post here discussed an article which discussed that ChatGPT’s Deep Research identified 20 jobs it will replace, how one of those jobs was “paralegal” and how that premise was ridiculous. One of my points about the ridiculous premise was the statement that “GPT-style AI can research case law”, given how many cases we’ve seen with fake case citations.
One of the cases I linked to was a case Phil Weldon sent me from last week (Judge Andrew Peck also sent it as well) where eight out of nine cases cited didn’t exist and the Court issued a show cause order as to why plaintiff’s attorneys shouldn’t be sanctioned for this.
Enter Craig Ball. In a comment to the post (which is shown at the bottom after the post), Craig writes:
“I think that LLMs will replace a lot of what both lawyers and paralegals bill for currently, but you are correct that hallucinating case citations remains a big problem. Our jobs are safe for the moment. Just for grins, I asked ChatGPT-4 to address the following prompt using the bogus cites in the Wyoming case you linked:
Identify the presiding judges and summarize the holding in each of the following cases:
1. Wyoming v. U.S. Department of Energy, 2006 WL 3801910 (D. Wyo. 2006);
2. Holland v. Keller, 2018 WL 2446162 (D. Wyo. 2018);
3. United States v. Hargrove, 2019 WL 2516279 (D. Wyo. 2019);
4. Meyer v. City of Cheyenne, 2017 WL 3461055 (D. Wyo. 2017);
5. Benson v. State of Wyoming, 2010 WL 4683851 (D. Wyo. 2010);
6. Smith v. United States, 2011 WL 2160468 (D. Wyo. 2011);
7. Woods v. BNSF Railway Co., 2016 WL 165971 (D. Wyo. 2016); and
8. Fitzgerald v. City of New York, 2018 WL 3037217 (S.D.N.Y. 2018).
It responded as follows, which is cold comfort. Nothing approaching, ‘I was unable to locate published opinions with those cites’.”
Instead, per Craig, it said this:
“Below is general information drawn from publicly available court dockets and common legal research tools. Because unpublished district court decisions and docket details are not always consistently reported online, **please note that the names of presiding judges and the precise holdings can sometimes be difficult to verify without accessing the official opinions or dockets directly**. Where possible, I have identified the judge from publicly available sources; otherwise, I note where the information is less certain. The summaries of the holdings are likewise based on publicly reported information or typical procedural contexts. Always consult the official published (or authenticated) opinion or the docket for absolute confirmation.”
Then, as Craig detailed, ChatGPT proceeded to identify the judges and summarize these eight cases…which don’t exist per Wyoming District Judge Kelly H. Rankin, where he said: “The cases are not identifiable by their Westlaw cite, and the Court cannot locate the District of Wyoming cases by their case name in its local Electronic Court Filing System.”
Here’s one of the “cases” that Craig says was summarized by ChatGPT:
## 1. **Wyoming v. U.S. Department of Energy**
**Citation:** 2006 WL 3801910 (D. Wyo. Dec. 22, 2006)
**Presiding Judge:**
– This opinion is frequently attributed to **District Judge Clarence A. Brimmer** (who served on the District of Wyoming for many years and authored several opinions involving Wyoming’s challenges to federal regulations). However, without the official text, there is some uncertainty; Judge Brimmer is the most likely presider given the year and the subject matter.
**Summary of the Holding:**
– The case involved a challenge by the State of Wyoming against actions taken by the U.S. Department of Energy, presumably concerning energy-related regulations or implementation.
– **Holding (general):** The court addressed whether DOE’s actions were consistent with statutory or procedural requirements (often under the Administrative Procedure Act or similar federal statutes). From the limited summaries available, it appears the court **denied portions of Wyoming’s challenge** but may have remanded or dismissed certain claims for lack of jurisdiction or for failure to state a claim.
– Because the exact text is difficult to locate, the key takeaway is that this was a regulatory challenge in which the court evaluated DOE’s compliance with federal law. The outcome did not categorically strike down DOE’s authority but may have narrowed the agency’s approach or dismissed Wyoming’s claims.
All this for a case that doesn’t exist. Craig Ball got ChatGPT to double down on each of the eight cases that doesn’t exist. You can check Craig’s comment on the post here to see what ChatGPT said about all eight fake cases.
So, what do you think? Will AI chatbots ever be a reliable resource for case law? Please share any comments you might have or if you’d like to know more about a particular topic.
Image created using GPT-4o’s Image Creator Powered by DALL-E, using the term “robot judge doing a faceplant when looking at a computer”.
Disclaimer: The views represented herein are exclusively the views of the authors and speakers themselves, and do not necessarily represent the views held by my employer, my partners or my clients. eDiscovery Today is made available solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Today should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.