There’s a term that illustrates what’s happening with all these fake citations cases: automation bias. What is it and how do we stop it?

I feel like embracing my inner Britney Spears and singing “oops, they did it again” every time I see another instance of fake citations in case filings. They just keep coming. I’ve had two sent to me in the last two days – from my colleagues Tom O’Connor and Judge Andrew Peck. I’ve also read about the one Judge Peck sent me from Kelly Twigger on LinkedIn (which I covered yesterday, though I’m not entirely sure that was a legit AI-generated error) and two that Bob Ambrogi covered yesterday (one of which was the one Tom sent me today).

In that one, attorneys from the law firms Ellis George LLP and K&L Gates LLP submitted a brief to Special Master Michael Wilner containing numerous hallucinated citations. Even after Wilner questioned two suspicious citations, the attorneys filed a “corrected” version – which still contained at least six other AI-generated errors. Here are the sanctions that the special master decided to impose:

• Striking all versions of the attorneys’ supplemental brief.
• Denying the discovery relief they sought.
• Ordering the law firms to jointly pay $31,100 in the defendant’s legal fees.
• Requiring disclosure of the matter to their client.

Ouch! That last one may be the worst one of all.

So, what is automation bias? The Center for Security and Emerging Technology (CSET) defines it this way:

“Automation bias is the tendency for an individual to over-rely on an automated system. It can lead to increased risk of accidents, errors, and other adverse outcomes when individuals and organizations favor the output or suggestion of the system, even in the face of contradictory information.”

According to the AI Overview in Google (yes, I get the irony), here are key characteristics of automation bias:

• Overreliance: People become overly reliant on the automated system and may fail to monitor its performance or verify its output.
• Reduced Vigilance: Reliance on automation can decrease vigilance, causing people to miss critical information or warning signs.
• Trust in Automation: People tend to trust automated systems, often seeing them as more accurate or reliable than they are, leading them to accept outputs without critical evaluation.
• Errors of Commission and Omission: Automation bias can lead to errors of both commission (following incorrect advice from the system) and omission (failing to act when the system doesn’t prompt them).

It also gives five reasons why automation bias occurs. Here are two of them:

• Cognitive Miser Hypothesis: People tend to prefer the path of least cognitive effort, making them more likely to accept the system’s recommendations without further thought.
• High Workload and Time Pressure: In situations with high workload or time pressure, individuals may be more likely to rely on automation to reduce their cognitive burden.

Bingo. I think those are the two biggest reasons why this happens.

So, what can we do to mitigate automation bias? Here are five suggestions the AI gives:

Training and Education: Educate users about the limitations of automated systems and the potential for automation bias.

Hey, we’re trying! I’ve written about it numerous times, Bob Ambrogi has written about it probably even more times. Isn’t everyone reading our stuff? 😉 But it’s hard to get the word out to 1.3 million attorneys (that’s just in the US) and who knows how many experts who file declarations (even AI experts do it).

Transparency and Explainability: Design systems that provide clear explanations for their decisions and allow users to understand the system’s reasoning process.

Alas, you’ll never get that with ChatGPT and other LLMs. The best you can do is to use the “Search the Web” option to at least ensure that it’s pulling information from web-based sources. Then, you still need to check those sources (which many people don’t) to ensure they say what the model says they say (which they often don’t). Oh, and those web-based sources are ever more frequently generated by AI today, so it’s becoming “the blind leading the blind”, so to speak.

Human-in-the-Loop Design: Ensure that humans retain oversight and control of automated systems, providing opportunities for intervention and overriding system decisions.

As Dr. Phil would say: “How’s that working for you?” 🤣 We, as humans, have the “opportunities for intervention and overriding system decisions” – many are just not taking advantage of them.

Cognitive Forcing Functions: Implement features that prompt users to critically evaluate the system’s output and consider alternative options.

I don’t expect we’ll see that, but it might be helpful if the models identify responses (or portions of responses) that are more likely need checking. For example, if I ask for ten ideas for a blog post title on AI hallucinations, there’s no right or wrong answer, just suggestions on choices to use from which I can subjectively choose the one I like best. On the other hand, if I ask for a citation to a case (or a quote from a case law ruling), the answer isn’t subjective – it’s either correct or it isn’t. Perhaps a little stronger warning for responses that are supposed to be based on fact could make some think twice about blindly trusting the model. Or not.

Promote Accountability: Emphasize the importance of human responsibility and encourage users to verify the system’s output, rather than blindly accepting it.

We’ve tried that too. Courts have issued standing orders and are starting to do so again (like this one I referenced yesterday) to remind lawyers they are responsible under Rule 11 of the FRCP to ensure the accuracy of their filings. Many people have said these standing orders aren’t necessary (I agree), but with the pace of fake citation filings increasing (not decreasing), I can understand why some courts feel it necessary to remind lawyers of their duty. Sanctions are the other way to promote accountability, but those don’t appear to be working either – case in point: it’s been 14 months since a lawyer was suspended for a year in Florida and we’re seeing more of these than ever.

How do we stop automation bias from happening with fake citations in filings? We can’t, because we can’t control how humans use these models. We can’t control it any more than we can control whether a person drives a car correctly and obeys traffic laws* or decides to speed, drive recklessly or drive drunk. We can take steps to minimize the instances of it – through training & education, stronger and more targeted warnings in the models and enforcing accountability – but that’s it.

Just as each of us is responsible for what happens when we get behind the wheel, we’re responsible for what we do with AI outputs. Stopping the “car wrecks” starts with each one of us.

So, what do you think? Is there a way to stop automation bias? Please share any comments you might have or if you’d like to know more about a particular topic.

*Surely, you’re not going to position self-driving cars as the answer to this, are you? 😉

Disclaimer: The views represented herein are exclusively the views of the authors and speakers themselves, and do not necessarily represent the views held by my employer, my partners or my clients. eDiscovery Today is made available solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Today should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.