How do you get from AI hype to AI accountability? A new report from Casepoint discusses how senior practitioners are using AI.
For the report titled (wait for it!) From AI Hype to AI Accountability: What You Can Learn From Legal and FOIA Teams About AI Modernization (available here for download), Casepoint interviewed dozens of senior practitioners across corporate legal departments, federal agencies, and law firms about how AI is being used in legal, FOIA, records, IT, and compliance work today. And guess what? The era of “AI is coming” has ended; the era of “AI is here and it is incredibly messy” has begun.
The report turns those conversations into practical guidance for building AI workflows that can be verified, explained, and defended.
The 22-page report is quite comprehensive, so I’ll touch on a few highlights:
- Prompt Engineering is Dead, Long Live Context: The obsession with “prompt engineering” – the idea that the perfect sequence of words is the key to AI value – is rapidly fading. In its place, mature legal teams are pivoting toward “context engineering.” The strategic shift is toward “bounded workflows”: tasks with known source materials, clear constraints, and reviewable outputs. By prioritizing the “where” and the “what” of the data rather than the wording of the prompt, teams are focused on making sure that AI-generated results are grounded in verifiable facts rather than the general hallucinations of a large language model.
- Your AI Prompts are Now Official Records (And You Aren’t Ready): Many organizations are failing to realize that AI-generated summaries, query histories, and draft analyses, etc. are living records. This data often resides within systems like Microsoft 365 that were never originally designed to manage them as official records. For example, one federal eDiscovery leader discovered Copilot query and response files sitting in mailbox collections during routine litigation hold processing – not an ideal time to decide how to treat that material.
- AI is Making FOIA Responses More Efficient, But It’s Also Creating More Work: While Federal agencies are looking to AI to manage staggering backlogs, requesters are using that same technology to flood agencies with a “tidal wave” of complex requests that traditional workflows cannot handle. One user at a large Federal agency reported receiving 100 requests in minutes from a single group – normally, it’s roughly 300 requests per year.
- Hallucinations Aren’t a “Bug”, They’re a Workflow Reality: Surely, you know this by now, right? One manager of litigation services at a law firm called hallucinations “super frustrating”. But they aren’t a reason to abandon AI; they’re a factor that must be built into your standard operating procedure – i.e., check the AI output against the underlying material. We all know what happens when we fail to do that.
- No “Broad Permission Slips for AI: Stop issuing “broad permission slips” for AI access. Giving an entire department a general AI tool rarely leads to maturity. The most successful organizations – like Starbucks, which moved from a zero-AI posture (no use of AI at all) to one of controlled internal use while strictly restricting vendor and outside counsel use – take a more targeted approach to deployments.
- Practitioners Are Skittish About Defensibility: One federal eDiscovery practitioner raised the concern about defensibility: “There’s no point using AI in the manner I described if we get to a production, go to court, and they say: this is not verifiable, this is not TAR 2.0, and I don’t have any of the statistics and measures I need.” Personally, I think you can use (and we have seen used) the same validation statistics and measures used for GenAI that we used for TAR. I’ll be part of a panel speaking on validation for GenAI at ILTACON and look forward to talking more about that.
Let’s face it: getting from AI hype to AI accountability is messy. But as Casepoint’s report concludes “AI accountability is now part of the work”. It was always supposed to be.
Again, their report is available here for download.
So, what do you think? Is your organization succeeding in getting from AI hype to AI accountability? Please share any comments you might have or if you’d like to know more about a particular topic.
Image created using DALL-E 3, using the term “a human supervisor checking the work of robot workers”.
Disclosure: Casepoint is an Educational Partner and sponsor of eDiscovery Today
Disclaimer: The views represented herein are exclusively the views of the author, and do not necessarily represent the views held by my employer, my partners or my clients. eDiscovery Today is made available solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Today should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.