Monday, EDRM conducted a webinar discussing the EDRM GenAI Survey Results with several thought leaders weighing in on the findings.
In the webinar (available here via EDRM’s webinar page for on-demand viewing), the findings and implications of the EDRM GenAI Survey Results were discussed by an expert panel including Hon. Ralph Artigliere (ret.), Maura R. Grossman, Bruce Hedin, and Jason R. Baron. The 15-page report discussing the findings of the survey is available for download from the webinar.
About the Survey
The EDRM GenAI Working Group survey was initiated in 2024 to explore the practical application of GenAI within the Electronic Discovery Reference Model.
- Survey Format: 42 questions followed by individual interviews.
- Respondent Pool: 19 self-selecting members of the EDRM GenAI Working Group.
- Demographics: Predominantly lawyers (legal/compliance roles) at large law firms (over 1,000 employees) with a primary geographical focus on the United States.
Limitations of the Survey Included: The sample is small and non-random. Respondents are “power users” highly engaged with technology; therefore, results represent the “vanguard” of the profession rather than the industry average.
Primary Use Cases
The most successful applications involve processing existing text rather than complex legal decision-making.
- High-Frequency Success: Text summarization (15/19 respondents) and document drafting (11/19). Tasks such as early case assessment, chronology construction, and deposition prep show “tangible” efficiency gains.
- Document Review (eDiscovery): While 14/19 respondents have used GenAI for document review, the results are “mixed.” Some find it superior to TAR, while others report it is not yet more effective or efficient than incumbent processes.
- The “Initial Draft” Model: Experts emphasize that GenAI is currently a tool for human augmentation. It provides a “first draft” or analysis that requires human refinement to be usable.
The Scrutiny Gap: GenAI vs. TAR
Part of the discussion focused on how the uncritical embrace of GenAI stands in stark contrast to the strong resistance to TAR a decade ago.
- The TAR Precedent: Despite peer-reviewed science and millions of dollars in studies (e.g., TREC Legal Track), TAR faced years of skepticism and resistance.
- The GenAI Wave: GenAI has been “embraced” without similar independent peer-reviewed science.
A couple of potential reasons for the difference are:
- Ubiquity: GenAI is used in daily life (e.g., ChatGPT), making it feel more accessible than the statistical complexity of TAR. (I asked a question about this potentially being one of the reasons for the difference during the webinar).
- Client Push: Unlike TAR, which was often a law firm-driven efficiency, clients are now “aggressively” encouraging GenAI adoption to contain costs.
Transparency and Disclosure
10 of 19 respondents involve clients in vetting tools or provide specific details on vendors and use cases. None reported making disclosures to opposing counsel or to courts regarding their use of the tools. Respondents expressed concern that disclosing GenAI “prompts” could infringe on attorney work-product protections. The panel discussed a “shadow GenAI” environment where the technology is used for backstage tasks (narratives, deposition prep) without formal protocol agreements.
Evaluation and Validation Frameworks
The level of evaluation and validation depends on the task:
- Document Review: Remains grounded in familiar metrics of recall and precision, often compared directly against TAR.
- Other Legal Tasks: Evaluation is described as “completely ad hoc” or “tricky,” often relying on “user satisfaction” rather than statistical validation.
Conclusion
Not surprisingly, the report concludes that the legal profession is in the “early days” of a transformative shift. While adoption is high, there is an urgent need for: 1) Sound Validation Frameworks: Moving beyond subjective “user satisfaction”; 2) Objective Peer Review: Developing scientific evidence for GenAI methodologies similar to those established for TAR; 3) Collaborative Education: All-hands-on-deck effort to increase judicial and practitioner competence.
These are just a few key points from the webinar and the survey results – again, the webinar is available here via EDRM’s webinar page for on-demand viewing.
So, what do you think of the EDRM GenAI Survey Results? Please share any comments you might have or if you’d like to know more about a particular topic.
Image created using DALL-E 3, using the term “robot lawyer dressed in a suit completing an online survey on their workstation in a law office”.
Disclaimer: The views represented herein are exclusively the views of the author, and do not necessarily represent the views held by my employer, my partners or my clients. eDiscovery Today is made available solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Today should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.