A new study – Comparing GenAI to Machine Learning and Manual Review for eDiscovery – challenges the findings of the Redgrave study from earlier this year.

The 8-page study (the last two 2 pages of which are the Bibliography, so it’s essentially a 6-page study), available here, was published by Gordon V. Cormack, Maura R. Grossman and Jeremy Pickens – three notable eDiscovery and data science experts (if you read eDiscovery content regularly, you probably know who they are). They examine the original empirical study by Redgrave LLP from earlier this year (covered by us here).

The original study examined 45,004 documents from the UCSF Opioid Industry Documents Archive. It reported that Relativity’s generative AI review tool, aiR for Review, achieved 87.7% recall and 28.6% precision, compared with 64.4% recall and 39.2% precision for a managed review process using Relativity Assisted Learning (RAL), which is an established managed review process utilizing a traditional TAR approach. Those estimates used an independent subject matter expert’s review of a random 1,000 document sample, containing 73 documents the expert considered responsive.

Cormack, Grossman and Pickens identified at least three issues with treating those results as a straightforward technology comparison:

1. Different workflows: the RAL process required first-pass reviewers to approve documents as responsive, while aiR’s classifications were evaluated without that additional human filter.
2. Different responsiveness determinations: The reviewers, prompt engineer, and evaluating expert were able to interpret responsiveness differently. First-pass reviewers classified 34% of the documents as non-responsive that the expert considered responsive. Conversely, the expert classified 61% of the documents as non-responsive that first-pass reviewers considered responsive. Pretty huge disparity.
3. Recall-Precision trade-off: The reported results represented different recall–precision trade-offs as the precision was higher in the RAL approach. Retrieving more responsive documents while also retrieving more non-responsive documents does not automatically establish that the higher recall approach is better.

When the authors applied the same human-review requirement (using gain curves to illustrate the results), the comparison substantially changed. In the “filtered” TAR workflow, where first-pass reviewers had the final say, RAL achieved approximately 64% recall, compared with only 59.5% for aiR for Review. As the authors noted: “CAL is superior to both at all points in the gain curve.” In the “unfiltered” workflow, where documents identified by the system were treated as responsive regardless of first-pass coding, RAL achieved 97.2% recall, compared with approximately 88% for aiR.

As the authors note: “Using RAL (or CAL) for unfiltered TAR review would most likely be considered disproportionate because of review cost.”

The authors also tested Scalable Continuous Active Learning (S-CAL) using a weighted statistical sample of only 2,000 first-pass coding decisions for training, excluding the expert-coded evaluation sample. S-CAL substantially matched both aiR’s more inclusive and more restrictive classification settings, while allowing additional choices along the recall–precision curve. It exceeded 90% recall and eventually reached 100% estimated recall against the expert’s assessments; a level aiR did not reach in the evaluation. At its self-calibrated recall–precision crossover, S-CAL agreed more closely with the expert than did the first-pass reviewers whose decisions trained it.

The authors stated this with respect to proportionality and the recall-precision trade-off: “The doctrine of proportionality occasions us to balance the burden of reviewing or producing more documents, with their value. We can inform this analysis by examining the slope of the gain curve, if it is known. For RAL and CAL, the gain curve is constructed incrementally, and the utility of proceeding can be assessed by the frequency with which responsive documents are identified (i.e., the estimated slope of the gain curve). From S-CAL’s self-estimate of the entire gain curve, we may identify the point of diminishing returns. For aiR, we have only two viable cut points – liberal or strict – and no principled way to choose between them. occasions us to balance the burden of reviewing or producing more documents, with their value. We can inform this analysis by examining the slope of the gain curve, if it is known. For RAL and CAL, the gain curve is constructed incrementally, and the utility of proceeding can be assessed by the frequency with which responsive documents are identified (i.e., the estimated slope of the gain curve). From S-CAL’s self-estimate of the entire gain curve, we may identify the point of diminishing returns. For aiR, we have only two viable cut points – liberal or strict – and no principled way to choose between them.”

The authors also noted significant limitations with the study: one matter, one first-pass review, and one prompt-engineering effort, using a small, non-random subset of a larger collection. They also flagged the litigation’s extensive publicity as a potential constraint on generalizability. Confidential prompts limited methodological transparency, and unavailable aiR pricing prevented a complete cost comparison.

When it comes to comparing GenAI to machine learning and manual review for eDiscovery in this study, here is the bottom-line conclusion from Cormack, Grossman and Pickens:

“For this particular task, the Redgrave Case Study shows aiR to be comparable to, but not superior to RAL, whether the tool or the L1 reviewer makes the final determination of responsiveness. When the L1 reviewer makes the final determination, CAL achieves the same recall as RAL, with less review effort. When the tool makes the final determination, S-CAL, trained on 2,000 L1 reviewer responsiveness determinations, is as effective as aiR, offers a calibrated adjustment of the recall-precision trade-off, and runs quickly and cheaply on commodity hardware. aiR offers only two viable recall-precision choices, offers no calibrated means to choose, and runs slowly on specialized hardware.”

Again, the study is available here. Notably Redgrave LLP made the case-study data available for independent research and assisted in helping the authors to understand the data and study methodology.

So, what do you think? Do you agree with the findings presented in Comparing GenAI to Machine Learning and Manual Review for eDiscovery? Please share any comments you might have or if you’d like to know more about a particular topic.

Image created using ChatGPT, using the term “robot lawyer looking at gain curves on a computer workstation”. Note: the statistics shown in the image were totally made up by the model and have no bearing on either actual study.

Disclaimer: The views represented herein are exclusively the views of the author, and do not necessarily represent the views held by my employer, my partners or my clients. eDiscovery Today is made available solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Today should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.