Remember the Redgrave study of aiR for Review? Relativity is now giving you a chance to weigh in on close calls: documents where reviewers disagreed.

The post by Aurora Bryant (Close Calls: An Interactive Look at the Redgrave Study of aiR Review, available here) discusses the study that Redgrave LLP published comparing aiR Review with a 24-person managed review team on the same 45,004 documents that Mallinckrodt produced in an opioid litigation, which are publicly available in the Opioid Industry Documents Archive at the University of California San Francisco. (Robert Keeling and Ray Mangum summarized it here). Their results showed that aiR Review found more of the responsive documents, with far less attorney time.

We covered it here, and also covered an additional study here from Gordon V. Cormack, Maura R. Grossman and Jeremy Pickens, which challenged the findings of the Redgrave study.

Aurora’s post takes a second look at those results: how the study ran; what Jessica Maurer, a second expert, made of the same documents; and the documents where the reviewers disagreed.

At the end of the post is a link to a short game: Close Calls, which contains 10 documents from the study, most of them close calls where the reviewers did not all agree, and a timer. It enables you to review those documents yourself, make your decisions and find out which “team” your calls best match.

By “teams”, there are four of them:

• Team Kevin: Kevin Reiss, Redgrave LLP counsel whose expert classifications established the study’s original benchmark.
• Team Jessica: Jessica Maurer, a member of Relativity’s applied legal intelligence team who independently reviewed the same sample.
• Team Ray’s aiR Review: Relativity’s generative AI system using instructions developed by Redgrave partner Ray Mangum.
• Team First-Pass Review: The 24-person managed review team—one manager, two team leads, and 21 contract attorneys—using active learning to prioritize documents.

The game provides review instructions, including definitions of several issue codes – for things like promotional statements regarding the products, discussions of drug samples provided, “diversion” (e.g., stolen or misused drug quantities), suspicious orders, and interactions with healthcare professionals.

I decided to play the “game”. Here’s how it went:

I reviewed 10 documents in 9 min 35 seconds, for a measured review speed of 63 documents an hour (which is about a document a minute – a common review rate for many collections). As the analysis notes: “At its review speed in the study, about 45,000 documents an hour, Ray’s aiR Review would have reviewed about 7,200 documents in that time.” Show-off! 😉

In the same amount of time, the First-Pass Review Team, its 24 reviewers working together at their study average of about 770 documents an hour, would have reviewed about 123 documents (or 5.39 docs per reviewer).

So, I was quicker than the average reviewer, just sayin’. 😉 Of course, quicker doesn’t necessarily mean better.

In terms of the “team” that I was closest to, I agreed with 8 out of 10 with Team Ray’s aiR Review, 6 out of 10 with Team First-Pass Review, and 5 out of 10 with Team Kevin and Team Jessica. So, either I think like AI does (not the first time I’ve done so 😉) or I think like Ray does.

In my opinion, however, the point of the exercise is to illustrate how difficult document classification can be. There were only two of the ten documents where all teams agreed on the classification (including me). Nearly half (four of the ten documents) had an even split between the teams (i.e., two classified as responsive, two classified as non-responsive). And these are (what I would consider to be) experts classifying these documents (the AI was prompted by an expert, so I’m counting that).

You can play the game here. There’s a lot more information in Aurora’s blog post and in the analysis when you complete the game (including how the 10 documents were chosen).

Classifying documents as responsive or non-responsive isn’t an exact science – even among experts. Responsiveness is in the eye of the beholder more often than we think. There are often many close calls.

So, what do you think? How has your organizations dealt with close calls in review? Please share any comments you might have or if you’d like to know more about a particular topic.

Image created using ChatGPT using the term “robot lawyer wearing a suit looking through a microscope at a workstation”.

Disclosure: Relativity is an Educational Partner and sponsor of eDiscovery Today.

Disclaimer: The views represented herein are exclusively the views of the author, and do not necessarily represent the views held by my employer, my partners or my clients. eDiscovery Today is made available solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Today should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.