Can the costs of Technology Assisted Review (TAR) be shifted from the producing party to the receiving party? Apparently so, if the court rules that the TAR process that the plaintiff insisted on was disproportionate to the needs of the case.
In Lawson v. Spirit Aerosystems, Inc., No. 18-1100-EFM-ADM (D. Kan. June 18, 2020), Kansas Magistrate Judge Angel D. Mitchell granted the defendant’s Motion to Shift Costs of Technology Assisted Review of ESI to the plaintiff, ruling “the ESI/TAR process became disproportionate to the needs of the case” after having previously warned the plaintiff that inability to focus ESI requests would result in the court shifting costs.
Case Background
This case regarded the defendant’s alleged breach of a retirement agreement with the plaintiff due to plans by an investment firm to install the plaintiff as CEO of an aircraft component manufacturer (“Arconic”) where the defendant withheld the plaintiff’s retirement benefits because the defendant claimed that he violated the non-compete provision in his retirement agreement.
The parties had a lengthy set of discovery disputes as the case continued, with the Court having previously established a search protocol for the parties to follow in April 2019, where the Court directed the parties to work together on search terms to try to achieve an 85% responsiveness rate. At one point, the defendant harvested ten custodian files, which returned 304,272 documents, or 468,595 documents including families. The defendant reviewed a 384 document sample and determined that only 7.8% were responsive. Of those, many were technically responsive but were actually irrelevant to the claims and defenses in this lawsuit.
At one point, the defendant conducted new searches of the ten custodians’ ESI using revised terms provided by the plaintiff, and this returned approximately 322,000 documents. The parties continued to negotiate back and forth and, around August 2019, the parties abandoned efforts to refine search terms to meet the 85% responsiveness-rate goal, and they began discussing the option of conducting a TAR of the 322,000 document set identified in July 2019. Despite the fact that the defendant’s sampling exercise suggested that only 5% of the 322,000 documents would be responsive to the plaintiff’s discovery and those “technically responsive documents are largely irrelevant to the dispute”, the plaintiff still pursued conducting the TAR protocol, which was completed in mid-January. Only 3.3% of the documents in the TAR set of 322,000 documents were responsive and the defendant estimated its TAR expenses to be approximately $400,000 in vendor costs and $200,000 in law firm fees.
Judge’s Ruling
Judge Mitchell noted: “At Lawson’s request, the parties spent months engaged in an ESI discovery process regarding the issue of business overlap between Spirit and non-party Arconic, Inc…using traditional ESI methods involving custodians and search terms. When that process repeatedly yielded low responsiveness rates, the court allowed the parties to proceed— again, at Lawson’s request—with a technology-assisted review (“TAR”) of approximately 322,000 documents, with the caveat that the court would decide whether to allocate the TAR expenses to Lawson.”
Ruling on the defendant’s motion, Judge Mitchell stated: “Spirit’s motion is granted. The court is mindful of the default rule that the producing party should ordinarily bear the costs of production, but the court finds good cause to allocate the TAR expenses to Lawson in order to protect Spirit from undue burden and expense. Early in the case, Lawson pursued a scattershot ESI approach on the issue of Spirit’s ‘Business,’ and the court repeatedly cautioned Lawson to better focus his ESI custodians and search terms because the court would, at some point, begin shifting costs. Spirit has already borne its fair share of expenses providing discovery on this subject matter by accommodating Lawson’s ESI requests for the custodians and search terms he selected, by running court-ordered sampling exercises, and by making targeted document productions on a separate path than the ESI process. That ESI process repeatedly yielded low responsiveness rates. But Lawson was unwilling to abandon the largely non-responsive ESI dataset and instead sought continued review via TAR that unnecessarily perpetuated and exacerbated ESI/TAR expenses. The TAR process ultimately yielded a responsiveness rate of only 3.3%. Even the documents that were technically responsive were of marginal (if any) relevance above and beyond what Spirit produced outside of the ESI/TAR process. Thus, the ESI/TAR process became disproportionate to the needs of the case.
The parties are directed to meet and confer to try to reach agreement on the amount of the TAR expenses. In the event they are unable to reach agreement, the court orders further briefing as to what dollar amount the court should award”.
So, what do you think? Should the judge have granted the motion to shift costs or could the defendant have done more to curb costs in the first place? Please share any comments you might have or if you’d like to know more about a particular topic.
Also, just a reminder that, next Tuesday, October 6th, EDRM will host the webcast Important eDiscovery Case Law Decisions for October 2020 at 1pm ET (noon CT, 10:00am PT). In this webinar, Tom O’Connor, Mary Mack, Hon. Andrew Peck (Ret.) and I will discuss key cases related to discovery about discovery, the efficacy of terminating sanctions, considerations for audio/video discovery and disputes regarding search terms and technology assisted review (TAR). Don’t miss it!
Case opinion link courtesy of eDiscovery Assistant.
Disclaimer: The views represented herein are exclusively the views of the author, and do not necessarily represent the views held by my employer, my partners or my clients. eDiscovery Today is made available solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Today should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.
As you pointed out in your commentary, 3.3% of the population that was to be subjected to TAR was responsive. Elsewhere in the opinion (Section E), it backs up what you wrote: “Only 3.3% of the documents in the TAR set of 322,000 documents were responsive.”
But as you also quote the judge in the intro: “The TAR process ultimately yielded a responsiveness rate of only 3.3%.”
Those are two massively different statements: (1) The collection on which TAR was to be run (aka the post keyword culled universe) is 3.3% rich, vs. (2) TAR was 3.3% precise (precision = “responsiveness rate”)
The only way those two statements are logically consistent with each other is if TAR were used to review the entire collection. Which essentially means it’s no better than linear review (aka reviewing documents at random). Which is simply not the case.
Therefore, until such glaring inconsistencies are resolved, it is very difficult to form an opinion on what was and was not justified. There are more details that are needed to make a proper assessment of what the “shoulds” should have been.
Dr. J, I expected you would have something to say about this case! Certainly not the first case that I’ve covered where not all of the details needed to understand the case ruling were included in it. For me, the most glaring item was the estimate of $600K to run TAR on 322K documents!! I thought TAR was supposed to save on review costs. Certainly doesn’t seem to have happened here.
I cannot reveal internal details, of course. But as a scientist, let me list the questions that I would ask if I were trying to understand that $600k price tag:
(1) How does that $600k break out? At one point it says: “Spirit estimates its TAR expenses total about $600,000, including $400,000 in fees and costs to Legility and $200,000 in fees to its law firms”. So right there, that’s not all TAR.
So how does the $400k break out? We don’t see a final value. But elsewhere an estimate is provided: $150,000 to $300,000 for costs incurred by Legility’s document review team [side comment: That’s a pretty wide ranging estimate.. so what was the actual?]; $25,000 to $30,000 in data promotion costs to migrate the ESI corpus into Legility’s review platform; $6,000 per month through the date of trial[3] in hosting costs [side comment: How many months was the data hosted?]; and $40,000 to $60,000 in attorneys’ fees to provide a quality control review of Legility’s document review, coordinate with Legility and its review team, and review the documents prior to production.
Many of those costs would have been costs no matter if TAR was used or not. So, how much was actually spent on the TAR review portion?
In other words, consider the baseline. The baseline would have been full linear review on all 322k documents, right? That review would have also incurred the same document loading costs, the same attorney fees, the same number of months of hosting fees, the same QC fees, etc. (Actually, if fewer than 322k documents were reviewed in TAR, then the QC fees would actually be lower for TAR as well.. fewer judgments = fewer wrong judgments.) So given that (almost, cf QC) all of those costs would have been the same under both conditions, linear and TAR, how does the actual TAR portion break out? There is not enough info in this document, but that’s the first question I would ask.
(2) Variously throughout the opinion, you can see that the review team hit different recall levels: First 65%, then 68.5% (almost the broiler chicken lower bound acceptable level of 70%), then 80% (the broiler chicken upper bound acceptable level), then 85%. Given that TAR doesn’t need to be held to a higher standard than keywords, one would want to know the costs at each of these recall points. I.e. how many documents had been reviewed to get to each of these recall levels?
In short, there is not enough public information to determine actual TAR efficiencies. I’ve read this three times, trying to see if there is anything I missed. If you do see something, let’s discuss.
All great points, Jeremy, and, no, I don’t see anything you missed. Agreed that (if we assume that Legility’s estimate is reflective of the actual cost) that some of those costs would have occurred whether TAR or another approach was used. But, it appears we won’t know how much of it would have been, unless there is another case ruling which gets into the details. Since the parties are supposed to meet and confer on costs and they haven’t been able to agree on just about anything so far, there could be.
The point this drives home for me is one that I’ve been hammering for years now, to very little avail. And that’s the notion that when it comes to review, one has to view review from a total cost perspective. Of what good does better TAR do you, if you have to do more (and more expensive) QC? Or if you spent hundreds of thousands of dollars developing keywords with which to cull the collection.
Unfortunately, folks tend to treat review as a line item request, where every line is independently optimized. Cheapest hosting is considered separately from cheapest review is considered separately from QC cost is considered separate from etc.
What is forgotten is that one plays in to the other. That saving a few pennies on hosting might not be most cost efficient, if you waste extra dollars because of a worse TAR algorithm. Or a good TAR algorithm might not be most cost efficient, if you stick poorly trained reviewers on the case and have to do more QC (at higher rates).
It all fits together and has to be considered holistically, and not as separate line items.
[…] (up to the motion previously granted by Judge Mitchell for cost shifting) is briefly synopsized here and details regarding the order granting in part and denying in part the defendant’s Application […]
[…] cost shifted to the plaintiff in the Lawson v. Spirit Aerosystems case (about which I’ve written three blog posts this year, including this morning) and whether those costs were in line or with the […]
[…] (D. Kan. Dec. 28, 2020), Kansas Magistrate Judge Angel D. Mitchell, having previously granted the defendant’s motion to shift the expenses it incurred in connection with a technology-assisted review (“TAR”) of approximately 322,000 […]
[…] to provide evidence of its burdensome objection” and rejecting the defendants’ citation of Lawson v. Spirit Aerosystems, noting in that case “the court repeatedly relied on several affidavits and declarations in the […]