In the case In re Insulin Pricing Litig., No. 23-md-3080 (BRM) (RLS) MDL No. 3080 (D.N.J. April 11, 2025), New Jersey Magistrate Judge Rukhsanah L. Singh resolved an “informal discovery dispute” between the plaintiffs and defendants regarding various aspects of defendant Express Scripts’ proposed TAR protocol, including training the model, establishing a stopping criterion, the validation process, disclosure of validation documents, and a validation recall target.
Case Discussion and Judge’s Ruling
The core of the dispute revolves around the adequacy of Express Scripts’s proposed TAR implementation, specifically its use of a continuous active learning (CAL) model. While Express Scripts agreed to train and validate its model and disclose certain metrics, Plaintiffs argued its proposed workflow was insufficient in several key areas. Plaintiffs sought a Court ordered TAR Protocol incorporating their proposed methodology.
Specific Areas of Dispute and the Court’s Rulings
Here are the specific areas of “informal discovery dispute” and what Judge Singh ruled on each area:
Training the TAR Model
Plaintiffs’ Proposal: Express Scripts should iteratively train the model using all attorney relevance-based coding decisions and retrain regularly.
Express Scripts’ Counter: While intending to use “all or nearly all” relevance-based coding initially, Express Scripts sought the “option to fine-tune the model’s performance towards the end of the review by limiting training input to QC’d records only.”
Judge Singh’s Ruling: Acknowledging the flexibility needed in complex ESI discovery while also considering Dr. Maura R. Grossman’s expert opinion against “‘cherry-picking’ of training examples” after the initial seed set, Judge Singh directed Express Scripts to proceed with its proposal but with the following conditions: “(1) it must use all reviewer determinations to train the TAR model for the majority of its review; (2) if and when it decides to limit training to only those documents reviewed for quality control, Express Scripts shall provide advance notice to Plaintiffs and the parties shall meet and confer as to whether there is consent for such limitation; and (3) to the extent the parties reach an impasse following the meet and confer described herein, then the parties shall raise the dispute with the Court within five (5) business days of such impasse and Express Scripts shall not limit its training to only those documents reviewed for quality control until the Court has issued a ruling on the parties’ dispute.”
Stopping Criterion for TAR Review
Plaintiffs’ Proposal: Implement a preset stopping criterion based on marginal precision (similar to In re Uber Techs., Inc.), where review pauses for validation once consecutive batches of top-ranked documents contain 10% or fewer responsive documents.
Express Scripts’ Counter: Opposed a preset stopping point, arguing it should be determined “at the point in time with the facts of the actual in-process workflow and its progress in hand.”
Judge Singh’s Ruling: Agreeing that “the better course is to determine the stopping point in real time as the process proceeds”, Judge Singh sided with Express Scripts, declining to adopt a preset stopping point at this stage.
The Validation Process
Plaintiffs’ Proposal: Validation should rely on samples drawn from the entire universe of documents subject to TAR.
Express Scripts’ Counter: Proposed validating using an “elusion set” that excludes documents that go through quality-control review. Express Scripts planned to use the recall measures and elusion sampling formulas from In re Broiler Chicken.
Judge Singh’s Ruling: Citing case law and expert recommendations, Judge Singh raised concerns that excluding QC’d documents from recall calculations could lead to “opaque and potentially unreliable” results. So, Judge Singh sided with Plaintiffs, ordering Express Scripts to validate their TAR model “consistent with the validation process and recall calculation ordered in in re Broiler Chicken…, including a sampling from the full TAR document universe.”
Disclosure of Validation Documents
Plaintiffs’ Request: Disclosure of all non-privileged responsive documents from the validation samples and information concerning the stratum from which each document was drawn.
Express Scripts’ Position: Agreed to produce the responsive documents but opposed disclosing stratum information.
Judge Singh’s Ruling: While acknowledging that the adequacy of a search involves both quantitative and qualitative factors, Judge Singh found it premature to require stratum disclosure at this stage, given that the TAR process had not yet begun and the volume and timing of validation document production were unknown.
Validation Recall Target
Plaintiffs’ Proposal: A target recall rate between 70% to 80%, or higher.
Express Scripts’ Position: Agreed to “target a reasonable proportional recall rate of 70 or higher” but opposed a binding ex ante target without mutuality.
Judge Singh’s Ruling: Judge Singh found Express Scripts’s commitment to target a recall rate of 70% or higher “sufficiently reasonable at this stage of the process”, “[c]onsidering the dynamic nature of discovery”. He also noted that “the parties have agreed to meet and confer following the validation disclosures. In that process, the parties can discuss the adequacy of the recall measures in the context of quantitative and qualitative factors.”
As a result, Judge Singh ordered that “Express Scripts shall proceed with its TAR workflow in accordance with the above” and terminated the “informal discovery dispute”.
Related Posts
So, what do you think? Was this a good way to resolve the “informal discovery dispute”? Please share any comments you might have or if you’d like to know more about a particular topic.
Case opinion link courtesy of Minerva26, an Affinity partner of eDiscovery Today.
Disclaimer: The views represented herein are exclusively the views of the author, and do not necessarily represent the views held by my employer, my partners or my clients. eDiscovery Today is made available solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Today should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.