Aligned Discovery PLLC, recently founded by noted eDiscovery expert Tara Emory to facilitate discovery between litigation parties as a court-appointed neutral, is offering a new eDiscovery Recall Calculator on its site.

The tool (available here) helps legal teams estimate document search recall using statistical sampling methods to support a variety of approaches and methodologies. As Tara noted in this LinkedIn post announcing the eDiscovery Recall Calculator a couple of days ago, it supports:

• Different search sets
• Nested search sets (like search terms + TAR)
• Multiple methodologies (search terms, traditional TAR 1 and TAR 2, and GenAI TAR)
• Visualization of outcomes across different search populations

Importantly, data submitted to the tool is ephemeral and not retained, so you don’t have to worry about anything that you’re doing being kept or re-used. To save your work as a PDF, you can export your results using the Export Report button.

Tara provided a walk-through of the Recall Calculator to me a couple of days ago. As she noted, the tool was born out of a need in her neutral practice to empower parties to understand their own validation processes rather than relying solely on an expert, and potentially receiving “bad news” about results late in the process. Having said that, it’s not a defensibility tool, and doesn’t replace experts. Instead, it’s designed to correct the industry’s often incorrect approaches and language regarding search validation by providing a “wizard-based” experience, and also includes plenty of click-through explanations that help educate users on the “why” behind the need for statistical analysis.

The tool offers two different validation modes – Production Validation and Search Effectiveness – which determine when family members are included or excluded. In Search Effectiveness mode, the tool cues users to exclude family members of hit (search-positive) documents, which can create “noise” (if they’re non-responsive) that obscures whether a search term is actually performing well. In Production Validation mode, the tool cues users to include family members for search results, so it focuses on what documents would actually be reviewed and produced.

Perhaps one of the most important aspects of the tool is that it enforces the requirement to test both the positive set (search hits/predicted responsive) and the negative set (non-hits/predicted not-responsive). As Tara noted, legal professionals often ignore the non-hits, which is critical for an appropriate recall estimate.

As you can see below, it also provides a link to a template for a Search Validation Protocol that you can consider adapting for your cases’ needs, to include within an ESI Protocol!

The eDiscovery Recall Calculator provides a couple of demos – Independent Set and Nested Set – that you can check out to get a feel for how the tool looks. Below are images that illustrate a sample analysis using the Independent Set demo, with two different data sources.

This image shows estimated results for the project across both data sets, factoring in the “Search Positive” and “Search Negative” results, as well as already-reviewed Known Responsive documents, based on per-sample confidence intervals at 95% confidence level (99% confidence could also be selected by the user).

This image shows those same estimates again with error bars reflecting lower and upper bounds for each category based on per-sample confidence intervals at the selected 95% confidence level.

This image shows Search Recall, with a comparison of retrieved vs. missed responsive documents (which reduce Recall), again including error bars for the calculated confidence intervals.

And this image shows Search Precision for the Search-Positive Set, with a comparison of responsive retrieved and not responsive retrieved (which reduce Precision).

Again, the eDiscovery Recall Calculator (available here) is designed to be used as an educational aid and a “gut-check” mechanism rather than as a replacement for legal experts or formal court evidence. Lawyers who claim “they said there would be no math” can use this tool to better understand the necessary “math” needed to validate search results – which is great!

Tara alluded to a couple of additional educational tools in the works, so stay tuned for that!

So, what do you think? Do you understand the best practices associated with search validation? Please share any comments you might have or if you’d like to know more about a particular topic.

Image created using GPT-4o’s Image Creator Powered by DALL-E, using the term “robot looking at various graphs and analytics on a computer”.

Disclaimer: The views represented herein are exclusively the views of the authors and speakers themselves, and do not necessarily represent the views held by my employer, my partners or my clients. eDiscovery Today is made available solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Today should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.