A new paper published by Maura R. Grossman, J.D., Ph.D. and Gordon V. Cormack, Ph.D. titled The eDiscovery Medicine Show seeks to invite an open and honest dialogue regarding the need for truth and science in eDiscovery. And they do so by drawing a parallel between how eDiscovery professionals currently approach culling of electronically stored information (“ESI”) and the medicine shows of the early 1900s, where “unscrupulous purveyors of magical elixirs promoted their wares”.
In the paper (available here), Grossman and Cormack set the stage for the discussion in the very first sentence by informing/reminding us: “As recently as 100 years ago, harmful practices such as bloodletting were still advanced as sound medical practice by expert practitioners. Bloodletting gradually fell into disfavor as a growing body of scientific evidence showed its ineffectiveness and demonstrated the effectiveness of various pharmaceuticals for the prevention and treatment of certain diseases.” And they quickly tie the parallel between those medicine shows and current eDiscovery methods in the second paragraph, as follows:
“eDiscovery methods, like therapeutics, are amenable to scientific evaluation. But practitioners and their “experts,” vendors, and clients often ignore empirical evidence, citing instead existing or past practice to justify, for example, culling electronically stored information (“ESI”) using untested search terms, establishing neither their necessity nor their efficacy. Or, they use pseudo-science to promote various potions marketed as “Artificial Intelligence,” “AI,” “technology-assisted review,” or “TAR.” Or, they employ pseudo-science and various logical fallacies to impugn scientific studies that contradict their claims. Or they point to the oft-cited Sedona Principle 6 as justification to do whatever they please. Or, sometimes, even all of the above. Trade shows and other “educational” activities sponsored by vendors promote their wares, complete with pseudo-scientific results, testimonials, sponsored receptions, prizes, and hospitality suites. The Continuing Legal Education (“CLE”) industry and the trade press often echo these testimonials, failing to discriminate between practice and sound practice—let alone best practice—or between science and pseudo-science. So far, neither the courts nor any other authority has taken up the mantle, leaving parties to fend for themselves in the eDiscovery Wild West.”
Wow.
Grossman and Cormack proceed to discuss recall and precision and what can be considered a good result (as well as the limits of what Boolean search can yield in a single query), the pitfalls of how the eDiscovery medicine show “frequently” promotes the sequential use of two or more information retrieval methods, including Boolean search, TAR, and manual review, the misapplication of effectiveness measures, the use of non-blind experiments and the “Clever Hans effect”, as well as misleading statistics and the “misconception…that only the TAR tool should be subject to validation, while keyword culling and manual review should be exempt, as they have always been.”
Grossman and Cormack also discuss what the Courts can do and should do to demand parties show “the reasonableness of their eDiscovery search and review processes… by hewing closely to tools, methods, and procedures that have been scientifically vetted and shown to be valid and reliable. Anything else belongs in a medicine show.”
In their eight-page paper (also available here in the Ohio State Technology Law Journal), Grossman and Cormack make an informative and compelling argument to stick to the science instead of the “eDiscovery Medicine Show”. Let the open and honest dialogue continue!
So, what do you think? Do you agree with their characterization of approaches today as “The eDiscovery Medicine Show”? Please share any comments you might have or if you’d like to know more about a particular topic.
Disclaimer: The views represented herein are exclusively the views of the author, and do not necessarily represent the views held by my employer, my partners or my clients. eDiscovery Today is made available solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Today should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.
This matches my observation. From what I’ve seen for over a decade, the percentage of both vendors and software users who are willing to vet is in the low single digits. And half those who then do vet rarely do so properly (i.e. following correct scientific processes). Often this is criticized as being “too ivory tower” and “out of touch”, but those levying such criticisms have no such complaints when similarly rigorous scientific testing is applied to, say, the medications they consume or the planes in which they fly. There is an extremely large, but currently incredibly sparsely populated, middle ground between old time medicine shows and ivory towers. That middle ground needs more people willing to resettle there.
[…] are definitively open and honest in their discussion of the industry (as I previously discussed here), comparing it to medicine shows of the early 1900s, where “unscrupulous purveyors of magical […]