Last month, I covered the case McMaster v. Kohl’s Dep’t Stores, Inc. where Michigan Magistrate Judge R. Steven Whalen gave what I thought was a well-considered ruling regarding the parties’ search term dispute (we also discussed it in our monthly case law webinar for EDRM earlier this month, which you can watch on-demand here).  He ordered the parties to consult an expert to assist them in resolving the dispute, but I’ve seen other judges rule on search term disputes themselves.  Is that a good thing?  Here’s why I don’t think so.

Recap of Judge Whalen’s Ruling on Search Terms

To recap, here is an excerpt of what Judge Whalen said in his ruling regarding the search term dispute:

“Here is another case in which the Court is called upon to decide whose competing list of search terms is better suited for the search of large amounts of electronically stored information”, citing United States v. O’Keefe, 537 F. Supp. 2d 14, 23–24 (D.D.C. 2008), which stated: “for lawyers and judges to dare opine that a certain search term or terms would be more likely to produce information than the terms that were used is truly to go where angels fear to tread.”  Judge Whalen stated: “I, for one, have no interest in going where angels fear to tread. Therefore, if the parties cannot agree on appropriately limited search terms, they will share the cost of retaining an expert to assist them. If they still cannot agree, then Plaintiff may renew his motion regarding the search terms and will provide the Court with an expert report substantiating his position.”

Lawyers, Judges and Search Terms Today

While a portion of the legal profession has embraced Technology Assisted Review (TAR) technologies like predictive coding/supervised machine learning as more efficient than and (at least) as effective as search terms and manual review, most of the profession has continued to prefer the use of search terms in their cases (obviously, in some cases, a multi-modal approach using both approaches is conducted).  So, why do many lawyers still prefer search terms over TAR?  Here are three reasons why I think lawyers have been reluctant to embrace TAR and still prefer search terms:

• Lawyers don’t understand how supervised machine learning works: While many lawyers may have some idea of how it works, many others don’t understand it, so they tend to avoid technologies (even proven ones) if they don’t understand how they work.
• Lawyers are concerned about having to defend the “black box” decisions of an algorithm: In my interview with Christine Payne of Redgrave LLP and Michelle Six of Kirkland & Ellis LLP (here are parts one and two), they discussed how much more “discovery about discovery” there seems to be when TAR is involved.  Many lawyers are reluctant to consider TAR for that reason as well.
• Lawyers feel they understand search terms and how to use them: Because lawyers learned to use Westlaw and Lexis in law school (and the syntax associated with them), they feel they understand how to conduct searches for discovery purposes.

Do Lawyers and Judges Understand Search Terms as Well as They Think They Do?

In one of my very first blog posts, I discussed how I was brought in to help a client that had already agreed on search terms with opposing counsel.  One of those terms – min* – had been agreed on to search for mining related terms, such as mine, mines, miners and mining.  But the search retrieved over 300,000 files with hits because it was retrieving all sorts of other words, such as mink, mind, mint and minister.  So, we had to try to negotiate a revision to that search, along with other terms that were improperly scoped as well.

While that’s an extreme (but not entirely uncommon) example, it illustrates the need for an approach to optimize search terms and test the results.  I spoke with Mandi Ross, CEO and Managing Director of Prism Litigation Technology about search term optimization, who said:

“Just as TAR requires a structured process to lead to a successful result, keyword search also requires a structured process to be successful as well.  Effective information retrieval is a science that’s best developed by experts who take a proven and structured approach to identifying the language that signals relevancy. It starts with an analysis of the issues associated with the case, working with the attorney to define what a relevant document is about, and then designing a series of complex Boolean searches to capture those documents. It also includes testing of the results to confirm that appropriate recall and precision measures are achieved.  Conducted correctly, keyword search can still be a viable, transparent and defensible process for companies looking to reduce the volume and expense of review.”

A while back, Prism published a white paper on search term optimization and best practices, which you can download here.  And, here is a recent blog post on search terms from Mandi and John Patzakis, Chief Legal Officer and Executive Chairman at X1.

Conclusion

The approach that Mandi describes is essentially the same approach that I’ve advocated and taken with clients over the years (when they get me involved in the case early enough, that is).  But, too many times, I’ve seen lawyers in cases identify search terms and simply proceed with them, without any process for confirming that all of the issues in the case are addressed or that the search terms are properly optimized for the case and the document collection being searched.  The same search term in two different cases or with two different document collections can be proportional in one and overly burdensome in another.  I’m simply amazed how often lawyers and judges decide on searches “in a vacuum” without a defensible process used to optimize the terms for the case and document collection.  Hopefully, more rulings like the one Judge Whalen made in the McMaster case will spur more lawyers to get expert assistance to optimize searches in their cases.  Then again, I’m an optimist, so we’ll see!  :o)

So, what do you think?  Do you have a structured approach to optimize keyword search terms in your cases?  Please share any comments you might have or if you’d like to know more about a particular topic.

Disclaimer: The views represented herein are exclusively the views of the author, and do not necessarily represent the views held by my employer, my partners or my clients. eDiscovery Today is made available solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Today should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.