[author: Michele C.S. Lange*]
It is well established that courts will support parties electing to use technology assisted review (TAR) to identify responsive documents in discovery. However, TAR methodologies and quality control (QC) measures are still very grey for many judges and parties alike. What technologies are most appropriate for identifying the scope of the data universe? How do you sufficiently train the system to properly classify documents? How do you demonstrate that the review method adequately found everything? A new order assists in bringing clarity to this grey space surrounding TAR.
On January 3, 2018, Magistrate Judge Jeffrey Gilbert and Special Master Maura Grossman released a detailed search protocol in In Re Broiler Chicken Antitrust Litigation, a complex, document intensive class action currently pending in the Northern District of Illinois. From deduplication and email threading to sampling and recall, the Order covers an array of topics. And rest assured, Grossman adds in plenty of references to transparency, cooperation, and the importance of regular meet and confers.
This new Order will quickly gain momentum outside of chicken industry lawsuits. In your next e-discovery matter, take heed of the Broiler Chicken search and QC parameters to avoid ending up with egg on your face.
Background of the Case
In September 2016, a food service distributor filed a class action suit on behalf of purchasers of broiler chickens from more than a dozen of the country’s top poultry producers. The action alleged producers conspired to manipulate the supply of chicken to keep prices artificially high, driving up profits.
In February 2017, the plaintiffs filed their first set of requests for production. As foreshadowed by Magistrate Judge Gilbert in a September 2017 discovery order, discovery in this case is likely to be complex and expensive, given 3 putative plaintiff classes, nearly 30 defendants, multiple theories of liability, and conduct covering nearly a decade in a $20-30 billion dollar industry. Perceiving potentially thorny ESI issues, in October 2017, Magistrate Judge Gilbert appointed Special Master Grossman to address and resolve disputes regarding e-discovery. Grossman wasted no time, issuing an Order in January 2018 outlining the document sourcing, search, and validation methods to be used by the parties.
Breakdown of the Order
Grossman begins the Order establishing that the parties will be reasonably transparent regarding the universe of documents collected for search, the search terms utilized, and the processes applied for TAR. Notably, throughout the Order, Grossman refers to this process as “TAR/CAL” (CAL signifying Continuous Active Learning) and never once uses the term “predictive coding."
Next, the Order addresses culling technologies that will be applied prior to searching. Laying this foundation ensures that processing details are clear, so there are no surprises in identifying the scope of documents selected for searching/TAR. Specifically:
The second section of the Order addresses search methods, dissecting this topic into two areas: TAR/CAL searching and keyword searching. If using TAR, Grossman requires the producing party to disclose the name of the software and vendor, how training will work, what categories of documents will be included and excluded, and what quality control measures will be taken. Similarly, the producing party must disclose comparable details about selected keyword search terms, including stop words, case sensitivity, metadata field searching, synonym searching, and misspellings, to name a few. In developing the processes for TAR and keyword searching, Grossman expects the parties to cooperate and work together in good faith to resolve any differences.
The third section of the Order is devoted to a document review validation protocol, which applies no matter TAR or “exhaustive manual review” is used. In detail, a QC sample is created, consisting of 500 responsive documents, 500 non-responsive documents as coded by a human reviewer, and 2,000 unreviewed documents as the result of TAR. These documents are combined into a blind validation set to be reviewed by a human subject matter expert (SME) in the case. From there, a table of each document and its coding will be prepared and provided to the receiving party and Special Master Grossman. The producing party must also provide copy of each responsive, non-produced document found in the sample, along with recall statistics. At this point, parties must meet and confer about the recall estimate, and the quantity and nature of the responsive documents identified through the sampling process to determine if the review is substantially complete. Grossman strongly warned parties about turning to the recall estimate percentage as the sole indicator of an adequate review. Obviously, if there is a dispute, Special Master Grossman can intercede.
Implications for E-Discovery Professionals
If there were any questions about the complexity of e-discovery, this Order lays those doubts to rest. In the coming days and months, this Order will likely be examined at length across our community, but here are three brief takeaways.
*Michele C.S. Lange is a freelance writer and attorney based in Minneapolis, MN. She has more than 15 years of experience in the legal technology industry and can be reached at email@example.com.