The State of Technology Assisted Review in 2017

by Nexsen Pruet, PLLC

Nexsen Pruet, PLLC

The State of Technology Assisted Review in 2017

It has been almost five years since Judge Andrew Peck of the Southern District of New York first approved the use of technology assisted review (“TAR” or “predictive coding” or “computer assisted review”) in Da Silva Moore v. Publicis Groupe.  Since Da Silva Moore, courts and government agencies continue to support the use of TAR to expedite discovery and reduce litigation costs.  However, attorneys have been slow TAR adopters even though they also recognize the benefits of TAR.  Despite statistics and new terminology, the good news is that TAR is not as complicated as one might think.     

Courts Continue to Give TAR a Thumbs Up

“[T]he case law has developed to the point that it is now black letter law that where the producing party wants to utilize TAR for document review, courts will permit it.”[1]  Those were the words of Judge Andrew Peck of the Southern District of New York in 2015.  And it is true that courts across the country continue to accept and even encourage the use of TAR to reduce costs and create efficiencies in the litigation process.  Federal or state courts in California, Delaware, Georgia, Illinois, Minnesota, Nebraska, New Mexico, New York, Pennsylvania, Texas, Tennessee, and Virginia have accepted TAR to search for relevant electronic information since 2012.  

Government agencies including the Department of Justice, the Federal Trade Commission, and the Securities & Exchange Commission may also authorize the use of TAR so long as the TAR methodology is communicated with the agency.  In 2016, courts in Ireland, England and Australia also approved TAR.[2]

In 2016, a producing party’s TAR results were challenged in Dynamo Holdings vs. Commission of Internal Revenue.[3]  The Commission claimed the TAR production was flawed and insufficient and moved to compel the production of documents by using key words and human manual review.  The court concluded that there was “no question” that Dynamo Holdings satisfied its discovery obligations by using TAR. 

The Commission’s motion was “predicated on two myths” according to the court: (1) that human review is accurate and complete and constitutes the gold standard for searching; and (2) that the Rules require a perfect response.  Citing studies undermining the accuracy of keyword searches and human review, the court countered that “[t]his myth of human review is exactly that: a myth.  Research shows that human review is far from perfect.”  Furthermore, the Rules require a “reasonable inquiry” as opposed to “perfection” and cited Judge Peck who stressed that TAR should not be held to a higher standard than keywords or manual review.[4]    

Several other courts have indicated that TAR can be superior to using search terms and/or human review.  In Hyles v. New York, Judge Peck described TAR as “cheaper, more efficient and superior to keyword searching,” and thus “the best and most efficient search tool” for most cases." [5]  The court in Malone vs. Kanter Ingredients observed that TAR is “promoted (and gaining acceptance) as not only a more efficient and cost effective method of ESI review, but a more accurate one.”[6]   Judge Denise Cote of the Southern District of New York has recognized that TAR “may be more reliable — not just as reliable but more reliable than manual review, and certainly more cost effective[.]”[7] The District of Minnesota entered an order in 2016 governing the use of computer assisted review which was designed to “reap the benefits” of TAR including “greater accuracy than keyword search or manual review, reduced costs, and the more rapid document delivery time frames[.]”[8]

Despite acknowledgment that TAR may be more accurate and certainly less costly, courts typically defer to the producing party to determine what search methodology is appropriate.  This approach is consistent with Principle 6 of the Sedona Principles: 

Responding Parties are best situated to evaluate the procedures, methodologies, and technologies appropriate for preserving and producing their own electronically stored information.[9]

Accordingly, several courts declined to dictate what search methodology a party utilizes.[10]

Lawyers Slow to Use TAR

While many courts are sold on the benefits of TAR, attorneys, in general, have yet to embrace TAR with the same enthusiasm and endorsement.  There is a lot of speculation as to why that is as well as how many and how frequently law firms are actually using TAR—but the use of TAR still seems to be the exception.  Attorney Ralph Losey, a TAR proponent, recently suggested that predictive coding is “for average lawyers today, still just a far off remote mountaintop; something they have heard about, but never tried.”[11]

Two other TAR thought leaders, Maura Grossman and Gordan Cormack, noted:

[A]doption of TAR has been remarkably slow, considering the amount of attention these offerings have received since the publication of the first federal opinion approving TAR use . . . .The complex vocabulary and rituals that have come to be associated with TAR, including statistical control sets, stabilization, F1 measure, overturns, and elusion, have dissuaded many practitioners from embracing TAR.[12]

However, the needle may start to move on the use of TAR.  In a 2016 Exterro survey, 90% of attorneys surveyed see benefits of using TAR.[13]  As the legal profession becomes more knowledgeable about TAR – and perhaps as TAR software becomes more user intuitive – the use of TAR should continue to increase.  

So How Does TAR Work?

Have you ever wondered how Pandora creates your music playlist, Amazon makes recommendations on the next book you should read, or your email filters out spam from your inbox?  These are examples of machine learning.  Likewise, TAR involves machine learning algorithms or systematic rules to determine which documents are relevant based on a review of smaller sets of documents by “subject matter experts” – i.e., the attorneys who know the most about the case as opposed to contract attorneys. 

TAR software is not developed to the point that a user pushes a “magic button” and out comes the relevant documents.  However, the TAR process is not overly complex.  At a very high level, the TAR process involves the following basic steps:

1. Define the parameters of the review including:

  • Data sources and/or custodians to be searched;
  • Time frames to be applied
  • Measurements for sampling, recall, and precision
  • Subject matter expert(s).

2. Prepare a review protocol that defines responsiveness and privilege.

3. The subject matter expert(s) review and “train” the documents (commonly referred to as the “seed set”).

4. Apply the subject matter expert training or seed set to a larger set of documents.

5. The subject matter expert(s) should validate the TAR results through different forms of testing and sampling to determine the recall and precision rates.

6. Evaluate the machine review results and then repeat Steps 3, 4, and 5 above until satisfied with the results.

Considerations Before Using Tar

  • Prepare a TAR protocol that is reasonable and proportional to the needs of the case – so that the search methodology is defensible if challenged.
  • Consider the requirements in your jurisdiction as well as the advantages and disadvantages of disclosing the use of TAR to opposing parties or entering into a stipulated order governing a TAR protocol.
  • Consider entering into a Rule 502(d) order to protect the inadvertent production of privilege documents.

[1] Rio Tinto PLC v. Vale S.A., 306 F.R.D. 125, 127 (S.D.N.Y. 2015).

[2] Irish Bank Resolution Corp. v. Quinn; David Brown v. BCA Trading; Pyrrho Investments Ltd. v. MWB Property Ltd.; Money Max v. QBE Ins. Group.

[3] Dynamo Holdings Ltd. P’ship v. Comm’r of Internal Revenue, No. 2685-11, 2016 WL 4204067 (T.C. July 13, 2016).

[4] See also In re Biomet M2a Magnum Hip Implant Products Liability Litig., No. 3:12-MD-2391, 2013 WL 1729682 (N.D. Ind. Apr. 18, 2013) (holding that the defendant’s keyword and TAR search methodologies satisfied its discovery obligations under Rule 26 and Rule 34).

[5] Hyles v. N.Y. City, No. 10CIV3119ATAJP, 2016 WL 4077114, at *2-3 (S.D.N.Y. Aug. 1, 2016).

[6] Malone v. Kantner Ingredients, Inc., No. 4:12CV3190, 2015 WL 1470334, at *3 at FN 7 (D. Neb. Mar. 31, 2015).

[7] Fed. Hous. Fin. Agency v. HSBC N. Am. Holdings Inc., No. 11 CIV. 6189 DLC (S.D.N.Y. July 24, 2014).

[8] In re Bair Hugger Forced Air Warming Prod. Liab. Litig., No. MDL 15-2666(JNE/FLN), 2016 WL 3702959, at *1 (D. Minn. July 8, 2016).

[9] The Sedona Principles: Second Edition, “Best Practices Recommendations & Principles for Addressing Electronic Document Production, Principle 6” (available at

[10] In re Viagra (Sildenafil Citrate) Prod. Liab. Litig., No. 16-MD-02691-RS (SK), 2016 WL 7336411, at *1 (N.D. Cal. Oct. 14, 2016); Kleen Prod. LLC v. Packaging Corp. of Am., No. 10 C 5711, 2012 WL 4498465, at *5 (N.D. Ill. Sept. 28, 2012); Hyles, 2016 WL 4077114.

[11] Ralph Losey, “Predictive Coding 4.0” available at:

[12] Grossman & Cormack, “Continuous Active Learning for TAR” (Practical Law, April/May 2016), available at:

[13] “2nd Annual Federal Judges Survey - Views from Both Sides of the Bench” available at:


DISCLAIMER: Because of the generality of this update, the information provided herein may not be applicable in all situations and should not be acted upon without specific legal advice based on particular situations.

© Nexsen Pruet, PLLC | Attorney Advertising

Written by:

Nexsen Pruet, PLLC

Nexsen Pruet, PLLC on:

Readers' Choice 2017
Reporters on Deadline

"My best business intelligence, in one easy email…"

Your first step to building a free, personalized, morning email brief covering pertinent authors and topics on JD Supra:
Sign up using*

Already signed up? Log in here

*By using the service, you signify your acceptance of JD Supra's Privacy Policy.
Custom Email Digest
Privacy Policy (Updated: October 8, 2015):

JD Supra provides users with access to its legal industry publishing services (the "Service") through its website (the "Website") as well as through other sources. Our policies with regard to data collection and use of personal information of users of the Service, regardless of the manner in which users access the Service, and visitors to the Website are set forth in this statement ("Policy"). By using the Service, you signify your acceptance of this Policy.

Information Collection and Use by JD Supra

JD Supra collects users' names, companies, titles, e-mail address and industry. JD Supra also tracks the pages that users visit, logs IP addresses and aggregates non-personally identifiable user data and browser type. This data is gathered using cookies and other technologies.

The information and data collected is used to authenticate users and to send notifications relating to the Service, including email alerts to which users have subscribed; to manage the Service and Website, to improve the Service and to customize the user's experience. This information is also provided to the authors of the content to give them insight into their readership and help them to improve their content, so that it is most useful for our users.

JD Supra does not sell, rent or otherwise provide your details to third parties, other than to the authors of the content on JD Supra.

If you prefer not to enable cookies, you may change your browser settings to disable cookies; however, please note that rejecting cookies while visiting the Website may result in certain parts of the Website not operating correctly or as efficiently as if cookies were allowed.

Email Choice/Opt-out

Users who opt in to receive emails may choose to no longer receive e-mail updates and newsletters by selecting the "opt-out of future email" option in the email they receive from JD Supra or in their JD Supra account management screen.


JD Supra takes reasonable precautions to insure that user information is kept private. We restrict access to user information to those individuals who reasonably need access to perform their job functions, such as our third party email service, customer service personnel and technical staff. However, please note that no method of transmitting or storing data is completely secure and we cannot guarantee the security of user information. Unauthorized entry or use, hardware or software failure, and other factors may compromise the security of user information at any time.

If you have reason to believe that your interaction with us is no longer secure, you must immediately notify us of the problem by contacting us at In the unlikely event that we believe that the security of your user information in our possession or control may have been compromised, we may seek to notify you of that development and, if so, will endeavor to do so as promptly as practicable under the circumstances.

Sharing and Disclosure of Information JD Supra Collects

Except as otherwise described in this privacy statement, JD Supra will not disclose personal information to any third party unless we believe that disclosure is necessary to: (1) comply with applicable laws; (2) respond to governmental inquiries or requests; (3) comply with valid legal process; (4) protect the rights, privacy, safety or property of JD Supra, users of the Service, Website visitors or the public; (5) permit us to pursue available remedies or limit the damages that we may sustain; and (6) enforce our Terms & Conditions of Use.

In the event there is a change in the corporate structure of JD Supra such as, but not limited to, merger, consolidation, sale, liquidation or transfer of substantial assets, JD Supra may, in its sole discretion, transfer, sell or assign information collected on and through the Service to one or more affiliated or unaffiliated third parties.

Links to Other Websites

This Website and the Service may contain links to other websites. The operator of such other websites may collect information about you, including through cookies or other technologies. If you are using the Service through the Website and link to another site, you will leave the Website and this Policy will not apply to your use of and activity on those other sites. We encourage you to read the legal notices posted on those sites, including their privacy policies. We shall have no responsibility or liability for your visitation to, and the data collection and use practices of, such other sites. This Policy applies solely to the information collected in connection with your use of this Website and does not apply to any practices conducted offline or in connection with any other websites.

Changes in Our Privacy Policy

We reserve the right to change this Policy at any time. Please refer to the date at the top of this page to determine when this Policy was last revised. Any changes to our privacy policy will become effective upon posting of the revised policy on the Website. By continuing to use the Service or Website following such changes, you will be deemed to have agreed to such changes. If you do not agree with the terms of this Policy, as it may be amended from time to time, in whole or part, please do not continue using the Service or the Website.

Contacting JD Supra

If you have any questions about this privacy statement, the practices of this site, your dealings with this Web site, or if you would like to change any of the information you have provided to us, please contact us at:

- hide
*With LinkedIn, you don't need to create a separate login to manage your free JD Supra account, and we can make suggestions based on your needs and interests. We will not post anything on LinkedIn in your name. Or, sign up using your email address.