The Heart of the Matter: Cutting E-Discovery Review Costs by Tracking Relevancy Rates (and Other Metrics)

by Exterro, Inc.

Exterro Comprehensive Guide to E-Discovery Review

With different fact patterns, laws, custodians, and companies, every review is going to be unique. When outside review teams are used for a company, multiple groups are doing different things in different ways, and it's very difficult for corporate counsel know exactly what is working.

A repeatable process allows in-house legal and the outside law firms and/or vendors doing the review to compare information or feedback, ensuring the result will be consistent. This requires planning and defining roles and responsibilities for all involved.

Create Communication and Transparency Between All Stakeholders

For checks and balances to work, multiple teams must have transparency and great communication within the preservation and collection process, which may include In-house legal, technology vendors, outside counsel, and the review team. Without communication, transparency, and a knowledge of scope, inefficiencies begin to show up in the process.

Some legal teams utilize an e-discovery lead or an e-discovery specialist that acts as a conduit between IT and legal. Legal knows what they want to preserve, at least at a fairly high level, and IT knows where all the data lives. Between the two of them, they must make sure they're on the same page and understand exactly what they're talking about. This is where a liaison, who can speak both legal and tech, can avoid a lot of problems.

Another way open communication can create a more repeatable process, thereby saving on review, is instead of thinking of each matter as its own unique instance with its own preservation and/or collection set, teams should strive for the kind of transparency that would give them knowledge of what has already been preserved for other matters, whether they are similar in nature or not. If there is custodian overlap between matters, some of the same data may already have been locked down, so there is no need to duplicate that effort.

Tracking Review Relevancy Rates

Even after narrowing the funnel of data in the preservation and collection stages, there is still irrelevant ESI that gets through to the review stage. Review Relevancy Rates measure how much of the data sent for review is relevant to the case. If the relevancy rate is 1%, it might be because this was a government investigation where a very broad preservation and collection couldn’t be avoided, or it may have been something where targeted preservation or targeted collection wasn’t used. Once the things contributing to that 1% relevancy rate are located, checks and balances can be put in place upstream in the hopes of improving those rates, which in turn will cut costs.

If review isn’t being done in-house, it’s important to ask the outside review team for these metrics. This goes back to creating transparent lines of communication. This knowledge should be transferred back and forth, so that reviews going forward can be more effective and efficient.

Using Metrics to Track Review Performance

A lot of flexibility and granular information is needed in order to evaluate performance, but more important is the ability to quickly collapse that information into a meaningful report. This goes back to the idea that every case and every review is unique, even when working with the same enterprise client or law firm. There are different angles and things that an attorney, or in-house counsel, or a service provider is going to care about as they're looking at this data to understand performance.

Another area where metrics can help are with budgets, which are surprisingly often overlooked, even with the knowledge that review is the most expensive e-discovery stage. With the metrics that review tools can provide, the project can be looked at when it comes in—data set, time it will take to review, the exact numbers in terms of relevant and privileged ESI. But what is forgotten in many instances is continuing to track budgets during the review. When budget concerns are addressed as the project goes along, then calls from in-house counsel saying, "What happened, this is three times what you projected in your budget," are avoided.

By keeping everything transparent with all stakeholders, then even if a budget projection based on a document set changes after getting into the data and conducting ECA, everyone can be kept up to date in real time. It's a collaborative approach. In-house counsel will be interested in changing budgetary issues. Outside counsel will be interested in how changes in the number of reviewers or size of the dataset will affect deadlines. The same numbers may be reported to both, but they mean different things to different players.

Getting More from your E-Discovery Software During Review

  • Custodian Interviews and In-Place Preservation

Just before issuing legal hold notices, and after as well, custodian interviews are vital for understanding what data they really have, and how much they are aware of the claims of the case. Not only can technology make the interview process easier and more transparent, it can also alleviate the concern about the growing number of data types, as well as data volume, with tools like in-place preservation (Office 365 is a great example of IPP capability, where all data stored in Exchange, One Drive, SharePoint, etc, is automatically preserved without making duplicate copies).

  • Sampling / Targeted Collections

Take advantage of Early Case Assessment tools, while using more advanced search techniques, automation for privilege calls, or leveraging machine-learning to help identify relevant documents. Other powerful tools can use analytics for things like concept clustering and email thread analysis.

  • Using a Unified Technology Platform

Using multiple e-discovery tools creates a lot of inefficiencies. The ideal is a single end-to-end platform that orchestrates all elements of the e-discovery process, putting the tasks together from preservation through review, including customizable reporting features for all those tasks, which can continue throughout the life of the project. Combine all this with managing and optimizing the processes for the human aspect review, and the result is full orchestration over the entire e-discovery process.


Much can be done before, during, and after the review stage to gain performance and cut costs, and much of that has to do with mending fractured processes and lines of communication between the stake holders so that knowledge can be more effectively shared, resulting in smaller datasets, fewer fire drills, smoother handoffs, and a more efficient approach overall.

For more on how you can create a more efficient and cost-effective review process, Download Exterro's Latest E-Book, The Comprehensive Guide to E-Discovery Review

[View source.]

Written by:

Exterro, Inc.

Exterro, Inc. on:

Readers' Choice 2017
Reporters on Deadline

"My best business intelligence, in one easy email…"

Your first step to building a free, personalized, morning email brief covering pertinent authors and topics on JD Supra:
Sign up using*

Already signed up? Log in here

*By using the service, you signify your acceptance of JD Supra's Privacy Policy.
Custom Email Digest
Privacy Policy (Updated: October 8, 2015):

JD Supra provides users with access to its legal industry publishing services (the "Service") through its website (the "Website") as well as through other sources. Our policies with regard to data collection and use of personal information of users of the Service, regardless of the manner in which users access the Service, and visitors to the Website are set forth in this statement ("Policy"). By using the Service, you signify your acceptance of this Policy.

Information Collection and Use by JD Supra

JD Supra collects users' names, companies, titles, e-mail address and industry. JD Supra also tracks the pages that users visit, logs IP addresses and aggregates non-personally identifiable user data and browser type. This data is gathered using cookies and other technologies.

The information and data collected is used to authenticate users and to send notifications relating to the Service, including email alerts to which users have subscribed; to manage the Service and Website, to improve the Service and to customize the user's experience. This information is also provided to the authors of the content to give them insight into their readership and help them to improve their content, so that it is most useful for our users.

JD Supra does not sell, rent or otherwise provide your details to third parties, other than to the authors of the content on JD Supra.

If you prefer not to enable cookies, you may change your browser settings to disable cookies; however, please note that rejecting cookies while visiting the Website may result in certain parts of the Website not operating correctly or as efficiently as if cookies were allowed.

Email Choice/Opt-out

Users who opt in to receive emails may choose to no longer receive e-mail updates and newsletters by selecting the "opt-out of future email" option in the email they receive from JD Supra or in their JD Supra account management screen.


JD Supra takes reasonable precautions to insure that user information is kept private. We restrict access to user information to those individuals who reasonably need access to perform their job functions, such as our third party email service, customer service personnel and technical staff. However, please note that no method of transmitting or storing data is completely secure and we cannot guarantee the security of user information. Unauthorized entry or use, hardware or software failure, and other factors may compromise the security of user information at any time.

If you have reason to believe that your interaction with us is no longer secure, you must immediately notify us of the problem by contacting us at In the unlikely event that we believe that the security of your user information in our possession or control may have been compromised, we may seek to notify you of that development and, if so, will endeavor to do so as promptly as practicable under the circumstances.

Sharing and Disclosure of Information JD Supra Collects

Except as otherwise described in this privacy statement, JD Supra will not disclose personal information to any third party unless we believe that disclosure is necessary to: (1) comply with applicable laws; (2) respond to governmental inquiries or requests; (3) comply with valid legal process; (4) protect the rights, privacy, safety or property of JD Supra, users of the Service, Website visitors or the public; (5) permit us to pursue available remedies or limit the damages that we may sustain; and (6) enforce our Terms & Conditions of Use.

In the event there is a change in the corporate structure of JD Supra such as, but not limited to, merger, consolidation, sale, liquidation or transfer of substantial assets, JD Supra may, in its sole discretion, transfer, sell or assign information collected on and through the Service to one or more affiliated or unaffiliated third parties.

Links to Other Websites

This Website and the Service may contain links to other websites. The operator of such other websites may collect information about you, including through cookies or other technologies. If you are using the Service through the Website and link to another site, you will leave the Website and this Policy will not apply to your use of and activity on those other sites. We encourage you to read the legal notices posted on those sites, including their privacy policies. We shall have no responsibility or liability for your visitation to, and the data collection and use practices of, such other sites. This Policy applies solely to the information collected in connection with your use of this Website and does not apply to any practices conducted offline or in connection with any other websites.

Changes in Our Privacy Policy

We reserve the right to change this Policy at any time. Please refer to the date at the top of this page to determine when this Policy was last revised. Any changes to our privacy policy will become effective upon posting of the revised policy on the Website. By continuing to use the Service or Website following such changes, you will be deemed to have agreed to such changes. If you do not agree with the terms of this Policy, as it may be amended from time to time, in whole or part, please do not continue using the Service or the Website.

Contacting JD Supra

If you have any questions about this privacy statement, the practices of this site, your dealings with this Web site, or if you would like to change any of the information you have provided to us, please contact us at:

- hide
*With LinkedIn, you don't need to create a separate login to manage your free JD Supra account, and we can make suggestions based on your needs and interests. We will not post anything on LinkedIn in your name. Or, sign up using your email address.