Search and Collection of Data Efforts Critiqued in FOIA Case

by Morgan Lewis

[authors: Stephanie A. Blair, Scott A. Milner, and Denise E. Backhouse]

Absent showing of sound methodology, Judge Scheindlin finds identification, self-collection, and keywords to be insufficient.

In her latest opinion in the National Day Laborer matter,[1] Judge Shira A. Scheindlin of the U.S. District Court for the Southern District of New York highlights many of the pitfalls encountered in identifying, collecting, searching, and producing relevant information in the discovery process. Although decided under the demanding standard of the Freedom of Information Act (FOIA),[2] many of the lessons apply in the context of the Federal Rules of Civil Procedure (FRCP). Judge Scheindlin applies insights from cases decided under the FRCP, noting that "much of the logic behind the increasingly well-developed caselaw on e-discovery searches is instructive in the FOIA search context because it educates litigants and courts about the types of searches that are or are not likely to uncover all responsive documents."


In National Day Laborer, the plaintiffs made a request under the FOIA seeking information from five federal agencies concerning the controversial Secure Communities program. In response to the plaintiffs' requests (and a court order), the defendant agencies performed "massive" searches and made voluminous productions. The parties then cross-moved for summary judgment based on whether those searches (and consequently, the productions) were adequate. To support their motions, each defendant agency filed a declaration describing its efforts. Those declarations varied greatly in level of detail and illustrated a multitude of different approaches. In the end, Judge Scheindlin ordered further action by the defendants, the extent of which was driven not only by the accuracy of the searches performed, but also by the degree to which those search efforts were documented and verifiable.

Scope of Required Searches

Much of the opinion focuses on which custodians' files should be searched. Several of the defendant agencies contended that a reasonable search had been performed when the records of those custodians most likely to have responsive information were searched. The court disagreed, noting that the government is "not required to search only the files of the . . . custodians who are 'most likely' to have responsive records; it must also search other locations that are reasonably likely to contain records." Paralleling her findings in Pension Committee[3] and other matters decided under the FRCP, Judge Scheindlin found that the government attorneys should contact potential custodians to determine whether they were involved in the activity at issue and have relevant information. The National Day Laborer opinion also reiterates the duty to follow up with custodians who fail to respond after receiving a request to provide responsive documents. The court noted that it is "plainly improper" to view "a non-response as a 'no records' response." The court also noted that the following inactions may render a search for responsive information inadequate:

  • Failure to search the records of former employees
  • Failure to search archived records
  • Failure to follow through on obvious leads, such as searching the records of individuals copied on key correspondence
  • Failure to search, or to describe the extent of the search, for materials in the possession of third parties, including independent contractors retained by the government who were involved in the decision-making process

Document Identification and Review Methodology

In attempting to defend its search for responsive materials as adequate, the Federal Bureau of Investigation (FBI) simply stated in its declaration that it had conducted a "manual review" of documents in order to find relevant records without describing what this actually entailed. Judge Scheindlin could not determine that the phrase "manual review" indicated that every document was reviewed by human eyes, as it was much more likely that the FBI custodians looked through specific categories of documents or searched using keywords. Absent evidence of what specifically had been done to locate relevant information, it was found that the FBI failed to carry its burden of showing under the FOIA standard that its search was reasonably calculated to uncover all relevant records.

Reliance on Custodian Self-Collection Without Appropriate Supervision

Several of the responding government agencies did not monitor custodians but instead relied on them to conduct appropriate searches to gather relevant information. Judge Scheindlin noted that "most custodians cannot be 'trusted' to run effective searches because designing legally sufficient electronic searches in the discovery or FOIA contexts is not part of their daily responsibilities." Quoting her opinion in Pension Committee, Judge Scheindlin reiterated, "'I note that not every employee will require hands-on supervision from an attorney. However, attorney oversight of the process, including the ability to review, sample, or spot-check the collection efforts[,] is important.'" The court will not simply take at face value unsupported assertions that custodians have "designed and conducted a reasonable search." Counsel must take appropriate steps to oversee and monitor the process, documenting each step along the way.

The Limitations of Keyword Searches

A number of defendant agencies provided custodians with lists of keywords to use in searching for relevant information. For the most part, the custodians were asked to run the searches themselves and provide the resulting relevant documents. However, these search terms were not tested, and documents without search-term hits were not sampled. Running parallel to cases decided under the FRCP, the opinion highlights the inadequacies of simple keyword searches and the need for parties using such terms to establish their efficacy.[4] Looking beyond keyword search, the court noted the "parties can (and frequently should) rely on latent semantic indexing, statistical probability models, and machine learning tools to find responsive documents," explaining that "[t]hrough iterative learning, these methods (known as 'computer-assisted' or 'predictive' coding) allow humans to teach computers what documents are and are not responsive to a particular FOIA or discovery request and they can significantly increase the effectiveness and efficiency of searches."[5]

The court further opined that "in order to determine adequacy, it is not enough to know the search terms. The method in which they are combined and deployed is central to the inquiry." Without such evidence, the court could not assess the adequacy of the searches. In order to rectify any shortcomings in the defendants' responses without incurring wasteful and redundant re-searching, the court ordered the parties to cooperate and work together to craft search terms and protocols, including methods to test search-term validity.


While this case is specific to the high standard imposed by the FOIA, lessons regarding the shortcomings of untested keyword searches, inadequately supervised custodian self-collection, and failure to search in appropriate locations also apply where discovery is governed by the FRCP. It is no less important in that context to have a defensible discovery plan and a sound process for its implementation.

[1]. Nat'l Day Laborer Org. Network v. U.S. Immigration & Customs Enforcement Agency, 10 Civ. 3488 (SAS), 2012 U.S. Dist. Lexis 97863 (S.D.N.Y. July 13, 2012), available here. Please note: on February 10, 2011, we issued a LawFlash regarding an earlier decision by Judge Scheindlin in the National Day Laborer matter addressing metadata issues; that decision was subsequently withdrawn and deemed to have no precedential value by an Order dated June 17, 2011. Please see our LawFlash here.

[2]. In responding to requests under the FOIA, the government is required to show "'beyond material doubt that it has conducted a search reasonably calculated to uncover all relevant documents.'" Nat'l Day Laborer, slip op. at 6–7 (quoting Morley v. Cent. Intel. Agency, 508 F.3d 1108, 1114 (D.C. Cir. 2007)) (emphases added).

[3]. Pension Comm. of Univ. of Montreal Pension Plan v. Banc of Am. Sec., LLC, 685 F. Supp. 2d 456 (S.D.N.Y. 2010).

[4]. See Nat'l Day Laborer, slip op. at 37-42 (reviewing issues with keyword search and gathering related case law and commentaries).

[5]. Id. at 40.


DISCLAIMER: Because of the generality of this update, the information provided herein may not be applicable in all situations and should not be acted upon without specific legal advice based on particular situations.

© Morgan Lewis | Attorney Advertising

Written by:

Morgan Lewis

Morgan Lewis on:

Readers' Choice 2017
Reporters on Deadline

"My best business intelligence, in one easy email…"

Your first step to building a free, personalized, morning email brief covering pertinent authors and topics on JD Supra:
Sign up using*

Already signed up? Log in here

*By using the service, you signify your acceptance of JD Supra's Privacy Policy.
Privacy Policy (Updated: October 8, 2015):

JD Supra provides users with access to its legal industry publishing services (the "Service") through its website (the "Website") as well as through other sources. Our policies with regard to data collection and use of personal information of users of the Service, regardless of the manner in which users access the Service, and visitors to the Website are set forth in this statement ("Policy"). By using the Service, you signify your acceptance of this Policy.

Information Collection and Use by JD Supra

JD Supra collects users' names, companies, titles, e-mail address and industry. JD Supra also tracks the pages that users visit, logs IP addresses and aggregates non-personally identifiable user data and browser type. This data is gathered using cookies and other technologies.

The information and data collected is used to authenticate users and to send notifications relating to the Service, including email alerts to which users have subscribed; to manage the Service and Website, to improve the Service and to customize the user's experience. This information is also provided to the authors of the content to give them insight into their readership and help them to improve their content, so that it is most useful for our users.

JD Supra does not sell, rent or otherwise provide your details to third parties, other than to the authors of the content on JD Supra.

If you prefer not to enable cookies, you may change your browser settings to disable cookies; however, please note that rejecting cookies while visiting the Website may result in certain parts of the Website not operating correctly or as efficiently as if cookies were allowed.

Email Choice/Opt-out

Users who opt in to receive emails may choose to no longer receive e-mail updates and newsletters by selecting the "opt-out of future email" option in the email they receive from JD Supra or in their JD Supra account management screen.


JD Supra takes reasonable precautions to insure that user information is kept private. We restrict access to user information to those individuals who reasonably need access to perform their job functions, such as our third party email service, customer service personnel and technical staff. However, please note that no method of transmitting or storing data is completely secure and we cannot guarantee the security of user information. Unauthorized entry or use, hardware or software failure, and other factors may compromise the security of user information at any time.

If you have reason to believe that your interaction with us is no longer secure, you must immediately notify us of the problem by contacting us at In the unlikely event that we believe that the security of your user information in our possession or control may have been compromised, we may seek to notify you of that development and, if so, will endeavor to do so as promptly as practicable under the circumstances.

Sharing and Disclosure of Information JD Supra Collects

Except as otherwise described in this privacy statement, JD Supra will not disclose personal information to any third party unless we believe that disclosure is necessary to: (1) comply with applicable laws; (2) respond to governmental inquiries or requests; (3) comply with valid legal process; (4) protect the rights, privacy, safety or property of JD Supra, users of the Service, Website visitors or the public; (5) permit us to pursue available remedies or limit the damages that we may sustain; and (6) enforce our Terms & Conditions of Use.

In the event there is a change in the corporate structure of JD Supra such as, but not limited to, merger, consolidation, sale, liquidation or transfer of substantial assets, JD Supra may, in its sole discretion, transfer, sell or assign information collected on and through the Service to one or more affiliated or unaffiliated third parties.

Links to Other Websites

This Website and the Service may contain links to other websites. The operator of such other websites may collect information about you, including through cookies or other technologies. If you are using the Service through the Website and link to another site, you will leave the Website and this Policy will not apply to your use of and activity on those other sites. We encourage you to read the legal notices posted on those sites, including their privacy policies. We shall have no responsibility or liability for your visitation to, and the data collection and use practices of, such other sites. This Policy applies solely to the information collected in connection with your use of this Website and does not apply to any practices conducted offline or in connection with any other websites.

Changes in Our Privacy Policy

We reserve the right to change this Policy at any time. Please refer to the date at the top of this page to determine when this Policy was last revised. Any changes to our privacy policy will become effective upon posting of the revised policy on the Website. By continuing to use the Service or Website following such changes, you will be deemed to have agreed to such changes. If you do not agree with the terms of this Policy, as it may be amended from time to time, in whole or part, please do not continue using the Service or the Website.

Contacting JD Supra

If you have any questions about this privacy statement, the practices of this site, your dealings with this Web site, or if you would like to change any of the information you have provided to us, please contact us at:

- hide
*With LinkedIn, you don't need to create a separate login to manage your free JD Supra account, and we can make suggestions based on your needs and interests. We will not post anything on LinkedIn in your name. Or, sign up using your email address.