A recent Third Circuit decision reinforces that courts must rigorously review - and opposing parties should challenge - the reliability of experts' methodologies. In In re Zoloft (Sertraline Hydrochloride) Products Liability Litigation, the Third Circuit affirmed the district court's decision to exclude the testimony of the plaintiffs' expert statistician, Dr. Nicholas Jewell, and grant summary judgment to the defendants. The Third Circuit said Jewell invoked generally reliable scientific methodologies, but he failed to explain his deviations from the standard application of certain scientific techniques under those methodologies. The Third Circuit also confirmed that statistical significance remains an important metric to distinguish between true associations and results occurring by chance.

Here are some practical tips for challenging an expert's methodology based on the Third Circuit's decision - no matter the type of methodology an expert employs.

Third Circuit's Decision

Expert Methodology Must Be Reliably Applied

Jewell claimed Zoloft caused certain birth defects based on his examination of medical literature. He cited Bradford Hill and "weight of the evidence" methodologies to support his opinions. Epidemiologists use Bradford Hill criteria to evaluate distinctions between an association and a causal connection, using such factors as strength of an association, consistency, specificity, temporality, coherence and plausibility. The weight of the evidence analysis, however, is "a flexible methodology" that does not adhere to a particular combination of techniques. The parties agreed that both methodologies are generally reliable.

Importantly, the Third Circuit emphasized that an expert's techniques to employ a methodology must not only be reliable, but also reliably applied. The court accepted that the techniques Jewell used to implement his analysis (meta-analysis, trend analysis and reanalysis) were generally reliable. But Jewell failed repeatedly in his nonstandard application of those techniques that he used without scientific justification:

• Ignoring known ways to analyze trends in insignificant results. Jewell admitted that there was a mathematical calculation for analyzing trends in statistically insignificant results. But he failed to use that calculation and instead merely commented on the data trend.
• Failing to explain selection of studies in meta-analyses. Jewell performed a meta-analysis on two studies, but did not explain why he was justified in doing so, despite critical methodological differences between the studies.  Jewell also did not explain why he excluded a third study that used a similar methodology to one of the studies that was in his meta-analysis.
• Using calculations that are not scientifically validated. Jewell analyzed the strength of association between Zoloft and birth defects by only looking at studies with results favorable to his conclusions. He admitted the calculation he used to analyze these studies was "not scientifically rigorous."
• Ignoring large studies inconsistent with his opinion without explanation. Jewell ignored a study that showed no significant association between Zoloft and birth defects, even though it included more data than the studies he relied on. This omission was particularly noteworthy because the larger study he ignored included much of the same data from the studies he relied on. "Dr. Jewell [also] did not meaningfully discuss" why the larger study showed an "insignificant finding from a presumably better-powered study" in contrast with the smaller studies. "Claiming a consistent result without meaningfully addressing [] alternative explanations . . . undermines reliability."

Importance of Statistical Significance Confirmed

The plaintiffs also raised a question in the appeal that led the Third Circuit to comment on the value of statistical significance. The court declined to make a "bright-line rule" that statistical significance is necessary to prove causality. Nonetheless, statistical significance "remains an important metric to distinguish between results supporting a true association and those resulting from mere chance." The Third Circuit's position is critical, given that other courts, and even the American Statistical Association, have recently questioned the value of statistical significance in assessing whether an association is causal.

Key Implications

The Third Circuit's decision in In re Zoloft underscores the importance of thoroughly interrogating an expert's responses to these questions:

• What does the methodology mean to the expert? Do not accept that an expert's definition of the methodology is the same as used by other scientists. How the expert defines the methodology is important. What steps are required in the methodology? If applicable, how does the methodology differ from other generally accepted scientific methodologies (e.g., Bradford Hill)? What steps were taken to ensure that the methodology produced replicable results? Has the expert's methodology been previously accepted in litigation?
• Did the expert consistently apply an analytical methodology? It is critical to understand how the expert implemented each of the steps in the utilized methodology. How does the expert evaluate the strength of an association? What criteria are used to weigh the studies that support an association against those that do not? What data allow the expert to evaluate whether a dose-response relationship exists?
• How did the expert select evidence? An expert's methodology is no better than the quality of the information being analyzed. Has the expert relied on the same type of data used in the expert's peer-reviewed publications? If not, why not? Outside of litigation, have others relied on the same type of data to support similar conclusions? To what extent has the expert relied on scientific information from peer-reviewed, publicly available sources?
• Is the expert's analysis capable of being tested? An expert's methodology cannot be generally accepted or scientifically reliable if it cannot be scrutinized and replicated.

The unmistakable message of the Third Circuit's decision is that courts should rigorously examine the techniques that experts use to arrive at their conclusions. It is not sufficient for experts to declare that their analyses used methodologies accepted as scientifically reliable in other cases. Experts' application of analytical techniques also must be reliable. If they are not, any departures from standard practice need to be explained. Failure to do so is grounds to exclude expert testimony.