Does Statistical Significance Stink?

Steve Ziliak emails:

Dear Mark,

When is anosmia - the permanent loss of smell - "significant"?

[There is] a Supreme Court case - to be argued on January 10, 2011 - involving the difference between "statistical" and "practical" significance.

Should drug manufacturers and other companies be required to report the adverse effect of a product on users, if the effect is not statistically significantly different from zero at the 5% level? Can people file claims against companies which fail to warn shareholders about adverse effects that are not statistically significant at the 5% level? These are two of the questions to be decided by the Supreme Court in light of the Securities Exchange Act of 1934.

The case at hand involves Zicam, a zinc-based common cold medicine from Matrixx which evidently causes some users to permanently lose their sense of smell. (It's the nasal form of Zicam, spray and swab, not the pill, that seems to be the main culprit. Over 200 users in the U.S. have filed law suits, with more pending.)

On November 12, 2010, the economist Deirdre N. McCloskey and I filed an amicus brief with the Supreme Court of the United States.

Matrixx, the maker of Zicam, claims that the adverse effect (loss of smell) was not in clinical trials "statistically significant" at the 5% level of statistical significance and thus they claim, incorrectly it turns out, that the adverse effect was not "important" to human health nor to shareholders' wealth. Goodbye, Italian beef! So long lovers, English gardens, and wines of Bordeaux! Your scent is not significant.

Our brief explains - as we do in many articles and in a book, The Cult of Statistical Significance (2008) - why the Supreme Court should not use a bright-line standard of statistical significance, 5% or other, to determine materiality. What matters is in a word "Oomph" (real practical, human, and economic effect) and the odds of getting it. We call it in the brief, "practical importance".

The Acting Solicitor General of the U.S., the AARP, and other economists and lawyers (such as Joseph Mason and Robert Litan) agree. They understand the danger and costs of the current "significance" rule, and contributed a brief saying so to the Supreme Court.

The oral argument is scheduled for January 10, 2011. It is said to be one of the top ten cases of the year as the Court's decision has general and large implications for liability, regulation, and reporting by the Securities and Exchange Commission. Billions of dollars are at stake, true. But so are billions of lives - live smells included. In lower court trials the company claimed it hasn't harmed shareholders or users for not reporting the adverse effect.

Here is our amicus brief. Here is a short paper describing Ziliak's and McCloskey's objection to statistical significance and advocacy of an "economic approach to the logic of uncertainty."

(The above paper, on The Cult of Statistical Significance, is a copy of a lecture I gave at the American Statistical Association/Joint Statistical Meetings, August 2009, in Washington, DC.)

And here are some related articles by me on randomization and significance in medical and economic trials, published in The Lancet (link, link)

Hope your New Year is peaceful and lovely, with all the best smells and tastes,

Sincerely,

Steve Ziliak