'Visualizing Economic Uncertainty: On the Soyer-Hogarth Experiment'

Stephen Ziliak, via email:

Dear Mark,

Does graphing improve prediction and increase understanding of uncertainty? When making economic forecasts, are scatter plots better than t-statistics, p-values, and other commonly required regression output?

A recent paper by Emre Soyer and Robin Hogarth suggests the answers are yes, that in fact we are far better forecasters when staring at plots of data than we are when dishing out – as academic journals normally do – tables of statistical significance. [Here is a downloadable version of the Soyer-Hogarth article.]

“The Illusion of Predictability: How Regression Statistics Mislead Experts” was published by Soyer and Hogarth in a symposium of the International Journal of Forecasting (vol. 28, no. 3, July 2012). The symposium includes published comments by J. Scott Armstrong, Daniel Goldstein, Keith Ord, N. Nicholas Taleb, and me, together with a reply from Soyer and Hogarth.

Soyer and Hogarth performed an experiment on the forecasting ability of more than 200 well-published econometricians worldwide to test their ability to predict economic outcomes using conventional outputs of linear regression analysis: standard errors, t-statistics, and R-squared.

The chief finding of the Soyer-Hogarth experiment is that the expert econometricians themselves—our best number crunchers—make better predictions when only graphical information—such as a scatter plot and theoretical linear regression line—is provided to them. Give them t-statistics and fits of R-squared for the same data and regression model and their forecasting ability declines. Give them only t-statistics and fits of R-squared and predictions fall from bad to worse.

It’s a finding that hits you between the eyes, or should. R-squared, the primary indicator of model fit, and t-statistic, the primary indicator of coefficient fit, are in the leading journals of economics - such as the AER, QJE, JPE, and RES - evidently doing more harm than good.

Soyer and Hogarth find that conventional presentation mode actually damages inferences from models. This harms decision-making by reducing the econometrician’s (and profit seeker’s) understanding of the total error of the experiment—or of what might be called the real standard error of the regression, where “real” is defined as the sum (in percentage terms, say) of both systematic and random sources of uncertainty in the whole model. If Soyer and Hogarth are correct, academic journals should allocate more space to visual plots of data and less to tables of statistical significance.

In the blogosphere the statistician Andrew Gelman, INET’s Robert Johnson, and journalists Justin Fox (Harvard Business Review) and Felix Salmon (Reuters) have commented favorably on Soyer's and Hogarth's striking results.

But historians of economics and statistics, joined by scientists in other fields – engineering and physics, for example – will not be surprised by the power of visualizing uncertainty. As I explain in my published comment, Karl Pearson himself—a founding father of English-language statistics—tried beginning in the 1890s to make “graphing” the foundation of statistical method. Leading economists of the day such as Francis Edgeworth and Alfred Marshall sympathized strongly with the visual approach.

And as Keynes (1937, QJE) observed, in economics “there is often no scientific basis on which to form any calculable probability whatever. We simply do not know.” Examples of variables we do not know well enough to forecast include, he said, “the obsolescence of a new invention”, “the price of copper” and “the rate of interest twenty years hence” (Keynes, p. 214).

That sounds about right - despite currently fashionable claims about the role of statistical significance in finding a Higgs boson. Unfortunately, Soyer and Hogarth did not include time series forecasting in their novel experiment though in future work I suspect they and others will.

But with extremely powerful, dynamic, and high-dimensional visualization software such as “GGobi” – which works with R and is currently available for free on-line - economists can join engineers and rocket scientists and do a lot more gazing at data than we currently do (http://www.ggobi.org).

At least, that is, if our goal is to improve decisions and to identify relationships that hit us between the eyes.

Kind regards,

Stephen T. Ziliak
Professor of Economics
Roosevelt University