Why can't we make good predictions? Maybe we've been doing it wrong:
Scientists devise new method to solve significant variables conundrum, Columbia University: Scientists at Columbia University, the University of California, San Diego (UCSD) and Harvard University have presented an alternative method to address the challenge of using significant variables to make useful predictions in areas such as complex disease.
Shaw-Hwa Lo and Tian Zheng of Columbia, Adeline Lo of UCSD and Herman Chernoff of Harvard present findings in ... Proceedings of the National Academy of Sciences ... that demonstrates that statistically significant variables are not necessarily predictive. In addition, very predictive variables do not necessarily have to appear significant and thereby evade a researcher using statistical significance as a criterion to evaluate variables for prediction.
Statistical significance is a traditional, long-standing measure in any researcher's toolbox but thus far, scientists have been puzzled by the inability to use results of statistically significant variants in complex diseases to make predictions useful for personalized medicine. Why aren't significant variables leading to good prediction of outcomes? This conundrum affects both simple and complex data in a broad range of science and social science fields.
In their findings, the authors demonstrate that what makes variables good for prediction versus significance depends on different properties of the underlying distributions. They suggest that progress in prediction requires efforts toward a new research agenda of searching for a novel criterion to retrieve highly predictive variables rather than highly significant variables.
They also present an alternative approach, the Partition Retention method, which displays strong power in prediction. ...
"What we're saying here is that using the previously very well-known methods might not be appropriate when we care about predicting important outcomes," says Professor Lo. "Our alternative approach seems to do very well in prediction, and is relevant for many scientific fields."