Releasing Preliminary Research to the Public

We present a lot of preliminary research on these sites, research that has not yet passed through the full peer review process. But should preliminary research be released to the public before it has been fully evaluated by experts in the field? As John Kay writes in the Financial Times:

Journalists in search of stories and scientists anxious for publicity and research funding issue early, oversimplified or downright misleading accounts of research. Unsubstantiated claims of a link between immunisation and autism have caused distress to millions of British parents...

What is the solution to this problem? The Royal Society, the UK national academy of science, released a report addressing this issue. Here's a small part of the press release and link to the full report:

Royal Society: Scientists need to consider the public interest for research results: Scientists should consider the public interest when deciding whether, when and how to communicate their research results, according to a report... The report outlines 15 questions that researchers should ask themselves to help them take the public interest into account.

The report, which deals with issues arising from recent controversies ... warns that the results of research presented at conferences may be preliminary, and that media coverage for them may not be in the public interest because they might be shown to be wrong after being checked by peer review. It also notes that some misleading media reports have occurred because researchers have not checked press releases for accuracy... [Download a copy of the report]

Here's an example of research that changed after it was discussed in a New York Times Economic Scene column (it's about the effect of Fox News on voting behavior).

Once research has gone through the peer review process, is it reliable? Should recent cases of research fraud, mistakes, and misinterpretation have been caught by the peer review process? Returning to The Financial Times article:

The Society's answers are self-restraint and peer review. Peer review is the process by which professions review their own work. Articles submitted to journals receive critical assessment from referees experienced in the field. Peer review is a bulwark against cranks, crooks and incompetents. But too much reliance on peer review carries its own dangers. Every profession defines its own concept of excellence in inward-looking ways. Successful academics learn how to trigger the buttons that win the approval of referees...

Referees in the peer review process are usually anonymous to the author of the paper. When you get reports, you rarely know who they are from and because of that the reports can be frank in their appraisals. The freedom to deliver truthful appraisals without fear of retaliation is an important part of the process.

Going back to the Royal Society report, would the process be improved by revealing the identity of referees after publication in cases where papers are accepted?:

Many journals are wary, for good reasons, about revealing the identity of referees involved in peer review. Some critics argue, however, that public confidence in the integrity of the process might be improved if referees allowed their names to be revealed. A debate is needed about lifting the cloak of anonymity when an article has been accepted for publication and has public interest implications. Other changes in journal operation and practice, such as the publication of papers on the world wide web before the completion of peer review, need to be considered in terms of the implications for the public. Similarly, great care is needed when results are communicated to the public, for instance via the media... Lay summaries ... could serve an important function when communicating to the public. Researchers should be encouraged to acquire the skills needed for such communication...

I'm not in favor having referees revealed ex-post. One reason is that it doesn't help with the problems involved with the release of work prior to full peer review. Another reason is the incentives it creates to accept papers from people in position to return favors later, the opposite of the retaliation problem.

In economics the quality of the peer review process is not our main problem anyway, it is the long time it takes to complete the peer review process and it is hurting our junior faculty. I've seen too many cases where the sluggishness of the refereeing process clouded judgment unnecesarily and distorted decisions.

This probably isn't a post that will draw a lot of comment, but our peer review process is not functioning optimally and it's something our profession needs to address before more junior faculty are hurt by the process.