« links for 2011-07-05 | Main | The *Brief* Return of Depression Economcs??? »

Wednesday, July 06, 2011

Are Working Papers Working?

Should we abandon working papers?:

Working Papers are NOT Working, by Berk Özler: ...It is common practice in economics to publish working papers. There are formal working paper series such as NBER, BREAD, IZA, World Bank Policy Research Working Paper Series, etc. With the proliferation of the internet, however, people don’t even need to use these formal working paper series. You can simply post your brand new paper on your website and violà, you have a working paper: put that into your CV! Journals are giving up double-blind refereeing (AEJ is the latest) because it is too easy to use search engines to find the working paper version (it’s not at all clear that this is good...). But, do the benefits of making these findings public before peer-review outweigh the costs? I recently became very unsure…
In economics, publication lags, even for journals that are fast, can be long: it is not uncommon to see articles that state: Submitted December 2007; accepted August 2010. ... But, research findings are public goods and working papers are a way to get this information out to parties who can benefit from the new information while the paper is under review.
But, that assumes that the findings are ready for public consumption at this preliminary stage. By preliminary, I mean papers that have not yet been seriously reviewed by anyone familiar with the methods and the specific topic. Findings, and particularly interpretations, change between the working paper phase and the published version of a paper: if they didn’t, then we would not need peer-reviewed journals. Sometimes, they change dramatically. ...
When a new working paper comes out, especially one that might be awaited (like the first randomized experiment on microfinance), people rush to read it (or, rather, skim it). It gets downloaded many times, gets blogged about, etc. Then, a year later a new version comes out (maybe it is even the published version). Many iterations of papers simply improve on the original premise, provide more robustness checks, etc.. But, interpretations often change; results get qualified; important heterogeneity of impacts is reported. And sometimes, main findings do change. What happens then?
People are busy. Most of them had only read the abstract (and maybe the concluding section) of the first draft working paper to begin with. ... The newer version, other than for a few dedicated followers of the topic or the author, will not be read by many. They will cling to their beliefs based on the first draft: first impressions matter. ...
There is another problem: people who are invested in a particular finding will find it easier to take away a message that confirms their prior beliefs from a working paper. They will happily accept the preliminary findings of the working paper and go on to cite it for a long time (believe me, well past the updated versions of the working paper and even the eventual journal publication). People who don’t buy the findings will also find it easy to dismiss them: the results are not peer-reviewed. At least, the peer-review process brings a degree of credibility to the whole process and makes it harder for people to summarily dismiss findings they don’t want to believe.
I have some firsthand experience with this, as my co-authors and I have a working paper, the findings of which changed significantly over time. In March 2010, we put out a working paper on the role of conditionalities in cash transfer programs, which we also simultaneously submitted to a journal. The paper was reporting one-year effects of an intervention using self-reported data on school participation. ...
What’s the problem? Our findings in the March 2010 version suggested that CCTs that had regular school attendance as a requirement to receive cash transfers did NOT improve school enrollment over and above cash transfers with no strings attached. Our findings in the December 2010 version DID. ...
However, the earlier (and erroneous) finding that conditions did not improve schooling outcomes was news enough that it stuck. Many people, including good researchers, colleagues at the Bank, bloggers, policymakers, think that UCTs are as effective as CCTs in reducing dropout rates – at least in Malawi. And, this is with good reason: it was US who screwed up NOT them! Earlier this year, I had a magazine writer contact me to ask whether there was a new version of the paper because her editor uncovered the updated findings while she was fact-checking the story before clearing it for publication. As recently as yesterday, comments on Duncan Green’s blog suggested that his readers, relying on his earlier blogs and other blogs, are not aware of the more recent findings. Even my research director was misinformed about our findings until he had to cite them in one of his papers and popped into my office.
Many working papers will escape this fate – which is definitely not the norm. But, no one can tell me that working papers don’t improve and change over time as the authors are pushed by reviewers who are doing their best to be skeptical and provide constructive criticism. But, it turns out that those efforts are mainly for the academic crowd or for the few diligent policymakers who are discerning users of evidence. ...
So, what if we chose to not have working papers? There is no doubt that the speed with which journals publish submitted papers would have to change. ...
If we didn't have working papers, we could also go back to double blind reviews again. No, it won’t be perfect, but double-blind was there for a reason. I see serious equity concerns with single blind reviews ...

Double blind has its problems as well. If you are active at NBER meetings, at a top university, etc., etc., and you get a paper to referee that you haven't already seen presented somewhere (or at least reviewed as as submission to a conference), often more than once, you will draw conclusions. But single blind removes all doubt, so it's no better on this score.

On the issue of working papers, sensational results -- the ones most likely to be costly if they change later -- are going to leak in their preliminary form, and they are going to be reported. When that happens, I'd rather that the experts in the field be aware of the paper already or have easy access to it so that they can qualify the results as needed, or at least try to. Without the checks and balances of other researchers to help reporters and policymakers with the interpretation of the results, etc., this could lead to even worse policy errors than before. More generally, I'm not convinced that the costs -- the times when economics working papers have caused changes in policy that are later regretted -- exceed the benefits to other researchers of having this information available sooner rather than later (if only, for example, to know what questions other people are working on, new techniques that are being used, and so on -- the results themselves are not the only way this information is helpful).

    Posted by on Wednesday, July 6, 2011 at 12:33 AM in Academic Papers, Economics | Permalink  Comments (4)


    Comments

    Feed You can follow this conversation by subscribing to the comment feed for this post.