'Economics Needs Replication'

Via Eric Weiner at INET, and continuing a recent discussion, this is Jan Höffler and Thomas Kneib on replication in economics:

Economics Needs Replication, by INET Grantees Jan Höffler and Thomas Kneib: The recent debate about the reproducibility of the results published by Carmen Reinhart and Kenneth Rogoff offers a showcase for the importance of replication in empirical economics.

Replication is basically the independent repetition of scientific analyses by other scientists... The principle is well accepted in the natural sciences. However, it is far less common in empirical economics, even though non-reproducible research can barely be considered a contribution to the consolidated body of scientific knowledge. ...

In the narrow sense, replicability means that the raw data for an analysis can be accessed, that the transformation from the raw data to the final data set is well documented, and that software code is available for producing the final data set and the empirical results. Basically, this comes down to a question of data and code availability, but nonetheless it is a necessary prerequisite for replication. A successful replication would then indicate that all of the material has been provided and that the same results were obtained when redoing the analysis.

In the wider sense, a replication could go much further by challenging previous analyses via changing the data sources (such as changing countries, switching time periods, or using different surveys), altering the statistical or econometric model, or questioning the interpretational conclusions drawn from the analysis. Here, the scientific debate really starts, since this type of replication isn’t concerned with simply redoing exactly the same analysis as the original study. Rather, the goal is to rethink the entire analysis, from data collection and operationalization to the interpretation of results and robustness checks.

Unfortunately, very few journals in economics have mandatory online archives for data and code and/or strict rules for ensuring replicability. Moreover, the incentive for making your own research reproducible, and for reproducing research done by others, is low.

In this respect, there are several interesting lessons to be learned by the Reinhart/Rogoff case.

One is that the impact of replication can actually be quite high, especially when replicating papers that have been influential... Still, it is important to remember that replications that question earlier results are not the only ones that are of value. It is also helpful to know if a specific study could be replicated...

Another important lesson is that involving students in replications can significantly change attitudes towards replication. For students, a replication is a perfect opportunity to perform their own analyses based on an already available paper. They get to learn how experienced scientists tackle applied-research questions and they also learn that the consolidated body of scientific knowledge is constantly changing as it is questioned and transferred to new contexts.

Finally, it is very important for the raw data to be made available so that every step up to the final results of a study can be replicated. ...

In recent years, we have been teaching replication to students at all levels (ranging from undergraduates to Ph.D. candidates) and have set up a large global network to support the idea of replication by students. ...

As a part of our INET project on empirical replication, we therefore are collecting and sharing a large dataset of empirical studies. These studies are all potential candidates for replication that meet the minimal requirements for replicability. Information on these studies, as well as additional information about already published replications, is available in a wiki shared with collaborators who join our teaching initiative. Moreover, we will soon provide via the same wiki website additional resources to support teaching replication seminars. We also started a working paper series on replication so that replication papers can be published as reports and provide a forum for discussing replicability as another part of the wiki website.

We welcome you to join our efforts... You can find more information and contact us here and here.

One place where replication occurs regularly is assignments in graduate classes. I routinely ask students to replicate papers as part of their coursework. Even if they don't find explicit errors (and most of the time they don't), it almost always raises good questions about the research (why this choice, this model, what if you relax this assumption, there's a better way to do this,here's the next question to ask, etc., etc.). So replication doers occur routinely in economics, and it is very valuable, but it is not a formal part of the profession the way it should be, and much of the replication is done by people (students) who generally assume that if they can't replicate something, it is probably their error. We have a lot of work to do on the replication front, and I want to encourage efforts like this.