Evaluating the Millennium Villages

I met with Gabriel Demombynes, a World Bank economist stationed here in Nairobi, this evening. Recently, he pointed out that there was a fundamental flaw in the way in which progress in the Villages is evaluated. Essentially, the evaluation looked at changes in measures of performance over time, but it did not make comparisons to any type of baseline. For example suppose an evaluation finds that the percentage of people engaging in a negative behavior falls from 20% to 10%. If the percentage of people doing this falls, that's a good thing, right? Yes, but it doesn't mean that the Villages should necessarily pat themselves on the back. To see why, suppose that in other comparative villages -- which are not Millennium Villages -- the percentage also falls from 20% to 10%, or perhaps even lower. Now things don't look as good. Measured against the baseline, performance has stayed the same, or even worsened.

That's essentially what Demombynes and his co-authors showed in their academic paper on this topic. The measures for Kisumu in the report are very highly correlated with measures at the national level. The response from Village supporters such as Jeff Sachs was to say that the evaluation wasn't the actual evaluation, and the real one would come soon. But when the second evaluation arrived, it's major finding that infant mortality had fallen significantly in the Villages, was subject to similar questions.

Gabriel, via an email, points to further discussions of these points:

Have a great time in Kisumu. An hour or so away is Sauri, the first Millennium Village. I've had a long back-and-forth with Jeff Sachs about the project's evaluation. Here's an early blog post I wrote about my visit to Sauri:

 http://blogs.worldbank.org/africacan/evaluating-the-millennium-villages More recently I wrote this blog post which led to the MV project retracting its major finding in a paper in Lancet: : http://blogs.worldbank.org/impactevaluations/the-millennium-villages-project-impacts-on-child-mortality 

The retraction on the MVP website: http://www.millenniumvillages.org/field-notes/millennium-villages-project-corrects-lancet-paper 

The retraction in the Lancet: http://press.thelancet.com/MVP.pdf 

…from the Lancet editors…http://download.thelancet.com/flatcontentassets/pdfs/S0140673612607879.pdf my letter with others published in the Lancet: http://www.thelancet.com/journals/lancet/article/PIIS0140-6736%2812%2960848-4/fulltext 

Related: http://retractionwatch.wordpress.com/2012/05/31/millennium-villages-project-forced-to-correct-lancet-paper-on-foreign-aid-as-leader-leaves-team/ 

http://www.economist.com/node/21555571 

http://www.economist.com/blogs/newsbook/2012/05/jeffrey-sachs-and-millennium-villages?fsrc=gn_ep 

http://blog.givewell.org/2012/05/18/millennium-villages-project/

Being here emphasizes how big the problem is, and how important it is to get these measures correct.