Are Economists Misleading? (Updated)

Language Log says economists are soooo misleading. I couldn't resist adding a few comments in response:

Why are economists so misleading?, Language Log: The "happiness gap" coverage continues over at the NYT, where today's Freakonomics blog picks it up ("Why are women so unhappy?", 10/1/2007), and hundreds of readers are once again pouring out their souls in the comments.

But the real question here is not why women are so unhappy, but why economists (and journalists) are so prone to oversell tiny group differences as if they were universal characteristics of the individual group members. ...

In a couple of earlier posts, I took at quick look at the U.S. General Social Survey data from the unpublished Stevenson and Wolfers paper "The Paradox of Declining Female Happiness", and concluded that the "happiness gap" is pretty underwhelming, despite the deep chord that the news has obviously struck in the public consciousness. ...

Last week, I failed in my search for the other unpublished paper, Alan Krueger's work on trends in sex differences in the amount of time spent on activity perceived as unpleasant. ...

But this afternoon, thanks to a tip from Matt Henning, I found Alan B. Krueger, "Are We Having More Fun Yet? Categorizing and Evaluating Changes in Time Allocation", draft version of 8/23/2007. Unfortunately, Prof. Krueger doesn't give us any information about the within-group (or even overall) distribution of his misery index. However, he does provide the table of time-series data that the NYT writer (David Leonhardt) called "an even starker pattern". Here it is -- I've plotted the data from his Table 4A "U-Index from Men and Women Combined". (Men's data is plotted with red 'm' letters, women's data with blue 'w' letters.)

Now, how many of you really think that the question to ask about this graph is "Why Are Women So Unhappy?", as opposed to the other two questions that I asked at the start of this post?

In Table 4B, Krueger massages the data a bit differently, using sex-specific estimates of activity unpleasantness rather than estimated from pooled evaluations, and comes up with slightly different time functions:

In both time-series, I continue to speculate that the within-group standard deviation of the underlying time-allocation data will be a substantial fraction of the mean values... But whatever the effect size, I leave it to you to judge how "stark" this trend in mean values really is. ...

If he wants to go after David Leonhardt, fine, do so, but don't paint economists with a journalist's interpretation of the data (according to a search of the paper, the words "stark" and "starker" do not appear). That is, dare I say, misleading. Here's what Alan Krueger actually says about these data:

Panel A of Table 4 reports the results. The activity-based U-Index shows very little trend over the last 40 years for men and women combined or for women as a group. For men, however, there has been a shift away from activities associated with unpleasant feelings. To put the estimates in context, note that the difference between the activity-based U-index on weekends and weekdays is about 3 percentage points. Thus, the one point drop in the U-index from 1965-66 to 2005 is about one third of the difference in unpleasant feelings associated with activities during the week and those on the weekend. ...

Panel B of Table 4 combines gender-specific U-indexes for each activity with gender-specific time allocations. The results are generally consistent with those in Table 4b, although they are noisier. The gender-specific weighted U-index displays no trend for women and has trended downward for men over the last 40 years.

Table 5 presents regressions to control for possible changes in the age and education composition of the samples, as well as the survey day and month. ... The regression-adjusted estimates reveal a similar pattern: very little shift toward or away from unpleasant activities, on net, for women, but about a one percentage point shift away from activities associated with unpleasant feelings for men since the mid 1960s.

What is misleading is the presentation of Krueger's results. Table 3, which supports Table 4, is conveniently omitted from the discussion, as is Table 5. In addition, Krueger doesn't say it's a stark pattern, he says it's a one percent change. I would be happier if standard errors and significance levels were presented, and I am not convinced the results will withstand such scrutiny (and even if there is statistical significance, we need to remember that may not translate into economic significance), but the term used by Krueger is "gradual," not "stark":

This paper provides two new methods for classifying activities and evaluating trends in time use. The results indicate that, for the population as a whole, changes in time allocation over the past 40 years have not led to a decrease in the amount of time people spend in activities associated with unpleasant feelings. For men, however, there has been a gradual shift away from activities that are associated with unpleasant feelings, primarily because of a downward trend in paid work and an upward trend in more “affectively neutral downtime” activities, such as “relaxing/doing nothing” and watching television

The other thing I hope we will remember is that this is a working paper, it has not been published in an academic journal and issues such as statistical significance will come up as part of the process (it is specifically identified as a "first draft" and was, apparently, hard to find). Substantial changes may be required before it's ready for publication and issues such as the statistical significance of the findings will surely be part of the refereeing and revision process. Comments such as those above are useful in identifying places in the paper that need more work, but it's unfair to treat this as though it has already been through the refereeing process and is hence representative of published work appearing in top academic journals in economics.

Justin Wolfers is guest blogging at Marginal Revolution this week, and if he chooses to respond, I will update this post. [Update: Justin's response is here].

Comments

[marcello]

>I would be happier if standard errors and significance
>levels were presented,

and if EVER these become regularly presented with _all_ reported economic data, I will die a happier man.

actually, _all_ us us would die happier, with much less stress and gnashing of teeth from, what would be finally correctly interpreted as statistically useless, monthly economic updates.

alas, we won't be so lucky ...

[General Specific]

Do economists ever study the productivity of angels dancing on pin heads?

[reason]

It clearly depends on which derivative you look at.

[Lafayette]

GS: Do economists ever study the productivity of angels dancing on pin heads?

Often, especially when regression doesn't give a plausibly deniable result. ;^)

[piglet]

I would say that even the word "gradual" is misleading. "For men, however, there has been a gradual shift away from activities that are associated with unpleasant feelings

The plots reproduced here, based on table 4A and 4B, do not indicate an even gradual "shift", only some very slight fluctuations. True, the journalist gets most of the blame. But the economist appears to be guilty of inventing a trend where there probably is none.