« Demand Creates Its own Supply | Main | Reich: The Baker Commission Report is a Non-Event »

Thursday, December 07, 2006

NY Fed: Tracking Productivity in Real-Time

This is a bit technical, but I have confidence you can handle it. One problem in macroeconomic policy is that data are often only available with a substantial lag making assessment of the state of the economy difficult. This article from the New York Fed describes how to estimate the permanent and transitory changes in productivity growth in real-time which helps to overcome the problem. Note, however, that real-time in this case means detecting "changes in trend productivity within a year or two of their occurrence" which, though still a long lag, is an improvement. The good news is that, according to these estimates, "the underlying trend remains strong despite recent weak productivity data":

Tracking Productivity in Real Time, by Authors: James A. Kahn and Robert W. Rich, Current Issues in Economics and Finance, November 2006, New York Fed View PDF version: Because volatile short-term movements in productivity growth obscure the underlying trend, shifts in this trend may go unrecognized for years—a lag that can lead to policy mistakes and hence economic instability. This study develops a model for tracking productivity that brings in additional variables to help reveal the trend. The model’s success is evident in its ability to detect changes in trend productivity within a year or two of their occurrence. Currently, the model indicates that the underlying trend remains strong despite recent weak productivity data.

Economists since Adam Smith have regarded productivity growth as the key driver of long-run improvements in living standards[1] But while productivity’s contribution to the population’s well-being is widely acknowledged, its underlying trend has proved to be difficult to track. Labor productivity—output per hour of work—has risen at an average annual rate of 2.3 percent in the U.S. nonfarm business sector during the post–World War II period.[2] With the benefit of hindsight, however, we know that trend productivity growth has differed markedly from this average over extended periods of time. In particular, it substantially exceeded 2.3 percent from 1948 to 1973, and again after 1996, and fell far short of 2.3 percent between 1973 and 1995. Significantly, these shifts in trend were not detected until well after their onset.

The difficulty in assessing the trend in productivity growth stems primarily from the extreme volatility of quarterly growth rates. In any one quarter, annualized growth rates in excess of 5 percent or below zero are common (Chart 1).[3] Moreover, the volatility is not confined to short-term movements in this series; productivity growth also fluctuates with the business cycle, typically declining during a recession and rising sharply at the onset of a recovery. Thus, economists cannot easily distinguish changes in trend productivity from quarterly or cyclical swings, and years may pass before a trend shift can be ascertained.

Chart 1
Quarterly Nonfarm Productivity Growth

Chart 1 - Quarterly Nonfarm Productivity Growth
Source: U.S. Department of Labor, Bureau of Labor Statistics.
Note: The data are expressed as annualized percentage rates.

Not surprisingly, the misidentification of trend changes can have significant consequences for the economy. It is widely believed, for example, that the inability to recognize the slowdown in trend productivity growth in the early 1970s led policymakers to overestimate potential GDP growth and set interest rates too low—actions that contributed to double-digit inflation over the next several years.

In this edition of Current Issues, we present a methodology for tracking productivity growth that is designed to distinguish between permanent and transitory movements. We apply this methodology to real-time data—the data historically available at each point in time—to discern trend shifts that are often obscured by the volatility of the productivity numbers. Specifically, we construct a statistical model that includes, in addition to productivity, two variables that economic theory predicts will move together with productivity over the long term: real (inflation-adjusted) consumption expenditure and real labor compensation. By looking at all three of these economic series at once, the model can more easily uncover the trend that underlies them all. In this respect, our approach resembles the way policymakers gauge the current state of the economy by extracting a signal, with the help of an analytical framework, from a wide array of noisy indicators.[4]

Our model allows for shifts between high- and low-growth productivity “regimes” and, at each point in time, yields an estimate of the probability of being in one regime or the other. We find that it is largely successful in detecting the shifts from one regime to the other, using only the data then available. Most notably, the model would have done well in recognizing the nature of the acceleration in productivity in the second half of the 1990s. While many economists at the time argued that the productivity gains of this period were cyclical, our model would have identified the faster productivity growth as a permanent change, and would have done so within two years of when the change occurred—well before any consensus had formed about the trend shift.

Our analysis also shows that the model can profitably be used in the current environment to monitor the risk of a return to a low-growth regime. Despite significant slowdowns in actual productivity growth after 2000, the model in real time continued to be optimistic—accurately, as it turns out—about the trend through the first half of 2005. Estimates in the third quarter of 2005 seemed to point to a shift toward lower trend productivity growth, but the most recent data from the model reaffirm that productivity growth is hewing to the high-growth path it has followed since 1996.[5]

Trend Shifts in the Postwar Period With the advantage of hindsight, economists have been able to chart the trend in productivity growth in the postwar period. Close to 3 percent from 1948 to 1973, the average growth rate of nonfarm output per hour fell to 1.5 percent between 1973 and 1995, then returned to approximately 3 percent from 1996 to the present. At the time these shifts in trend occurred, however, they were difficult to detect. The trend shift in 1973 went unrecognized for many years.[6] During the late 1990s, the notion that the trend growth rate had picked up found favor with those who believed in the advent of a “new economy,” but many economists continued to dispute the notion that the productivity gains of the post-1996 period represented a return to permanently higher growth.[7] Gordon (2000), for example, attributed about half of the acceleration to a “cyclical” effect, and argued that much of the rest was confined solely to the information technology (IT) sector. However, others (for example, Stiroh [2002]) found evidence that the productivity acceleration had spilled over into other sectors through investment in IT equipment and hence was unlikely to be merely a cyclical or isolated sectoral phenomenon.

Since 2000, productivity growth has slowed during some intervals—in the months leading up to the start of the 2001 recession, in 2004 after a big surge in the early part of the recovery, and as recently as the third quarter of 2006. Each of these episodes has given rise to concern that trend productivity growth might be falling back toward post-1973 rates. Our work provides one way to evaluate the validity of such concerns as they arise.

Estimating Trend Productivity Growth As we noted earlier, much of the uncertainty surrounding trend shifts has stemmed from the volatility of quarterly movements in productivity growth. Consequently, efforts to estimate trend productivity growth that focus on just this series are bound to run into difficulties. Our approach in this article, as in the more technical work (Kahn and Rich, forthcoming) on which it is based,[8] is to incorporate in our analysis variables that move together with productivity over the long run. By looking at multiple variables simultaneously, we obtain a more precise estimate of their common trend.

More specifically, using a theoretical model of economic growth, we argue that productivity (output per hour of work) should share a common trend with real labor compensation per hour and real consumption expenditure relative to hours of work. We identify the common trend as technical progress.[9] If a common trend indeed exists, then the three series will not drift too far apart over the long run. Formal statistical tests confirm this property in the data, which is illustrated by the parallel movements of the three series in Chart 2. Note that in the chart, the three series all slow down from the mid-1970s to the mid-1990s and then accelerate.

Chart 2
Common Trends in Real Output, Consumption, and Labor Compensation

Chart 2 - Common Trends in Real Output, Consumption, and Labor Compensation

Source: U.S. Department of Labor, Bureau of Labor Statistics.
Notes: The series are divided by hours of work in the nonfarm sector. Nominal output and compensation are deflated by the price index for nonfarm output. Nominal consumption is deflated by the price index for personal consumption.

The identification of real labor compensation and real consumption expenditure as series that can help uncover the underlying trend in productivity plays a critical role in our analysis. Nevertheless, we still need to specify a model that can be used to estimate the components of interest. Because we are especially concerned with the issue of changes in trend productivity growth, it is important to adopt a model that can account for this feature of the data. The choices are numerous, but we select a model that has gained considerable popularity in recent years, the so-called regime-switching model, and apply it to productivity growth.

The regime-switching model, introduced in Hamilton’s (1988) study of nominal interest rates, has been applied to a number of economic and financial time series.[10] The model is motivated by the observation that many variables go through periods in which their behavior changes dramatically. To capture such changes, the model allows for switches in the equation governing the movements of a variable. A different equation is associated with a different regime (or “state”), with changes in the behavior of a series reflecting a transition from one regime to another. The number of regimes is typically kept small to ensure that the model remains manageable for estimation purposes, although the design of the model is flexible enough to allow the shifts between regimes to occur often or rarely.

The regime-switching model is especially useful for our purposes because it will allow the common trend in productivity, real labor compensation, and real consumption expenditure to shift periodically between high-growth and low-growth states. Significantly, this methodology does not impose the dates of the shifts beforehand but instead estimates them. The estimates take the form of a time series of probabilities indicating the likelihood of the trend component being in one state or the other.

Although we seek to identify the trend component in productivity growth, our ability to draw meaningful conclusions about its behavior requires that we properly account for other sources of movements in the series. As we noted earlier, productivity growth fluctuates with the business cycle. To help identify the influence of the business cycle on the series, the model includes as an additional explanatory variable the growth rate of hours of work, which has an important cyclical component.[11] The model also separates out near-term, or “idiosyncratic,” movements in each of the series that are unrelated to the trend or cyclical components. The idiosyncratic movements in productivity growth account for much of the short-term volatility noted earlier. The box contains a brief description of the statistical model.

Components of the Statistical Model

We consider four time series over the period from first-quarter 1947 to third-quarter 2005: (1) output per hour of work, (2) real labor compensation per hour of work, (3) real consumption expenditure divided by hours of work, and (4) hours of work. The standard one- sector neoclassical growth model of economic theory implies that the first three variables should have a common trend related to technical progress, while any trend in the fourth variable, hours of work, is unrelated to technical progress. Consequently, we remove the trend of this last series so that only its cyclical movements enter into the model.

     Let Qit denote the i th individual time series (i =1,…,4). It is assumed that the movements in each series are governed by the following process:

where Xt denotes a permanent (or “trending”) component that is common to series (1)-(3) listed above, xt denotes a transitory (or cyclical) component that is common to all four series, and zit is an idiosyncratic error term. The permanent component grows over time, but its growth rate can shift periodically between high and low growth. The cyclical component does not grow over time, but instead displays oscillatory behavior. The idiosyncratic component reflects the remaining movements that are unique to each individual series. The parameter indicates the extent to which the series moves with the common permanent component. Similarly, the parameter indicates the extent to which the series is affected by the transitory component. The common trend implied by theory means that . We have verified this property in the data and illustrate it again in Chart 2. We also set to ensure that the (detrended) hours series does not affect the estimate of the common trend.

What We Know Now: Retrospective Assessments of Trend Productivity Growth Using our model, we obtain estimated probabilities of being in the high- or low-growth state over time. We first examine this issue from a retrospective viewpoint: Given everything we know today, what can we say about the likelihood of having been in the high-growth state at some past date? Because this retrospective inquiry is based on all available data, it provides the most informed assessment of the probabilities.

This assessment is plotted in Chart 3, through the second quarter of 2006. The vertical axis is the probability of being in the high-growth regime. With hindsight, we see that the economy was in the high-growth state (estimated to be 2.9 percent) until 1973, then underwent a twenty-four-year period of low growth (a 1.3 percent estimated trend), and finally returned to the high-growth regime in 1997. The slight dip at the very end partly reflects the unavoidable uncertainty at the end of the sample because confirmatory subsequent data are lacking.

Chart 3
Probability of Being in a High-Growth Regime: Retrospective Estimates

Chart 3 - Probability of Being in a High-Growth Regime: Retrospective Estimates

Source: Authors’ calculations.

What We Would Have Known Then: Real-Time Estimation and Forecasts

Having gauged the probabilities of the growth states in a retrospective manner, we now estimate these same probabilities using only the information that was available at the time. Thus, the question becomes: At each point in time, using only the data then available, what probability would one have assigned to being in the high-growth state? This inquiry sheds light on the model’s practical usefulness for policymakers, who must make decisions without the benefit of hindsight. The extent to which this inquiry and the retrospective inquiry give similar answers can tell us whether our model yields reliable results in real time. If the model were to indicate a trend shift toward higher growth in the 1990s only with several years of hindsight, then we would have little confidence in the accuracy of its current assessments.

With this in mind, we collected vintage data sets consisting of each of the time series as they would have appeared in August of each calendar year from 1993 onward. Given publication lags, there is typically a one-quarter differential between the data vintage and the last observation available. (Hence the August vintage will have data through the second quarter of that year.) This analysis enables us to see whether the model would have detected the trend shift toward higher productivity in the second half of the 1990s. A successful performance would also provide evidence of the model’s ability to identify trend changes in the current environment.

We find that the model would have picked up the change in trend within two years of when it is now known to have occurred. Chart 4 plots the estimated regime probabilities using data from vintages 1997, 1998, 1999, and 2000. By 1998 the estimates show a distinct rise in the high-growth regime probability, and by 1999 the probabilities are almost indistinguishable from those estimated retrospectively, using data through second-quarter 2006 (compare Chart 3).

Chart 4
Probability of Being in a High-Growth Regime: Real-Time Estimates

Chart 4 - Probability of Being in a High-Growth Regime: Real-Time Estimates

Source: Authors’ calculations.

We next consider the model’s forecasts of productivity growth. These forecasts take into account the transitory or cyclical components of productivity growth in addition to the trend. Chart 5 shows the ten-year-ahead forecasts of productivity growth, again based on data available as of August, for each of the years 1993 through 2006. It also includes a forecast based on the most recent observations of November 2006, which add data through the third quarter of 2006. To get a sense of the accuracy of the forecasts, we plot them against actual year-over-year productivity growth (shown by the bars). As we would expect given our earlier real-time estimates, through 1997 there is no real change in the long-run forecast. One achievement of this model—particularly in comparison with more conventional analyses[12]—is its ability to detect the permanence of the productivity acceleration. In line with our earlier findings, we see a noticeable permanent increase in productivity growth beginning with the forecast in 1998, followed by a large increase in 1999 to the level that has persisted ever since. A second accomplishment is the model’s recognition of the transitory nature of the productivity growth slowdowns in 2000-01, 2003, and 2004. As Chart 5 indicates, there is little noticeable response in the long-run forecast profile to the declines in productivity growth (shown by the bars) during these episodes.

Chart 5
Forecasts of Productivity Growth Ten Years Ahead

Click here for larger (readable) version

Sources: U.S. Department of Labor, Bureau of Labor Statistics; authors’ calculations.
Notes: Unless otherwise indicated, forecasts are as of August for each year. The bars show year-over-year actual productivity growth. Real-time data were unavailable in 1996 because a change in indexation methods resulted in a temporary discontinuity between historical and then-current data.

What does our regime-switching model suggest about the possibility of a trend shift in the current environment? Interestingly, for a period of roughly a year beginning in mid-2005, an apparent slowdown in real compensation growth caused the model to give warning signs of a return to lower trend productivity growth. Subsequent data releases from the Bureau of Labor Statistics, however, revised compensation growth upward, with the result that the model is again indicating strong productivity growth going forward. The shifts in our model estimates are evident in Chart 6, which depicts the probability of being in the low-growth regime (trend growth of 1.3 percent as opposed to 2.9 percent in the high-growth regime), tracked over recent data vintages, including quarterly updates since the third quarter of 2005. We see noticeable increases in this probability beginning in third-quarter 2005, reflecting the sluggish growth in real labor compensation. The latest estimate plotted in the chart, however, assigns a probability of less than 0.05 to productivity’s being in the low-growth state. Thus, the model once again places high probability on a trend path that is roughly 2.9 percent, close to the trend over the past ten years and to that in the interval between World War II and 1973.

Chart 6
Probability of Being in a Low-Growth Regime: Real-Time Estimates

Chart 6 - Probability of Being in a Low-Growth Regime: Real-Time Estimates

Source: Authors’ calculations.

Productivity and the Stock Market One variable conspicuous by its absence in our analysis is the stock market. One might think that stock market values would act as a natural leading indicator of a change in trend productivity, since asset prices in general can respond much faster than measures of real activity. We find, however, that the stock market is simply too volatile to be of practical use in detecting a change in trend productivity. In Chart 7, we show the regime assessments when the stock market is included as an additional variable in our model. If we accept the view that the 1973-96 period was one extended low-growth regime—as the retrospective estimates in Chart 3 suggest—then adding the stock market to the analysis largely results in more volatile assessments, even with hindsight.

Chart 7
Probability of Being in a High-Growth Regime,
with Stock Market Values as an Explanatory Variable

Chart 7 - Probability of Being in a High-Growth Regime, with Stock Market Values as an Explanatory Variable

Source: Authors’ calculations.

In fact, it may be more useful to consider the interplay between the stock market and trend productivity in the reverse direction. In other words, tracking the trend in productivity growth may illuminate trends in asset values. Chart 8 shows how long-term fluctuations in stock market values (as measured by the S&P 500, adjusted for inflation, and normalized by hours of work) appear to move in line with the level of trend productivity estimated from our original four-variable model. As we have seen, the model identifies changes in the growth regimes of productivity relatively quickly and therefore could be informative about future movements in stock prices. Such a result would accord with the findings of Ludvigson and Lettau (2001), who determine that consumption expenditure, because it responds only to “permanent” changes in wealth, predicts movements in stock prices.

Chart 8
Stock Market Values and Trend Productivity

Chart 8 - Stock Market Values and Trend Productivity

Sources: Standard and Poor’s; authors’ calculations.
Note: S&P 500 values are adjusted for inflation and hours of work.


Volatile short-term growth rates make the tracking of the trend in productivity difficult. But by looking at productivity in conjunction with labor compensation and consumption expenditure, one can discern the trend that is common to all three series. Our regime-switching model proves effective in tracking trend shifts in the postwar period—particularly the jump to trend productivity growth of nearly 3 percent that occurred sometime around 1997. While the immediate detection of a change in trend is virtually impossible, our model could have provided at least a preliminary sign within a year of when the change occurred and a conclusive signal within two years—well before any consensus had emerged. Although the idea of a “new economy” with robust productivity growth had gained many adherents well before 1999, there were also plenty of nay-sayers, and few of the optimists would have ventured to base their views on objective statistical analysis. Indeed, the view that higher productivity growth was likely to be sustained won acceptance only during the 2001 recession. When productivity growth remained relatively strong despite the downturn in the economy, it became clear that the acceleration was not just an artifact of the boom years of the 1990s.

Going forward, our model also provides a means of monitoring, in real time, the risk of a shift to lower trend productivity growth. Significantly, after a brief “false alarm” in the third quarter of 2005, the model’s most recent estimates continue to show strong trend productivity growth of nearly 3 percent.


1. Smith ([1776] 1937) emphasized specialization as the source of productivity growth: In The Wealth of Nations, he wrote, “It is the great multiplication of the productions of all the different arts, in consequence of the division of labour, which occasions . . . that universal opulence which extends itself to the lowest ranks of the people.” Later economists have focused on the development of new technologies as a spur to productivity gains.

2. Researchers typically focus on the nonfarm business sector because it comprises most of the private economy and eliminates the farm sector, where short-run productivity is dominated by transitory movements attributable to weather and other nontechnological factors.

3. Such readings have occurred 43 percent of the time since 1947. The standard deviation of quarterly nonfarm productivity growth (annualized) has been nearly 4 percentage points around the mean of 2.3 percent in this period.
4. For example, Federal Reserve economists Carol Corrado and Lawrence Slifman (1999) base their views of underlying productivity growth on the behavior of profits and prices.

5. Our approach to tracking productivity complements the work of Gordon (2000) and Stiroh (2002), who focus more on growth accounting methods for productivity. Their studies have the advantage of providing detailed information about the sectoral sources of productivity growth, but with significant time lags and only at an annual frequency.

6. See, for example, Sims (2001), who writes that during the 1970s, “unemployment rose and inflation rose because of real disturbances that lowered growth. . . . Since such ‘stagflation’ had not occurred before on such a scale, they [policymakers] faced a difficult inference problem, which it took them some years to unravel.” Sims and Zha (2006) make a similar point.

7. Optimistic views go back as early as 1997 (see, for example, Louis Uchitelle, “Measuring Productivity in the 90’s: Optimists vs. Skeptics,” New York Times, August 2, 1997). The optimism appears, however, to have been based on something other than the actual productivity data, about which there was much skepticism (see Corrado and Slifman [1999]). Of course, pessimists (for example, Roach [1998]) have also based their views on skepticism about the data.

8. Interested readers can find a much more detailed discussion of the theoretical and statistical models in the forthcoming paper.

9. This approach is closely related to the work of Ludvigson and Lettau (2001, 2004), who examine the relationship between wealth, labor income, and consumption expenditure. Their common trend is not specifically limited to technology; it also includes demographic and labor supply trends.

10. We estimate a version of the model developed by Kim and Murray (2002).

11. The hours of work variable is “detrended” because it is unrelated to the common trend of productivity, real labor compensation, and real consumption expenditure.

12. In Kahn and Rich (forthcoming), we compare the forecast performance of our model with the performance of an alternative model that has no regime switch. We find that our model provides more accurate forecasts, particularly during the late 1990s.


Corrado, Carol, and Lawrence Slifman. 1999. “Decomposition of Productivity and Unit Costs.” American Economic Review 89, no. 2 (May): 328-32. Papers and Proceedings of the 111th Annual Meeting of the American Economic Association.

Gordon, Robert. 2000. “Does the ‘New Economy’ Measure up to the Great Inventions of the Past?” Journal of Economic Perspectives 4, no. 14 (fall): 49-74.

Hamilton, James D. 1988. “Rational-Expectations Econometric Analysis of Changes in Regime:
An Investigation of the Term Structure of Interest Rates.” Journal of Economic Dynamics and Control 12, no. 2-3 (June-September): 385-423.

Kahn, James A., and Robert W. Rich. Forthcoming. “Tracking the New Economy: Using Growth Theory to Detect Changes in Trend Productivity.” Journal of Monetary Economics.

Kim, Chang-Jin, and Christian J. Murray. 2002. “Permanent and Transitory Components of Recessions.” Empirical Economics 27, no. 2 (March): 163-83.

Ludvigson, Sydney, and Martin Lettau. 2001. “Consumption, Aggregate Wealth, and Expected Stock Returns.” Journal of Finance 56, no. 3 (June): 815-49.

_____. 2004. “Understanding Trend and Cycle in Asset Values: Reevaluating the Wealth Effect on Consumption.” American Economic Review 94, no. 1 (March): 276-99.

Roach, Stephen S. 1998. “No Productivity Boom for Workers.” Issues in Science and Technology, summer.

Sims, Christopher A. 2001. Comment on “Evolving Post-World War II U.S. Inflation Dynamics,” by Timothy Cogley and Thomas J. Sargent. In Ben S. Bernanke and Kenneth S. Rogoff, eds., NBER Macroeconomics Annual 2001. Cambridge, Mass.: MIT Press.

Sims, Christopher A., and Tao Zha. 2006. “Were There Regime Switches in U.S. Monetary Policy?” American Economic Review 96, no. 1 (March): 54-81.

Smith, Adam. [1776] 1937. An Inquiry into the Nature and Causes of the Wealth of Nations. New York: Random House.

Stiroh, Kevin J. 2002. “Information Technology and the U.S. Productivity Revival: What Do the Industry Data Say?” American Economic Review 92, no. 5 (December): 1559-76.

    Posted by on Thursday, December 7, 2006 at 12:28 PM in Economics, Methodology, Monetary Policy, Technology | Permalink  TrackBack (4)  Comments (7)


    TrackBack URL for this entry:

    Listed below are links to weblogs that reference NY Fed: Tracking Productivity in Real-Time:

    » William J. Polley from William J. Polley

    New York Fed economists James Kahn and Robert Rich have a new model for estimating productivity growth in "real time." Real time in this case still implies a year or two lag. However since productivity growth is notoriously volatile, there will always ... [Read More]

    Tracked on Thursday, December 07, 2006 at 02:22 PM

    » Productivity on Strong Growth Trend from Businomics Blog

    Productivity--output per hour of labor worked--remains on a strong growth trend. That's hard to see in the data, which were weak last quarter. When the data first came out, I commented that separating short-term cycles from the underlying trend is [Read More]

    Tracked on Saturday, December 09, 2006 at 08:51 PM

    » Kahn and Rich: Tracking Productivity in Real-Time from Brad DeLong's Semi-Daily Journal

    Mark Thoma watches Jim Kahn and Bob Rich try to track changes in trend productivity growth in "real time": Economist's View: NY Fed: Tracking Productivity in Real-Time: The good news is that, according to these estimates, "the underlying trend r... [Read More]

    Tracked on Sunday, December 10, 2006 at 08:12 PM

    » Slowing Productivity Growth? from Economics Unbound

    As you folks know, I'm still worrying about a slowdown in productivity growth. Here's an interesting new chart from the New York Fed The bar represents the probability of being in a "low productivity growth" state, given observable data on... [Read More]

    Tracked on Monday, December 11, 2006 at 07:51 AM


    Feed You can follow this conversation by subscribing to the comment feed for this post.