Why Did Risk Models Fail?

I've been meaning to write about this, but haven't had the chance to think it through thoroughly. The topic is why risk models failed to inform regulators, and failed to protect investors during the recent crisis. Many of these strategies rely upon mining data for correlations that can be used to construct portfolios that (supposedly) have low variance returns.

I've wondered if the problems these models encountered can be explained using standard criticisms concerning the use of reduced form approaches to look for correlations in the data, correlations that can breakdown when there is any change in the underlying structure of the model, or a change in how policy reacts to macroeconomic variables (i.e. a standard Lucas-type critique of relying on reduced form estimates rather than deeper structural parameters and shocks). There's a difference between mining data for correlations and understanding why those correlations occur. The former can be accomplished through a variety of approaches, some borrowed from other disciplines. The goal isn't to understand how things work, the models generally make no attempt to explain why particular correlations are in the data, they simply look for the existence of correlations and attempt to exploit them. That may work fine as a money-making strategy in the short-run when the structure of the economy is fairly constant, but for understanding why things work the way they do, and to understand how things might suddenly change, theoretical models are necessary. There may also be a problem with the difference in responses to systemic and more localized shocks, particular since large economy-wide shocks are relatively rare and hence hard to evaluate using data-based methods.

But as I said, I haven't given this enough thought, so here are some views on why risk models failed just when they were most needed from people who have spent more time thinking about this than I have. Let's start with Alan Greenspan since the explanation that comes after this references this article:

How did we go so wrong?

The essential problem is that our models – both risk models and econometric models – as complex as they have become, are still too simple... A model, of necessity, is an abstraction from the full detail of the real world. In line with the time-honoured observation that diversification lowers risk, computers crunched reams of historical data in quest of negative correlations between prices of tradeable assets; correlations that could help insulate investment portfolios from the broad swings in an economy. When such asset prices, rather than offsetting each other’s movements, fell in unison on and following August 9 last year, huge losses across virtually all risk-asset classes ensued.

The most credible explanation of why risk management based on state-of-the-art statistical models can perform so poorly is that the underlying data used to estimate a model’s structure are drawn ... from ... periods of euphoria and periods of fear, that is, from regimes with importantly different dynamics.

The contraction phase of credit and business cycles, driven by fear, have historically been far shorter and far more abrupt than the expansion phase... Over the past half-century, the American economy was in contraction only one-seventh of the time. But it is the onset of that one-seventh for which risk management must be most prepared. Negative correlations among asset classes, so evident during an expansion, can collapse as all asset prices fall together, undermining the strategy of improving risk/reward trade-offs through diversification. ...

I do not say that the current systems of risk management or econometric forecasting are not in large measure soundly rooted in the real world. ... But these models do not fully capture what I believe has been, to date, only a peripheral addendum to business-cycle and financial modelling – the innate human responses that result in swings between euphoria and fear that repeat themselves generation after generation with little evidence of a learning curve. Asset-price bubbles build and burst today as they have since the early 18th century, when modern competitive markets evolved. To be sure, we tend to label such behavioural responses as non-rational. But forecasters’ concerns should be not whether human response is rational or irrational, only that it is observable and systematic.

This, to me, is the large missing “explanatory variable” in both risk-management and macroeconometric models. ... We will never be able to anticipate all discontinuities in financial markets. Discontinuities are, of necessity, a surprise. ...

Here is Avinash Persaud, writing on Willem Buiter's blog, with his take on the problem:

Why Bank Risk Models Failed and the Implications for what Policy Makers Have to Do Now, by Avinash D. Persaud: Sir Alan Greenspan, and others have questioned why risk models, which are at the centre of financial supervision, failed to avoid or mitigate today’s financial turmoil. There are two answers to this, one technical and the other philosophical. Neither is complex, but many regulators and central bankers chose to ignore them both.

The technical explanation is that market-sensitive risk models used by thousands of market participants work on the assumption that each user is the only person using them. This was not a bad approximation in 1952, when the intellectual underpinnings of these models were being developed ... by Harry Markovitz and George Dantzig. ...

In today’s flat world, market participants from Argentina to New Zealand have the same data on the risk, returns and correlation of financial instruments and use standard optimization models, which throw up the same portfolios to be favoured and those not to be. Market participants don’t stare helplessly at these results. They move into the favoured markets and out of the unfavoured. Enormous cross-border capital flows are unleashed. But under the weight of the herd, favoured instruments cannot remain undervalued, uncorrelated and low risk. ...

When a market participant’s risk model detects a rise in risk in his portfolio, perhaps because of some random rise in volatility, and he tries to reduce his exposure, many others are trying to do the same thing at the same time with the same assets. A vicious cycle ensues of vertical price falls prompting further selling. Liquidity vanishes down a black hole. ...

Policy makers cannot claim to be surprised by all of this. The observation that market-sensitive risk models ... were going to send the herd off the cliff edge was made soon after the last round of crises*. Many policy officials in charge today, responded then that these warnings were too extreme to be considered realistic.

This brings us to the philosophical problem of the reliance of supervisors on bank risk models. The reason we regulate markets over and above normal corporate law is that from time to time markets fail and these failings have devastating consequences. If the purpose of regulation is to avoid market failures, we cannot use ... risk-models that rely on market prices, or any other instrument derived from market prices such as mark-to-market accounting. Market prices cannot save us from market failures. Yet, this is the thrust of modern financial regulation, which calls for more transparency on prices, more price-sensitive risk models and more price-sensitive prudential controls. These tools are like seat belts that stop working whenever you press hard on the accelerator.

In terms of solutions, there is only space to observe that if we rely on market prices in our risk models and in value accounting, we must do so on the understanding that in rowdy times central banks will have to become buyers of last resort of distressed assets to avoid systemic collapse. This is the approach we have stumbled upon. Central bankers now consider mortgage-backed securities as collateral for their loans to banks. But the asymmetry of being a buyer of last resort without also being a seller of last resort during the unsustainable boom will only condemn us to cycles of instability.

The alternative is to try and avoid booms and crashes through regulatory and fiscal mechanisms designed to work against the incentives ... for traders and investors to double up or more into something that the markets currently believe is a sure bet. This sounds fraught and policy makers are not as ambitious as they once were. ...

Regulatory ambition should be set now, while the fear of the current crisis is fresh and not when the crisis is over and the seat belts are working again.

Comments

[odograph]

As a reader of "Against the Gods: The Remarkable Story of Risk," by Peter L Bernstein, as well as the Taleb books, my reaction is to expect very little from such models.

(Donald Rumsfeld caught a lot of deserved flak for a lot of things, but "unknown unknowns" was a serious and correct answer to risk problems.)

[STS]

Quantitative answer: correlated trading behavior is barely studied and poorly understood.

Qualitative answer: conflicts of interest leading to selective blindness.

[bakho]

Garbage in. Garbage out. If the data is unreliable or if the people providing it are not truthful (lying?) then they may sell something as AAA when it is permeated with bits and pieces of the "Big Sh!tP!le". If you plug "AAA" into your model instead of "BS" it won't give the right result. This is not necessarily a problem with the model.

[jm]

The problem with economics in the US seems to be that American economists attempt to model the world mathematically, mostly with linear partial differential equations, assuming reversibility. But you can't model the real world that way, because critically important elements of it are completely non-linear, and many processes are irreversible.

There seems to be no concept of "phase change" in economics, and little awareness among economists that systems regulated by negative feedback are as likely to oscillate violently as they are to come to a stable equilibrium. The Econ 101 description of how market prices stabilize at the intersection of the supply and demand curves is fundamentally a description of a negative-feedback regulated system, but never mentions that the price will stabilize at the crossing point only if the gain around the loop is less than unity at the frequency where lags and deadtimes around the loop equate to 180 degrees of phase shift (which turns the negative feedback into positive, regenerative feedback).

Economists attempt to do things that amount effectively to engineering, but completely ignore the accumulated knowledge and wisdom of the engineering professions.

[James]

Its difficult to add anything substantial to the insightful comments so far posted, in particular the comments of STS and jm.

All one can do is recommend The (Mis)Behavior of Markets: A Fractal View of Risk, Ruin, and Reward by Benoit Mandelbrot for additional insights.

[jm]

There was a time hundreds of years ago when the mathematical models and engineering knowledge of bridge designers were as feeble as those of present-day economists. Their bridges frequently collapsed with catastrophic injury to the public -- like present-day economist-guided public policies.

[Mark Thoma]

Actually, that's all well known (e.g. see Samuelson's correspondence principle).

The problem is that the things in the feedback rule are intelligent and can respond to stimuli. When you feedback to tune a TV (monitor the output then design an optimal filter for the inputs), the TV doesn't anticipate the treatment and takes no action to avoid or encourage it. People do - this comes in through expectations and this is exactly what the Lucas critique is all about. If I change tax rates, people take action to avoid the taxes, if I do something to a machine, it just sits there and takes it. We rarely worry that a TV will exhibit moral hazard if we help it. In any case, what you are talking about was known long ago - it's much more complicated to do stability analysis in the presence of expectations.

Yes, we do use techniques from engineering - in grad school, for example, I took a grad level stochastic processes course in the electrical engineering department and learned all the techniques engineers use to analyze this stuff (and got the second highest grade - I found it easier than time-series econometrics which is much of the same material). As I noted, control models for the economy are much like control models for physical processes (monitor the outputs, e.g. inflation and output, then feedback to the inputs, monetary and fiscal policy, to achieve optimal control), but then we realized what I noted above, that these models would not work if we simply brought them over and used as is, they had to be amended to account for expectations (hence the development of techniques to solve expectational difference equations - much harder than ordinary difference or differential equations).

Anyway, yeah, we know about all that...

[I should add that we also have lots of "phase change" models, chaos and catastrophe models as well, regime switching models, models with switching stochastic inputs, etc., etc., you name it... again, nothing new to economists there.

As another example, New Keynesian models have a point (a value for the parameter on inflation in the monetary policy feedback rule) where if the response of monetary authorities is greater than that value, the model is stable, but below that value, you get indeterminacy and values can shoot off in one direction or another. This is just the Taylor principle, and there are all sorts of examples like this]

[save_the_rustbelt]

Math models apparently do not have the ability to tag some data as fraudulent, or at least beyond the pale of reality.

Financial auditing fails for the same reason, lack of healthy skepticism. Lack of context.

Or as grandma used to say...

"you can't see the forest when you are staring at the trees"

[Mark Thoma]

You won't like this because it's a linear differential equation set-up, but just to illustrate how it fits together in the most basic case, here's the simplest example I can think of off the top of my head.

Let demand be Qd = a+bP
Let supply be Qs = c+dP

Then P* (the price where the lines cross is)

P* = (c-a)/(b-d)

Now, let prices out of equilibrium be governed by dp/dt = k(Qd - Qs)

That is, when demand exceed supply, prices rise, when supply exeeds demand, prices fall. This assumption (which can be changed if you don't like it) governs the behavior of prices out of equilibrium. The parameter k is the speed of adjustment. [In models where there is more than one market you use a matrix version of this, one equation for each endogenous variable specifying how it acts out of equilibrium - below, simply replace the condition "negative" needed for stability with "negative definite" and everything pretty much goes through the same.]

The question is, when prices are away from equilibrium, will they go back there (your stability question)? [This would be so much easier to explain if I could draw a graph]

Sub in for Qd and Qs:

dp/dt = k[(a+bP)-(c+dP)]

dp/dt = k(a-c) + k(b-d)P

Solve the differential equation:

p = p* + exp{k(b-d)t}

[doing this fast and in my head, so hope I didn't make a silly error...]

The question is, if we are away from p*, the equilibrium value (i.e. p does not equal p*), will we get back there given enough time (if we let t go to infinity).

That is, is the limit of p as t goes to infinity equal to p*?

lim p = p* if (b-d)<0.

So, ordinary supply and demand curves (b negative, d positive), will be stable since the coefficient attached to exp is negative. As you can see, though, it's easy to go on from here and develop all sorts of dynamic paths to (or away from ) the equilibrium value by amending the model in various ways.

And as you know, getting asymmetries and phase shifts is harder, but not that many steps away.

We can also handle non-linear systems, both locally and globally, though mostly we use linear approximations in the neighborhood of the steady state to analyze dynamic behavior. The dynamics in these models can be complex.

[Update: I meant to note, but forgot, that I assumed the initial price shock was unitary to simplify. But the full solution, p = p* + [p(0)-p*]exp{k(b-d)t} is probably more intuitive since the p(0)-p* term is the size of the deviation from equilibrium, and the exp term is how fast the deviation decays or grows.]

[odograph]

We have what is probably a biological drive to predict the future. If we knew just a little more, we could make so much better choices. Our survival and prosperity would be improved. So we constantly press hard for better answers.

50K years ago the the choices were simple and fuzzy: Should I hunt in the north valley or the south?

In those days if answers seemed similar, they probably were. A partial differential equation was probably not going to tell you that much more about the presence of deer (or lions) than you already knew.

What I see really, in our modern quest for prediction, is that our ancient need for an answer tempts us to not just mine the data with new tools, but to over-mine it with wild abandon.

Actually I think we all know that at some level, that no one knows that future. We know that much of what is called risk is really uncertainty ... but so many of us keep coming back for more.

It is hard for humans to admit uncertainty (esp. when they are paid not to).

[odograph]

BTW, should the Fed (etc.) prepare for risk? Or uncertainty?

[a]

Two points: (1) Historical data is often used as the basis for the scenarios of the models. This data usually comes from the recent past, because data that is too old is considered irrelevant. So events that happen very rarely do not get captured. (2) End of the world scenarios are not considered, and until now there has been little discrimination in these scenarios.

I'll try to expand on (2) in a little bit...

[a]

OK let me continue. I think it is probably incorrect to say that models "failed". The models simply did not consider the scenario that we are living through.

There are *always* scenarios which will make a bank go bust. Consider a conservatively managed bank in the scenario where housing prices overnight go down 50% and people starting mailing in their keys. It would go bust. There's no way around it, so long as a bank is the banking business.

Or consider the case of the Nasdaq bubble. At the height of the bubble if the market had gone down 80% in one day, then at least one IB would have gone bust. At least one, maybe more.

These are extreme scenarios, and they are not considered in the analyses that IBs run. At the height of the Nasdaq bubble suppose someone thought a scenario of -50% to be possible. Now, after the fact, we can say that it didn't happen. Was the person wrong? Or was it a possibility that needed to be considered?

So banks can't and won't consider all possibilites. Some extreme scenarios will sink the bank; that's not news. The hard part is to decide, beforehand, what is *plausible*.

For the proverbial "black swan" (or, for want of a better description, End of the World) events, the only thing that proves the plausibility of the event is if it happens. That makes it very difficult to know, beforehand, exactly what degree of scenario needs to be considered. At the height of the Nasdaq bubble what was the proper extreme scenario, for which a bank needed to be prepared? -25%? -40%? -50%? - 75%? -90%?

Now I don't have any pity for IBs, but the current scenario is unprecedented. It's a genuine "black swan" event. So post hoc it's very easy to say, "Hey guys, you should have considered this." But before the event, it's very hard - no, it's impossible - to calibrate.

So I'd say it's not a question of tweaking the models. Post hoc there's obviously ways to do that, but I imagine the IBs models are actually robust enough, and what was lacking was considering the present scenario as plausible, as an extreme event which needed to be considered and to be reserved against (and here the regulators, with their allegiance to VAR, were not helpful).

[Thomas]

Following on from JM comment 1 about "...critically important elements of it are completely non-linear, and many processes are irreversible..."

An oft-used example given in Chemistry is that you can't unfry an egg. Perhaps we should listen to Chemists over Economists (though I'm sure the Physicists would have a problem with that!)

[hari]

AG and Avinash are professional analytikers of financial markets...but one thing they don't focus upon is the advent of 24/7 hi fi market without national boundaries. In other words, globalization of hi fi markets has also introduced uncertainty in valuation and execution of portfolios. I see that more in FX market in which daily volume is in trillions!
Multinationals are evidently the one's who make the market daily when they execute their executive board orders for own accounts - leveraging currencies against one another.

How can a systemic modelling of the global hi fi market become realistic and predictable - I don't know. What I do know is that it's *unrealistic* to expect theoretical modelling to provide anything more than fundamental policy guideline based on data that are verifiable/credible. A lot of data today is not *current* and subject to *revisions* and whatnots by authorities.

AG is reconfirming his intuitive intelligence (from WallStreet) that it's unrealistic to consdier that one can predict market failure....So why not find ways and means to SLOW the system - even a bit - with enough regulatory uptick?

[Correlations]

"...they simply look for the existence of correlations and attempt to exploit them..."

The more successful models tend to use data over as many business cycles as possible, reject correlations that don't have any fundamental reason (super-bowl predictor), and limit leverage (to allow recovery when some future correlations inevitably vary from the historical pattern). Highly leveraged strategies can work in the short run, but tend to self destruct over long periods of time. There is a reason most hedge funds are not good long term performers. There are also reasons that a few hedge funds are.

Systemic risk occurs when subsidized low short rates place vast sums at the disposal of people who make highly leveraged bets that can only work if the historical pattern holds. When variations inevitably occur, cascades of credit failure can occur. Perhaps subsidized short rates should only be provided for direct investment (starting a new business, or expanding an established one), rather than for purposes of making highly leveraged bets based on historical correlations. Artificially low short rates can magnify the systemic risk by encouraging too many highly leveraged bets.

[jamzo]

are they saying: we relied too much on models in our oversight

or

are they saying: don't blame me, it was caused by the models that those guys are giving us, if they gave us better models these problems would not have occurred

either way, we cannot help but to be impressed by the
stnnd-up capitalists taking responsibility for their actions

the wonders of "free" markets!

[dd]

Isn't it possible that the models did not fail but incorporated Fed assurances to backstop systemic failure? The speed of financial innovation provided a basis to argue that increased risk-taking was now appropriate but the new level of "excessive" risk taking would be unknown until an actual failure. There was ever incentive to push to the limit because:
"The management of systemic risk is properly the job of the central banks. Individual banks should not be required to hold capital against the possibility of overall financial breakdown. Indeed, central banks, by their existence, appropriately offer a form of catastrophe insurance to banks against such events."
Greenspan 2/26/1998
http://www.federalreserve.gov/boarddocs/Speeches/1998/19980226.htm

[Mark Thoma]

On irreversibilities, we have had those for a long time, so nothing new there either. The earliest model I can remember that has this are putty-clay models. That stands for two types of capital. Clay, once it is in place, cannot be remolded or reused elsewhere. Putty can re reused and remolded. So once capital in the clay class in installed, it can't be reversed and used elsewhere. Those models are 40 or 50 years old, but there have been lots of other examples since then of various sorts.

[GT]

The issue isn't correlation of the exposures, it's that the underlying price for the exposure didn't match the expected payouts.

This situation is similar to one that arose in the Workers Compensation market a few years back. One company started providing cheap reinsurance and quickly cornered the market. They immediately repackaged the risk and sold it to other reinsurers.

Once the primary insurers realized that the reinsurer would accept any and all business, they cut the rates they charged their policyholders in order to attract more business. At some point, the insurance industry discovered that the premiums being collected didn't cover the indemnity payments, and the whole set-up went bust.

[kharris]

Think back to the early days of the EMBI. It started life as a portfolio strategy. Find a bunch of high-yield emerging market debt. Run correlations to determine which debt instruments produced the portfolio with the lowest expected risk. (Then sell the portfolio strategy to clients, but that has to do with motive, and may not be a proximate source of error.)

The notion was that things which had a lot in common - default risk, economic risk, interest rate risk, currency risk, portfolio risk (many were held together in emerging market debt funds), the very fact that they were all sovereign debt instruments from high yielders, so that the perception of linkage could erupt suddenly - could be put together to reduce risk because the math said so. There was no lack of complexity. There was the perception that reason could be put aside because the math said so. Any of us, not knowing what the correlation tables looked like, would have though this was a bad idea. Why, after looking at the correlation tables, is it suddenly a good idea?

Both Persaud and Greenspan, in different ways, argue that our assumptions about the underlying conditions are wrong. Greenspan says we build models based on data from normal (good) times, and that they fail in bad times. Persaud says that we cannot assume that our behavior in response to observations about correlation are irrelevant - techniques which may work when the data and techniques are not generally know fail when they are used by everybody. Heisenberg would understand perfectly.

Both gentlemen make the case that the flaws were known, but that portfolio managers and policy makers didn't make use of that knowledge. Why do we think they will next time?

One tentative conclusion would be that we should not expect these models to work when we need them most. They can improve risk-adjusted returns in normal times, but may even add to the problem in abnormal times. Our very reliance on them makes them dangerous. That seems to be what Persaud is telling us. It certainly seems to be the message from the EMBI episode.

The history of investment is a history of booms and busts. We regulate, and still we have booms and busts. We innovate, and still we have booms and busts. We go through a Great Moderation, we "delink" through (despite?) globalization, and still we have booms and busts. The question asked is "Why Did Risk Models Fail?" My question is, why did you think they wouldn't?

[Barry]

Two comments - first, echoing DD, perhaps Greenspan isn't so good a source of advice - he rode things on the way up, like so many other people. Probably 100x smarter than me on it, but in the end people who took his advice would have gotten hosed.

Which leads to the second idea (echoing Rusty and others): fraud. As long as things go well, making bets that they'd continue to go well would make the decision-makers much money. Leveraging them highly would make the decision-makers even more money. Others frequently had little access to information, or were bought off (e.g., every single frickin' rating agency, auditing agency, and those others whose job was to bring quality of information to the market).

[STS]

Avinash Persaud is on a better track. (Via Yves Smith via Willem Ruiter's blot at FT)

He seems to have been influence by a more game theoretic approach (Stephen Morris and Hyun Song Shin) to the process by which trading decision are arrived at.

More generally, the idea of an ODE model seems inherently "off" to me. The modeling needs to start from a more discrete model of decision making by traders which then is aggregated by means other than rote application of the Central Limit Theorem. Particle systems theory or mean field theory would be more promising technical approaches.

[Tom L]

What makes everyone assume these risk models are firmly based on economics? Much has been made about the non-statistical neural network type models, which are entirely empirical in nature. Such models can stop working at any time, as inputs move outside of the space used for model estimation.

[johnchx]

I reject the premise: the risk models did not fail. Pretending that they did is the last refuge of scoundrels -- in particular, senior executives at investment banks who either deliberately ignored warning signals or ensured that serious risk-management systems weren't implemented in the first place.

There is no mystery here. If you take the battery out of your household smoke detector ("it makes such an awful noise!") you don't get to ask "how did smoke detector technology fail?"

[VG]

Yes, exactly. Risk models are not used to avoid risk. They are used to avoid Risk Control.

Risk Control exists to prevent traders from gambling with the investor's money. But traders HAVE to gamble the investors' money---- nobody gets paid million-dollar bonuses for stuffing money into T-bills.

The result is hedging. Hedging claims to reduce risk and this lets clever traders borrow more money to invest. More money means more profits inherently. In fact, since it allows you to "make markets", it even generates a higher percentage of profit--- for a while.

But, except for certain special cases, hedging does not actually reduce risk. It simply moves all risk to some random point in the future. Preferably some point after you've gotten your bonus.

That "random point in the future" was six months ago.