Foolish Sophistication: Blame The Way Risk Models Were Used

I've been meaning to write about this topic, why risk models break down when we need them the most, but this saves me the trouble since it covers most of the points I wanted to make:

Blame the models, by Jon Danielsson, Vox EU: A well-known American economist, drafted during World War II to work in the US Army meteorological service in England, got a phone call from a general in May 1944 asking for the weather forecast for Normandy in early June. The economist replied that it was impossible to forecast weather that far into the future. The general wholeheartedly agreed but nevertheless needed the number now for planning purposes.

Similar logic lies at the heart of the current crisis

Statistical modelling increasingly drives decision-making in the financial system while at the same time significant questions remain about model reliability and whether market participants trust these models. If we ask practitioners, regulators, or academics what they think of the quality of the statistical models underpinning pricing and risk analysis, their response is frequently negative. At the same time, many of these same individuals have no qualms about an ever-increasing use of models, not only for internal risk control but especially for the assessment of systemic risk and therefore the regulation of financial institutions.[1] To have numbers seems to be more important than whether the numbers are reliable. This is a paradox. How can we simultaneously mistrust models and advocate their use?

What’s in a rating? Understanding this paradox helps in understanding both how the crisis came about and the frequently inappropriate responses to the crisis. At the heart of the crisis is the quality of ratings on structured investment vehicles (SIVs). These ratings are generated by highly sophisticated statistical models.

Subprime mortgages have generated most headlines. That is of course simplistic. A single asset class worth only $400 billion should not be able to cause such turmoil. And indeed, the problem lies elsewhere, with how financial institutions packaged subprime loans into SIVs and conduits and the low quality of their ratings.

The main problem with the ratings of SIVs was the incorrect risk assessment provided by rating agencies, who underestimated the default correlation in mortgages by assuming that mortgage defaults are fairly independent events. Of course, at the height of the business cycle that may be true, but even a cursory glance at history reveals that mortgage defaults become highly correlated in downturns. Unfortunately, the data samples used to rate SIVs often were not long enough to include a recession.

Ultimately this implies that the quality of SIV ratings left something to be desired. However, the rating agencies have an 80-year history of evaluating corporate obligations, which does give us a benchmark to assess the ratings quality. Unfortunately, the quality of SIV ratings differs from the quality of ratings of regular corporations. A AAA for a SIV is not the same as a AAA for Microsoft.

And the market was not fooled. After all, why would a AAA-rated SIV earn 200 basis points above a AAA-rated corporate bond? One cannot escape the feeling that many players understood what was going on but happily went along. The pension fund manager buying such SIVs may have been incompetent, but he or she was more likely simply bypassing restrictions on buying high-risk assets.

Foolish sophistication Underpinning this whole process is a view that sophistication implies quality: a really complicated statistical model must be right. That might be true if the laws of physics were akin to the statistical laws of finance. However finance is not physics, it is more complex, see e.g. Danielsson (2002).

In physics the phenomena being measured does not generally change with measurement. In the finance that is not true. Financial modelling changes the statistical laws governing the financial system in real-time. The reason is that market participants react to measurements and therefore change the underlying statistical processes. The modellers are always playing catch-up with each other. This becomes especially pronounced when the financial system gets into a crisis.

This is a phenomena we call endogenous risk, which emphasises the importance of interactions between institutions in determining market outcomes. Day-to-day, when everything is calm, we can ignore endogenous risk. In crisis, we cannot. And that is when the models fail.

This does not mean that models are without merits. On the contrary, they have a valuable use in the internal risk management processes of financial institutions, where the focus is on relatively frequent small events. The reliability of models designed for such purposes is readily assessed by a technique called backtesting, which is fundamental to the risk management process and is a key component in the Basel Accords.

Most models used to assess the probability of small frequent events can also be used to forecast the probability of large infrequent events. However, such extrapolation is inappropriate. Not only are the models calibrated and tested with particular events in mind, but it is impossible to tailor model quality to large infrequent events nor to assess the quality of such forecasts.

Taken to the extreme, I have seen banks required to calculate the risk of annual losses once every thousand years, the so-called 99.9% annual losses. However, the fact that we can get such numbers does not mean the numbers mean anything. The problem is that we cannot backtest at such extreme frequencies. Similar arguments apply to many other calculations such as expected shortfall or tail value-at-risk. Fundamental to the scientific process is verification, in our case backtesting. Neither the 99.9% models, nor most tail value-at-risk models can be backtested and therefore cannot be considered scientific.

Demanding numbers We do however see increasing demands from supervisors for exactly the calculation of such numbers as a response to the crisis. Of course the underlying motivation is the worthwhile goal of trying to quantify financial stability and systemic risk. However, exploiting the banks’ internal models for this purpose is not the right way to do it. The internal models were not designed with this in mind and to do this calculation is a drain on the banks’ risk management resources. It is the lazy way out. If we don't understand how the system works, generating numbers may give us comfort. But the numbers do not imply understanding.

Indeed, the current crisis took everybody by surprise in spite of all the sophisticated models, all the stress testing, and all the numbers. I think the primary lesson from the crisis is that the financial institutions that had a good handle on liquidity risk management came out best. It was management and internal processes that mattered – not model quality. Indeed, the problem created by the conduits cannot be solved by models, but the problem could have been prevented by better management and especially better regulations.

With these facts increasingly understood, it is incomprehensible to me why supervisors are increasingly advocating the use of models in assessing the risk of individual institutions and financial stability. If model-driven mispricing enabled the crisis to happen, what makes us believe that the future models will be any better?

Therefore one of the most important lessons from the crisis has been the exposure of the unreliability of models and the importance of management. The view frequently expressed by supervisors that the solution to a problem like the subprime crisis is Basel II is not really true. The reason is that Basel II is based on modelling. What is missing is for the supervisors and the central banks to understand the products being traded in the markets and have an idea of the magnitude, potential for systemic risk, and interactions between institutions and endogenous risk, coupled with a willingness to act when necessary. In this crisis the key problem lies with bank supervision and central banking, as well as the banks themselves.

References

Danielsson, Jon (2002), “The Emperor has no Clothes: Limits to Risk Modelling”, Journal of Banking and Finance, 26(7),1273—1296.

Footnote

1 For example, see Naseem Taleb (2007). "Fooled by randomness: the hidden role of chance in life and the markets" Penguin Books.

Comments

[lark]

Yves Smith had a post that seems related to me, namely the fact that liquidity from an information theory point of view is inherently destabilizing. This relates to my field (software) so I find it interesting.

Perhaps a lesson to be learned here is that liquidity acts as an efficient conductor of risk. It doesn't make risk go away, but moves it more quickly from one investment sector to another.

From a complex systems theory standpoint, this is exactly what you would do if you wanted to take a stable system and destabilize it.

One of the things that helps to enable non-linear behavior in a complex system is promiscuity of information (i.e., feedback loops but in a more generalized sense) across a wide scope of the system.

One way you can attempt to stabilize a complex system through suppressing its non-linear behavior is to divide it up into little boxes and use them to compartmentalize information so signals cannot easily propagate quickly across the entire system.

This principle has been recognized in the design of software systems for several decades now, and is also a design principle recognizable in many other systems both natural and artificial (c.f. biology, architecture) which are very robust with regard to exogenous shocks. Stable systems tend to be built from structural heirarchies which do not share much information across structural boundaries, either laterally or vertically. That is why you don't die from a heart attack when you stub your toe, your house doesn't collapse when you break a window, and if your computer crashes it doesn't take down the entire internet with it.

Glass-Steagall is a good example of this idea put into practice. If you use regulatory firewalls to define distinct investment sectors and impose significant transaction costs at their boundary that will help to reduce the speed and amplitude of signals which will propagate from one sector to another, so a collapse in one of them will be less likely casue severe problems in the others.

It worries me that we’ve torn down most of these barriers in the last several decades in the name of arbitrage, forgetting that the price we paid for them in inefficiency was a form of insurance against the risk of systemic collapse. This is exactly what I would do if I wanted to take a more or less stable, semi-complex system and drive it in the direction of greater non-linearity.

I think this was to some degree inevitable - it is a symptom of the decay and loss of trans-generational memories from our last great systemic shock in the 1930s. I suspect that something like this is bound to happen every 3-4 generations as we unlearn the lessons our grandparents and great-grandparents learned to their cost.

link http://www.nakedcapitalism.com/2008/05/hoisted-from-comments-greater-liquidity.html

[kthomas]

Thanks for the post,lark.

Ignorance is not bliss, after all.

[robertdfeinman]

Sorry, I think Danielsson gives the game away with this:

And the market was not fooled. After all, why would a AAA-rated SIV earn 200 basis points above a AAA-rated corporate bond? One cannot escape the feeling that many players understood what was going on but happily went along.

This wasn't a failure of "models" this was a fraud. Either the ratings agencies were abetting a fraud by granting an AAA rating which wasn't really an AAA rating in the conventional sense or they were committing a fraud by rating investments as sound when they had a conflict of interest.

Those who then sold the "AAA" rated securities to the unsuspecting or to those not acting as prudent investors were also in on the scam. The only way that the models can be blamed for anything is by assuming that those involved were complacent enough to think that the house of cards wouldn't collapse during their watch.

Even Ponzi bought into his own hype for awhile.

The regulatory agencies don't get off without blame either, they were unwilling to regulate these new markets or even oversee them properly. Ideology or greed take your choice.

We can use Katrina as an example. The risks were known (the local newspaper ran a detail series of articles way in advance of the storm). There was nothing wrong with the models. The levees failed just as the experts cited in the articles predicted. The failure was that of the authorities who gambled that the storm wouldn't happen during their administration and chose not to take the advice of the experts and reinforce the containment system.

I don't think models get blamed, just the usual culprits of greed and fraud. The economists can go back to being happy with their work.

[CathyG]

Actually, in quantum physics the measurement of a thing does change characteristics of the thing being measured. That's what give rise to the Heisenberg uncertainty principle.

[Eric Dewey, Portland OR]

Great post, Mark (and lark). From my experience, these thoughts are directly on the mark, and explain most of why we saw the recent banking behavior in mortgage markets.

Danielsson's point goes well beyond SIVs, however. He seems to have a great deal of faith in the internal risk modeling taking place in individual banks. Having worked in risk management at a couple of large banks, I'm not as confident in those models, primarily because of the "garbage-in, garbage-out" syndrome. All of the models require inputs, and those inputs are very often more subjective than objective, and thus quite subject to interference from management at all levels, which is usually unwilling to allow inputs to reflect negatively on their own team's performance.

Lark's review of Yves Smith's post also brings up another fascinating possibility: does the uncertainty principle in physics apply to complex information systems like financial markets? The famous experiments demonstrating that principle suggests that light quanta "know" when a measurement is being taken.

Is it much of stretch to be able to demonstrate that sentient actors in the financial markets are "quanta" that "know' when a measurement is being taken, and change their behavior accordingly? Isn't that part of the underlying reason why observed behaviors such as the "January effect" in stock can't be consistently exploited over time?

Philosophically, the fundamental issue seems to be that management seems to have generally lost confidence in it's own ability to judge risk in financial markets, and instead is religiously relying on objective, quantifiable data elements.

While objective data is undoubtedly a critical input in the decisionmaking process, the point seems to be that some level of human judgment is required, if only in order to allow for the fact that no model can contain all possible relevant factors.

[Copycat]

A point I've seen make (but cannot quickly find the reference) is that the kinds of models under discussion implicitly assume that the rest of the world isn't using them. If everybody uses the output from the same models to decide to buy and sell, the prices that are the inputs become meaningless.

[Bernard Yomtov]

lark,

Interesting comment. Thanks.

[Realist]

"Is it much of stretch to be able to demonstrate that sentient actors in the financial markets are "quanta" that "know' when a measurement is being taken, and change their behavior accordingly? Isn't that part of the underlying reason why observed behaviors such as the "January effect" in stock can't be consistently exploited over time?"

Yes, it is much of a strech. The impact of quantum phenomena on everyday life is zero. Don't even think about it.

[Anon]

rdf, How's this?

The models are a tool help a few criminals perpetrate a fraud on a credulous public. Of course, we'll never be able to prove that the criminals didn't simply believe in the accuracy of the models themselves.

On a bad day, I have to admit that I feel that economic models are used this way all the time.

[Mark Thoma]

Very quickly - the idea behind the Lucas critique is that our models did not account for the behavioral response of agents to policies. When you take actions that affect people, say it imposes costs, they react in a way that minimizes, as much as possible, the costs you are trying to impose upon them (they use the rules to shift income among categories when the tax structure is changed).

It's easy to extend this to ordinary observation. Measurement tells the agent what state of the world they are in, the trajectory, their position, etc., and the agent can them take actions to change their state, position, or trajectory so as to improve their personal outcome. Here, then, the mere act of measuring something and providing information motivates a response that changes the state of the world.

Sorry to be in a rush...

[Bill Jefferys]

The footnote got copied incorrectly from the original blog. It's "Naseem Taleb," not "Naseem Tale."

[Not sure how that happened - but it's fixed. Mark]

[Anon]

Based on the Lucas critique as described above, it seems to me that Lucas should have promoted a movement away from mathematical modelling and towards qualitative economics. (Since he couldn't possibly believe that any model would ever get enough of the behavioral responses correct, could he?)

[Jack]

"Indeed, the current crisis took everybody by surprise..."
Really? I remember reading quite a bit from a few economists (Paul Krugman, Dean Baker) that predicted the current crisis pretty accurately. Isn't this just hogwash?
Of course, in the current business culture the goal is to find some way of placing the blame on someone else when something goes wrong while retaining the opportunity to take credit when something goes right (whether or not credit is due). In the current academic culture the goal is to make whatever you're doing looks as much like hard science (physics) as possible, which means creating models and mathing them up as much as possible so only those with specialized expertise can "interpret" them, which means something like "know what they're looking at." This creates potential for a nice symbiotic relationship--business managers, many of whom are cretans, can point to models that no one understands to vindicate their terrible decisions, while those who created the models can easily beg off taking responsibility by losing people in jargon. You might call it the "Honorlessness/Blind Scientism" model. Now all I need is a mathematicain. . .

[jult52]

lark's first comment and quotation is lengthy but well worth reading.

[Masonik]

"Indeed, the current crisis took everybody by surprise in spite of all the sophisticated models, all the stress testing, and all the numbers."

These 2 articles argue there was a strong incentive to ignore any risks especially when you have a tails I win heads you lose sort of environment.

================

Why the Crisis?
Ironically, most of the buyers and sellers were graduates of elite business schools. Better than most, they should have known enough to avoid making loans with no down payments to borrowers who had few assets and poor credit records. The reason they didn’t is that their incentive structure encouraged them to ignore the quality of what they sold and bought.

The financial industry pays enormous bonuses for “performance.” For several years, the profits for subprime traders were great and the bonuses were large (and, for many, irresistible). Their supervisors had the same incentives. The main incentive was to increase the size of one’s bonus by increasing the firm’s bottom line. Failure to play the game could cost you not only your bonus, but also your job. The financial industry’s compensation system created these incentives. By the time the crisis came, the people who fueled it had sold their subprime assets to someone else, along with assurances from the rating agencies.

http://american.com/archive/2008/february-02-08/why-the-crisis/?searchterm=credit%20crisis

=================

Wall Street, Run Amok

Weren’t fail-safe devices in place to guard against risk? Weren’t government watchdogs there to make sure that catastrophes could not happen? Weren’t ratings agencies on the job to police what was going on in the canyons of Lower Manhattan?

First, Maestro Einhorn points out that the fellows who run big investment banks have a strong incentive to maximize their assets and leverage themselves into deep trouble because their pay is a function of how much debt they can pile on. If they can use relatively low-interest debt to generate slightly higher returns, the firm earns more revenue and executive pay increases. Often, an astonishing 50 percent of total revenue goes to employee compensation at Wall Street firms.

http://www.nytimes.com/2008/04/27/business/27every.html?dlbk

[Alex Tolley]

Mathematical models are still probably better than nothing or guesswork. The problem is that they are relatively poor models of human behavior.

The reliance on models is the result of being able to do "what if" modeling on spreadsheets. The problem is that this becomes a crutch - you can spend time running numerical scenarios fooling yourself that this is more important and realistic than qualitative ones. Financial folks and MBAs who are very comfortable with numbers often fail to see how futile this is.

[Anon]

Alex T.

Ah, but are mathematical models better than qualitative research? Surely that will depend mostly on the quality of the mathematical model and the quality of the qualitative research.

[RW]

Models don't only have rules they possess a domain, a set of boundaries, under which the rules can be expected to operate reasonably reliably. By that same even the most robust models must have a breaking point, a point over a boundary where they break, or they would be so general as to possess little worth as models. Most risk managers probably did not feel a pressing need to push their models over the edge, to purposely apply "what if" inputs that would break them and then imagine if those inputs were within the realm of possibility (clearly they were), and even in cases where risk managers did so push and alerted higher management I suspect they would have been ignored: There was just too much money sloshing through the system and scooping it up was very cheap, in the shorter term at least (and when results are measured monthly and quarterly what other term is there).

[donna]

Oh, come on. I knew we were in trouble when our math and physics major friends ended up getting ultra high paying jobs on Wall Street instead of doing math and physics. Doesn't that in itself tell you the basic structure of our economy is messed up? It certainly indicated to me that there were risky financial games taking place.

Scientists should be doing science, not making big bucks by making models for other people to make big bucks. The problem is financial systems that create false wealth, instead of real growth, not that use the wrong models. It's only a matter of time before any Ponzi scheme fails.

[paine]

mark apparently i'm not as impressed with this piece
as you are
after reading it
i'm not enlightened
i feel shimmy jimmy -ed

"even a cursory glance at history reveals that mortgage defaults become highly correlated in downturns"
so far so good

"One cannot escape the feeling that many players understood what was going on but happily went along."
okay but why

"Underpinning this whole process is a view that sophistication implies quality: a really complicated statistical model must be right. "

that's silly
sure the rubes might fall for the magic model scam
the total risk neutralizer
the perpetual motion machine of investment
profits for all
now and forever
but the inside "players" didn't and don't believe their own marketing claims
btw
is that what this ultra vague phrase amounts to

"the importance of management"

the old guard recalling the last crisis
and making
back room provisions for the next one

the explanation
of endogenous risk
strikes me as thin and arm waving

hiding behind the regress of
"he knows that i know that he knows
gets tedious "
and to conflate it with
the correlation change
created by panic ...
come on

okay so timing the inevitable panic is impossible

so cutting the gordian knot for regs
means huge equity margins for players
and thus forever lower return rates

so what elsethe players go back to the old black magic
building even better risk detectors
ie
more p t barnum flim flams
while re corrupting the regulators

all this is big stuff
ready to reload and happen again
so we plebs coulda used a starker clearer
finger in the eye presentation
then this account provides

[paine]

jack:

"Indeed, the current crisis took everybody by surprise..."
Really? I remember reading quite a bit from a few economists (Paul Krugman, Dean Baker) that predicted the current crisis pretty accurately"

key word "pretty"

timing the turn
is okay fpr a private investor
with no comparative performance peers to worry about
timing is not for super high leveraged agents
of huge pools of equity
if they call it wrong by getting out ahead of it
into lower return lower geared placements
or say agressively shorting the boat race to soon
i mean only ahead of the real final tumble
by say
a few months
and
hey that could cost you your job

no one had the means to accurately time
this inevitable ponzi bust
not closely enough at any rate
not to less then say
a quarter
or to where
the futures markets end

it was all
guts
and
guess work

come gain
or
goof
u are an agent
playing other folks chips
so u keep at the plate
keep swinging

[paine]

folks don't confuse the novel flim flam
for the age old back story reality

keep your eyes on the dealer;s hands
when it matters the next card
will come out of a sleeve
or
off the bottom of the deck

[paine]

moral
don't play
unless you're the dealer

[ampersand]

Every model is built on assumptions. Given A, B, C ... then X, Y, Z.

What every bubble watcher could make out - and none these elite academic economists and Fed researchers could ever admit- was the fact that A, B, C did not hold true anymore. The ordinary man in his daily grind could see the fraud, deceit and lying that went around him, because he knew what the buyers earned, what kind of people they were, and what price hosues they were buying. He knew that these people could not afford these houses.

The Fed and academia looked at the data in the applications, and data in the markets and pretended - PRETENDED - that there was nothing wrong. After all when one is being bombared by NINJA loan ads 24x7 on the radio, TV, and internet, it takes willfull and criminal malfeasance to ignore that and still pretend that the data is good, and the model assumptions are good.

Even now, these people pretend that all this was unexpected.

The fit thing for them, if they really think that all this was unforseen, is to admit their incompetence. And then to shut up. You have ZERO credibility.

Even thieves have better honour.

[zinc]

Please ask the sophisticates to input into their model that the potential borrower has a FICO score in the lowest 10 pct, has no income or earns less than $25000 per year, has no savings, owes on home appliances, automobiles, and credit cards.

Do you think the model will classify this borrower as AAA ?.

Clustered in a downturn my ass. The US taxpayer has been fleeced big time, mafia style, with the aid of the John Birch republicans.

[ken melvin]

... you knew I was a when you ...

The credit rating of the buyers played little role.

This sort of thing will be more prevalent as there are fewer ways to skin the cat for those who have climbed the ivy. Once, perhaps, they'd built a plant and made widgets. You see, there's tremendous pressure.

[ken melvin]

out damned snake

[evagrius]

The Shell Game’s Allure
At the end of the 1800s one of the best known shell men in the United States was Jim Miner, also known as Umbrella Jim. He used to introduce his game with a wonderful little song, thankfully recorded in Gambling and Gambling Devices by John Phillip Quinn (a reformed gambler), which went as follows:

A little fun, just now and then
Is relished by the best of men.
If you have nerve, you may have plenty;
Five, draws you ten, and ten draws twenty.
Attention giv’n, I’ll show to you,
How umbrellas hide the peek-a-boo.
Select your shell, the one you choose;
If right, you win, if not, you lose;
The game itself is lots of fun,
Jim’s chances though, are two to one;
And I tell you that your chance is slim
To win a prize from ‘Umbrella Jim’!

[Bruce Wilder]

"Blame the way risk models were used" is a bald attempt to cover up the crime. Reference to presumably arcane mathematical modelling is just a fancy way of saying, "Pay no attention to that man behind the curtain." To take up the shell game analogy, it is playing third man in the con, the one who comments on the second man's skill, or lack of skill, in playing the game.

It seems eerily parallel to the glib complacency, which excuses John Yoo's Torture Memo as a "good faith" error.

[a]

Economists' use models to predict how the economy will behave on a macro level, and they are trying to predict systems at least as complicated as financial markets. I would hope that scepticism to how models actually perform in financial markets would give economists pause about their own attempts.

[reason]

It seems to me we should have remembered to add uncertainty about the model to the uncertainty about the securities.

[reason]

Still it goes with a general theme I have - redundant exactitude. Because we can express a number to 4 decimal places, it doesn't mean that that number carries information content at that level of exactness. Almost all economic measures are inexact proxies. And over time concentrating on maximising proxies makes the proxies themselves less good at measuring what we really want to measure. Forgetting that is the cause of a lot of the problems we have today.

[reason]

P.S. Leveraged arbitrage, by the way makes redundant exactitude a REALLY serious problem.

[Masonik]

"I knew we were in trouble when our math and physics major friends ended up getting ultra high paying jobs on Wall Street instead of doing math and physics."

The search is on to find the "unified theory" to explain the markets. They are throwing everything at it including theories on genetics, biology, physics, and etc. Big money at stake no doubt.

==============

The New Math

For a man whose flagship hedge fund is running on fumes, Marek Fludzinski couldn’t be calmer. The founder and CEO of New York–based Thales Fund Management has watched his firm’s assets plummet by more than $1 billion during the past year, as Thales, like most quantitative managers, has suffered as a result of the global credit crisis that began last summer. But Fludzinski, who has a Ph.D. in theoretical physics from Princeton University and was one of the first two dozen employees at famed quant shop D.E. Shaw Group, is on a mission that means far more to him than profit and loss. He believes science is the key to unlocking the inner workings of the markets, and he intends to devote significant resources to prove it.

http://www.alphamagazine.com/Article.aspx?ArticleID=1897101

[Blissex]

«Ultimately this implies that the quality of SIV ratings left something to be desired. However, the rating agencies have an 80-year history of evaluating corporate obligations, which does give us a benchmark to assess the ratings quality. Unfortunately, the quality of SIV ratings differs from the quality of ratings of regular corporations. A AAA for a SIV is not the same as a AAA for Microsoft.»

As they kept selling AAA ratings on CDOs and other synthetics, Moody's business boomed, as new issuances were encouraged mightily by those generous ratings.

This is how Moody's stock price has gone in the past 30 years, and look at it benefit from the enormous success of its CDO ratings business:

http://money.cnn.com/quote/chart/charts.popup.html?symb=MCO&time=30yr

S&P and Fitch are part of other companies, so the data are not not so obvious, but you can expect their numbers to be similar.

Moody's management sold a lot of ratings on a lot of CDOs, and had a chance at becoming very very wealthy.

[Darren]

I like the phrase "foolish sophistication". My own phrase for this phenomenon, coined during my MA in Economics, was "pseudo-rigour".... analysis that covers a whole blackboard with squiggles and LOOKS rigourous, impressive, and irrefutable, but really isn't because the domain of applicability of the analysis has been forgotten. There is a certain kind of person prone to this kind of error, who is very comfortable with mathematics and quantitative analysis but isn't much of a critical thinker (critical thinkers tend to be driven mad by academic economics), and it is exactly THIS kind of person who is selected for in advanced quant finance/econ programs, and goes on to develop risk management models. They really DO believe what the models are telling them.

[reason]

Darren,
there is a simple phrase for this - "the map is not the territory".