Timothy Geithner explains how the stress tests were conducted:
How We Tested the Big Banks, by Timothy Geithner, Commentary, NY Times: This afternoon, Treasury, the Federal Deposit Insurance Corporation, the Office of the Comptroller of the Currency and the Federal Reserve will announce the results of an unprecedented review of the capital position of the nation’s largest banks. ...
We ... chose a strategy to lift the fog of uncertainty over bank balance sheets and to help ensure that the major banks, individually and collectively, had the capital to continue lending even in a worse than expected recession.
We brought together bank supervisors to undertake an exceptional assessment of the strength of our nation’s 19 largest banks. The object was to estimate potential future losses, and ensure that banks had enough capital to keep lending even in the face of a deeper recession.
Some might argue that this testing was overly punitive, while others might claim it could understate the potential need for additional capital. The test designed by the Federal Reserve and the supervisors sought to strike the right balance.
The Federal Reserve marshaled hundreds of supervisors to spend 45 days rigorously reviewing the banks’ detailed loan data. They applied exacting estimates of potential losses over two years, along with conservative estimates of potential earnings over the same period, and compared them with existing reserves and capital. The results were then evaluated against strict minimum capital standards, in terms of both overall capital and tangible common equity.
The effect of this capital assessment will be to help replace uncertainty with transparency. It will provide greater clarity about the resources major banks have to absorb future losses. It will also bring more private capital into the financial system, increasing the capacity for future lending; allow investors to differentiate more clearly among banks; and ultimately make it easier for banks to raise enough private capital to repay the money they have already received from the government.
The test results will indicate that some banks need to raise additional capital to provide a stronger foundation of resources over and above their current capital ratios. These banks have a range of options to raise capital over six months, including new common equity offerings and the conversion of other forms of capital into common equity. As part of this process, banks will continue to restructure, selling non-core businesses to raise capital. ...
Banks will also have the opportunity to request additional capital from the government through Treasury’s Capital Assistance Program. Treasury is providing this backstop so that markets can have confidence that we will maintain sufficient capital in the financial system. For institutions in which the federal government becomes a common shareholder, we will seek to maximize value for taxpayers and enable these companies to attract private capital, thereby reducing government ownership as quickly as possible.
Some banks will be able to begin returning capital to the government, provided they demonstrate that they can finance themselves without F.D.I.C. guarantees. In fact, we expect banks to repay more than the $25 billion initially estimated. This will free up resources to help support community banks, encourage small-business lending and help repair and restart the securities markets. ...
This is just a beginning, however. Our work is far from over. The cost of credit remains exceptionally high... The ultimate purpose of these programs is to ensure that the financial system supports rather than impedes economic recovery.
We have not reached the end of the recession or the financial crisis, but the bank stress tests should advance the process of repairing our financial system and provide a better foundation for recovery.
I've given a lot of tests over the years, and I can pretty much make the mean on a test come out how I want through the design of the questions and how I score the answers. If I want a mean of 70, or around there, I can get it, and if a mean of 50 is the target, that's possible too. The other thing that I've learned is that the relative rank of students in the class doesn't change much, the A, B, C, D, and F students will line up pretty much as they always do. But a mean of 50 will lead to much more complaining, lower evaluations, and general dissatisfaction with the course than a mean of 70 even if you grade on a curve so that the grades are identical. So, since I learn the same thing either way in terms of relative understanding of the material, targeting a mean of 70 works out better.
But you have to be careful since there is also an absolute standard to worry about. Someone who gets the mean score and a grade of, say, C, is being stamped with a particular level of competence, and the questions cannot be so easy that a C is awarded to students who do not have this minimal level of understanding.
The people designing the stress tests have the same freedom. Depending on the "questions" they ask the balance sheets, and how the "answers" are scored, they can get whatever mean they desire (e.g. how are assets that cannot be valued in the marketplace are "scored" makes a crucial difference in the outcome). If we choose a score of "70" as the dividing point between being solvent and being insolvent, then the percentage of banks passing the test is a function of the difficulty of the stress test how the items on the balance sheets - the answers to the questions - are interpreted.
That's where, as with the class and awarding a passing grade of C, an absolute standard comes in. If the test is just hard enough, but not too hard, if it hits the Goldilocks sweet spot, or close enough anyway, then the results can be trusted. But does anyone know for sure what that absolute standard is? And if we don't, what do the tests really mean? There will always be uncertainty.
There is a way out of this box, I suppose, and that's to give a test that everyone can plainly see is hard, no doubt about it, and then have most of the banks still pass it. So if officials can convince everyone that this was, in fact, a really, really hard test and banks did well anyway, that could work. But that's not what they did, they sought balance and in doing so, Geithner explicitly admits they are open to the criticism that the test was too easy (of course, if they had designed a hard test and many banks failed, they'd be accused of trying to use a biased test to force politicians to support nationalizing troubled banks).
And there are other problems too. Again, how did they value the toxic assets that nobody has been able to find a way to value? Were the questions biased, i.e. was it proper to apply the same test to the different institutions that were forced to take it, or should the test have been varied to fit the profile of particular banks? A big part of the problem with bank balance sheets is the things we cannot see, do not know about, and cannot predict. How were those things accounted for in the stress tests? If we had such a hard time predicting how bank balance sheets would change until now, then what changed that makes the tests credible? Why should we believe we can now predict what we haven't been able to predict previously? Why did the government negotiate the outcome with banks and how lenient were they in those negotiations? There are always students who want to argue about the result of a test, to have sections regraded, and how you respond to attempts to "negotiate" a grade can affect the percentage passing the class, particularly when - as with the stress tests - there aren't a lot of students/banks taking the test.
I don't think there's an outcome here that is all that helpful. If banks pass the test, for the most part - as government officials are assuring us they will - people will argue the test was too easy, or biased, or invalid for some other reason. If many banks fail (an unlikely event given the rosy talk from Fed and Treasury officials), that will cause even more fear and uncertainty in markets and the tests, which were supposed to bring certainty and calm to financial markets, will backfire (a reason to avoid this outcome as you are scoring the stress test results). I hope I'm wrong and Treasury can, in fact, convince people that the tests are credible, and the outcome is optimistic, but with all the uncertainty surrounding this process, that won't be easy to do.