Model Doubts

"Statistical modelling increasingly drives decision-making in the financial system while at the same time significant questions remain about model reliability and whether market participants trust these models. If we ask practitioners, regulators, or academics what they think of the quality of the statistical models underpinning pricing and risk analysis, their response is frequently negative. ... To have numbers seems to be more important than whether the numbers are reliable. This is a paradox. How can we simultaneously mistrust models and advocate their use? ... A single asset class worth only $400 billion should not be able to cause such turmoil. And indeed, the problem lies elsewhere, with how financial institutions packaged subprime loans into SIVs and conduits and the low quality of their ratings. ... [E]ven a cursory glance at history reveals that mortgage defaults become highly correlated in downturns. Unfortunately, the data samples used to rate SIVs often were not long enough to include a recession. Ultimately this implies that the quality of SIV ratings left something to be desired. ... After all, why would a AAA-rated SIV earn 200 basis points above a AAA-rated corporate bond? Underpinning this whole process is a view that sophistication implies quality; a really complicated statistical model must be right. That might be true if the laws of physics were akin to the statistical laws of finance. However, finance is not physics. ... Taken to the extreme, I have seen banks required to calculate the risk of annual loses once every thousand years, the so-called 99.9% annual losses. However, the fact that we can get such numbers does not mean the numbers mean anything. The problem is that we cannot backtest at such extreme frequencies. ... Fundamental to the scientific process is verification, in our case backtesting. Neither the 99.9% models, nor most tail value-at-risk models can be backtested. ... If we do not understand how the system works, generating numbers may give us comfort. But the numbers do not imply understanding", Jon Danielson (JD) at http://www.voxeu.org/, 8 May 2008.

I agree with JD. Without understanding concepts like arbitrage and discounted present value, the models are worse than worthless, they are "soma" pills. Mark Thoma has a post about JD's article at http://www.economistsview.com/, 8 May 2008 which is worth reading.