Financial managers are rewarded for results which are imperfectly aligned with the interests of investors. Managers will receive a bonus for high returns and for amounts of money invested, and are therefore rewarded for risky positions with good short-term performance - the high returns will attract investors and give the manager lucrative bonuses. Because the manager still walks away with their bonus from previous years if there is a crash, they have little incentive to avoid taking on low-probability, high-magnitude risks.
Managers are drawn to badly-investor-aligned behaviors in multiple ways. There is surely some amount of outright fraud. However, I expect that this is dwarfed by perceptual filtering with no outright evil intention. Because the manager is rewarded for attracting investors and achieving a high rate of return this year, they will focus on aspects which enable this; they will not dig as deeply into the risks because the risks are not as important in their minds. If the risks are incomprehensible, they are more likely to go ahead than to walk away. Many of them probably view risk management the way most people view speed limits - a bureaucratic obstacle to what they really want to do, requiring lip service at times, and for which other people get caught. We can be confident that many financial managers will behave this way because the situation selects them for these traits.
There are several things managers can do to achieve this type of profile. During stable, calm economic times with steady growth, they can increase leverage to extraordinary levels with little short-term risk. By including difficult-to-value assets which do not have a liquid market in their portfolio, and using mark-to-market accounting, they can use their "judgment" about the value of the assets to show astounding growth and get their bonuses for several years; a collapse, if there is one, will not come until later when the assets are sold.
Investors are not completely stupid, and have learned to look for excessive leverage. When they find it, they discount the returns appropriately. Thus, in order to achieve an advantage, financial managers must not only use additional leverage, but somehow hide or obfuscate the fact that they are using leverage. The same applies to low-probability, high-magnitude events. Everyone understands that insuring against natural disasters has real costs even though the disasters are unlikely. The same is not true for other types of purely abstract financial events, especially those which are not presented as insurance, and whose downside is downplayed.
New types of products and derivatives are harder to understand and value because the intuition and tools have not been developed yet, providing both a shield from investor scrutiny and more freedom to set values. Therefore, they will allow financial managers to increase leverage and expose themselves to low-probability high-magnitude events with less comprehension on the part of their investors (or even themselves). Even if the new product or derivative is not originally created for this purpose, the emergent behavior of the system assures that the product will soon be picked up and used for it.
The base asset of the day is residential real estate. Created in part by Greenspan's policy of leaving interest rates low after the dot-com crash, cheap credit allowed people to pay more for houses while leaving their monthly outflow the same. This trend was assisted in part by historical observations about the lack of a nationwide housing price fall and by fundamental arguments about exponential population growth, all of which were correct in essence but incorrect in magnitude.
The illiquid product is the mortgage-backed collateralized debt obligation. In 1999, the financial services industry was finally able to obtain a repeal of the separations between banking and investment, originally enacted in 1933 as part of the Glass-Steagall act. Known as the Gramm-Leach-Bliley act, this repeal was bought and paid for by the financial services industry, and was a product of extensive lobbying. This reconnection between banking and investment allowed mortgages to be packaged up and held as investments. This product met all the requirements for boosting returns - it was illiquid, new, difficult to objectively value, and would not crash until later. Once the market for these products appeared, it provided financial motivation to relax mortgage lending standards.
The derivative is the credit-default-swap. Essentially these are a sort of insurance on bond defaults. (ssh! don't say the I-word! People might get the right idea!) Originally created for bondholders (and other credit holders) to hedge against the risk of default by the issuer, they were, like other derivates, picked up by those not involved in the primary transaction, seeking leverage instead of a hedge. In fact, the standard language for credit-default-swaps had to be rewritten to allow settlement without the surrender of the original bond, because many of them were transacted by people who never had any of the original bonds at all. The value of credit-default-swap contracts grew to 10x to 100x the amount of actual outstanding credit.
Thus, the inevitable happened. Housing prices went down. Instead of a 10% loss, the leverage ensured that everybody was taking a 300% loss. Nobody could come up with capital to cover that kind of loss. The resulting disaster resembled not so much a train wreck but a train running full speed straight into the side of a mountain because that was where the tracks had been built from the very beginning.
We cannot know what the next round of assets, products, and derivatives will be. The new regulation which will be introduced over the next year will address the symptoms of the problem and ensure that these particular products will not have another run; their successors will look different. We will know them not by what they are made of but by the roles that they fill.
...
A story is both more and less than a collection of observations. I know I have left out pieces that are undeniably important, like the role of the credit rating agencies, because their effect was secondary to the story I wanted to tell. If you want to know more, or want to look into the events yourself and develop a different narrative, I can recommend the following resources: