Smilodon is perhaps the most famous of all the saber-toothed cats, but the level of notoriety it has received has led to a number of misunderstandings. As a child I remember hearing in a documentary (complete with somewhat hokey stop-motion giant sloths) that sabercats became extinct because their teeth grew so long that they could not close their mouths. I did not know it at the time, but this fallacious idea had been around for quite some time, and was quite surprised to find an effective refutation of it when I read G.G. Simpson's popular book The Meaning of Evolution (1950);
The sabertooth is one of the most famous of animals just because it is often innocently supposed to be an indisputable example of an inadaptive trend. In fields far remote from paleontology the poor sabertooth has some to figure as a horrible example, a pathetic case history of evolution gone wrong. Its supposed evidence is thus characteristically summarized in a book on (human) personality: "The long canine tooth of the saber-toothed tiger grew more and more into an impossible occlusion. Finally, it was so long that the tiger could not bite effectively, and the animal became extinct." Now, like so many things that everyone seems to know, this is not true... Throughout their history the size of sabertooth canines varied considerably from one group to another but varied about a fairly constant average size, which is exactly what would be expected if the size were adaptive at all times and there were no secular trend in adaptive advantage but only local and temporary differences in its details. The biting mechanism in the last sabertooths was still perfectly effective, no less and probably no more so than in the Oligocene. To characterize a finally ineffective a mechanism that persisted without essential change in a group abundant and obviously highly successful for some 40,000,000 years seems quaintly illogical! In short, the "inadaptive trend" of the sabertooth is a mere fairy tale, or more fairly, it was an error based on too facile conclusion from imperfect information and it has since been perpetuated as a scientific legend.
One of my favorite "hypotheses" for how sabertooths lived: they actually spilled pools of blood from their victims and then drank from the pools. (Wish I could remember the source.)
When I interviewed Dr. Chris Shaw about Smilodons, he made an interesting observation. With most carnivorans, perhaps as many as 25 percent of the population will be found to have chipped or broken canine teeth, showing that survival is fairly easy even with at least one nonfunctional canine. With sabertooths, the percentage is more like one or two percent. The canines were so critical to their hunting success that if one got broken, the cat apparently had little chance of surviving for very long.
I again submit that the hypertrophied canines had a social display purpose.