How do you avoid having sex with your close relatives? Well, not you, specifically, but how is it done generally, or perhaps among mammals in particular?
One obvious way would be to use a part of the genome that seems to evolve rapidly, and that would be able to distinguish between even moderately closely related individuals. For instance, the Major Histocompatibility Complex (MHC) is big, diverse, and changes rapidly. If two potential mates could somehow compare their MHC's, they could estimate their degree of relatedness, and thus avoid inbreeding.
An article just out in Current Biology examines this question in mice. Noting that the MHC is a commonly invoked likely candidate for kin recognition, Sherborne et al (2007) also note that very little direct evidence indicating exactly what mechanism is used exists. SHerborne et al also notes that there is another highly variable genetic system, the Major Urinary Protein Gene Cluster (MUP).
This is a pretty obscure gene cluster. You won't even find it in Wikipedia. (Enter "MUP" and you get things like "Melbourne Universityi Press" and Ministarstvo UnutraÅ¡njih Polsova -- the Bosnian Ministry of the Interior.)
These researchers designed an experiment to test whether mice use the MHC or the MUP to avoid inbreeding. They used "wild" house mice (Mus musculus domesticus). (Which kind of brings up the degree to which biological scientists get the concept of "wild." ... house mice are never wild. They are commensal. But that is another story.) The point of this was to avoid genetically homogeneous lab mice all of whom have undergone severe bottlenecking in their histories. They independently measured actual genetic similarity as a proxy for relatedness (of course, in real life, relatedness is a proxy for actual genetic similarity ... proving that sometimes tautologies are useful). They also let the mice run around in a "natural" enclosure ... meaning very large ... and do whatever they wanted. They ran this experiment for a few generations.
The result was a reduction in the frequency of successful matings betwen full sibs, but a statistically weak one. There was no evidence for avoidance on the basis of MHC similarity. But, the MUP genetic complex had a "strong and highly significant effect on the likelihood of successful mating ... there was no [reduction] when only one MUP haplotype was shared, but there were many fewer matings between mice that shared both [of the] MUP haplotypes ... than expected under random mating conditions."
This experiment was carried out among four populations, and the same results found in all cases.
In the end:
The use of MUP alone was sufficient to explain inbreeding avoidance in this study. Although the deficit in mating between mice of the same MUP type was very strong, there was no evidence that animals used genetic markers other than MUP to improve their level of inbreeding avoidance.
This interesting study challenges the commonly held assumption that the mammals would use odor cues linked to the MHC to avoid inbreeding. This idea was very attractive because, while inbreeding has its special problems, avoiding inbreeding (i.e., out breeding) has the independent benefit of enhancing MHC variability, which confers enhanced immune system capacity on the offspring of such out-matings. However, MUP cues for out breeding result in the same sort of MHC diversification anyway, according to this study.
Furthermore, while the MHC is related to immune system function and also could be (but maybe is not) used for inter-individual recognition, MUP appears to do only one thing: Signal individuality via scent. Nobody knows of any other purpose of MUPs.
Sherborne, Amy L., Michael D. Thom, Steve Paterson, Francine Jury, William E.R. Ollier, Paula Stockley, Robert J. Beynon and Jane L. Hurst. (2007) The Genetic Basis of Inbreeding Avoidance in House Mice. Current Biology In Press. doi:10.1016/j.cub.2007.10.041
Is there any evidence of any influence of MUPs on pheromones? This study would appear to indicate that a connection there should be investigated.
CH: MUP's are, essentially, Mouse pheromones. I don't know off hand what the human homolog is ... I'll bet we have that gene cluster, but I don't know what we do with it.
OK, so you can tell who is too related to mate to by smell? The response is not to be attracted to/not mate with a related individual? I breed dairy goats and my herd is fairly (in human terms) inbred. I don't see any indication that there is any unwillingness to breed with ones relatives, even siblings, and have to make a lot of effort to prevent it when I want to avoid it. Same thing with sheep so spare me the goat jokes :-) Maybe the difference is that goats and sheep are herd animals and close breedings are common where there are many females and only a few/one males? But then, my turkeys don't seem to care how closely they mate either, even parent child matings are common. I know this because I had one tom and all hens, even his daughters hatched chicks. I guess I'm just not seeing this thing in the barnyard.
yeila: There are likely to be other factors as well, such as imprinting. In humans, that takes the form of the famous Westermarck effect, where we imprint on our siblings during a specific period during the toddler years.
Also, the instincts in question may well have been damaged during domestication (almost certainly for the birds, quite likely for the goats).
MUP's are, essentially, Mouse pheromones.
It's actually even more interesting than that. MUPs are not mouse pheremones; they are pheremone-binding proteins, that slowly release their bound cargo of pheremones over time.