Take a look at these photos of Jim and Nora:

They’ve clearly been distorted (using the “spherize” filter in Photoshop), but in opposite directions. Jim’s been “expanded” to make more spherical, while Nora has been “contracted” to look more concave. If you look at these photos for a while, you might have difficulty recognizing how Jim and Nora look normally.

This is an aftereffect. Aftereffects can be experienced in a number of ways, with dizziness being perhaps the most frequently observed — spin around a few times with your eyes open and the room will appear to be spinning in the opposite direction. Researchers have found a number of different aftereffects in face perception. If you see a lot of spherically expanded faces, then normal faces will start to look contracted. If you look at a face that has the opposite characteristics of Jim’s face, then you might think an “average” face looks like Jim.
Two aftereffects can actually be induced simultaneously. If you look at pictures of Caucasian expanded faces and Chinese contracted faces, then normal Caucasian faces will seem contracted and normal Chinese faces will seem expanded. But there is still some transfer between races: If you look at only contracted Chinese faces, then both Chinese and Caucasian faces will seem expanded (though Caucasian faces won’t seem as expanded as Chinese faces).
So a 2005 study by a team led by A.C. Little was rather surprising. They found that some face aftereffects did not transfer from one sex to another. For example, if observers saw male faces with wide-set eyes, then normal male eyes seemed narrow-set, but normal female faces just seemed normal. Similar effects were found for masculinized and feminized faces of both genders.
Emma Jacquet and Gillian Rhodes decided to test these findings with a different type of transformation: the expansion and contraction I showed you above.
They divided 64 students into four groups. Each group saw only either male or female faces, which were only either expanded or contracted. So, for example, the male/expanded group saw six different expanded male faces, for two seconds at a time, flashed in random order, repeatedly for two minutes. Then they were shown a pair of faces of the same male or female. One face was expanded a small amount (10%, compared to 70% during the adaptation phase), and the other was contracted a small amount. They were instructed to choose the more “normal” face. This was repeated for 11 different faces, both male and female. Here are the results:

The graph shows the aftereffects divided by sex and expansion/contraction of the original images seen. A significant aftereffect was observed in every case except when viewers saw contracted female faces and then tried to judge male faces. What’s more, there was no significant difference in aftereffects observed when rating faces that were the same sex as the originally viewed faces and faces that were the opposite sex.
Jaquet and Rhodes repeated the study with simultaneous adaptation to male and female faces (e.g. male contracting and female expanding). This time, their results were mixed. They found some sex-specific aftereffects and some that were not.
The researchers say this suggests that while there are some sex differences in aftereffects, for the most part male and female faces are processed in the same way. Little’s team’s results, which consistently point to sex-specific aftereffects, may be due to the fact that most of the manipulations they did were sex-related (e.g. masculinizing or feminizing a face). Sex clearly plays a role in face recognition and processing, but some aspects of face processing are independent of the sex of the face being processed.
Jaquet, E., & Rhodes, G. (2008). Face aftereffects indicate dissociable, but not distinct, coding of male and female faces. Journal of Experimental Psychology: Human Perception and Performance, 34 (1), 101-112 DOI: 10.1037/0096-1523.34.1.101