Take a look at these two pictures of the Mona Lisa:
They’re derived from a series of images of the famous painting that had been obscured by random noise filters (like when your old analog TV wasn’t getting a signal), like this:
Each picture appears to have a slightly different facial expression — some happier, some sadder, depending on the random alteration of the image due to the visual noise. The two color pictures above are composites, made by picking the saddest (for picture B) and happiest (picture C) from over a hundred random images — rated by 12 volunteers — and combining them into a single image. So what makes the picture B seem so sad and picture C so happy? Let’s make this a poll:
To me, just eyeballing the picture, it seems like the mouths and the eyes are changing. But perhaps that’s just an illusion. Leonid Kontsevich and Christopher Tyler, who generated these images based on the ratings of the 12 observers, conducted a statistical analysis of the original randomly-generated images. They considered 7000 different regions of each random face and identified the ones that were similar when most observers rated the face happy or sad (as opposed to neutral). This image shows the results:
The only relevant region identified on each picture is around the corners of the mouth. The other regions that the statistical analysis identified are clearly not relevant to emotion (their statistical model assumed that up to seven irrelevant regions would be identified). If only the mouth is changing in the pictures, then why (to my eye, at least), does it look like the eyes are different too?
The researchers created composite images of just the bottom halves of the “sad” and “happy” composites and the original, unaltered tops, like this:
They asked a new set viewers to rate the expression of the the eyes in each of these images, and the rating for the “happy” face (B) was significantly happier than for the “sad” face (A). I have to say, B’s eyes look much happier to me, despite my knowledge that they’re identical to the eyes in A.
While there are a few possible explanations of this effect, the most likely one is that we extrapolate a “happy” or “sad” eye expression when we see it in the mouth.
One thing I wished the researchers had asked viewers is whether the smiles in these paintings (and the filtered composites) look authentic. We know that the eyes are the key to a true “Duchenne smile” — I’m not sure any of these pictures really convey a truly “happy” expression. They’re happier than the neutral or sad faces, but they don’t look especially authentic to me. Perhaps that is reflective of the random way in which the pictures were generated: we might need many more different random pictures to generate a truly authentic-looking smile.
Kontsevich, L., & Tyler, C. (2004). What makes Mona Lisa smile? Vision Research, 44 (13), 1493-1498 DOI: 10.1016/j.visres.2003.11.027