Cognitive Daily

Take a look at these two pictures of the Mona Lisa:

They’re derived from a series of images of the famous painting that had been obscured by random noise filters (like when your old analog TV wasn’t getting a signal), like this:

Each picture appears to have a slightly different facial expression — some happier, some sadder, depending on the random alteration of the image due to the visual noise. The two color pictures above are composites, made by picking the saddest (for picture B) and happiest (picture C) from over a hundred random images — rated by 12 volunteers — and combining them into a single image. So what makes the picture B seem so sad and picture C so happy? Let’s make this a poll:

To me, just eyeballing the picture, it seems like the mouths and the eyes are changing. But perhaps that’s just an illusion. Leonid Kontsevich and Christopher Tyler, who generated these images based on the ratings of the 12 observers, conducted a statistical analysis of the original randomly-generated images. They considered 7000 different regions of each random face and identified the ones that were similar when most observers rated the face happy or sad (as opposed to neutral). This image shows the results:

The only relevant region identified on each picture is around the corners of the mouth. The other regions that the statistical analysis identified are clearly not relevant to emotion (their statistical model assumed that up to seven irrelevant regions would be identified). If only the mouth is changing in the pictures, then why (to my eye, at least), does it look like the eyes are different too?

The researchers created composite images of just the bottom halves of the “sad” and “happy” composites and the original, unaltered tops, like this:

They asked a new set viewers to rate the expression of the the eyes in each of these images, and the rating for the “happy” face (B) was significantly happier than for the “sad” face (A). I have to say, B’s eyes look much happier to me, despite my knowledge that they’re identical to the eyes in A.

While there are a few possible explanations of this effect, the most likely one is that we extrapolate a “happy” or “sad” eye expression when we see it in the mouth.

One thing I wished the researchers had asked viewers is whether the smiles in these paintings (and the filtered composites) look authentic. We know that the eyes are the key to a true “Duchenne smile” — I’m not sure any of these pictures really convey a truly “happy” expression. They’re happier than the neutral or sad faces, but they don’t look especially authentic to me. Perhaps that is reflective of the random way in which the pictures were generated: we might need many more different random pictures to generate a truly authentic-looking smile.

Kontsevich, L., & Tyler, C. (2004). What makes Mona Lisa smile? Vision Research, 44 (13), 1493-1498 DOI: 10.1016/j.visres.2003.11.027

Comments

#1 Todd
March 18, 2009

I find the later A and B sample the most interesting as it leads us to conclude as you noted that we draw eye expression from the lower facial region as well.

In the upper example the researchers seemed to not take into account the stochastic distribution of their noise and the way it would interact with the underlying image.

To my eye, in the B vs C sample, the upper right brow is slightly darker suggesting the beginnings of a furrow, additionally the bridge of the nose is also showing signs of darkening leading to a crease indication.

The lower region of the face also shows signs of filling out, either intentionally or through the noise, giving an impression of slackening in the cheeks, jowls, and neck. this gives the overall image a “saddened” look when combined with the changes to the mouth area.

thoughts?
#2 humorix
March 18, 2009

If I remember myself well original of Leonard de Vinci, the Mona Lisa pulled the tongue, carried glasses and lived in Alaska.
#3 amar
March 18, 2009

i think there are 2 important things to remember here. the first is that the face is not a simply a mr. potato head whose pieces can be switched out independently. the muscles on the face are all connected so usually, when one moves a lip up or down, not only does it slightly alter the position of other muscles on the face, but it can change the depth of the flesh beneath the skin which will affect the way that light is reflected off of it, which, especially in a picture, can alter one’s perception of it. The second thing to remember builds on the problem perception. appearance are always altered by perception, and perception can be influenced by more than just the actual objects themselves. in this case, one’s perception of the eyes can be different based on their relationship to the mouth, even if the eyes have not changed at all.
#4 Mark P
March 18, 2009

I had pretty well convinced myself that the eye region looked different only because the mouth looked different. Pictures A and B pretty much confirm that; The eye region in B looks different – happier – than in A but is obviously not.
#5 Nigel
March 18, 2009

Initially I thought that the mouth and maybe the eyes showed a difference. The more I looked at the eyes trying to see exactly which pixels were different the less convinced I was. So I covered the mouth and nose with a piece of paper and decided that if given only the eyes I really couldn’t tell which one was smiling or not. Another example of what see is 25% visual input, 75% what our brains think should be there?
#6 Logan
March 19, 2009

Wow … I even took a piece of paper to cover the lower half of the image before voting, and still the happy image seemed to me as if the eyes were happier. My brain remembered the smile, even though I did not see it?
#7 Gard
March 19, 2009

In the B/C comparison at the top, there is a slight difference in the CHEEK. While the 2D edge of the hair is identical, the difference in shading give a different 3D interpretation of how the state of facial muscles.

In the composite variation, I do no interpret the eyes as happy. Instead, I read her face as “forced smiling” whilst trying to hide a grief or insecurity.

Then again, having read about faces when I was very small, where I trained myself to read the various parts of the face separately without having to cover them with pieces of paper may have influenced my interpretation – for which I am ever so grateful.
#8 buy wow gold
April 13, 2009

I think the main reason why you were able to achieve a happy face was because of mouth movement. Once you made changes there, it will automatically adjust the whole face.
#9 Ibrahim | ZenCollegeLife.com
April 18, 2009

This is really interesting. I always assumed that the eyes would be a tell, but I guess this isn’t necessarily the truth. Crazy how our brains work.