Mixing Memory

Blogs and the mainstream media have been filled with neuroscience news lately. First we learned that sarcasm happens in the brain, and then that sexual orientation is in the brain too. There was even an attempt (sarcastic, I hope) to account for sports fandom with mirror neurons (I've heard that the actual reason we like watching sports is because we have retinas(1)). Neuroscience is all the rage, man.

I haven't, however, seen much coverage of what I think is the coolest recent neuroscience finding. That finding was reported in a paper in the May 30 issue of Science, titled "Predicting Human Brain Activity Associated with the Meanings of Nouns" (2). And the finding is even cooler than the title suggests! They used fMRI data from some nouns, along with data from a huge corpus of texts, to predict the activity of new nouns, so that they could actually tell you what noun a person was looking at by looking at the person's brain activity, despite the fact that the model they were using to predict fMRI signals had never actually "seen" that noun before.

To understand how they did this, let's start with their model (the image below presents a schematic explanation), which is really pretty simple. To start, Mitchell et al. collected "intermediate semantic features" for a bunch of words using a huge corpus (a trillion words, give or take). "Intermediate semantic feature" is just a fancy way of saying co-occurrence information. In essence, Mitchell et al. compute a metric that tells you how often two words co-occur. If you do that for a bunch of words, what you get is a high-dimensional space with a bunch of vectors representing the co-occurrence relationships between the various words in the space.

From Mitchell et al., Figure 1, p. 1191

After collecting "intermediate semantic features" (I just like typing that) for 25 verbs, they began feeding the model the data for a bunch of nouns. If we consider each "intermediate semantic feature" a dimension, then what they did in essence is look at the value on each dimension for each noun. They then compare the values for each noun fMRI data from participants who looked at that noun. Using a fancy equation, this allows them to associate each "intermediate semantic feature" with activity at each voxel (a 3D pixel in an fMRI image) in the fMRI data. They then use the associations between semantic features and brain activity to predict the brain activity associated with new nouns, given those nouns semantic features. Their model was able to accurately predict the voxel by voxel brain activity associated with new nouns between 68 and 83% of the time (the average accuracy was 77%), all of which are reliably above chance (which would be 50%). Even when they gave the model new nouns from semantic categories that it hadn't been trained on (e.g., if it were trained on animals, plants, body parts, and clothing, it might be tested on nouns in the tools category), and it was still accurate above chance (the average accuracy dropped to 70%, but that's still pretty good).

In summary, then, by comparing semantic information for nouns with the brain activity associated with looking at those nouns, Mitchell et al. were able to derive associations between meaning and brain activation, and use those associations to predict the brain activation of nouns not used in training their model. Once trained, then, their model can accurately guess what word you're looking at by looking at your brain activity, even if you're looking at a word that the model's never seen before. Perhaps most interesting of all, because they were using fMRI data averaged across several participants (9, to be exact), the ability of the model to predict words from activation suggests that their is a lot of overlap in how and where particular words are represented in the brain across individuals. That is, it suggests that we all represent words in our brain similarly. I don't know about you, but that strikes me as a much cooler finding that one showing that sarcasm happens in the brain.

¹Laux, J. (Personal communication, June 16, 2008)
²Mitchell, T.M., Shinkareva, S.V., Carlson, A., Chang, K.M., Malave, V.L., Mason, R.A., & Just, M.A. (2008). Predicting human brain activity associated with the meaning of nouns. Science, 320(5880), 1191 - 1195.