LOLbachia
Modern brain-scanning technology allows us to measure a person's brain activity on the fly and visualise the various parts of their brain as they switch on and off. But imagine being able to literally see what someone else is thinking - to be able to convert measurements of brain activity into actual images.
It's a scene reminiscent of the 'operators' in The Matrix, but this technology may soon stray from the realm of science-fiction into that of science-fact. Kendrick Kay and colleagues from the University of California, Berkeley have created a decoder that can accurately work out the one image from a large set that an observer is looking at, based solely on a scan of their brain activity.
The machine is still a while away from being a full-blown brain-reader. Rather than reconstructing what the onlooker is viewing from scratch, it can only select the most likely fit from a set of possible images. Even so, it's no small feat, especially since the set of possible pictures is both very large and completely new to the viewer. And while previous similar studies used very simple images like gratings, Kay's decoder has the ability to recognise actual photos.
Training the brain-reader
To begin with, Kay calibrated the machine with two willing subjects - himself and research partner, Thomas Naselaris. The duo painstakingly worked their way through 1,750 photos while sitting in an fMRI scanner, a machine that measures blood flow in the brain to detect both active and inactive regions. The scanner focused on three sites (V1, V2 and V3) within the part of the brain that processes images - the visual cortex (right).
The neurons in the visual cortex are all triggered by slightly different features in the things we see. All of them have different 'receptive fields'; that is to say that they respond to slightly different sections within a person's field of vision. Some are also tuned to specific orientations, such as horizontal or vertical objects, while others fire depending on 'spatial frequency', a measurement that roughly corresponds to how busy and detailed a part of a scene is.
By measuring these responses with the fMRI scanner, Kay created a sophisticated model that could predict how each small section of the visual cortex would respond to different images.
Kay and Naselaris tested the model by looking at 120 brand-new images and once again, recording their brain activity throughout the experience. To account for 'noisy' variations in the fMRI scans, they averaged the readouts from 13 trials before feeding the results into the decoder.
The duo then showed the 120 new images to the decoder itself, which used its model to predict the pattern of brain activity that each one would trigger. The programme paired up the closest matches for the predicted and actual brain patterns and guessed the order of the images that Kay and Naselnaris had looked at.
Success!
It was incredibly successful, correctly identifying 92% of the images that Kay looked at and 72% of those viewed by Naselnaris. Obviously, using the average of 13 scans is a bit of a cheat, and if the machine were to ever decode brain patterns in real-time, it would need to do so based on a single trial. Fortunately, Kay found that this is entirely feasible, albeit with further tweaking. Even when he fed the decoder with fMRI data from a single trial, it still managed to correctly pick out 51% and 32% of the images respectively.
The decoder could even cope with much larger sets of images. When Kay repeated the experiment with a set of 1,000 pictures to choose from, the machine's success rate (using Kay's brain patterns) only fell from 92% to 82%. Based on these results, the decoder would still have a one in ten chance of picking the right image from a library of 100 billion images. That's over a hundred times greater than the number of pictures currently indexed by Google's image search.
Obviously, the technology is still in its infancy - we are still a while away from a real-time readout of a person's thoughts and dreams based on the activity of their neurons. But to Kay, his experiments are proof that such a device is possible and could be a reality sooner than we think.
Reference: Kay, K.N., Naselaris, T., Prenger, R.J., Gallant, J.L. (2008). Identifying natural images from human brain activity. Nature DOI: 10.1038/nature06713
Comments
Where is this published?
Posted by: Mike P | March 5, 2008 1:24 PM
Nature, in the issue that comes out tomorrow. When the paper is published online, it will go into the DOI database and I can stick up a citation using the Research Blogging network.
Posted by: Ed Yong | March 5, 2008 2:00 PM
Cool, thanks! Good post!
Posted by: Mike P | March 5, 2008 2:24 PM
Very cool, thanks for sharing!
The first thing I thought of wasn't The Matrix. I grant that it's been a long time since I've watched The Matrix, but from what I recall this technology is less Matrix and more Final Fantasy: The Spirits Within. In the latter film, the main character's dreams are recorded for later review. In The Matrix, it was my understanding that the digital readout observed by the operators is a reflection of what is happening in the matrix. True, the crew's mind was plugged into the matrix, but the operators could observe events in the program beyond the crew's virtual perceptions.
Um... Anyway, definitely looking forward to seeing how this research progresses.
Posted by: Kevin L. | March 5, 2008 7:24 PM
Reference now up. And you know, the point about the Matrix did run through my head while I was writing this, but it was 1 in the morning and the part of my head that contains the sci-fi references was asleep and I was sorta counting on people being awesomed by the machine and not noticing. :-p
Posted by: Ed Yong | March 5, 2008 8:13 PM
I wrote an article about fMRIs and fear conditioning last year, and consider myself somewhat literate on the subject, but this simply blows my mind. I have to remain skeptical until I look at the primary data. And even then...ha
Posted by: caynazzo | March 5, 2008 10:15 PM
This is off-topic, for which I apologise -- but I figured, if anyone can help on this, it would be Ed Yong or readers here. ;-)
My husband is going to be teaching a graduate seminar on clear writing and communication. For illustration purposes, he needs to dig up a really badly written physics paper. Not bad research -- just impenetrable and unclear writing. My field is biology and computing, and I've already been able to give him examples of that for biology and medicine, but nearly all physics papers are jargon-filled, unclear and impenetrable as far as I'm concerned, so it's difficult for me to select one or two.
Sorry about sticking this on an unrelated thread, again.
But would you be able to make any nominations for this?
Posted by: Luna_the_cat | March 6, 2008 6:39 AM
Sadly, I'm not sure I can - there's a reason that there aren't any physics posts on this blog and that's because anything physics that doesn't have the name "Feynman" or "Davies" on it is as impenetrable to me as it is to you. Any readers have suggestions?
Good luck to your husband though - a worthy cause indeed. For his info, I'd highlight this paper which I wrote about here as a really nice example of a very well written paper that still retains academic integrity while being a good read. And, obviously, the Origin of the Species, which is arguably the most important biological text for several centuries and can be read by a schoolkid.
Posted by: Ed Yong | March 6, 2008 6:51 AM