Recent advances in functional neuroimaging have enabled researchers to predict perceptual experiences with a high degree of accuracy. For example, it is possible to determine whether a subject is looking at a face or some other category of visual stimulus, such as a house. This is possible because we know that specific regions of the brain respond selectively to one type of stimulus but not another.
These studies have however been limited to small numbers of visual stimuli in specified categories, because they are based on prior knowledge of the neural activity associated with the conscious perception of each stimulus. For example, we know that the fusiform face area responds selectively to faces, and so we can predict that a subject is looking at a face if that area is active, or some other visual stimulus if it is not.
Researchers from the ATR Computational Neuroscience Laboratories in Kyoto, Japan have now made a significant advance in the use of fMRI to decode subjective experiences. They report a new approach which uses decoded activity from the visual cortex to accurately reconstruct viewed images which have not been previously experienced. The findings are published in the journal Neuron.
Yoichi Miyawaki and his colleagues exploited the functional properties of the visual system for their method. Specifically, they utilized a feature called retinotopy, whereby the spatial relationships between components of an image are retained as visual information passes through the visual system. Adjacent parts of an image are encoded by neighbouring neurons in the retina, and the topography remains in place when the information reaches the primary and secondary visual cortical areas (areas V1 and V2, respectively). Here, the so-called "simple" cells of the visual cortex encode the simplest components of the image, such as contrast, bars and edges.
Thereafter, the visual information is processed in a hierarchical manner through higher order visual cortical areas (V3, V4 and so on). Thus the "raw" data relating to the simple image components is combined; more features are added at each successive processing step, and the same information is encoded at increasingly larger scales. Thus, the initially crude representations of an image become more refined with each point in the hierarchy, until eventually an accurate reconstruction of the visual scene emerges into consciousness.
The researchers used functional magnetic resonance (fMRI) imaging to analyze the activity of the neurons involved in the earliest stages of visual processing, whilst their participants viewed a series of around 400 simple visual images, including geometric shapes and letters, during a single "training" condition. They then presented to the participants a series of completely new images, and combined the decoded fMRI signals from neurons in V1 and V2 with those from V3 and V4, all of which contain neurons that encode image contrast at multiple scales. By analyzing this activity using a specially developed algorithm, they were able to accurately predict the patterns of contrast in the novel images observed by the participants.
The major advance over similar neuroimaging studies carried out in the past is the ability to accurately reconstruct images that the particpants had not previously seen. This was possible because the activity recorded was that of neurons involved in the earliest stages of visual processing. These cells encode a small number of features, so their activity is limited to a small number of different states, and can be decoded with relative ease. Their combined activity can therefore encode a huge number of combinations of the same simple features, and so could be analyzed to predict and reconstruct the novel images, from a set of millions of candidate images.
As the film clip above shows, the reconstructed images are accurate but not too detailed - they consist of 10 x 10 patched reconstructions of the viewed images. However, as the algorithms and devices used for neuroimaging become more sophisticated, and as our knowledge of how the brain processes visual information advances, the ability to reconstruct images in this way will improve, and the reconstructed images will become more detailed.
The authors note that their new approach could be extended to reconstruct images that include other features such as colour, texture and motion. A similar approach could possibly be used to predict motor function from brain activity, and so could eventually lead to significant improvements in the capacity of neural prostheses and brain-computer interfaces. They even suggest that the method may one day be used to reconstruct hallucinations and dreams, which are not elicited by external stimuli, but which are also associated with activity in the visual cortex. Even if this was realized, it still would not constitute "mind-reading", because reconstructing visual images from brain activity is one thing, but deciphering the activity underlying a complex stream of consciousness is another.
Related:
- fMRI: What is it good for?
- The eye tells the brain when to plasticize
- An eye-opening view of visual development
Miyawaki, Y. et al (2008). Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image Decoders. Neuron 60: 915-929. DOI: 10.1016/j.neuron.2008.11.004
Brain & Behavior
Comments (12)
This is probably one of the most amazing things I've ever seen.
Posted by: Matt | December 12, 2008 12:38 PM