September 25, 2009
[ 
, Cognitive Neuroscience ]
A number of very smart people (and smart communities) seem like they might be under the impression that the "voodoo correlations" scandal in the neuroimaging community is somehow related to recent work by Bennett et al, who used fMRI to show task-related neural activity in a dead fish.
These two things have almost nothing to do with one another.
1) The Bennett work is, in the words of a friend, "a cute way to make a point" that every fMRI paper I've ever read has failed to explicitly acknowledge. The reason they've failed to acknowledge it is that it's standard to run equivalent statistical tests to the ones that Bennett et al recommend (of course, that doesn't keep a "sizable minority" of studies from failing to do so - in Bennett's estimation between 25-35%; I suppose I'm not reading those studies). Anyway, the Bennett point is simple: when you run a large number of statistical tests simultaneously, even on a random dataset, you're bound to find some percentage of tests that turn up "significant" just as a result of chance, and with some probability those significant results will randomly cluster together in 3D space. If one fails to correct the significance threshold for the large number of statistical tests performed, then you get unreliable results, even if you only consider those significant results that cluster in 3D space. (it's this latter point that makes the study interesting, worthwhile, and worthy of publication in a high profile journal, in my opinion). Regardless, the potential issue was already well known, perhaps explaining the difficulty the authors reportedly have in publishing their work. The problem they identified is why virtually everyone everywhere uses, and for a long time has used, both multiple comparisons correction and cluster-based correction when reporting fMRI results. As Bennett et al noted in their poster, such corrections are widely available in all the major neuroimaging analysis packages and are the default in one major package, FSL.
2) The "Voodoo correlations" work, on the other hand, is principally about the non-independence of multiple tests. Simply put, even when you do both types of the corrections discussed above in point #1, it's not OK to take the results of that analysis (clusters in 3D space) and then run additional analyses of the same clusters in the same dataset because the data is now biased by the first analysis.
An example from the original Vul paper should make this problem clear:
We (the authors of this paper) have identified a weather station whose temperature readings predict daily changes in the value of a specific set of stocks with a correlation of r=-0.87. For $50.00, we will provide the list of stocks to any interested reader. That way, you can buy the stocks every morning when the weather station posts a drop in temperature, and sell when the temperature goes up. Obviously, your potential profits here are enormous. But you may wonder: how did we find this correlation? The figure of -.87 was arrived at by separately computing the correlation between the readings of the weather station in Adak Island, Alaska, with each of the 3315 financial instruments available for the New York Stock Exchange (through the Mathematica function FinancialData) over the 10 days that the market was open between November 18th and December 3rd, 2008. We then averaged the correlation values of the stocks whose correlation exceeded a high threshold of our choosing, thus yielding the figure of -.87. Should you pay us for this investment strategy? Probably not: Of the 3,315 stocks assessed, some were sure to be correlated with the Adak Island temperature measurements simply by chance - and if we select just those (as our selection process would do), there was no doubt we would find a high average correlation. Thus, the final measure (the average correlation of a subset of stocks) was not independent of the selection criteria (how stocks were chosen): this, in essence, is the non-independence error.
To summarize, the dead fish study is a point about first-pass analysis, which almost every paper I've ever seen does correctly. The papers that don't always note that the result failed to pass multiple comparisons or cluster correction, and typicallly discuss those results with caution. On the other hand, "voodoo correlations" is a point about nonindependence in statistical tests. This has not always been done correctly, and has not always been reported clearly. Moreover it primarily affects only a subset of correlations between brain and behavior - and not the vast majority of work in fMRI, which has to do with task-brain relationships.
Posted by Chris Chatham at 10:10 AM • 4 Comments • 0 TrackBacks
[ , Cognitive Neuroscience ]
Something's afoot when a massively parallel and distributed system shows a bottleneck in performance. We've known that numerous bottlenecks plague cognition since the 1940's, but only with recent advances in neuroimaging have we been able to say whether these bottlenecks reflect the intrusion of executive operations (for managing goals and organizing cognitive processing) or a more passive "queueing" processes inherent to the selection of responses. Thanks to a number of very helpful (and interesting) reviews on a recent paper of mine, I've been pointed towards a fascinating study (by Jiang, Saxe and Kanwisher) suggesting that a queueing process might actually be to blame.
The particular "bottleneck" in cognitive performance investigated by Jiang et al is known as the "psychological refractory period," which can be observed with a very simple manipulation: subjects must simply give separate responses to two different stimuli, presented in rapid succession (<500ms). Reaction times to the second stimulus are longer than when the stimuli are presented at a more leisurely pace (>500ms), as though the selection of responses is subject to a bottleneck in information processing. This bottleneck, or refractory period, persists even when the tasks require responses in different modalities (i.e., verbal and manual), suggesting that a relatively "central" source is to blame for the refractory period.
Jiang et al present two possible explanations for this effect: according to the passive queueing account, the second task "is held in a passive queue until the bottleneck is freed" from processing the first task. Alternatively, the active monitoring account holds that a number of executive operations must occur: the tasks must be ordered for processing, the first task must be checked for completion while the second task is "halted" (inhibited?), and the second task must be triggered when the first has passed through the bottleneck. Jiang et al assume that this active monitoring involves "significantly increased executive functions" and as such that they should be more invoked when the refractory period is observed.
Under the reasonable assumption that executive functioning of this kind would engage the prefrontal cortex, Jiang et al used neuroimaging during a refractory period paradigm. Their results show that the only part of the prefrontal cortex to show increased activity during fast as opposed to short stimulus presentation was theright inferior frontal gyrus. From this, one might conclude that the right inferior frontal gyrus actually instantiates Jiang et al's "active monitoring" process. Maybe.
Instead, Jiang et al hold the right inferior frontal gyrus to a higher standard: they predict that a region involved in active monitoring should show also greater BOLD response according to individual differences in the refractory period. Specifically, the region responsible for the refractory period should be greatest among those individuals showing the largest refractory period.
In a way, the prediction seems natural (particularly if you're coming from an inhibitory perspective on this brain region). But what if the right inferior frontal gyrus actually reduces the psychological refractory period - say, if greater executive functioning actually helps you manage the "refractoriness"? After all, greater use of a putative executive function should probably improve task performance. Moreover, the refractoriness itself (i.e., the braking of motor output) might actually be the direct but involuntary result of a completely different, though connected region (say, the subthalamic nucleus). And the greater recruitment of "active monitoring" might help one detect the stimuli earlier, ultimately streamlining performance in the task. Of course, under this redefinition of monitoring, we'd expect the opposite effect: greater use of monitoring should be associated with a reduced bottleneck. Jiang et al observed this exact effect, but counter-intuitively interpreted it to mean that the activity in the right inferior frontal gyrus was not due to executive functions!
In a second experiment, they suggest that while the right inferior frontal gyrus does not implement an executive function, it may show greater activity due to effort. To manipulate effort, Jiang et al instructed subjects to employ a conservative strategy (i.e., "take your time") in some trials and a "daring" strategy in others (i.e., "respond as quickly as possible!").
A number of regions in the prefrontal cortex showed increased activation as a result of the effort manipulation (including the anterior cingulate, the pre-supplementary motor area, and bilateral middle frontal gyrus...) but the right inferior frontal gyrus was the only gray matter showing more activity both when the refractoriness was induced and when subjects were instructed to effortfully minimize it.
From this, Jiang et al conclude that "the effort to reduce the postponement is active" but suggest that the activity in the right inferior frontal gyrus is not related to executive functioning. I'd like to propose an alternative:
1) activity in the right inferior frontal gyrus is related to executive functions: it's under conscious control, it's engaged in a task-appropriate fashion, and this engagement predicts better task performance
2) activity in the right inferior frontal gyrus is related to an active monitoring process, and this is its executive function, but it's of a different kind than considered by Jiang et al: right inferior frontal activity doesn't reflect the monitoring of working memory (consistent with what Jiang et al seem to be arguing, and with previous work in primates (thanks, Reviewer #2 ;)), but does reflect active monitoring for the occurrence of stimuli in the environment. Detection of these stimuli provoke an involuntary "refractoriness" - perhaps by triggering activity in the subthalamic nucleus - but this itself is not the role of the right IFG (as reflected in the negative correlation between rIFG activity and the refractoriness observed here).
Of course, I don't expect this account of right IFG function to be commonly accepted until there's more neuroimaging data directly supporting it. Now, back to writing up that data of ours...
Posted by Chris Chatham at 9:32 AM • 3 Comments • 0 TrackBacks
September 2, 2009
[ Artificial Intelligence, Developmental Psychology, Link Posts ]
I've been busy writing up a new paper, and expect the reviews back on another soon, so ... sorry for the lack of posts. But this should be of interest:
The Dana Foundation has just posted an interview with Terrence Sejnowki about his recent Science paper, "Foundations for a New Science of Learning" (with coauthors Meltzoff, Kuhl & Movellan). Sejnowski is a kind of legendary figure in computational neuroscience, having founded the journal Neural Computation, developed the primary algorithm in independent components analysis (infomax), contrastive hebbian learning, and played an early role in linking the mathematical concept of "prediction error" to dopamine function.
One snippet from the interview:
Q: In what ways has the study of how children learn been used to solve engineering problems?
A: Children's brains are still developing and we need to understand how that helps them to learn. One example is imitation learning, which has been studied by Andrew Meltzoff, Ph.D., at the University of Washington in Seattle, who is trying to understand what makes children such effective learners. Babies and children are really good at imitation. Right out of the womb, babies can imitate facial expressions. If you stick out your tongue, a baby who can barely see will repeat your action. Children have fantastic abilities to mimic actions and behaviors. They learn a lot simply by observing and mimicking, and they will try to repeat not only the action itself - say, reaching out with the arm - but the purpose of the action - say, picking up a ball. This is something humans do much more effectively than any other animal.
Engineers, having seen that imitation is highly effective in humans, combined imitation learning with reinforcement learning to boost the performance of control systems. In apprenticeship learning, for example, a powerful computer tracks the actions of an expert human controlling a complex system, and then programs the reinforcement system to imitate and learn the very complex motor commands that the human makes. Engineers are now able to reproduce human skills that were previously thought beyond the reach of machines. For example, Andrew Ng, Ph.D., at Stanford has used apprenticeship learning with reinforcement to automatically control helicopters that do stunts like flying upside down.
Read more of the interview here.
Posted by Chris Chatham at 4:05 PM • 1 Comments • 0 TrackBacks
July 28, 2009
[ , Cognitive Neuroscience ]
Children assigned to chew sugar-free gum purportedly score 3% higher on standardized tests of math skills (as widely reported in the press). But is this just one of the 5% of all possible untrue hypotheses statistically guaranteed to have some significant result in its favor (in fact, it's worse than that)? Is the effect due to some other aspect of gum chewing (as Michael Posner asks)? Or might there be a real effect here of chewing (i.e., "mastication"), and if so, how can you use it to your maximum advantage?
Read on »
Posted by Chris Chatham at 9:41 AM • 4 Comments • 0 TrackBacks
July 20, 2009
[ Cognitive Neuroscience ]
The UCLA Neuroimaging Summer Education Program starts today at 8:30 am Pacific. Standard Time - and is going to be streaming live at this address (video embedded below). The schedule is quite impressive, including talks from Rick Buxton, Mark Cohen, Russ Poldrack, Vince Calhoun, and Jose Hanson among others. Topics include everything from causal modeling to network analysis and multivariate pattern recognition.
Monday, July 20
08:30 Intro & overview (Russ Poldrack & Mark Cohen)
09:30 MRI acquisition: basics (Mark Cohen)
11:00 Ethical issues in cognitive neuroscience (Russ Poldrack)
12:00 lunch
13:15 MRI acquisition: advanced (Mark Cohen)
15:00 Laboratory: Neuroanatomy (Susan Bookheimer)
Tuesday July 21
08:30 Spikes and BOLD: Can they get along? (Dario Ringach, UCLA)
09:30 Hemodynamics and fMRI signals (Rick Buxton, UCSD)
11:00 Neural basis of imaging signals (Rick Buxton, UCSD)
12:00 lunch
13:15 Basic experimental design (Susan Bookheimer)
14:30 Advanced experimental design (Russ Poldrack)
15:30 Lab: Introduction to Mac and FSL
Wednesday July 22
08:30 Preprocessing: Image registration and motion correction (Russ Poldrack)
09:30 Preprocessing: EPI unwarping, intra/intersubject registration (Russ Poldrack)
11:00 Statistics I (Jeanette Mumford, UCLA)
12:00 lunch
13:15 Statistics II (Jeanette Mumford, UCLA)
14:15 Lab: Statistics by hand in MATLAB
15:30 Lab: FSL Preprocessing
Thursday, July 23
08:30 First-level fMRI modeling (Russ Poldrack & Jeanette Mumford, UCLA)
09:30 First-level modeling, continued (Russ Poldrack & Jeanette Mumford, UCLA)
11:00 Group fMRI modeling (Jeanette Mumford, UCLA)
12:00 lunch
13:15 EEG/MEG (Charan Ranganath, UC Davis)
14:15 Lab: First-level statistical analysis
Friday July 24
Software package comparisons (Russ Poldrack)
08:30 Multiple testing problems (Jeanette Mumford, UCLA)
09:30 Data quality control (Mark Cohen)
11:00 Q&A session
12:00 lunch
13:15
14:15 Lab: Group modeling and Multiple testing
Saturday July 25
all day EEG lab, or MRI physics lab
WEEK 2
Monday July 27
08:30 Advanced fMRI modeling: percent change and power analysis (Jeanette Mumford)
09:30 TBD
11:00 fMRI Design Optimization (Tom Liu, UCSD)
12:00 lunch
13:15 Lab: working with datasets, and power analysis Tuesday July 28
08:30 Improving reliability (Gary Glover, Stanford)
09:30 Reporting fMRI data (Russ Poldrack)
11:00 Connectivity analysis (Russ Poldrack)
12:00 lunch
13:15 Computational anatomy (David Shattuck, UCLA)
14:15 Diffusion tensor imaging (Nathan Hageman, UCLA)
15:30 Lab: working with datasets
17:30 EEG/fMRI demo (Brain Mapping Center)
Wednesday July 29
08:30 Network analysis (Steve Petersen, Washington University)
09:30 Dynamic causal modeling (Marta Garrido, UCLA)
11:00 ICA (Vince Calhoun, University of New Mexico)
12:00 lunch
13:15 Graphical causal modeling (Clark Glymour, CMU)
14:15 Lab: Connectivity exercises
18:30 Group Dinner (Napa Valley Grill)
Thursday July 30
08:30 Pattern classification (Steve Hanson, Rutgers)
11:00 Pattern-information and representational similarity analysis (Niko Kriegeskorte, NIMH)
12:00 lunch
13:15 Avoiding statistical circularities in brain-activity analysis (Niko Kriegeskorte, NIMH)
14:15 Lab: working with datasets
Friday July 31
08:30 Psychophysics for fMRI (Don Kalar, UCLA)
09:30 Setting up an imaging lab (Mark Cohen)
11:00 Imaging difficult populations (Susan Bookheimer)
12:00 lunch
13:15 Q&A session
15:30 Presentation of results from data analysis projects
Posted by Chris Chatham at 10:36 AM • 1 Comments • 0 TrackBacks
June 30, 2009
[ , Cognitive Neuroscience, Computational Modeling ]
How many times did Pavlov ring the bell before his dogs' meals until the dogs began to salivate? Surely, the number of experiences must make a difference, as anyone who's trained a dog would attest. As described in a brilliant article by C.R. Gallistel (in Psych. Review; preprint here), this has been thought so self-evident "as to not require experimental demonstration" - yet information theoretic analysis suggest the idea is incorrect, at least when the time from the bell to the food is constant. More problematic is the fact that the whole issue is ill-formed for experimental verification: technically speaking, one can never actually accept the (null) hypothesis that some experimental manipulation has no effect. But as Gallistel says, while "conventional statistical analysis cannot support [the null hypothesis]; Bayesian analysis can."
Read on »
Posted by Chris Chatham at 11:00 AM • 13 Comments • 0 TrackBacks
June 29, 2009
[ , Cognitive Neuroscience ]
Don't think of a white bear. Doesn't work so well, does it? Yet under some circumstances, people appear to be able to do precisely this: as described last week, young adults are thought (by some) to actually suppress the neural activity related to to-be-ignored stimuli, and even delay the peak of this neural activity, relative to a situation in which stimuli are to be just passively-viewed. In a follow-up paper at Nature Neuroscience, Gazzaley et al report that cognitively-intact older adults (60-77 years of age) show an impairment in this ability, without concomitant impairments in the enhancement effects normally observed to to-be-attended stimuli.
Read on »
Posted by Chris Chatham at 12:35 PM • 1 Comments • 0 TrackBacks
June 26, 2009
[ , Cognitive Neuroscience ]
The cognitive neurosciences have had high frequency oscillations on the brain: so called "gamma-waves", as recorded on the scalp, have been linked to working memory processes (via their interaction with slower "theta waves"), to cognitive insight, and even to consciousness. (I think there's an unwritten rule that whenever someone mentions consciousness, they'll be made to look foolish by a subsequent paper). In the midst of these "inflationary accounts" of the role of gamma oscillations, a debate has emerged: could these oscillations (at least, as recorded on the scalp) reflect simply the movements of the eyes, as now appears to largely be the case? More interestingly, is that necessarily incompatible with the more lofty interpretations of gamma? And what is the relationship between these movement-related artifacts on scalp recordings, and the well-established importance of gamma waves as recorded on or inside the brain?
(UPDATE: first para edited to emphasize scalp- vs. i-EEG distinction; thanks Alex & farraway).
Read on »
Posted by Chris Chatham at 11:28 AM • 7 Comments • 0 TrackBacks
