When analyzing data, understanding the limitations of your data is critical. One of the things we need to understand is significance: how strong does an effect have to be to considered not a result of random chance. Typically, we assume that if an effect has a five percent or less probability of occurring due to random chance, then it is "significant." But significance becomes very problematic when making many simultaneous assessments. If we make one hundred assessments (e.g., comparisons) and not a single one is actually different (assume that the Omniscience of the Mad Biologist is operating here), then, on average, five of the 100 assessments will be "significant" even though there is no real phenomenon underlying the supposed significance.
I recently finished Cordelia Fine's Delusions of Gender: How Our Minds, Society, and Neurosexism Create Difference. An area of research that's in vogue (perhaps unjustifiably so) is seeing if male and female brain activation responds differently to certain stimuli, such as an emotionally disturbing picture. Then the results are ridiculously overinterpreted, and we conclude that boobies mean girls can't do math. Or something. So someone wanted to test just how reliable assessing significance of these scans (from Fine):
Could the sex differences in brain activation be spurious? When looking for changes in blood flow between two conditions, researchers search in thousands of tiny sections of the brain (called voxels), and many researchers are now arguing that the threshold commonly set for declaring that a difference is "significant" just isn't high enough.
And here's the Bestest Control Experiment EVAH! (italics mine):
To illustrate this point, some researchers recently scanned an Atlantic salmon while showing it emotionally charged photographs. The salmon-which, by the way, "was not alive at the time of scanning"-was "asked to determine what emotion the individual in the photo must have been experiencing." Using standard statistical procedures, they significant brain activity in one small region of the dead fish's brain while it performed the empathizing task, compared with brain activity during "rest." The researchers conclude not that this particular region of the brain is involved in postmortem piscine empathizing, but that the kind of statistical thresholds commonly used in neuroimaging studies (including Witelson's emotion-matching study) are inadequate because they allow too many spurious results through the net.
But did the salmon speak English? DUH! BAD EXPERIMENT.
This is awesome.
- Log in to post comments
It appears your cited article has a bad link.
OMG WIN!!!!
The link incorporates a user ID, so it won't work for anyoneelse, sadly. But if you go to the www.sciencedirect.com site, it is possible to search for author Bennett, journal NeuroImage, vol 47, and find it anyway. Whether or not you can access the full thing unfortunately depends on whether or not you have a subscription to the ScienceDirect articles, I think.
http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf
Found a better link:
http://www.jsur.org/ar/jsur_ben102010.pdf
Thanks. I liked the idea of a dead fish resting. ;)
As a Bayesian, I am not too impressed with Fisherian hypothesis testing. I do not think that it is useless, but that it should be interpreted differently. If we believed that empathy was an activity of specific parts of a dead salmon's brain, then the experiment indicates which parts they might be. Establishing our initial assumption is a separate question.
Often Fisherian hypothesis testing is applied to "null" hypotheses that no one believes, anyway. In such a case, no harm, no foul. :) I do not think that that applies to hypotheses about men's and women's brains, however.
I want to be the grad student that asked the salmon questions. This should end up on YouTube.
Wait, so was the salmon resting, or perhaps pining for the fjords?
Neither, it was an EX-salmon!
h/t Monty Python for those in the slow lane.
It shouldn't be difficult to design MRI tests for (human, living) emotion and cognition where the testers don't know the gender of the subjects.
Have the testers in one room, giving instructions by telephone or internet to an assistant in the MRI room. The assistant does the tests and transmits the results, without mentioning the gender of the person they're scanning.
The tester on the other side of the hospital interprets the data 'at one remove' from the subject, who's identified onlt by a number. Their own biases and expectations of gender are bypassed.
Just a thought.
But was it a male salmon or a female salmon? That really could account for all the test results. And did they cook and eat the test subject after the experiment was done? Just how thorough was this testing?
As Withnail put it: "Don't threaten me with a dead fish!"
Brilliant. There ought to be whole departments of people slinking around with bags over their heads. 'Course, this will be ignored with a sniff of hauture.