In the eyes, of anti-vaccine advocates, vaccines bear the brunt of blame for a variety of conditions, including autism, asthma, neurodevelopmental disorders, autoimmune disorders and a wide variety others. Often this link is based on retrospective data, in which parents or patients recall and self-report how many vaccines they've had and which ones. This self-recall is then correlated with the health condition under study, and sometimes correlations are found. However, it's long been known that self-reporting has a tendency to be unreliable, with a tendency to conflate incidents that may or may not be related. That's why a study in which a patient's recall is cross-referenced as much as possible with objective and concrete records, such as the study participants' medical records. Human memory is very fallible, but we human beings often have a very hard time realizing that, admitting it, or accepting it. Each of us tends to think that his or her memory is far above average, at least when it comes to issues like this. Combine the fallibility of human memory with the natural human tendency to find correlations and relations where none exist, and it's not too surprising that the myth that vaccines cause autism, for instance, is so persistent, all science refuting notwithstanding.
Published just this week in the British Medical Journal is a study that demonstrates how relying on self-reporting can result in investigators finding correlations that are probably not there, particularly when it comes to relating vaccines to health problems. What's interesting about this study is that it's not about vaccines and autism. Rather it looks at the relationship between vaccines given to British soldiers serving in Iraq between 2004 and 2006. That no correlation between vaccination status (specifically multiple vaccines) and various health complaints was not particularly surprising, but there was a twist. The authors analyzed the data both using self-reported data and actual medical records about vaccination:
LONDON (Reuters) - A British study has found no link between illness among British troops sent to Iraq and multiple vaccinations. Instead, it says the troops tend to blame poor health on multiple jabs even when they did not have them.Several studies have linked ill health in forces in Iraq to multiple vaccinations, but Dominic Murphy, a psychologist at King's College London, said these relied on troops' own recollections, and should be re-evaluated.
"Multiple vaccinations given to personnel in the UK armed forces in preparation for deployment to Iraq are not associated with adverse health consequences when vaccinations are recorded objectively from medical records," said the study carried out by Murphy's team, published in Tuesday's British Medical Journal.
The twist showed that soldiers tended to blame poor health on multiple vaccines, whether they in fact received multiple vaccines or not. Here's how the study was set up. The investigators examined a cross section of the British military serving in Iraq between June 2004 and March 2006. Soldiers who had deployed to Iraq either during the 2003 invasion or subsequently were selected for the study, and they completed detailed questionnaires. One of the questions on the questionnaire was: What was the maximum number of any vaccines that you received in one day in preparation for your deployment?" Examples of the specific vaccinations administered to service personnel included vaccines against anthrax, tetanus, typhoid, and yellow fever, all of which, with the exception of the anthrax vaccination, were administered routinely. The exception was the anthrax vaccination, which was offered under a separate program that included information and written consent.
Other questions on the questionnaire were designed to elicit health information by asking about a wide variety of health measures, including a 13 item fatigue scale, a general health questionnaire (GHQ-12), a 53 item physical symptom checklist, self perception of health with a single item from the SF-36, and symptoms of post-traumatic stress disorder measured with a 17 item checklist (PCL-C). In addition, a 10% subset of soldiers was randomly selected for verification of vaccination records. Investigators actually visited military medical facilities and collected data on vaccination, recording the maximum number of vaccinations received on a single day before deployment. To avoid missing data, both deployment medical records and standard medical records were examined.
When investigators looked for agreement between reported vaccinations and actual vaccination status, they found pretty poor agreement. Then, they did their analysis two ways. First they looked for correlations between self-reported vaccination status and a number of health complaints. They found several, with veterans remembering receiving more than one vaccine in a day being more likely to report feelings of stress, fatigue, and other health problems than those who recalled never having more than one vaccine in a day. They then repeated the analysis using objectively determined records of vaccination status and found--well, let the investigators say it:
We repeated the analysis using the number of vaccinations recorded from participants' medical records (table 4). Without exception, we found no health differences between individuals whose medical records indicated they had received no more than one and those who had received two or more vaccinations on a single day.
The conclusion is that prior reported associations between vaccination and health problems in British veterans are probably due to recall bias. The study is not without its weaknesses, perhaps the most troublesome of which is that the investigators only used one question to assess the subjects' vaccination status. Also, followup has only been less than four years, which, if there were indeed a link between vaccination and adverse health outcomes in the form of chronic disease, may not sufficient time. Even so, however, this study is pretty strong evidence for an effect of recall bias in British soldiers on prior studies.
The phenomenon of recall bias can occur when there is intentional or unintentional differential recall of information about an exposure or a health outcome of an association by subjects in one group compared to subjects in another group under study. This differential recall can lead to a systematic misclassification of the study subjects with regards to the exposure or outcome variable. This bias can result in a study either finding an association between an exposure and a health outcome when there is none or in failing to find an association that does exist. Indeed, it's a problem for studies of sun exposure and melanoma, abortion and breast cancer (where recall bias causes a real problem), and, of course, vaccines and a variety of health problems. Unfortunately, for some study designs (sun exposure and melanoma, for example) there is no satisfactory way to get at the exposure data without relying on self-reporting.
Recall bias tends to be the worst when either the exposure or health outcome meet one or more of these conditions:
- The event under study is highly significant in the life of the subject (such as cancer).
- The patient has a preconception that the exposure and the health outcome are related.
- The media reports an association between the exposure and the health outcome.
- The exposure or behavior are socially undesirable or illegal.
Number four, of course, would tend to decrease an association, as people would tend not to recall or report socially undesirable or illegal behaviors. The other three tend very strongly to link a risk factor and a health outcome when there may in fact be no link. Indeed, #1, 2, and 3 are highly applicable to any retrospective study in which self-reporting of vaccinations received by babies by their parents are linked to autism rates. Having a child with autism is a highly significant event in the life of the parent; many parents now have a preconception that vaccines cause autism; and the media, fueled by useful idiots like Jenny McCarthy, is constantly reporting scientifically dubious "associations" between vaccines and autism. Indeed, Generation Rescue's pseudoscientific telephone survey was nothing if not an exercise in harnessing recall bias to the antivaccinationist cause.
Recall bias is a real potential problem in any retrospective study design that includes self-reporting of exposures or factors that the investigator is testing as a risk factor for the disease in question. It can, however be controlled for. What's critical in evaluating any retrospective study is to look at (1) whether the investigators admitted the possibility of recall bias and (2) how much effort they took to try to minimize it. The best studies, of course, rely as much as possible on objective data and not patient recall. The medical record and data sources used may have their own problems, but nowhere near the level of problems that recall bias can cause.
REFERENCE:
Murphy, D., Hotopf, M., Wessely, S. (2008). Multiple vaccinations, health, and recall bias within UK armed forces deployed to Iraq: cohort study. BMJ, 337(jun30 1), a220-a220. DOI: 10.1136/bmj.a220
1) Thanks for the description of a very nicely done study, and a good primer on recall bias.
2) I think it is likely that the older British studies you described are less well-controlled, thus it's probably less reliable. That said, you make a serious logical error in your statement here:
"The conclusion is that prior reported associations between vaccination and health problems in British veterans are probably due to recall bias. ".
You yourself point out that recall bias is more likely once "The patient has a preconception that the exposure and the health outcome are related.". Now, who's more likely to have a preconception that multiple vaccine exposure and stress/fatigue/ect. are linked- the British veterans in the original studies, or the soldiers in this study (who could have been aware of the results of the first studies)?
Just because this study provides excellent evidence for recall bias in this population, does not imply that recall bias was the cause of the effect seen in the other populations.
I agree with your conclusion, but that line of argument wasn't quite right.
Good article!
On the topic of anitvaccine freaks, check out the Age Of Autism website and look what they have to say about Bill Gates. Whatever you may feel about Microsoft, Bill Gates has to be respected (not only as an entrepreneur, but as the richest man in the world at one time). Now, his focus is on his foundation which aims to wipe out infectious diseases in the third world. But these people are such lunatics, they have co-opted this fantasy that he is some how in bed with big pharma to make more money. One person actually thinks that it his intention to give vaccines to make these people 'dumber' so that he can maintain intellectual superiority. Wow! Way to take a great man who aspires to help so many others and twist it into a sad, sad little world.
If you are interested in reading the article, it's the one titled Bill Gates and the moral imperative...the one with the picture of the gates of hell. Nice folks at AoA.
How does recall bias affect self-reports of abortions and breast cancers? I can understand not being able to accurately estimate sun exposure or forgetting exactly how many vaccines I was given on a single day. But abortions are major, singular events, and it seems unlikely that someone would misunderstand or forget how many she had.
The reason is #4 on my list: Abortions are considered socially unacceptable in many quarters and thus women are less likely to report them. Indeed, it's been shown that healthy women are less likely to reveal that they have had an abortion to an interviewer or to report one in a study, while women with cancer are more likely to do so, probably because they are searching for causes of their cancer and/or because of the media reporting that breast cancer may be linked to abortion. This causes differential reporting, where women with cancer are more likely to report that they had an abortion.
In a case like this, perhaps a better term to use would be "reporting bias," but it's all lumped together under "recall bias."
Ever have a traffic accident ? years later you remember how it wasn't really your fault -- the other driver just did something wrong!
Human memory is rarely as good as we would like to believe.
That works both ways. Years back, I was proceeding slowly through an intersection on green (right lane clear, left two blocked by traffic stopped for the red that had ended as I watched) when I was struck by someone turning right on red.
For days afterwards, I lost sleep replaying it from an "how could I have prevented it?" perspective -- despite the fact that there was, simply, no freaking way to prevent some idiot from making an illegal turn into you. In slightly different circumstances, I can easily see someone remembering it as their fault.
It's sad, what people blame. Personally, I always feel *great* after having had my shots, because I know I've taken a proactive step to safeguard my health!
Recall bias cuts both ways. An awful lot of psych studies and assessment tools are based on subjective recall, often of events that occurred decades previously, including in childhood.
Also, in a least some conditions, the fatigue scale used in the study (Chalder, but also other fatigues scales like the Checklist Individual Strength and Krupp Fatigue Severity Scale) have a tendency to produce a high proportion of patients with maximum scores, making them unreliable measures.
I am not a fan of check list assessment of this kind, it has a lot of problems and limitations, and often seems to fail in really capturing the patient experience. Its value, such as it is, is probably more one of convenience to clinicians than anything else.
BTW, the Chalder scale is a 14 item scale, not 13.
All that said, the numbers in this study do give it some statistical power.