Effect Measure

My sciblings at Scienceblogs have done a pretty thorough fisking of the Andrew Wakefield affair.To recap breifly, a paper by Wakefield and others in The Lancet in 1998 raised an alarm that the widely used measles-mumps-rubella (MMR) vaccine was the cause of some cases of childhood autism and a chronic inflammatory bowel disease. The incriminated agent was alleged to be measles virus contained in the vaccine (MMR has never contained mercury preservative). The impact was dramatic and this issue became a powerful engine propelling the anti-vaccine movement. The result has been a real public health crisis as falling vaccination rates in the UK and elsewhere have allowed measles and mumps to make a comeback after being almost eradicated. While all three of these diseases are usually relatively mild childhood maladies, some case are serious or even fatal. They are also totally preventable.

So over the weekend when an investigative report by The Times in London was published that seemed to show fairly conclusively Wakefield had doctored the data to have in come out the way he wanted, it was a big deal. This explains why no one has been able to replicate his findings: in fact there was no demonstrable relationship between the autism seen in his 12 child case series and the vaccination itself. Since others here at Scienceblogs have covered this extensively I want to discuss another aspect of it, the role (or lack of it) of the respected medical journal, The Lancet, in publishing what is apparently bogus if not fraudulent research. Before I do so, you need to know the nature of the doctored evidence.

The Times has published a fairly extensive catalog of each case but I will only present two here, so you can get the idea of what was involved:

The first, in the Lancet tables, concerned the first child in the paper: Child One, from Cottesmore, Leicestershire. He was 3½ years old and the son of an air force pilot. In November 1995, his parents had been devastated after receiving a diagnosis of autism.

?Mr and Mrs [One]?s most recent concern is that the MMR vaccination given to their son may be responsible,? their GP told the hospital in a letter.

In the paper this claim would be adopted, with Wakefield and his team reporting that Child One?s parents said ?behavioural symptoms? started ?one week? after he received the MMR.

The boy?s medical records reveal a subtly different story, one familiar to mothers and fathers of autistic children. At the age of 9½ months, 10 weeks before his jab, his mother had become worried that he did not hear properly: the classic first symptom presented by sufferers of autism.

Child One was among the eight reported with the apparent sudden onset of the condition. So was the next child to be admitted.

This was Child Two, an eight-year-old boy from Peter-borough, Cambridgeshire, diagnosed with regressive autism, which, according to the Lancet paper, started ?two weeks? after his jab.

However, this child?s medical records, backed by numerous specialist assessments, said his problems began three to five months later.

The difference between 14 days and a few months is significant, according to experts. Autism usually reveals itself in the second year of life, when the vaccine is routinely given. If there was no sudden onset after the MMR injection, as claimed for the ?syndrome?, the condition could be ascribed to a conventional pattern. (TimesOnline)

The dreary record goes on, through all twelve cases, none of which supported Wakefield’s conclusions. So why didn’t the “peer reviewed” scientific journal, The Lancet, catch it before it was published? First, a disclosure. I quite like The Lancet and admire its tendency to take courageous stands on controversial issues. I have also been a peer reviewer for them, on multiple occasions. Second, I am also co-Editor in Chief of a peer reviewed scientific journal, so I have both sympathy and understanding of the position they are in, but perhaps more of a tendency to defend them because of that. You’ll have to decide if what I say makes sense.

A word about peer review. This is the process whereby journal editors send manuscripts to experts in the field for their evaluation of scientific soundness. Based on the comments, editors then make a decision as to whether to publish or not. That decision may or may not be the same as the reviewers’. There are many considerations whether to publish something or not (is it of sufficient interest to the readership or does it make enough of a contribution to the field, for example). In general, however, depend on reviewers for the science. Most journals do closed, anonymous reviews. This means that the authors don’t know who the reviewers are and the reviews are not provided to the readers. Often the names of the authors are also kept from the reviewers so as not to prejudice their judgment. Some journals (like the one I edit) practice open review, meaning that reviewers’ names are known to the authors (and vice versa) and that the reviews themselves are available to readers when the paper is published. In the case of the Wakefield paper we don’t know the names of the reviewers or what they said.

Reviewing papers is a chore and if done conscientiously, hard work. You get paid nothing for it and you do it on a deadline. All you have is the submitted manuscript and any other supplementary materials that may be submitted along with it. In 1998 supplementary data files were uncommon because journals were mostly read in hard copy via the journal. Today it is common for readers to have access to supplementary files via the journal’s website, and in my experience as an editor many reviewers consult these files. But even if they had been available in 1998 it wouldn’t have made any difference. Wakefield provided the case summaries (which we now know were doctored) and a reviewer would not have had access to or had the time to look at the original medical records. The same is true for the journal. Accurate representation of raw data is taken on trust. I don’t think The Lancet can be taken to task for not catching this. This kind of scientific misconduct is only found after the fact.

Still, one might wonder at The Lancet’s penchant for controversial topics. Science journals are not just about science. They compete with each other for readership, public recognition and prestige.It used to be that in the UK publishing world The British Medical Journal (BMJ) was the dull, conservative journal and The Lancet was more “out there.” As one of the leading medical journals in the world, it perhaps could afford to be. Now they seemed to have switched places. BMJ is tending to push the envelope with more aggressive contrarian articles and The Lancet has gotten duller. In general I like a bit of daring, but there is a risk, too, and in this case the result was bad for public health.

It is still an uphill battle to repair the zombie undead idea that MMR causes autism. I have quite a lot of sympathy for the parents of these children, sympathy that is personal in nature. They want to know “why their child.” I want to know why their child, too. The answer to “why?” isn’t MMR vaccine, however, and while having an answer might satisfy a need, when the answer is wrong but keeps getting spread, it can cause great harm. On the other hand, I can easily imagine a circumstance where the answer to the question,”why did my child get a life threatening pneumonia?” could easily be: “Andrew Wakefield.”

Comments

  1. #1 Sam C
    February 12, 2009

    Review is fine for good science and for research where errors will not have far-reaching impact (so the normal process of correction and extension by future workers is appropriate).

    But cases like this need audit. An audit can not always detect deliberate fraud (just as in financial auditing), but it might pick up errors of protocol (like the O’Leary lab’s inadequate controls in their DNA/RNA work) or substandard or imperfect work (inappropriate statistical techniques, equipment whose limitations are not understood, results transferred incorrectly, etc.).

    Engineering organisations use ISO standard QA systems, but these only work if applied correctly and conscientiously.

    Perhaps any grant award should require that some percentage of the award be allocated to an independent audit of techniques, results and conclusions?

    And the role of the extra authors on multi-author papers needs to be clarified – what are all these people doing if they take no interest in the correctness of the results?

  2. #2 Orac
    February 12, 2009

    Actually, if there’s one thing I’ve learned, it’s that the “cutting edge” journals or journals that like to take on controversial topics are the ones most likely to get it wrong. However, usually they get it wrong because the newest, most cutting edge, most controversial findings are far more tentative and often are not verified by later studies.

    I suggest reading the actual original Wakefield paper as an exercise, though. The indications that it’s pretty thin gruel are all there. However, you are right in that peer reviewers cannot have been expected to have uncovered the fact that (1) Andrew Wakefield had been paid over £400,000 by a trial lawyer interested in suing vaccine companies for MMR-induced “damage” in the form of autism; (2) that the 12 children in the original report were not consecutive referrals but had all been referred by a network of parents convinced that MMR had caused their children’s autism; (3) that Wakefield had falsified data. Indeed, it took a General Medical Council investigation and Brian Deer’s doggedness over nearly 11 years to uncover that.

    I had the privilege of seeing Brian Deer speak the other day. What he said was that the data presented in the paper looked even to him “too good to be true.” He was right. It did. Peer review could potentially have picked that up.

  3. #3 Anne
    February 12, 2009

    Wait a minute; Wakefield never said MMR causes autism. Not then, not now. You can find his rebuttal to Deer’s piece in the Times all over the internet, and it’s worth reading– and by the way, in the US, the complainant in a case like this (Deer) would never get away with then approaching a major paper and offering to write up a prejudicial piece about it. Especially using illegally obtained documents from the proceedings. Poor Deer seems to be suffering from a colossal obsession.

  4. #4 D. C. Sessions
    February 12, 2009

    Engineering organisations use ISO standard QA systems, but these only work if applied correctly and conscientiously.

    Which is what an ISO audit service checks (having been through our regular audit last week.)

    Perhaps granting organizations might consider adding points to proposals coming from researchers subject to audited ISO procedural formality. IMHO the procedural rigor of some research institutions could be improved, based on reports from my children — whom I taught a bit myself on matters such as notebook maintenance.

  5. #5 AF
    February 12, 2009

    Anne said:

    by the way, in the US, the complainant in a case like this (Deer) would never get away with then approaching a major paper and offering to write up a prejudicial piece about it. Especially using illegally obtained documents from the proceedings.

    I find it very striking that so many who obviously wish to defend Wakefield cannot do so by denying the facts that constitute the case against him, but can only accuse Brian Deer of obtaining those facts or presenting those facts in some manner that is incorrect. It would be striking even if they didn’t also frequently express mistaken ideas about what Brian Deer’s position is, or about the laws relevant to the publication of documents that were not voluntarily released. (Hint: might want to look into a little bit of history called the Pentagon papers.)

  6. #6 iayork
    February 12, 2009

    Sam C: “Perhaps any grant award should require that some percentage of the award be allocated to an independent audit of techniques, results and conclusions?”

    Sam, this is an interesting idea I haven’t heard floated before. I think it’s not doable as a portion of every grant, but I wonder if there could be a separate fund set aside specifically for audits. I’m not sure how it could work, and it would be a real problem to get it balanced properly, though — if there were a standing committee or organization, I could see it getting bogged down in bureaucracy, dinging every paper they come across for trivial procedural errors (“Patient #214 signed the form but failed to initial the 17th page”) so that genuine problems would be hidden anyway.

    “And the role of the extra authors on multi-author papers needs to be clarified – what are all these people doing if they take no interest in the correctness of the results?”

    Many, if not most, journals have already started doing this. Either in the published version, or behind the scenes, there’s a paper that at least nominally breaks down the contributions (“JH designed experiments and analyzed data, BP performed experiments, AM clipped the gerbils’ toenails”).

  7. #7 Tom DVM
    February 12, 2009

    Then the question becomes…what is causing the epidemic of Autism, Multiple Sclerosis, Food Allergies,asthma, CJD,learning disabilities etc. etc. etc.?

    If we can’t answer this question, we sure aren’t going to solve many other of our current problems.

  8. #8 kevin
    February 12, 2009

    Tom DVM — those have always been the questions. The anti-vaccine crowd are, and have been for a long time, done nothing but obstruct that investigation. Every dollar and every minute spent dealing with the anti-vaccine idea could have been better spent on other leads.

  9. #9 Orac
    February 12, 2009

    Wait a minute; Wakefield never said MMR causes autism.

    Poor Anne appears to be parroting antivaccine talking points. What Wakefield said was that MMR vaccination is associated with a syndrome that includes regressive autism and enterocolitis and tried to link it with the presence of the strain of attenuated measles virus in the MMR. There was a lot of “wink-wink, nudge, nudge” in the actual scientific paper, but his public statements made it very clear that he was promoting the idea that vaccines cause autism.

    As for the antivax claim that Brian Deer made the complaint to the GMC and then reported on the hearings, it’s a massive distortion bordering on a lie. Do check out Brian Deer’s response and the list of links at the end of my post:

    http://scienceblogs.com/insolence/2009/02/keith_olbermann_played_for_a_fool.php

    Make no mistake, Brian Deer’s reporting was instrumental in getting British health authorities interested in investigating Wakefield, but to claim that Deer made the complaint in order to have something more salacious to report on is a distortion bordering on a lie.

  10. #10 revere
    February 12, 2009

    Tom: I don’t understand the logic. Leaving aside the claim of an “epidemic” in those diseases we don’t understand (e.g., I think we know what caused the increased in vCJD and for the others, we cannot easily estimate incidence for many reasons, including questions of outcome definition, reporting, etc.), the idea that if we cannot understand these very hard questions we can’t understand anything is like saying, If we can’t produce a unified field theory uniting relativity and quantum mechanics we can’t produce any science.

  11. #11 qetzal
    February 12, 2009

    Coming back to revere’s original question, I think the best that can be expected of a journal in a case like this is to 1) make sure any ‘suspicious’ paper gets higher scrutiny, e.g. extra, more senior, and/or more diligent reviewers; and more importantly 2) require all authors to endorse that they stand behind the research described, and that they are not aware of anything that might suggest negligence or misconduct relating to the research.

    Does The Lancet require such statements now? Did they in 1998? That seems to me a key question, especially given the allegations that Wakefield improperly reported the timing of onset of symptoms versus vaccination, and that he reported abnormal findings in colon biopsies that were not reported by the original hospital pathologists (without disclosing that the paper’s findings conflicted with the original pathology reports).

    The original paper says that clinical histories were mostly taken by the senior clinical investigator, J A Walker-Smith, and that histopathologies were carried out by A Anthony, A P Dhillon, and S E Davies. Were those co-authors unaware of the above conflicts when the paper was published? Orac’s post mentioned that “most of the authors of the 1998 Lancet paper later withdrew their names from it.” Didn’t any of those co-authors have doubts about the paper’s claims before publication?

    Assuming all authors saw approved the final manuscript before publication, it’s certainly possible that no one but Wakefield had any reason to suspect problems with the data, but I think that’s pretty unlikely. More likely, at least some co-authors did have concerns, but chose not to act on those concerns, for fear of the professional conflict it might generate.

    There is a limit to the ability of any journal or reviewer to detect well-executed fraud. Even a detailed audit of all supposedly raw data wouldn’t always be sufficient. The best that can be done, IMO, is to incentivize co-authors to speak up if they have significant concerns. They are in the best position to know if misconduct is a reasonable possibility.

  12. #12 qetzal
    February 12, 2009

    Adding to my previous thoughts:

    If The Lancet did require such an author statement in the Wakefield paper, and if all co-authors did endorse the paper before publication, then they should have to answer why they had no concerns before hand, especially given that they (apparently) withdrew their names later.

  13. #13 catgirl
    February 12, 2009

    what is causing the epidemic of Autism, Multiple Sclerosis, Food Allergies,asthma, CJD,learning disabilities etc. etc. etc.?

    There are already various causes for all of these things which are being investigated. Many of these diseases involve several different causes, so it’s not as straightforward. However, the rise in asthma is at least partially caused by pollution. The hypothesis for food allergies is that children are not being exposed to the specific foods and not being exposed to enough germs in general, which prevents them from developing a healthy immune system.

    CJD, if you are referring to Creutzfeldt–Jakob disease, is caused by infectious proteins called prions. CJD is mostly genetic but can be spread from person to person by tissue transplants. It can also arise spontaneously by a specific gene mutation. If you are thinking of vCJD, that can be spread by eating infected meat.

    Being born prematurely increases a child’s risk of having learning disabilities. Because we now have the ability to save many premature babies, this is one contributing factor to the rise in learning disabilities. Some new evidence suggests that being born prematurely also increases the risk of autism, so that may also be a factor in the epidemic. Changing diagnostic criteria and better screening may also play a role.

    All of these questions have been at least partly answered and are still being investigated. It’s wrong to claim that we can’t answer them just because some of them haven’t been thoroughly answered yet.

  14. #14 Tom DVM
    February 12, 2009

    When it comes down to it, why aren’t discussing the industry ‘whores’ who have been cooking data for twenty years while we all conveniently look the other way!

    Whether or not Dr. Wakefield did or did not ‘cook’ his data set therefore is not the issue in my opinion.

    The issue is that we continue to have an unexplained increase in a number of diseases, possibly linked by it seems environmental (industrial) rather than genetic factors.

  15. #15 revere
    February 12, 2009

    Tom; We’re talking about it because that’s what the post was about. We have covered you other concerns (about corporate influence) quite often here.

  16. #16 catgirl
    February 12, 2009

    Tom DVM,
    Those increases in diseases are not unexplained and the environmental factors are certainly being considered. All of this information is easily available on Wikipedia, so there is no excuse for your ignorance. Read a little bit about asthma, CJD, and allergies before you claim that they are unexplained.

  17. #17 Jeorge
    February 12, 2009

    In our small family group, all the grandparents smoked. None of the children did but every single grandchild has asthma
    or some bronchial problems that their doctors hinted was caused by parental smoking. When the docs were told none of the parents smoked but all the grandparents did, they were shocked into silence. Two of the doctors wondered aloud if perhaps there was some genetic changes going on that passed from grandparent to parent but manifested itself in the grandchildren.
    My parent’s generation was the group raised on the theme that smoking was good for you and very cool. They had stopped smoking by the time our children were born, but I have often wondered if their smoking contributed to their grand children’s asthma.
    And who knows if the same thing might be true for autism as well.

  18. #18 daedalus2u
    February 12, 2009

    There is some (very limited actually) evidence that exposure to smoking in utero reduces the incidence of autism.

    http://www.pediatrics.org/cgi/content/full/107/4/e63

    My explanation is that the carbon monoxide in tobacco smoke binds to some of the same heme containing enzymes that NO does and produces “cross-talk” that mimics a higher NO state.

    Asthma and allergies are part of the spectrum of diseases that are all worse in the developed world and virtually unknown in the rural undeveloped world. There is the hygiene hypothesis that posits removal of some agent associated with living in the rural undeveloped world is causal.

    I think the bacteria I am studying are an important causal agent of the hygiene hypothesis. There are others that are known to be effective. An infestation of parasitic worms in the gut has quite powerful healing effects on inflammatory bowel diseases.

  19. #19 paiwan
    February 12, 2009

    “There is the hygiene hypothesis that posits removal of some agent associated with living in the rural undeveloped world is causal.”
    “An infestation of parasitic worms in the gut has quite powerful healing effects on inflammatory bowel diseases.”

    Thank you for this post.

    As the latest findings from Metagenomics have advanced, our understanding of micro-organism will be modified. The problem was due to the methodology of identifying microbes; we need culture medium to grow the colony for bacterium, with microscope and electro microscope, then we can identify them. But there are lots of mcro-organism can not be cultured by our existing media.

    PCR doesn’t need the culture medium; nevertheless prior to it, we need the primer. Metagenomics has provided much broader methods to detect the existence of nucleotides- genome. In marine world, the un-identified micro-organism perhaps are more than 99%.

    My experience in hatchery hygiene has shown that the disinfections by powerful ozonation or strong chlorination eventually will lead to the situation that the cultured species tends to be very naive in immunity; therefore it will incur big problems in the ongoing production.

    Ultra-clean and hygiene are not proper for marine case; perhaps this will provide a clue for human public health.

    http://www.nytimes.com/2009/01/27/health/27brod.html?_r=1&scp=1&sq=Babies%20knows:%20A%20little%20Dirt%20is%20Good%20for%20you.&st=cse

  20. #20 daedalus2u
    February 12, 2009

    Not all of the adverse effects of being too clean are mediated through the immune system.

    http://hyper.ahajournals.org/cgi/content/full/51/3/784

    Bacteria living on the tongue reduce nitrate from the diet (well absorbed and concentrated 10x in saliva from plasma) to nitrite and that nitrite has demonstrable physiological effects.

    Anti-bacterial mouthwash kills these bacteria and decreases the production of nitrite. I suspect that the adverse effects of alcohol consumption on mouth and throat cancer might be mediated through changes in these oral flora.

    The bacteria I am working with make nitrite from ammonia in sweat. Ammonia in sweat (including urea) is 3 orders of magnitude higher than the nitrate level of sweat.

    I blogged about how skin biofilms of these bacteria can suppress skin infections, including those from MRSA and resistant Pseudomonas.

    http://daedalus2u.blogspot.com/2008/06/suggestion-to-reduce-antibiotic.html

    Much of what suppresses infections is the chemical warfare between commensals and pathogens. When the pathogens get the upper hand, they express quorum sensing compounds and then become virulent. That is why antibiotic resistant pathogens are so dangerous. The antibiotic kills off the helpful commensals that are keeping the pathogens in check, with the niche now open, the resistant pathogens can expand exponentially.

  21. #21 Antaeus Feldspar
    February 13, 2009

    “When it comes down to it, why aren’t discussing the industry ‘whores’ who have been cooking data for twenty years while we all conveniently look the other way!

    Whether or not Dr. Wakefield did or did not ‘cook’ his data set therefore is not the issue in my opinion.”

    Tom DVM nicely demonstrates the usage of the tu quoque fallacy.

  22. #22 Path Forward
    February 13, 2009

    Revere wrote:

    At the age of 9½ months, 10 weeks before his jab, his mother had become worried that he did not hear properly: the classic first symptom presented by sufferers of autism.

    Sometimes I think that in this age of cheap, accessible video, routine developmental exams should be video’d (with informed consent, of course), and kept as part of a child’s pediatric records.

    Kids often act shy or strange in the doc’s office, so not every video would be a perfect representation of the child’s developmental and interpersonal capacity.

    But it would be helpful in efforts to track back the onset or first observation of a developmental problem, and not just for cases that end up labelled “autism.”

  23. #23 Sergio Stagnaro
    February 18, 2009

    I stop reading further, when I’ve read: “…..measles-mumps-rubella (MMR) vaccine was the cause of some cases of childhood autism and a chronic inflammatory bowel disease”. How is it possible (i.e., MMR vaccines is the cause, if milion of children around the world have had vaccination favorably? I cannot agree at all with this a-scientific statement, due to the fact that today’s Middle Ages of Meeicine is ruled by Economics.

  24. #24 Annie H
    February 1, 2010

    I dont know about the arguments for or against what Dr Wakefield did or did not say or do. All I know is that from the time my child had the MMR vaccination I had a different child. When he had the jab I wanted him to have it, I was confident in the jab. He had it and was really ill for a week afterwards, he needed to go into the hospital as we could not get his temperature down. Afterwards, he changed. Everyone said it… he was different. He went from a talkative, bubbly inquisitve, loving little boy to being withdrawn, isolating himself and ‘different’ to other kids. He is 12 now and has been diagnosed with Aspergers Syndrome. I have no doubt at all that the MMR caused it. I dont care about the arguments for and against, I only know what happened to my child. I think the vaccines should be offered singly. If I had known then what I know now I would never have had him have the MMR.

  25. #25 john
    February 2, 2010

    Silenced Witnesses Volume II: The Parents’ Story http://www.whale.to/vaccine/silenced_witnesses2.html

  26. #26 Carl
    February 19, 2010

    Annie H:

    That is the kind of thinking which resulted in superstition. “All I know is old lady Gretchen’s black cat ran across my field, and then my corn died, and I don’t care about any arguments for or against witchcraft because I know what happened to my corn.”

    The funny thing is that you will surely recognize how silly that logic is on any issue you don’t already believe, but you can’t stop doing it yourself.

    And don’t complain that I am mocking you, because it would be logically inconsistent to recognize that as mockery without proving yourself to be wrong.

  27. #27 Carl
    February 19, 2010

    Annie H:

    That is the kind of thinking which resulted in superstition. “All I know is old lady Gretchen’s black cat ran across my field, and then my corn died, and I don’t care about any arguments for or against witchcraft because I know what happened to my corn.”

    The funny thing is that you will surely recognize how silly that logic is on any issue you don’t already believe, but you can’t stop doing it yourself.

    And don’t complain that I am mocking you, because it would be logically inconsistent to recognize that as mockery without proving yourself to be wrong.

  28. #28 Paula
    February 19, 2010

    I don’t see the point of Carl’s argument here. Annie H. is suffering and Carl is indeed mocking her. (And please note, Carl, that is not inconsistent for this, or another, observer to point this out to you.) Currently, I am dealing with a situation of county commissioners’ involvement with a windturbines project that would make unlivable the homes of myself and several dozen other persons; we citizens find ourselves struggling against a 5-county consortium, state industry-backed energy department, and apparently national energy bigwigs, not to mention multinationals, who, pushing the “research” results of industry-funded studies, claim that those independent studies showing serious health effects from windturbine noise (not to mention effects on birds, bats, large resident mammalian wildlife, etc.) aren’t real. This gives one a great deal of sympathy for persons like Annie H.–or, for that matter, Lois Gibbs and her fellow residents, once upon a time, on Love Canal–who must claim that yes, a position backed by those in power, including a “scientific” position (and perhaps some research), obviously is not true to what is actually happening to them. But, in all events, mocking another commenter online seems inappropriate, especially on a blog site such as this where comments are met with warmth.

  29. #29 William Wallace
    March 23, 2011

    Anti-vaccine doctor meets with Somalis.

    (For the record, and in the interest of full disclosure, I am against many vaccines for my children, but I do hope everybody else gets all of their children vaccinated for everything recommended by the CDC.)