Effect Measure

Vaccines is a topic I don’t like writing about so much for many reasons. Vaccination programs are important to public health but we (all the Reveres, including this one) have always interested either in basic science or programs that are applied to the whole population at once, such as clean water, air or food or safe products in the marketplace. But vaccines keep coming up so we talk about them. Since this blog has spent a lot of time on flu, most of it has related to influenza vaccine, but not always. This is a “not always” post, and it is partially about the latest news that US Army and Thai researchers are reporting a modest success with an AIDS vaccine. We mentioned it in passing a week or so ago, and while the effect, as reported, was indeed modest it was the first time any AIDS vaccine looked like it worked at all. The trial was controversial at the outset, but the results seemed to vindicate it as a proof of principle for a vaccine, which some thought might never be a workable way to deal with AIDS. Like everyone else, the idea filled us with hope, even though we hadn’t seen the paper it was based on. Now a controversy has erupted about whether the data really support the hopefulness of the Army’s press release. It turns out we weren’t the only ones who hadn’t seen the data. Now that some people have, there are new questions. So first a little about the specifics and then a comment about the press release.

Jon Cohen over at ScienceInsider seems to have broken the story:

The press conference and press releases discussed an analysis that included all 16,000 people who participated in the trial, except for seven who were infected before receiving any doses of the two vaccines that were used in combination. Seventy-four people in the placebo arm of the study became infected with HIV, while the similarly sized vaccinated group only had 51 infections?a 31.2% efficacy. The analysis indicated that there was about a 96% level of confidence that the effect was real and not due to chance?just above the 95% cutoff that is widely used as a measure of statistical significance.

In the private briefings, researchers learned that a second analysis, which is usually performed in vaccine studies and was part of the Thai study?s design, also found that vaccine recipients had fewer infections, but the reduction was not statistically significant and the level of efficacy was slightly lower. This analysis eliminated people in both groups who did not rigorously follow the protocols. ?Anything that really works, you?ll have enough robustness in results to be significant with both analyses,? says Douglas Richman, an AIDS researcher at the University of California, San Diego, a longtime critic of the study. Richman did not discuss the specific results with Science. (Jon Cohen, ScienceInsider)

It’s not exactly correct to say that the 95% cutoff is one between the effect being “due to chance or not due to chance” as this implies, but the more important issue is that the data were not completely reported and it sounds like the conventional level of significance was very sensitive to moving a case or two from one group to another. It’s the uncertainty about this that is involved with the second (unreported) comparison.

Failing to achieve statistical significance doesn’t mean that there was no effect, but it leaves open the alternative possibility that the lowered AIDS in the vaccinated group was a chance fluctuation rather than the vaccine. This was still not very likely, but more likely than a conservative but conventional criterion would permit and certainly not the stuff of a press release and press conference. And that, it seems to us, is the real story here. The details about the analysis will get sorted out one way or another in the peer review and publication process (a paper with both analyses is supposedly under review at the New England Journal of Medicine).

Comments

#1 Felix
October 6, 2009

Credit also to The Lay Scientist (http://www.layscience.net/) who flagged up concerns about this press release as soon as it was in the news:

Are Recent AIDS Vaccine Claims Seriously Flawed?
http://www.layscience.net/node/640
#2 Manny Kimmel
October 6, 2009

Shame on you for regurgitating a poorly sourced blog rife with anonymous sources. If the sour-grapes naysayers (you know who you are) feel so strongly, why do they not go on the record instead of whispering to their journalist friends, anonymously?
#3 Felix
October 6, 2009

@Manny,

Hi,
I presume that your comment is referring to mine.

The blog I cited _was_ lacking in details (presumably because the only source of information was a press release), however to my mind the point is that the media was full of this story which _sounded_ brilliant but was/is not backed by published data.

The result being, as usual, hype and poor quality reporting.

That Martin was able to calculate the P value for himself and let us know not to e too excited was very useful for me when I was trying to get past the headline.

As for naysayers etc etc I know nothing, perhaps you could enlighten me?
#4 Manny Kimmel
October 6, 2009

“Felix”

No my comment was directed to Revere, who has glommed onto a (largely) anonymously sourced blog that shames Science. If the detractors of Rv144 (and the way the data were presented) feel so strongly, why do they not go the record with their concerns. The DoD investigators took all questions in an open forum – it sounded pretty transparent to me.

Regards,

Manny K
AIDS Activist
#5 Jon Cohen
October 6, 2009

Manny Kimmel,

I prefer not to quote people blindly, but the researchers who spoke with me violated a confidentiality agreement and could face legal ramifications if their names were used. No one has questioned the accuracy of the facts they told me and that I reported.

Speaking of blind sourcing, is “Manny Kimmel” your real name? I encourage readers to google “Manny Kimmel.”

Jon Cohen
#6 Jonathon Singleton
October 6, 2009

Revere, “Once the genie is out of the bottle there is no excuse for keeping the data from the scientific community. Both the researchers and the journal editors deserve censure…”

Revere, maybe that’s the whole point — legal censure… What if this issue really is about MedImmune’s FluMist!?! The legal implications are astounding…

Admittedly, what you write is utterly correct on all levels. My base instincts — not factual analysis — suggest trouble with regard to this research led by Danuta Skowronski of the British Columbia Centre for Disease Control and Gaston De Serres of Laval University… The news leaks connect last year’s North American LAIV-based seasonal flu vaccine with an increased chance of getting this year’s version of swine flu.

Yes, once the genie is out of the bottle there is no excuse for keeping the data from the scientific community… Midst the present roll-out of US vaccination campaigns (of which, a significant portion are LAIV-based), the release of relevant data to a public health operation which may (or may not) go tits-up on a truly grand scale would be rather helpful…
#7 Manny Kimmel
October 6, 2009

“Jon Cohen”

I’m not THAT Manny Kimmel. Not to worry –no underworld connections…….http://en.wikipedia.org/wiki/Manny_Kimmel

Googling Jon Cohen yields 15 millions hits. You are very prolific.
#8 revere
October 6, 2009

Manny: If you read the post carefully you will see that my problem is with science by press release. The point of sourcing is so you can check the source. In those terms and in science terms, the press release was unsourced, and I object to doing science on those grounds. Presumably if you object to unsourced material, so would you. But you don’t object. Maybe that’s the difference between AIDS activism and AIDS science, a difference I respect but which I recognize.
#9 Manny Kimmel
October 6, 2009

Yes, I understand that the theme of “science by press release” is one of your favorite hobby horses, one that you have ridden many times.

Science by press release is generally not a good thing. Thanks for the reminder.

My point here is the anonymity issue (“Revere”— HELLO!) and my objection to scientists like you and the ones “Jon Cohen” uses in his “scoop” who are are too chickensh*t to use a real name when voicing their opinions.

Thanks for the gratuitous swipe at AIDS activists, doctor. Your endless arrogance is, at the end of the day, amusing to us little people.
#10 raven
October 6, 2009

Without seeing the data, it is hard to evaluate the claim Given the history of AIDS vaccines, this is in the category of extraordinary claims requiring extraordinary proof.

Any secondary endpoints? How robust is the immune response? Neutralizing antibodies? T cell mediated immune response. Killer CD8+ T cells induced?

It would be nice to see this clinical trial replicated. Going to be hard with 16,000 patients over a multiyear period. Gets expensive fast.
#11 Manny Kimmel
October 6, 2009

“Jon Cohen” writes:

“The researchers who spoke with me violated a confidentiality agreement and could face legal ramifications if their names were used.”

Very honorable of them. Not.

Please be assured that a FOIA request is on its way to determine the names of the individuals who signed the agreement. That list will be made public. Then we can begin the process of determining who among them are the leakers of privileged information. Perhaps the SEC will be interested in this because of the market implications. Will you receive a subpoena for your hard drive and phone records? Perhaps. But, hey, you were just doing your job. You’ll be okay.
#12 Felix
October 6, 2009

Manny,

I don’t understand your position, as an AIDS activist (which sounds very commendable) why would you want this hyped story falsely raising expectations? Why would you object to bloggers warning that the reality may not live up to the headline?

(This is all, of course, assuming that the reported effects are unreproducable. If the vaccine actually works at 30% as reported than everybody here will be happy.)
#13 Rich
October 6, 2009

Interesting exchange and one that is misleading, in places. There is a “chance” that a significant effect (as opposed to any old “effect”) went undetected. The trial probably was powered at level that the likelihood of this would be 20%. This is the usual parameter.

The second analysis, including only those who followed study procedures is concerning. On the one hand, it is less conservative (in practice, lots of people don’t follow clinical regimes), but one would have expected it to yield a larger odds ratio. Given the small numbers of HIV+s, even in the presence of an adequately powered trial, the findings seem pretty unreliable. One concern I’d have is that there will be pressure to do underpowered subgroup analyses that will lead to the kind of controversies that doggged the US VAXGEN trial, which yielded misleading suggestions that the vaccine “worked” for under-represented subgroups who had not beeen sampled in a particularly systematic way.

The 95% “cutoff” is commonly treated as Cohen used it, whereas the strict statistic use of of a p-value is more indirect and simply provides the likelihood of false rejecting a null hypothesis.

Cohen is probably the most credible journalist on HIV issues and has been following HIV vaccines for 20 years. he has had incredible access to people over time. Hence, his credibility is great, but clearly, he misses nuances. Still, he’s better than the usual science journalists who seem incapable of simple addition and subtraction.

The Walter Reed PIs did do a reasonable job of explaining the initial findings and doing Q&A on a conference call sponsored by an advocacy group, but clearly this was no substitute for a peer reviewed paper. Revere is right that they should have waited for the NEJM paper to be in print.
#14 Curiou
October 6, 2009

Revere,

I wondered if you’d comment on this:

http://www.bloomberg.com/apps/news?pid=20601101&sid=aFY7sRohP0hM

(And I apologize for not being on topic.)
#15 caia
October 6, 2009

We have a bad example currently with the alleged “Canadian problem” suggesting that getting last year’s seasonal flu vaccine increased your chances of getting this year’s swine flu.

And while they’re sitting on that, according to actually published research [BMJ], having gotten last year’s seasonal flu vaccination may provide some protection against this year’s pandemic strain.

If the Canadian data were actually published, one could compare the robustness of the two data sets. But it seems like the unpublished data will get more attention, if only because that which surprises/frightens us gets more attention.
#16 Manny Kimmel
October 6, 2009

Felix,

At the end of the of the day, I don’t disagree with you, Revere, Rich or others who think that a better presentation of the data would have been useful, either in a journal or at a meeting (such as the big AIDS vaccine meeting coming up), before or at least simultaneous with the PR blitzkrieg.

My point from the beginning was that I object to the anonymous sourcing — and repetition of anonymous sourcing — upon which Jon Cohen’s story was hung and which Revere perpetuated.

These sources are no doubt big shot professors/clinicians/researchers who should have the balls to air their views without hiding behind big, bad Jon Cohen, he of the magic rolodex. And if they signed a confidentially agreement, they should honor it. Seriously, STFU and stop with the secret gossiping and kvetching. This is the terrain of high school girls, not grown men and women who one would think would have a bit more dignity.

AIDS activists do stupid things all the time, but for the most part aren’t sneaky like this mob.
#17 revere
October 6, 2009

Manny: I didn’t just repeat it. I checked with a couple of biostatisticians familiar with the issue, although they hadn’t seen the data. They agreed that leaving out the other analysis wasn’t good practice. It isn’t, and you don’t need sources to tell you that. But I checked anyway to verify from practitioners in clinical trials. So I don’t really see what your beef is. The presser was a mistake and may have been inappropriate. I am criticizing my own profession here for these practices and I don’t apologize for it or think I should STFU (I know you weren’t referring to me specifically, but your logic would still apply to me).

And this is a fucking blog, not the New York Times (that’s why we are more accurate). I also write under a pseudonym to protect my colleagues and students from being harrassed because of what I say or penalized by funding agencies. So everything here is unsourced. So whatever you say I said is unsourced, too. Big deal. You didn’t like the content of the story so you made it a matter of principle.
#18 Manny Kimmel
October 6, 2009

Revere,

This I don’t get:

“I also write under a pseudonym to protect my colleagues and students from being harrassed because of what I say or penalized by funding agencies.”

I can’t think of anything you have written (and I have read hundreds if not thousand of your posts) that would cause your colleagues to be harrassed. Seriously. Give me one example of a post and how, why and where these other people would be harrassed. I can’t imagine the really great progressives — eg Noam Chmsky — saying something like that. You rarely say dumb things, but that is dumb.

Ditto for the funding agencies. What, you think the guys and gals in study sections at NIH or NSF or wherever you get your $ are really paying that much attention and would hold your opinions in an open forum against you? Do you know of such circumstances? Or is it more of a plausible scenario that COULD happen.

I am not just cracking wise; I am really am curious where you are coming from.
#19 Jon Cohen
October 7, 2009

Manny Kimmel,

People spoke to me off the record about a serious issue. They were not gossiping or making ad hominem attacks on an individual. They were pointing out that the scientific standard is to present the data, be they positive or negative, that are spelled out in a trial’s protocol. They were advocating for transparency, and citing other similar studies that handled the initial reporting of results differently.They were highlighting that AIDS advocates had called for the same details, before the results were presented.

The press conference and press release highlighted the positive, and the critics were upset that they omitted the negative. Had the two been concordant, it wouldn’t be an issue, but they are not. So this raises the serious point that I think you are failing to address: The claims of efficacy have to be examined with all of the data. The less robust the results, the more difficult it becomes to extrapolate from them and build better vaccines.

Your carping about people speaking off the record here would have more weight if they made unsupported accusations. As far as I can tell, everything they told me and that I reported is accurate. The confidentiality issues are another matter and have nothing to do with the truth. (And your FOIA gambit is absurd: Literally dozens of people heard the data.) My protecting sources who obviously could suffer harm? Come on. That is the purpose of granting people anonymity. I do not do it often, but in this case, I think it’s justified–and an ethical violation to do otherwise.

The bottom line is this: data speak. And the people who discussed “confidential” data with me on the condition that I not name them heard results that surprised them–and they felt the community should have heard them from the get go. Maybe in the final analysis it won’t matter one wit in the the design of an effective HIV vaccine. But I gather from your complaint about blinded sources that you don’t believe ends justify means. So there’s at least one point on which you and the critics of the press conference and press release agree.

One final question. Is Manny Kimmel your real name? I ask because of your attacks on Revere for wanting anonymity, and because I can’t find an e-mail address or anything else about you. I’m at jcohen1@cox.net. I have a website at http://www.joncohen.org. Where can people reach you?
#20 revere
October 7, 2009

Manny: The Reveres (however many and whoever we are) have written literally thousands of posts here (slightly above 3200). Among them were posts severely critical (and often using language that would never be used in academia) of our funders (Directors of NIH institutes, Director of CDC, well known and established scientists whom we have said have acted unethically). Whichever Reveres say these things here say the same kinds of things in public, and have for many years, but we use far different language. We are well established and long in the tooth and have been jailed, beaten by police and called all sorts of names. I can take it. My wife was almost fired from her job because of what I said in the newspaper about the health commissioner (she worked at the health dept.). The only thing that saved her was that she was union. I also am a friend of Noam’s. He is in an entirely different position and I admire him greatly. But my students work for gov’t agencies when they graduate, are funded by gov’t agencies, are reviewed by scientists I criticize and my institution likewise. I lived through the McCarthy days and I know well what can happen and I see it happen still today, in other ways. I’ve paid my dues but I have two generations of students and colleagues I won’t put at risk because of what I say on a goddam blog I’m sometimes so tired while writing after a long day at work I can hardly tap the keys. When I say it at a meeting or in conversation I know who is listening and I know how to choose my words. On a blog everything is different. The only activists that would harrass me (outside of zealots who have tunnel vision on a particular subject) are wingnuts, and I can take that. I don’t care about them. They won’t deny my students a job or grant funding. I hope this is sufficient explanation. Either way, the conversation is over. I think Jon’s response is sufficient on the content of your point. There is a time when a reasonable person would say, “OK, I get it. Thanks for the explanation.” That time has come for you.
#21 Manny Kimmel
October 7, 2009

Okay, thanks for the explanations, “Revere” and Jon Cohen.

Both cogent.

I wonder: in your liberal souls, is there something about an accomplishment by our military that you find irksome? What with the police beatings and so on that at least one of you have suffered. Uniforms …… ugh!
#22 SusanC
October 7, 2009

Catching up…

thanks, revere for both an informative post – it is informative on happenings or lack thereof, even though the central story has to do with lack of data – and a cogent argument about (my words) the dumping down of science and scientific practice because some people can’t wait to get their 2 minutes of fame.

I also want to say that story was reported with BIG headlines and had a 2 page spread in the Times in the UK. I didn’t look into it, and assumed that the data had gone through peer review and was published or being published around the same time as the press releases. I should have known better – how many times do I have to be surprised before I learn not to take people’s word for it?!!

So, thanks, for hitting me on the head with a 2 x 4…
#23 albatross
October 9, 2009

I know confidentiality agreements are very common, and I’ve dealt with some in my own field. But an agreement that prevents scientists involved in some research from openly discussing their own concerns or reservations about their results strikes me as a horrible idea. Is there some kind of justification for such an agreement? (I understand why the funding source or employer might like it, but it seems like a pure lose for the world as a whole–like an agreement that prevents someone from telling the police about crimes committed by their employers.)

Manny:

One reason for pseudonymity is just to allow yourself to clearly separate your own opinions from those of your employer, students, teachers, etc. You might be concerned about consequences for p–sing people off with your opinions, but you also might prefer to argue, say, your political opinions without any impliction that those are somehow shared or supported by the university or government agency or company you work for. I don’t see anything evil or bad about this.

In fact, some very informative and good blogs, which improve the quality of discourse in the world, are written pseudonymously. Orac and the reveres and the guys at Emergent Chaos are all good examples, and there are many more.
#24 Manny Kimmel
October 9, 2009

Albatross,

I take your point that anonymity/pseudo-anonymity can allow for freer discourse on blogs.

Was Jon Cohen’s piece that Revere abstracted a blog or a news story? A close call. With a news story, I prefer (as a reader) to have sources identified such that I can assess their biases, their agendas, their conflicts.

For instance, why exactly did Mr/Dr Cohen’s sources divulge these details to him? He says they were “advocating for transparency” — I am willing to take his word about this, but do they also have competing agendas, perhaps with regard to their own scientific endeavours or commercial interests? I would like to know. Perhaps forthcoming stories by Cohen and his peers will illuminate this.