Scientific Publishing

The Chronicle morning round-up

terrasig — Thu, 17 Jun 2010 04:15:28 +0000

The Chronicle morning round-up

Waiting for that coffee to take effect but want it to appear you are doing something scholarly?

Have a look at this pair of highly-read posts at The Chronicle of Higher Education:

We Must Stop the Avalanche of Low-Quality Research
The most-viewed article of the last two days at the online presence of the nation's leading higher ed publication, this team-authored position piece has been a magnet for criticism. The thread of 102 comments (thus far) is as worthy of your time, if not more, and the humorous and insightful payoff by commenter #100 is clever and spot-on, IMHO.

Why 'Female' Science Professor?
Written by the Grande Dame of science professor blogging (yes, I used a gendered term but it is one of respect as coined by DrugMonkey), the "midcareer female professor of the physical sciences at a major research university" holds forth on the reasons she uses "Female" in her blog 'nym and title. So deal.

terrasig Thu, 06/17/2010 - 00:15

Tags

Academia

Blogging community

Scientific Publishing

Portrait of a Reviewer as a Young Man

pontiff — Mon, 07 Jun 2010 21:11:58 +0000

Portrait of a Reviewer as a Young Man

Science is dynamic. Sometimes this means that science is wrong, sometimes it means that science is messy. Mostly it is very self-correcting, given the current state of knowledge. At any given time the body of science knows a lot, but could be overturned when new evidence comes in. What we produce through all of this, however, at the end of the day, are polished journal articles. Polished journal articles.

Every time I think about this disparity, I wonder why different versions of a paper, the referee reports, the author responses, and all editorial reviews aren't part of the scientific record. In an age where online archiving of data such as this is a minor cost, why is so much of the review process revealed to only the authors, the referees, and the editors?

pontiff Mon, 06/07/2010 - 17:11

Tags

Scientific Publishing

The Loony Bin Called Academia

Completely agreed. My dream setup for a journal would be that whenever a paper gets accepted, the reviews and the reviewer's names get published with it. If the paper is rejected, then everything stays anonymous. This way, reviewers have the same incentives as authors. In addition, a good review could be viewed as a valuable contribution to the process (like it should be right now, but since reviews inevitably end up buried, very few people seem to bother)

Carlos, the problem with that I think would be that most people would go the 'safe' way and reject papers, since that would protect their anonymity and wouldn't present them with the risk of being associated with a potentially wrong/trivial/.. paper. Also, since there are several reviewers per paper, at least for some journals, how would you deal with the person who would raise objections (or reject) but would still have his/her name published if the other reviewers said 'publish'? Should their name be published too? Moreover, blind reviews protect more vulnerable members of the community like early-career and tenure-track scientists.

Amen. I've spent a lot of time working in the history of science and one of the things we have lost in the digital age is "rough notes." For papers written thirty years ago or so, notes - from scraps of paper to entire notebooks - frequently can be found in archives and private collections that detail the "messy" process of science. The other thing we have lost, particularly with the advent of e-mail, is written letters as a record. Some of the best ideas have come out of these letters (I cited several in my PhD thesis). But electronic mail not only gets lost to the cyber-wastebin, even when it is saved it is rarely as informative as a written letter. The latter often included hand-drawn diagrams, easier-to-decrypt equations (LaTeX has its limits), and other tidbits not found in the relatively cold form of an e-mail.

Some of the open access journals do this - PLoS comes to mind. I want to say that the BMC journals also do, but I can't be sure offhand.

IJQI experimented with this for the special issue(s) on distributed quantum computing. My contribution:
http://quantalk.org/139
Apparently, only a couple of people were willing to go through the public humiliation :-). I'm an author, and reviewed a paper publicly, too.

I'm in favor, at least on an experimental basis.

I had this same thought a little while back. Couple of comments:

(1) This could also be done for conferences (in CS we don't use journals unfortunately).

(2) The reviewers comments could be anonymous. This would still be valuable.

Also, don't they already do this with the Royal Society Journal of Statistics? It seems to work well.

Nature asks eight "experts" about Venter's prosthetic genome work

terrasig — Sun, 23 May 2010 08:45:02 +0000

Nature asks eight "experts" about Venter's prosthetic genome work

Men. Check.

White. Check.

Grey. Mostly. That dude must use color.

Beards. Only two. Maybe three. Aw, hell, Church's makes up for the rest.

Article here.

Ed Yong (Asian-British, man, young, dark hair, no beard) also lists a great wrap-up of the week's commentary on the work.

terrasig Sun, 05/23/2010 - 04:45

Tags

Race in Science and Society

Scientific Publishing

Underrepresented Groups

Women in science and medicine

*sigh*

just b/c they're old white dudes doesn't make them not experts

and just b/c ed wrote an article about it doesn't make him an expert

8 old white dudes and Woese not among them? That's just BORING. (Woese on Venter is HILARIOUS, always)

All those white people look the same to me ;-p

It's all about the beards. Doesn't matter the specialty - sociology (Durkheim), economics (Marx), psychology (Wundt), biology (Darwin)... I could go on, but I don't have to. So, only a couple of them are credible... amirite?

Aside from the beard, Church also deserves an award for his penetrating stare. If I were a gene faced with that look I'd give up my secrets too.

It's b-a-a-a-c-k. And I'm glad.

revere — Wed, 21 Apr 2010 04:31:24 +0000

It's b-a-a-a-c-k. And I'm glad.

When two of the most loathsome members of the US Senate bring back again a bill that won't die, you'd think I'd be in high dudgeon. But I'm not. I hope the bill isn't killed or is allowed to die -- again -- and we finally get it. I'd much rather that the two right wing whack jobs, Senators Joe Liberman (morally corrupt Independent neÃ© Democrat) and John Cornyn (morally corrupt Republican), spent their time sponsoring this kind of legislation than making their usual mischief that hurts everyone. What is this miracle legislation that brings me together with these usually worthless publicly supported time server water carriers for the rich? FRPAA!

FRPAA is the Federal Research Public Access Act which requires taxpayer supported research from designated federal agencies to be publicly accessible online within 6 months of publication in a scientific journal. To be fair to the way the world usually is, the current revival of FRPAA is not from Liberman/Cornyn in the Senate but Mike Doyle (D-PA) in the House, but it sounds like it is essentially the same as the 2009 Senate version from the Brothers Grimm:

The bill's open access mandate would apply to the Departments of Agriculture, Defense, Energy, Health and Human Services, the Environmental Protection Agency, and the National Science Foundation, among other agencies.

The first version of FRPAA was introduced in Congress in 2006 (later dying in committee), and was modeled, like the current bill, after an open access mandate at the National Institutes of Health, which requires that all NIH-funded research be deposited in PubMed Central within 12 months of publication.

"FRPAA reflects the growing trend among funding agencies -- and college and university campuses -- to leverage their investment in the conduct of research by maximizing the dissemination of results," read a statement from the Scholarly Publishing and Research Coalition, which is urging its members to support the bill. (The Scientist)

While the big publishers and some scientific societies still oppose it, the principle that taxpayers shouldn't have to pay twice for access to research they funded has gained a lot of traction, with big universities like Harvard, MIT and Boston University solidly behind it with their own open access policies. Private funders like the Wellcome Trust are also onboard as are many scientists, including us.

Congress finally got over the health insurance hump, albeit with a version that's pretty weak tea. Maybe they can at last do better with open access for work the public paid for. That would seem to be a no brainer, if you aren't a publisher raking off profit by privatizing public money.

If Liberman and Cronyn can swallow it without gagging, it should go down easy for a person with normal morals. Unless the publishing special interests have a choke hold on their campaign finances.

revere Wed, 04/21/2010 - 00:31

Tags

Media

Open Access

Scientific Publishing

How much does it really cost to maintain a peer-reviewed on-line journal? Including maintaining availability into the future?

Welcome back, the Fonz of Public Health, lol

Here are the Canadian Report on malfunction and lack of control quality of Tasers as brought up by CBC journalists (in english this time) that prompt the Royal Canadian Mounted Police to destroy hundreds of tasers because it emitted more volts and has cause many deaths.

Sorry here is the link

http://www.cbc.ca/canada/story/2009/03/18/f-taser-faq.html

Excerps;

Tasers are designed to incapacitate a person through up to five centimetres of clothing. Taser International says the electrical pulse is delivered at a high voltage because the electric current has to pass through clothing and air â neither of which is a good conductor of electricity â to make a complete circuit with the target's skin.

Taser International also says that while its device can deliver up to 50,000 volts, it does not deliver that much voltage to a person's body. The company says its Advanced Taser M26 delivers an average of 1,500 volts.

As well, the high-voltage pulse of a Taser is said to carry only a small current, typically 0.002 to 0.03 amps.

By comparison, electrical outlets in Canada deliver 120 volts of electricity, and the current they carry depends on the appliance that's plugged into them. A 60-watt light bulb, for example, pulls 0.5 amps, while a toaster pulls about five amps.

It's possible to suffer a fatal shock from a household electrical socket, at just 120 volts, if enough current passes through the body

Tests conducted for CBC News/Radio-Canada, however, found that some stun guns produced higher-than-advertised current.

The procedures, conducted by U.S.-based lab National Technical Systems, found that 10 per cent of the X26 model Tasers produced more electrical current than the weapons' specifications.

In some cases, the current was up to 50 per cent stronger than specified. The X26 Tasers were manufactured before 2005 and are one of the most commonly used models.

Taser International said CBC made scientific errors by failing to spark-test the weapons before firing them, a process the company recommends police officers do on a regular basis. But engineers who reviewed the testing protocol for CBC said the tests were based on solid practices.

In late March, 2009, the Quebec government decided to pull all its stun guns off the street after their own testing found five of 52 Tasers made before 2005 performed outside the normal range.

I'm a strong supporter of public access to science (regardless of whether it's taxpayer-funded or not). Philosophically, I like seeing the details of science out in the open where they can be readily scrutinized by anyone, and not just those who happen to belong to an institution subscribing to a particular journal or willing to pay for each paper they view.

There are plenty of open-access journals that seem to be doing just fine. Maybe they aren't attractive to researchers operating on a shoestring budget, since the author usually has to pay for publication in this case (there are, after all, costs associated with the process), but it's a start.

Hopefully as time goes on the journals can further minimize costs and streamline the process. To me, hard copy journals are pretty much obsolete, and it makes sense to ditch them completely in favour of online-only publication (I guess the one use for hard copies is that if the journal isn't open access, a member of the public can walk into a university library and read a copy at no cost). This would also make it easier to include more supporting data in appendices or supplemental files, without worrying about page counts.

In the last 7 years, I had received a couple of thousand requests for medical information from Africa, India, Sri Lanka and many doctors.

It had helped a lot. The Peer Review Journal is trusted in countries as far as in the suburb of Dakar Senegal, where villae doctors have access to the internet only via Dakar University and for about 30 minutes a week.

One thin is evident too, even if the information is there on the web it dosn't mean it is accessible, wich motivate me to athered the informations as much as possible on one site.

In India, a Provincial Health department ask for the latest datas on how to eradicate mosquitoes to avoid Chickununya infections.

It worked and three provinces act swiftly with the good information to et rid as much as possible of mosquitoes larva and thus they reduced morbidity and mortality from Chikungunya infections.

The problem in emergency, like the H1N1 pandemic, some datas where under embaroed waiting for Peer Review. On this aspect they should do their best to accelerate the process.

Still, despite this, last fall a study on the use of Lipitor in ICU lost cases gave great results, tit reduce mortality of 50% in ICU after everything else has been tried.

They brouht up on line swiftly the results of ICU attemps, allowing me to forward this to Intensivists in many country.

Yes Peer Reviews papers should be put on-line and free as soon as possible, it does indeed saves lives.

I would assume this would stretch to the entire files on glowball warming too? Indeed when I tried to gain access to the information from one Mr. Hansen, the Democrat weather super scientist that Mr. Gore has been so recalcitrant to back lately, I was denied.

The sword cuts both ways. It is a good point though Revere. If we are planning to keep this country intact there is going to have to be a fundamental shift back to the center. This kind of stuff is just one thread, the cloth though is starting to unweave and its not just healthcare either.

But, they have to have time to publish. They have to have time to reap their benefits and the universities are the biggest culprits. But flat denial of FOIA requests are going un-actioned by this government and not the universities. Conspiracy? Yeah, maybe but its not Kansas any longer Toto. This is the least of our worries as the real indications of a revolution are starting to show.

When a government refuses to accede to the wishes of the people, when is it sedition to act against said government? To even speak out against this government is considered to be seditious. I did and I had an IRS audit notification within two days.

Coincidence?

Maybe, but now a pair of Senators are asking the questions instead of me. You are correct to demand that this info be released Revere and I applaud it. I found it interesting that I was on a list after asking for the weather data.. Now how does that happen?

Question Revere: Was Liberman morally corrupt when I assume you voted for him on the Democratic Kerry ticket? Also, if morally corrupt then isnt all decisions he makes also morally corrupt? Sorry dude cant have it both ways.

BostonERDoc: That's silly. Yes, he was morally corrupt then and I voted for Kerry. Like a lot of things I had to take Lieberman as part of the package, a package that had "not George Bush" written on it. The idea that all decisions made by a morally corrupt person are invalid doesn't make sense. If he decides to give a beggar a dollar does that mean all other decisions are OK? Obviously not. Nor are the converse or contrapositive true.

Hmm, I'm all for the dissemination of scientific knowledge, but still, a little knowledge can be a dangerous thing. I suspect that those genuinely interested in the science will be far outweighed by those looking to data-mine and cherry pick from them.

For instance, many climate change skeptics pick out things that, to a layman, sound contradictory, but are actually explainable by those who understand what the data means. A hyperbolic example would be the whole "hah there's lots of snow, therefore global warming is false" sentiment that spread around this winter.

I worry that all we'll really end up accomplishing is dumping more fuel on political fires. Since often it's easy to latch on to one thing that sounds questionable in a sound-bite, but explanations of what's really going on require a much more in depth look.

Lieberman was Gore's VP on the 2000 Democratic Party ticket, not Kerry's in 2004. I don't like him now and I didn't like him then either but the top of the ticket is more important(Gore preferred over Bush) and if we look at the VP candidates Lieberman may still come out ahead as morally corrupt seems a wholly inadequate descriptor for Dick Cheney

A scientific ethics of code

revere — Wed, 10 Feb 2010 05:35:08 +0000

A scientific ethics of code

I'm a scientist and my research is supported by NIH, i.e., by American taxpayers. More importantly, the science I do is for anyone to use. I claim no proprietary rights. That's what science is all about. We make our computer code publicly available, not just by request, but posted on the internet, and it is usable code: commented and documented. We ask the scientists in our program to do the same with the reagents they develop. Reagents are things like genetic probes or antibodies directed against specific targets mentioned in the articles they publish. There is an list of the reagents on the internet and instructions on how to get them if you are another researcher. Since giving you the link would also reveal the identity of one of the reveres, you'll just have to trust me that this is true. It is. And I mention it because I am in full agreement with a piece in The Guardian [UK] by Darrell Ince, a professor of computing at The Open University in the UK (hat tip Slashdot):

One of the spinoffs from the emails and documents that were leaked from the Climate Research Unit at the University of East Anglia is the light that was shone on the role of program code in climate research. There is a particularly revealing set of "README" documents that were produced by a programmer at UEA apparently known as "Harry". The documents indicate someone struggling with undocumented, baroque code and missing data - this, in something which forms part of one of the three major climate databases used by researchers throughout the world.
Many climate scientists have refused to publish their computer programs. I suggest that this is both unscientific behaviour and, equally importantly, ignores a major problem: that scientific software has got a poor reputation for error. (Darrell Ince, The Guardian)

I do not have a moment's doubt about the basic science of climate change. There are too many convergent lines of evidence and some really convincing science to back it up. But like a lot of science -- including a great deal of molecular biology and sophisticated engineering and much else -- it depends on complex computer code that can't be checked or verified because it isn't made available to other scientists.

One of the things we know about software -- even critical software that runs important medical devices like radiation therapy machines -- is that it is frequently in error. Checks of commercially produced software has found a high rate of error and inconsistency. Imagine what you'd find with software produced by academic researchers who aren't software engineers. The trouble is you can't find it because the code isn't always made available.

I feel the same way about data. As an epidemiologist there are some problems related to subject privacy with our data sets, but they can be overcome. Many of my colleagues object to releasing their data sets for a different reason: usually it has taken them years and a great deal of money to collect and they don't want someone else scarfing it up without lifting a finger and using it to scoop them. My colleagues -- and I -- want first crack at it. The same thing is true for sequence data in virology and other disciplines. I'm sympathetic because I'm in the same boat, but I think this can be dealt with, too. One way would be to grant a grace period before requiring release of data to allow the scientist who collected it to use it. Once published it must be made available, preferably as online supplementary material accompanying the research where it is used. Another solution would be to have some requirement for crediting the data collector via authorship or data origination credit, credit that would count for academic or professional purposes like promotion and tenure.

Whatever the solution, the principle should be that scientific data, like other information, wants to be free and has an even greater claim because science is an open process. It can't be if the tools that generate the data and the data themselves are not accessible for confirmation or verification. I agree with Ince:

So, if you are publishing research articles that use computer programs, if you want to claim that you are engaging in science, the programs are in your possession and you will not release them then I would not regard you as a scientist; I would also regard any papers based on the software as null and void.

We now use only open source statistical software suites like R because it can be checked, improved and corrected by a large community of users and by our scientific colleagues around the world. We make our own code available, too.

Because we like to consider ourselves scientists.

revere Wed, 02/10/2010 - 00:35

Tags

computers

ethics

Intellectual Property

Scientific Publishing

Technology

Nature science writing paywalls are pissing me off

terrasig — Thu, 04 Feb 2010 20:02:03 +0000

Nature science writing paywalls are pissing me off

Please forgive me for the cranky. I am still confined to bed and am only writing between fits of coughing that still occasionally drive me near unconsciousness due to hypoxia. I'm stuck at home trying to read some research literature across the VPN and proxy servers from my three faculty appointments that give me access to much biomedical research literature.

However, some journals are now no longer granting access if one's IP address does not come directly from the university, even if you are using the university VPN server. And then there's my love-hate relationship with Nature Publishing. I absolutely loved when Nature expanded to Nature Medicine and the Nature Reviews journals have been spectacular, particularly Nature Reviews Drug Discovery and Nature Reviews Cancer. But as each of these came out, it was costing another $199-265/year or so for each of these sources. Still, the content was worth it.

But today, I am cranky. Not because of lack of access to research publications but rather because I have been shut out to a book review and a news review by two of my favorite science writers, Steve Silberman and Melinda Wenner Moyer, respectively. One of these episodes on any other day and I'd probably be fine. But two? On the same day?

Steve just wrote for Nature a review of Rebecca Skloot's new book while Melinda apparently has a killer article in Nature Medicine on the search for drugs beyond statins to manage cardiovascular disease.

But to gain access to these, I need to pay $32. Each.

This is really just plain bullshit.

Don't get me wrong - I support Steve's and Melinda's need for livelihood more than the average scientist might. They are each gifted writers and deserve to be rewarded handsomely for their craft. But $32? Each?

Now, I don't want anyone sending me PDFs of these articles. Really. No. Don't.

I raise this issue because I am a (reasonably) well-compensated faculty member at a small state university so I can afford to buy some personal subscriptions but I didn't take Nature this year and I've usually relied on my institutional subscription to Nature Medicine. As I said earlier, I do have access to online resources at a couple of R1 institutions yet because I am holed up in bed on doctor's orders, I cannot access these articles. As much as I love Steve and Melinda, I'm not paying $64 to read their work.

This is where the OpenAccessEleventy movement has me converted: if I can't get access to this information for a reasonable cost, what does that say about the rest of my colleagues at universities here and abroad who may not even have institutional access if they are physically on-campus?

Listen, I'm not begrudging Nature or any paywall publication for paying their staff, hiring freelancers, covering their costs, and making a profit. But for God's sake? $32? Each?

Yes, I know that Steve and Melinda were not researchers who, as with a research publication, would pay "page charges" to have their work published. Instead, they had to be paid some magnitude of compensation for their work. But even without me being an economist, I am certain that Nature could make these works available for, say, $3.99 and still make a decent profit. And if they're charging $32 each, Steve and Melinda had better be having filet mignon and Silver Oak cabernet tonight. I'm sure that Nature is rewarding them handsomely. Right?

I'm sorry - I'm just furious. And, no, I'm really not feeling any better. In fact, I'm more frustrated these days because my pneumonia does not seem to be improving. The longer I am home and struggling to do work, the more cranky I shall become. But for those of you who come here for positive and enlightening posts, I'll try not to make a habit of this.

terrasig Thu, 02/04/2010 - 15:02

Tags

Journalists, Awesome

personal

Scientific Publishing

Right now, I'm *wishing* I was "a well-compensated faculty member at a small state university." Instead what I am is a 52-year old journalist who recently lost his contract at Wired after 14 years of full-time writing for the magazine in a wave of cost-cutting measures that I would bet don't include a cut in the editor-in-chief's extravagant CondÃ© Nast-style salary. I have been enthusiastically invited to continue writing for the magazine... as a freelancer. As far as the payment from Nature goes, after doing research for about two weeks on that little review, I will be paid what Nature accurately described as a "token honorarium." In this case, I consider simply being published in Nature for the first time to be an honor. But yes -- if about four people pay to download that article, it covers what I made from it.

I'm not against paywalls in general because I grew up paying for magazine subscriptions and hope, someday, to have a real job again. But first, magazines have to figure out how to make money. Nature has figured it out, though most of their subscribers are probably institutions.

(By the way, instead of cabernet and filet mignon, we're eating cheap Chinese food tonight, because my partner is currently supporting us on a middle-school science teacher's salary.)

I hope you feel better!

Steve, you are an amazingly talented writer and hearty congratulations on having your review published in Nature. In my rant, I neglected to note that it is a great honor to be have work in a Nature publication regardless of whether it's research, commentary, news, or reviews. Same for Melinda.

It goes without saying that the payment model has to work for the writers, first and foremost. This is a challenge for all press as you have sadly witnessed firsthand (also reminds me of my 50-ish relatives being let go by tech and mfr companies then welcomed back as contractors).

I'm lucky to have a good job, for now, but my point was that even academics of reasonable means and resources might not otherwise have access to your work. As I said, you'd think that more downloads at a more reasonable price would work but I guess NPG has already run the pricing and demand models.

In any case, it's an honor for me to field your comment. By the way, if any editors reading are looking for a great freelance science writer, commission Steve Silberman.

Best to you and Keith - I'm buying dinner next time I'm out in S.F. - when I'm better, of course. Thanks for the kind wishes.

Oh, one other thing: while most readers here will know Steve as a science writer, he is an accomplished music writer as well with some remarkable credits. From his website:

Music-Related Links:
I got a gold record in 1999 for co-producing the Grateful Dead's 5-CD box set of previously unreleased recordings, So Many Roads (1965-1995) -- which was Rolling Stone's Box Set of the Year -- and a second one in 2001 for my liner notes for the Dead's Workingman's Dead and Europe '72 in The Golden Road (1965-73). I also wrote essays for Dead Set in Beyond Description (1973-1989); the Dead Ahead DVD; Reflections in the Jerry Garcia box set All Good Things; the Jerry Garcia Band album How Sweet It Is and concert DVD Live at Shoreline; and David Crosby and Graham Nash's Wind on the Water, Whistling Down the Wire, Crosby/Nash Live, and Another Stoney Evening. A biographical essay called "A Thread from the Weave" is the primary text in David Crosby's Voyage, a box set spanning his career from the Byrds to Crosby, Stills, Nash and Young, CPR and beyond. An abridged version of "Singing Their Way Home" appears in Crosby, Stills and Nash's Greatest Hits. Beck: The Wired Outtakes is a mostly-unpublished interview about Guero, multimedia art, and the future of music from April 2005. Also see the interviews with Crosby, Trey Anastasio, Mickey Hart, and Bruce Hornsby linked below.

Pretty awesome, eh?

I'm not arguing any of your points about costs and access. I think there's actually an issue with your institution's vpn or proxy server because I can authenticate to those from off campus using EZ Proxy from my institution.

If you are on VPN, then your IP does appear to be from the university. Some very expensive clinical tools don't allow off campus access (Up to Date) but NPG stuff is not in that category. I hope when you contacted your librarian they were able to help you?

The VPN problems have been really, really annoying me, too. Wiley appears to now have a blanket 'no access on VPN' policy: my affiliation shows on the page but I can't get the papers. [Even some of *my* papers, which is especially vexing!]

@Chrstina, #4: I would have thought so, too [i.e. VPN being specifically designed to avoid these issues] but if multiple folks at multiple institutions are having issues, I guess they're real. I tried via multiple VPN systems on which I am authorised, also, with identical results.

Hope your pneumonia is getting better soon. I believe that the $32 per article are meant to discourage people to purchase individual articles. DeepDyve lets you rent articles for $0.99, but doesn't cover a lot of journals.

As of this week you have another option if you own an iPhone (Android phones in April). The Nature.com iPhone app currently gives access to the fulltext articles without institutional subscription.

@Ewan - Wiley does not have a no VPN policy. First, I hope they would have e-mailed me because that's a big deal. Second, I just connected via SSL VPN to my institution and had no problem wandering around JASIST and pulling articles

Are you sure your institution still subscribes to that journal? Remember: Wiley does *not* mark subscriptions in any way (no green dot, no check mark, etc) so you may not have access. If your catalog, metasearch, or link resolver says you're supposed to, you should contact the library.

as an unemployed scientist, this has been *my life* for the past 5.5 years -- and there's no relief in sight, either. so tell me how an unemployed and impoverished scientist is supposed to "stay in the game" when she can't access the fucking literature??

Gee, if you are getting some of these Nature Journals for $199-265 a year then you are getting a steal. Please be aware that the starting price that Libraries pay for some of their publications is MUCH higher. Nature Medicine costs libraries $3,060 a year while their subscription to Nature will cost $2,920 a year. Don't forget that Nature Publishing is also responsible for raising the price of Scientific American, which publishes popular science articles that might inspire future doctors and scientists. Libraries now pay somewhere between $1,000 to $1,5000 a year (and were given no warning about the price increase). While I understand the need for writers to make money off of their product, I firmly believe that knowledge is a common pool resource that should be available to all. Next time, find an open access journal please.

Abel, thanks for your generous comments and your link to my "other" writing life, which will discredit me in the science community for decades . This year I will be doing two long-overdue things: launching a blog (I had one in 1996 at HotWired, before the term existed, but not since) and writing a book proposal, both science-related. Neither is the road to riches, but they will give me avenues of expression that do not depend upon Wired, as this Nature review did. Thanks so much.

Heh, I was kidding about the "discredit" comment.

Yikes! I had no idea Nature was charging $32 for my piece. Those pay walls are just as annoying for reporters, by the way. Yesterday I must have spent three hours emailing the authors of studies I don't have access to (studies relevant to pieces I'm reporting), asking them if they would be kind enough to send me PDFs. Half of the time I'm sure my emails end up in their spam boxes, because I don't hear back from them, and then those studies (however important) don't make it into my reporting as they should. The end result? My writing suffers. It's soooo frustrating.

I had leftovers last night, by the way.

Also, thanks for the kind words about me. Hope you're feeling better today!

$32 for a PDF of a single article is definitely in WTF territory.

I don't have any copies of Nature on hand at the moment, but I do have a subscription to Science through my AAAS membership. According to the front cover of a recent issue I have on my "to read" pile (22 January 2010), the cost of a single issue of Science is $10. That's the entire issue, dead tree version, including shipping costs and retail markup. I assume that Nature has a similar price structure.

I get the idea of charging more for institutional subscriptions than for individual subscriptions (my primary scientific society, the American Geophysical Union, has been doing this since before I entered the field), but I don't see how they can justify a subscription price that exceeds the cost of buying single issues (in the case of Science, that would be $510 per year: weekly issues, but they take a week off at Christmas).

As with most publishers single article downloads at Nature are most probably priced for pharma reps who want reprints. "normal" readers are pushed towards buying a personal subscription or asking their library for an institutional subscription instead.

To add to Christina's comments, if your library does not have these titles and you need them, please do share that with your librarian. Also, see about your interlibrary loan options - we can often get our users electronic copies of articles from unsubscribed titles in just a day or two. Here, we subsidize ILL requests for things we don't subscribe to, so the article would be free to our primary users - you may have something similar (free or a cost

Sorry, that last one was cut off - you may have a free or reduced cost option available through your library even if they don't have a subscription.

And, I ran into an article a week or two ago for which the charge was $73! I don't know what the reasonable charge threshold is, but I'm pretty sure $73 is above it.

Abel- I note that, adding insult to injury, the book review costs more than the book!

I am surprised that you don't have off-campus access to those journals.

Firstly, I hope you feel better soon. I am a big fan of your blog and I was sorry to hear that you were feeling under the weather.

Second, as an undergraduate interested in medical anthropology and currently deciding on a grad school, access to medical journals is a major factor. My current school doesn't have a med school, and the schools in the consortium (DC area if you're keeping score) that do have access don't have the right to lend them to students from other consortium schools. I kind of thought that was the point of having a consortium but there you have it.

The thing is that I can get these articles if I physically go to the other school to pick them up--my AU student ID gets me into the GWU library--- but their library staff aren't allowed to email me their electronic version. The limitation doesn't prevent me from getting the article, it just makes it a pain in the #@! to get the article.

I mean, I guess I'm an example of the paywalls working, since schools buy subscriptions to attract students, but I don't like this tactic. It seems like they're annoying people into paying them, kind of like an electronic version of a bad street musician.

@culturegeek, you should see about inter-library loan (ILL) through your school. Today it is very fast, and you may be able to fill out the forms OnLine.

The same for Abel, even if your library doesn't have subscriptions, ILL is fast and free (to you, the cost comes out of the library budget, so the journal/author is not cheated). You may even be able to do it OnLine, from home, via a form or your e-mail account.

I don't mean to tell you stuff you already know. However, all the time that I did research I was totally self-sufficient in the library (pre- electronic era) and I had no idea of the services they offered. I thought librarians just checked-out books and pointed us to the correct shelves.

I'm fortunate to have off-campus access to journals...however, I have to use a different university than the one I teach at, so I use a friend's passcode to her university library (I review her stats and papers, and she provides me with her code). For whatever reason my university no longer subscribes to Nature or Science--I'm just sessional for a year so I haven't bothered finding out why they stopped subscribing to these journals.

However, it makes me wonder if Nature is slowly pricing themselves out of the market. When universities stop subscribing maybe you're doing something wrong?

I never know when or where (or if) my next contract is coming from so I can't afford to start laying out money for subscriptions. And that leaves me rather frustrated--I need to stay up-to-date in my field, I need to do literature searches to publish papers, but I can't access the very sources that will help me be more competitive (e.g. publishing more papers, doing more research) for full-time jobs. I can't even legitimately access the journals in which we just published two papers last year.

Without friends, contacts and generous authors who send me their pdfs I'd be more than frustrated, I'd be angry.

Actually, I am angry...companies, universities, gov't ministries would rather hire contract workers than replace retiring workers, and us contract workers have no benefits, no retirement plan, no job security, no way to plan a vacation (I might need to cancel and take that 1-year contract), and we have difficulty buying a house or even renting long-term because we move to where the jobs are (so if there's no field house or sponsored housing available, that means renting another place in another province, and most places want you to sign a 1-year lease, which is not feasible if your job only goes 2 to 8 months), and on top of all that, journals price their wares as if people had steady high-paying jobs (so do various training conferences, but that's another complaint...). Oh, and I didn't mention our gov't is cutting science funding for the environment, biology, natural resources, monitoring programs, climate......

Sigh. Ok, rant done. Think I need to change careers to medicine of some type. At least politicians recognize the need for health care.

Wow, that IS a lot of money. Outrageous! Also... Why not pay the writers a little better if they charge that much per article? o.O
Thankfully I live in student society housing here in Trondheim, meaning that the Norwegian University of Science and Technology is my ISP, and it appears they do have a subscription to Nature that works even off-campus. As a poor university student, there's no way in hell I could afford two articles like that, even as good as they were. I couldn't even justify the cost because while I'm an all-round geek and I love this sort of thinig, I'm still in computer science. Thanks for the links, and I think I'll go hug my University (or at least, go hug the Electro building on campus) tomorrow for letting me read interesting articles like that.

Also, more importantly: Get well :)

@Joe
First, thanks for the reply.
Unfortunately, the ILL system is part of the problem---I have to do ILL requests twice for virtually everything, because I have to do a CLS (local mini-ILL) request first for anything that any consortium school has. This request is often rejected because a school that is supposed to be sharing information with us, and has the article, is not permitted to share that information.
ILL is not particularly fast for articles from journals that guard their content so jealously either, because it takes a long time to find a library that has the right to make the journal available through ILL.

I maintain adjunct or clinical faculty status each year in large part so that I can have full library access. (OK, I also like to have students around, I admit...) The cost of articles is now such that I probably benefit more from the library access than the actual stipend.

Nature is pricing itself out of MY library's market, however, for some of their publications. I ended up subscribing to Nature Neuroscience on my own. We have ended up starting a little mini-library of specialty journals in the medical staff lounge. I think there are more NPG ones in there this year than last; I'll have to look.

I am mystified by the "one-year lockout for online access" for certain publication groups. Since I live many miles away from the actual campus and work weird hours, it's often difficult for me to make it to the physical library. I'm not going to use my students' time to fetch articles. So the outcomes are either a) I go to different articles, when the topic permits, or b) I contact a friend at a better-funded institution, and she or he finds the article for me, and I enlist him or her into the project.

I take the ILL desk people cookies or similar treats every month, though. I think being nice to them is in my best interests.
As to whether or not they sometimes come raid our staff lounge journal collection for stat-request/clinical-urgency-request articles, cough, I really couldn't comment. Other than to say they have the code to the lounge.
As I have online access, I would probably just donate my NN subscription, but I'm not sure if they're allowed to accept it, let students reproduce it, etc.?

On a related point, I just noticed that a scientific obituary was posted for a well-known cannabinoid researcher Billy R. Martin from the Medical College of Virginia who died in the summer of 2008 at the young age of 65. He was involved in some research I wanted to post on so I dialed up the obituary in the journal Neuropsychopharmacology.

But, alas, Neuropsychopharmacology is a Nature journal - that'll be $32 please - for an obituary.

@Abel,

Hope you're on the mend!

@Daniel,

If it helps, I know where your rant is coming from. Working independently is very hard.

@Silver,

I'm investigating something akin to this for myself as a solution for related reasons.

@All (and sundry, the cat, the dog, the lab animals, whatever),

As an independent scientist / consultant, access to the literature is an issue (and one I have meant to blog about for some time). Certainly buying the articles I need one by one given the costs the journals charge is out of the question.

Peer review problems: just in the Nature of things

revere — Wed, 03 Feb 2010 05:04:43 +0000

Peer review problems: just in the Nature of things

The Nature blog, The Great Beyond, has an interesting although not surprising report of accusations on BBC that a cabal of researchers has been impeding publication of important stem cell research to help themselves or help their friends:

Truly innovative stem cell research is being suppressed by a clique of peer reviewers for high profile journals, several researchers claimed today.
They told the BBC that the problem lies with those responsible for producing the reviews of research that journals such as Nature use to decide whether to publish the work.

Two scientists told the BBC they believe that in some cases reviewers are submitting negative comments or demanding additional and unnecessary experiments to delay publication and allow their friends to publish first. (Daniel Cressey, Daniel Cressey, The Great Beyond)

It's interesting because Nature's blog is in essence calling out Nature, the journal, although it's not the journal's sole responsibility. As a journal editor myself, it's not easy to police behavior of this type, assuming it's true. And that's the "not surprising" part. I don't know if it's true or not but if it were, I wouldn't be surprised. Getting published in a high profile journal like Nature is good for one's career in a way that publishing the very same paper in an excellent but lower profile specialty journal isn't.

For its part, Nature and similar Big Time journals like Science and The New England Journal put a premium on publishing results first. All three journals published so-so papers on swine flu because they were the first papers on swine flu. In some instances the papers added little to what we knew already. It was a race for high visibility and press coverage. These journals are too often interested in newsworthiness than science worthiness. They have active media operations and tantalize journalists and reporters with embargoed papers, making them think that because they are embargoed they are getting some kind of hot science news and with it visibility. The more a journal is mentioned in the news the more scientists want to publish there first, thus setting up the kind of dynamic at issue here.

Nature, Science and The New England Journal are among the most important scientific journals in the world. I subscribe to the first two and read the third at work. But they are also businesses. Sometimes business works with science and sometimes against it. This appears to be a case where the business side has indirectly worked against the science side. The indirectness, though, makes the effect more insidious and pervasive than just the stem cell story.

Remember you heard it here first. Well, not exactly. You heard it on Nature's blog, first. Just goes to show you.

revere Wed, 02/03/2010 - 00:04

Tags

Scientific Publishing

Correct Nature blog link here

They are the "top" journals for many values of "top" of course. But I have to say, in my practical work I more often than not end up not citing the Nature/Science paper but other, often later papers by the same group published in more specialized journals. Perhaps it's the relative lack of pressure or simply the larger available space, but those papers tend to simply be better written and more comprehensive.

On the subject of brevity, one scientist (solar physicist) I know mentioned he has to pay a page fee for his published papers. His last paper cost him 11,000 dollars ( Ithink the paper was 6 pages), so they try to keep the papers as short as possible. Not sure this applies to all journals, but you would think with the subscription fees, advertisements and pay per view, that charging the scientists for publication (paid for by government or private grants presumably) is a bit much.

Me thinks a poor patent officer today with a a paper on quantum physics and special theory of relativity would have trouble getting published today, unlike in 1905. Even if he got past the peer review (and their "who is this guy anyways"), he would not be able to afford to get the paper published (unless the page fee is the exception and not the rule).

"The most important scientific journals in the world ... are also businesses."

And as businesses, their editorial decisions may be unduly influenced by the profits to be made from drug companies who support the journals by spending on advertising and article reprints.

Not the best possible system.

The 50 Best Health Blogs

Journals that hide the peer-review process from scrutiny are just asking for these kinds of criticism. (And yes, I know that's all of them.) Transparency is the way forward, although researchers, being ultra-conservative, haven't realized it yet and will probably fight it every step of the way.

I think the day will come when article reviews are signed like book reviews are now.

Zen; It's not all journals. Many of the Open Access journals published by BMC practice Open Review. The journal I edit does as well. Reviewers and authors are known and the reviews are accessible to readers via a separate link on the website. Our experience, going back years, is that this improves the review process, making it more constructive and helpful and producing better papers.

@pft: pay for publish journals remain rare in most fields and often they are low voltage or highly specialized journals.

A somewhat muddled presentation here. Most papers I see as a reviewer for several journals, associate editor for one (spanning public health and social science) are awful. Most of the stuff that gets rejected outright should be, although I sometimes worry that junior investigators get discouraged too easily--many of their papers show signs of insufficent mentoring. When it comes to good papers, matters of preferred methodology do block publication, as does politics. I had a rather innocuous paper spiked because of someone who clearly didn't like the results. In another case, statistical analysis was the sticking point--obviously the same reviewer at 2 different journals, who wanted me to use an analysis that was completely inappropriate to the level of measurement in the data. Fortunately, I knew the editor at journal #2 and made a successful appeal.

This kind of shit goes on *every* *fucking* *day*!

In Medicine both the British Medical Journal and the PLoS Medicine practice open peer review.

The authors know who the reviewers are and vice versa.

It changes the dynamic and I personally like it. But I don't know if it's better for science or not.

I agree with comrade. And shit it is. Problem is it not only involves the journals, but grant reviewing process for the NIH etc. These folks bring home the grant proposals from the committee work that are suppose to be confidential, hand them out to their post docs, grad students, and quickly assign projects to the underwings with the intent of scooping the grant submitters idea or approach. Peer review needs to be revamped.

*Feigning shock/surprise*

Echoing Comrade PhysioProf sentiments - everyone involved in research has dealt with this behavior from reviewers/editors high on some kind of power trip! The only way to get around it sadly, as Rich mentioned is knowing someone higher up to appeal to.

Why isn't the integrity of a journal questioned when science and research takes a back seat in favour of other matters (conflict of interest anyone?!) Instead we jump through all these hoops and our motivation is unchanged in seeking to publish in these same journals.

Who is held accountable for delaying publications?

Why are there no measures in place that name and shame reviewers/editors who engage in this behaviour? Certainly, it's hard to prove until after the point (almost identical paper published a month later).

Punishment not enough?

What about rewarding editors/reviewers of journals with a better impact factor/waiving of page fees for submission/invitation to submit, etc.

I'll go back to dreaming...

@Revere: Check out this cartoon about publishing in Nature/Science from PhD Comics:

http://www.phdcomics.com/comics/archive.php?comicid=1201

Is it acurate of how editing works?

Alex: Only big journals can afford this. Most journals use their editors and associate editors to distribute to peer reviewers which they select from their editorial boards or rosters of peer reviewers or through authors' suggestions or looking at the references or through pubmed searches. Journals vary in size and editorial support, but most specialized journals depend on volunteer labor. Only the publishers make out and some of them make out like bandits. So a rich journal can do this, most journals don't. They get along with rudimentary editorial support and some help from the publisher.

Thanks for clarifying.

Hmmm...Aren't the journals pretty desperate for reviewers? I say volunteer your services for a while before submission. Then you case for appeal is much likelier to be approved.

@pft: "one scientist (solar physicist) I know mentioned he has to pay a page fee for his published papers. His last paper cost him 11,000 dollars ( Ithink the paper was 6 pages)"

What journal? I know that ApJ, which is high-impact for solar physics, charges $110 per page, $350 for color. Solar Physics, the journal, has no page charges. Who charges $1800 a page?

Randomized trial versus observational study challenge, V: pre-randomization

revere — Tue, 12 Jan 2010 05:15:31 +0000

Randomized trial versus observational study challenge, V: pre-randomization

[Previous installments: here, here, here, here]

We'd like to continue this series on randomized versus observational studies by discussing randomization, but upon reviewing comments and our previous post we decided to come at it from a slightly different direction. So we want to circle back and discuss counterfactuals a little more, clarifying and adapting some of what we said for the coming randomization discussion.

Let me change the example to a more recent controversy, screening mammography for breast cancer. Should women under 40 get routine screening given that there is said (on the basis of RCTs) to be small benefit on the one hand and on the other, putative or real risks? I don't want to get into the actual details of this dispute but just look at the logic without concerning ourselves with the size of the benefit. Let's simplify to asking if there is any benefit to mammography at all. This is like the question about our blood pressure drug: does it work? I'm changing the example because I think what is involved is easier to see when we talk about screening.

Consider what this means in the case of Jane Doe who is considering whether to get a mammogram or not. She wants to know if it will prolong her life in quantity and quality. Let's do a thought experiment. The ideal experiment would be this. We screen Jane Doe, follow her for a number of years and then use an appropriate measure of outcome, let's say age at death (we are not only including breast cancer death but risk of death from any cause). Then, we turn the clock back and not screen Jane Doe and do the same thing and compare them. If they are different, most of us would say that's what it means for "mammography to work."

In the real world, though, only one of these scenarios can happen. The thing that gives it meaning for causality, the comparison with the impossible anti-trial, can't happen. This is a conundrum. We need a work around. It won't be perfect but it's the best we can do. We will claim that there isn't just one work around but many and you use what you can in terms of feasibility, ethics and resources.

So what are the work arounds for what seems an insoluble situation? Jane Doe can't be both screened and not screened. They are mutually exclusive. For screening, unlike our blood pressure example, there is no possibility of using a cross-over design, i.e., first screening her (or not screening her) to see what happens for a few years and then not screening her (or screening her) and then seeing what happens for several more years. Her risk changes with time so the later Jane Doe is clearly not equivalent to the earlier one. We don't have an identical Jane Doe. Even if she had an identical twin, Alice Doe, Jane and Alice will have the same genome but different histories, making them differ in many ways (e.g., one might be a radiologist and the other an accountant; one might live in Denver, the other in Charleston, SC). Not even their genetics will be identical because the genome becomes modified after birth. These "epigenetic" changes are like changing the Preferences on a software program. The underlying program is the same but two users might set things up quite differently after opening the shrinkwrap. The problem with a whole bunch of Jane Does (a population) is no different than one Jane Doe. We can't both screen them and not screen them. At this point many of you will want to have two populations, one screened and one unscreened and compare them. That's obviously where we are heading, but before we do, let's stay with the counterfactual problem just a bit longer.

What if we could turn the clock back for a population of Jane Does? What would we look at? We want something that measures the relevant (for our purposes) differences between the screened and unscreened population. Epidemiologists are adept at finding these measures. It might be total mortality after a suitable follow-up period (incidence proportion), survival after screening, breast cancer mortality per person year of observation (incidence density), etc. Which one we choose may be subject and setting specific and let's not worry about which one we settle on. We are only interested in the difference in the measure between the screened and unscreened population after turning the clock back. That difference is called a causal contrast (or effect measure(!) or causal parameter). Say we are using the arithmetic difference as the causal contrast. Let's choose risk of dying from breast cancer and call the risk when the population is screened R1, and R0 the risk when it isn't screened.

We can observe only one of these, however, because in the real world we can't turn the clock back. While we want to measure (R1 - R0), we can only observe one of R1 or R0, not both together, so we can't measure what we want, the causal contrast (the measure of effect). Faced with this, we try to do the next best thing: find a substitute for the unobservable counterfactual population (the one that was unscreened). So far we haven't said anything about randomization and for good reasons. This is a general framework for all kinds of studies about whether something works or causes disease (etiologic studies). In experimental situations, like clinical trials, the investigator gets to assign the treatment (screened or not screened, drug or no drug). In observational studies the assignments are given to us or the contrast is with some prior experience (e.g., Rind's HIV or rabies examples mentioned in our last post or our fictitious blood pressure trial). Study design differences are differences are related to how we choose the original population for the study question (the Jane Does), how we choose an appropriate substitute population for the causal contrast (the pseudo Jane Does), how well these decisions represent the world of Jane Does and then all sorts of ancillary decisions about related to costs, study time and other technical factors.

The sampling problem is an added complication. The study group of screened Jane Does is meant to represent the world of Jane Does for whom we want to know if screening is a good thing. The way we have chosen them might or might not be representative -- and here that means "good stand-ins" -- for that larger group. The same for the unscreened pseudo Jane Does. Let's call the larger group the target population. So there is a double substitution going on here. One is the pseudo Jane Does (unscreened) for the real Jane Does (screened). The other is both of these populations for the bigger external target population. None of this (so far) involves randomization. It just involves the notion of substitution of one population for another so we can observe something unobservable (in one case the entire population, in the other the counterfactual), thus allowing us to get a causal contrast. If either or both of these substitutes don't represent the target population or the counterfactual pair, then we run the risk of misreading the contrast. Epidemiologists call [the non comparability of the counterfactual substitute] confounding (or we say the causal contrast is confounded; this is a more general notion of confounding than seen in many textbooks but amounts to the same thing). What it means in plain language is that when we used an imperfect substitute we weren't really getting an accurate picture of what we would have seen if we'd been able to turn the clock back. We aren't seeing what we want to (but can't) see. Our "work around" was faulty.

Let's consider our blood pressure trial (see previous posts here, here, here, here). The substitute for the same patient but without treatment by the drug was in fact the same patient at a previous time when they were being treated with the usual therapies and not responding. This would seem to be a pretty good substitute, although one can conjure up reasons why there might be confounding (see the comment threads and also the post by David Rind in Evidence in Medicine). Most of those problems could probably be remedied with a technical fix for this unblinded single-arm trial and wouldn't require randomization. What is the target population? It could be all refractory hypertensives or just refractory hypertensives in this doctor's practice or none at all, just a report of what happened to these patients. So the double substitution is visible here, too.

A randomized clinical trial (RCT) is another kind of "work around" for the counterfactual problem. You still have to worry about how good the substitute for the counterfactual is and how representative a substitute the study population is for the target population. Thus when you study seasonal flu vaccine effectiveness in the elderly with an RCT, they are a substitute for the elderly in general. If you extend that to swine flu vaccine in the young, you are changing the nature of the substitute. That may or may not be a reasonable thing to do. It requires justification. You have to check what the target population is and that the study population provides a fair substitute for it. Thus even a pristine RCT isn't a pure gold standard, but an alloyed gold standard. You always have to check how much base metal there is.

Although randomization doesn't deal with the target population substitution, there is a reasonable expectation that it will make the counterfactual substitute (the pseudo Jane Does for the screened Jane Does) roughly comparable, i.e., that the substitute will be a good one. Even here, however, complications arise independent of randomization gone bad (i.e., that by odd chance the two groups will not be comparable on some factor of importance). This requires a longer discussion, though, and is best delayed to the next installment. How many installments will there be? I have no idea. I am just following the argument as it goes. I'm surprised it has taken this many.

If you are interested in counterfactuals, this article is useful: Estimating causal effects. Maldonado G, Greenland S. Int J Epidemiol. 2002 Apr;31(2):422-9. It's a deeper subject than most people give it credit for. As for the next installment, you don't have to wait. The comment threads are open.

revere Tue, 01/12/2010 - 00:15

Tags

Scientific Publishing

I am liking your discussion on random and counterfactuals very much.

Yet, it would be of help to me, and many others, to see how you would explain random versus fixed effects models. We seem to be having a real upsurge in folks who are taking samples in the environment that are not random and they honestly don't understand the difference between fixed versus random effects so they generalize (inappropriately) with fixed effects models just as they would with random effects models.

Please go into the diffferences in the way variances are partitioned in fixed versus random models and please be very concrete in your examples.
dh

Dwight: First, thanks for the kind words. I'm not sure how much I want to get into the weeds on study design. I have another end in view and this has gotten unexpectedly long already. But I'll see. I'm still not sure where it is taking me.

Maldonado and Greenland mention early in the article that a relative risk is not necessarily a biological constant; this is worth emphasizing too. The half life of a chemical isotope is assumed to be a physical constant (otherwise the dating of a carbon-containing compound would be futile). But the ârelative riskâ of an âexposureâ may not be an analogous biological constant. Even if the substitutions for the counterfactual were perfect, and the external validity were perfect, the estimation of the relative risk could be subject to variation, depending on the distribution of other factors in different populations.

As they mention at the end of their article, this means that meta-analysis may not be so much an estimate of a fictional âcommon effectâ as it is a search for sources of systematic variation among study results. Making âmeta-analyses with homogeneityâ a âgold standardâ for medical evidence could, in this reading, be a bit of a fallacy. Since the homogeneous meta-analysis has an oracle-like status in many interpretations of the hierarchy of evidence for EBM, Maldonado and Greenlandâs article could reframe the meaning of this hierarchy.

To me, that seems like a pretty big deal, especially when some people get all persnickety about what constitutes good evidence for a medical hypothesis.

Ed: I'll have to dig up my copy, but a RR isn't a "property" of an organism or even an exposure and an organism. It's the product of a comparison and depends on the reference group so could hardly be a biological constant. But maybe I'm misreading what you were saying.

As Obama says of Harry Reid, my expression was inartful.

The fact that RR depends on so many aspects of the comparison being made has consequences for the interpretation of meta-analyses. Homogeneity and consistency are not the same thing. The consistency of an association across populations should not be viewed as necessary for a causal relationship.

Generally, when meta-analysis is described to general audiences, it comes across as an attempt to pool the association measures of smaller studies into a single association measure with a smaller standard error than those of the individual studies. This is what I read the article to be rejecting when it speaks of meta-analysis as a search for sources of systematic variation rather than "an exercise in estimating a fictional common effect."

To me, this interpretation of the Maldonado and Greenland paper suggests that it is a mistake to place meta-analyses with homogeneity at the top of the hierarchy of study designs for evidence in medicine. I have heard learned people speaking as if heterogenity somehow weakened the evidence for causation. This article suggests that this need not be true. Homogeneity may arise from pooling studies of similar populations with similar distributions of potential confounders, an artifact of various investigators' research agendas rather than of truth itself. That is worth remarking on.

Ed: Ahh. Yes. There are two uses of the term meta-analysis. One of them is indeed a pooled analysis. This is sometimes possible but not often. The other is an observational study of experimental studies, and that's usually what the Cochrane Collaboration produces. It is indeed what we will have to examine. You have discovered some of where I am going.

revere,
I liked your original post but I've been getting more and more restless since. All the discussions analyse the original study as if it were the end of the investigation. It should be the start. You have a hypothesis. You have some evidence to support the hypothesis. You don't have a clear biological mechanism. There are some problems with the design, as you correctly show there are problems with all designs. You don't have a huge effect or overwhelming statistical support. What to do? Get more data. Do more studies.

In the real world of medical research, wouldn't at least a few cardiologists in teaching hospitals somewhere in the world follow up this work? Wouldn't one try four groups (placebo, one pill a day, two pills a day, three pills a day)? Wouldn't another do a double blind randomized test comparing your treatment to alternate treatments? This is certainly what happens in other areas of scientific research.

ECaruthers: The assumption was there was a biological mechanism. That's why she tried it on her patients. So you can consider this confirmation of the biological hypothesis. How do you know there wasn't good statistical support? I didn't give you the results of the F-test. And what do you consider "good statistical support"? Should any result, whether an RCT or observational study be confirmed and replicated? Of course. The question here was about the logic of this trial and we still have more to explore. If you follow up on this, who is going to pay for the follow-up? Doctors in teaching hospitals can't do this without money. It's quite expensive to do an RCT. Does it happen in other areas of scientific research? All the time? I rather doubt it. Every paleontology finding? Every botanical finding? Unlikely.

revere,
I understand that your main interest is in the the logic of your test compared to other tests. But you intentionally set up your hypothetical to be suggestive but not conclusive. No one responded by saying this should become the new standard of care for similar patients. I actually agree with you in believing the results shouldn't be dismissed.

When I suggest 'do more studies' I don't only mean RCTs with clinical endpoints. I include exactly the kind of follow-up that David Rind described after the initial report of the effects of ritonavir and that quickly developed into triple therapy "cocktails" for HIV.

Your hypothetical 'magnitude of effect' is much smaller than in the ritonavir tests, so I wouldn't expect as many scientists to follow up. But every scientist I know just loves a new result in his or her area. Even a theory that the community strongly disbelieves gets tested by someone, just to say it's been disproved. See http://vosshall.rockefeller.edu/reprints/KellerVosshall2004.pdf

ECaruthers: It's possible for other docs to do the same off label thing, but actually setting up a trial, RCT or not, costs money and it has to come from somewhere. So while someone might get funded for this, it would take a while. But I agree that confirming findings is critical, although it is done much more rarely than you'd think.

revere,
Perhaps you already intend to discuss the comparitive costs of different study types and experimental designs. If not, I would suggest it as an additional future topic. If ramdomized controlled tests of drug effects are so expensive that they are only done when funded by drug companies, then it seems to me that you have another argument for alternative types of study.

Randomized trial versus observational study challenge, III: metaphysics

revere — Fri, 08 Jan 2010 05:56:54 +0000

Randomized trial versus observational study challenge, III: metaphysics

Let me start with an apology. This post is again fairly long (for a blog post). Blog readers don't like long posts (at least I don't). But once I started writing about this I was unable to stop at some intermediary point, although I might have made it more concise and less conversational. I haven't done either. Even worse, I didn't quite finish with the single point I wanted to make, so it will be continued in the next post. Hence the apology. Now to recap a bit and then get down to business.

My "challenge" from 10 days ago has drawn quite a response: over 40 quite substantive comments on the two posts (not counting mine) and a long and substantive post on the issues I raised from David Rind on his blog, Evidence in Medicine. It has also challenged me, since you have all raised many issues that deserve comment. Since I can't comment on them all, I am going to pick certain themes that appear to me as more fundamental or that implicitly underlie many of the concerns you have raised. I'm going to have to do multiple posts, so if you don't see your issue addressed, even obliquely in this post, it may very well appear in a subsequent one. If you are coming in in the middle of this and want to catch up, you can go back to the two previous posts (here and here) and Rind's observations (here). But if you don't want to do that or want a reminder, here's the basic idea.

In my challenge I described a fictitious inquiry made by a cardiologist who had reason to believe, on the basis of emerging scientific literature, that an acceptably safe and FDA approved drug for epilepsy might be useful in treating refractory hypertensives (people with high blood pressure who do not respond to any of the conventional treatments). The science suggested this would work for a certain subset of refractory hypertensives, so she drew up a list of criteria obtainable from her practice's medical records and tried it by giving the drug to patients who met the criteria, measuring their blood pressure at baseline and after a month or so of taking the medication. There was a statistically and clinically relevant drop in blood pressure in her convenience sample of 29 patients. If you read the original post you will find more details of how this fictitious cardiologist went about making information collection systematic, reliable and relatively unaffected by the fact that this was an unblinded trial -- both the blood pressure measurer and the patients knew they were being treated. There was no placebo group. My challenge was for you to say whether it was reasonable for the cardiologist to alter her practice on the basis of this single arm, non blinded, uncontrolled, non randomized small trial of a convenience sample. I raised the stakes by saying she later published this clinical trial in a peer reviewed medical journal and hence it might also affect the practices of others.

I had two motives for this particular construction, one of which I will keep for later. The other was to use the polar opposite of what is considered the gold standard for medical evidence -- a double-blind randomized controlled trial (RCT). The question was not whether an RCT was the best form of evidence (something that seems to be assumed by many people although it isn't always true). It was whether you could rely at all on anything from an inquiry which had none of these characteristics.

My first move in this game will probably be unwelcome to some (most?) readers. I intend to make a tiny foray into the field known as the philosophy of science. I hope it will be interesting and not to onerous. A disclaimer: I am not a philosopher of science (although I confess to have given a couple of papers in venues inhabited by this academic species). To the extent that it informs my practice (and it does), it is always from the perspective of a scientist. But there are some philosophical questions that should not be avoided, if only because they are always there and avoiding them actually commits you to one or another view, sometimes multiple and incompatible ones. The philosophical question I want to raise is fundamental to science and the challenge itself, the nature of causation. There is a big philosophical literature on this, only a fraction of which I am familiar with, but much of it is so abstract and technical I can't relate to it as a practicing scientist. So instead I will reframe the questions in the terms we meet them everyday in the clinic or laboratory or daily life.

First let me anchor the question in the challenge example. The underlying question is whether this anti-epilepsy drug is "doing anything" to lower blood pressure (we will have to deal later with the ultimate objective, helping the patient). At first blush, it appears to do something, but many of your comments and critiques give reasons for why what seems to be true (after patients took the drug their blood pressure went down) may not in fact have anything to do with the drug. This is a causation question. Did the drug cause the decrease in blood pressure or is there some other explanation? What do we mean by this question?

Let me pause for a moment to call your attention to something that is seldom mentioned. What philosophers call "causal necessity" is a very weird animal. We are used to hearing that epidemiology can only show association, not causation. This always infuriates me, because while not false, it is true of every empirical science. Whether it is chemistry, physics, molecular biology or whatever, all we see is associations. The correlations or associations we see in the field, clinic or lab don't come with labels on them that say "causation." We have to infer, i.e., make a judgment, that certain associations are of a particular kind we like to call "causal." The philosophy of science, and more specifically what we call "scientific method," is about how we make those judgments and what warrants making them. Said another way, scientific method is about how to trick Nature into revealing to us which associations are causal and which aren't.

There is both a way in which the meaning of causality seems obvious and a way that it isn't obvious and we are able to see them both simultaneously in the form of a "yes-but" response. One of the (many) problems with the notion of "cause" is that the word has multiple meanings and connotations and we often don't keep them separate. Consider this from a famous philosopher of the law, HLA Hart (in work with Tony HonorÃ©):

Human beings have learnt, by making appropriate movements of their bodies, to bring about desired alterations in objects, animate or inanimate, in their environment, and to express these simple achievements by transitive verbs like push, pull, bend, twist, break, injure. The process involved here consists of an initial immediate bodily manipulation of the thing affected and often takes little time. (Hart HLA and HonorÃ© A, Causation in the Law, Oxford University Press, 1959; h/t DO for cite)

While this is a very intuitive notion of causation it isn't sufficient (or necessary) for scientific investigation. But it is often lurking in the background and carries with it both connotations of "agency" (an agent that does something) and responsibility (what got done is due to something). Assuming that cause includes natural laws and not just human agency, it also raises the question of whether every event has a cause. Philosophers and scientists have tried to sharpen the notion of causation for better application at least since Hume's time (18th century and the "scientific revolution") and arguably before. I'll just add two other versions, since they are the ones that we scientists seem to use most as our unspoken mental models (if we have a mental model; a lot of science is done by rote, unfortunately). Both of these versions are directly applicable to the example. Unfortunately when you lift up the hood to see what's inside, you immediately see deep problems, a realization which makes most scientists quickly slam the hood down again. Nothing to see here. Move along. We choose to assume there are or have been experts who know how that mechanism works and we don't want to be bothered with metaphysical nitpicking. But I want to bother you with it, at least a little.

So consider these to typical scientist models of causation:

Version I: A causes B if, all things otherwise the same, a change in A brings about a change in B.
Version II: A causes B if, all things otherwise the same, but for A, B wouldn't happen.

Version I. is the classic form of an experiment. Version II. is often the way we think about observational studies, although it also applies to experiments. In either case we have the problem of "all things otherwise the same." Version II. also has the counterfactual problem: it relies for its content on an event that didn't happen (that A didn't occur and therefore B didn't occur but everythng else was the same). Neither version requires placebo controls, randomization or blinded measurement, except insofar as they try to deal with either the association itself, "all things otherwise the same" or the counterfactual condition.

Note that there are two comparisons here: one is the whether A and B are associated (move together); the other is related to whether "all things are otherwise the same" (where the opposite of "the same" is "different," that is, a comparison between two states of affairs). Placebo control is more related to the A and B association, especially in the sense of Version I., as is blinding (it relates to the validity of measurement). Randomization is more related to the second association (are all things otherwise the same). However there are interacting influences, so this isn't a hard and fast distinction.

The most mind bending problem here for most people (once they think about it) is the counterfactual condition. Suppose patient A has refractory hypertension. I give him the anti-epilepsy drug. There is no meaningful change in blood pressure. What are the possibilities? One is that the drug has no effect on this patient. Another is that something else raised his blood pressure so the effect of the drug was masked (the something else could be measurement error or some external real effect). On the other hand, suppose his blood pressure changes. This could be because he is responsive to the drug (the drug "works") or because of some other factor (which our commenters have been quite ingenious in positing). In either case we have the possibility of a causal effect or a non-causal one. How do we decide?

The two favored techniques by commenters are placebo control or a cross-over design. A placebo control doesn't completely address the "all things otherwise the same" criterion of an experiment because there is one other thing that is extremely different: the placebo is given to a different person. A randomized placebo controlled trial doesn't get around this. Instead of giving it to one different person it just gives it to a whole group of different people. Randomization doesn't make things otherwise the same, either. It just makes the two groups approximately the same on average. But the counterfactual or "otherwise the same" criterion isn't about groups, it is about individuals. That's why a persistent concern about RCTs in general and meta-analyses in particular is that they might hide responsive or susceptible groups in the average.

A cross-over design is one where the person stays the same but the treatment changes. In this case we might wait a month or two for drug's effect to dissipate and then see if the patient's blood pressure goes back up. This has the advantage that it is the same person so more "other" things are the same. But of course over the span of time other things might have changed and there are still those concerns about whether lack of blinding might affect how good the measurements are. Also observe that the original challenge design is implicitly a cross-over design and is in an important sense also controlled for alternative treatments, because these were people who had been treated with medications by the same cardiologist but the medications didn't work. One might argue there is no need for a placebo because we have evidence of something even better, comparison with a medication known to work for blood pressure control.

Note also that a cross over design requires the ability to see what happens if you don't treat someone. For example, consider a new chemotherapy drug where we are measuring effect by 5 year survival. If we treat someone with the drug and they die much later than expected, we don't know if was the drug or they were just lucky. We don't have a do-over where we don't treat them with the new agent to see of they would be a long term survivor regardless. That's why the counterfactual is a bit mind bending. It depends on something we can't ever see directly. We have to infer it by trying to find a "substitute" for the treated patient to see what happens if they aren't treated.

I was hoping to finish this subject in one gulp, but this post is already too long, so I'll have to finish the subject of causation in the next installment, where I will connect it up with some comments of Rind's on the importance of size of the effect, although I will take a different tack.

But that's for next time. Assuming you are still hanging in there with me.

revere Fri, 01/08/2010 - 00:56

Tags

epidemiology

New drugs

Philosophy of Science

scientific method

Scientific Publishing

The Tamiflu doesn't work non-story

revere — Thu, 10 Dec 2009 05:03:02 +0000

The Tamiflu doesn't work non-story

The other day the British Medical Journal (BMJ) published a clutch of articles about whether Tamiflu was as useful a drug as some have touted. I read the main article, another one of the Cochrane Collaborative meta-analyses of the studies they deem useful about any particular subject, and it didn't seem to make much news. It confirmed what their previous review had said about the neuriminidase inhibitor antivirals for influenza (Tamiflu and Relenza): these drugs work but their effect is modest. We've been saying the same thing for years here, not because we did a fancy meta-analysis, but because that's quite clearly what the literature said. They confirmed it. Again. Not very interesting, I guess, so the BMJ, quickly becoming medical tabloid central, fastened on the one scientific aspect of the paper that might remotely have a news hook: the meta-analysis didn't have enough information to show -- according to Cochrane Collaborative standards, that is -- that healthy people who got flu and were given a neuriminidase inhibitor avoided more serious complications like pneumonia. It didn't show the antivirals didn't work for this. It just alleged there was no Cochrane-required-level-of-evidence they did. The data -- in their hands -- showed the evidence was compatible with either outcome. Yawn. But yawn was good enough. It was elevated to a different story: that one reason we didn't know is that the drug companies were hiding the data. That is a news story, I agree, but not a news story about whether the drug works or doesn't. While just an allegation (because they didn't get to see the data), the medical journal was doing this in collaboration with television channel 4 in the UK and the Cochrane Collaboration itself. Conflicts of interest?

The paper was co-authored by, among others, Dr. Thomas Jefferson and graduate student Mr. Peter Doshi, both of whom I have criticized here on one occasion or other. I'm not making any serious complaints about this paper (although why, on scientific grounds it should have been published in a high profile journal isn't clear to me since it didn't provide new evidence), but I do know enough about this kind of work to know that a great deal of judgment is used in accepting or rejecting papers (indeed that's why this follow-up was done; someone objected to a paper that was considered in the previous review). However the overwrought point-counter-point between drug maker Roche, the authors, BMJ reporters and the editors over access to the data and who was supported by whom had the effect of entangling use of an important class of drugs for influenza -- a class of drugs that everyone seems to agree works to some extent when no other does (and during a pandemic, no less) -- and the important but unrelated issue of transparency over data involving pharmaceuticals. Let me be clear that on this issue I am on the BMJ's side. I think it's a scandal that we don't have access to information used to license drugs given to the public with an official sanction of safety and efficacy. But it's an issue that would likely crop up with almost any drug they sought to examine. Doing this in tandem with a media outlet whose objectives are not science but snagging viewers and the Cochrane Collaborative itself is unseemly at best and borders on the unethical. It makes it look like the BMJ was again engaging in self-promotion (OK, I understand it's a business enterprise, but let's recognize what's involved). It didn't hurt that the self-promotion seemed to serve everyone's purpose (except for Roche's, and frankly I can't bring myself to feel sorry for them).

Well, maybe not everyone's. I don't think it served the public purpose or the public health. Tamiflu and Relenza work. They are in fact they only therapeutic modality we have other than supportive care or in critical cases heroic methods (mechanical ventilation). Vaccines are preventive, not therapeutic. So here we are in the middle of a pandemic and the Cochrane folks, aided and abetted by the BMJ and television producers, are saying, "How do you know for sure that they will prevent pneumonia in an otherwise healthy person who gets flu?" There is evidence about this, even if the Cochrane zealots don't recognize it:

And observational data on these drugs' usefulness in patients hospitalized with severe cases of flu - seasonal and H1N1 - points toward benefit, Dr. Tim Uyeki of the U.S. Centers for Disease Control recently reported in the New England Journal of Medicine.

"We don't want people to stop using the antivirals in the way that we're recommending they're used because we believe they are having a beneficial effect on hospitalization, on severity of illness and indeed (preventing) death," [antiviral expert Charles] Penn said Tuesday from WHO headquarters in Geneva.

"And if people stop using them, then the consequences of that will be an increased burden on the health-care system and worse outcomes." (Helen Branswell, Canadian Press)

But as long as we're talking about the need for randomized trials -- something I'm not against but know enough about not to think they are automatically better than observational studies -- let's really talk about the need for them. The Cochrane Collaborative exists to do systematic reviews of the medical literature for the purpose of improving medical outcomes (note that they set their own rules for this). To be at all effective they have to be read and understood and used by practicing clinicians. Do we really know that clinicians who follow the recommendations of the Cochrane Collaborative have better outcomes for their patients? Isn't it possible that some Cochrane recommendations actually do harm by inhibiting the use of treatments that are helpful in certain cases? Remember that their judgments, even if valid, are about average effects. If we wanted to find out if anyone should pay attention to a Cochrane review, shouldn't we subject them to a randomized clinical trial? It would be easy. We could give a randomized groups of physicians real and sham Cochrane reviews (no one can really understand how they are really done from the descriptions in the papers anyway so the placebo ones would look just like the real ones). Then we could see if they do result in better medical practice.

And I know just who would publish it.

revere Thu, 12/10/2009 - 00:03

Tags

Antivirals

big pharma

epidemiology

Pandemic preparedness

Publishing

Scientific Publishing

Categories

Medicine

Ah. What would we do without you, Revere? Love it.

I am interested in hearing more about the idea that an RCT is not automatically better than an observational study. This warrants greater discussion. In addition, more comments about the Cochrane way of doing things would be helpful. I have had some concerns about Cochrane, and just what it takes for them to say that an intervention is effective. If you have any links to these topics, they would be of use to many readers.

I could not disagree more. This so called "argument" about clinicians basing judgments on "averages" is ridiculously unfair because you are not complaining about the MANY MANY times greater problem of clinical interventions that use those same "averages" to do something.

Opportunity costs and negative outcomes from interventions are real whether those intervening admit responsibility or not.

There are ways to continue to make evidence better and the Cochrane Collaborative is not perfect. The above does not help.

Also the ridiculous ahistorical nature of your complaints about statistical bias against medical interventions. I guess you do not know about the thousands of times medical leaders, academic institutions, research orgs, and big pharma have cooked the books (or is that so boring nowadays that it is conveniently not worth mentioning).

I amazed constantly about the hypocrisy of people who always fight against the "un-science" of "alternative" and even "anti-" medicine who have convenient blinders on about the medicalization of every aspect of life and the constant "up-selling" of disease and intervention. They are both problems but the relatively minor one gets all the blows. That is bullying in my book, but everyone seems to want to be a bully these days.

Thank you, revere. You've hit it right on the head, on every single point. I was going to write something about this, but now all I need to do is say "Ditto"!! ;-D

Additional information - there are at least 2 studies that show reduction in mortality for hospitalized patients in seasonal flu.

Antiviral therapy and outcomes of influenza requiring hospitalization in Ontario, Canada.

Treatment with antiviral drugs active against influenza was associated with a significant reduction in mortality (odds ratio, 0.21; 95% confidence interval, 0.06-0.80; n=100, 260).

Severe Human Influenza Infections in Thailand: Oseltamivir Treatment and Risk Factors for Fatal Outcome

Treatment with Oseltamivir was statistically associated with survival with a crude OR of .11 (95% CI: 0.04-0.30) and .13 (95% CI: 0.04-0.40) after controlling for age.

Not all useful information has to come from randomized controlled trials. It's the sum total of information from different sources that is often the most valuable.

Floormaster: I could not disagree more. This so called "argument" about clinicians basing judgments on "averages" is ridiculously unfair because you are not complaining about the MANY MANY times greater problem of clinical interventions that use those same "averages" to do something.

I don't think you understood my point. The "averages" I referred to were the meta-analysis estimates, not those used by clinicians. So using your argument, all the more reason to require an RCT for Cochrane. Notice that is where my point was headed, so you are supporting it.

Opportunity costs and negative outcomes from interventions are real whether those intervening admit responsibility or not.

Again, you missed the point. You are supporting a demand for an RCT of Cochrane. I said nothing about medical interventions being harmless. On the contrary, I have good reason to know how harmful they can be and how often it occurs.

There are ways to continue to make evidence better and the Cochrane Collaborative is not perfect. The above does not help.

So shouldn't it be submitted to an RCT? Does it really make medical practice better? Can they (or you) support the claim? Or is what's sauce for the goose not sauce for the gander?

Also the ridiculous ahistorical nature of your complaints about statistical bias against medical interventions. I guess you do notknow about the thousands of times medical leaders, academic institutions, research orgs, and big pharma have cooked the books (or is that so boring nowadays that it is conveniently not worth mentioning).

Not only did I not make such claims, but I explicitly said I agreed with BMJ that the data secrecy is a scandal. You may either wish to re-read, or if you are confident you read correctly, point us to where you think we said anything like what you claim.

I amazed constantly about the hypocrisy of people who always fight against the "un-science" of "alternative" and even "anti-" medicine who have convenient blinders on about the medicalization of every aspect of life and the constant "up-selling" of disease and intervention. They are both problems but the relatively minor one gets all the blows. That is bullying in my book, but everyone seems to want to be a bully these days.

I don't think we can be accused of any of this. But if you have specific instances, rather than blanket accusations, we'd be glad to discuss them with you. We hardly ever talk about CAM or quackery here, and myself have tried acupuncture (as far as I could tell it didn't help, but neither did anything in the medicine cabinet). We have been involved in very public disputes with Orac over the meaning of scientific method and it is always been our position that the "demarcation problem" (how do you separate science from so-called pseudoscience) is unsolved. We consider astrology a science (it makes testable claims and is based on theory). It's just that the claims it makes are false (they have been tested in large scale data studies about things like vocation and birth sign), so as science it is bad and should be discarded. The theory also has no basis in what we know today either. But that just makes it bad science, not pseudoscience. But apparently you have a different view. I'd like to know what it is. Can you tell science from pseudoscience, or perhaps more to the point, in over five years and 3400 posts written here can you find a single instance where we claimed to be able to do so?

As for the medicalization of everyday life, you will get no argument from me. When our children were born the battle cry among progressive young parents was not to medicalize pregnancy. Now that our children our having babies, it is a hundred times more medicalized than ever before. Mrs. R. and I just shake our heads. So we too, are amazed, but you seem to be amazed at us. Perhaps, again, you can give me examples, because it is also one of our concerns and we do not wish to feed it. But if you want absolute zealots on the issue of what is proper and what isn't in medicine, look to the Cochrane Collaborative, not us.

Any comment regarding the quality of these numbers:

CDC: Nearly 10,000 deaths linked to swine flu
About 50 million Americans have been infected with the H1N1 virus

It was 4,000 last month. A suspicious person might imagine the numbers were being massaged in order to justify the flu "emergency"

Our favorite journalists, Shannon Brownlee and Jeanne Lenzer, have come out with a story about this - published just today on Atlantic Online. I was waiting for them to weigh in on this important matter, since it seemed just up their alley, but they did it sooner than I expected.

Here's the link: http://www.theatlantic.com/doc/200912u/tamiflu

I just read the first paper posted by SusanC @5, which was funded by Hoffman-LaRoche. What I thought was interesting was that 71% of their subjects (those admitted to hospital for influenza) had received the flu vaccine. Maybe some RCTs to establish vaccine efficacy would be worthwhile.

Thank you S. Lakshmi for that link. An excellent article.

Sid Offit: As of today, 10,402 H1N1 deaths have been reported worldwide, according to the European Centre for Disease Prevention and Control. The number of people infected is at best a rough estimate. According to WHO stats, seasonal influenza kills 250,000-500,000 people annually. It would take a lot more number massaging to justify all the media attention and money that's been given to the flu this year.

(Those statistics and sources are all available from wikipedia - "2009 flu pandemic")

Revere: Evidence-based medicine is the most reliable, and double blind, randomized controlled trials are the strongest form of evidence we have. We can't keep recommending drugs just because we think they work to some extent. These drugs have side effects. Evidence of their safety and efficacy is required to justify their use. You sound a bit like a homeopath.

"And if people stop using them, then the consequences of that will be an increased burden on the health-care system and worse outcomes." (Helen Branswell, Canadian Press)

Not if they cost a fortune and they don't really work.

Having been able to peak into the sausage factory of what goes on in the creation of a Cochrane Systematic Review, I'd like to endorse the worries expressed in this post.

It also chafes me when a Cochrane Review allows for an "Editor's Conclusion" where words like "Promising!" or "Merits further research!" are bandied about for treatments which clearly flop if you even just quickly skim the data. The CAM reviews seem particularly guilty of this editorial legerdemain, but they're not alone.

I'd *love* to see an RCT on the effectiveness of Cochrane meta-analyses and systematic reviews. And you know what, I bet Archie Cochrane himself would have been very open to the idea. Then again, he was Cochrane and not a Cochraenist.

Quinn: I guess that means that sciences that don't use experimental method (seismology, astronomy, natural history, cosmology) don't have reliable evidence. A good observational study is better than a bad randomized trial any day. And there are a lot of bad randomized tirals. They are experiments only in a provisional sense. Once treatment has been allocated they are just another observational study. They are difficult to do, often done poorly, and full of pitfalls. If this weren't true, you would only need one RCT for each drug or intervention. Instead we get many and they conflict. That's why Cochrane has a big Handbook on how to judge RCTs. What is your evidence that they are the most reliable form of inquiry? Or don't you need evidence for that?

Revere:
"What is your evidence that they are the most reliable form of inquiry? Or don't you need evidence for that?"

What would you accept as evidence? My observations? An anecdote? Maybe this?
http://faoj.wordpress.com/2009/02/01/introduction-to-evidence-based-med…

Quinn: Well I think I'd rather listen to an epidemiologist or statistician (for different views) than a podiatrist, especially one who is addressing EBM in podiatry exclusively and doesn't even get all oit right. I and my colleagues actually do these studies and teach about how to do them. Most clinicians don't know a thing about them or how to evaluate them.

Quinn: So to summarize. You present as your evidence the opinions of a podiatrist backed you no data. Do you think this article would be considered evidence by the Cochrane folks. Evidence of anything? Not because he's a podiatrist but because there's no evidence there. Just an opinion. You made a statement but have no data for it. I use evidence from all sorts of study designs and I know about them all. They each have strengths and weaknesses. But the one thing we don't teach our students is that one design is automatically better than another. The design has to be fit to the question and the situation. There are questions for which RCTs are impossible, others for which the RCTs are of very poor quality or almost infeasible, others where they really are good or even the best evidence. I hpe that's not too nuanced for you.

Revere,
I agree that the study design must fit the research question. "Levels of evidence" aren't the sort of thing that can be easily tested using RCTs. But there is a solid rationale for this hierarchy and it is well accepted in the medical community.

I thought the article I offered did a nice job explaining this rationale. The levels of evidence and the reasons for them are not the author's opinion, and he references his source - the Centre for Evidence Based Medicine at the University of Oxford. Here's the link:
http://www.cebm.net/index.aspx?o=1025

When it comes to the safety and efficacy of drugs, there are very good reasons why we consider double blind RCTs to be superior to uncontrolled studies, case reports, and anecdotes.

Quinn: many kinds of adverse event data are not easily uncovered by RCTs unless they are very large, run for a very long time, and have protocols that describe what the definition of an adverse event is, minor, serious, or life-threatening, and so forth.

Quinn: The other study designs are not uncontrolled. They are carefully designed (or should be; like RCTs they can be done badly), they use the same logic as RCTs (the randomization only serves to control but not eliminate confounding which can be dealt with in observational studies in both the design and the analysis. The Cochrane Collaboration accepts them in principle but often do studies like this with arbitrary exclusions for observational studies that can be highly informative (read: reliable evidence).

Revere: by "uncontrolled", I mean there's no control group.

Marissa: I completely agree. RCTs are not useful in every situation. When it comes to establishing the efficacy of antivirals, however, properly designed, double blind RCTs would provide the strongest possible evidence.

Quinn: Well you are in error. Both cohort and case-control designs have comparison groups (what you call a control group). They are designed to take advantage of natural experiments that usually can't be performed.

Revere,

When I said "there are very good reasons why we consider double blind RCTs to be superior to *uncontrolled studies*, case reports, and anecdotes", I wasn't referring specifically to the observational studies in question.

But they could be included here as well. Well designed, double blinded RCTs provide stronger evidence than well designed cohort and case-control studies.

Of course you have now hedged in your contention with qualifications, but even so it is not necessarily true. If the purpose of randomization is to reduce (although not eliminate) the problem of residual confounding, you can often design an observational study so that confounding is not a likely explanation for what you see. More importantly, perhaps, is that the observational studies are of people living real lives in the real world and RCTs are artificial. That is the reason that the Cochrane analysts and others have to make a distinction between efficacy and effectiveness. Observational studies measure effectiveness (if you want to use their terms and concepts) and RCTs measure efficacy. Since it is usually effectiveness you are after, observational studies are superior. All this says is that the design has to be fitted to the question and the answer properly interpreted. You get certain kinds of important and reliable information from well designed studies of some kinds and other kinds of important and reliable information from well designed studies of other kinds. The idea that you can put the reliability of study designs in a linear order, while a common misconception, is contrary to how evidence is actually used by scientists and fallacious. Remember, too, that RCTs are only a tiny, almost miniscule fraction of the science that is done. Toxicologists and chemists and physicists rarely randomize. Yet they produce highly reliable science.

So your problem with this article is that they restrict the EBM review to RCTs only? Or that it (a mere EBM review, not new data) was published in a high profile rag? Or that it concluded something that you "know" isn't the case?

People in RCTs are not living in the real world? Where are they living? Of course they are in the real world.

Of course some sorts of evidence are of better quality than others.

Observational studies can be designed to make it such that confounding is not a likely explanation (albeit often are not) but cannot eliminate the possibility like a true RCT can. Observational studies should count and are often the best evidence we've got, but the quality of evidence from observational studies is not as good as that from a well powered and well run RCT. I have hard time understanding your arguing otherwise.

The evidence that Tamiflu does much is moderate strength evidence at best (of a sort not considered by Cochrane for reasons that are unclear) for a very modest effect. Re-emphasizing this fact is important as some have this season handed out Tamiflu like it is candy corn and as some panic over the emergence of Tamiflu resistance in Pandemic A/H1N1-SOIV.

Don: There was nothing much (if anything) new in this article. They don't even claim there is. One claim: NIs reduce the symptoms of flu "modestly." That's not new. They say that they don't work for non-flu disease (ILI that's not flu). Duh. Evidence that they prevent complications is insufficient is something else they claim is "new." They updated a search for new literature but didn't find any that met their standards. Nothing was good enough for them (out of 1400 studies whose evidence they disregarded). They restricted it to studies of people otherwise healthy (even though it has been mainly recommended for people with risk factors), who may not have had flu because they included use in any ILI, they excluded challenge studies because they didn't consider them equivalent to field studies even though they are maximally informative for the biological efficacy of the drug and the studies had to include 75% of subjects with the lowest risk of getting complications from flu (ages between 14 and 60). So if you are at low risk for complications, they didn't have enough data to show that the NIs helped, even though the data were compatible with major reductions in complications as well as with no reductions. And if you were at high risk for complications (risk factor or age), then they didn't look. Not that you'd know it by the way it was spun by them and the journal and now the two journalists at The Atlantic, both of whom have a financial interest in this because they are paid to write articles that attract readers (and if you compare their version with the original article they did a lot of spinning and obfuscation) and one of them also has been -- in the language they like to use -- a paid consultant to the journal where it was published.

As for RCTs versus experimental studies, yes they live in the real world but it's not a real world intervention. The populations are different, compliance is different, outcomes are judged differently and the people behave differently. This is in addition to the restrictions on study populations I mentioned above. An RCT cannot be done with these drugs given what we know when seriously ill people are involved, something explicitly stated by the authors and obviously true. But some doctors, not having read the study, might hesitate to use them given current indications and they would put themselves at risk of liability (correctly) and perhaps increase the risk of dying of their patient. This story is being spun as saying that the drugs don't work. That's false. They do for healthy adults and animals and the idea that they decrease complications is highly plausible. They have not established the contrary but have excluded much evidence from observational studies indicating this.

Their conclusion? It should not be used for routine treatment of seasonal influenza. The only country that does this, as far as I know, is Japan. That has never been the practice in the US. They make no recommendations about pandemic influenza, people with risk factors or the elderly for seasonal or pandemic influenza. Not that you'd know this from reading about it. And you have to read the article carefully and parse it to even know this from the journal itself, not just news media.

The data on how much the NIs decrease symptoms has been known for a long time and there is clinical and observational evidence that they help seriously ill patients. Would you withhold them from a patient seriously ill with flu on the basis of this article? I would hope not. But some doctors will because all they read was the news articles or the crap put out by The Atlantic.

Actually the was another country in which Tamiflu was being handed out to healthy adults and children complaining of ILI - the UK. The UK ran a phone hotline and if the algorithm placed the individuals' descriptions as being likely H1N1 they were instructed to go and pick up a Tamiflu pack. No need to be seen by the doc.

This has been an issue in the British press especially since the algorithm used by these low level workers sucked - http://www.dailymail.co.uk/news/article-1234135/Just-diagnosed-swine-fl…

"Just 20 per cent of all cases diagnosed by the National Pandemic Flu Service were actually cases of swine flu, HPA scientists found. Everyone diagnosed by the service was given vouchers to get Tamiflu"

Moreover some serious illnesses including meningitis and malaria were missed. http://info-wars.org/2009/11/29/sick-patients-being-misdiagnosed-by-swi…

It isn't all about the US. The BMJ is addressing this context.

Don: Note that the BMJ piece was not about that. It was about seasonal flu, only. The use of Tamiflu for containment was incredibly stupid and much criticized, here and many other places. We also criticized WHO years ago for even broaching it as a possibility for H5N1. But the UK made a complete hash of the whole business.

Not that my opinion is worth much, nor that anyone needs my endorsement. But FWIW I've now read every single comment, and still fully support all of revere's arguments. Thank you, for your perseverance. There are people out there who need this treatment, and there are doctors out there who can be easily misled. When these 2 meet, the outcome may depend on how much stock the doctor puts in what they read in BMJ (less likely) or in the media (more likely, unfortunately)..

SusanC: Your views are worth a lot, as anyone who knows your work at Flu Wiki and elsewhere can attest. So thank you!

Oh c'mon revere. This review wasn't about the use of Tamiflu in seasonal influenza (which is almost all Tamiflu resistant by now); it was about using the evidence of seasonal influenza as a proxy for Pandemic A/H1N1. As the editorial described the history (http://www.bmj.com/cgi/content/full/339/dec07_2/b5164) "Following the outbreak of influenza A/H1N1 in April 2009, the UK NHS National Institute of Health Research commissioned an update of the Cochrane systematic review of neuraminidase inhibitors in healthy adults." A narrow question of great importance in a country in which the decision was to pass out Tamiflu to all healthy individuals with any hint of ILI.

Yes you are right that the methods they used are needlessly narrow, not considering good evidence that failed to meet their arbitrary definition of acceptability.

Yes you are right that the resultant article was written poorly enough as to allow misinterpretation of the results and that the media has magnified that error by misrepresenting it completely. But even so ...

I guess our reactions come from two different perspectives. You (and Susan) see the risk that some doctors will be idiots and not use Tamiflu for those who have a good chance of benefiting from it because they have read of this report in the press. I see my fellow idiot physicians currently overusing Tamiflu beyond the CDC guidelines which are themselves a bit of a Tamiflu carpetbombing approach. They need the reminder that there is no solid evidence that it does all that much of importance in otherwise healthy adults and that there is some risk of harm.

I am operating from the clinician's POV of being presented with an individual with ILI (in my case a child, often that under 2 year old with a febrile URI or croup which could be H1N1 or not) and having to decide if Tamiflu is indicated. And we have established that my clinical judgment (along with every other clinician's) sucks and that the available testing is worse yet. If the evidence was solid that Tamiflu was going to significantly reduce the risk of complications that the individual was at high risk of if they did have H1N1 then I should call it H1N1 in order to not miss a treatment opportunity. The cost of a false negative in that case would be high and the benefit of a true positive meaningful. But given that that evidence is only moderate of a small benefit then the cost of my false negative and the benefit of my true positive are both less. I should hold off on making the call more often in order to avoid the cost of false positives (side effects and dollars both).

I think you can appreciate the position we clinicians are in.

Double blind RCTs are conducted to establish that a medication works. *If* the beneficial effects of the medication are demonstrated, observational studies can be helpful in evaluating its usefulness in routine clinical practice. However, if double blind RCTs fail to show any beneficial effect, the drug shouldn't be incorporated into clinical practice. If it is, and observational studies subsequently suggest that itâs beneficial, the apparent benefits shouldn't be attributed to the drug.

For example, say RCTs indicate that an expensive medication has no therapeutic effect, but observational studies suggest that it does. If physicians tend to prescribe the drug more often to insured wealthy people, who receive better health care in general, the drug may appear to be beneficial because it was given to people who were more likely to recover anyway.

There are strategies that can reduce confounding in an observational study, but such studies will always be more likely to be confounded than well designed, double blind RCTs.

The idea that you can put the reliability of study designs in a linear order, while a common misconception, is contrary to how evidence is actually used by scientists and fallacious.

Are we talking about âlevels of evidenceâ here? Did you check out the link I gave to the Centre for Evidence Based Medicine at the University of Oxford? They seem to have bought in to the âmisconceptionâ too. As did the professors who taught me evidence based medicine, and most of the medical community.

Remember, too, that RCTs are only a tiny, almost miniscule fraction of the science that is done. Toxicologists and chemists and physicists rarely randomize. Yet they produce highly reliable science.

Double blind RCTs arenât the *only* way to generate reliable data, but they generate the *most* reliable data. RCTs arenât always applicable depending on the branch of science and the given research question. We want to know if Tamiflu works. Double blind RCTs are the best way to find out.

Don: Read the actual article. It is quite explicit. The commentaries are all add ons to dress up the fact they didn't find much and make a big to do about the data (which I agree is a serious problem). Read the conclusions and what thee article adds.

Don S,

I live in the UK. The "just 20% had swine flu story" is one example of epidemiology taken out of context. I don't know how exactly they arrived at that 20%, but I can tell you that the HPA reported that for people calling NHS Direct who were sent a self-swabbing kit, the positive rate was only 7%.

On the other hand, in an outbreak at Eton College, after they identified an index case, they notified all students and staff and asked people to self-identify. Out of 102 people who self-identified, 63 were still symptomatic at that point. PCR tests on all 63 came back positive. I repeat, the detection rate was 100%. You don't see such kinds of information in the Daily Mail, do you?

The accuracy of such tests depends on who is taking the test, whether the samples were processed properly etc etc. Of course, that is too complicated for the likes of Daily Mail.

references:

1. HPA Pandemic (H1N1) 2009 in England: an overview of initial epidemiological findings and implications for the second wave

2. Smith et al An outbreak of influenza A(H1N1)v in a boarding school in South East England, May-June 2009.

@Don S,
I hadn't read your most recent comment before I posted @28. I really appreciate your perspective and comments.

Quinn: Yes, that's what RCTs are meant to do and they may or may not be able to do it. Whenever someone talks this way about RCTs they omit all the qualifications they are then forced to stick in: "adequately powered," "properly conducted," "properly analyzed," "sufficient follow-up", etc., etc. That's because once allocation is made they are just another kind of observational study where confounding and other biases can return. And randomization does not eliminate confounding. It just reduces its probability according to population size. The problem I am speaking against is that the notion that uttering the magical words "randomized clinical trial" seems to befog the minds to many otherwise knowledgeable people. I don't know how much you were taught about two things: how to do an RCT; and how to do a meta-analysis. But as Meat Robot @10 says, once you've seen the sausage made you don't think it's so magical anymore.

I think that this discussion could go on for a long time and that a better way for me to approach it will be to write a post about RCTs so we can isolate our points of agreement and disagreement. Because we are now repeating ourselves.

Revere,
I've repeatedly used the qualifier "well designed" when speaking of RCTs. I'm not saying well designed, double blind RCTs are perfect or magical, I'm saying they're the best way to tell if a drug works.

Quinn: Yes, you have repeatedly used that qualifier. But you and most people not engaged in this work don't know what it means and even if you did, it's not the end of the story. It's the beginning. Any of my students can produce a well designed study. But they can't execute one. On paper. In the real world they can be extraordinarily difficult to conduct, analyze and interpret properly. So what you are presenting me is either a platitude or a tautology. If a study is really good, it will show you what you want (that's what it means to be really good, after all). Randomization doesn't make that happen. It's just another tool in the tool box. It has been elevated to magical status. I don't think you understand that.

I'm not going to go down the RCT vs whatever track cos that horse has been beaten to death many times over.

What gets my ire is the tone with which all this is being reported, starting from the BMJ itself. Breathlessly, with 'shock and awe', as if there's been some major discovery. Well, guess what? There hasn't. There isn't anything in there that we don't already know, period.

So my next question is, why the big fuss?

Let's look at the caricaturization of tamiflu being handed out like candy. Granted there IS some truth to that, anecdotally, and I don't support the UK policy, but things happen in the fog of war that don't necessarily stand well to armchair scrutiny. So let's look at some real numbers here. The UK has a population of 60+ million. Unlike the US, the summer wave in the UK was several times bigger than the previous seasonal flu wave (see more detailed analysis here http://www.newfluwiki2.com/diary/4115/are-we-done-here ). Granted not all of them were due to H1N1, but then by definition this was ILI tracking. Comparing like-for-like, this was an extraordinary ILI wave in the middle of the summer when not much ILI generally happens.

So how many people got infected by H1N1? It's anybody's guess. But let me tell you about the outbreak at Eton (because it was thoroughly documented). When they identified the index case, the school had already broken up for a short break, so self-identified cases (102 of them) were dispersed all over the country (Eton being a boarding school). They were swabbed by whoever in their local area. These tests came back at an astonishing 100% positive for PCR. Because they were not taken at a single center, I tend to believe the results are probably reasonably accurate. But the more interesting part is this. After it was all over, they did some seroprevalence testing, and found 39% of the students tested positive. (see HPA report, link in previous comment).

In other words, in a school with 1300+ students, 39% were seropositive, 8% (102) self-reported clinical illness, of whom those who were still symptomatic tested 100% positive, but only 3% (39) sought medical treatment. Granted that the seroprevalence results may overestimate the true level of infection, we are still looking at a lot of people having been infected during that single outbreak over a 4 week period, of whom only a small fraction sought medical care. (FYI, tamiflu prophylaxis was offered to the whole school, with an uptake of 48%. Whether that is high or low depends on where you are coming from...)

Anyway, the point of all this is, an AWFUL lot of people were infected. Now, the Daily Mail (and others) complain that >1 million doses of tamiflu were given out. We all know that compliance is never 100%, but even if 1+ million doses of tamiflu were consumed in a country with 60 million people, we are still talking about only 2%, give or take, of the population.

Where is the carpet-bombing?

We can debate whether this was/is a good policy. And we will likely never agree here, which is fine. What I dislike most is hyperbole and grandstanding. As if Cochrane, BMJ et al have suddenly caught the UK government red-handed doing something terrible that they didn't tell the public about.

Guess what? Nothing happened. It was a policy. It may be a good one, or a bad one. Only time will tell. But there's no call to conflate the issue (and confuse the public) with grandstanding and hype.

In the meantime, just for the record, even Jefferson et al, after reporting 567 serious neuropsychiatric adverse events (the most serious AE under discussion) from Japan, agrees that the risk is RARE:

It is, however, estimated that more than 36 million doses have been prescribed since 2001, making such harms (even if confirmed) rare.

Oh I agree that this article doesn't say much. I can't say that I have seen the big to-do about it: the first I heard of it actually was here. No docs I know are talking about it. But then I don't get out much. :)

2% of a national population getting a medication for dubious indications to me is carpetbombing.

To use your specific example, offering Tamiflu prophylaxis to the entire school and having nearly half the population take them up on it, is carpetbombing. The benefit gained by that? Any? By your numbers most had already uneventfully recovered. Most would have done just fine without the medicine. Was there a single hospitalization or even outpatient case of pneumonia likely prevented by doing that? In fact, revere, assuming that the studies excluded by the Cochrane review do indeed convincingly show a small but real benefit in preventing hospitalization then what is the number needed to treat to prevent a single hospitalization? How many would experience mild side effects (nausea, nightmares, etc.) of that number? How many would experience more significant reactions (I had one kid with a serious anaphylactic reaction for example, but also other allergies, etc.)?

Let's stick to your UK numbers Susan. The NHS cost is allegedly 20 pounds per course. That would come to 20 million pounds spent but online I find that the actual cost to the UK was 500 million pounds. Now how many hospitalizations did that money prevent? Could its use have been restricted to a much higher risk group and prevented nearly or exactly as many hospitalizations for a fraction of the cost?

Tamiflu is held up by some to be this magic drug, much like how revere believes some hold up the RCT. While the review was not a great example of EBM its message that Tamiflu may be overused and is of only limited value is important for the public to hear. More important than reassurance that it does work a little bit.

And yes revere I have read the article, such as it is.

FYI, the tamiflu prophylaxis happened at the very beginning, when the UK was following a policy of 'containment'. I don't agree with that policy, I don't see the point. But it was what it was.

Containment was given up in the beginning of July.

Just so you know.

Oh, I love that cost debate. Sure we spend however many hundred million pounds on tamiflu. Let me remind you that the tamiflu was stockpiled several years ago. The UK was among the first to start stockpiling, in anticipation of a possible pandemic against H5N1. Even with extensions, these meds were just about ready to expire. So even if we didn't use it, the money was down the drain.

As I said, it's easy to be armchair generals. Pandemic planning is complicated business, and things never happen the way you planned. If a pandemic with H5N1 happened, would the same people complaining now, consider the costs justified? Probably yes.

For me, I'd rather the money was wasted, either the drugs expired and there was no pandemic, or somehow some of it was used in a mild pandemic that probably didn't need that level of treatment. I'd MUCH rather that money was 'wasted', than if it was 'well spent'.

But then that's just my little house-wifely logic...

Don: Let me summarize what the paper seems to say. Let's see if we can agree:

The data suggest that neuraminidase inhibitors are modestly effective at reducing the symptoms of influenza (about one day reduction) for healthy individuals who get flu. Generalizing this to very ill people in hospital seems reasonable to them, although they don't have data. However they believe it is unlikely ethics committees would permit a trial that compared no treatment for people with influenza who have life threatening disease.

The authors believe NIs should not be used in routine control of seasonal influenza. They believe use for pandemic is reasonable but have no data on it.

Yeah well I am sure there is a pile of antibiotics that is about to expire somewhere in a warehouse. We probably should hand it out to a lot of people with colds before it goes bad, y'know? Afterall treating every cold with an antibiotic is sure to prevent a pneumonia or two.

Sorry, I come from the pediatric side, and in peds we learn that often the best thing to do is to do nothing, just so long as you do it well. Very seriously. You need to know when to not test and when to not treat at least as much as you need to know when to to test and when to treat. An article that messages to a treatment happy populus that doing nothing (other than TLC) may be the better choice if you are an otherwise healthy individual is a good thing to me.

Yup revere, not much really to it. Modest benefit and no EBM comment on hospitalized patients. Poorly written as they really should just have said that that question was outside the scope of the review which focused only on healthy adults. As for pandemic use they think it is reasonable to generalize from seasonal to pandemic and that its role in controlling the spread of a pandemic is limited at best - if used then only as part of a "package of measures to interrupt spread".

Don, you can agree or disagree with the UK policy. FYI I don't support it either, not all of it, and I've been a severe critic over the years.

But that is not the issue under discussion. The issue is the BMJ articles, whether they were justified. I think they are technically justified on content, but not justified in presentation. Irrespective of whether the UK policy is a good one, there was NOTHING that was new and unknown, and nothing shockingly unethical or dangerous that justified the level of grandstanding accompanied by uninformed accusations in the media.

Hype is bad for a sane debate of policy. It's also bad for science. It diminishes the credibility of BMJ and Cochrane et al.

That's just my personal opinion. I'm sure there are plenty who disagree. Fine by me.

Don: Let's talk about antibiotics. If you use them for something they aren't effective for, does that mean you shouldn't use them at all? Because the BMJ paper says NIs are not effective for ILI and that essentially is being interpreted don't use them for real flu. And we are talking about doing nothing well -- or poorly.

Good question, revere.

I'd also like to know, Don, how exactly you are approaching the use of tamiflu, as a clinician in a pediatric practice (I assume)? What are the criteria you are using to prescribe or not prescribe?

You see I see the article being spun as that perhaps and poorly phrased but actually not claiming that Tamiflu should not be used at all.

I follow the CDCs guidance but allow myself the flex as described by deciding where I draw the line as to labeling a particular febrile URI in an under two as likely H1N1 or not. The issue as I presented it earlier is very real to me. If I was convinced that there was a significant lost opportunity to treat with a false negative I'd label more readily; without that I am less likely to overdiagnose even knowing that by doing so I will also miss some. Asthmatics and those with neurodevelopmental/neuromuscular issues OTOH I call more and Rx more readily - they are less likely other viral with a fever that high and more likely to not just have hospitalization but death.

What I'm hearing is there are actually no hard and fast rules. It's a judgment call.

You're worried about people over-prescribing. I'm worried about both over- and under-prescribing. What I'm most worried about, is those clinicians with the least capable judgments are probably the same ones most likely to be persuaded by media hype.

That is my concern.

Revere,
re: post 34
Your suggestions that I lack knowledge and understanding wouldn't be helpful even if they were true. Please keep in mind that you donât know much about my educational background or professional experience. As Susan quite rightly pointed out, the âRCT vs whateverâ issue has been beaten to death, so Iâll leave it alone.

But I have another question. In the first study that Susan posted (at comment 4), 71% of the patients hospitalized for influenza had received the vaccine. Less than a third of the population (in Canda, at least) is typically vaccinated for influenza. What do you make of this?

Susan: I've been referring to you in the third person here, but I'd appreciate your thoughts too ;-)

I'm sorry, what exactly is the question for me?

The middle paragraph. I was wondering what you and others think about the percentage of hospitalized flu patients who've been vaccinated.

Quinn,

If you are referring to the vaccination part, where did you get your figure for the level of vaccination in Canada? I'm not disputing it, I just want to make sure I know what you are referring to.

The level of vaccinating differs by age group. It's likely to be much higher in the elderly, than the national average.

Here're the stats for Canada, flu shot vaccination trends

http://www.statcan.gc.ca/studies-etudes/82-003/archive/2006/9195-eng.pdf

Overall, for 2003, around 28%, rising to 62% among elderly with no risk conditions, and 75% among elderly with at least 1 risk condition.

Thanks for the link, Susan. I got the figure from recent news reports that I've seen and heard, but I didn't have a specific source in mind. You're right that the figure increases with age and risk factors.

About 36% of the subjects in the study were under 15, though. Even if 75% of the elderly in the province had been vaccinated, the finding is hard to explain.

Quinn: May I remind you that you called me presumptuous? But I don't really mind. I'm happy to engage you on the substance now as I did then. But for the record, I said that you and most people like you (meaning people with training and knowledge but not experience actually doing these studies) don't know what's involved -- if I am in error (which on the evidence of what you have been saying I don't think I am but I will take your word for it), you can respond to what I actually said, not merely take umbrage. When you called me presumptuous I responded by asking what was presumptuous about what I said (and got no answer) and in this case I have spelled out what you don't seem to understand: "Randomization doesn't make that happen. It's just another tool in the tool box. It has been elevated to magical status. I don't think you understand that."

So you can just take umbrage at my inference (that you didn't understand something which I took care to spell out) by demonstrating you do understand it or ask for clarification as I did with your (I believe unfounded and not particularly constructive) accusation I was "presumptuous," a clarification I never received. Not to put too fine a point on it, but I did indicate what I thought you didn't understand while you merely said I was presumptuous.

So I'll say it again. Randomization is just a tool to deal with one kind of bias, residual confounding by uncontrolled confounders. One tool among several. By itself it works no magic. I don't think you appreciate that (I'll say it that way if the word "understand" bothers you). The issues with randomization are deep and somewhat controversial but I am assuming we will remain skating on the surface since that's the level it is usually treated in these discussions, and usually without harm. But not always, so if you want to get into the weeds about randomization I'm ready.

May I remind you that you called me presumptuous?

That isn't true. I said your statement was presumptuous and I offered a very clear, itemized explanation as to what the statement assumed. Here's a recap:

you said: "This seems to me to be another argument in favor of the vaccine: it prevents DNA damage from the virus."
I said "This is presumptuous."
and explained:
"This assumes that a) the H1N1 virus is mutagenic, and b) the vaccine is not. Neither is a safe assumption." There's more of an explanation as to why the second isn't a safe assumption in the post (the first is obvious).

Randomization doesn't make that happen. It's just another tool in the tool box. It has been elevated to magical status. I don't think you understand that.

This spells out what you think I don't understand? Randomization doesn't make *what* happen? I never claimed that randomization makes a study perfect, or that it hasn't been "elevated to magical status" by some.

My experience with research methodology (or lack thereof, if you choose to believe) isn't relevant to the argument I've made. The Evidence Based Medicine folks at the University of Oxford share my views on "levels of evidence" - do you think they don't understand randomization? It's widely accepted in the medical community that double blind RCTs are the gold standard for research design. I'm shocked that you'd contest this.

Revere,
I've calmed down a bit and I'll retract my statement that "I'm shocked that you'd contest this". I think I do know where you're coming from.

We may be arguing two slightly different things.

I actually agree on the following:
-observational studies can be useful and there are ways to reduce confounding, other than randomization.
-RCTs may be poorly designed and don't always generate reliable data.
-the hierarchy of evidence does not eliminate the need for careful evaluation of study design in weighing evidence.

My argument has been that properly designed RCTs are the best way to answer the question "do NIs work?". In other words, if we were going to conduct a new study to answer the question, a large, properly designed, double blind RCT would be the best choice. And if we're assessing a bunch of different studies that are all well designed, the RCTs would get more weight.

But, if we want to answer the question "do Nis work?" on the basis of existing evidence, we need to carefully evaluate the studies we have. And it's likely that some observational studies will be more useful than some RCTs. It would be inappropriate to weigh the evidence on the basis of study type alone.

I'm guessing that you don't disagree so much with the "hierarchy of evidence", but with its rigid application(?) I can appreciate that.

Susan, I have answered only for myself and while that is judgment it is also as messy of sausage making as any meta-analysis. And while it may be good medicine (in my assessment) it does risk medicolegal exposure - the "safer" defensive medicine thing to do is to think less and treat everyone in a CDC risk group who even might have an ILI. But that is another discussion.

How badly did the BMJ misrepresent or sex it up and how much was the media coverage of it?

I read those reports as saying in several places that this is a question about treating otherwise healthy adults (as was widely done in the UK during this pandemic), not treating those with risk factors.

I see a separate article discussing the role of observational studies http://www.bmj.com/cgi/content/full/339/dec07_2/b5248

"...health policy makers often have to make important decisions about new drugs when not all relevant trials have been undertaken. This is the situation currently regarding the use of antiviral therapy in H1N1 influenza. It may be argued that at such times we should be informed by all available evidence and not constrained by randomised trials alone. ...

... Results of multivariable analyses were broadly in line with those reported in the randomised trials. For participants with clinically diagnosed influenza, the estimated number who needed to be treated with oseltamivir to avoid one diagnosis of pneumonia was always over 100, and may be as high as 1000. Only one of the observational studies reviewed here considered safety issues. ...

... generally support the conclusion that oseltamivir may reduce the incidence of pneumonia and other consequences of influenza in otherwise healthy adults. However, these events are rare, so for most otherwise healthy adults treatment of influenza with oseltamivir is not likely to be clinically important.

A potential advantage of observational studies is that they can provide evidence on the use of a drug in a realistic setting. However, this advantage was largely undermined by the studiesâ selection criteria ...

... The estimated effect of antiviral drugs in people with existing cardiovascular disease was substantial, and the difference in rates of death and serious morbidity were potentially clinically important. ...

... Our rapid review of these "real life" data suggests that oseltamivir may reduce the risk of pneumonia in otherwise healthy people who contract flu. However, the absolute benefit is small, and side effects and safety should also be considered. None of the studies examined the role of oseltamivir in patients with H1N1 influenza, which may be associated with higher rates of pneumonitis than seasonal influenza.14 We did not consider the evidence for the use of oseltamivir in high risk patients, although several of the studies identified by Roche were in special populations. Other observational studies suggest that early intervention with antivirals for influenza may benefit a range of high risk patients and potentially improve survival rates, but these studies are also open to residual confounding.13 14 15 16 17 18 19 20"

The BMJ article seems to be a bit much perhaps but this beating them up also seems uncalled for.

Quinn: Thanks for this good summary of the situation. I think that you ably set out where we are both coming from, and, as usual, when the dust settles it turns out people are not so far apart as it may have looked at the outset. I appreciate your willingness to see the conversation through to the point where we each have been able to clarify our views (noting that blog posts and comment threads are not ideally suited to well organized, nuanced and coherent arguments but still work if time is taken).

Don: The issue with BMJ isn't just this article but a pattern of provocation and sensationalism that appeared when the new Editor arrived. There was the notorious Jefferson piece on vaccines which became the pretext for The Atlantic article which took up so much time and space here (and around the net and the MSM in general) and those same conflicted journalists have tried to leverage this piece, too. The Cochrane Collaborative is a network of volunteers, many of whom do an excellent job of survey and summarization, producing reviews which are useful for others looking to get a quick overview of a topic and guide to some of the literature. Because, like any review, they are limited in scope and full of selection biases of their own (sometimes clearly evident and sometimes not) they must be used judiciously.

The problem is that some of the volunteers have become zealots and have even made their reputation writing contrarian and attention getting pieces and BMJ has taken advantage of this for its own attention getting purposes. Sadly, high profile journals are doing this in all sorts of ways these days, including using embargoed pre-pub seeding of articles to get journalists' attention, deliberate selection of articles for public newsworthiness rather than scientific value, etc. That's a fact of life. But what irritated som many people about the BMJ practice was both the pattern and the lack of responsibility in recognizing that doing this regrettable practice has a context and in this case the context was the first influenza pandemic of the new millennium. This has been a trying period for everyone, including clinicians, policy makers, public health, the public, the media and many others and therefore "sexing it up" has a consequence in this case that it doesn't have in other contexts: it seeds the last thing we need, more confusion, confusion that isn't the result of a confusing science but the result of a public message that is not meant to enlighten but to gain viewership (the colluding TV station), market share (The Atlantic, BMJ) and attention (the paper's authors, who are least culpable for the paper itself but guilty of blowing this up in the commentaries and other material that turned a not very interesting or enlightening paper into a media event.

Let me say again that I have sympathy for the generic data issues that arise with drug companies and their own agendas as they overlap with and interfere with scientific publishing. But entangling that issue with the use of the only class of drugs we have for treating influenza in the midst of a pandemic was not responsible.

Don, see revere's response. I agree entirely. I don't always see eye to eye with the reveres. We've had some vigorous debates where we finished up neither convincing the other. But on this one, the BMJ articles annoy me for precisely the same reasons cited by revere, especially the issue of context. I hate opportunism in science, particularly opportunism that doesn't care about the unintended (erring on the charitable side) consequences.

Quinn,

About 36% of the subjects in the study were under 15, though. Even if 75% of the elderly in the province had been vaccinated, the finding is hard to explain.

About half of those under 15 were infants less than a year old. Infants and young children are another age group for severe influenza and hospitalization. In this cohort, none of the children died.

So to recap: your problem isn't this particular article and its narrow conclusions which as it turns out is, while not earth shattering, is also not unfair or inaccurate (if a bit breathless in its style), but in the pattern of articles by the newest BMJ editor. And in how it was played by the media misinformation machine.

Okay. Fair 'nuff.

Revere,
I'm glad we could find some common ground. Thanks for the stimulating debate.

Revere, Don, Quinn, & Susan,

Thanks so much for these excellent comments on a very important topic!

Despite the lack of definitive supporting data from randomized controlled clinical trials, patients with severe H1N1 disease do seem to benefit from early antiviral therapy.

Starting treatment with a neuraminidase inhibitor within 2 days after symptom onset was significantly associated with a lower risk of ICU admission or death in hospitalized 2009 H1N1 patients (N=272; median age, 21 years), as compared with later treatment (P <0.05).

In a multivariable model that included age, admission within 2 days or more than 2 days after the onset of illness, initiation of antiviral therapy within 2 days or more than 2 days after the onset of illness, and influenza-vaccination status, the only variable that was significantly associated with a positive outcome was the receipt of antiviral drugs within 2 days after the onset of illness.

At the bedside, physicians taking responsibility for the outcome of seriously ill patients often do not have the luxury of always having well-designed, definitive randomized controlled clinical trials to guide their therapeutic decisions (the science of medicine), and clinical decision-making is made even more difficult because young healthy patients can quickly succumb to severe H1N1 disease, and early in the course of their illness - when they may have only mild symptoms - is the optimal window of opportunity for greatest possible benefit from antiviral therapy (the art of medicine).

Good investigative journalists know that often when there's smoke there's usually fire - in this case, it seems, perhaps one more example of pharmaceutical companies witholding and distorting data.

I believe there is a special place in Hell reserved for those engaged in profit-driven dishonesty, deceit & deception that endangers patients' lives.

I hope the media will continue to shine the light on this important issue and encourage badly-needed reform to restore scientific integrity in clinical research.

I hope the media will continue to shine the light on this important issue and encourage badly-needed reform to restore scientific integrity in clinical research.

Hear! Hear!

The story isn't that Tamiflu is of little use; that would be a non-story indeed, everybody knows it's a waste of time.

The story is that Roche destroyed data from their own research, then their PR department pre-wrote the conclusions of the studies that were actually published and instructed the researchers and paper writers to arrive at those conclusions.

If you think this is a big story, wait until people start going to prison.

ewan: You are right. The story isn't that Tamiflu or Relenza are of little use. Because that's not what the paper says (although the story has mistakenly been interpreted that way). As for the rest, I have problems with Roche and other Big Pharma handling of data. If some people go to jail over it, fine with me.

I am really happy I work in the "hard" sciences. Seems these poor Cochrane guys only fault is that they disprove someones pet ideas with hard data.

George: Except when they don't. They "disprove" nothing and they have no hard data. They don't do any studies of their own. They summarize other studies using a specific protocol. They, in effect, are doing observational studies where the data points are RCTs. That means all the problems of observational studies are on their doorstep, too, for example, observer bias, selection bias, etc. So their main fault is hubris. Not uncommon, even in the "hard" sciences (where, BTW, it is rare to find anybody doing a randomized trial; what's the matter with you people? you don't do "science"?).

I thought the Tamiflu neurologic side effects resulted from a bad batch of Tamiflu made by a specific factory in Japan. If this is so, it is pretty silly to pretend otherwise.