IRB shopping

Via Inside Higher Ed comes news that the Food and Drug Administration has changed its mind (do administrative bodies have "minds"?) about rules it recommended on how scientists get approval for their research projects from IRBs (institutional review boards). In particular, the rules were intended to head off abuses of the approval system that might come from "shopping around" for the IRB most likely to respond favorably to one's research proposal. From the IHE article:

[The FDA] announced in the Federal Register that it would withdraw a 2002 plan that would have required scientists seeking approval for a particular piece of research to inform "institutional review boards" on their campuses of any previous attempts to gain that approval.

The FDA, which oversees significant amounts of federal research funds and regulates IRB's, which are the campus panels charged with approving clinical trials involving human subjects, said it was considering the 2002 rules because of concerns raised in a 1998 report by the Department of Health and Human Services's inspector general about what it called "IRB shopping." The report suggested that in at least "a few" cases, researchers "who were unhappy with one IRB's reviews [of their proposed study] switched to another without the new IRB being aware of the other's prior involvement." Many large universities and medical centers have multiple review boards.

FDA officials sought comments on a proposed change in the rules governing IRBs that would require researchers to include in their research proposals information about prior attempts to seek approval for the experiments. "These disclosures," the FDA wrote at the time, "could help ensure that sponsors and clinical investigators who submit protocols to more than one IRB will not be able to ignore an unfavorable IRB review decision and that IRBs reviewing a protocol will be aware of what other IRBs reviewing similar protocols have concluded."

What was the worry with "IRB shopping", and why has the FDA decided to stop worrying about this?

Researchers proposing research would like their research to be approved. IRBs are supposed to make sure that the proposed research conforms to all the applicable federal, state, local, and institutional rules about research with human subjects, research with animals, intellectual property issues, managing potential conflicts of interest, etc., etc.

How the regulations apply to a specific piece of proposed research is, it seems, a matter of interpretation. For example, here are the first three principles (out of thirteen) in my own university's policy for protection of human research subjects:

1. No human subject is to be exposed to unreasonable risk to health or well-being whether physical, psychological or social.

2. Commensurate with the principles of protection of human subjects, the procedures for assessing and minimizing risk to human subjects shall respect and protect the academic freedom of the faculty and students in their pursuit of knowledge.

3. The risks to an individual must be outweighed by the potential benefit.

Just what qualifies as an "unreasonable risk" to health or well-being? What potential benefits are beneficial enough (and likely enough to come to pass) that they outweigh the risks? What is the proper balance between "academic freedom" and a certain level of risks to human subjects? These are the sorts of questions an IRB has to take up in evaluating research proposals.

An IRB is not an exceptionally well-programmed computer through which one feeds research proposals for evaluation. (The IRB 9000! "Dave, I'm sorry but I can't let you use that protocol.") Rather, it's a group of human beings, doing their best with their understanding of the regulations and their prior experience to make reasonable judgments about what might happen in research that hasn't been conducted yet. They're making bets about the future. And, they are charged with making those bets in such a way that they are being fair to the interests of the folks proposing the research and to the folks who could be affected, positively or negatively, by the research.

How you do this isn't so clearly prescribed as to have a "right" outcome (which makes being an IRB different from being a blackjack dealer, who knows for sure whether he's supposed to hit on 16). Different groups of people can take the same regulations and the same proposals and end up making different bets about them.

If you're a researcher, this might mean that, in the interests of getting your research approved, you'd like to find the IRB that's more likely to bet on fewer bad outcomes and more good outcomes. You want to find the "easier" IRB. In the case that your research proposal gets shot down by a "hard" IRB, you'd like to find out whether it could get approved by the "easier" one. Is there anything wrong with this? After all, having your proposal put before a hard IRB rather than an easy one might have been just the luck of the draw, in which case it could just have easily ended up before the easy IRB. If the easy IRB -- which is still a full-fledged IRB -- approves the research, shouldn't that be enough?

Here, it's worth noticing what the proposed FDA rule change would have required. The reseachers making the proposals would have had to "include in their research proposals information about prior attempts to seek approval for the experiments." In other words, if they had trouble with a "hard" IRB initially, they would have to disclose these initial unfavorable reviews to the "easier" IRB evaluating their proposal. Possibly this wouldn't change the latter IRB's assessment of the proposal at all. But perhaps, seeing the concerns raised by the initial IRB would change the way the subsequent IRB made its bets about forseeable risks and benefits. Seeing how the hard IRB scored the proposal might make the easy IRB harder. (The first IRB's rejection of the proposal might also hang the "loser" sign on the proposal, although I'm sure each IRB strives to evaluate proposals on their merits rather than on what the other kids are saying about them.)

Is this a problem? After all, you might think an institution would be better served if its many IRBs were pretty much on the same page in terms of how they judged research proposals. (It would certainly make it easier for researchers to figure out what they need to do in formulating proposals if was a uniformity of judgment rather than IRB roulette.) Maybe every research proposal should be evaluated by multiple IRBs, with a running tally sheet of evaluation from each one.

You realize, of course, that members of the IRBs have other professional duties (including doing their own research) and that they, too, only have 24 hours per day. Having multiple IRBs at an institution is a strategy to make it possible to get through all the research proposals on something faster than a geological time scale. And indeed, the increase in the amount of work (for researchers and IRBs) of the FDA's proposed rule change would be significant. This, combined with the fact that there is no evidence that "IRB shopping" happens in any significant numbers, is why the FDA said, "Never mind."

But ... it might not be a bad idea for institutions to look at whether they have reliably easy and reliably hard IRBs. It might not be a bad idea to bring members of such IRBs in communication with each other on a regular basis to see if these differences flow from fundamentally different understandings of the regulations, or if they just represent different reasonable bets one could make working from a shared understanding of the regulations. Reshuffling the members of the various IRBs every now and again might be a useful control, too.

As far as the researcher is concerned, while it's probably not unethical to take a proposal that was shot down by one IRB and see if a different IRB will approve it, it might be a good idea to spend a few moments with the first IRB's review. Are there any reasonable ways to address their concerns and still do the research? If so, working those into the proposal before submitting it to the next IRB seems like the responsible thing to do, whether or not you include the first IRB's report with your submission.