Deciding the conclusion ahead of time

Mark Thoma links to a report by Michael Shear on a leaked memo from the U.S. Chamber of Commerce:

The U.S. Chamber of Commerce and an assortment of national business groups opposed to President Obama's health-care reform effort are collecting money to finance an economic study that could be used to portray the legislation as a job killer and threat to the nation's economy, according to an e-mail solicitation from a top Chamber official.

The e-mail ... proposes spending $50,000 to hire a "respected economist" to study the impact of health-care legislation ... would have on jobs and the economy.

Step two, according to the e-mail, appears to assume the outcome of the economic review: "The economist will then circulate a sign-on letter to hundreds of other economists saying that the bill will kill jobs and hurt the economy. We will then be able to use this open letter to produce advertisements, and as a powerful lobbying and grass-roots document."

Don't worry, though, they didn't really mean it:

Randy Johnson, the Chamber's senior vice president who handles health-care issues, called the e-mail "inartfully worded" and said the group never intended to suggest that the outcome of the study would be preordained.

"It's not saying that we would tell the economist how it should come out. . . .

Of course, you don't need to tell the economist how it should come out, you can just let him or her know ahead of time that you'll only pay if it comes out a certain way. Of you don't even need to do that, really, you can just make your expectations clear.

Hey, wait a minute!

OK, OK, it's easy to laugh at the Chamber of Commerce here, and I think what they're doing here is bad: the goal is to influence policy via a scientific-seeming study whose conclusions are decided in advance. In this particular case, it might be OK--maybe their pre-chosen conclusion is correct--and maybe this issue is important enough that it's worth a little lying and cheating, all's fair in love and war and all that. But this particular defense isn't going anywhere; I mean, you can defend almost anything by arguing that it benefits a larger cause.

The more serious issue is that this predetermined-conclusions thing happens all the time. (Or, as they say on the Internet, All. The. Time.) I've worked on many projects, often for pay, with organizations where I have a pretty clear sense ahead to time of what they're looking for. I always give the straight story of what I've found, but these organizations are free to select and use just the findings of mine that they like.

This also reminds me of something I've noticed on legal consulting projects: typically, the consultants on the other side seem incompetent, sometimes extremely so. I have a few hypotheses here:

1. Most statistical consultants are incompetent. Lawyers etc. tend to find statistical consultants via word of mouth, and they're likely to get a professional consultant who is really bad at statistics but good at promoting himself. (Even when academic consultants are hired, the lawyers often pick the wrong academic--somebody who might be a top researcher but who has no expertise in the topic at hand and doesn't know enough to turn down the project.)

2. Maybe the side that hires me is actually typically in the right, and they hire me because they want their argument to be made transparently. Conversely, if you're a lawyer and you don't really think the numbers support your case, maybe you'd rather hire a hack who will say anything you want--he's dependent on you for the money--than a professor who feels free to draw his own conclusions.

3. Maybe I wasn't really on the right side in many of these cases; maybe I'm just fooling myself to think so.