Some commenters on my last post seem to be of the view that it is perfectly fine for scientists to pull numbers out of thin air to bolster their claims, at least under some circumstances.
I think it’s a fair question to ask: In which circumstances are you comfortable giving scientists the go-ahead to make up their numbers?
One suggestion was that scientists ought to be “permitted to use ordinary language meanings of words and colloquialisms in their non-peer reviewed discourse” — or at least, to do so in discussions on blogs. I take it this assumes that “ordinary language meanings of words and colloquialisms” covers totally made-up statistics (which itself seems to assume either extreme innumeracy or disregard from actual facts among people talking in anything other than a peer-reviewed context).
I’m guessing that adopting this standard would undercut scientists’ attempts to communicate with non-scientists about matters of interest or importance. Why on earth would a non-scientist be swayed by a scientist’s claims about the impact of carbon emissions on global climate, or about vaccination on disease prevention (or the incidence of autism), or about the age of the earth, or about the size of an atom, if every number that appears in such claims can be presumed to have been made up?
Another suggestion was that it is permissible to use “joke statistics to make a rhetorical point”. In the case where the cited statistic is clearly made-up (and I think there’s some room to discuss whether joke statistics not explicitly flagged as such might be taken as authoritative), it may advance a rhetorical point. Does the rhetorical point depend on the joke statistics, though? Could the joke statistics (by their very jokey-ness) actually undermine the rhetorical point?
Ultimately, can the rhetorical point end up distracting attention from the question of how things actually are in the situation being discussed?
Let me repeat my worry as I elaborated it in a comment:
My point is, when scientists do it — whether in the context of a discussion of their findings, or of the prevalence of fraud in their field, or of mentoring strategies, or what have you — when they make up data rather than admitting “I don’t know the exact prevalence of X (nor does anyone, to the best of my knowledge)”, that’s lying. Scientists, of all people, should understand the importance of distinguishing between what you know and what you guess. Scientists, of all people, should understand the value of admitting ignorance in situations where good data has not yet been collected.
Doing otherwise may be human, but it’s not scientific.
So, I’d be very curious to hear an elaboration of when you think it’s OK for scientists to pull numbers out of thin air.
Those of you asking mentors (or bloggers) for career advice, tips for getting grants, and so forth: which bits of that advice would you like to have grounded in reality, and which bits not? How about when they’re giving you advice about running experiments?
Those of you depending on scientists for reliable information about the world: in which non-peer-reviewed contexts would you prefer that scientists only cite numbers grounded in reality, and in which are you fine with them making it up? Letters to the editor or op-ed pieces? Statements at school board or city council meetings? Conversations at the doctor’s office? Chatting in line for coffee or at a soccer game?
Scientists: do the made-up numbers make the point you’re trying to make better than would saying, “I don’t know for sure but my hunch is this is not a big problem,” or “My impression, from what I’ve seen, is this is more frequent than you might guess”? If so, how do the made-up numbers make the point better?
Would the point be bolstered better still by actual numbers? Would your point be undermined by your audience noticing that you’ve pulled numbers out of thin air?
It’s quite possible that my prejudice about scientists leaning on made-up numbers is mistaken, at least in certain contexts. But I’d like to see a reasoned argument to that effect.
Preferably an argument that doesn’t rely on numbers pulled out of thin air.