Mister P

The Secret Weapon

The Superplot

The Folk Theorem

The Pinch-Hitter Syndrome

Weakly Informative Priors

P-values and U-values

Conservatism

WWJD

Theoretical and Applied Statisticians

The Fallacy of the One-Sided Bet

Alabama First

The USA Today Fallacy

Second-Order Availability Bias

The "All Else Equal" Fallacy

The Self-Cleaning Oven

The Taxonomy of Confusion

The Blessing of Dimensionality

Scaffolding

Ockhamite Tendencies

Bayesian

Multiple Comparisons

The background of the study is a delightful tangle. Seth reports:

One of the authors of the Duke study is a professor of psychiatry, Susan Schiffman. An earlier study of hers had pro-Splenda results. . . . Drs. Abou-Donia and Schiffman admitted that some of the results recorded in their report submitted to the court were not actually observed or were based on experiments that had not been conducted. . . .

Results in the report that were based on experiments that had not been conducted . . . that seems pretty bad to me! On the other hand, as Seth points out, maybe "the only way doctors learn about bad side effects of this or that drug is when drug reps selling competing drugs tell them." In this case, it's the Sugar Institute, not a drug rep, but maybe the same idea.

It reminds me of what Phil and I said when trying to publicize our work on decision making for home radon exposure. There's no radon lobby (radon is a radioactive gas that occurs naturally) and so there's an asymmetry, with various organizations motivated to oversell radon risks and scare people, and not too many people on the other side.

P.S. I'd never actually heard of Splenda before, but I do remember the controversy in the 1970s about saccarhin--I seem to recall that rats were getting cancer after being fed the equivalent of 800 bottles of diet soda a day--and then I remember there was something called Nutrasweet, so I guess Splenda is another one of these. It's pretty funny that I'm so removed from pop culture to be unfamiliar with Splenda, a substance that I'm assuming is omnipresent, given that Seth discussed it without feeling the need to identify it at all to his readers.

P.P.S. It says in the press release that a trial has been set for January 2009, so maybe there's more news on this.

The [Senate] Finance Committee bill also includes language requested by industry lobbyists (pages 1138-1139) that threatens to withdraw federal funding for 5 years from any investigator who publishes a report on research funded by the proposed institute that is not "within the bounds of and entirely consistent with the evidence." Determinations regarding such consistency would be made by the newly created research entity, which would have industry involvement both in its governance and in study design.

Selker and Wood continue:

To allow scientists -- and their institutions, which receive the support for the conduct of research -- to be punished for the publication of work that is not approved by this entity is essentially to cede authority over the dissemination of government-funded research to a body that is at least partially controlled by persons with a potential commercial interest in its outcome. This move would be a major retrograde step that would both inhibit the conduct of CER and call its integrity into question. In addition, because researchers and their institutions will seek to avoid such punishment, this provision is likely to result in prolonged arguments, taking place out of public view, regarding which data are acceptable to publish, thereby impeding and delaying publication.

I don't really know anything about the details here--this was just something sent to me by email--but I'm inclined to agree that this kind of gag order can't be a good idea.

If any of you know more about this, feel free to comment and educate me on this.

P.S. The second author of this article works for Symphony Capital, a New-York based company that describes itself as a "private equity firm dedicated to collaborating with leading innovative biopharmaceutical companies, helping them capture more of the value in their most important clinical development programs." I don't know where that puts them on any conflict-of-interest scale. (As a sometime consultant myself, I certainly wouldn't say that working for a private investment firm should automatically disqualify someone from opining on these topics.

Check out the GDP chart under "The New Triad" here:

It's supposed to compare GDP in China, India, and the US for three time periods but for my money, it's composed wrong. The bars should be for the years, not the countries. That way we could see total GDP in each year and how it was composed of the respective GDPs. It would even be fairly easy to scan across and see how the country's GDP grew or shrank. What's there is just confusing.

I agree. Better to put time on the x-axis if possible. Then you don't need a bar plot at all, you can use a line plot, which I generally prefer.

My only suggestion beyond what Ioannidis wrote has to do with potential solutions to the problem. His ideas include: "being cautious about newly discovered effect sizes, considering some rational down-adjustment, using analytical methods that correct for the anticipated inflation, ignoring the magnitude of the effect (if not necessary), conducting large studies in the discovery phase, using strict protocols for analyses, pursuing complete and transparent reporting of all results, placing emphasis on replication, and being fair with interpretation of results."

These are all good ideas. Here are two more suggestions:

1. Retrospective power calculations. See page 312 of our article for the classical version or page 313 for the Bayesian version. I think these can be considered as implementations of Iaonnides's ideas of caution, adjustment, and correction.

2. Hierarchical modeling, which partially pools estimated effects and reduces Type M errors as well as handling many multiple comparisons issues. Fuller discussion here (or see here for the soon-to-go-viral video version).

The U.S. Chamber of Commerce and an assortment of national business groups opposed to President Obama's health-care reform effort are collecting money to finance an economic study that could be used to portray the legislation as a job killer and threat to the nation's economy, according to an e-mail solicitation from a top Chamber official.

The e-mail ... proposes spending $50,000 to hire a "respected economist" to study the impact of health-care legislation ... would have on jobs and the economy.

Step two, according to the e-mail, appears to assume the outcome of the economic review: "The economist will then circulate a sign-on letter to hundreds of other economists saying that the bill will kill jobs and hurt the economy. We will then be able to use this open letter to produce advertisements, and as a powerful lobbying and grass-roots document."

Don't worry, though, they didn't really mean it:

Randy Johnson, the Chamber's senior vice president who handles health-care issues, called the e-mail "inartfully worded" and said the group never intended to suggest that the outcome of the study would be preordained.

"It's not saying that we would tell the economist how it should come out. . . .

Of course, you don't need to tell the economist how it should come out, you can just let him or her know ahead of time that you'll only pay if it comes out a certain way. Of you don't even need to do that, really, you can just make your expectations clear.

Hey, wait a minute!

OK, OK, it's easy to laugh at the Chamber of Commerce here, and I think what they're doing here is bad: the goal is to influence policy via a scientific-seeming study whose conclusions are decided in advance. In this particular case, it might be OK--maybe their pre-chosen conclusion is correct--and maybe this issue is important enough that it's worth a little lying and cheating, all's fair in love and war and all that. But this particular defense isn't going anywhere; I mean, you can defend almost anything by arguing that it benefits a larger cause.

The more serious issue is that this predetermined-conclusions thing happens all the time. (Or, as they say on the Internet, All. The. Time.) I've worked on many projects, often for pay, with organizations where I have a pretty clear sense ahead to time of what they're looking for. I always give the straight story of what I've found, but these organizations are free to select and use just the findings of mine that they like.

This also reminds me of something I've noticed on legal consulting projects: typically, the consultants on the other side seem incompetent, sometimes extremely so. I have a few hypotheses here:

1. Most statistical consultants are incompetent. Lawyers etc. tend to find statistical consultants via word of mouth, and they're likely to get a professional consultant who is really bad at statistics but good at promoting himself. (Even when academic consultants are hired, the lawyers often pick the wrong academic--somebody who might be a top researcher but who has no expertise in the topic at hand and doesn't know enough to turn down the project.)

2. Maybe the side that hires me is actually typically in the right, and they hire me because they want their argument to be made transparently. Conversely, if you're a lawyer and you don't really think the numbers support your case, maybe you'd rather hire a hack who will say anything you want--he's dependent on you for the money--than a professor who feels free to draw his own conclusions.

3. Maybe I wasn't really on the right side in many of these cases; maybe I'm just fooling myself to think so.

I was pretty dismissive of the article; in fact elsewhere I gave my post the title, "Some ESP-bashing red meat for you ScienceBlogs readers out there."

Dr. Bancel was pointed to my blog and felt I wasn't giving the full story. I'll give his comments and then at the end add some thoughts of my own. Bancel wrote:

I find it disappointing that a Columbia faculty member should, in his public blog, be content to substitute facile derision for informed argument in criticizing a research article. It is an unfortunate choice, as it merely adds to today's wearisome environment of ad hominem public discourse, while missing an opportunity to educate.

I won't bother to explain here the errors in your post. Such explanations are all in the article - you would only need to spend more than "a few minutes" to appreciate that. There we explain why the RNGs are shielded, and we emphasize that the effect size is very small, which is essential to understanding why the experiment is run and analyzed the way it is. We do verify our results (as you suggest) with a re-sampling analysis over the full database of 4,000 days. All of this, again, is detailed in the paper.

You say these issues are incidental to your main critique. But it is not clear just what your main objection is. You indicate that the article is "very professional", but flawed, because we propose no theoretical framework (my interpretation of your second paragraph). This might be the entry point for an interesting discussion. But then - after tangential remarks - you pick this up at the end by suggesting (if I correctly read past the polemics) that we blindly manipulate our data, which is grossly wrong, and inconsistent with your opening comments.

It is regrettable that you have used a public forum to misrepresent work which you have, as you state, spent but a few minutes reviewing. It is also unfortunate if you have passed these misrepresentations on to a journalist. In your American Scientist article this summer, you warn journalists not to be misled by brash statements and to seek the best advice of scientists when writing about science. That works only if the scientists go the whole nine yards and make the honest effort to give good advice.

Lastly, I would say that I am doubly disappointed in your post since your own expertise is complementary to our own, and we benefit from any valid criticism based on a careful reading of our paper. Without hesitation, I'd say we welcome it.

I replied:

Thanks for the response. My blog represented my opinion based on a quick look, but I agree this is not my area of expertise. I would like to run your response (without comment from me) on both of my blogs. Would that be ok with you? I would like the readers to get both sides of the story.

To which Bancel replied:

If you feel my personal email to you is appropriate to post, please do so, but a brief response of explanation might be more interesting.

Perhaps you could suggest a couple of issues that I could address as it is still not clear to me just what your objections are.

I did appreciate two points you indicate in your post.

One is that you distinguish between the analysis and the topic itself. Most researchers conflate the admittedly questionable GCP hypothesis and the quality of analysis. These are, of course, separate issues.

The other is the difference in style between the social sciences and physics. This leads to unnecessary misunderstandings. I benefit enormously from interacting with scientists in different disciplines but the challenge is always to understand the mindset, since it determines how people frame the questions they ask.

As far as experimental physics (and the hard experimental sciences in general) and statistics, there are really 3 worlds here: laboratory research, where one works hard to achieve huge effect sizes. In this world one usually doesn't need to have much statistical sophistication. The second is modeling and simulation which is highly coupled to theory. The last is the experimental study of "natural records". This includes astronomy, geology, climate science, etc. Here you often take what you get and data can be noisy, heteroscedastic, etc. so that statistical sophistication is key. This is a caricature and of course these all overlap and interact. My point is that physics obviously isn't a monolith and good physicists may need some skills and not others. Presumably their training allows them acquire new skills as needed, often with the helpful guidance of colleagues in other fields.

I don't think it's appropriate for me to give long reply in response, so I'll just make a couple of general comments.

1. I think the biggest issue is that ESP is something that Bancel and Nelson are particularly interested in, but it's not something that I care about much at all. I don't want to go around claiming that ESP isn't real, or anything like that--I think it's enough to say that whatever effects are there, are very small, so small that they don't particularly interest me.

In contrast, I get much more irritated when people do bad science on topics that are potentially important (for example, the crappy studies I've mentioned on the blog on occasion, on topics such as political effects of the number of cabinet ministers in a country, or the alleged irrationality of voting, or the purported liberal voting tendencies of rich people, or, hmmm, was there something once about engineers having beautiful babies, or something like that . . . I can't quite remember . . .). Some statisticians get particularly outraged about shaky medical claims, but I don't know enough about medicine to get involved in such fights.

2. ESP statistics is pretty sophisticated. When you have large effects, you don't necessarily need sophisticated methods. But when effects are very weak, you might need very large sample sizes, sophisticated corrections for nonsampling errors, multiple comparisons adjustments, and so forth. I respect the statistical methods that have been developed in ESP research (and in psychometrics more generally), but I think they're still in a tough spot because of the small magnitudes of the effects they're studying.

As I've always said, what makes a statistician look good is not teasing out a small effect, but finding a huge effect that hasn't been noticed before. Sometimes fancy methods can help us find big effects (as in Red State, Blue State), but then we should be able to go back to the raw data and find these as well (again, as in Red State, Blue State). In that sense, the fancy methods are helping us do a more effective job of exploratory data analysis.

I don't have anything more to say about the Bancel and Nelson article in particular. It's out there, and youall can make your own judgments of it. (Please be polite in any comments. I appreciate that Bancel responded to my blog, and I don't want to reward him with a bunch of rude replies. Thanks.)

This discussion from Keynes (from Robert Skidelsky, linked from Steve Hsu) reminds me of a frustrating conversation I've sometimes had with economists regarding the concept of "risk aversion."

Risk aversion means many things, but in particular it is associated with attiitudes such as preferring a certain $30 to a 50/50 chance of having either $20 or $40. The standard model for this set of attitudes is to assume a nonlinear function for money. It is well known that reasonable nonlinear utility functions do not explain this sort of $20/30/40 attitude (see section 5 of this little article, for example); nonetheless the curving utility function always comes up in discussion, requiring me to waste a few minutes before going on, explaining why it doesn't explain the phenomenon.

It would be as if any discussion of intercontinental navigation required a preliminary discussion of why the evidence shows that the earth is not flat. . . .

I spent a few minutes looking at the article, and, well, it's about what you might expect. Very professionally done, close to zero connection between their data and whatever they actually think they're studying.

Harlem Children's Zone (HCZ) is arguably the most ambitious social experiment to alleviate poverty of our time. We [Dobbie and Fryer] provide the first empirical test of the causal impact of HCZ on educational outcomes, with an eye toward informing the long-standing debate whether schools alone can eliminate the achievement gap or whether the issues that poor children bring to school are too much for educators to overcome.

Their conclusions are extremely positive:

Harlem Children's Zone is enormously effective at increasing the achievement of the poorest minority children. Taken at face value, the effects in middle school are enough to reverse the black-white achievement gap in mathematics and reduce it in English Language Arts. The effects in elementary school close the racial achievement gap in both subjects. Harlem Gems and The Baby College, the only two community programs in HCZ that keep detailed administrative data, show mixed success. We conclude by presenting three pieces of evidence that high-quality schools or high-quality schools coupled with community investments generate the achievement gains. Community investments alone cannot explain the results.

Here's how they address the potential concern that kids in the program will be better-prepared than the control group of kids not in these schools:

We implement two identification strategies. First, we exploit the fact that HCZ charter schools are required to select students by lottery when the demand for slots exceeds supply. Second, we use the interaction between a student's home address and cohort year as an instrumental variable.

Here's the punch line:

"Winners" here are students who receive a winning lottery number or who are in the top ten of the waitlist.

They also show results for English tests which are positive, but less impressive. They remark that, "Interventions in education often have larger impacts on math scores as compared to [English] scores (e.g. Decker et al., 2004; Rockoff, 2004; Jacob, 2005). This may be because it is relatively easier to teach math skills, or that reading skills are more likely to be learned outside of school. Another explanation is that language and vocabulary skills may develop early in life, making it difficult to impact reading scores later (Hart and Risley, 1995)."

What does this all mean?

I haven't looked at the statistical details of this paper--that's hard work!--but I do have a few comments, to be made on the assumption that Dobbie and Fryer's analysis is essentially correct.

My first comment is that my mindset, before reading this paper, was that more effective teaching methods do exist--KIPP and the like--and that the way they work is by getting the teachers and students to work harder and longer than is usual during the school day. The Dobbie and Fryer paper did not change my view on this; they write "Our rough estimate is that Promise Academy students that are behind grade level are in school for twice as many hours as a typical public school student in New York City. Students who are at or above grade level still attend the equivalent of about fifty percent more school in a calendar year."

This is not to dismiss the findings--it's not so easy to motivate teachers and students to work twice as hard--but just to connect these results to other things that I've heard.

My second comment is that these schools are described as a way to close the gap between whites and blacks in school performance. But if they're so effective, maybe they'd be applied to white kids also? Or is the point that these school changes would really only be applied as part of a package of interventions in predominanty-minority neighborhoods? I'd like to hear more about this issue in the Conclusion section of the article, which raises the idea of following up in regular public schools.

Silly little things

Dobbie and Fryer's paper has excellent graphs--something you don't always see in work by economists. I'm happy to see that the top economists are presenting their work graphically--this seems like an excellent sign. I just have a couple of minor comments:

I'd prefer if Figure 1 (the map) were shown in a non-distorted way and with more information that is relevant to the study. For example, more information about exactly where the kids live, where the schools are, etc. The existing map is hard to read partly because it is distorted (or so it looks to my eyes), meaning that the distance scale is not so meaningful, also the orange background color makes it hard to see any details at all. Beyond this, the map includes irrelevant information such as the path of the Central Park road; this is the sort of thing that Ed Tufte correctly calls "chartjunk." In this case, the authors didn't add the chartjunk; they just put their info on an existing map. Nonetheless, the end result of this otherwise-potentially-useful map is to show nothing much more than that the Harlem Chlidren's Zone is, indeed, located in Harlem.

Figure 2 is just great. I have only three small suggestions:
- Reduce the y-axis scale. There's no reason to go all the way from -.6 to +.5; you can restrict to the range of the data, which is from -.4 to +.3. Even a small change like this will help a lot, actually.
- There's something weird going on with the y-axis. You can't put "percent enrolled" on the same scale as test scores! That's like saying that my groceries cost $25 and it's 15 degrees out, so my groceries are higher than the temperature. Also, you have to be careful with the whole "percentage" thing. Does ".2" on the percentage scale correspond to 0% or to 20%.
- Also, once you get rid of the percentage thing, you can really expand the scale, because the red and blue lines are all between -.4 and .02 on the y-axis.
- Beyond this, how to we interpret a test score of -.2? That doesn't seem right. I assume that the actual scores are positive, and that this is all explained in the text, but I really think that graphs should be as self-contained as possible.
- The color scheme is great (once you can explain how percentages and test scores fit on a common scale). I'd recommend labeling the lines directly rather than using a legend. Once you fix the scale, the lines will be farther apart also.
- 2003 should come before 2004. In the graph shown, 2004 is on the left and 2003 is on the right, which is counter to the conventional way of displaying time ordering.

I won't go over the other graphs line by line, except to say that they're basically fine. I would prefer, however, that they use a consistent color scheme throughout. In Figure 2, blue represents Math score and red represents English score; in the other figures, blue means Lottery Winners and red means Lottery Losers.

And then there are the tables. I think you know already what I'm going to say, so I won't bother to say it. (I mean, 10.424 with a standard error of 7.167? What are these people thinking?) I know, I know, default choices don't need to be justified. But, still . . .

It's worth emphasizing, at this point, that I think the authors present their results very well, both graphically and in the text of their article. It's only because they took the leap to make these solid graphs, that I can take the next step and try to help them do even better next time. I think one of the roles of a statistician such as myself is to help researchers do their jobs even better--and this is particularly satisfying in settings such as this, where there's no way I would've been doing the research myself.

The last line of the acknowledgments says, "The usual caveat applies." I have no idea what that means--something in economics-speak? I have noticed in general that econ papers have longer acknowledgment sections than stat papers do. My theory has always been that economists write fewer articles and put more time into each one, whereas statisticians spit out articles at a machine-gun rate and don't look back. The two fields have different systems: my impression is that in econ, it's a big deal to be published in the American Economic Review or wherever, whereas, in stat, an article in JASA or Annals of Statistics or wherever won't necessarily get noticed anyway.

For an English-born reader, America is written in a language deceptively similar to one's own and full of pitfalls and 'false friends'. The word nature, for instance, means something different here - so do community, class, friend, tradition, home (think of the implications beneath the surface of the peculiarly American phrase 'He makes his home in ...').

I can't tell if Raban is being serious or if he is making some sort of joke. The paradox of the statement above is that very few readers will be qualified to assess it.

In any case, if someone can explain to me how nature, community, class, friend, tradition, and home have different meanings in English and American, I'd appreciate it. I've read a lot of things written by English people but I have no idea whatsoever what he's taking about.

The thing that I find hard to believe is that the number of hits on most of these articles is so low. For example, if I google "World War II," the Wikipedia entry comes up first. But according to the list linked to here, there were only 30,000 visits to the World War II Wikipedia page in all of 2009. I have similar problems with the other numbers. Could they really be so small as all that? Or am I thinking about this all wrong?

Curious, I looked up "Martin Voracek" on the web and found an article about body measurements from the British Medical Journal. The title of the article promised "trend analysis" and I was wondering what statistical methods they used--something more sophisticated than lowess, perhaps?

They did have one figure, and here it is:

Voracek and Fisher, the critics of lowess, are fit straight lines to data to clearly nonlinear data! It's most obvious in their leftmost graph. Voracek and Fisher get full credit for showing scatterplots, but hey . . . they should try lowess next time! What's really funny in the graph are the little dotted lines indicating inferential uncertainty in the regression lines--all under the assumption of linearity, of course. (You can see enlarged versions of their graphs at this link.)

As usual, my own house has some glass-based construction and so it's probably not so wise of me to throw stones, but really! Not knowing about lowess is one thing, but knowing about it, then fitting a straight line to nonlinear data, then criticizing someone else for doing it right--that's a bit much.

Not just lowess

Just to be clear, when I say "lowess is great," I really mean "smoothing regression is great"--lowess, also splines, generalized additive models, and all the other things that Cleveland, Hastie, Tibshirani, etc., have developed. (One of the current challenges in Bayesian data analysis is to integrate such methods. Maybe David Dunson will figure it all out.)

I currently have no home internet service, and so may not be able to answer your message swiftly. Thank you for your patience.

This is sort of funny, partly because of the implied expectation that everyone has home internet service, and partly because I send him the message at around noon in his time zone on a Monday, so I wouldn't expect him to be at home anyway!

This also reminds me that I'm toying with the idea of removing myself from the internet for a period of three days each week, so I can get more work done. I have enough backlog blog entries that I could still leave timed posts every day, so that wouldn't be a problem.

Of course, I could start by taking just one day internet-free and seeing how it goes, but there's something appealing about committing to the three-days-a-week policy. Or maybe I could stay on skype (that way, my postdocs could check my email and notify me if something really important happens) but stay completely removed from browser and email. I'm still not sure what to do on this one.

For example, Kevin Drum describes the chart as "pretty cool" and writes, "I think I'm more interested in the placement of senators themselves. Democrats are almost all bunched into a single grouping, with only four outliers. Republicans, by contrast, are spread through considerably more space on both the economic and social dimensions."

Matthew Yglesias also labels the chart as "cool" and answers Drum by describing the pattern as "an illustration of the importance of setting the agenda. The Democratic leadership has only brought to a vote bills that unite the overwhelming majority of Democrats. . . ."

Yglesias may well be right on this point, but before going further I'd like to stand athwart history and yell Stop" for a moment.

My first reaction when seeing the above graph was, Huh? it doesn't look right to me. The graph seems to imply that Dems and Reps have a huge huge overlap on social issues, with the median positions of the two parties being virtually identical (and a Democratic senator in Vermont being quite a bit more socially conservative than Republican senators in Indiana, Tennessee, and two senators in Arizona). Can this really make sense?

I asked Brendan, who responded:

The graph is an auto-generated plot of the Lewis-Poole optimal classification scores for the 111th congress generated by Royce Carroll, one of Poole's students. So the important thing to keep in mind is that it's only being run on part of one Congress (rather than say, DW-NOMINATE, which Poole runs on all the Congresses as a batch), so the estimates may be screwy depending on the set of available votes. In this case, there apparently haven't been a lot of votes dividing the Senate Dems internally so their estimated ideal points are tightly clustered, whereas GOP divisions on the votes to date have caused their ideal point estimates to spread in two dimensions (this is not true for the House, where the Dems have had more internal division). Also, the second dimension that's being recovered for the last ten months in the Senate may or may not be the "social issues" dimension that Carroll labels it. The second dimension is always an interpretive mess in the post-civil rights period, and it's even worse for <.5 of one Congress. . . . This is my (Brendan's) best guess at what's going on and I don't know exactly what Carroll and/or Poole are doing behind the scenes.

OK, this makes sense. My take-home message here is that we should ignore the second dimension of the above graph, at least until someone can come up with some interpretation of it. The problem isn't simply an artifact of sparse data or agenda-setting; more fundamentally, we have to know the meaning of a variable before we start talking about it! A key step in any statistical analysis is to connect the inferences back to what is already known about the underlying system (in this case, the positions of senators on social issues).

Or, to put it another way, don't believe everything you see on a graph.

P.S. I don't mean this to be intended as some sort of devastating critique of Carroll's work. I've presented enough mistaken graphs on my website that I certainly can't blame others for posting things without making a sanity check first. Actually, posting stuff quickly on the web is a great way to get others to find your mistakes! And I hope that this post and others will be helpful to Carroll as he continues his research (and also helpful to me once I receive the inevitable corrections of whatever mistakes I'm making here).