Doubts About the STAR Study: How Much Is Kindergarten Really Explaining?

Since I've been writing a lot about education, I have some brief thoughts about the NY Times report by David Leonhardt about some findings from Tennessee's Project STAR which tracked the long-term outcomes about a randomization trial of kindergartners (slides from a presentation are available as a pdf):

Just as in other studies, the Tennessee experiment found that some teachers were able to help students learn vastly more than other teachers. And just as in other studies, the effect largely disappeared by junior high, based on test scores. Yet when Mr. Chetty and his colleagues took another look at the students in adulthood, they discovered that the legacy of kindergarten had re-emerged.

Students who had learned much more in kindergarten were more likely to go to college than students with otherwise similar backgrounds. Students who learned more were also less likely to become single parents. As adults, they were more likely to be saving for retirement. Perhaps most striking, they were earning more.

All else equal, they were making about an extra $100 a year at age 27 for every percentile they had moved up the test-score distribution over the course of kindergarten. A student who went from average to the 60th percentile -- a typical jump for a 5-year-old with a good teacher -- could expect to make about $1,000 more a year at age 27 than a student who remained at the average. Over time, the effect seems to grow, too.

First, my reading of the presentation is that the jump is only around $500 per year (see slide #46); ironically, the $1,000 increase is attributed to teacher seniority (see slide #33). Second, the researchers admit that they can't distinguish between the influence of the teacher and of their peers in the class (slide #39)--to claim that "some teachers were able to help students learn vastly more than other teachers" is not supported by the experimental design.

But there are other issues too:

1) While teacher quality accounts for ~$500, what accounts for the rest of the $15,000 difference on slide #9? If anything this suggests that teacher/class quality accounts for very little of the variation in future income.

2) Related to point #1, the researchers should show the details of the ANOVA for student demographics and KG test scores. It would be helpful to know the extent to which different demographic variables contribute to test scores (and, thus potential future income).

3) On slides 41-42, if I'm reading it correctly, it looks like the demographic variables, which exclude include teacher quality, are doing a really good job of predicting test scores. So how does class quality supposedly become so important?

It's hard to tell just from the slides (maybe the paper will have more detail), but it seems that, while there is a class/peer effect, demography seems to be much stronger.

A snooty aside: This is why we need more people trained in analysis in our political discourse (even if we write for shit). Most scientists, much to Randy Olson's chagrin, would never make the claims that Leonhardt does; he's straying into McArdle territory.

Bonus prediction: Having banged this out late Wednesday afternoon, I predict that 'progressives' will greet this with renewed assaults on teacherscalls for education 'reform.' Sure as shinola, this will happen.