Math performance in the US: boys and girls have same mean, different variance

Sorry for the light blogging everyone. It has been a busy, busy week.

Some of you may have caught Janet Hyde's latest paper looking at data from the No Child Left Behind Act and math performance in the US. Under the No Child Left Behind Act, states are required to test children for a variety of skills on a yearly basis. The paper looked at math performance across grade-level broken down by gender for 10 states from these tests.

Here is the key graph:

The data includes a measure of effect size called Cohen's d (I discussed it here) and a measure called the variance ratio (VR -- which is just the ratio of the variances).

You can see from the data that the difference between boys and girls for math performance is statistically insignificant across grade-level. (A Cohen's d less than about .2-.3 is considered basically insignificant.) Hyde's work has always been important in the debate to explain the relative dearth of women in the math and the sciences. Some people -- notably former Harvard President Larry Summers -- attributed that dearth to differences in innate ability in mathematics between men and women. This data would argue against that assertion, and that interpretation of these findings in a variety of publications.

On the other hand, as many of my friends and several bloggers pointed out, this is not the end of the statistics. The observed variance between the boys and the girls is consistently different across grade level. One of Larry Summers other arguments was that the difference in participation between men and women could be attributed to this difference in variation. While at the mean ability of men and women is statistically indistinguishable, the tails of the distribution would have greater men than women. (Men would have more dullards and more geniuses due to high variance.) This upper tail effect could explain the relative participation between men and women in the sciences.

Because of this point, Alex Tabarrok at Marginal Revolution was highly critical of the coverage of this paper and suggested that the authors attempted to specifically downplay this observation:

Notice that the greater male variance is observable in the earliest data, grade 2. (In addition, higher male VRS have been noted for over a century). Now the study authors clearly wanted to downplay this finding so they wrote things like "our analyses show greater male variability, although the discrepancy in variances is not large." Which is true in some sense but the point is that small differences in variance can make for big differences in outcome at the top. The authors acknowledge this with the following:

If a particular specialty required mathematical skills at the 99th percentile, and the gender ratio is 2.0, we would expect 67% men in the occupation and 33% women. Yet today, for example, Ph.D. programs in engineering average only about 15% women.

So even by the authors' calculations you would expect twice as many men as women in engineering PhD programs due to math-ability differences alone (compare with the media reports above). But what the author's don't tell you is that the gender ratio will get larger the higher the percentile. Larry Summers in his infamous talk, was explicit about this point:

...if one is talking about physicists at a top twenty-five research university, one is not talking about people who are two standard deviations above the mean...But it's talking about people who are three and a half, four standard deviations above the mean in the one in 5,000, one in 10,000 class. Even small differences in the standard deviation will translate into very large differences in the available pool substantially out.

If you do the same type of calculation as the authors but now look at the expected gender ratio at 4 standard deviations from the mean you find a ratio of more than 3:1, i.e. just over 75 men for every 25 women should be expected at say a top-25 math or physics department on the basis of math ability alone (see the extension for details on my calculation). Now does this explain everything that is going on? I doubt it. As Summers also pointed out it takes more than ability to become a professor at Harvard and if there are variance differences in characteristics other than ability (and there are) we can easily get a even larger expected gender ratio. (Link in original.)

While I agree with Tabarrok that not talking about the variance is a significant omission in the coverage, I understand why Hyde et al. decided to downplay the point. Further, I disagree with his assertion that "higher male VRS have been noted for over a century" without caveats.

A couple problems:

• First, it is indeed true that higher male variance has been noted in some studies comparing males and females on psychological traits. Tabarrok notes this paper. However, a broad survey of psychological differences between men and women found that the variance depends heavily on what attribute you are measuring (I discussed this here). Further, the difference in variance between men and women may be consistent in the US, but international comparisons show different VRs in different countries. The median internationally is that boys have slightly greater variability for math, but there are several countries where girls have greater variability than boys. The present study also indicated that the variance ratio for boys and girls is inverted for Asian American students -- the girls have higher variance. Thus, higher variance in boys is not always a robust finding.
• Second, the suggestion that differences in participation are related to upper tail effects in math ability is predicated on the notion that people in math and sciences are drawn from the upper tail. This is demonstrably untrue (I discussed this here). Catherine Weinberger from that study found that individuals with bachelor's degrees in technical disciplines like science and engineering were drawn from the upper 40% of the mathematics ability distribution. This is a far-cry from the 4 SD upper tail that Tabarrok notes would be necessary for a 3:1 split between men and women. (This is approximately the difference in participation in between men and women in disciplines like physics.) The notion that there is an upper tail (and a very high upper tail too) from which professors are universally drawn is predicated on the idea that there is some quantifiable "right stuff," and that professor search committees are capable of adequately assessing that to the exclusion of women. "We know the upper tail, and you, Madam, are not in it." Yet, anyone who has looked around at their academic department notes a large variation in abilities. There is no "right stuff," or rather there are many right stuffs that are applied to lesser or greater success.
• Third, the observation of higher variance in boys says nothing about whether this is innate or can be modified by the right environment. A recent comparison across countries found that countries with more gender equity have smaller disparities between boys and girls in math performance.
• Fourth, I am much less suspicious of Hyde et al. in downplaying the higher variance finding than Tabarrok. Dr. Hyde is an expert in this field, and it is reasonable to expect to her to know that variance differences in these studies are a sometimes fleeting phenomena. That the authors downplayed this finding is not surprising in light of the other work in this field.

Let me conclude by saying that I agree that the coverage of this article should include some statement about the observed difference in variance. Statistical distributions are not just means. That is just intellectual honesty. However, considering the other data in this field I still do not believe that differences in upper tail effects can account for the gender difference in participation in math and sciences.

Hyde, J.S., Lindberg, S.M., Linn, M.C., Ellis, A.B., Williams, C.C. (2008). DIVERSITY: Gender Similarities Characterize Math Performance. Science, 321(5888), 494-495. DOI: 10.1126/science.1160364