Debunking the Upper Tail: More on the Gender Disparity

Earlier this week I argued that the gender differences in cognition, while real, are not substantial enough to explain gender disparities in science. We talked about the work of Janet Hyde; it shows that -- contrary to the popular conception of women and men as psychologically disparate -- men and women are actually quite psychologically similar in most respects.

Some of the commenters brought up the issue of the upper tail, and I want to talk about that specifically. It has been suggested that even if the size of the effect -- the differences in averages between a trait like mathematical ability -- is not very large, it becomes magnified at the edges of the distribution. Thus, even a barely statistically significant effect, when observed at the threshold of 4 SD, could result in a several fold skew towards men. The argument goes on that if such a trait were required to participate in a discipline -- like competence in mathematics to become a physicist -- women would then be acting at a natural disadvantage.

This argument assumes two things:

1) A high measure of mathematical ability as measured by cognitive testing is required for technical disciplines like engineering, mathematics, computer science, and physical science college majors and careers. (I shall henceforth refer to these as EMS disciplines because that is the terminology used in the paper I want to talk about.) This is not to suggest that math itself is not involved; rather, I am just pointing out that this argument is predicated on the idea that A) the mathematical ability required for the disciplines is something we can measure and B) the thing we measure is the thing required.

2) Mathematical ability so measured is correlated with participation.

Neither of those propositions are true, as demonstracted by work for NBER by Catherine Weinberger. Her study looked followed high school seniors from 1972 and 1980 through college to correlate mathematical performance with later participation in an EMS discipline.

This study uses data from nationally representative samples of 1972 and 1980 high school seniors, followed longitudinally through the college years to answer the following questions: Is the EMS participation of young women similar to that of young men with the same ability? and: Did most of the men who pursued EMS careers have very high mathematics test scores, or unobserved ability, as high school seniors?

Here is a brief summary of the main point of that paper:

• The level of EMS attainment by women is less than that of men, in line with Census data:
  

  Among 1972 high school seniors, 4.5% of the men and 0.9% of the women graduated college with
  an EMS major by the 1979 resurvey. Among 1980 high school seniors, 5.8% of the men and 1.8% of the women completed EMS degrees by the 1986 resurvey.

• The numbers of women at the upper tail of the mathematical distribution is less than that of men for both the entire sample and the subset of bachelor's degree graduates:
  
  

  I now turn to the question of whether EMS participants are drawn from the extreme upper tail of the math score distribution...Figure 3 shows what we already know: there are more men than women at the upper tail of the math score distribution. Figure 4 shows that this is also true within the (more highly selected) group of college graduates. But, within the even more highly selected group of EMS college graduates, the distributions of men's and women's math test scores are much more similar (Figure 5). In fact, among EMS graduates in both cohorts, the women have a higher mean math score and smaller variance than the men. Note that if the relationship between mathematics test scores and EMS participation were the same for men and women, then female EMS participants, drawn from a lower test score distribution, would tend to have lower average scores than male EMS participants. The finding that the mathematics test scores of white women with EMS degrees are not lower than those of the men suggests that women are morecautious about entering unless they have very high scores. (Citations have been removed. Emphasis mine.)

• Here is where it gets interesting. The levels of math achievement for those males that ended up in the EMS discipline were not espcecially high, and by no means prerequisite:
  
  

  Among all white male 1972 high school seniors employed in the 1979 workforce, only 6% had high school SAT-M[ath] scores greater than 650, while 76% had scores no greater than 550. Although men with high scores are overrepresented in the EMS workforce, they are not the majority. No more than 25% of the EMS workforce had scores above 650. High scores were somewhat more common in the college-educated EMS workforce. Among college graduates employed in EMS occupations, 29% had high scores as high school seniors. Further restricting attention to those who earned an EMS bachelor's degree by 1979 and were employed in an EMS occupation in either 1979 or 1986 did little to change the estimate. Although this group would include those who attended graduate school before entering the EMS workforce, no more than 30% had scores above 650, while 32% had scores no greater than the average humanities major. These estimates lead to the surprising conclusion that less than one-third of the EMS work force had SAT-math scores above the threshold previously presumed in the economics and cognitive psychology literatures. (Citations have been removed. Emphasis mine.)

  It would appear that the imputed threshold is not nearly the price of admission that some people would argue. Either we are not measuring the mathematical acumen that is required, or it is not as required as we think.

• Some people might argue with the above point that "Hey, well those people who do poorly at math are probably not being as good a physicist right? Maybe they just got a degree but they are not any good at it." Well, Weinbeger shows that if you look at the math scores compared with later earnings, the individuals with low math scores are doing just as well:
  
  

  The result that the lowest scoring EMS graduates enjoy the same economic returns as graduates with high scores is extremely robust. This finding suggests that many individuals with lower ability, as measured by the SAT-M[ath], are able to complete college level coursework in EMS subjects, and to enjoy the associated wage premium.

  Granted this isn't a direct measure like publishing frequency of working physicists, but considering that this is fundamentally an economic issue I think it is a good proxy for success.

• Some housekeeping. Some might argue that SAT-Math is not a good test. Well if you substituted another test, the Cognitivie Test of Mathematics (CTM), results don't change:
  
  

  Within this sample, the estimated gender differential does not change when SAT-M scores are substituted for CTM scores.

• Say we are just not measuring the right thing. Say there is some "right stuff" that men have that women don't that accounts for their higher participation. How much of a boost would be required to give them this disparity? It turns out it would have to be quite large:
  
  

  Of course, any possible gender differential could be explained by gender differences in an unobserved ability of some sort. In fact, this is the model underlying the notion that the demographic composition of the EMS workforce reflects the underlying distribution of attributes. In an illustrative example, equal participation conditional on ability is imposed. This exercise allows us to estimate the gender differential in the dispersion of unobserved ability required to sustain this model...In order to sustain the premise of this model, we have to believe that women with math scores at the men's 90th percentile have the same average EMS ability as men scoring at the 78th percentile of male high school seniors. The gender differential in participation can be explained away if we assume that men have quite a bit of "the right stuff," but this is no longer an argument based on gender
  differentials in math scores.

• Finally, we can mathematically calculate the "threshold" for participation in the EMS field:
  
  

  The estimated threshold for EMS participation is surprisingly low, 578 SAT-M points. And the estimated dispersion of unobserved ability is so high that 57 percent of men in this sample are estimated to have true EMS ability above the threshold. This estimate of the proportion of men above the EMS ability threshold is more than 50 percent higher than that obtained using CTM scores with the same sample and model. This finding means that, if SAT-M scores were available for a representative sample, the estimated true ability threshold would be far below the 78th percentile estimated using CTM scores.

  There remains little room for doubt that many individuals with scores, and ability, well below the "top few centiles" complete EMS college degrees and enter well-paid occupations.

  For a threshold that is that low, the disparity between men and women would have to be insanely large to justify the differences in participation. No cognitive test for mathematical ability between men and women has ever found a difference that large.

Here are her conclusions:

It was previously widely believed that entry into bachelor's degree level careers in engineering, mathematics, computer science, or physical sciences (EMS) was limited to individuals who had ability (as measured by high school mathematics test scores) in the top few centiles (Paglin and Rufolo, 1990, Benbow and Arjmand, 1992, Hedges and Nowell, 1995). This belief was so well accepted that no empirical analysis testing this assumption has yet been published.

This study shows that bachelor's degree level EMS participants are drawn from throughout the upper 40% of the mathematics test score distribution. While it is true that there is a high correlation between high school mathematics test scores and EMS participation, many EMS participants had high school mathematics test scores that were not exceptional. For example, fewer than one-third of college-educated white men working in EMS occupations had high school SAT-M scores above 650 (out of 800), while more than one-third had SAT-M scores below 550--the score of the average humanities major--at the 76th percentile of all white men.

The lowest scoring white men with EMS college degrees have the same earnings advantage, relative to college graduates with the same scores, as high scoring EMS participants. This suggests that EMS education is equally beneficial for this lower ability group. When an unobserved component of ability is explicitly modeled, white male EMS participants are estimated to come from the upper 22% of the "true" ability distribution. It can not be determined from this analysis whether the unobserved component of ability is due to under-performance on the math test, or whether the unobserved ability important to EMS success is something that simply cannot be measured by these tests.

The result that relatively few EMS participants were drawn from the top few centiles of the mathematics test score distribution is robust to basing the definition of EMS participation on either educational or occupational attainment, and to using scores from two different standardized mathematics tests. Because EMS participants are not drawn from the very top of the distribution, there are many non-participants with comparable math scores or ability. (Emphasis mine.)

What conclusions do I draw from this?

1) I continue to believe that the cognitive differences between men and women are a genuine and important area of study. The effects of gender on the brain are another important part of understanding the totality of brain function. However, the evidence continues to suggest that the gender differences between men and women in several cognitive traits are not sufficient to explain the disparity of participation between them in scientific occupations.

2) The entire idea that a single axis model of mathematical ability could explain achievement was always fundamentally flawed. Do you sincerely believe that whether someone is a good physicist or a good mathematician can be explained so simply? I don't.

The reality is that many things go into making a good scientist, most of them complicated. Part of it is a raw ability, but that ability is can measured in a variety of different ways and can certainly not be boiled down to a single number (if it can be measured at all). But also part of it is also intuition, drive, wisdom, and experience. These things are not well quantified by tests, and there are certainly not documented differences in them between men and women.

The sooner that we abandon a simplistic notion that single value X equals ability to do physics, the sooner that we will realize that the impediment to women in science is us -- and it is so easily amended.