Standards, standardized tests, and society

In cleaning out my open webpages, I came across the video above in an important post at Wired blogs, and it hardly matters that the post is from last October (yes, I keep too many tabs open in Firefox). Rhett Allain argues that there's an inverse relationship between how much standardized testing students experience, and how much learning they experience, and Ken Robinson, in the video, argues that standardized testing assumes that there is a single standard way of learning, or that such standardization is desirable. The video touches on a wide range of points beyond that, and is well worth watching, but this issue of standards and standardized tests is well worth pondering.

In my day job, I spend a lot of time thinking about science standards in the different states, and about how they translate into standardized tests, into textbooks, and into classroom practice. In terms of seeing evolution taught, and taught accurately and honestly, the movement to establish strong state education standards has been very good. From the mishmash of the infamous "10,000 democracies" (there are actually about 17,000 largely autonomous school districts across the country), these standards give some degree of order. Having clear standards gives textbook authors a clear guide, and as the coverage of evolution has increased in standards, so too the coverage of evolution in textbooks has improved. It's hard to measure what's happening in each science classroom in each district, but better textbooks and better standards and better test coverage of evolution can all be expected to translate to better evolution education, and that means better science education.

But the standardization trend also worries me. If you establish clear standards, it only makes sense to have some sort of assessment to ensure that people are learning what they're supposed to be learning. And once you have an assessment, it seems foolish not to insist that the assessment ought to carry some sort of consequences. So you get the current mess of No Child Left Behind/Race To The Top, in which teachers are punished for even modest year-to-year variations (I recall Brad Delong running the numbers on one example, but can't locate it now that I need it!), and class time is devoted to test preparation, not learning. I worry especially that in science classes, standardized testing and an emphasize on standards-based education could too-easily result in a profound miseducation: teaching science-as-encyclopedia (a list of things to memorize), rather than science-as-process (a way of testing claims that requires a particular mindset, creativity and insight, and also subject knowledge).

There are surely ways around that challenge, tests involving lab practicals and opportunities to design experiments, and standards that emphasize those aspects of science, and I remain hopeful that some reasonable middle ground can be found. Science is the same in every state, and in every country. Having clear standards makes sense, since students often move between districts within a state, and increasingly across state lines, and it's helpful for teachers to know what students are already supposed to know. And some sort of testing based on those standards makes sense, because those standards should represent something real about what we think students should know, and standardized tests are an important tool for seeing whether that important content is actually being learned.

The problem comes when the tests take on a bigger role. We go too far when student performance comes to dictate school funding or teacher salary or retention, or when the desire to do well on tests causes teachers to drop important lessons. Or, as Ken Robinson observes in the video above, when the testing reinforces harmful inequities in society, or reinforces harmful mental habits and forces medical treatments on children with learning styles and mental skills ill-suited to a regime of standardized testing (and the dull lecturing style such tests tend to force on teachers).

All of which reminds me of a classic Cosma Shalizi post from 2007, about the heritability of intelligence, and what sort of intelligence we could plausibly expect selective breeding to increase. Cosma begins by noting:

Doing well on standardized, multiple-choice tests calls for certain sorts of cognitive skills, certain kinds of abstract problem solving. Maybe more exactly, you need both aptitude at understanding explicit rules for manipulating symbols, communicated to you through the medium of writing, with very little contextual information to help you make sense of the message, and you need to be willing to follow those rules, even when they are pointless. These are skills that come from making your way through an industrial, or more precisely bureaucratic and mass-literate, society. (Shades of Luria!) These are skills you are more apt to learn if you grow up in a household which is already highly literate, etc., than if your parents and neighbors are all displaced peasants or harassed proles. It's certainly not surprising if someone who grows up in a household of intellectuals (that is, clerks) finds these habits easy to learn.

Q: So the analogy suggests that IQ scores are...?

A: A proxy for the skills and habits encouraged by a bureaucratic society; skills and habits which can be at once highly heritable (because of strong transmission through family and neighbors) and highly learned (within the scope of what it is biologically possible for humans to learn and internalize). Innate ability needn't enter into it at all. The implications for democracy would be nearly nil.

And then extends the analogy:

Suppose that our new alien overlords showed up tomorrow, and after demonstrating that resistance is futile, decide to institute a selective breeding program. They tie everyone's tubes just before puberty, and then at age 25 everyone is given a test in which they must prove certain theorems about non-Abelian Yang-Mills field theories; those who pass are allowed to breed, those who fail are permanently sterilized. If this persisted for, say, a thousand years, I am quite confident that a randomly selected human being from 3007 will be much more likely to be able to prove those theorems than a randomly selected member of the present population.

Q: In this scenario, whatever you call it --

A: How about "Classical Lumps and Their Quantum Descendants"?

Q: I'm going to pretend you didn't say that. So, how would that come about?

A: How should I know? On the basis of the Baldwin effect, and the other sorts of coupling between individual- and population-level adaptation, I'd expect it to happen somehow. It could be that people will evolve to have geometric intuitions about non-commutative Lie groups. Or they might get really good at memorizing proof-length bits of text, if that kind of thing would get past the alien overlords. Or -- and somehow this sounds more plausible to me -- they might just find studying really interesting and enjoyable.

Q: And why does that last sound more plausible?

A: Because if you look at the actual scientific literature on expertise, the thing which stands out for importance is massive, deliberate practice. This takes interest.

...

If IQ really correlates with the ability to flourish in an industrial society (and I'm quite prepared to believe that), then it is, as I said last time, a measurement of the ability to navigate paper-pushing bureaucracies -- to learn to manipulate arbitrary abstract explicit rules, and to do so on command. Presuming that people who don't manage to pull off at least some minimum level of this make very unattractive mating partners, and so have below-average reproductive success, then those of us in developed countries have spent the last one or two centuries breeding for docility, in both senses of the word.

Q: You're saying that --

A: We have met our alien overlords, and they are us.

So many aspects of success in 21st century America rely on ability to succeed on standardized tests, that this is certainly not implausible. The more weight we place on these tests, the more we'll reinforce the genetic correlates of standardized test success, and the more we'll select against (or medicate away) diversity of learning styles, of teaching styles, and mental habits - diversity which is not inherently harmful but which doesn't fit neatly into a bubble sheet.

So this is the tension: I want to know what tests say, but I don't want the tests.