Standards Are the Enemy of Achievement

The US Chamber of Commerce has a education website, which provides "grades" for states based on various measures of their educational performance. One category is "Academic Achievement," based on the percentages of students scoring at or above grade level on the NAEP test. Another is "Rigor of Standards," which is a little fuzzier, but is based on official standards for graduation in that state-- state curricula, exit exams, and that sort of thing.

What's interesting about this is their correlation: if you click back and forth between the two, looking at their spiffy map, you can watch the colors change, and it jumps out (at least to me) that states with a lousy grade in one, tend to have a good grade in the other. That is, "Academic Achievement" seems to be negatively correlated with "Rigor of Standards."

This mostly holds up if you look at the numbers, too (you can download the data sets as Excel files if you click on the little question mark). Out of the 48 states with a grade reported for both categories, there were 17 states that got a D or F grade for "Rigor of Standards." 12 of those states (70%) had either an A or B in "Academic Achievement." Six of the eleven (55%) states with an A or B grade for "Rigor of Standards" got a D or F in "Academic Achievement."

Only one state (Massachusetts) managed an "A" in both, and only one state (Virginia) got a "B" in both. There were two double D's (Missouri and Rhode Island), and one D-F (Hawaii).

So, what the hell is going on here? Why is it that rigorous standards lead to lousy achievement, and vice versa? I can come up with some possible explanations, but I'd like to hear other people's opinions, first.

In the meantime, the data at least give you a quick way of knowing where you want your kids to go to school. And, more importantly, where you want them not to go...

(The original link was from Ezra Klein.)

Tags

More like this

Barack Obama Defeats Barack Hussein Obama | The Onion - America's Finest News Source "Though few had heard of the freedom-hating extremist before, Barack Hussein Obama quickly garnered attention in several key regions of the country, and saw his popularity buoyed by conservative talk-radio hosts…
There's a paper in the Journal of Political Economy that has sparked a bunch of discussion. The article, bearing the snappy title "Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors," looks at the scores of over 10,000 students at the US Air Force Academy over…
I write about education and educational data a lot, and I'm always struck by the insistence that the U.S. K-12 educational system is DOOMED! This is a staggering display of willful ignorance that rivals creationism (and, arguably, is more pernicious). Without going through the entire backstory (…
...too bad MA Governor Deval Patrick, and for that matter, President Obama, don't. The recent educational regression reform plan in Massachusetts and the Obama Administration's educational proposals both seem to misunderstand what has made Massachusetts' educational system one of the best in the…

It seems pretty obvious to me: The states that have higher expectations of what students will learn at each grade level have fewer students who can meet those expectations, and the ones with lower expectations have more kids who can because they're not as demanding.

My suggestion is that states that are failing in the achievement category seek to improve performance by raising standards. The students' performance is either lagging behind the rising standards, or the standards do little to nothing to improve performance.

This doesn't surprise me at all. Fixing a poorly performing education system is hard, but announcing "tough new standards" is an easy substitute for doing the real ground level work required.

As education performance slips, administrators and politicians continually devise new standards in order to create the illusion that they're doing something.

By BiophysicsMonkey (not verified) on 18 Dec 2008 #permalink

So if I'm understanding this right, there's a negative correlation between doing well on some national standardized test, and the local grades assigned by teachers in the state?

If so then I'd guess that the states with overall tougher teachers with more willingness to give students bad grades when they are doing poorly end up teaching their students better and getting higher grades on the standard exams.

Although it could also be that the states where they do well on the exam are so focused on the exam that they don't do as well in normal classes. Except that the standard for what is needed for doing well in class is set by the same teachers that are "teaching to the exam", so I'd think they would also make the exam and class focus similar... so that doesn't make as much sense.

Standards in and of themselves are not evil. Consider the International Bureau of Weights and Measures, and the standard kilogram in Paris. Or what used to be the National Bureau of Standards (now NIST).

But Pedagogy is not a science the way that Physics, Chemistry, Biology, Geology, and Astronomy are. Or even Economics.

Bad standards are worse than no standards at all.

There's a sweet spot between crystalline overstandardized rigidty and random anarchy in the classroom.

In the terminology of The Santa Fe Instituite, I call that "Students On The Edge of Chaos."

That's the short form of the working title of my planned Doctorate in Educational Leadrship.

Mary,

I don't think your interpretation can be correct. As far as I can tell, "Academic Achievement" is measured by a common national test, not by each state's individual expectations. Different states don't give harder or easier tests with this metric.

By Ambitwistor (not verified) on 18 Dec 2008 #permalink

Converting the letters to a 4-point scale and plotting achievement vs. rigor, I get a slight negative correlation:

y = -0.5187x + 2.9127
R^2 = 0.1114

Except that the standard for what is needed for doing well in class is set by the same teachers that are "teaching to the exam", so I'd think they would also make the exam and class focus similar... so that doesn't make as much sense.

Not true at all. There are some teachers involved in setting the standards (or at least I hope there are), but the teachers who have to do the work in the classroom are the ones most opposed to the standards. The 4th grade math teacher at my daughters' school complains that she spends 85-90% of her time teaching MAP testing and almost none teaching actual math.

To clarify a bit: the "Rigor of Standards" category has nothing to do with grades. It purports to measure the quality of the state-mandated curriculum, and whether or not the state mandates exams before graduation.

New York gets an "A" for "Rigor of Standards," for example, because of the Regents curriculum and the mandatory exams in Regents courses. The "A" has nothing to do with how the teachers grade their students.

Maybe it's because states with high "rigor of standards" scores put a whole lot of emphasis on adherence to a specific curriculum, which (in my experience) is generally unbelievably broad--too broad for any reasonable student to possibly do anything other than memorize stuff. And if they also have to pass exams to graduate (giving the state a higher rigor score), then the teachers probably spend a lot of time teaching kids how to pass those tests (since it's more than likely that the teachers are graded, at least in part, on how well their students do on the tests).

If you assume that the NAEP is actually a valid measure of academic achievement (which I don't, necessarily, but bear with me), then it follows that states that put a lot of weight on students adhering to their state curricula and standards (which, as I said, are probably so broad and comprehensive that they don't actually get to master concepts, only memorization) would not prepare the students as well for a test that actually measures understanding, as opposed to memorization.

Put more generally, I think it might be because "rigorous" states put more effort into getting kids to pass their exams, so they're less likely to pass a general exam. Whereas states that are less focused on testing and have less "comprehensive" curricula might allow students more freedom to explore and master concepts, so they're more likely to succeed on any given test (again, assuming the test is a valid way of measuring achievement).

I can only speak for my own state -- MN... The achievement was an A, standards were a C. They were critical of poor alignment of high school standard with college requirements, and they are right. Students come to my CC thinking they are ready for college, but end up in remedial classes or not doing well in regular classwork.

By PhilosopherP (not verified) on 18 Dec 2008 #permalink

If metrics are flawed conclusions are no better. What does the market want as end product? Is it getting it? What is to be done with production rejects (aside from diversity hires and political office)?

New York State annual Regents exams in major subjects worked flawlessly through 1970. Any kid could buy a review book, get educated on the sly, and pass specs. Intelligence (rate the engine in horsepower) testing worked, as did aptitude (the transmission) testing. In 1970 the Community demanded an end to racism (objective evaluation). Every kid went to CUNY, whose diploma then had the impact of single ply toilet paper.

There will always be a bottom 10%. The goal of education - academic and vocational - is for the bottom 10% not to constitute the bottom 80%.

Implementing standards are usually a reactionary measure to a problem that already exists. Why would you implement tough standards in a state that's already exceeding expectations?

Maybe it's because states with high "rigor of standards" scores put a whole lot of emphasis on adherence to a specific curriculum, which (in my experience) is generally unbelievably broad--too broad for any reasonable student to possibly do anything other than memorize stuff. And if they also have to pass exams to graduate (giving the state a higher rigor score), then the teachers probably spend a lot of time teaching kids how to pass those tests (since it's more than likely that the teachers are graded, at least in part, on how well their students do on the tests).

This is, more or less, what I suspect is going on. The states that have good standards spend time teaching to those standards, and not the NAEP. The states without broad state-wide standards can spend more effort on teaching to the NAEP, and thus do well on those tests.

This is further complicated by differences in populations. States with big drops from Rigor to Achievement include California (A to F) and New York (A to C), and both of those states have student populations that will include large numbers of poor urban districts. States with big positive swings include North Dakota (D to A) and Wyoming (the lone F in Rigor, but a B in Achievement), which are more homogeneous.

But I think that the "teaching to the test" effect is a big part of it. Which end you think the problem is on is open to debate-- is it better to have students who do well on math and reading tests, but know nothing about science and history, or students who know a little bit of everything?

"...both of those states have student populations that will include large numbers of poor urban districts."

And large populations of English language learners. Something that is rarely acknowledged, but probably plays an increasingly large role, is that every standardized test is an English test, no matter what other content it is testing. Several studies have suggested that ELL are among the fastest growing "minority" group in the public schools; I can't believe that's not going to affect test scores (but not necessarily standards or graduation requirements).

I'd be interested to see data on the achievement of students in each state relative to that state's standards and requirements. I wonder if states with high standards also have low student achievement relative to those standards, as Mary (#1) suggested.

Critiques of Mathematics Framework for California Public Schools
by
Jonathan Vos Post
EDSE 421M Special Instructional Methods (Math) in Secondary Subjects
Dr. Fred Uy, Thursday, KH-A2027 office; KH-C2091 classroom

Due 30 Oct 08: Reaction Paper [to California Math Framework
(downloaded as PDF) at least 2 pages, commentary cites to specific page numbers

http://csmp.ucop.edu/downloads/cmp/math.pdf

In general, these is much to praise and much to provoke annoyance in Mathematics Framework 2005. Mathematics Framework for California Public Schools, Kindergarten Through Grade Twelve (2005). Complete Mathematics Framework ...
ISBN 0-8011-1474-8

Developed by the Curriculum Development and Supplemental
Materials Commission

Adopted by the California State Board of Education

Published by the California Department of Education

www.cde.ca.gov/ci/ma/cf/

I'd like to make a few quick comments, as to the seriously flawed definition of Mathematics in the opening sentence of the Framework, and then some philosophical comments (quoting David Corfield, Michael Polyani, George Polya, Terry Tao, Yau, Witten, Feynman, and others) on the relationship between Mathematics and Science, and then finally append one of the better papers (as an appendix, pp.11-28 below) which agrees with what I have observed about the difficulty of following the advice of the Framework with the urban Spanish-speaking students similar to those whom I have encountered in the Pasadena Unified
School District's middle schools and high schools.

DEFINITION OF MATHEMATICS
========================

"Mathematics--using abstract symbols to describe, order, explain, and predict--has become essential to human existence."
[Framework, p.4]

I simply disagree with this definition. It is shocking to see such foundational confusion in the opening sentence.

I agree, about half the time, that Math is MERELY a tool "to describe, order, explain, and predict". The other half the time I partly believe the alternative philosophical position: Realism/Platonism (that Mathematical objects exist, but not in the same sense that stars and people exist), or that Math is a vast game/conspiracy/social construct for people great at Math to get money, sex, and power.

SUBJECTIVELY it sometimes seems that I glimpse Platonic reality, and "see" a 4-dimensional shape or the like, but that is merely a psychological quirk -- I think. As to social construct, one can say that a Triangle or a Divergent Series is real the qualified way that Washington State Law or a Black Queen in Chess is real, i.e. follows
certain rules, which people can break by cheating.

The deepest mystery is that Math has some use in solving physical problems, which reference to the evolution of our brains does not fully explain. Wigner famously inquired as to the "the unreasonable efficacy of mathematics in explaining the physical world."

Does Math -- as a tool -- "govern reality"? No, it does not. It is brainwashing by an academic elite to say that it does. My mentors, including Feynman, knew better, and rejected String Theory on that and other bases.

I often adhere to my mentor's dictum about Physics being about experimental fact, not algebraic slight of hand.

The greatest error in that dreadful definition is the conflating of Mathematical Truth (axiomatic truth) with Empirical Truth (Scientific truth).

Let me amplify on that by first quoting from David Corfield, a Philosopher of Mathematics with whom I correspond.
[truncated; goes on for 25 more pages]

I don't think you can trust the data on rigor at all. They gave full points for having an exit exam, period, without evaluating its rigor. In our state, you can pass the exit exam in math and still earn an Epic Fail (placement into middle-school algebra) when you try to place into college level math. And if that is the case, what does it matter whether there exist beautiful "education standards" that look great on paper?

The real problem with the tests is what #8 mentioned: the emphasis is on a PARTICULAR kind of test, one with a specific design that is well known to the teachers in that state. Teaching to that test, including test-taking tactics, will not help with a more general test with a different design, or any other test for that matter.