The Testing Myth and NCLB

Another credulous article on the No Child Left Behind (NCLB) law appears today in the Washington Post. As someone who knows many teachers who have had experience with similar stupid laws in Virginia, and the history of the Bush administration pushing for these kinds of laws based on the "Texas Education Miracle", I'm far more skeptical about any real gains in learning as a result of standardized testing.

But first, you have to understand what Bush, and his education secretary Rod Paige, really did as governor of Texas for education (take a guess), and how standardized testing is a cynical political tool.

First of all, the "miracle" in Texas was that no investigative reporters bothered to actually figure out until Bush was already in the White House was that they improved drop out rates by fudging the stats.

All in all, 463 kids left Sharpstown High School that year, for a variety of reasons. The school reported zero dropouts, but dozens of the students did just that. School officials hid that fact by classifying, or coding, them as leaving for acceptable reasons: transferring to another school, or returning to their native country.

"That's how you get to zero dropouts. By assigning codes that say, 'Well, this student, you know, went to another school. He did this or that.' And basically, all 463 students disappeared. And the school reported zero dropouts for the year," says Kimball. "They were not counted as dropouts, so the school had an outstanding record."

Sharpstown High wasn't the only "outstanding" school. The Houston school district reported a citywide dropout rate of 1.5 percent. But educators and experts 60 Minutes checked with put Houston's true dropout rate somewhere between 25 and 50 percent.

And it's true, statewide to this day Texas reports that it has a dropout rate in the single digits. However, independent analyses, like those by the Manhattan Institute (a right-wing think tank) show that the actual drop out rates are closer to 40%!(1) One major way the Bush administrations fudged the stats was by changing the definition of the metric. They counted drop outs as those entering the 12th grade who failed to graduate, whereas the real metric of drop out rates is how many kids enter high school and fail to graduate. That combined with creative accounting of dropouts by the schools and it becomes obvious the "mirace" is a big lie.

WaPo has a similar article on the fudged stats. Only this is how the schools in Houston under Bush and Paige were able to fudge test scores:

They note that the Texas test is administered in the sophomore year. Austin High, like many other Houston schools, routinely holds students back in the ninth grade under a policy that effectively allows school administrators to exclude weaker students from the 10th-grade test results. In 2001, for example, there were 1,160 students in the ninth grade and 281 in the 10th grade.

Perla Arredondo, the daughter of Mexican immigrants, took ninth grade three times before being moved up to 11th grade. By then, she was so discouraged she dropped out of Austin High, along with many of her friends. She regrets her decision, after discovering she needs a high school diploma even for jobs such as secretary or cashier.

"I felt school was a waste of time because I had to go over the same thing over and over again and wasn't moving up," she said.

Because Arredondo skipped 10th grade, she was never included in Austin High's accountability statistics. According to Robert Kimball, a former Sharpstown High assistant principal who provided KHOU with much of its information, that is common practice in Houston. "The secret of doing well in the 10th-grade tests is not to let the problem kids get to the 10th grade," he said.

This is a far more general problem and something people don't get about "accountability" measures. When you base accountability on metrics - it doesn't matter what the field is - people will fudge the results to get ahead. If you try to hold police accountable based on crime statistics, they will charge people with more minor offenses to make it appear as if violent crime is down. If you try to hold doctors accountable based on metrics culled from their records they will fudge to make their patients look healthier (don't read the authors conclusions - they're nuts - look at the data). Careless use of measures of accountability are simply not effective means of ensuring you are getting what you want out of people, because people faced with a difficult student population, or criminal population, or patient population, will find it easier to fudge the stats than fix the problem. Additionally, when you don't actually give them any money to fix the problem, they have a dilemma, fudge the stats or lose your school and/or job. It's not usually outright dishonesty either, people have many implicit biases that really make it difficult not to try to manipulate systems to make themselves look better. In science, it's something you have to be constantly vigilant about, it's why trials of drugs are double-blinded, it's why we don't believe things until they're repeated multiple times, it's why you take steps to constantly try to eliminate bias from your experiments. It's not because people are fundamentally dishonest, but they are biased, often in very predictable ways, and bias can make people affect results without their even realizing it - or thinking they're doing anything wrong. I think that's what is going on here.

How did the schools fudge the data for NCLB? Well, it's an open secret that the state tests that ensure their schools continually get federal funding were initially challenging tests - but when all the kids failed they dumbed down the tests. Independent testing has consistently shown that the state tests are routinely failing to actually assess student performance. They're fudging the stats.

The worst aspect of all of this testing and accountability crap, is that it give the implementor a false appearance of accomplishment. It goes like this. The politician says the schools need accountability so they implement a standardized testing scheme. The first year, the students do terribly, the politician can claim proof of a problem (and immediately blame their predecessor). Then the next year, the schools under pressure to maintain funding, or in the case of NCLB, the states, dumb down the tests or spend the whole year teaching to it (as they did here in Virginia) and suddenly test scores are way up. The implementing politician can now claim success! In one year test scores are up 40% 50% etc., they must be a great leader! This is exactly what happened in Texas, this is exactly what happened here in Virginia, and now, it's happened nation-wide. Expect Bush to declare "mission accomplished" on education sometime soon, even though it's all just a scam.

This is the scam that helped propel Bush to the white house as an educational reformer. In reality it's just more Enron-based policy from this incompetent administration. It's about appearances, not real success. And the worse part of it, is the cynical use of the nation's children to make it appear as though politicians have reformed education, when all they've done is swept it under the rug.

1. Greene, J.P., High School Graduation Rates in the United States, November 2001(Revised April 2002), The Manhattan Institute for Policy Research.

Tags

More like this

In the midst of an article about Uncommon Schools, a non-profit charter school company, we discover this interesting effect of holding back students who fail a grade: High retention rates can help to boost test scores at charter schools, at least in the short term. Students may do better on tests…
It's a new pilot program in a few dozen New York City schools: students are given cash rewards in exchange for higher test scores. Jennifer Medina reports: The fourth graders squirmed in their seats, waiting for their prizes. In a few minutes, they would learn how much money they had earned for…
Via a EurekAlert release with the catchy headline "As graduation rates go down, school ratings go up", a new study of the Texas school system, which provided the inspiration for "No Child Left Behind". It's not pretty: A new study by researchers at Rice University and the University of Texas-Austin…
Before one worries about how teachers are performing, it would seem that getting children to attend school regularly would be an important first step, which is a reason one Floridian teacher opposed teacher student testing: When Florida proposed strict accountability measures, teachers, parents and…

This is a far more general problem and something people don't get about "accountability" measures. When you base accountability on metrics - it doesn't matter what the field is - people will fudge the results to get ahead.

Man, this is so true, and it's often a nightmare. "Accountability" is one of this bureaucratically misused words that really gets under my skin.

It's a generalization of teaching to the test.

-Rob

I agree with everything here, but it seems to leave one big question left unsaid:

How, exactly, can we measure the success (or lack therof) of any educational reform?

Clearly NCLB is junk, and needs to be done away with, but if a different path is chosen to improve education, how do we measure its resuts? Some form of independent analysis?

By General Woundwort (not verified) on 06 Jun 2007 #permalink

Testing of itself isn't the worst thing in the world. The problem is viewing testing as a solution to a problem. Testing will never fix a problem, it will only show you a problem (and then often only once - because the system will adjust to the test in these cases).

The secret is to have occasional testing with rigorous standards and auditing of their application. Not every year in every grade. Further, poor performance should result in more aid, rather than punishment. Schools aren't trying to fail kids, and not every school is blessed with the perfect upper-class motivated population of students. Tests should be used to identify trouble spots than can use increased aid and intervention.

Finally, we must be suspicious of rapid improvements in metrics, as they are almost always a sign of fudging.

My kid's High School Principal (Suburban Chicago area) was against the NCLB from the start for various reasons, including what was mentioned by markH & Rob. She finally said F**k you, we don't need your stinkin' Federal Money, so stick it up your George Bush. And they did.

What I hate about the SOLs in Virginia is that they force my daughter's teachers to concentrate most of their efforts on material that can be easily assessed via multiple choice tests that can be graded by scantron readers. Of course, the teachers then adapt their teaching methods to the material, so there is a heavy reliance on rote 'learning'.

I took a whole bunch of AP classes in high school, the classes where you take a national, standardized test at the end of the year and get a score for which some colleges will give you course credit. The "teaching to the test" phenomenon was impossible to avoid.

Of course, you'll say, the AP classes are full of all the smart kids, who'll probably do just fine whatever happens, and if they can get into colleges more easily where they'll see the real thing, so much the better.

And the NCLB came along and made it everybody's problem. I got a little smug satisfaction out of that.

I wrote extensively on the Texas Miracle fraud before the 2000 election:
http://www.skirsch.com/politics/election2000/ad.htm

I think I was the only person who looked into why the two RAND reports reached different conclusions. Everyone else just reported that they were different. I was the only one who explained 1) why they were different and that 2) Bush is distorting the facts to make things appear to be opposite that of what they really were.

In 2000, I spent millions trying to get the word out before the election, including full page ads in the New York Times.

And here's the summary of how he fooled everyone:
http://www.skirsch.com/politics/election2000/quick_version.htm

And here's the gory detail of how it was done:
http://www.skirsch.com/politics/election2000/myth.htm

Here's a summary of things I found most disturbing:
http://www.skirsch.com/politics/election2000/mad.htm

My grasp of school budgeting is probably very naive, but I've never understood why various entities attempted suing the federal government over NCLB and characterizing it as an unfunded mandate. Isn't the fed contribution to the state school budget typically around 7%? (Hello, seven percent solution imagery). Can't the states just say, thanks but no thanks to NCLB and 7% of their budgets?

First, a response to Mel. It is possible for the states to say no thanks, but 7% is often a huge amount of money, and most school systems are already struggling for cash (at least that is my belief I get from my mother who was on a school board). Giving up even just that much is unrealistic.

Second, this is off-topic of education, but a perfect example of metric fudging. I don't know how many of you have read this elsewhere, but the government has released numbers that show the death count in Baghdad as going down. This shows the surge must be working, right? Of course, as soon as people started to look more closely, it turned out that they had decided, right around the time the surge started, that counting people who had been killed by car bombs is exactly what the terrorists wanted, and thus stopped including them in the official count. Surprisingly, all of the sudden, the counts showed that less people were dying!

By CaptainBooshi (not verified) on 07 Jun 2007 #permalink