Harry Brighouse at Crooked Timber has a very good post about schools that appear to "beat the odds", getting good results with populations that don't typically do well in school. It does an excellent job of laying out the problems with the vast majority of attempts to determine which schools are "beating the odds," let alone what methods are best to use for this. It turns out to be a lot harder to measure than most people think-- I was particularly struck by this bit:
It gets worse, thanks to my colleague Doug Harris, in his paper, “High flying schools, student disadvantage, and the logic of NCLB”. Using the School-Level Achievement Database and the Education Trust’s definition of high-flying schools (high-performance in either reading or math in the grade and year selected by ET for analysis), Harris has estimated how many schools remain high flying over time, and what the characteristics of those schools are. Using a sample of 18,365 schools he finds that when the definition of high performing is changed to require consistency over time fully 93% of schools identified as high-performing for a year drop out of the category. And whereas low poverty schools are only 3 times more likely to be high performing than high poverty schools on the single-year definition, they are 22 times more likely to be high-performing on the definition that requires consistency over time. Using data on schools with high minority populations he finds that, on the more demanding definition, the “likelihood that a low-poverty-low-minority school is high-performing is 89 times greater than for a high-poverty-high-minority school”.
This highlights one of the things that make education so damnably difficult to measure: there is a lot of year-to-year variation in the population you're dealing with. The students we're teaching this year are not the same as the students we were teaching last year, or five years ago. It's not a system well suited to yearly measurements, but longer-term studies are subject to larger-scale drifts in the populations and attitudes of the students being served. On the short term, you're mostly seeing statistical fluctuations, and on a longer term, you've got to worry about demographic shifts.
It's kind of depressing in a lot of ways. What it really shows is that the problems of education are inextricably coupled with the problems of society in general, and that the odds of a quick and simple fix are very, very low.
- Log in to post comments
As an old teacher, I get disgusted by the "fix-it" mentality of politicians. We have to "close the achievement gap" without dealing with the money gap , literacy gap, parent gap, food gap, etc. It is a tribute to many kids that they achieve at all in spite of the long odds.
Not to be too simple about it, but a big part of achievement comes from beliefs of the student. They need to believe:
1. You (the teacher) know what the hell you are talking about.
2. You (the teacher) are interested in them and want them to achieve.
3. You (the teacher) are convinced that they (the students) can do it!
Kids have an infallible BS detector. I learned a long time ago, if I don't know something, I say "I don't know."
So why do education studies always compare this year's third graders to last year's third graders? If you are trying to track how successful a school is in teaching the children, wouldn't you want to compare this year's third graders to last year's second graders? If the school is doing a good job, you would expect this year's third graders to be doing better on third-grade tests than the same students, last year's second graders, did on the second-grade tests.
It's like looking for climate trends by comparing the temperature in New York last year to the temperature in California this year. How does anyone think that comparing different cohorts is at all meaningful? I have just never understood this.
It's always bugged me, too, that education studies don't tend to be cohort studies, that they compare this year's third graders to last year's third graders. Even in a school with a fairly stable population, there can be quite a variation from one cohort to the next.
An additional problem with high poverty schools is that the population there tends to be much more transient than in a low poverty school. The kids you had in third grade are somewhere else entirely in fourth grade, because rents went up, or a local source of jobs closed, or whatever.
The more I'm around schools, the more I think that "closing the achievement gap" is absolutely meaningless. I don't even know where to begin to address the differences between my daughter's fairly affluent suburban school, where the problem I'm fighting is unreasonable homework loads and an expectation that teenagers should perform better than high-functioning adults at juggling AP classes and extracurricular activities, and a high poverty school. I've been in classrooms in both, and I'm really don't have a clue as to how to "fix" education to make the two perform similarly. I'm pretty sure, though, that standardized testing and national standards don't do it.
Scott @2 nailed it, except it is actually worse than he said. The size of the cohorts (and thus the size of the sqrt of N) isn't even the same, sometimes varying wildly according to a HS friend who has taken up teaching as a second career.
It could be the fluctuating year-to-year success of the low-income schools results from year-to-year variability of a given class of students.
Also, low-income schools have much more migration in and out during the year as families move in and out of different districts. There is no guarantee that this year's 3rd grade was in last year's 2nd grade even if they all passed 2nd grade.
The solution is what is called "longitudinal" tracking, but this is not common and then only within a state or district.
BTW, everything written @1 is true but none of it is new. What bugs me is that the curriculum in low-income schools has not been developed to account for the lack of books and other reading material in the home, the issues raised decades ago in "Cultural Literacy". Correction: what bugs me the most is that math skills keep getting worse and math is the most culturally neutral thing you can teach and the most objective to measure, yet one local school was so bad that math scores went DOWN as one cohort moved through middle school.
One of the reasons I'm increasingly disenchanted with the whole "close the achievement gap" focus is that the assumption is that what is best for white middle class students is best for everybody. So, since white middle class students go to college, we should set the standards and the requirements to graduate so that all high school students are "ready" for college when they graduate, as if by just declaring it it will be so. In other words, we set kids up for failure, then blame the school systems when the inevitable happens.
I certainly don't mean to imply that you must be white and middle class to go to college, and I'm all in favor of programs that help kids who aren't that get to college, but in the real world, even in the best schools in the world, not everybody belongs in college, and pretending that we're going to prepare them for it is a lie.