The New SAT, Reading, Gaming, and Jargon

An achievement test's goal is to exclude. A social activist's goal is to include. Given one million grams of diamond ore, one gram therein is product. How much exclusion is equitable?

End arithmetic; ask Siri. Do patois Siri done speaks?

Is herbivory really a standard word in that field? It seems too cute to my ears, I guess. Like describing something as "engineery" or "physicsy". I'm not really that knowledgeable of the field, though.

I've never really considered the idea of vocab as a proxy for being well-read, and the games played by ETS and the test prep folks. That's an interesting perspective. I remember the GRE more clearly than the SAT, and the thing that really bugged me on that was the disparity between the level of the math section and the level of the vocab. To get into engineering grad school, I have to demonstrate a mastery of algebra and geometry (stuff I learned in high school) and a mastery of vocabulary like mulct (and I still don't know what that means). I know there's a GRE subject test in math that goes deeper, but UF didn't require it and I didn't take it. (Not sure if it was even around in 2002, although I know the SAT subject tests were back in 1998.) It makes more sense from the perspective of being well read, and also being able to differentiate between the people at the end of the bell curve, but I still don't understand why the math test was so low level.

We capitulated to the "SAT-prep vision of the world" the very moment that it became expected of a student to spend time, effort and ... money, explicitly preparing to take the SAT in the first place. What you remark about your own experience, one which I relate to greatly, is that the bulk of your ability did not come from specific preparation, it came from your own intellectual pursuits. You might say it came naturally to you, even if there were specific activities which lent themselves to having prepared you to take the SAT throughout your entire youth.

Or, as the bulk of mankind refers to it, you were a smart kid.

So neither would it be a surprise that your kids should also be smart, whether by nature or nurture. Accordingly also not a surprise that your kids will know the meaning of "disturbance", "spectrometer", become fluent in the metric system and adapt more readily to the notion that light at a wavelength of 540nm is green.

What also does not surprise me, is that considering all of this, you appear to be out of touch with what level an (average / normal / random) 11 year old would be able to communicate at. The reason I am not surprised is not anything relating to your strengths, which are what keep me reading your blog religiously, but rather that this is just exemplar of the raison d'etre of the Alan Alda Communication Center in the first place.

Science does a generally poor job of communicating science to non-science people. That said, while I believe this to be true, it is also true that the other end of this un-bridged gap does not reach as far across as we would like - and this is precisely how I feel it all relates to the notion of changing SAT format.

But as far as that is concerned, what does it matter anyway? We only relate these scores to other students, and even then in the overall context of other information like grades, extra-curricular activities, etc. I suppose my point is that the real flaw here has nothing to do with the test and everything to do with some combination of our K -12 education system and our cultural values.

Test prep becomes nothing but a veil between the "smart kids" and the kids with the resources and motivation to prepare them as meticulously as possible. I took the SATs a year before you did. I did well, though not as well as you. My best friend at the time was coached and tutored and otherwise prodded by his well-to-do parents, and still did worse on his 4th attempt than I did on my only, un-prepared attempt. That he is now an MD-PhD, and I am only now finishing a B.S. should say all there really is to know about the worth of any standardized testing at that age.

This whole business of SAT etc. seems to miss the whole point of identifying if a potential student has remembered (or perhaps knows) the background information needed to continue on to college/university. I never had to take this, (I graduated from high school in 1947, went to local city and State colleges, paid my own way, gt a few scholarships, and still got my degrees and a good professional job) and the whole idea that people are making money out of figuring out how to ace these tests rather appalls me, among other things. Just for fun, I took an SAT-type test in the 80's, and found the math part really hard (I received only about 550 for a total score - embarrassing), and don't even remember any of the `lit' parts, that seem to disturb people these days. This trend of SAT etc. as a guide to a student's ability is basically pointless and demeaning to teachers, and just seems like another money-making scheme. Knowledge as a commodity? .

Great point.

The thing I loved about the original Up-Goer Five comic was how it seems superficially simple but really expects a lot of background knowledge. If you don't know what oxygen is, you probably also don't know that air has "parts" or that it becomes "wet" and takes up less space when you get it cold enough.

Tom@2 - That's because "engineering" and "physics" are already nouns. You don't need to create “engineery” or “physicsy” to get a noun referring to an entire field of things. In contrast, what single word would you use to refer to the entire set of things which are herbivorous? ("Herbivorousness" would be the property, not the set of things with that property.)

I'm guessing your impressions of cuteness are based on the predilection of baby talk to add -y to the end of words, e.g. "are you comfy-womfy in your jammy-wammies?", and the (related?) use of -y to make imprecise adjectives (sugary, splintery, fuzzy, etc.) That's distinct from the precedented usage of ending class words with -ry, though: Archery, forestry, chemistry, carpentry, gadgetry, pottery, etc.

rehana@5 - True. "In defense of jargon" arguments typically make rather similar points. Jargon is often impenetrably dense to people not in the know, but rather precise to those who are. You can try to replace it with "simpler" language, but you either end up with something that is vague and confusing, or something that's accurate but so long and tortuous that even experts get lost halfway through. If you're going to discuss something like oxygen in any depth, there's no substitute for having prior experience with the topic enough to know what all is implied when one says "oxygen".

Thanks to Obi-One in Star Wars, I suspect all 10 year olds already know disturbance (in the force).

...while I did very well on the Verbal portion of the SAT back in the day (780, and this was in 1988 before they started “recentering” things), I never spent time memorizing anything in preparation for the test.

Neither did I (and I scored in the low to mid 700s, a few years earlier than you), but there were companies offering SAT prep services in my day (at least in well-off suburban areas like where I lived during my high school years), and I remember at least one of the mailed ads claiming substantial score improvements attached to names of students I recognized as being in the class a year ahead of me.

I agree with William @3: once it became clear that, ETS propaganda notwithstanding, a student could improve his SAT score by using these test prep services, that became the preferred route for any student in a family with the resources to pay for it. IIRC ETS was claiming that the best preparation method for the verbal SATs was the method you and I used: read extensively, so that you would learn to decipher unfamiliar words in context. An ability to recognize Latin and Greek roots was also helpful, so they recommended taking a foreign language (preferably Latin, but Spanish or French would do--I took Spanish, which also had obvious practical value where I was living). But once it became clear that rote memorization of words produced the same effect with (for most students) less effort, it was game over.

As for SteelyKid, I also agree with William that her situation is not typical. She is at least average, and probably well above average, in intelligence, and she lives with two parents who have high expectations for her ability to learn words like "disturbance". Lots of kids live with parents who don't have such expectations.

For the record, from other context, I suspect "herbivory" was not an adjective, but a reference to the action of herbivorous behavior. Not "My cat eats grass, so she's kind of herbivore-y" but "Ed Begley Jr. is well known for his herbivory." That kind of thing.

I'm also not attempting to contest the claims that I used words that would be unfamiliar (though I'm inclined toward the "disturbance in the Force" position of Omega Centauri advocates); the oldest kid I have regular experience of is five-and-a-half, and she's pretty awesome (though not an off-the-charts above-grade-level reader, based on the report card that came home yesterday). I was genuinely surprised by those comments, though, for the reasons stated.

I also have no problem with the claim that test-prep courses and the like are engines of inequality. For that reason alone, it's good to shake up the test format every now and then, just to give smart kids with fewer resources a fighting chance. But short of doing an individual in-depth assessment of each kid as an individual, there's not a whole lot that can be done to prevent wealthy families from using the resources they have available to give their kids every possible advantage. I think we can and should do more to provide better resources to a wider range of families, though, and I should've commended the small steps ETS is taking in that direction, also announced yesterday.

I thought about the Star Wars reference before I commented about "disturbance". I had even seen it by that age. With that said, I still think that many might not fully grok the meaning of the word. I realize you could often figure it out from context, but if you're trying to explain a new concept I'd warn against it--they'll have less context for the word and they may need to break to think about the meaning of the word, which could cause them to lose focus from the actual lesson at hand.

They'd definitely have zero context for nanometer, though.

I read a lot with kids, mostly elementary and preschool level, and never have I had a problem with words being too hard. Often I've had problems with the simple text making the book boring.

This is both for books I'm reading aloud/share-reading with the kid, and for books the kids read independently and then talk about later. So I always recommend using the word that fits, even if it is a "big" or "hard" word, and trusting the readers to figure it out. Nothing gets my hackles up more than a review that warns that the text might be too advanced for kids because of a few unusual words, because it completely goes against my experience. But I don't know any actual data about this.

The kids I know range from smart (in the schools Gifted program) to learning-disabled (in the school's resource program). The oldest is now 15, so my experience goes back about 13 years. I still hang out with KG age kids too.

But short of doing an individual in-depth assessment of each kid as an individual, there’s not a whole lot that can be done to prevent wealthy families from using the resources they have available to give their kids every possible advantage.

If tomorrow we replaced the college admissions process with a detailed background investigation, during which time a kid's portfolio of work was analyzed, the academic resources that they had access to over their lifetime was inspected in order to calibrate for advantage or disadvantage, and extensive interviews were conducted to gauge intellectual curiosity, and the whole thing was done at no expense to the student, the rich would still win. They'd hire coaches to help the kid prepare for the interview, and also to ensure that their school essays and projects did not just get them an A but also demonstrated originality on whatever rubric the college interviewers will use.

Inequality can be addressed on the margins, but in the end it is a timeless fact of human experience. I say that not as an apologist, but rather as an skeptic. So much of what is ostensibly done for the purpose of combating inequality in education is really done to make the upper-middle class feel that at least they tried. I know that my own work with disadvantaged students is not a systemic fix and thus is really of the "at least I tried nature." But at least I do know it, so I don't fool myself. And by just doing what I can instead of making a show of trying to revamp society, I probably waste less money in the process.

Mencken had it right:

“No one ever went broke underestimating the intelligence of the American public.”

http://www.thisdayinquotes.com/2011/09/no-one-ever-went-broke-underesti…

How well the S.A.T. correlates with future academic performance is only important when it's the only criterion for making that prediction. Which, obviously, it isn't, ever. Rather, its value has historically come from how much it improves prediction of academic success when added to an existing evaluation process.

College admissions already consider a student's demonstrated academic achievement. If the SAT is made to correlate better with future academic success by making it correlate better with past academic success, its value will not have been improved. From the College Board's press release touting how the changes better align with what students have been taught, this seems to be exactly what is happening.

Inequality can be addressed on the margins, but in the end it is a timeless fact of human experience. I say that not as an apologist, but rather as an skeptic. So much of what is ostensibly done for the purpose of combating inequality in education is really done to make the upper-middle class feel that at least they tried.

I don't disagree with this, but I think there are things you can do to minimize the effects of coaching or other purchased aid. Oral exams are an effective assessment tool for a reason, after all-- when you can see how somebody approaches a new problem in real time, that gives you a lot more information, and it's much harder to fake it-- not impossible, but much harder, so that it would be harder to get by just on prep classes. The problem is, they're also really resource-intensive, and the gain in fairness might not be worth the cost.

If the SAT is made to correlate better with future academic success by making it correlate better with past academic success, its value will not have been improved. From the College Board’s press release touting how the changes better align with what students have been taught, this seems to be exactly what is happening.

I think the main value of the tests even now is not so much the scores themselves, but as a consistency check on high-school grades. That is, if somebody has all A's on their school work, but an 820 on the SAT, that tells you that either something is fishy about those grades, or the school is just really terrible. Likewise, a student who runs a C average but has 1400 SAT's is either the beneficiary of some amazing test-prep classes, or at a truly amazing school. In either of those cases, you know you need to look a little more closely at that student when making an admissions decision.

There's also a sense in which it can help students from districts with fewer resources. If a student from a poor rural area where they can't afford tons of AP classes and test-prep programs gets a really high SAT in addition to running an A average, that tells you that they're really very good. Provided, of course, that the distortion from test-prep gaming of the test is not too great. Which is why (as someone who came from a rural district in which I was the only student in two of the three AP courses I took) I think it's worthwhile to shake up the tests now and again to stay ahead of the test prep industry to the greatest degree possible.

I think the main value of the tests even now is not so much the scores themselves, but as a consistency check on high-school grades. That is, if somebody has all A’s on their school work, but an 820 on the SAT, that tells you that either something is fishy about those grades, or the school is just really terrible. Likewise, a student who runs a C average but has 1400 SAT’s is either the beneficiary of some amazing test-prep classes, or at a truly amazing school.

In each case there's a third possibility that will inevitably be pointed out with great enthusiasm, since this is the internet.

For the anomalously low score, the person might be going through a truly awful personal crisis that made it impossible for them to concentrate on the day of the test, or have some sort of severe test anxiety. Of course, the solution is to look more closely at the application, but justice demands that we point this out.

For the anomalously high score, the person could be in the "smart but lazy" category. Usually they say something like "I'm just too creative for this system, man. I don't do assignments, you know, in the conventional way." I say that if they don't do academic work in the, you know, conventional way, man, then they should not do the conventional upper-middle class thing and be admitted to college. They should go out and do something, like, unconventional, man. This will make them happier and also make some college professors happier.

Memory is hugely important for intelligence. Obscure words are just a vehicle and diligence is also being tested.

The problem is, [oral exams are] also really resource-intensive, and the gain in fairness might not be worth the cost.

It's not just the resource cost, either. Expect your legal department to have kittens if you seriously advance an oral exam requirement, especially if your school isn't already doing field interviews (and even if they are, it's a much higher demand on your field interviewers). They'll probably insist on background checks for everybody doing the oral exams. At least, this is what I have heard, unofficially, from the office that runs my undergraduate alma mater's applicant interview program.

Nor will it stop the super-rich from trying to game the oral exams the way they already try to game interviews (you had a rant on that subject a few years back). The "educational consultants" would have to work a bit harder for their paycheck, but they could give their clients practice drills, so that they'll be less susceptible to panicking in the actual oral exam. And unlike under the current system, they actually would be offering a tangible service.

Alex - Smart kids who are lazy in high school, where they are highly regimented and/or bullied and are expected to do what seems like arbitrary makework, may blossom in higher education when they are in the classroom voluntarily and are, to a much greater extent, choosing what they wish to study.

I'm horrified to hear that an editor of - presumably - a biology paper thought "herbivory" was too big a word for his audience. Good grief. But there is a horrific trend of trying to avoid having students encounter any new vocabulary. I have a friend in a humanities field who helped to edit a textbook in the late 1990s. The publisher asked that they remove the term "foil" - in the dramatic or literature sense, of A serving as a "foil for" B, because they thought *college* students would not be familiar with it. Well, now's their chance to learn it, right? Nope. And what we will wind up with is Newspeak. You can't think thoughts for which you haven't been allowed to learn the vocabulary.

And, for all the biases of standardized tests, the biases of face-to-face interactions can be just as bad. One person's "Energetic and dynamic" is another person's "Pushy and demanding", often depending on gender and race.

I don't want multiple-choice tests to be the be-all and end-all, and I'm fine with periodically mixing it up to keep the gamers on their toes, but I'm not sure we should completely do away with them either, for certain purposes.

Well, one thing that is different from before is the importance these tests are- or at least portrayed- for those taking it. Basically all tests can be gamed as long as they have a pool of questions and it is worth doing so. It is unlikely that something like this will escape being gamed.
My suspect feeling of why the Writing section was removed is due rather to the increased load of essay readers who previously only had to deal with PSAT and SATII.

Aside- I feel that one way to emphasis context clues and reduce memorization is to flat out invent words. (ex: for herbivory, rather than like a herbivore- have the meaning be for those that do not pursue a goal (prey) and are fine taking what's available on the "ground" for them- which is kind of how it is used in Japanese slang).
However, that might possible cause trouble given people's dislike of abstraction and those who don't understand and use the fake words seriously.

Also, there's a paradox of sorts in this where:
A) If they make the test more correlated with future performance, it will be more valued as a criteria.
B) If it is more valued as a criteria, the higher incentive it is for people to game (or outright cheat) the test making it less correlated.

Often a metric's good until it is tied to a quota.

"Starting in 2015, the essay section will no longer be mandatory, ..."

No wonder! Essays are still not susceptible to a computer' program's machine-reading and interpretation and grading. Maybe when Johnny's computer can read and grade essays, they'll return to the test material.

Oh, well, that's progress in the fool's paradise that is the technopoly.

"the vocabulary section was never intended to be a memorization test, but a proxy test for wide reading. I say this mostly because while I did very well on the Verbal portion of the SAT back in the day (780, and this was in 1988 before they started “recentering” things), I never spent time memorizing anything in preparation for the test. I didn’t really need to, because I spent a lot of time reading. A lot."

Fortunately, kids still read a lot. Many have read every single book in the Harry Potter series---and so did a lot of young adults without kids, by the way.

The essay question wasn't added to the SAT until long after I took it (yah, I'm old) and I always wondered how they could possibly claim to objectively put a numerical score on so many essays - especially if it's not just "A" vs. "B" etc. but a score with two significant figures.

Back when we did the Blogger SAT Challenge, Dave found some volunteers who did actual SAT essay grading. I don't recall the exact details, but I believe each essay got read by two human graders, each of whom gave it a score on a scale from 0 to 6. The scores got added together for the final grade.

The time they spend reading each is pretty short-- less than 10 minutes per. One of those old links might have more detail, but I couldn't find it quickly.

"For the anomalously low score, the person might be going through a truly awful personal crisis that made it impossible for them to concentrate..."

The solution to some brief temporary crisis is to reschedule or retake the test, which I believe is already allowed. I remember that I had the option of taking the SAT again a few weeks later.

For a really big crisis... well any such circumstance which would make it impossible to do well on a test during a several month window will very likely also make it wise to simply delay school because the ongoing crisis would disrupt that as well. There is some flexibility on the course of higher education.

I was one of those 'very good test scores, B+ grades'kids in high school.. and really the same in college. So I guess it was predictive. My 'test prep' was to buy a little booklet with a couple sample tests and take them timed to get used to the format.

It's not hard to game grades and extracurricular activities either. There were some rather dim bulbs with great GPA's at my highschool. Hard workers taking easy classes. I expect they did the same in college.

I'm rather dubious that we can sort kids intellectual prospects better than +/-20 percentiles with any cheap procedure, so outcomes become very arbitrary if rating differences smaller than that have strong effects on opportunities, which I think they do.