Casual Fridays: Most citizens don't pass the citizenship test

Last week we wondered how thorough news reporters were being when they conducted "person on the street" interviews with questions from the U.S. citizenship test. We decided to administer the test a bit more systematically (but still not scientifically). Over 680 people responded to our study, allowing us to get some pretty solid results.

The headline is what we stated above: Most U.S. citizens didn't get a passing grade on the test -- even though we were very generous in grading the tests. We didn't even count off for spelling errors and accepted answers that were only partially correct. But how did U.S.-born citizens compare to naturalized citizens, who themselves had to take the test at some point? This graph shows test results by citizenship status:


No group's average exceeded the passing grade of 80 percent. Interestingly, there was no significant difference in passing scores between naturalized citizens and US-born citizens. Also, non-citizens living in the US did just as well on the test as US citizens living abroad. As you might expect, non-citizen non-residents got the fewest correct answers on the test.

But how many people actually passed the test? This next graph shows the portion of each group receiving a passing grade:


Just 33 percent of US-born citizens passed, slightly fewer (but not significantly different) than naturalized citizens. This figure is actually biased on the high end -- our readers are more educated than average, and many, many people (714 out of 1,392) took a look at the test and decided not to complete it -- very likely because they didn't know the answers to the questions on the test.

So what factors did matter the most in passing the test? Perhaps younger people would score better because they took their high school civics and history classes more recently. This graph plots test score against age:


In fact, as the trendline shows, age was associated with higher scores on the test. There was a significant positive correlation between age and score (r=.13). But perhaps the age effect was really just an education effect. Indeed, there was a positive correlation (r=.13) between years of education and test score. Even more important, however, was the number of years of education in the U.S.:


Here you can clearly see the very strong correlation (r=.44) between U.S. education years and test score. I find this result especially interesting when you consider that the vast majority of our readers have at least a bachelor's degree, meaning that the effect is partially driven by postgraduate education level. Even highly specialized graduate education appears to be associated with higher scores on citizenship tests covering material that most of us learned in high school (if we went to high school in America, that is).

There's one more calculation we can make with this data, albeit a speculative one: what if those 714 people who took a look at the survey had actually tried to complete the test? In all likelihood the overall passing rate would decrease, but by how much? We can't know how well these people would have done, but we can chart what the expected overall success rate would be depending on our assumptions about the people who decided to quit:


The graph shows the portion of US born test-takers would have passed given several different assumed success levels for the quitters. If they would have passed at half the rate of actual participants, for example, this would result in an overall success rate of just 24 percent. Assuming all quitters would have failed the test, that computes to just a 16 percent pass-rate of the U.S. citizenship test among US-born citizens. Yikes!

So what questions were the hardest for our test-takers? Here's a graph of accuracy on all the test questions:


Finally, here's a link to all the questions and answers.

More like this

Last week I created a survey that was truly humbling. The goal was to find out if time and money invested in preparing for the SAT and other standardized college admission tests is worth it. The first thing I learned from the study was that Cognitive Daily readers are incredibly smart -- much…
Imagine that you're taking a test in a large public hall. Obviously, your knowledge and confidence will determine your score, but could the number of people around you have an influence too? According to psychologists Stephen Garcia from the University of Michigan and Avishalom Tor from the…
Some people—even people who really know their stuff—just don't "test well." You can talk to them face to face, and they seem perfectly well informed and intelligent, but when the money's on the line, when they've sharpened their number 2 pencils and it's time to sit down for the big exam, they just…
Yesterday we discussed several experiments offering converging evidence that exposure to the color red, even for brief periods before taking a test, can result in lower achievement. It's startling research, but as my daughter suggested at breakfast this morning, maybe people are just intimidated by…

I am so glad I got my citizenship before they made the new test. I got 8 out of then when it mattered. I would have flunked it today. And I consider myself well versed in US civics.

I thought the "name a territory" question was ambiguous. I wasn't sure if it wanted a current or historical territory (would Orleans, Franklin, or Northwest have been correct answers?). Only one of those five current US dependencies is technically a Territory (Guam). Puerto Rico and the Northern Marianas are Commonwealths and the Virgin Islands and Samoa are unorganized territory. I suppose that's an example of too much knowledge being a bad thing. In any case, I got all ten right (but only because spelling doesn't count).

are these questions randomly taken from test as a whole, or are these the 10 hardest questions pulled by the news media to be sensationalist? just curious.

I just reviewed the complete list, and it seems to me that the questions about the number of amendments and the number of U.S. Representatives in particular are probably the hardest (not to say stupidest) ones there, which is quite enough to make this a harder than average selection overall. These are simply not facts that are worth knowing with any sort of precision, or that anyone (barring, perhaps, a member of the House trying to put together a majority for some vote) has any use for whatsoever, outside the arbitrary demands of this sort of test. They are both completely arbitrary and meaningless numbers that can only be known by pure rote memorization. In neither case is knowing them in the least indicative of any understanding of the nature, workings, history, or ideals of American government or society. I would never bother to memorize such facts unless I was about to take the citizenship test. (Of course, tests given in schools also often demand useless, pointless information of this sort, which is both a symptom and a contributory cause of the failings of the American education system.)

I think I had 8/10 correct, but I can't remember if I said 26 or 27 for the first answer. I know I got #3 wrong, and agree that numerical questions of this sort are not good predictors of good citizenship.

"These are simply not facts that are worth knowing with any sort of precision"

Reminds me of the Illinois Driver's license written. Lots of questions like "What is the penalty for a second DUI offense?". Isn't the point not to drink excessively and then drive? A better question on the above test would be "How many Senators does the US have?" At least that doesn't shift every ten years.

Pssst, just FYI then number of representatives has been fixed since 1911 IIRC.

Non-US citizens may have been more likely to look at the test but not take it of course.

I think it's pretty important to know the number of representatives. That way you have a good idea of the power of a single representative. Plus, it's important in the election of presidents -- the number of presidential electors is the 435 representatives plus 100 senators.

I agree that the number of amendments to the constitution is a little more like trivia. A lot of people were off by just one on the amendment question -- the percent correct would have increased from 19 to 36 percent if we'd accepted answers that were off by one.

But increasing the range for the representatives question didn't get as dramatic a result. Accepting answers ranging from 425 to 445 only increased correct responses from 29 percent to 36 percent.

the number of presidential electors is the 435 representatives plus 100 senators

Not quite: the District of Columbia suffers from taxation without representation (ie, has no voting congressmen), but gets a number of electors for President equal to the number of electors granted to the state with the fewest electors. In other words, DC gets three electors. So there are 435 congressmen, 100 senators, and 538 electors. 50% plus 1 vote of 538 is 270.

I agree with Dave. It's important to know the number of members of the House and Senate. Both because the two numbers don't change, and because they are relevant to the respective power of their members and to the electing of the President. I'm less certain of the importance of knowing the number of amendments: their content and historical context is certainly important, but the exact figure of 27 doesn't seem that helpful. An order of magnitude answer is useful: there are 27, rather than a couple hundred.

I didn't look at the test, but perhaps the reason for the question having to do with the House, rather than the Senate has to do with the fact that as the President of the Senate, the Vice President of the US votes, but only when there is a tie to be broken. One could argue correctly that the VP is a voting member of the Senate (101 voting members); others might argue that it only occurs in a rare instance, otherwise the VP does not vote (100 voting members).

I don't know whether the 10 questions were generally representative of the 100, but my score on the short test was 20 percentage points below my score on the long test. And my score on the long test probably would have been higher if I could have applied good test taking strategies: filled in the ones I knew immediately and come back to the rest. That was impossible when I saw the answer as soon as I scrolled down. Come to think of it, a camera and microphone don't make the best test environment either.

It is nice to know I'd have passed the whole test.

By Stephanie Z (not verified) on 07 Oct 2007 #permalink

I said it was not worth knowing these numbers with any sort of precision. The fact that less than a score of amendments have been passed since the Bill of Rights helps to give some sense of how difficult it is to alter the constitution in this way (or how reluctant Americans have been to do it), but knowing that it has been done 17 times rather than, say, 15 or 19, really does not add anything to one's understanding.

Likewise, it may be worth knowing that there are several times as many Representatives as Senators, and perhaps even that there are round about four times as many. Arguably this gives one a very rough sense of the relative power of a Representative as compared to a Senator. However, that sense is very rough indeed, because all sorts of other factors impact the power of a Representative (or a Senator, come to that), ranging from constitutional differences between the powers and responsibilities of the House and Senate to individual differences in the personalities and political circumstances of individual members, and their position within various formal and informal political groupings. Knowing there are 435 Representatives rather than 445, or even 300 or 500, is not going to help anyone understand these real and significant, but unquantifiable, political facts. Knowing the precise number, for anyone except, occasionally, Representatives themselves, is quite worthless for either citizens or non-citizens.


By aboveaverage a… (not verified) on 24 Jun 2009 #permalink