Cognitive Daily

Last week’s Casual Fridays study was inspired partially by reports we see in the mainstream media so often, proclaiming that Americans are ignorant about the rest of the world. If the rest of the world really is better than Americans at geography, then maybe they can beat us even on our home turf. Our quiz compared non-Americans’ knowledge of U.S. states to Americans’ knowledge about the rest of the world (you can see the maps and answers here).

I randomly chose ten countries and U.S. states, then created a multiple-choice quiz where a map was displayed and five plausible choices were given. (For example, the distractors for the map of Ukraine were nearby Poland, Romania, and Belarus, and similar-shaped Kazakhstan.) Non-Americans were given the U.S. State quiz, while Americans took the Countries of the World quiz. Both groups had the option of completing the other quiz when they finished. So, did non-Americans demonstrate vastly superior ability?

Ummm… not exactly. In our test of 500 individuals, Americans performed significantly better on the Nations quiz than non-Americans did on the U.S. State quiz.

But as many commenters pointed out, this isn’t really a fair test. Why should Europeans learn the internal divisions of the U.S.? A better comparison would be a direct one — how do Americans compare to the rest of the world when it comes to knowledge of national boundaries? Considering only the people who chose to take both tests, here are the results:

It does appear that non-Americans have a slight advantage, and given our large sample size, that advantage actually verges on (but does not attain) significance. Does this mean if we tested a larger sample, we’d find a significant advantage for non-Americans? I don’t think so: I believe I have found the source of that slight advantage. Take a look at these results for the single quiz question on Denmark, the only Western European country on the test:

As you can see, Europeans were nearly perfect on this question, significantly better than Americans and others. But there was no difference in scores on any other question, even the one on the European country of Ukraine. In fact, the European advantage on the single Denmark question explains the entire difference between non-American and American scores on the Countries of the World quiz. So while Europeans may enjoy an advantage identifying (some) European countries, that advantage disappears in the rest of the world.

Clearly, there are some problems with this study. First, non-Americans may have learned the country names in their native language, so they might have been at a disadvantage taking the quiz in English. Second, it’s casual. Participants weren’t randomly selected, but instead are chosen from the biased group of individuals who happen to read Cognitive Daily. In our defense, our readers probably come from similar backgrounds, so if there were underlying differences in the educational systems from nation to nation, our study should have uncovered it. But still, there’s a clear selection bias in our results: our test was only taken by people who chose to take it.

This all brings up an interesting point. Can these results reveal any differences in how people choose to take online quizzes? One thing I was wondering about is whether people who took the test early were better than those who took it later in the testing period. Take a look at this chart of the average scores for the entire test:

The first 100 participants scored significantly higher than the last 100! Could it be that people who choose to participate in online quizzes early on are those who feel they are likely to perform better?

It turns out, probably not. I analyzed the composition of those groups and found that there were 72 Americans in the first 100 participants, but only 58 in the last 100. This is probably due to the time difference — at the end of the sampling period it was Friday night in America, but Saturday morning in Europe, and daytime in other parts of the world. Since Americans did so much better on the State quiz, this pulled the early scores up in comparison. As you can see by this chart which shows only the results from the Country quiz, where everyone did about the same, there are no significant differences in scores, regardless of when the test was taken:

Nonetheless, I think you’ll agree these are interesting results, suggesting that educated, thoughtful people like CogDaily readers have a similar level of geography knowledge, no matter where they’re from.

Comments

#1 Scott Belyea
January 26, 2007

I’ve enjoyed and participated in a number of these Friday studies, but I found this one to be by far the worst – poorly thought out, misleading, and pointless.

Example – contrast the heading of this post (“Are Americans geo-ignorant?”) with “…suggesting that educated, thoughtful people like CogDaily readers have a similar level of geography knowledge, no matter where they’re from.”

I don’t think that the “stereotype” of the geo-ignorant American ever had much of anything to do with “educated, thoughtful people.”

A blunt comment, I know … but this one was not worthy of your more usual standard.
#2 Lantern Bearer
January 26, 2007

I am happy that I was able to hold up my end of this exercise in both the world and US sections. My advantage is that I was nearly six years in a military intelligence service and I have had the pleasure of traveling to remote and relatively unknown places on someone else’s nickel.

A couple of years ago, I had the pleasure of conversing with a young bellman at an Orlando resort. I never ask a direct, “Where were you born? question, but rather, “Where was your grandfather born?” My bellman replied, “My grandfather was born in Eritrea and so was I.” To which I was able to reply, “Ah, yes. The break away northern province that was once a part of Ethiopia. The capital is Asmara.”

His jaw dropped. We had several pleasant conversations over five or so days about his escape from there by donkey and camel thru Sudan to Egypt and then to America. He was 6 years of age. His family was originally Yemeni. In his long family history, they had been animist, Jews, Christians and Muslims. This fellow grew up in Boston and was a budding transcendentalist and Sufist. He was supremely geographically literate because he was an IT fall-away and had taken up course work in hospitality management. All his future employers in that field require a depth of geographical knowledge where they operate anywhere on the earth ball.

At the moment I am working my way thru classical Greek in the ancient tongue and plotting my way around the trails and byways of the ancient cities. The groves, grottoes, and temples of the gods come with their unique geographical and geologic locales. The islands are intriguing as well. I have a project on the back burnner of quartering up the moon and becoming familiar with it more intimately. My family name is attached to a crater there.

My secret weapon? No TV and no pop fiction.

Lantern Bearer
#3 Dave Munger
January 26, 2007

Scott,

I think what you’re saying is that because only educated people are likely to read Cognitive Daily, it’s pointless for us to use our readership to try to compare American knowledge of geography to the rest of the world.

The point is well taken, but I’d submit that we didn’t know that before we started. Perhaps the stereotype of American provinciality has a basis in the American educational system, at all levels.
#4 John McKay
January 26, 2007

I think that the results of this test will be skewed from a that of a broader ample of society because, as you said, only firly well-educated people are likely to read Cognitive Daily. I think there might also be another self-selection bias in that many people are more likely to volunteer for a quiz that they think they will do well on than on that they think they will do badly on. Like Lantern Bearer, I believe I’m very strong in political geography (and I got 100% on both quizes). I would be far less inclined to volunteer to take a general knowledge quiz on sports or contemporary music.
#5 John McKay
January 26, 2007

I also wouldn’t volunteer for a test on spelling, typing, or proof reading.
#6 Scott Belyea
January 26, 2007

…pointless for us to use our readership to try to compare American knowledge of geography to the rest of the world.

That’s essentially it, Dave. To turn the point around, it seems to me fairly obvious that most of the population who would give rise to the “ignorant American” syndrome weren’t included to begin with.

A secondary point – as a non-American (Canadian), the equation of US states to other countries seemed to me to be another example of the sort of US-centric approach that is all too common. I’m sure that nothing like that was intended … but that’s really the point. It’s all too often a “built in” sort of attitude in my experience. Example – I cannot imagine a similar quiz done by a Candadian or a Brit comparing knowledge of provinces or counties to other countries.

As I said, this was not up to your standard in my opinion.
#7 ocmpoma
January 26, 2007

Having seen several ‘it’s unfair to use states vs. countries’ comments, I think I’ll weigh in.

The reason that US states were chosen was to see Americans who took the quiz were so poor at geography relative to non-Americans that the non-Americans would actually know more US states than Americans know other countries. It was the whole point: “Americans are so ignorant, they can’t even identify countries as well as foreigners can identify US states!”

It was not an attempt to give the Americans an edge, it was an attempt to see how ignorant Americans actually are; furthermore, I think that if the test was administered to a group that is not so self-selected for education and interest as is the ScienceBlog readership, the results would be quite different.
#8 Dave Munger
January 26, 2007

The equation of US states to other countries seemed to me to be another example of the sort of US-centric approach that is all too common

I see where you’re coming from there, Scott, and you’re right, that certainly wasn’t what’s intended. Maybe we should have compared U.S. knowledge of Canadian provinces to Canadian knowledge of U.S. states — but this probably would have excluded too much of our audience.

We’ll try to get back to something closer to Cognitive Science with next week’s study.
#9 Barry Kelly
January 26, 2007

Indeed, I agree with Scott’s point: as an Irishman, when I got to the test it seemed to me to be colossally ignorant, and entirely slanted to favour Americans.
#10 chezjake
January 26, 2007

Excuse me, but IIRC you had another European country in that quiz besides Denmark and Ukraine — Serbia. So what did the results show up there? Does the fact that its borders are relatively recent put everyone at a disadvantage?
#11 "Q" the Enchanter
January 26, 2007

Yikes. A lot of harsh commentary here. The complaints seem to spring from a priori expectations about what what the tests were designed to prove rather than from the actual, subsequent analysis. Are there really any terribly unreasonable inferences drawn? I don’t think so.

At all events, remember folks, these polls are sprung on *casual* Fridays. Relax and enjoy the show.
#12 coturnix
January 26, 2007

I grew up in Serbia (then Yugoslavia) so I knew the Serbia question, but would expect anyone who’s finished high school earlier than 5 years ago not to be sure about the new borders.

Also, Denmark was a no-brainer because in many European countries kids (including me) spend a whole year in geography (6th grade or so) studying EVERY European country in detail. Such level of detail is not given to the study of other countries (including North America)- all other continents were studies as continents, not country-by-country.
#13 Jose Braga
January 27, 2007

In my opinion, your test is nonsense. It is like trying to compare apples to bananas. There is no support in statistics for an experiment designed like that, and no serious conclusions can be drawn from it. Although it is clear that your intention was not to prove anything…
#14 irony miner
January 27, 2007

Looks to me as if that long-term research on Internet irritability and querulousness is collecting considerable data. I look forward to its April 1 publication of results.
#15 Dave Munger
January 27, 2007

First, the Serbia data:

There was no difference between accuracy on this question between America and the rest of the world, even with Europeans averaged in. However, Europeans did have a slight advantage, answering the question accurately 63 percent of the time compared to 50 percent for America and the rest of the world. This may support Coturnix’s argument that all Europeans get European geography drilled into them in school.

Second, the continued complaints about bias in the study:

Yes, the original comparison was an “apples to bananas” comparison. It was intended as a bit of a joke, but we were also attempting to address two stereotypes: 1, that Americans are ignorant about world geography, and 2, that the rest of the world is well-informed about America in general, and American geography in particular.

The first stereotype, if untrue, would presumably reflect poorly on anyone who held it — Americans or otherwise. If true, it would reflect poorly on Americans. We found no evidence to confirm the stereotype, so if you held this stereotype, you might be disappointed in our results.

The second stereotype, if true, would reflect poorly on Americans, because it would imply that Americans care less about the world than the world cares about Americans. If untrue, it would reflect poorly on those who held the stereotype. Again, we found no evidence to support this stereotype.

Interestingly, all of my jokes in the posts about this study have been about Americans — are they “ignorant,” etc. Yet it is non-Americans who are doing all the complaining. Is this because they held these stereotypes, which weren’t confirmed by our study?

I’m as disappointed as anyone when a hypothesis isn’t supported by the data, but I’m surprised that more people don’t find it interesting how similar the test results were, from people around the globe. Shouldn’t that be cause for optimism, not complaint?
#16 Slocum
January 27, 2007

“A secondary point – as a non-American (Canadian), the equation of US states to other countries seemed to me to be another example of the sort of US-centric approach that is all too common.”

But why? Are US states really not comparable to European countries for these purposes? Consider California and Denmark. California is about 7 times the size of Denmark in population, more than that in area and arguably California’s cultural and economic influence exceeds Denmark’s by an even wider margin. So wouldn’t it be a more serious manifestation of geographical ignorance not to be able to locate California on the globe than Denmark?
#17 G.O.Graphic
January 27, 2007

A much better and less biased survey was this 2002 National Geographic quiz, comparing Americans’ knowledge of global geography to that of several other countries.

http://news.nationalgeographic.com/news/2002/11/1120_021120_GeoRoperSurvey.html

Americans were 2nd worst, just ahead of Mexico. Americans came in dead last, for instance, in being able to estimate the population of the USA.
#18 mike
January 27, 2007

I would like to see American geographical knowledge broken out by race and class. I wouldn’t be surprised if our large percentage of academically-underperforming, low-income ethnic minorities isn’t a significant contributor to our differences in scores with Europeans.
#19 P-A
January 31, 2007

I am not american, but European working in Asia.
– I did enjoy the idea of the test, and found it, did not find it **that** american centric.
– I did find the results interesting, showing that for a specific profile of the population, the stereotype does not seem to hold
– There might be other surveys with other sample groups and testing methodologies (such as the Nat Geo one) but I don’t expect any survey to come up with results in line with other surveys. To many variables, influencing the end result greatly.

Overall, a good addition to the already existing surveys.

Its value add for me? Promotion of a few key messages: Never take a stereotype for granted. Always be aware that stereotypes reflect the behaviour of a % of the population only, in a specific context, at a point in time only. Being french -and often subject to stereotypes- I wish we could promote that message a bit..

thanks for the great work
#20 Chris Tregenza
January 31, 2007

I would be interested on a different take on this question. Instead of using a self-selecting sample for a survey, look at what oppertunities the average person has to learn about geography.

An examination of international news coverage on analog broadcasters would prove interesting. How many different countries are identified in a week’s news in the USA compared to the rest of the world?

My experience with US news is that, apart where American troops are fighting, there is very little international coverage. Where as here in the UK it is always featured strongly. This may be because of our colonial past and large imigrant population means that we are more interest in and accustom to news from outside the UK.

Chris