How To Evaluate Science Stories

I'm on my way to a taping of the Humanist Views with Host Scott Lohman. I do these now and then and have done so since I first moved to Minnesota back when it was still cold here. We'll be talking about science knowledge, and why basic science knowledge is important. We'll also be talking about how to go about evaluating science stories you encounter in the news, or more likely, on your Facebook feed or in other social media.

Pursuant to this, I wrote a blog post that talks about how science stories go out to the general public. I also report on a request I sent out a few days ago to my own Facebook Friends for their thoughts on which Internet sites are good science sources, and which are not so good.

So, here goes...

How a scientific finding comes to you

A first year graduate student comes up with a project. The idea is that change in A causes a change in B, and this could be important, although in truth the natural phenomenon being studied is a bit esoteric. After a year or so of experimentation, learning, literature search, and thinking about the problem, the graduate student comes to understand that a change in the level of disorder in the state of A is associated under certain conditions, some known and some unknown, with a threshold change in B, but it doesn’t always happen. The threshold itself is as yet unmeasured, but seems like a threshold. In the end, more questions have been raised than answered, but also, more is known about A and B and related things than before.

Eventually, there is a paper, peer reviewed, and about to be published. The University Press Office is informed. The University writer who covers this area of science is on vacation, so a different person not so familiar with that area of science takes on the job of writing the press release. An interview with the graduate student doesn’t go too well, because scientists have dialects that are sometimes more difficult for a non-specialist to understand than are the diverse dialects of a widely spoken language (like English) by someone unaccustomed to them.

During the conversation the writer presses the graduate student for more on the significance of the study. The graduate student claims the study results are significant. But the writer is thinking “cures cancer” or “a better mousetrap” significance, and the graduate student is thinking about statistical tests and p-values. But, during the conversation something is said about something that sounds significant to the writer. The paper is about statistical variation in ATP use in a muscle fiber, and muscle fibers are what’s messed up in many different diseases, as well as in aging. So now the writer contacts a couple of scientists unrelated to the exact research project and asks about its significance. During that conversation it is made clear that curing heart disease is important, even though this research really has little to do with it. But it could be related in the sense that the more we know about muscle and ATP in muscle fibers, the more we know in general, and that can’t be bad when it comes to heart disease, or a long list of other problems.

So the writer writes up the story, and focuses on the value this new research will have in curing heart disease and multiple sclerosis. The real meaning of the original research, which is that we should be measuring the order and disorder of the state of a particular molecule in muscle fibre, instead of measuring, for instance, how much the muscle twitches in a test tube, is not even mentioned in the writeup because it is too difficult to understand and too esoteric.

Under deadline, the writer asks the editor if the near final copy should be run by the graduate student to see if it is right. The editor says no, explaining that “we don’t let the people we interview see the copy because it would not be fair to the other people you interviewed,” or some such excuse. So the copy moves along in the process. The editor creates a title that makes the research look sexy. The writer, feeling the title might be misleading, asks that the title be toned down a bit, and the editor agrees. But the process of putting the press release onto the University web site has already begun, and the original, overstated, title is still in the HTML Metacode where it will show up as the title on a Facebook post about the research.

Then, somebody spots the research and posts it on their Facebook feed. It gets shared and shared and shared and shared, with the original bogus title on top of every share. Almost nobody reads the text under the title; had they done, they would notice a conflict between the title and the text. Even fewer people click through and read the original text of the press release, so almost no one notices that there may be more, or really, less, to the story than the title suggests. Even fewer people, maybe one in 1,000, have a look at the original article, and if they do, they don’t understand much of it because the process of publishing peer reviewed papers also involved making science being reported less, rather than more, understandable. Also, it is only an abstract because the paper is behind a firewall.

Everybody is now stupider than they were before this whole thing started.

(See a cartoon version of this here, hat tip: Michael Tobis.)

And, importantly, this is how science gets muddled even when there are sincere efforts to not muddle it, and in the absence of nefarious muddling by anti-science operatives.

This is not how it goes with all scientific stories. Many scientists, often those once or twice burned, are more careful in dealing with press offices. Many press offices are actually pretty good, and have great writers, and the press releases they produce are better. Many stories get picked up by crack science writers and bloggers who bother to read the original paper, talk to experts, contact the author with questions, then do a good job of presenting the material. But often, something like the above, or a subset of the above, happens. Stupider, many become.

How does the average person who is interested in science, or a particular topic important to them because of something in their life, avoid becoming stupider, and maybe, just possibly, become even smarter? Here are a few guidelines, most of which have to do with encountering this information on the Internet.

1) Do not assume that a title reflects the research. It often does not.

2) Do not assume that a third party writeup is not messed up. It often is.

3) The internet is made of tubes. Some of these tubes are little more than conduits of original press releases, scraped from myriad sources and turned into what look like news stories. These are good places to find out about newly published research. They are entirely unreliable to find out what that research is about. They are like search engines that lie.

4) Find interpretive outlets you can trust. There are many science writers and science bloggers (overlapping entities) who regularly do a good job of describing current or recent research.

5) Time is your friend. Often, even among the better interpretive sites, mistakes are made and research is accidentally mis-represented. But usually, eventually, corrections are made. An absolutely fresh report of new research may be misleading, while just a week or so later, the reporting gets straightened out.

6) In some fields, there are people who are involved in the research (specifically or generally) who also write about it in a blog. The best example I can think of has to do with climate change. RealClimate blog is written by climate scientists. Very often, the blog posts they produce are written by the actual authors of the new papers. They write these blog posts specifically to inform the general interested (and at least somewhat field-aware) public of their findings. Sometimes they write blog posts specifically designed to address misunderstandings that have emerged, as described above, or as is often the case in climate science, because nefarious science deniers have muddled up the message on purpose. Similarly, there are science based medicine sites that write about health and medicine related news, though in my experience these bloggers are experts in their fields but not generally the authors of the work they are writing about, as is often the case with RealClimate.

PLEASE NOTICE THE TWO SPECIALIZED SEARCH ENGINES IN THE SIDEBAR TO THE RIGHT, ONE FOR GOOD SCIENCE SITES IN GENERAL (SKEPTICAL SEARCH ENGINE) ONE FOR CLIMATE SCIENCE SPECIFICALLY!!!

7) In some fields, there are relatively reliable web sites that cover everything encyclopedia style. Again, with Climate, SkepticalScience.com covers every aspect of climate change, as well as denial of climate change science. If something isn’t there, it is because it is so new it hasn’t been covered yet, but will be. You can even contact the authors of this site and ask for more, or for clarification. Other sites are more like topical sites. This is trickier. There are bogus health and diet sites and there are good health and diet sites. Nature News is crap according to everyone I know (I don’t track that site). WebMD tends to be reasonably good, The Mayo Clinic’s site is very reliable. The CDC does a good job of covering disease. These sites will be less current, and very cautious. They won’t say stuff if they are afraid you will misuse the information, but they go out of their way to address common goofs people make in their thinking about the issues they cover.

8) This should be number 1, but in fact, applies to very few people for various reasons, so I put it down here. If you want to be able to evaluate new scientific research in a given area, learn all about that area and become an amateur expert on it. That is not easy. People will tell you it is easy, and claim they have done this. It is not and they did not – if they thought it was easy they missed something. But if your sources are good, you are honest with yourself, have a bit of training or experience with thinking about things in a scientific way (and haven’t simply told yourself you can do this) then you can make this happen.

9) Pursuant to number 8, use sources like Google Scholar to find actual peer reviewed research of interest to you and read it. Many peer reviewed papers will not be easily available to you because they are behind firewalls, but many are OpenAccess. Others, probably all others, can be obtained at a good library, though that can be a lot of trouble. For something really important, where your need for a paper goes beyond your own interest – maybe you are a teacher teaching about the topic – go ahead and contact the paper’s “corresponding author” and ask for a copy. If the paper is an older one, go first to the authors’ web sites and see if there is a downloadable copy there, often this is the case. Try Googling the entire title of the paper, in quotes, followed by the words “download” and “PDF.” Every once in a while this works, just like magic.

There are some great science communicators some of whom are also scientists.

A couple of quick tips on how to tell a good communicator:

• They communicate in the field they work in, or at least, communicate a lot in. So they know stuff.
• When they talk they make sense (by itself not a good clue, but helpful).
• They manage to use some big words or concepts but make them fully understood.
• They are often interviewed on comedy central, the only really good news network.

Caution: self styled skeptics are often bad sources because they really do think they understand the science, but may not.

• As a rule if a non-specialist or highly experienced writer tells you that a certain area of science is simple to understand, check your wallet.
• If a skeptic tells you that “many peer reviewed studies” have proven/disproven something, check your wallet. Then check for the studies.
• If an argument is the counter to the argument that the science is controlled by big business, chances are both the original argument and the counter argument are worthless.
• Notice how self styled skeptics often follow a party line that is as much derived from authority as any other argument they may reject because it is derived from authority.

So what are some good science sources, and what are the bad ones?

A few days ago I asked my Facebook friends to suggest what they thought were good, vs. bad, sources on science. Below I’ve placed their recommendations, without links. That is partly because I don’t want to have links to bad sources on this site. If you enter the term supplied here you can find the referenced resources easily.

If you disagree with anything on this list, or want to add to it, just drop a comment below.

I have not included sites like Physorg and other science news aggregator sites. See above for my opinion on those sites. Interestingly, these sites were listed by Facebook friends as either bad or good. In truth, they are probably either bad or good depending on what you do with them.

Not everything here is exactly a science site but you can see where those listings are still relevant.

Science Sources People Say Are Good

The Global Warming Fact of the Day Facebook Page

RealClimate

SkepticalScience.com

Science Based Medicine

Bad Astronomy

PolitiFact

Christian Science Monitor

Wikipedia (Especially as a really smart search engine)

Talk Origin

SCOTUSBlog

Federation of American Scientists

Cultural Cognition Project

Questia

Mayo Clinic

Carl Zimmer

XKCD

Science Sources People Say Are Bad

Whats Up With That

Briebart

InfoWars

Natural News

The Truth Wins

Thunderbolts.info

Answers in Genesis

Discovery Institute

Real Science

Dr. Oz

Mercola

Collective Evolution

Food Babe

Spirit Science

International Medical Council on Vaccination