How To Evaluate Science Stories

I'm on my way to a taping of the Humanist Views with Host Scott Lohman. I do these now and then and have done so since I first moved to Minnesota back when it was still cold here. We'll be talking about science knowledge, and why basic science knowledge is important. We'll also be talking about how to go about evaluating science stories you encounter in the news, or more likely, on your Facebook feed or in other social media.

Pursuant to this, I wrote a blog post that talks about how science stories go out to the general public. I also report on a request I sent out a few days ago to my own Facebook Friends for their thoughts on which Internet sites are good science sources, and which are not so good.

So, here goes...

How a scientific finding comes to you

A first year graduate student comes up with a project. The idea is that change in A causes a change in B, and this could be important, although in truth the natural phenomenon being studied is a bit esoteric. After a year or so of experimentation, learning, literature search, and thinking about the problem, the graduate student comes to understand that a change in the level of disorder in the state of A is associated under certain conditions, some known and some unknown, with a threshold change in B, but it doesn’t always happen. The threshold itself is as yet unmeasured, but seems like a threshold. In the end, more questions have been raised than answered, but also, more is known about A and B and related things than before.

Eventually, there is a paper, peer reviewed, and about to be published. The University Press Office is informed. The University writer who covers this area of science is on vacation, so a different person not so familiar with that area of science takes on the job of writing the press release. An interview with the graduate student doesn’t go too well, because scientists have dialects that are sometimes more difficult for a non-specialist to understand than are the diverse dialects of a widely spoken language (like English) by someone unaccustomed to them.

During the conversation the writer presses the graduate student for more on the significance of the study. The graduate student claims the study results are significant. But the writer is thinking “cures cancer” or “a better mousetrap” significance, and the graduate student is thinking about statistical tests and p-values. But, during the conversation something is said about something that sounds significant to the writer. The paper is about statistical variation in ATP use in a muscle fiber, and muscle fibers are what’s messed up in many different diseases, as well as in aging. So now the writer contacts a couple of scientists unrelated to the exact research project and asks about its significance. During that conversation it is made clear that curing heart disease is important, even though this research really has little to do with it. But it could be related in the sense that the more we know about muscle and ATP in muscle fibers, the more we know in general, and that can’t be bad when it comes to heart disease, or a long list of other problems.

So the writer writes up the story, and focuses on the value this new research will have in curing heart disease and multiple sclerosis. The real meaning of the original research, which is that we should be measuring the order and disorder of the state of a particular molecule in muscle fibre, instead of measuring, for instance, how much the muscle twitches in a test tube, is not even mentioned in the writeup because it is too difficult to understand and too esoteric.

Under deadline, the writer asks the editor if the near final copy should be run by the graduate student to see if it is right. The editor says no, explaining that “we don’t let the people we interview see the copy because it would not be fair to the other people you interviewed,” or some such excuse. So the copy moves along in the process. The editor creates a title that makes the research look sexy. The writer, feeling the title might be misleading, asks that the title be toned down a bit, and the editor agrees. But the process of putting the press release onto the University web site has already begun, and the original, overstated, title is still in the HTML Metacode where it will show up as the title on a Facebook post about the research.

Then, somebody spots the research and posts it on their Facebook feed. It gets shared and shared and shared and shared, with the original bogus title on top of every share. Almost nobody reads the text under the title; had they done, they would notice a conflict between the title and the text. Even fewer people click through and read the original text of the press release, so almost no one notices that there may be more, or really, less, to the story than the title suggests. Even fewer people, maybe one in 1,000, have a look at the original article, and if they do, they don’t understand much of it because the process of publishing peer reviewed papers also involved making science being reported less, rather than more, understandable. Also, it is only an abstract because the paper is behind a firewall.

Everybody is now stupider than they were before this whole thing started.

(See a cartoon version of this here, hat tip: Michael Tobis.)

And, importantly, this is how science gets muddled even when there are sincere efforts to not muddle it, and in the absence of nefarious muddling by anti-science operatives.

This is not how it goes with all scientific stories. Many scientists, often those once or twice burned, are more careful in dealing with press offices. Many press offices are actually pretty good, and have great writers, and the press releases they produce are better. Many stories get picked up by crack science writers and bloggers who bother to read the original paper, talk to experts, contact the author with questions, then do a good job of presenting the material. But often, something like the above, or a subset of the above, happens. Stupider, many become.

How does the average person who is interested in science, or a particular topic important to them because of something in their life, avoid becoming stupider, and maybe, just possibly, become even smarter? Here are a few guidelines, most of which have to do with encountering this information on the Internet.

1) Do not assume that a title reflects the research. It often does not.

2) Do not assume that a third party writeup is not messed up. It often is.

3) The internet is made of tubes. Some of these tubes are little more than conduits of original press releases, scraped from myriad sources and turned into what look like news stories. These are good places to find out about newly published research. They are entirely unreliable to find out what that research is about. They are like search engines that lie.

4) Find interpretive outlets you can trust. There are many science writers and science bloggers (overlapping entities) who regularly do a good job of describing current or recent research.

5) Time is your friend. Often, even among the better interpretive sites, mistakes are made and research is accidentally mis-represented. But usually, eventually, corrections are made. An absolutely fresh report of new research may be misleading, while just a week or so later, the reporting gets straightened out.

6) In some fields, there are people who are involved in the research (specifically or generally) who also write about it in a blog. The best example I can think of has to do with climate change. RealClimate blog is written by climate scientists. Very often, the blog posts they produce are written by the actual authors of the new papers. They write these blog posts specifically to inform the general interested (and at least somewhat field-aware) public of their findings. Sometimes they write blog posts specifically designed to address misunderstandings that have emerged, as described above, or as is often the case in climate science, because nefarious science deniers have muddled up the message on purpose. Similarly, there are science based medicine sites that write about health and medicine related news, though in my experience these bloggers are experts in their fields but not generally the authors of the work they are writing about, as is often the case with RealClimate.

PLEASE NOTICE THE TWO SPECIALIZED SEARCH ENGINES IN THE SIDEBAR TO THE RIGHT, ONE FOR GOOD SCIENCE SITES IN GENERAL (SKEPTICAL SEARCH ENGINE) ONE FOR CLIMATE SCIENCE SPECIFICALLY!!!

7) In some fields, there are relatively reliable web sites that cover everything encyclopedia style. Again, with Climate, SkepticalScience.com covers every aspect of climate change, as well as denial of climate change science. If something isn’t there, it is because it is so new it hasn’t been covered yet, but will be. You can even contact the authors of this site and ask for more, or for clarification. Other sites are more like topical sites. This is trickier. There are bogus health and diet sites and there are good health and diet sites. Nature News is crap according to everyone I know (I don’t track that site). WebMD tends to be reasonably good, The Mayo Clinic’s site is very reliable. The CDC does a good job of covering disease. These sites will be less current, and very cautious. They won’t say stuff if they are afraid you will misuse the information, but they go out of their way to address common goofs people make in their thinking about the issues they cover.

8) This should be number 1, but in fact, applies to very few people for various reasons, so I put it down here. If you want to be able to evaluate new scientific research in a given area, learn all about that area and become an amateur expert on it. That is not easy. People will tell you it is easy, and claim they have done this. It is not and they did not – if they thought it was easy they missed something. But if your sources are good, you are honest with yourself, have a bit of training or experience with thinking about things in a scientific way (and haven’t simply told yourself you can do this) then you can make this happen.

9) Pursuant to number 8, use sources like Google Scholar to find actual peer reviewed research of interest to you and read it. Many peer reviewed papers will not be easily available to you because they are behind firewalls, but many are OpenAccess. Others, probably all others, can be obtained at a good library, though that can be a lot of trouble. For something really important, where your need for a paper goes beyond your own interest – maybe you are a teacher teaching about the topic – go ahead and contact the paper’s “corresponding author” and ask for a copy. If the paper is an older one, go first to the authors’ web sites and see if there is a downloadable copy there, often this is the case. Try Googling the entire title of the paper, in quotes, followed by the words “download” and “PDF.” Every once in a while this works, just like magic.

There are some great science communicators some of whom are also scientists.

A couple of quick tips on how to tell a good communicator:

  • They communicate in the field they work in, or at least, communicate a lot in. So they know stuff.
  • When they talk they make sense (by itself not a good clue, but helpful).
  • They manage to use some big words or concepts but make them fully understood.
  • They are often interviewed on comedy central, the only really good news network.

Caution: self styled skeptics are often bad sources because they really do think they understand the science, but may not.

  • As a rule if a non-specialist or highly experienced writer tells you that a certain area of science is simple to understand, check your wallet.
  • If a skeptic tells you that “many peer reviewed studies” have proven/disproven something, check your wallet. Then check for the studies.
  • If an argument is the counter to the argument that the science is controlled by big business, chances are both the original argument and the counter argument are worthless.
  • Notice how self styled skeptics often follow a party line that is as much derived from authority as any other argument they may reject because it is derived from authority.

So what are some good science sources, and what are the bad ones?

A few days ago I asked my Facebook friends to suggest what they thought were good, vs. bad, sources on science. Below I’ve placed their recommendations, without links. That is partly because I don’t want to have links to bad sources on this site. If you enter the term supplied here you can find the referenced resources easily.

If you disagree with anything on this list, or want to add to it, just drop a comment below.

I have not included sites like Physorg and other science news aggregator sites. See above for my opinion on those sites. Interestingly, these sites were listed by Facebook friends as either bad or good. In truth, they are probably either bad or good depending on what you do with them.

Not everything here is exactly a science site but you can see where those listings are still relevant.

Science Sources People Say Are Good

The Global Warming Fact of the Day Facebook Page

RealClimate

SkepticalScience.com

Science Based Medicine

Bad Astronomy

PolitiFact

Christian Science Monitor

Wikipedia (Especially as a really smart search engine)

Talk Origin

SCOTUSBlog

Federation of American Scientists

Cultural Cognition Project

Questia

Mayo Clinic

Carl Zimmer

XKCD

Science Sources People Say Are Bad

Whats Up With That

Briebart

InfoWars

Natural News

The Truth Wins

Thunderbolts.info

Answers in Genesis

Discovery Institute

Real Science

Dr. Oz

Mercola

Collective Evolution

Food Babe

Spirit Science

International Medical Council on Vaccination

Categories

More like this

So, you accept the science of climate change and global warming as legit. But you often encounter people, at family gatherings, on your Facebook page, on Twitter, at social events, etc. who don't. Do you keep your mouth shut when someone says something clearly wrong that brings the science into…
Continuing the current discussion of the questionable quality of popular science journalism, British researcher Simon Baron-Cohen weighs in at the New Scientist with his personal experiences of misrepresented research. Baron-Cohen complains that earlier this year, several articles on his work…
Ever have one of those times when you have a cool new blog post all ready in your head, just needs to be typed in and published? Just to realize that you have already published it months ago? Brains are funny things, playing tricks on us like this. I just had one of such experiences today, then…
About once a day, someone tells me that human caused climate change is not real because this or that thing in the latest report by the Intergovernmental Panel on Climate Change (IPCC) contradicts something I, or some other scientists or science writer, has said. I've noticed an uptick in references…

Some other fact check sites:
http://climatefeedback.org/
“Climate Feedback organizes scientists from around the world to comment on the accuracy of a variety of climate change media articles using the emerging technology of web annotation. Scientists’ comments, or ‘annotations’ are layered directly onto the original texts allowing readers to easily identify where and why the coverage is consistent (or inconsistent) with state-of-the-art thinking and knowledge in climate science.
Climate Feedback assigns each article a “credibility” rating based on the annotations and evaluations made by the participating scientists, giving readers an overall guide to the accuracy of the text.”
http://climatefeedback.org/about/

“FactCheck.org’s SciCheck feature focuses exclusively on false and misleading scientific claims that are made by partisans to influence public policy.”
http://www.factcheck.org/scicheck/

http://ossfoundation.us/ has a section on Climate Myths v. Facts.
http://scholarsandrogues.com/2007/07/23/anti-global-heating-claims-a-re…
Post that debunks myths. (Angliss has also written about Spencer/Braswell, Tom Harris, Venus, libertarian science denial, and much more.
http://scholarsandrogues.com/author/angliss/ )

Some sites give information about sources.
http://www.realclimate.org/wiki/index.php?title=RC_Wiki
http://rationalwiki.org/wiki/Main_Page
http://www.desmogblog.com/global-warming-denier-database
http://www.sourcewatch.org/index.php/SourceWatch

SourceWatch doesn't cover all topics equally well:

“Their coverage of global warming denialism, oil and coal interests, and tobacco front groups is superb.
By contrast, their information pertaining to animal testing is atrocious, calling the whole exercise vivisection. They also downplay and whitewash PETA's insanity, hypocrisy, and media stunts, as well as the shenanigans of ecoterrorists and the radical fringes of the animal rights movement. Their coverage of Big Pharma and alternative medicine is rather schizophrenic. Some of their alt-med and pharma pages actually link to good skeptical resources like Quackwatch and The Skeptic's Dictionary debunking nature woo and quackery. However, some other pages are loaded up with anti-vaccination crankery and alt-woo references, including some really egregious quacks like Dana Ullman.
http://rationalwiki.org/wiki/SourceWatch

The following is an updated list of papers on ACC. I'm not qualified to judge how good or comprehensive it is.
https://agwobserver.wordpress.com/
The next includes a number of papers that have been debunked, and includes papers that been misused by climate septics.
https://agwobserver.wordpress.com/anti-agw-papers-debunked/

One of the best sites covering attacks on science was
http://www.climatesciencewatch.org/
After Rick Piltz's death it was inactive. It has been reactivated and has expanded its coverage:

“The next phase of Rick’s program – now aptly renamed: Climate Science & Policy Watch (CSPW) – is to assess the actual implementation of the current administration’s stated support for a comprehensive climate policy and critique its contradictory, behind-the-scenes actions which support accelerated fossil fuel development.”
http://www.climatesciencewatch.org/2015/08/25/7113/

By cosmicomics (not verified) on 09 Sep 2015 #permalink

For understandable fitness, health, and nutrition, I find Dr. Gabe Mirkin's site and e-newsletter very helpful. He has no agenda except helping people become healthy and fit, sells nothing, and gets all his information from peer reviewed professional journals. His clear explanation of published information is welcome and useful . He reviews and updates his explanations as additional information becomes available in published papers.
Here's a link to his "about" page.
http://drmirkin.com/about-dr-mirkin

By Justa Retiree (not verified) on 09 Sep 2015 #permalink

There's a word for second-hand and third-hand (etc.) stuff that's spread via Facebook, and the word is "rumor." The best preventive is to not spread things without checking them first. The penalty for spreading rumors is loss of one's own credibility. Better yet, exit Facebook entirely and hang out on blogs with smaller and more focused communities of people interested in particular subjects.

---

Obvious way to spot quack sites of all kinds: they use Lots Of Capital Letters and/or they often say they have "The Truth" or "The Real Truth" about one thing or another. Often they have flashy animated clutter on their pages, but just as often they may try to look like legitimate media. Avoid like plague.

Beware of anything that says "breakthrough" or uses similar hyperbolic language, unless you see it on a known high-quality site. Beware of emotion-heavy language that plays on common human hopes such as for cures for diseases or answers to fundamental questions.

Study the quack sites and silly nonsense sites for the purpose of learning their subcultural dialects, so you can spot the various keywords in other places that aren't quite so obvious. For example "health freedom" is a buzz phrase for anti-vaccination garbage and also for "integrative medicine," which itself is a buzz phrase for "integrating" nonsense into medicine.

Find places such as Scienceblogs where working scientists write for the public or hang out, and ask questions. "I just saw this article on (subject) at (link), and I was wondering (question)." Do it under a pseudonym if you're afraid you might embarrass yourself by saying something silly.

Working scientists and their grad students are often willing to answer questions sent in via email, but be concise and right to the point. In some cases they'll also be willing to send you copies of papers they've written or otherwise been involved with. But before you ask a bunch of questions, try finding answers for yourself online. Ask the questions in email that you don't see addressed in online discussions.

Re. search engines, beware the Google Bubble effect, whereby Google and some others track you and try to predict what you're looking for, and give you more of the same. That might be helpful in some cases, but more often it ends up leading people down rabbit holes they can't get out of due to Google's persistent cookies. Better, use Ixquick.com or DuckDuckGo.com, which don't do that. Experiment with ways of phrasing searches to get to the best sources of information quickly.

Some working scientists give talks that are recorded on video and available online. These can be useful for background information about a field, or to hear these individuals speaking for themselves in an informal context without someone else trying to interpret their words for them. Searching the person's name on a video search page (Ixquick.com has this) is useful. Long-form interviews of people who work in a field are also useful if & where you can find them on credible sites.

Funny, I would avoid a medical site that insisted on replacing neutral terms such as "integrative medicine" with namecalls that take for granted that which should be demonstrated, i.e., that all activites undertaken by medical practitioners that do not derive from fashionable *American* conventional medicine are worthless. For that matter, I generally avoid any purveyor of the type of binary thinking in which one particular field or approach to research is defined as good and every competing field or approach is treated as belonging to a single field, defined by opposition, that is entirely bad and worthless. It then becomes necessary either to conceal the existence of apparently meaningful research in those areas or to demonize the researchers.

Jane, exactly how does research qualify as 'meaningful' to you?

Jane, it's probably obvious to most readers that the sources that are disparaged, i.e., the ones advocating "integrative medicine", hooky alternative stuff (like magnets & pyramids), etc. are the very ones that are guilty of calling or implying that conventional medicine is "worthless" and that their brand of healing is defined as "good" and others approaches are implied as "bad" or "worthless".

There's an uncomfortable undertone to your comment that suggests that old "false balance" that "all medical approaches are equal" -- but quite certainly they're not. Only those based on solid science are reliable as being "true". Saying so is not to be disparaged (given that such claims are backed up by evidence, subjected to peer review, etc., which is a given if you're referring to real science rather than pseudo-science).

By Brainstorms (not verified) on 09 Sep 2015 #permalink

OT but sorta not: Murdoch apparently just acquired National Geographic. While we're all recovering from that, consider this: one assumes that means ScienceBlogs, too?

bill:

Murdoch apparently just acquired National Geographic.

Oh dear, that is worrisome. I'm a longtime subscriber, and I've been pleased with the NGS's reporting on conservation and environmental issues. I'll be watching for signs of Murdoch's influence.

By Mal Adapted (not verified) on 10 Sep 2015 #permalink

Bill: Murdoch did not acquire, as in buy or own, NG. Rather, the long term relationship between NGS and FOX (the two together have produced most of the NGS documentaries as a partnership for many years now) has been extended to the magazine and some other assets. I have no idea what the new relationship means to the magazine. I suspect little. FOX owns and produces a lot of stuff, only some of which includes the highly politically charged crap we associate (justifiable) with FOX.

I have heard that a climate change related documentary in in the works now. That will be an interesting test of the whole thing.

This has nothing to do with scienceblogs. Scienceblogs.com has always had about the most editorial independence of any similar network of bloggers, and as far as I know that continues. Most other networks are highly coordinated with a central authority determining what happens and does not happen, or follow a standard model with typical editorial control or direction, or has no model but occasionally bloggers get slapped or tossed out having had personal discretion up to that moment.

Greg @ 11: Excellent news, as I was worried about Fox also. I was at the point of asking a colleague who's a full-time professional website developer if he'd be willing to do a pro-bono site that could fully replace ScienceBlogs, if it looked like there was about to be an exodus from here. Good to hear that won't be necessary any time soon.

Also and importantly, apparently one of Murdoch's sons is ferociously on our side about climate change, and has managed to sway Rupert himself on that issue. The son also has a serious dislike for Roger Ailes. Sorry that I don't have a cite and a link for it, but if you Ixquick-search the relevant terms, the articles will probably pop up. This stance hasn't been reflected on Fox Noize USA yet, but after Rupert's gone, that son is lined up to take over the empire. At that point we should expect to see some changes, including at Fox Noize.

Jane @ 5: All language has emotional associations and implications. Today's neutral word is tomorrow's hypercharged buzzword and vice-versa. In the 1960's the respectful word for African Americans was "Negro" with a capital N, and the word "black" was considered disrespectful. Today "black" as an adjective is a respectful word, and any use of the word "Negro" is considered suspect.

But beyond that, there's medicine and there's not-medicine and there's outright quackery, and a lot of the latter two categories fly under the banner of "integrative medicine." Anti-vax is not only quackery, it's a danger to public health. Homeopathy and so on have been responsible for numerous individuals not getting real medical care and ending up disabled or dead as a result. Steve Jobs would probably be alive today if he'd gotten "standard of care" treatment for his pancreatic cancer when he was diagnosed. Instead he spent precious time diverted into "integrative medicine."

When "alternative medicine" is found to work, it's no longer "alternative," it's just "medicine." Today's conventional wisdom about the benefits of eating more vegetables and fruits, were 40 years ago considered "alternative," but the public health outcomes validated the more reasonable claims for improved diet. Yoga is useful as a form of exercise, and meditation is useful for improving concentration and reducing stress, though neither of them can cure life-threatening diseases. When someone takes something out of its known-good applications and promotes it as a panacea, that's not medicine any more. Using steroids for cheating at athletics is also a case in point though we usually don't think of it that way.

And nobody is concealing any research, though as for "demonizing," sloppy research in any field is fair game for criticism. There is however a general problem of the under-funding of research across all scientific disciplines. What we need is to triple the budget for all forms of basic and applied research, from astronomy to zoology, and I'd be happy to accept more funding of "wild stuff" into the bargain because there would be no shortage of funding overall. So the lesson here is, whoever you're supporting in this election, let their campaign know that you support a major increase in federal funding of science.