As much as I like to deconstruct pseudoscientific claims, particularly about health, medicine, and health care, Sometimes it gets a bit draining. There’s just so much pseudoscience, so much credulity, so much sheer idiocy out there that trying to refute them and encourage a more skeptical mindset often feels like pissing into the ocean, for all the effect it has. In the age of fake news and Donald Trump, it even feels as though we’re going backward—and not slowly, either. That’s why I felt it was time for a bit of a break, a bit more optimism than I’ve been able to muster before. So it was a good thing that I happened across an article by Julia Belluz and Alvin Chang entitled This researcher may have discovered the antidote to health bullshit.

I’ve always suspected that the key to combatting bullshit of all stripes, be it related to health (one of the more dangerous forms), conspiracy theories, or one of the many other varieties of nonsense, pseudoscience, conspiracy theories, and quackery, will be starting young. Let’s face it. By the time we’re adults, changing the way we think is very difficult. That’s not to say that it can’t be done, but in general you have to want to change. Again, let’s face it. Most people don’t want to change. They resist it. That’s part of the reason why adults are so good at motivated reasoning, in which they become very good at finding observations and evidence that support their preexisting point of view while downplaying or discounting evidence that does not. In essence, they develop only part of the skills needed to be a skeptic in that they are very good at deconstructing ideas they disagree with but remain not so good at critically examining ideas that they do agree with that might lack good evidence to support them.

Yes, as Andy Oxman shows, you have to get ’em young. He relates a story about his visit to his then 10 year old son’s class in 2000.that shows how children actually have a proclivity for becoming skeptical:

“I told them that some teenagers had discovered that red M&Ms gave them a good feeling in their body and helped them write and draw more quickly,” Oxman said. “But there also were some bad effects: a little pain in their stomach, and they got dizzy if they stood up quickly.”

He challenged the kids to try to find out if the teens were right. He split the class into small groups and gave each group a bag of M&Ms.

The children quickly figured out they had to try eating M&Ms of different colors to find out what happens, but that it wouldn’t be a fair test if they could see the color of the M&Ms. In other words, they intuitively understood the concept of “blinding” in a clinical trial. (This is when researchers prevent study participants and doctors from knowing who got what treatment so they’re less likely to be biased about the outcome.)

In a short time, they were running their own blinded, randomized trials — the gold standard for testing medical claims — in the classroom. By the end of their experiment, Oxman said, “They figured out that there was little if any difference in the effects of the different colors and they asked me if the teenagers who made the claim really believed that.”

As a result of this experience, Oxman began working with other academics to develop curricula designed to teach critical thinking skills in children. The idea is to arm them with the skills needed to spot “alternative facts” more readily and, as importantly, how to keep fake news from spreading in the first place. He’s not the only one who asked this question. So did Imogen Evans, Hazel Thornton, Iain Chalmers and Paul Glasziou, who wrote a book called Testing Treatments that’s available for free as a downloadable PDF. In 2012, Oxman teamed up with one of the authors of the book, Sir Iain Chalmers, asking him to help to develop curricula based on the concepts in the book appropriate for primary school children

Because Oxman had ties with researchers in Uganda, he tested the new materials there on children between 10-12 years of age. Personally, when I first saw that, I thought that ten years old is a bit old to start the process. I would have thought that age eight or even six would be the time to start. Be that as it may, Oxman ran a randomized trial in which a representative sample of eligible schools were randomized either to control (no change in curriculum) or including the Informed Health Choices (as the program came to be known) primary school resources (textbooks, exercise books, and a teachers’ guide) in the lesson plan. Teachers teaching the Informed Health Choices curriculum attended a 2 day introductory workshop and gave nine 80 min lessons during one school term. The lessons addressed 12 concepts essential to assessing claims about treatment effects and making informed health choices.

Twelve key concepts were emphasized:

Claims

  • Treatments might be harmful
  • Personal experiences or anecdotes (stories) are an unreliable basis for assessing the effects of most treatments
  • Widely used treatments or treatments that have been used for a long time are not necessarily beneficial or safe
  • New, brand-named, or more expensive treatments may not be better than available alternatives
  • Opinions of experts or authorities do not alone provide a reliable basis for deciding on the benefits and harms of treatments
  • Conflicting interests may result in misleading claims about the effects of treatments

Comparisons

  • Evaluating the effects of treatments requires appropriate comparisons
  • Apart from the treatments being compared, the comparison groups need to be similar (ie, “like needs to be compared with like”)
  • If possible, people should not know which of the treatments being compared they are receiving
  • Small studies in which few outcome events occur are usually not informative and the results may be misleading
  • The results of single comparisons of treatments can be misleading

Choices

  • Treatments usually have beneficial and harmful effects

These are indeed key concepts that any medical skeptic needs to know and understand. I’ve empnasized pretty much all of them at one time or another over the last 12 years. Heck, we even have names for some of them, such as the appeal to antiquity to describe the concept that just because a treatment is old does not make it better. After all, many of these “ancient” remedies date back to times when medicine was anything but scientific and the very basics of what causes disease were not understood and instead diseases were attributed to “imbalances” in humors, “bad air,” or even the intervention of malign spirits. Any of them that might work were basically discovered by sheer accident, and ancient herbal remedies that work have mostly already been picked over and turned into purified drugs.

The result was a publication in The Lancet. At the end, results from students at non-curriculum schools were compared with those of students at Informed Health Choices schools on a 24-question multiple choice test on the 12 concepts (two questions per concept). The results were striking:

The average score for children in the intervention schools was 62·4% (SD 18·8) compared with 43·1% (15·2) in the control schools. The adjusted mean difference (based on the regression analysis) was 20·0% (95% CI 17·3–22·7; p<0·00001) higher in the intervention than in the control group. Appendix 1 shows the distribution of test scores. In the intervention schools, 3967 (69%) of 5753 children had a passing score (≥13 of 24 correct answers), compared with 1186 (27%) of 4430 in the control schools (table 2). The adjusted difference (based on the odds ratio from the logistic regression analysis) was 50% more children who passed (95% CI 44–55; p<0·00001) in the intervention than in the control group.

Noting:

Use of the Informed Health Choices primary school resources had a large effect on the ability of primary school children in Uganda to assess claims about treatment effects. This effect was larger for children with better reading skills, but the intervention was effective for children lacking basic reading skills, as well as for children with basic or advanced reading skills. This effect was achieved even though the learning materials and the tests were in English, which was not the children’s first language. Based on findings from pilot testing both the resources and the test used to measure the outcomes, we were surprised by the size of the effect, which is also large in comparison to other education interventions in primary schools in low-income and middle-income countries,20 and other interventions to teach critical thinking for all ages in high-income countries.11 In addition, the intervention had a positive effect on the children’s intended behaviours and the teachers’ mastery of the key concepts.

Overall, about one-fifth of the children achieved a test score indicating that they had mastered the key concepts (getting at least 20 questions out of the 24 correct), while less than 1% of the children in control schools achieved that high a score. This is quite an effect.

The authors do acknowledge weaknesses in their study. One thing I tend to question is whether the questions on the multiple choice test actually correspond to what the authors say they do. In other words, we don’t know if the children learned how to answer test questions but at the end of the teaching didn’t know how to apply the concepts they learned to real life. We also have no way of knowing what the long term effects of these interventions are and whether they will actually have measurable effects on the health choices made by these participants when they are adults. I also hate to be the pessimist, but I find it disappointing that, even after this intervention, only one-fifth of the children mastered all of the concepts and the average score among those who underwent the teaching still got more than 1/3 of the questions wrong.

As much as there is an emphasis on starting to learn critical thinking at a young age, all is not lost for us old farts:

Separately, the researchers also created a podcast on critical thinking concepts for parents, and tested that approach in another randomized controlled trial, also published in the Lancet. They were successful here as well: Nearly twice as many parents who listened to the podcast series passed a test on their understanding of key health concepts compared with parents in the control group.

Or, in a bit more detail:

We recruited parents between July 21, 2016, and Oct 7, 2016. We randomly assigned 675 parents to the podcast group (n=334) or the public service announcement group (n=341); 561 (83%) participants completed follow-up. The mean score for parents in the podcast group was 67·8% (SD 19·6) compared with 52·4% (17·6) in the control group (adjusted mean difference 15·5%, 95% CI 12·5–18·6; p<0·0001). In the podcast group, 203 (71%) of 288 parents had a predetermined passing score (≥11 of 18 correct answers) compared with 103 (38%) of 273 parents in the control group (adjusted difference 34%, 95% CI 26–41; p<0·0001). No adverse events were reported.

Studies like these, for all their messiness and shortcomings that are unavoidable in carrying out studies like this, give me hope. We human beings have cognitive wiring that leave us prone to making all sorts of incorrect inferences and latching on to all sorts of pseudoscientific beliefs, particularly about health. Critical thinking, while second nature in some areas (such as economic; e.g., when buying a used car), is not natural to most humans, particularly when it comes to health claims, where anecdotes can profoundly mislead and we are very quick to confuse correlation with causation. It can be taught, however. The problem is that there has to be the will and resources to teach it, as well as a large enough core of motivated teachers trained to do it. Then there’s the issue of competing for time among existing subjects that have to be taught. I can’t help but wonder if, rather than teaching critical thinking like this as a module separate from ohter subjects, it would be more effective to find a way to weave training in critical thinking into all subjects, especially science, math, and history, but by no means limited to them and including topics like writing, literature, and other humanities.

However it’s done, the need is acute. 2016 was the year fake news and bullshit appeared to have reached a tipping point. The need to be able to recognize and combat it is more acute than ever, and it’s never too early to start inculcating critical thinking skills into our children, who will need them more than our generation ever did.

Comments

  1. #1 Alain
    May 23, 2017

    Hi Orac,

    The statements about school curriculum in this post got me thinking about an idea I’ve had for a while. As we all know, the statistics about employments in autistic persons are poor (58% unemployed, ht_tps://www.ncbi.nlm.nih.gov/pubmed/25891885 and ht_tps://www.ncbi.nlm.nih.gov/pubmed/24077909 from that search: (“Unemployment”[Mesh]) AND “Child Development Disorders, Pervasive”[Mesh] ) and it’s not for lack of motivation or education. On the other hand, I do know our governmental agency here in the province will provide a small stipend to both the employee and the employer for part-time work (20 hours per week) and there are other programs for salaried workers depending from where the pool of money come from.

    In the case of the 20 hours program, my brother is going for an interview today and tomorrow for a position where he will assemble utensils in bag for restaurants (I’m not sure of the status of the company in question: non-profit or for profit) but in my case, the idea consist of something like Cochrane collaboration coupled with an educational program for (autistic) workers with the result of that work given to the population in the form of scientific publications, layman publications and maybe more as the idea or business grow.

    Of course, my comment is a tangent to critical thinking but I have that in mind as a goal too, which is to teach about it, especially in the layman publications area.

    Basically, I want to kill many birds (critical thinking, jobs for autistic person, taming the many firehoses of scientific publishing) with one stone.

    Alain

  2. #2 Dorit Reiss
    May 23, 2017

    I hope they start providing this curriculum to schools here, and in other countries, too. It may not be perfect, but it sounds like it helps.

  3. #3 HeatherVee
    Next to the radio
    May 23, 2017

    I would be pleased to have my kids exposed to more critical thinking skills in school (in the context of health choices and in other areas), but I can also imagine the protests that some parents would make, especially if their kids started applying those critical thinking skills to dearly-held political or religious beliefs.

  4. #4 Michael J. Dochniak
    Minnesota
    May 23, 2017

    Orac writes,

    We human beings have cognitive wiring that leave us prone to making all sorts of incorrect inferences and latching on to all sorts of pseudoscientific beliefs, particularly about health.

    MJD says,

    Q. Can critical thinking eliminate the placebo effect.

    If the answer is NO, integrative medicine is in it’s infancy.

  5. #5 Eric Lund
    May 23, 2017

    @Heather: Indeed, schools in many US states, most notoriously Texas, are de-emphasizing critical thinking. Because if the kids learn to think critically, they might end up like Molly Ivins:

    I believe all Southern liberals come from the same starting point–race. Once you figure out they are lying to you about race, you start to question everything.

  6. #6 darwinslapdog
    The Beagle-it might be sinking!
    May 23, 2017

    Good news–now, how do we get it to go viral? How do we make critical thinking “cool”?

    My college student grandson is taking every logic class available and we are having great fun discussing and applying what he’s learning. Grammie is learning some new bits as well! I have to add that he feels he’s pretty much alone in this effort and that he is surrounded by (how to say it?) idiots at his university. They actually have a “pre-chiropractic” major! This is a very well-respected State University.

  7. #7 Eric Lund
    May 23, 2017

    They actually have a “pre-chiropractic” major!

    Which I presume is distinct from a “pre-med” major. Compared to the latter, the “pre-chiropractic” major would presumably have less stringent requirements for organic chemistry, which you don’t really need if you are just going to be manipulating your patients’ spines. But I’m not sure I want to know the details.

    I have been fortunate never to be affiliated with a university that had a separate “pre-med” major (there is a nursing program at my present university, but for many in that program it’s a terminal bachelor’s degree). If you want to follow up your undergraduate education with medical school, you have to take an actual major (for obvious reasons biology is the most common choice, but there is no set requirement; humanities, chemistry, physics, and engineering majors have been known to go on to med school) plus whatever additional courses the med schools require that aren’t part of your major or general requirements. I like to think it helps to train potential doctors in critical thinking (at least in the area covered by the major), but it’s not foolproof. Pre-meds are notorious, at least among physics types, for doing minimum work for maximum grade (the ones who actually major in physics are exceptions, because unlike most pre-meds they are actually interested in physics).

  8. #8 JeffM
    May 23, 2017

    Who needs critical thinking courses (and civics, I might add) when we have Internet sites like Infowars and Natural News?

  9. #9 Jazzlet
    May 23, 2017

    I used to work for a local authority promoting road safety and environmental awareness (yes an odd combination, but that’s local authorities for you). We found that resources which embedded the messages within the existing curriculum were most likely to be adopted by teachers. When a national curriculum was introduced there was a huge opportunity to provide resources that made it easy for teachers to ‘tick the boxes’ they needed to tick. Teachers are under a lot of pressure both in terms of time and money for lesson materials so it is important that resources are of good quality and either provided free or copyright-free especially as you are competing with materials provided by large corporations which can have unhelpful messages alongside the maths english or whatever.. There is a lot to be said for providing resources in a variety of subjects which cover the subject requirements, but which also teach critical thinking.

  10. #10 Eric Lund
    May 23, 2017

    @JeffM: As Sam Goldwyn said, “If I want your opinion I’ll give it to you.”

  11. #11 Denice Walter
    May 23, 2017

    In light of my own areas of expertise ( aw… rather than recount, I think you know the drill), SURE, you have to get them young- the foundation of abstract and scientific thought ( think its mathematic roots) lies in pre-formal operational thought so of course the earliest school levels.

  12. #12 Denice Walter
    May 23, 2017

    re Orac’s last line in the OP
    my comment above should also clarify the problems the adults had this year..

  13. #13 Old Rockin' Dave
    In the Lair of the White Worm, who sadly turns out to be a white worm nationalist
    May 23, 2017

    When my kids were just kids I would make an effort to get them to question what they saw.
    One time stands out in my memory. The kids were in elementary school, and the Fox network (of course) had a credulous special with Uri Geller. When he did his spoon bending I told them to concentrate on watching only his hands. When the host said, “Now we will see…” my daughter cut in with “a man bending a spoon.” They both cracked up. I stopped worrying that they wouldn’t learn to think critically.

  14. #14 Robert Blew
    United States
    May 23, 2017

    Suspected that you would be writing about this article but did not expect it to be the next day. However would like to point out that it also requires “education” to get people to not think. This idea is covered in Suki Kim’s work, “The Art of Essay Writing for the Brainwashed” {https://www.pri.org/stories/2014-12-29/art-essay-writing-brainwashed}, Without You, There Is No Us: My Time with the Sons of North Korea’s Elite {http://www.goodreads.com/book/show/20685373-without-you-there-is-no-us}.

  15. […] Source: Teaching critical thinking to combat fake news and bullshit: You have to start young […]

  16. #16 Lighthorse
    May 24, 2017

    Earlier this year, a colleague sent me the following:

    At the University of Washington, biologists were recently deluged with over 20,000 applications for enrollment in a new course titled “Calling Bullshit in the Age of Big Data”. In 20 minutes, the course was entirely booked up. The syllabus and course is free for other colleges and universities to teach, provided acknowledgement of the originators.

    The web site for the course offers a bevy of useful resources. As you can see from the course descriptions, one of the subjects to be covered is homeopathy. For a laugh, read the disclaimer on the home page, but don’t mistake the course as a joke.

    http://callingbullshit.org/index.html

  17. #17 Daniel Corcos
    May 24, 2017

    Is it it critical thinking when you are given the right information? I would rather say that critical thinking is when you understand that what you are told cannot be true.

  18. #18 C.C.
    May 25, 2017

    This study comes out at an interesting time for me. I’ve actually been considering changing careers and going into elementary education for a number of reasons. The chance to introduce critical thinking skills to young people is certainly high on pro side of my pro/con list.

  19. #19 shay simmons
    May 25, 2017

    C.C. – I hate to be a downer, but one of the reasons the spousal unit is very glad to be retiring next year (middle school science, civics, geography) is that he does very little actual teaching any more. Most of his classroom time is devoted to preparing the kids to take standardized tests.

  20. #20 Kristina Saunders
    June 1, 2017

    Isnt Orac worried that the Teachers Union, or worse, Berkeley kids and faculty, read this piece? After all, you are indicting public school ehjuhmakayshun and gawd knows they might come after you for criticizing their safe spaces

    So much for American education.

  21. #21 Panacea
    June 1, 2017

    Kristina,

    You think far too little of higher education.

  22. #22 Chris
    June 2, 2017

    Panacea, she also thinks too little of teachers. Especially the challenges dealing with a diverse population, and the hormones starting in middle school.

  23. #23 Rich Bly
    Just back from Chiang Mia to lovely Ocean Shores
    June 6, 2017

    As a former Jr, High Life Science teacher, It can be very hard to even try and teach. I had one class that the average reading percentile was 27. You don’t make much progress with most of the students but work hard to help ones you can.