The Skeptics Society & Skeptic magazine

Why is Critical Thinking
so Hard to Teach?

by Kevin Mccaffree & Anondah Saide

Critical thinking has long been recognized as the vehicle by which individuals make informed decisions. Yet, shockingly little understanding exists of how critical thinking strategies are best diffused to the public. In the U.S. there are several regional grassroots organizations such as the Center for Applied Rationality that exist to encourage the development of critical thinking skills. Strategies are numerous and varied, ranging from straightforward group discussions of cognitive biases to thought experiments designed to improve objectivity and to develop the ability to see things from another’s perspective. In addition to such organizations that target individuals, groups and corporations, many colleges and universities offer classes that teach critical thinking strategies.

The Skeptics Society’s own Skeptical Studies Curriculum Resource Center, informally known as Skepticism 101, provides hundreds of resources from professors across the country actively teaching their own critical thinking courses. The skeptical and secular community feel the high percentage of the general public who believe pseudoscientific claims is worrisome, and education is seen as the means by which believers can be reasoned out of their misconceptions. Indeed, with survey data showing that between 67 and 73 percent of adults in the U.S. subscribe to at least one paranormal belief^{1, 2} this topic needs empirical clarification.

Education and Paranormal Belief

Unfortunately, the empirical relation between educational attainment in general, and belief in the paranormal (e.g., in ghosts, astrology, telepathy) is a murky one. The results of research on whether education (as measured by number of years of formal education received) decreases belief have been mixed. Sociologist Erich Goode³ has shown that educational attainment doesn’t necessarily reduce belief in supra-empirical ideas, but rather it appears to moderate it. Educated people tend to simply believe different (demonstrably false) things than less educated people. For example, in a study by Tom Rice, college educated individuals were more likely to believe in psychic healing and déjà vu, while those with only a high school education were more likely to believe in traditional religion and astrology.⁴ The Baylor Religion Survey found that individuals with less than a high school diploma were more likely to have consulted a psychic, while college graduates were more likely to claim an out-of-body experience.¹ This suggests that rather than decreasing belief, education influences the nature of the beliefs a person holds (e.g., belief in homeopathy v. astrology).

Critical Thinking and Paranormal Belief

Given that educational attainment in general is not a prophylactic against holding supernatural or paranormal ideas, researchers have zeroed in on critical thinking training. However, research on critical thinking indicates that current training strategies in general do not necessarily decrease belief in the supernatural. An Austrian study that utilized both the Cornell Critical Thinking Test and Watson-Glaser Critical Thinking Appraisal found no significant relation between these measures and belief in the paranormal. On the other hand, there is some evidence showing that individuals with an analytical cognitive style subscribe to distinctly less conventional views of God (e.g., deistic, pantheistic).⁵ Other research has shown that individuals usually endorse supernatural beliefs simply because of a perceived consensus among others that these beliefs are, in fact, justified.^{6, 7} Thus, individuals may not necessarily—or at least consistently—engage their critical faculties in the assessment of supernatural beliefs; they may evaluate only the probability of their truth given the beliefs of others in their environment, and choose to believe (or not) on that basis.

This paper provides some evidence in support of the view that critical thinking may be as social as it is psychological. For the most part teaching critical thinking has focused on imparting specific cognitive skills to an individual thinker. What many critical thinking seminars and college courses overlook is the role of “fitting in”—critical thought may be as much about avoiding judgment and punishment from others as it is about the deployment of some “toolbox” of thinking strategies.

Despite the commonly held view that being aware of our cognitive biases is useful in combating faulty thinking, we argue that critical thinking is not strictly a cognitive issue. Too much focus on the psychological aspects that influence critical thinking may obscure the role played by a strong need to be social and to fit in. We present a meta-analysis that combines the results of multiple peer-reviewed studies published over the last several decades that evaluate the success of teaching critical thinking strategies in the classroom. In addition, we discuss some reasons for their limited impact.

Data and Methods

The purpose of this research is to consider the effectiveness of college courses in reducing belief in the paranormal and supernatural. These courses all had one or more of the following primary objectives: (1) to teach what science is, (2) to teach how to distinguish science from pseudoscience, and/or (3) how to think critically about new information. Our search criteria included peer-reviewed empirical studies that: (1) measured belief in the paranormal pre- and post-course content, (2) took place at a university or college within the United States, and (3) in most cases, also measured critical thinking pre- and postcourse content. One caveat to this last criterion is that although the critical thinking tests used were not the same across studies (e.g., Cornell Critical Thinking Test v. Watson-Glaser Critical Thinking Appraisal) they were all administered for the same purpose (i.e., to measure critical thinking) and have been independently statistically validated by other empirical work.

Each of the courses utilized in these studies took a slightly different approach and placed a different degree of emphasis on various paranormal phenomena. For example, course titles included: “Parapsychology,” “Science & Pseudoscience,” “Paranormal Phenomena,” “Paranormal Statistics,” “Research Methods in Psychology,” and “Psychology of Critical Thinking.” Researchers from each study gave students a survey to measure their belief in the paranormal (e.g., using the Paranormal Belief Scale) before and after exposure to the course content. Although research exists on the relation between critical thinking and religious belief⁸, we were more interested in how successful college level courses specifically designed to increase critical thinking were in decreasing belief in the paranormal (though “traditional religious beliefs” is one of seven subcategories measured in the Paranormal Belief Scale [PBS]⁹ that many researchers use).

We were able to collect statistics for only eight courses¹⁰ that measured the magnitude and direction of the change in paranormal belief. No other studies matched the search criteria listed above. Of the courses that did match, most had been taken by psychology undergraduates, and the studies contained significance tests to determine if paranormal belief scores changed in a statistically significant way after students were exposed to the course content. Basically they asked the question “Did the students’ general belief in paranormal phenomena decline?” In five out of those eight studies, critical thinking was also measured both pre- and postcourse.¹¹ The significance tests in these studies answered the additional question: “Did the students’ critical thinking scores increase?” We were most interested to see if belief in the paranormal decreased along with an increase in critical thinking ability. With the few studies that met our criteria we conducted meta-analytic procedures that converted the significance tests to correlations between the pre- and post-scores. This allowed us to combine and contrast the studies as well as ascertain the strength of the relation between the preand post-change in scores.

Results

The first set of analyses explored whether or not these courses decreased students’ paranormal belief. The second set examined whether or not critical thinking scores increased.

First, the average effect size associated with a change in level of belief in paranormal phenomena pre- and post-course content was r=.67 which is very high, and statistically significant. The students’ purported belief in the paranormal declined significantly and substantially from the time they started the course to the time it ended. The reduction in paranormal belief was so significant that over 200 studies showing no such relationship would need to exist in order for these results to be statistically questionable.¹² Therefore, it appears that these courses decrease purported belief, at least in the short term.

On the other hand, the average effect size associated with a change in critical thinking, as opposed to paranormal belief pre- and post-course content was r=.08. This tiny correlation wasn’t statistically significant—that is, this effect may well have shown up by mere chance. Taken together, we find that although students’ paranormal beliefs decline by the end of a course, their actual ability to think critically exhibits no corresponding increase. This suggests that they did not abandon paranormal beliefs because they became better critical thinkers.

It also suggests there may be other variables lurking here: tribal identity and social inclusion.

The Social Dimension of Critical Thinking

There are several reasons why students may report decreased levels of paranormal belief despite little or no increase in critical thinking. First, a caveat. It is possible that these students have actually employed their new critical thinking “toolkits” in the service of reducing paranormal beliefs, and that, for whatever reason, this increase simply wasn’t picked up in post-testing. However, this is highly unlikely to have occurred consistently across five studies. What more likely occurred is what we suggested above—a reduction in paranormal belief without any parallel increase in critical thinking ability.

Social Mechanisms

Why did paranormal beliefs decrease across these studies without an increase in critical thinking? We suggest three social mechanisms. First, the content of these courses might have raised more cognitive dissonance for some individuals than typical course content in psychology and philosophy. Calling upon students in an introductory course to question their “sacred” views on karma, astrology, spiritual healing and the like is probably more emotionally complicated than learning about Freud or Socrates. As a result, students may disengage from the course (consciously or not) and experience something akin to apathy. They may report a decrease in paranormal belief simply because they know this is what the course was designed to do. And they want to avoid the discomfort created by the introduction of conflicting new material— they just don’t want to think about it. While it seems desirable that they reported that their belief in the paranormal has declined that may be entirely motivated by apathy, due to a mildly uncomfortable social environment (in this case, the course and the classroom).

A second social mechanism might be fear of group exclusion. A classroom (or critical thinking “workshop”) is intrinsically hierarchical. In these instances, a leader (e.g., professor or organizer) disseminates knowledge about how to think to a group of students, who are expected to understand the information and internalize it as true. In this kind of social environment, hierarchies are rigid—there is a teacher and there are learners. In such a setting, self-reported beliefs may not be reliable if they simply reflect fear of reprisal or punishment for disagreeing with the views of the teacher and class. Fear of punishment or of ostracism may motivate students to report lower levels of paranormal belief at the end of the class. They may also be motivated by a powerful need for social inclusion, acceptance, and rewards—to “fit in” with a class that, it is assumed, tacitly endorses the professors’ views as the correct ones. Unlike the socially-induced apathy described above, fear of social exclusion may actually be sufficient to change the beliefs of students. Fear of reprisal or punishment may be enough to motivate genuine belief change. But such a belief change would be emotionally motivated, without the need for developing a critical thinking “toolbox.”

The third social mechanism is related to the second. In social environments with a rigid hierarchy (such as classrooms or workshops), students might report reduced beliefs in the paranormal simply due to an appeal to the authority of the hierarchy. That is, students might report lower levels of paranormal belief because they believe authority figures in general (i.g., professors) tend to be correct, whether or not the student understands the reasons for this (i.e., the professor’s supposed superior level of critical thinking). The appeal to authority is also a social environment mechanism because the professor is almost always the sole authority in the environment. If classrooms had two professors instead of one, and each had a different opinion about the validity of paranormal beliefs students might have responded differently. Again, it isn’t necessary that students learn to think critically in order to jettison supernatural beliefs. They could simply be responding to a generalized trust of authority figures.

If any one of the above mechanisms is operating in these classrooms or workshops, student belief in the paranormal will likely return to its original level as soon as they are removed from: (1) an environment that makes them apathetic, (2) an environment that ties critical thinking to social inclusion, or (3) an environment that contains an authority figure who promotes critical thinking about paranormal beliefs. Thus observed decreases in paranormal belief among students who take courses aimed at increasing critical thinking may be real in the short term, but not in the long term. Further study is required to understand how best to teach critical thinking that more permanently reduces belief in the paranormal and supernatural.