The Skeptics Society & Skeptic magazine

The public is frequently told to “trust the science,” and then ridiculed for holding any views that differ from what is reported to be the scientific consensus. Should non-experts then naively accept the authorized narrative, or are there good reasons to be skeptical?

Is sugar-free gum good for your teeth?

When we’re told that four out of five dentists recommend sugarless gum, we assume that five dentists independently examined the evidence and four of them concluded that chewing gum is good for your dental health. However, those dentists aren’t examining completely independent evidence. They sat through the same lectures in dental school, they have ready access to the same studies, they go to the same conventions, and they talk to each other, so we should worry about correlated errors.

Even worse, most dentists may have never even read a study about chewing gum, let alone conducted one of their own. Suppose they heard that most dentists recommend sugarless gum; they might well figure those other dentists are probably doing so for good reason, and so they would recommend it too. In other words, the dentists are following the herd mentality and just going along to get along. Perhaps most dentists believe chewing gum is good for dental health because they believe that most other dentists believe this, even though few if any of them have any good, independent reason to think this is true.

Herding can be a rational behavior. It would not be a good use of time or money for every dentist to conduct an independent study to assess the evidence and determine whether sugarless gum is good for dental health. However, herding can lead an entire scientific community to converge on the wrong answer, and they typically won’t know whether they’ve converged on the right or the wrong answer.

We can see how a dangerous emperor-has-no-clothes situation could easily arise. Suppose a dentist questions whether chewing gum really is good for dental health. He or she considers raising the issue at a convention but then remembers that most dentists recommend gum and worries that they’ll be mocked for questioning the consensus view. So they decide to keep quiet, the field moves on, nobody’s beliefs are challenged, and no new evidence is collected.

This may be a low-stakes example, and there probably are good scientific reasons to believe that chewing sugar-free gum is good for dental health. But herding is a problem in many scientific fields, including those studying arguably more important questions, such as the health of democracy.

How We Vote

Consider this example from an academic subfield I happen to know well. Among scholars of political behavior, there is a broad consensus that American voters don’t know or care much about policy, and their voting decisions are largely driven by party identity. Such claims are commonplace in academic papers, conferences, classrooms, textbooks, and public writings. To a member of the general public who has never taken a political science class, this claim might seem absurd. The average American may not be as informed as we would hope, and their policy preferences might diverge from ours. Yet even a brief conversation with a voter would likely reveal that they know and care about policy and think about it when they decide which candidates to support in elections. How can such a strong claim unsupported by good evidence be the scientific consensus?

When I challenged this scientific consensus,¹ I received significant public and private criticism from scholars of political behavior. A few of my critics engaged with my arguments and evidence, but most did not. Instead, they typically made appeals to authority, such as, “How dare you challenge what’s been established wisdom for seven decades?”.

In other words, they were herding. They assumed that something must be right because that’s been the consensus view in their field for a long time. They were not able or willing to provide further evidence or arguments in support of their position, and they simply dismissed anyone who challenged them, thereby creating a strong incentive for other scholars to uphold the consensus.

The Good and the Bad Scenario

Roughly speaking, there are two different ways in which an apparent scientific consensus might arise. In the good scenario, scientists are conducting genuinely good work, rigorously vetting each other’s work, and the theory, the evidence, and the analyses supporting the consensus view are all really strong. In this scenario, if reasonable, objective, intelligent individuals from outside the field examined all of the evidence, they too would be provisionally confident in the consensus.

In the bad scenario, the scientists are not always conducting good work, don’t rigorously vet each other’s work (or they engage in selective vetting based on whether or not they like and/or agree with the conclusions of a study), and the theory, the evidence, or the analyses supporting the consensus are not robust. In this scenario, a reasonable, objective, intelligent individual from outside the field who examined the evidence, the analyses, and the theory would be, at best, genuinely uncertain. Nevertheless, some scientists and all too many media pundits and politicos repeatedly state that there is a scientific consensus in support of their preferred view. Dissenters, whether scientists themselves or not, are ostracized.

Unfortunately, the bad scenario occurs too often— much more often than many scientists, commentators, and cultural leaders presume. We already saw one way in which the bad scenario can arise—herding. Here are some additional ways in which the bad scenario can arise and why skeptics should view appeals to scientific consensus, on their own, as uncompelling. I also discuss how non-experts can better distinguish between the good and bad scenarios, and how scientists can do more to avoid the latter.

The Illusion of Scientific Consensus

Commentators and leaders often assert that their position is the consensus view, but without providing direct evidence of that consensus. Just as social media and public discourse don’t accurately reflect the views of regular Americans, they also need not accurately reflect the views of scientists. Making it even more difficult to assess scientific consensus, those who do not hold the views of the purported consensus are often dismissed as not being legitimate members of the scientific community.

In the rare cases in which we are presented with systematic evidence on the views of the scientific community, the results are often underwhelming. Doran and Zimmerman conducted a survey of earth scientists to assess the extent of scientific consensus on climate change, and they concluded that “the debate on the authenticity of global warming and the role played by human activity is largely nonexistent among those who understand the nuances and scientific basis of long-term climate processes.”² Specifically, in one question, they asked earth scientists “Do you think human activity is a significant contributing factor in changing mean global temperatures?” and 82 percent of them said yes. The meaning of significant is open to interpretation, and even among people who answer yes, there could be genuine disagreement about the extent to which climate change is a problem and the right ways to address it. Furthermore, the survey’s response rate was only 31 percent, and we don’t know if those responding are representative of all scientists who were contacted. Even still, nearly one in five scientists surveyed did not answer yes to this seemingly anodyne question. So maybe the consensus isn’t as strong as we’re frequently told.

Doran and Zimmerman further find that the apparent scientific consensus on climate change gets stronger as they restrict their sample. For example, if they focus on scientists who actively and primarily publish papers on climate change, 97 percent of those scientists answered yes to the question above. One potential interpretation is that when people become immersed in climate science research, they increasingly converge to the truth. Another is that earth scientists who do not hold the desirable view on this question are prevented from publishing papers on climate science. The recent admissions of one climate scientist suggest that journals indeed will not publish the papers of authors who do not conform to the preferred narrative.³

Broad Consensus Doesn’t Mean High Certainty

Scientists in a particular field all have access to essentially the same information, so I would expect many of them to have similar beliefs on many scientific questions. How confident are they in those beliefs?

Even if 100 percent of earth scientists agreed that human activity is a significant contributing factor to an increase in mean temperature readings from around the globe, it would still tell us nothing about the certainty with which they held those beliefs. If someone is only 51 percent sure of a claim, they might answer yes to the forced-choice two-option question. So for all we know, although 82 percent of earth scientists answered yes, all of those individual scientists might still be genuinely uncertain.

For this reason, the percentage of scientists who agree with a statement is not a very informative statistic. How sure are they that human activity influences global mean temperature? (Also, how much do they think human activity influences temperature? If it’s a small effect, we’ll want to consider the other costs and benefits before making any rash decisions; if it’s a large effect, we should allocate more resources to accelerate the transition away from fossil fuels.) For some questions, 49 percent certainty might be more than enough to warrant taking a costly action— if you were 49 percent sure that your car was going to explode in the next minute, you would get out and run. For other questions, 51 percent certainty is not nearly enough—if you were 51 percent sure that you were going to win the lottery, you wouldn’t quit your job.

Unfortunately, surveys of scientists typically elicit no information about the certainty with which the respondents hold their beliefs. However, since 18 percent of scientists did not agree that human activity is a significant factor in changing mean global temperatures, and since those 18 percent have access to largely the same information as the majority, I would be surprised if all of the 82 percent who agree with the statement hold that belief with strong certainty. Indeed, it would be quite strange if 82 percent of experts were virtually certain while 18 percent of experts weren’t even sure enough to say yes to the binary question.

Correlated Errors

A scientific estimate can diverge from the truth for many reasons, but the hope of the scientific community is that if we conduct a lot of studies, the errors will cancel out, and when we conduct meta-analyses, our estimates will converge to the truth.

The problem with this logic is that not all errors cancel each other out. Often, scientific studies are biased, meaning that even if we repeated them over and over with infinitely large sample sizes, we still wouldn’t get closer to the truth. Further, the biases of different, related studies are likely correlated with one another.

Consider the increasingly common claim that diet sodas are bad for your health. Although we currently lack a compelling biological explanation as to why, dozens of scientific studies report that consuming diet soda and other artificially sweetened beverages causes a host of health problems including obesity, diabetes, and heart attacks. What’s the evidence for this claim? People who regularly consume diet soda typically have more health problems than people who don’t consume sweet beverages (people who drink sugary beverages are usually excluded or analyzed separately).

Why is there a strong correlation between diet soda and health problems? It could be that diet soda causes health problems. Alternatively, health problems might cause people to drink diet soda. For example, perhaps people switch from regular soda to diet soda after they become obese or diabetic. Or there could be confounding factors that influence both diet soda consumption and health. For example, perhaps people with a sweet tooth are more likely to consume diet soda and also more likely to consume sugary desserts, which cause health problems. These latter possibilities are sources of bias. Because of reverse causation and confounding, the correlation between diet soda consumption and health is not, in and of itself, convincing evidence that diet soda is bad for you. For all we know, it could be good for you insofar as it’s a substitute for sugary foods and beverages.

It doesn’t matter how many observational, correlational studies we conduct on this topic. They will likely all yield similar results, and we still would not learn much about the actual effects of diet soda on health. If all the studies are biased in the same direction, a scientific consensus could emerge that is based on hundreds or even thousands of studies and still be wrong.

Selective Reporting

Scientific results that happen to align with the predispositions of journal editors and peer-reviewers are more likely to be written up and published than those that go against the accepted wisdom in a field. Scientists often conduct multiple tests and selectively report those that are the most publishable, meaning that the published record is often biased, even if each individual analysis is unbiased. In some cases, scientific results might be skewed in the direction of sensational, surprising, or newsworthy findings. However, once a field has settled upon an apparent consensus and desires to maintain it, perhaps we should worry that results affirming the consensus are much more likely to be published than those that conflict with that consensus.

Archives of Sexual Behavior, a scientific journal published by Springer Nature, recently retracted an article on rapid onset gender dysphoria in response to criticism from activists.⁴ The retraction note says nothing about the scientific validity of either the data or analysis in that article. Rather, the paper was purportedly retracted on the grounds that participants in a survey did not consent to participate in a study, a claim that the author of the study contests.⁵

In 2017, Hypatia, an academic philosophy journal, published a paper entitled “In Defense of Transracialism” [that is, changing one’s racial identity].⁶ Hundreds of academics signed an open letter asking the journal to retract the paper.⁷ The open letter did not seriously engage with the arguments in the paper; rather, it asserted that the availability of the paper causes harm. The associate editors of the journal issued an apology and condemned the paper. The editor-in-chief criticized the associate editors and defended the journal’s review process but resigned soon after.⁸ Ultimately, the paper was not retracted, but the philosophy community has signaled that certain arguments and conclusions would not be allowed in their field.

There are many more examples of academic studies being retracted or condemned for reasons unrelated to merit, credibility, integrity, or validity. And unfortunately, these cases are just the tip of the iceberg. For every public retraction, there are likely many more studies that never make it through peer review because of their undesired or unpalatable results. And for every one of these, there are likely many more studies that never get written up and submitted because the author reasonably infers that a paper with such results would either not be published or would harm their reputation.

Some scientists and journal editors openly admit to engaging in this kind of selective reporting. In 2022, the editors of Nature Human Behavior published new ethics guidelines for their journal.⁹ They reserved the right to decline publication of any submitted paper and retract any published paper that might cause “substantial risk of harm.” In other words, the editors of the journal can reject or retract any study for reasons that are completely unrelated to its scientific validity. So, if the results and conclusion of one scientific result are deemed to be safe by journal editors, while those in another are deemed harmful, only the safe study gets published. This would lead to a scientific consensus around the safe result, even if it were factually wrong.

Fraud

Another obvious but important reason the scientific record might fail to reflect the truth is that some scientists engage in fraud. They might manipulate data points to make their results more favorable or even fabricate entire data sets whole cloth. We would all hope that this kind of outright scientific misconduct is rare, but it does happen. Two prominent behavioral scientists from Harvard and Duke University both independently appear to have intentionally manipulated or fabricated data in different parts of the same study—ironically, a study about dishonesty.¹⁰ The president of Stanford recently resigned (but kept his faculty appointment) after evidence came to light that strongly suggests he intentionally manipulated images in his neuroscience studies.¹¹ And these are just recent, high-profile examples that made the news. There are likely more cases of fraud that don’t come to light, don’t make the news, and do not lead to a correction of the scientific record.

Career Incentives

Partly because of the phenomena discussed above, we can’t know if a scientist who publicly supports a conclusion genuinely holds that view. To publish papers, secure grants, get a good job, get tenure, receive praise, and avoid banishment, scientists must not question the key tenets of their field. Some of this is natural. A biochemist is not likely to make much progress in her field if she doesn’t accept the atomic theory or the periodicity of elements, and biochemistry as a field won’t make much progress if it has to devote significant journal space and lab time to questions that are already well settled. However, most scientific claims aren’t nearly so well-established, and we’ll never know if they’re truly right or wrong if scientists aren’t able to publish novel theoretical perspectives, data, or analyses that challenge them. Paradigms, as defined by Thomas Kuhn, could simply never shift.

In addition to the incentives for individual researchers, scientific fields as a whole often have a strong incentive to collectively uphold a consensus. Virologists won’t be able to secure as much funding and support for their research if the public and the rest of the scientific community were to think that virology researchers caused a global pandemic. As a result, others outside the field shouldn’t necessarily be persuaded by the sheer fact that virologists oppose the lab-leak theory of COVID-19 origins, which, it just so happens, would be very bad for their careers.

Science vs. Values

Not all important questions are scientific questions. What is the effect of eating bacon on my chances of having a heart attack? is a scientific question. Should I eat bacon? is not. When you consider whether or not to eat bacon, you’ll want to think about a lot of things that can be scientifically quantified such as health risks, nutritional value, economic costs, and so on. However, you’ll also want to think about other questions such as How much do I enjoy eating bacon?, What are the ethical implications of eating pig products?, and Does my enjoyment of bacon outweigh the health risks and ethical downsides?. These latter questions are about your personal values, and by the personal experiential nature of the questions, scientists are probably less equipped to answer them than you are.

Just as individual decision-making involves values, so does public policy. So, If we banned the sale of bacon, how much would it increase unemployment in Iowa? is a scientific question, while Should we ban the sale of bacon? is not. And scientists’ values aren’t necessarily any more enlightened than those of citizens, elected officials, and bureaucrats in deciding the latter. So we should consider scientific evidence when assessing the costs or benefits of different policy decisions, but science alone cannot dictate which policies to implement.

Unfortunately, the difference between science questions and value judgments is often forgotten or ignored by scientists themselves. We’re often told things such as, there is a scientific consensus that we should raise the gas tax, economists support surge pricing for parking, education researchers oppose standardized testing, or a scientific journal endorses a political candidate,¹² and similar statements should be roughly as persuasive as anthropologists prefer mayonnaise over mustard. Scientists shouldn’t be in the business of telling people what to do. They should provide people with information so that they can make better decisions and policies conditional on their values.

Politicization

This article presents a number of different explanations for the potential emergence of an unreliable scientific consensus. All of these concerns are exacerbated when a scientific question becomes politicized or is of great public interest. If a particular scientific claim happens to align with the values, policy preferences, or political objectives of scientists, you can imagine that the incentives for misrepresenting the scientific consensus, selectively reporting results, accepting the conclusions of biased studies, and herding become even greater. And if an undesirable result can lead a scientist to be ostracized by not just their peers but by journalists, friends, family, and activists, the distortionary incentives become even stronger.

This poses a vexing problem for the otherwise-promising practice of evidence-based policy. All else equal, the more relevant science is for policy, the less reliable it will likely be. This is because scientists, like everyone else, are individuals with their own values, biases, and incentives. They probably already had strong views about policy before they analyzed any data, which means they’re even more likely than normal to report results selectively, publish biased studies, herd on a politically desirable conclusion, and so on. Unfortunately, this means that we should be more skeptical of scientific findings when that question is particularly politicized or policy-relevant.

All that said, avoid nihilism or worse.

Consumers of scientific information should be skeptical of an apparent scientific consensus, and they should think about some of the factors discussed here when deciding how skeptical they should be. How politicized is this topic? What are the career incentives for the scientists? How easy would it be for scientists to selectively report only the favorable results? Would a study have been published if it had found the opposite result or a null result? The answers to these questions will not definitively tell us whether the scientific consensus is right or wrong, but they should help us decide the degree to which we should simply trust the consensus or investigate further.

Although skepticism is warranted, nihilism is not. Even when a topic is highly politicized and when there are good reasons to worry about biased studies, selective reporting, herding, and so on, the scientific community can still find the right answer. The debate over evolution by natural selection would seem to feature many of the problems I’ve discussed, and yet, the scientific consensus is almost surely right in that case. However, you shouldn’t think evolution is right just because it’s the scientific consensus. You should think it’s right because the evidence is strong. And if scientists want to convince more people about evolution, they shouldn’t simply appeal to scientific consensus. They should present and discuss the evidence.

Science is a process, not a result.

If we want to learn more about the universe for the sake of enjoyment or with the goal of improving our lives, science is our best hope. So don’t become a nihilist, and don’t replace science with something worse such as random guessing or deference to authority, religious or political. Remember that science is just the word we use to describe the process by which we generate new knowledge by questioning, experimenting, analyzing, and testing to see if we are wrong rather than confirming that we are right. It involves repeated iterations of hypothesizing, experimenting, analyzing, empirical testing, and arguing.

If a group of so-called scientists stop theorizing, testing, and challenging, then they’re no longer engaged in science. Perhaps they’re engaged in advocacy, which is a respectable thing to do, particularly if the theory, evidence, and arguments on their side are strong. Yet advocacy and science are distinctly different activities and shouldn’t be conflated.

Science is not a specific person¹³ or even a group of people. Science is not a particular result or conclusion. It is not content but method. It is to remain always open to skepticism while never succumbing to cynicism. The goal of science is not for everyone to agree or behave in the same way. To the extent that there is a goal or purpose of science, it’s for us to challenge what we thought we knew, to obtain new information, and thereby get successively closer to the truth. Of course, we don’t know what the truth is, and the scientific process is imperfect, so, as part of a healthy scientific process, we can sometimes move away from the truth. However, if we’re doing this correctly, we will get successively closer to truth more often than not.

This article appeared in Skeptic magazine 28.4
Buy print edition
Buy digital edition
Subscribe to print edition
Subscribe to digital edition
Download our app

At various points in our history, there has been a scientific consensus that the sun revolves around the earth, that humans do not share a common ancestor with other animals, that force equals mass times velocity, and that bloodletting is an effective medical treatment. More recently, doctors told people for decades to treat soft-tissue injuries with ice, while the most current evidence now suggests that cold therapy delays healing.¹⁴

To its credit, the scientific process has allowed us to correct these mistakes. However, the scientific record is imperfect and ever-changing. So even if the scientific consensus might be right more often than not, we should not accept it on faith alone.

* * *

The scientific community should actively work to address the problems discussed here. It should try to set up better institutions and career incentives to reduce the prevalence of biased studies, selective reporting, and herding. It should do a better job of conveying the uncertainty associated with any scientific claims and beliefs. And it should not impose its values on others. In the meantime, members of the public should continue to be skeptical, but not cynical, while asking for better evidence and arguments before reflexively accepting a reported scientific consensus.