The Skeptics Society & Skeptic magazine

When you drive past any American school, you’ll see signs telling you to reduce your speed and declaring the area to be a “drug-free zone,” with draconian penalties for violators. While we can all agree on keeping drugs away from school children, drugs are not the only thing we keep out of schools. Unfortunately, when it comes to educational policy and practice, research findings have also found themselves banned from schools. Why is that?

The State of Education Research

Getting your measurements and calculations right matters immensely when building an airplane that is unlikely to crash or a building unlikely to collapse. In Turkey and Syria, when a 7.8 magnitude earthquake hit, outdated building methods contributed greatly to the death toll.¹ Engineers and builders need to make sure that the evidence they bring to the table is factually correct. Once you leave the concrete world where accurate facts are prized—or at least clearly have consequences you can detect—things get a lot fuzzier. In the realm of social science, particularly education and policy research, it isn’t always clear to a policy maker, practitioner, or parent what constitutes good evidence, especially when experts disagree.

Does that mean that researchers in the social sciences and education don’t think they have accumulated important evidence? No. So, from the perspective of those who recognize the value of accumulated knowledge and research evidence, it’s confusing why those in education policy and practice don’t appear to listen to researchers or fail to use what is considered the “best evidence” to date on a particular topic. When he realized that most research doesn’t impact policy or practice, educational psychologist David Berliner lamented: “Once upon a time, early in my career, when the world seemed quite a bit simpler than it really is, I believed that my research, and that done by my fellow educational psychologists, would influence what happens in America’s classrooms and in teacher education. I believed in the model of research that famous researchers often espoused.”² And that’s often the belief many graduate students from the social sciences initially hold, and that many distinguished scholars in their specific subfields still hold. Why?

Education is filled with fads and myths. Hot topics such as learning styles,³ multiple intelligences,⁴ grit,⁵ and mindset⁶ have, at best, only weak support, even though they continue to be trumpeted by the media and have become a part of the popular conversation. Though these are well-recognized examples, the history of education shows^{7, 8} that they are by no means exceptions.

The replication crisis in which many published research results have proven difficult or even impossible to reproduce, has sent shock waves across all areas of science,⁹ especially social science,¹⁰ including the oftentimes policy-influential domain of economics.¹¹ A paper published in Science that estimated the reproducibility of psychological science research was downloaded over 40,000 times and covered in over 231 news outlets,¹² for example, “Over half of psychology studies fail reproducibility test” (Nature¹³) and “Psychology’s replication crisis is running out of excuses” (The Atlantic¹⁴). Not only are social scientists themselves justifiably skeptical whether some seemingly established findings will stand the test of time, but the broader public has become cynical regarding the value of expert opinion in general.

Within the social sciences, different fields have different theoretical, empirical, and tool-based approaches they employ based on their niche-specific promotion incentive structures (pay, promotion, awards, recognition), typically linked to publishing in particular field-valued journals. Generally, the more publications you have in the more prestigious journals, the greater your chance of receiving pay raises, promotions, prizes, and other perks. Since this translates into the need to write for the handful of peers in one’s field, the disciplines are largely siloed, i.e., publications and information get stacked up in specialist journals, encased in technical language, equations, and symbols. Only rarely, and at risk, do scholars dare build on the work of those outside their own discipline, or in some cases even within them. The unfortunate reality is that the use of research-based evidence in formulating education policy is quite limited because politics and personal values dominate. For example:

Ron Haskins, a respected former Republican committee staffer in Congress and now a Brookings Institution scholar, was asked several years ago about the role research played in what was, at the time, a contentious congressional debate about welfare reform. Without missing a beat, he responded that, based on his personal experience, the best research might exert five percent of the total influence on the policy debate, with an upside potential of 10 percent. Personal values and political power, Haskins went on to say to his now silent and disappointed audience, were what really mattered in Congress.¹⁵

Why Research Carries Little Weight in Policymaking

Policymakers^{16, 17} have explained that research use is not really linear in the way that most researchers hope.¹⁸ On the playing field of hardball politics, research is more often used to: (a) support and justify a favored, pre-existing ideological, and long-held point of view, or (b) help inform the thinking around a decision-making process in a way that is quite specific, context-dependent, and disconnected from the findings in a journal article. Simply stated, research results usually just sit on the bench during the policy-making process.

Moreover, the rigor of the methods employed is rarely the primary concern of those using the research. In making policy, what counts is whether a given piece of research provides support for a predetermined decision, in a particular on-the-ground context. Bill Knudsen, former Deputy Assistant Secretary in the U.S. Department of Education, noted in a personal communication that, based on what he saw in working with legislators, perhaps at most 10 percent of decision making in education policy is evidence-based, and the definition of what is considered as evidence is quite loose, with little distinction made between mere qualitative evidence and the ascending levels of scientific rigor such as Randomized Controlled Trials (RCT) evidence and above. (See Figure 1.) Of course, the unfortunate fact that evidence often fails to impact practice is also true in health care¹⁹ and numerous other fields: “Yet even today, health care experts maintain that 80 percent to 90 percent of daily medical practice is not anchored in such evidence because the specific, detailed information practitioners need still does not exist.”²⁰

Figure 1. Research Design & Evidence Chart, redrawn based on a chart by CFCF [CC BY-SA 4.0] (See https://en.wikipedia.org/wiki/Evidence-based_education)

One reason for this lacuna is that in U.S. education policy a small set of individuals, often dominated by education economists or graduates from certain types of education policy or reform programs, tend to cite each other while ignoring a lot of the broader social science evidence that has important bearing on particular topics.²¹ And this is probably not intentional. When you are trained to think in a certain way and exposed largely to many others who also think that same way and value similar research methods and approaches, groupthink tends to take hold. While this problem is inherent to all academic disciplines, not just education policy, some are better than others at being truly multidisciplinary.

Academics who produce research evidence across social sciences and in education believe their subfield has much to offer those in education policy and practice. So they often feel frustrated that decision makers don’t usually read their publications. And the public doesn’t read research publications either: A.K. Biswas and Julian Kircher, who measured the impact of academic conferences and publications on real-world practice, noted, “Practitioners very rarely read articles published in peer-reviewed journals. We know of no senior policymaker or senior business leader who ever read regularly any peer-reviewed papers in well recognized journals like Nature, Science or Lancet.”²²

The history of education reform shows that most efforts have not proven successful.⁷ This is largely because top-down education reform efforts tend to evaporate at the point of impact, namely, the classroom.⁸ This is true even for efforts such as common core, which enjoyed wide bipartisan support.²³ The disconnect between research and policy/practice is the rule, not the exception. Nor is such lack of success confined to education. Reform efforts in criminal justice and welfare policy often have gone awry.¹⁵ The education research and policy community often avoid discussing failure,²⁵ perhaps in part because many look to education as the solution to those, and, increasingly, most perceived problems in society. Thus, though educational research has accumulated and, in some ways, has become more rigorous, the disconnect between research on the one hand and policy and practice on the other has remained quite consistent over time.² Realistic policy scholars argue that this doesn’t mean policy reform should be abandoned, but only that incremental change is more likely to prove effective than any quick-fix “silver bullets.”²⁴

Solutions

Some simply accept the verdict of history—the disconnect between research and practice is to be expected as the default condition in any field. This is especially true given that experts in education policy are unclear as to what a genuine solution might look like. Nonetheless, there continue to be important efforts to join the two. Both history and common sense suggest that gradual steps, monitored, measured, and revised, hold greater promise than one massive attempt to bridge the chasm. Here are some suggestions.

Improve the quality of evidence.

A necessary first step is to improve the research evidence base in education policy. Sadly, replications in the field of education are not standard at present. When researchers looked at the top 100 education journals they found that only 0.13 percent of education articles were replications.²⁶ And though economists are highly influential in education policy as a research field, as are political scientists and to some extent sociologists, the research of psychologists and other social and behavioral scientists is noticeable only by their scarcity. Economic thinking and approaches are influential in all areas of policy, including in education,²⁷ though they provide only one of the toolsets available for researching social science, education, and policy issues.²⁸

Additionally, a small group of education policy scholars and influencers in think tanks dominate and serve as gatekeepers determining which ideas gain entry to shape the research, which topics are discussed, and which never receive a fair hearing.²¹ A truly multidisciplinary approach to integrating evidence from every possible discipline relevant to education research and policy would be another important step in improving the evidence base in education. New research is not always necessary—just integrating research evidence that has accumulated in as yet unincorporated fields would be productive.

Multidisciplinarity, of course, faces the incentive structure constraints in academia that arise from the silo effect described earlier, but hopefully that too can change incrementally over time.²⁹ One way would be to first get the broader public, especially policymakers and practitioners, to read research.²² Even when the most rigorous relevant research is collected in an educational repository such as What Works Clearinghouse, policymakers and practitioners often don’t take the time to read it. They have different incentives and interests than researchers, who are trained and then rewarded in designing experiments and evaluating scientific evidence.³⁰

Engage the public.

Academics should publicly engage and teach scholars in other fields as well as the broader public about their research, whether through writing for the news and in magazines, going on podcasts and doing interviews, writing popular books and articles, using social media, and other methods.³¹ Some go so far as advocating giving research away to the public by making clear accounts of research methods freely available on the Internet, along with the data and results so that they can be replicated easily. However, this must be done responsibly, given that the replication crisis has made it unclear in some areas whether the cumulative evidence is strong enough to communicate or be useful in policy and practice.³² The challenge here is to explain just how research results published in an academic journal are actually relevant to the average person in their everyday life.

Build better relationships.

If they want to influence policy, researchers should get to know state-level policymakers and form mutually beneficial relationships with them. Policy making at the national level is usually out of reach, so there is a greater chance at the state or more local level.¹⁵ Doing so, however, is a two-way street. Building relationships with state or local-level policymakers and politicians requires learning how the political process works and being available to help solve real, on-the-ground problems within short time frames.

Communicate in plain language.

Academics also need to be able to communicate their findings in plain language so that those outside academia—or even their particular discipline—can understand and use that knowledge.³² History professor Patricia Limerick³³ made this case forcefully in her poignant article titled, “Dancing with Professors: The Trouble with Academic Prose.” Other scholars argue: “If academics want to have an impact on policymakers and practitioners, they must consider popular media, which has been ignored by them.”²²

New research is not always necessary—just integrating research evidence that has accumulated in as yet unincorporated fields would be productive.

The technical jargon used in each academic subfield often prevents integrating knowledge across subfields and hinders those outside of academia from using relevant research findings. Writing policy briefs and other publications in plain language is not incentivized by traditional academic positions where a Darwinian calculus rules in the form of “publish and get grants or perish.”³⁴ However, it is precisely the mass media that can transmit knowledge so that research findings could better find their way into policy decision-making. Learning why, when, and how to enter the public arena should be integrated into graduate training programs across the physical, biological, and behavioral sciences if researchers have any hope of impacting policy and practice.

Publishing for the general public needs to be incentivized.

An additional challenge is that academia rewards producing research that those in policy and practice just don’t think addresses their needs.¹⁶ One solution would be for academia to reward scholars in the tenure and promotion process for communicating and publishing the relevance of their research findings to a broader audience. Doing so could be included as part of the service component of the usual research-teaching-service pay-and-promotion criteria.

Work with practitioners in research-practice partnerships.

Some education researchers are embracing Research Practice Partnerships (RPPs),^{16, 35} for example, between researchers at a university and practitioners in local schools. (Full disclosure: I’m involved in an RPP in Northwest Arkansas,³⁶ and it is well worth it). RPPs work because practitioners are a part of the research process so that their research needs are met. While academic research questions are often disconnected from practice, there are some cases where not only the answers but also properly framed questions can be useful to practitioners. And for some questions, the results are publishable in an academic journal and so do reward the researchers.

Since RPPs are a partnership, the typical independence of the researcher for purposes of evaluation is not present and a clear conflict of interest could arise. However, because policy changes often are constrained by many moving parts, when the time comes for implementation, gradual yet positive change can move the needle provided that the practitioners understand why the research is useful and so are eager to use it to help kids in their schools.

Top-down solutions often won’t work.

Eric Kalenze,³⁷ a leading authority in the field of curriculum and content development, argues that top-down reform efforts don’t work largely because there isn’t either the infrastructure or buy-in from schools that are necessary to make it happen. He explains how in the school where he taught, there was a period when a supportive principal and a group of dedicated teachers could truly make effective education reform work, but the confluence of these positive factors is hard to scale. Despite this, he argues that bottom-up efforts are worth pursuing to help kids, including efforts that build on evidence use.³⁰

Involve the teachers.

Educational psychologist David Berliner argues:

It is the tinkering by teachers and researchers, and the study of their craft by the teachers themselves, that seems to me the most likely to pay off in improved education. If those in the research community can learn to do more design experiments in real-world settings, and join teacher-researchers to produce knowledge about how things work in real-world classrooms, the great disconnect might become a much smaller disconnect. Educational research would end up being less a field of traditional scientific research, and much more a field of engineering, invention, and design.³⁸

This perspective aligns with the focus on RPPs through getting teacher buy-ins and developing mutual respect to bridge the disconnect.

* * *

This article appeared in Skeptic magazine 28.3
Buy print edition
Buy digital edition
Subscribe to print edition
Subscribe to digital edition
Download our app

In their book Gradual: The Case for Incremental Change in a Radical Age, authors Greg Berman and Aubrey Fox explain that there have been successful policy reform efforts, such as Social Security, where the confluence of a large number of positive changes fortuitously came together.²⁴ Perhaps by approaching all of these possible solutions while thinking of novel ways to end the separation between research and policy/ practice can lead to gradual but positive change. Avoiding unintended consequences will require a clear-headed understanding of all of the relevant research and facts that influence education.³⁹

However, given the current climate of political polarization and the culture wars in education policy, the use of evidence in policy and practice remains an ongoing challenge, but one that might be overcome gradually, if more people better understood both the history of failures and the fact that radical changes are much less likely than positive small ones to help educate children — whether inside or outside of schools.⁴⁰