Strengthening how governments define and prioritize evidence

By Sara Kerr, Patrick Carter, Pete Bernardy and Maia Jachimowicz

Last week, Results for America released a new paper spotlighting the progress the federal government has made over the past decade — under both Republican and Democratic administrations — to build and use evidence of effectiveness.

During an event releasing the paper and celebrating the fifth anniversary of the Evidence Act, we heard from policymakers and government officials about how evidence has become an integral part of federal decision making processes, and is helping leaders make more informed decisions. We also heard what needs to happen over the next 10 years at all levels of government to realize the full potential of evidence-based policymaking. Robert Gordon, Deputy Director for Economic Mobility at the White House Domestic Policy Council, said this:

“I think the challenge we have now is that we’re at a point where some of this work is built in and it does run the risk of becoming another box that needs to get checked and that isn’t what it should be. It should be a tool for change, a tool for improvement. And I think for that to happen, both researchers and policymakers always need to be pushing themselves — what are the most important questions right now and how can evidence help us answer them.”

The importance of continual improvement is something we’ve been thinking about in regards to how we define evidence. Results for America has long supported leaders at all levels of government to define and prioritize evidence in their budgets, grants and more. We think clearly defining evidence is a crucial first step in embedding it in decision making. And we’ve seen great results with this approach. When the U.S. Department of Education defined evidence in the Every Student Succeeds Act (ESSA) in 2016, state investments in programs that delivered results for students increased. In one school in Nevada, this meant double the number of third graders reading proficiently in a single year.

But in gathering feedback over the last two years, we’ve heard that the current definitions of evidence could do more to address today’s important questions and challenges. Through conversations with nearly 100 policymakers, community advocates and experienced leaders from a broad range of backgrounds, we’ve identified ways for our definitions of evidence to better support government leaders to learn, improve and invest in what works.

With the goal of building on the strong foundation of evidence definitions already in use, we are excited to release new definitions of “evidence-based” and “evidence-building” programs that we believe will help pave the way for better results for all. The new definitions are designed to continue to steer public funding to programs and policies with a track record of success while also encouraging leaders to:

• Consider the full body of evidence for programs, including all relevant studies about results or implementation;
• Elevate information about why, how and for whom programs have worked in the past as they consider future investments;
• Examine how local context may affect implementation and results, including through feedback from relevant stakeholders; and
• Incentivize ongoing evaluation and improvement.

Notably, the new definitions:

• Recognize that relevant, credible knowledge is generated by multiple methods. The definitions clarify that there are different forms of evidence — impact evidence and implementation evidence — which help decision-makers understand both whether a program has been effective and how it can be implemented well. This is important because it recognizes that all rigorous research methodologies, including quantitative, qualitative, and mixed methods approaches, can help inform government decisions in different ways, as long as they are well-designed and well-implemented.
• Emphasize the importance of matching the type of evidence with the specific question being asked. The definitions are designed to encourage policymakers to consider which type of evidence is best for their particular use case and context — including the populations being served — and match this evidence to the specific issue or question they are seeking to address. This includes considering capacity to implement and the relationship between the proposed program and existing investments.
• Promote continuous learning and improvement. The definitions include “evidence-building” programs so that policymakers can design incentives and preferences for programs that have an informed rationale, but are still undergoing evaluation or, with the right support, could be evaluated. Results for America strongly believes that governments should invest in testing innovative and evidence-based programs and build evidence about their effectiveness. For example, in a policy area with weak or no evidence, government leaders may choose to conduct an implementation evaluation to inform improvements and ensure a program is being implemented well before conducting an impact evaluation to build new knowledge about whether the program produces a positive impact.
• Require there to be shared logic and supporting information for why a program will work in the proposed context. These two pieces are crucial to building confidence that a program is likely to work. If the intervention shows positive effects in one evaluation, it does not necessarily mean it will work in a different setting unless there is a theory — informed by research, participant experiences and diverse stakeholder perspective — for why it will work in similar settings with similar populations.
• Are designed to be “shovel-ready” for use in a variety of ways to help leaders advance evidence-based policymaking across the government. For example, we provide examples for how these definitions can be used to simultaneously prioritize programs with impact evidence in budget decisions, highlight implementation evidence for service providers who will be implementing the funded programs, and award bonus points in grant applications for programs that have implementation evidence and are currently undergoing an impact evaluation, or are willing to commit to one as a condition of funding.

Investments in evidence-based programs are key to increasing the likelihood that a government will deliver positive results for residents but, on their own, are insufficient. Policymakers must also take care to ensure their decisions are designed to meet their particular community’s needs and that there are sufficient resources to invest in quality program implementation and ongoing evaluation for continuous improvement. This approach invites governments to select and invest in evidence-based programs in a new, more nuanced, and more participatory way, taking care to both attend to prior research and to the voices of those closest to the challenges they seek to address.

We encourage you to explore these definitions and consider the many examples of ways results-driven governments can define and prioritize evidence to achieve better results for all. If you would like to learn more about how you can use these definitions in your jurisdiction, please contact our team.