Making Federal Research Meet the Needs of the American People

By Melissa Patsalides and Barry Steffen

USAID leaders receiving feedback on the Feed the Future Learning Agenda

Every organization must learn to improve, or fall into disrepair. Over the past few decades, federal agencies have used multiple strategies to guide improvement including evaluation, performance management, and data analysis. Today, spurred by new legal requirements, they are adding a new tool to their tool-box: learning agendas.

The Evidence Act, a bipartisan law passed in 2019, requires federal agencies to create learning agendas. Put simply, a learning agenda is a set of questions that, when answered, improve the effectiveness of an agency’s programs. One important element of the Evidence Act is that it also requires agencies to consult with stakeholders when formulating their learning plans. This helps make sure that agencies’ research is focused on issues that are important to their constituents. Or, in the words of the White House Office of Management and Budget, the purpose of such engagement is to make sure that the learning agenda addresses questions that are “relevant, salient, and meaningful” to those people served and impacted by the agency.

Some agencies already have a head start on this work and have been turning to the public and internal stakeholders to help them improve and stay relevant. This work can inform other agencies just getting started.

The U.S. Agency for International Development (USAID) has utilized multiple learning agendas across various units within the agency since 2011. In 2017, USAID conducted a landscape analysis to determine how those plans were working and how to improve them. In terms of engagement, USAID made some important discoveries:

• The most common format for engagement was in-person meetings and virtual meetings or webinars (e.g., conferences, workshops, and individual meetings or consultations).
• The level of engagement varied based on audience, need, and resources (e.g., staffing or time allocation).
• A participatory process was crucial to engaging external stakeholders, while consistent, regular communication was key for internal agency staff.

USAID also identified different types of engagement and how they contributed to a learning agenda (illustrated below). An agency that just is just beginning its stakeholder engagement work could create a similar type of map by 1) identifying engagement types, 2) identifying key stakeholders and their roles, 3) assigning tasks and timelines to each category, and 4) distributing resources accordingly.

When developing its current agency-wide learning agenda, USAID took significant steps to increase stakeholder engagement. Learning questions were gathered from across the agency to capture questions that were utilization-focused and important to staff in Washington as well as the field. The final selection of learning questions was informed by consultations with Washington operating units, 27 USAID Missions, and Agency leadership. An initial list of 260 questions was winnowed and refined to the final 13 questions.

In sum, USAID did not take these steps to increase stakeholder engagement simply for the sake of doing so, rather it was because the agency was developing a cross-agency learning agenda that would cover a broad swath of its work. Therefore, the more intensive stakeholder engagement was imperative. Today USAID’s internal and external stakeholder engagement continues through periodically revisiting the questions, undertaking learning activities to generate evidence for the questions, and to disseminate and complete action planning around the findings.

The U.S. Department of Housing and Urban Development (HUD) has also been a longtime leader in external stakeholder engagement. In 2013 HUD created its first Research Roadmap, the agency’s version of a learning agenda. The Roadmap used extensive stakeholder engagement to identify critical, policy-relevant research questions and guide HUD research investments. The Roadmap was updated in 2017 and is being updated again for 2020–2024 with enhancements for the Evidence Act.

HUD officials gather for Stakeholder HUDdle

HUD views the Roadmap as a continuous and iterative effort to keep up with evolving challenges. HUD’s participatory process engages internal and external stakeholders including program partners in state and local governments, the private sector, researchers, academics, policy officials, and the general public (for additional details see pages 14–16 of Roadmap Update 2017).

As illustrated in the chart above, stakeholders are engaged at each of the four stages of formulating the HUD learning agenda, including:

1. Gather research questions: External stakeholders identify research questions through email, web forums, conferences and webcasts, and targeted meetings. Internal staff participate in listening sessions. Roadmap coordinators record suggestions and catalogue them (without individual attribution) by the session or medium in which they were received.
2. Organize questions: Coordinators compile a database of the suggestions from stakeholders, categorizing each suggestion as a research question, project idea, data need, or other type of comment. Suggestions are divided into topical focus areas to help HUD leadership prioritize the questions.
3. Prioritize questions: Subject matter experts review research questions and assign a priority rating of 0 to 3 to each suggestion. A heat mapping process is used to elevate the top tier of questions for review and discussion by HUD’s leadership and management team.
4. Develop research proposals: HUD research staff complete brief project proposals for each prioritized research question, which become the core of the Research Roadmap.

Over time, HUD has adapted its stakeholder engagement strategies to solicit potential research questions in a variety of ways from different stakeholders. For its 2014 Research Roadmap, HUD held several conferences with researchers and stakeholders, hosted a web forum, and hosted numerous listening sessions with internal staff, other federal agencies, and industry partners. This multi-mode approach garnered nearly 1,000 comments on the learning agenda, but also demanded a high level of dedicated resources for administration and staffing.

Specifically, the researchers’ conferences produced many ideas, but were relatively costly and required much staff time to define the topical agendas. A web forum was relatively easy to set up and administer but yielded fewer useful comments, seemingly because it was not visible or salient to highly-informed stakeholders. Listening sessions in general — and internal listening sessions in particular — tended to yield the most useful suggestions because session participants are motivated, have shared programmatic interests and expertise, and can be nudged by facilitators to identify research gaps that underlie their more pragmatic or aspirational concerns. Internal listening sessions also posed significantly lower administrative burden on learning agenda coordinators.

Applying these lessons, HUD did not host a researchers’ conference for its 2017 Research Roadmap but added email outreach, which yielded a medium level of comments for relatively little effort. To engage stakeholders for its 2020 Roadmap, HUD added a shorter general audience conference at HUD headquarters to kick off the outreach phase and stimulate more-focused email submissions. A panel of invited experts at the conference was asked to share their thoughts about research gaps to stimulate ideas and increase the salience of the email campaign. This general conference required less resources than researchers’ conferences and was broadly available to a diverse group of stakeholders through the webcast.

Another outreach innovation for HUD’s most recent learning agenda, intended to create a more dynamic and creative atmosphere, was to host a session in which subject matter experts staffed a table for each topical focus area and recorded research questions and data needs on flip charts from participants that visited their table. The success of this approach supports the idea that more interactive, technology-supported events in which participants can react to new inputs in real time could be a less costly and more productive way to engage external stakeholders.

Reflecting the idea that learning should be continuous, HUD’s stakeholder engagement does not end with the publication of the Roadmap. The electronic mailbox for research suggestions remains open. HUD also holds quarterly research briefings that include expert panels and are webcast to maintain dialogue about major research findings and evidence-building priorities. This frequent and consistent engagement is designed to keep the Roadmap more relevant, and to provide a stable venue for ongoing participation.

Based on the leading work of USAID and HUD, several lessons can be drawn to inform the work of other agencies meeting the requirements of the Evidence Act for stakeholder engagement in learning agendas:

• Stakeholder engagement is an ongoing, integral process not a standalone effort.
• Use engagement to create a culture of learning and growth. Ask stakeholders to help the agency figure out what is working and not working, and include a “no surprises” clause to ensure agency researchers talk to stakeholders before releasing findings.
• Engage important internal stakeholders (e.g., senior leadership, program offices, regional and field offices) as well as external ones (e.g., grantees, T.A. providers, researchers, policymakers).
• Categorize stakeholders and engagement strategies to maximize limited resources. Stakeholders could fall into those an agency wants to inform, to consult, to participate/involve, or to actively collaborate. Agencies can also link capacity/resources to each category.
• Map stakeholders, find out where they convene, and go to them (when possible) for input and help with dissemination.
• Provide clarity about role and outcome of engagement to stakeholders.
• Dynamic and interactive modes of engagement can be valuable to increase participation and improve usefulness of suggestions.
• Make equity a priority; think about fairness and representativeness, who may be missing or unable to engage, and whether agency work has a disproportionate impact on certain populations that need to be engaged.
• Engage stakeholders about dissemination and use of research, not just for generating potential research questions.

Learning and improvement require sophisticated ongoing efforts, and learning agendas are one key effort. As agencies grapple with how to engage stakeholders in the learning agenda process, they can take heart that others have already learned a lot about learning, starting with smart ways to engage stakeholders.

Melissa Patsalides is the Agency Evaluation Officer at the U.S. Agency for International Development. Barry Steffen is a Social Science Analyst at the U.S. Department of Housing and Urban Development. The statements of the authors do not represent the views of the U.S. Government.