Emphasizing Evidence to Support Youth in Texas’ Workforce System

By Laney Umland

The Texas Workforce Commission (TWC) recently approved the inclusion of a new Evidence Framework in training grants, focused on young workers interested in building and construction trades. The Framework will prioritize evidence of effectiveness when awarding the $1.5 million in training grants for the 2020 Building and Construction Trades program.

The Evidence Framework was created in collaboration with TWC, the Office of the Governor, Texas Workforce Investment Council (TWIC) and Rural Capital Area Workforce Board over this past year with support from Results for America (RFA). Defining and prioritizing evidence of effectiveness will target public workforce development funding towards more impactful programs while simultaneously building the evidence base and capacity to develop rigorous evidence within Texas’ workforce system. The Framework will accelerate the discovery of innovative approaches and replicable models that lead to better outcomes for workers and employers. See the full evidence definitions below.

As part of RFA’s State and Local Workforce Fellowship, Courtney Arbour, Director, Workforce Division with Texas Workforce Commission; Jennifer Troke, Director of Grants and Contracts with Texas Workforce Commission; Lee Rector, Director of Texas Workforce Investment Council; Paul Fletcher, Chief Executive Officer of Workforce Solutions Rural Capital Area; and Diane Tackett, Chief Operating Officer of Workforce Solutions Rural Area have been working together to identify and implement priority strategies from RFA’s 7 Ways to Improve Workforce Outcomes Using Evidence: 2019 Policy Roadmap for State & Local Officials. Along with seven other state teams in the RFA fellowship cohort, the Texas Team members are honing their data and evidence skills and implementing policies and practices that will strengthen TX’s public workforce system and improve the lives of the people they serve.

The new Evidence Framework includes following evidence tiers:

High evidence programs are ones that are supported by rigorous evaluations of the program or of an essentially similar program design and outcomes. The program or essentially similar program must have:

• conducted two well designed and well implemented Randomized Controlled Trial or Interrupted Time Series studies that include both a comparison group and a statistically valid technique to assess causation that eliminates or minimizes confounding factors. The study must have had minimal attrition. This study must show that the program has both a positive and meaningful outcome, and that there is a high degree of confidence that the outcome is primarily caused by the program.

Moderate evidence programs are ones that are supported by rigorous evaluations of the program or of an essentially similar program design and outcomes. The program or essentially similar program must have:

• conducted at least one study that includes both a comparison group and a statistically valid technique to assess causation that eliminates or minimizes confounding factors. This study must show that the program has both a positive and meaningful outcome, and that there is a modest degree of confidence that the outcome is primarily caused by the program.

Performance Programs offer outputs and outcomes data and information as evidence conduct assessments of participants to demonstrate effectiveness of their programs, and at least one post program follow up to track the outcomes of participants. Primary support for these programs’ effectiveness is provided through historical data showing that the program creates an intended change in participants, and that participants show a positive outcome following participation in the program. To be reviewed on the basis of previously implemented programs, an applicant must have historical output and outcome data for at least two years, either directly collected or from the similar program being used as evidence. In addition to past output and outcome data, an evaluation performed by an external entity on a program with a very similar design and anticipated outcomes may also be used to support this evidence tier.

Experience programs do not perform evaluations of participant success in the program, do not collect performance data or follow up with participants, or evaluate the effect of the program on participants (though they may do a satisfaction survey of participants). Primary support for these programs’ effectiveness is provided through anecdotal participant success stories or other testimonials. Experience programs must have been providing services for at least one year prior to grant application.

New programs are entirely new and are not similar to an existing program. New programs have no evidence of effectiveness and have not been evaluated. An applicant must explain why the proposed program will achieve the outcomes specified in the main body of the application and demonstrate that there is capacity to collect sufficient data to track outcomes from the program.

Please contact Celeste at celeste@results4america.org if you’d like to learn more about Results for America’s State and Local Workforce Fellowship and about how your state or local workforce board/agency can adopt similar evidence definitions.

Laney Umland is an Associate for Education and Workforce Development at Results for America.