Are Survey Data Still Useful? Yes, But…
By Ben Harris
On April 15, 2019, Results for America and AEI hosted a provocative discussion of whether survey data are still useful to inform public policy. Results for America Chief Economist and Senior Adviser Ben Harris, who served as Chief Economist and Economic Adviser to former Vice President Joe Biden, gave the keynote address. Video of the address and a lightly edited version of his remarks are below:
I’m excited to be here at this event co-hosted by AEI and Results for America — discussing the critically important topic of survey data. At Results for America, we see this data as critical to helping policymakers at all levels of government — federal, state and local — make the best decisions and invest in the solutions that will have the greatest impact in improving the lives of young people, families and their communities.
Unfortunately, I suspect that many people see the topic of survey data as an esoteric issue. That questions of data quality and availability are topics best left to economists and statisticians and other academics who spend most of their time in the weeds of empirical study. Perhaps because so many see this as an issue best discussed in the basement of government agencies or in university conference rooms, it seems clear that the role of survey data is greatly underappreciated.
But, as I suspect most people in this room know, survey data underpins so much of what we know about public policy and the economy. Certainly no one would argue that the unemployment rate is an esoteric topic. Or retirement preparedness. Or business confidence. Or income equality. Or poverty or health insurance coverage or household income. And yet in the absence of survey data, we would have a much more limited understanding of these topics and others.
When it comes to program evaluation, survey data continues to be a paramount in our understanding of the impact of government programs. It’s difficult to imagine, for example, how little we would know about so many public programs without the Current Population Survey, the National Health Interview Survey, the Survey of Consumer Finances, and so many others. And more recently, researchers are just starting to grasp the magnificent power of linking survey and administrative data.
My hope is that today, we can all stipulate the vital importance of quality, widely available survey data. And if we all agree that survey data is one of the most important tools to understanding our world, then we can use this time to discuss some of the most pressing challenges facing survey data more broadly. And, to be sure, there are substantial and widespread challenges facing survey data. I’ll lay out four.
First, many policymakers remain unconvinced that the investment in survey data is worth the cost. I worry deeply that the profound importance of survey data to understanding programmatic impacts is lost on those making budget decisions. Results for America has long advocated for a 1 percent set-aside for evaluation, making the case that it’s worth investing $1 to determine if the other $99 are being spent wisely. And underlying this call is the assumption that available and quality data are a necessary complement to this evaluation — we simply can’t know if programs are working if we don’t have the data. And often times it is survey data we rely on.
The recent shutdown highlighted how invisible data can be to policymakers. During the longest government shutdown in history, policymakers were rightfully worried about government workers getting paid, public facilities being reopened, and public services — ranging from food safety inspections to airport security — staying on track. But it seemed, as least from the sidelines, that limited attention was paid to what PEW calls “the data casualties” of a government shutdown — including everything from new home sales data to BEA’s estimate of 4th quarter GDP.
My second concern is that the quality of survey data appears to be on the decline. As today’s panelist Bruce Meyer and his co-authors argue convincingly in a 2015 article in the Journal of Economic Perspectives, survey data are in peril: they find that nonresponse rates have been rising for decades while measurement error appears to be on the rise — leading to marked concerns about systematic bias in transfer program data. Meyer and his coauthors argue that this bias has “profound implications” for our understanding on how government programs impact low-income families, and this this bias is likely not limited to transfer programs.
Somewhat depressingly is that these challenges, ranging from respondent saturation with surveys to stigma over particular answers, have been around for years without a suitable solution — and many of the underlying problems appear to be getting worse.
My third concern is that we have yet to fully appreciate the power of linking survey and administrative data. As Robert Doar and Linda Gibbs wrote in a 2017 RFA paper, linked data can allow researchers to glean remarkable insights that would never be possible with using administrative or survey data alone. And the promise of this approach has only strengthened with the increased use of administrative data — Raj Chetty found that the share of articles published in economics journals using administrative data rose from around 20 percent to 60 percent between 1980 and 2010. But linked survey and administrative data is probably on the rise as well. To take one small sample example, a brief by Steffen Kunn found that only an average of one IZA working paper per year utilized linked data between 2001 and 2008, but that this grew to an average of six papers per year over 2009 to 2014.
Still, linked administrative and survey data still faces formidable obstacles, not the least of which is privacy concerns. Solving these challenges and making linked data commonplace is one of the most promising avenues towards better understanding of the efficacy of government programs.
Fourth, there is not a clear understanding of how the rapid technological change that has given rise to big data and AI interacts with survey data. Admittedly, I have yet to wrap my head around this concern, but it does seem as though the forces that have propelled big data to the forefront of private-sector innovation have largely escaped our approach to survey data. It seems, and this is just an impression, that we have become more adept about learning about people when they least expect it, but have gotten worse about learning about people when they willingly share information. The unfortunate byproduct is a cycle by which people become more guarded with this information, which only leads to private companies becoming even better at collecting data — and people guarding their information even more closely. The end result could not only be the death of survey data, but also data which is voluntarily and consciously shared — leading to a blurred line between research and spying.
Of course, it doesn’t have to be this way. Technology should make collecting data easier, not harder, and it can potentially help protect privacy, rather than destroy it.
I’ll conclude with this. I see enormous potential in the rise of evidence-based policy. It feels obvious, natural, and inevitable that the American public and policymakers across the county would like to know whether the trillions spent on government programs are achieving their desired objectives.
It is encouraging to see the progress that’s been made just in the last few months. With the signing of the bipartisan Foundations for Evidence-Based Policymaking Act, also known as the “Evidence Act,” earlier this year, federal agencies will be designating chief evaluation officers and chief data officers, and establishing learning agendas to help accelerate their generation and use of evidence and data. The new law also contains other key recommendations by the Commission on Evidence-Based Policymaking to help increase the secure access to data and to strengthen privacy protections.
The bottom line is that there is no evidence-based policy without evidence, and there is no evidence without data. I firmly believe that survey data will always have a central place in academia and policymaking, but the degree to which depends on how we solve the very real challenges facing survey data today.
