Web Policy  |  Intranet  |  Contact SCHEV    
SCHEV Research Data Blog

The official blog of @SCHEVResearch at the State Council of Higher Education for Virginia. Discussions about our work, national higher education data policy, and highlights about the data we publish.


Recent Posts Blog Archives Subscribe Feed All Blogs

Survey results that probably should be ignored

by Tod Massa 31. August 2018 22:52

This article at Inside HigherEd on the recent MLA survey on Language Ph.D.s: A Jobs Snapshot is really the kind of article that shouldn't be written. The survey simply does not provide enough responses to be meaningful. It covers PhD recipients that were contacted in a 2015 study. There were 311 responses out 1,946 contacts, a 15% response rate. The respondents represented PhD completions 1996 to 2015, with about 75% representing the years 1996 to 2005, 21% between 2006 and 2015, and the rest with unknown years. With distributions like these and such a small number of responses, it strikes me as difficult to say much of anything about the results.

While some might take heart from the finding that 68% of the respondents held a tenure-track position somewhere at any point in their career, knowing that almost a third did not raises questions. Are the responses skewed by those who were in the lucky two-thirds, i.e. the ones likely most easily contacted? The report does include comparisons to a 2013 survey with 2,198 respondents, which shows that about 48% were in tenured or tenure-track positions, so perhaps this is not far off.

I think the challenges researchers face today in conducting good survey research are just increasing. As SCHEV continues its study of the possibility of conducting a major post-college outcomes survey, we are hearing a lot about the difficulty of achieving good response rates and the cost of doing so. The US Census is struggling with the same issues and thus looking to opportunities to connect to administrative data systems, such as they did with the University of Texas System, which was not just about matching graduate data to wage data for the System's benefit, but to evaluate the possible added value to Census to reduce it's reliance on surveys.

Basically, we are all trying to find more and better data. And cheaper data.



blog comments powered by Disqus
Follow SCHEV Research on Twitter