Web Policy  |  Intranet  |  Contact SCHEV    
SCHEV Research Data Blog

The official blog of @SCHEVResearch at the State Council of Higher Education for Virginia. Discussions about our work, national higher education data policy, and highlights about the data we publish.


Recent Posts Blog Archives Subscribe Feed All Blogs

Trust, Big Data, and Bad Timing

by Tod Massa 5. December 2014 20:08

I read this story before Thanksgiving. A principal in New York finds that the state's education department (NYSED) issued reports on high school graduates attending college seems to be terribly flawed. Principal Burris was not shy about letting people know. The report is based on data from the NYSED student records matched to data from the National Student Clearinghouse. Read the original essay and then associated linked article and see what conclusion you reach.

This evening, a Twitter user @EduBenM rwtweeted @klarkcollege tweet of the the story:

I retweeted this just to draw attention to how people respond to these stories and because it is of interest. This drew a responding tweet from Richard Torres of NSC to read the comments to the article as the fault does not belong to NSC. Rick and I exchanged tweets as I pointed out that the comment from NSC was just an assertion and did not explain anything, and he then directed me to the response from the principal, which I still thought did not explain things. At least to my satisfaction. This tweet, however, does explain things:

Now, I understand especially in the context of a prior tweet:

Hah! Now I understand. Apparently the NYSED is only looking at six months post-graduation while the high schools are looking a year later. A basic problem of definitions, one that we struggle with all the time. At SCHEV we try to be clear about the definitions we use because we know it can cause confusion on campus. We are not always successful. There are also times when institutions have long-standing definitions and metrics that conflict with ours and with national standards. I encountered one of those last summer on Twitter in following one of public university presidents.

Now, I am not a big fan of NSC. I do think there are problems with their data. These are primarily a result from being flexible in collecting data from thousands of institutions, some of which appear to make no effort to think about using standard codings or abbreviations. However, NSC does know who its participating institutions and does a lot of work to understand the limits and possibilities of the data it has. I think it is important that we discuss this issue as a community and bring some awareness to the story. VDOE uses data from NSC for its federal reporting. SCHEV uses it to enhance the knowledge of what we don't know about post-completion outcomes. 

This is a Big Data story. As well a High Stakes Accountability Reporting story. There are lessons to be learned, for all of us. 

Especially for those who dare to build accountability reports.

Speaking of which, I had planned to write about specifics of calculating a graduation rate for master's level students. I think I will save that for next week.

Secretary Duncan told reporters Tuesday that he would be making an announcement on the ratings system “probably by the end of the year, so the next couple weeks.”

Under-secretary Ted Mitchell is interviewed by the Chronicle (paywalled

Q. The new proposed college-rating system was supposed to come out this fall. By the calendar I think you have about two weeks until the equinox. Will you make the time frame, and, if so, what will it look like?

A. [Laughs.] We’re planning on meeting our deadline. We are in fact in daily discussions about how we’re going to engage the field in the next part of this process. We will put out a schema for public comment, and the schema, as we talked about before, will not be populated with data but will instead give us an opportunity to consult with the field about some of the data elements, some of the specific metrics, that we’re looking at. We’re very much interested in responses from stakeholders over the next couple of months. There will be a 60-day comment period.


Q. We’ll have a new Congress in January run by Republicans who have not always been that friendly to the Obama administration’s higher-education policies. Does that portend trouble for thing like the ratings, or for the "gainful employment" regulation, or spending on financial aid, or even the department itself?

A. We feel that there’s a broad base of common ground with our colleagues on the Republican side, and look forward to exploring that with them over the coming weeks.


Q. Does the department plan to present any of its own priorities for the Higher Education Act reauthorization, and is there a timeline for that?

A. Yes. We’re working on a document that will help identify those and clarify those. We’re going to try to get through ratings and a couple of other things we’ve got on the plate before we turn to that.





blog comments powered by Disqus
Follow SCHEV Research on Twitter