Web Policy  |  Intranet  |  Contact SCHEV    
SCHEV Research Data Blog

The official blog of @SCHEVResearch at the State Council of Higher Education for Virginia. Discussions about our work, national higher education data policy, and highlights about the data we publish.


Recent Posts Blog Archives Subscribe Feed All Blogs

Happy Ratingsmas!

by Tod Massa 20. December 2014 08:58

After 16 months of hearings, public comment, expert testimony, a lot of hand-wringing, and many Microsoft-like promises, the feds have released a framework of sorts for the Postsecondary Institution Ratings System (PIRS). The press and select others were briefed Thursday and thus had their stories ready to go at 12:01 am this morning. Robert Kelchen wrote a very good, informative blog post here that was published by 1:00 am. Kevin Carey also did a nice job sizing up the ratings and the politics.

Relevant articles:

Official blog post.

Chronicle article.

InsideHigherEd article and side story - this is my favorite since I am quoted. Actually, I said some things that were more interesting...and more pointed...in the interview, but this works.

Of course, the real meat is the framework itself

I'm going to use this space just to look at the intended purposes of the ratings.

• To help colleges and universities measure, benchmark, and improve across shared principles of access, affordability, and outcomes.

• To provide better information about college value to students and families to support them as they search for select a college,

• To generate reliable, useful data that policymakers and the public can use to hold America’s colleges and universities accountable for key performance measures. In the future this can be used to help align incentives for colleges to serve students from all backgrounds well by focusing on the shared principles of access, affordability, and outcomes; ensuring wise and effective use of $150 billion in financial aid.

 • In additional to federal efforts, and those of individual institutions, we believe the ratings system can help inform policy, accreditation and funding decisions by states education authorities, policies and practices of accreditors and others.


These are nice ideas. I just don't think they require a federal ratings system. The first bullet certainly does not. There are so many benchmarking projects going on and so many professional papers on benchmarking and comparison, I just don't see the need for lessons from the feds. Generally, I suspect my institutional colleagues need more time and resources to reflect on effective measurement and its implementation, and the benchmarks they are already using. One of the big problems with the analytics movement is that we haven't solved the basic problem that is the needed time for reflection about the meaning of what we measure.

Ratings by themselves don't necessarily create "better information." Many of us made clear to the Department that it needs better data to create a rating system. Why not just accumulate better data, create better information, and make that available? Further, let the private sector focus on how best to package and provide it. After all, as Aneesh Chopra is fond of pointing out, the weather industry is fundamentally based on the repackaging federal data. 

Again, the ratings themselves are not necessary to the generation of reliable and useful data for use by policymakers and the public. All sorts of data can do that without having any kind of a rating component. Maybe the implied argument is that, without moving in the direction of a ratings system, the Department, perhaps all of the federal government, is incapable of fulfilling a vision of better data without it being the necessary component of something else.

And finally, this is where I have the most heartburn. Any ratings system will have an underlying structure and philosophy that can lead to a very myopic view of higher education policymaking. It doesn't have to be this way, but it is certainly possible. Too often in my opinion, policymakers seem to want as little information as possible to make a decision. Does this really lead to better policy? I understand the whole thing about a filling a tea cup with a firehose, but that doesn't mean a single cup of tea is adequate nutrition.

Finally, a bit of wisdom from Stephen Porter:

A valid institutional performance measure is like the Holy Grail of higher ed. No seems to have considered that there are many brilliant researchers in econ, education and sociology who would love to develop one. That one has not yet been developed should illustrate how difficult this issue is.

On second thought, it is exactly like the Holy Grail:

1. It does not exist.

2. Good people are wasting their time and energy questing after it.

3. Hucksters are selling cheap goblets to a gullible public.

No, there is no single valid performance measure to address the entirety of the simplest institution. We have done more than enough performance management work in Virginia to know this lesson quite well. 



The Wart, young King Arthur was talking to the Badger.

“He would go to war, if King Uther declared one. Do you know that Homo sapiens is almost the only animal which wages war?”

“Ants do.”

“Don’t say ‘Ants do’ in that sweeping way, dear boy. There are more than four thousand different sorts of them, and from all those kinds I can only think of five which are belligerent. There are the five ants, one termite that I know of, and Man.”

“But the packs of wolves from the Forest Sauvage attack our flocks of sheep every winter.”

“Wolves and sheep belong to different species, my friend. True warfare is what happens between bands of the same species. Out of the hundreds of thousands of species, I can only think of seven which are belligerent. Even Man has a few varieties like the Esquimaux and the Gypsies and the Lapps and certain Nomads in Arabia, who do not do it, because they do not claim boundaries. True warfare is rarer in Nature than cannibalism. Don’t you think that is a little unfortunate?”

“Personally,” said the Wart, “I should have liked to go to war, if I could have been made a knight. I should have liked the banners and the trumpets, the flashing armour and the glorious charges. And oh, I should have liked to do great deeds, and be brave, and conquer my own fears. Don’t you have courage in warfare, Badger, and endurance, and comrades whom you love?”

The learned animal thought for a long time, gazing into the fire.

In the end, he seemed to change the subject.

“Which did you like best,” he asked, “the ants or the wild geese?”

--T.H. White, The Once and Future King

Have a great holiday! I will be off on a quest of performance measurement. After I pull this big knife out of a rock in my backyard.



blog comments powered by Disqus
Follow SCHEV Research on Twitter