College Rankings are Pure Kluges

The college rankings are out! US News, the Wall Street Journal, Princeton Review, Washington Monthly, FIRE, and others have made their opinions public.

College rankings should be a good source of information, but…

New College of Florida, which I joined last year as admissions director, responds like every other college in America. We survey the rankings to find new talking points, update our ad copy, and buckle down to answer questions about any movements since last year.

In our case, those questions assume a particular salience. In early 2023, amidst great publicity, the State of Florida installed new leadership charged with returning this small, public institution to the ranks of America’s best liberal arts colleges.

We’ve since instituted many changes in the service of that goal, including (but not limited to) the elimination of the DEI office, the termination of an enormous Gender Studies program, the introduction of NAIA athletics, a recalibration of our entire approach to student recruitment, and a significant turnover in both faculty and the student body.

There’s thus no shortage of people—insiders and outsiders, boosters and critics, alumni and prospective students—seeking indicators of our progress. Annual college rankings seem like a reasonable place to look.

The operative word, however, is “seem.” In point of fact, college rankings provide minimal information about recent changes at our—or any other—institution. Worse, they’re kluges, burying information students and families might find useful behind meaningless if clear answers.

Rankings based on prior leadership say nothing about the present

Take US News, likely the most prominent of the bunch. For liberal arts colleges like ours, the 47% of the score looking at graduates considers students entering between Fall 2014 and Fall 2017. Standardized test scores (5%) assess the classes entering Fall 2022 or 2023. Criteria pertaining to faculty (15%) draw upon data for the 2022/23 or 2023/34 academic years. Financial data and debt (13%) look at various sources from 2020-22.

From our perspective then, 65-80% of the US News data reflect upon prior leadership. Recent changes won’t begin to play a major role for at least several more years. Data fully reflective of the reforms begun in early 2023 will first appear in the rankings published in 2034.

And US News is hardly alone in its reliance on classes that entered long ago. Washington Monthly’s new rankings study the classes entering 2012-14. Any of the other rankings trying to understand retention, graduation, or post-graduation outcomes necessarily do the same.

The remaining 20% of the US News score—a “peer assessment” surveying college presidents, provosts, and deans of admissions conducted each spring—is indeed relevant to us. Of course, given that we’re the vanguard of a movement to reform higher education, we can’t fairly complain if the successful, prestigious people we’re out to displace don’t much like us.

These rankings thus hit us with a double whammy: The decline that necessitated a change in leadership and the reform character of our new leadership both count against us.

What’s a student to do?

Our idiosyncratic challenges, however, pale in comparison with those of students, parents, or faculty members trying to compare colleges. Rather than sharing the valuable insights the data make possible, these publications present only meaningless rankings designed for headlines and advertising revenues.

To be useful, a score must measure something static—that is, a fixed quantity at a fixed point in time. A score that combines data collected on students who entered ten years ago, faculty salaries from two years ago, financial data from four years ago, and six-month-old opinion surveys represents an attempt to measure something that can’t be defined. From the perspective of mathematical or statistical modeling, that’s a classic error.

While that may sound a bit technical, it’s easy to illustrate—particularly to parents of college-bound children. If a form asks how tall your child is, you can probably provide a clear numeric answer. Suppose, however, that the question asked how tall your child has been over the past decade. Would any number you provided be meaningful? That’s the sort of information college rankings provide: A single score pretending to measure how well each institution has performed at different tasks over the course of a decade.

It gets worse. The weightings used to combine factors are unlikely to align with any particular family’s concerns. By putting the ranking front and center while burying the data, the publications make it hard for families to find the information that most interests them.

College rankings less meaningful than scores

Finally, a focus on rankings rather than scores inflates the significance of movement within the ranks. Raw scores tend to cluster—many schools may have “about” the same score. When that happens, minor year-to-year changes in a school’s factor can push it toward the top or the bottom of a group that is essentially tied. There’s no way to know whether a college’s drop of dozens of places stemmed from major or minor year-to-year changes.

All told, as eagerly as so many of us await each year’s rankings, it’s beyond time to admit that they’re meaningless. If you’re trying to compare or select colleges, these rankings likely do more to confuse than to inform. In this task, as in so many others, there’s no substitute for doing your own research. Fortunately, many of these publications have done a lot of it for you. All that you have to do is find where they’ve hidden it.

This article was originally published by RealClearEducation and made available via RealClearWire.