School

All the Questions the U.S. News “Best Colleges” Rankings Can’t Answer

Families deciding on schools need something better.

A smiling woman and teenage girl sitting close together lean toward a laptop.
Alfredo Hernandez Rios/iStock/Getty Images Plus

This essay was adapted from Timothy Burke’s newsletter, Eight by Seven. Subscribe here. 

The announcement of the updated U.S. News and World Report college rankings has come and gone for this year, with the usual (completely valid) critiques of the methodology and uses of the rankings appearing in response. One particularly important critique, by Timothy Knowles of the Carnegie Foundation for the Advancement of Teaching and Ted Mitchell of the American Council on Education, announced upcoming changes to the Carnegie Classifications, a descriptive system that U.S. News uses as a basis for its own rankings. Knowles and Mitchell believe these changes will have a knock-on effect, forcing U.S. News’ approach to rankings to shift in years to come.

Advertisement

Whether or not that effort succeeds, it’s clear that some prospective students and their families rely on the rankings because they don’t know how to sort out the information available on colleges and universities. The problem runs deep. Many of them don’t really know what it is that they might want to know—what kinds of information could be important for making a good decision. Some families and applicants prefer some system of hierarchical ranking because they want the most exclusive, selective, elite institution that is known to be that way, because they believe (incorrectly) that attendance at such a school will secure the best future outcomes in life. But most applicants and families just want to know how to identify the experience that might be best for them—the education that will be most satisfying, most useful, most inspiring, most encouraging, most transformative.

Advertisement
Advertisement
Advertisement
Advertisement
Advertisement

What do applicants and families need to know about colleges and universities that a meaningful, well-tended, and relatively transparent system might try to evaluate for them? It might be best to start with an observation that Daniel Chambliss and Christopher Takacs make in their study How College Works: The two most important determinants of student satisfaction with the outcomes of a college education are whether the student made at least one friend relatively quickly after starting school and whether at some point in their time there they felt strongly connected to at least one faculty or staff mentor who helped them identify their talents, aspirations, and challenges in a way that was both useful and accurate.

Working backward from there, what might a college that would provide these kinds of connections look like? Is it possible to find out before you enroll? Here are the questions that I think could, or should, matter to prospective students and their families.

Advertisement
  1. Can I get in? (This should be first, because why bother looking at the rest of the information if the answer is no?)

  2. What we call financial aid is just a discount off the sticker price. So, then, what offer might I receive? How much is it actually going to cost for me to go there? Does financial aid cover costs and fees that I might encounter during the year beyond the discount to tuition? If not, what are those costs and exactly how much are they? What’s the cost of living in the area?

  3. How hard will it be for me to graduate from this institution (e.g., what are its graduation rates?) Are some programs of study harder to complete than others? Are some parts of the curriculum gated off, or restricted to a subset of matriculated students? Are there programs of study firmly under the control of a small number of faculty who are basically not accountable to the central administration (for good or ill)? 

  4. Roughly speaking, what are the outcomes of being a graduate of this institution? How satisfied are graduates? What kind of reputation does this institution have in the wider world? What are the employment outcomes from graduates one, five, 10, and 20 years out?

  5. What’s it like to be a student at this institution? Are students happy or unhappy, on average? Are some kinds of students happier or unhappier in relative measure? What do students do during the week and on the weekend? How much is there a match between the kind of person I think I am at 17 and the kind of people the students are? Is there a single “culture” or many “subcultures”?

  6. How well-run is this institution? How well-maintained are its facilities? (What facilities does it have?) How financially secure is the institution? Is it obsessed with cost-cutting, or are there plenty of resources to do the things I want to do? Are the administrators I am likely to deal with skilled, friendly, and responsive? Do they share my values, goals, and beliefs?

  7. How big is this place, physically? How does it relate to the communities around it? What’s it like, weather-wise, to be there from September to May? How complicated is transportation around this institution, and how hard is it to get there from where I live? Does it offer housing to undergraduates? What’s the housing like, if so? 

  8. Can I study the things I think I want to study? Are there lots of curricular choices available so that if I change my mind about what interests me, I have lots of options? If not, is there a coherent alternative vision about why the institution and the curriculum are the size that they are? Are there a lot of faculty members in the areas that interest me now? What about in areas that I don’t even know about that might become interesting to me later? Are the faculty available, inspiring, and engaged by the students and the institution? Or are they disaffected, disengaged, burned out, or working under highly unfavorable conditions? Will the faculty member that I have strongly connected with be there in the future, or are they on a short-term contract? Will I just be stuck talking to teaching assistants for most of my time? Are the departments and majors I am interested in accessible, engaging, and responsive? Will I be supported if I turn out to be less prepared than other students? Are the faculty snobbish or discriminatory? Are there lots of “weed-out” classes? Are there pre-professional or vocational courses in addition to more academic or liberal arts courses? How real are the programs and departments that appear on the website or in the catalog?

U.S. News’ rankings are useless for answering almost all of these questions. But how hard are they to answer, if you try? Do colleges and universities generally provide data and information that could be used to answer some of them?

Advertisement

Here and there. Sometimes not at all, because the answers are so qualitative and subjective that there couldn’t be a definitive answer, anyway. Sometimes they’re holding data that’s relevant that they absolutely don’t want to share (say, qualitative assessments of student, faculty, and staff satisfaction; exit interviews; and more—often they don’t want to share anything about the formula they use to assess financial need when they make aid awards). Sometimes institutions are actively pushing out deceptive information on these questions—for example, university websites will sometimes make programs of study look much more regularized, supported, and viable than they actually are. A skilled reader can detect that if they look hard enough.

Advertisement
Advertisement
Advertisement
Advertisement

Part of the problem is also that making sense of those questions requires the applicant and the family to do a thoroughly introspective audit of their own perceived needs and desires when it comes to going to college. That’s very difficult for almost everybody. Parents and children frequently can’t talk straightforwardly to one another about how they are thinking about this decision. Parents or families may not want to disclose the reality of their financial situation, they may not be fully aware of the anxieties or biases they have regarding the future lives of their children, and they may have some harmful or dysfunctional views of the whole situation. Sixteen- and 17-year-olds often have no idea what they want from universities or life in general. They often, quite reasonably, have no idea what to study. They may have received some serious misinformation about college (and life), or they may be actively wrong about their own preferences in a way that can’t be fixed until they’re actually in college and get a better idea of the person they really are.

Advertisement
Advertisement
Advertisement

Mostly, this is the kind of decision that you can’t get clarity about in advance. It’s not as simple as visiting a Wirecutter article to figure out how to buy a new lawnmower. A lot of this information only makes complete sense when you’re finished with college, and either informs your satisfaction with the experience or your bitterness about having made a bad choice. I only know what I’m looking for in a lawnmower because I’ve bought enough regrettable, shitty, malfunctioning lawnmowers in the past 30 years. You only buy an undergraduate education once, or maybe two or three times (if you start over or transfer).

If I were going to start to build a compact, easy-to-use evaluation system from scratch that spoke to the things that often really matter to students, I’d work from easiest to hardest:

Easiest:

  • How big is the institution?

  • What’s the weather like?

  • How hard is it to get there from your hometown, and to get around once you’re there?

  • In basic terms, how financially secure is it? (Look at size of endowment, budgetary model, revenue sources.)

  • How selective is it? (As proxy for “Can I get in?”)

  • What are the graduation rates and average time to graduation?

Medium difficulty (may require using public data in ways that are not enabled or encouraged by colleges and universities):

  • What’s the average financial award? How many students are receiving some form of discount? How deep are the discounts, and where’s the income cutoff for discount eligibility?

  • Evaluate some visible aspects of student culture: athletics and athletic facilities, how many sports are varsity and how many are club sports, whether there are Greek organizations and how many students live in houses controlled by them, how many students live in dormitories and how much off-campus housing there is (and how expensive it is). How many places are there to eat on campus and off campus? How expensive is the meal plan, if there is one? Are on-campus residences managed by the university or by an outside contractor?

  • What are the most common majors? How many majors and minors are there? What is the average class size? (Not faculty-student ratio, which is a bad proxy for average class size.) What are the requirements to enter majors? What are the general education requirements, and how are they structured? What is the average number of credits/courses taken per semester? How many interdisciplinary programs are there, and how easy is it to customize a course of study? I think maybe it would be possible to build a crude quantitative tool that would measure how “open” the curriculum is (how many requirements, the intervals of requirements, how many choices students have available, how variable openness is across different majors) so that this wasn’t just an empty marketing word.

  • What percentage of classes are taught by tenured and tenure-track faculty versus faculty on short-term contracts? (One of the worst weaknesses of the U.S. News rankings is that they don’t use this information.) Look into faculty compensation by rank compared with other institutions in the same region and against a comparison group. How much of the curriculum does faculty really control, and how consistent or unified are the faculty in their vision of the curriculum? 

Advertisement
Advertisement

Medium-hard (would take critical reading “against the grain” of some public information, news stories, and social media discussions to measure or evaluate, or a thoughtful combination of existing data to create an approximating metric):

  • Consider budgetary austerity (whether justified or not by the financial health of the institution), ease of access to needed resources, and availability of up-to-date facilities.

  • How is maintenance of the facilities? What is the average comfort level, and how is maintenance of dormitories? (Could use proxies like “numbers of one-room triples,” etc.)

  • How many listed departments, programs, and majors are “real” (e.g., have steady governance, long-term faculty commitments, dedicated facilities, and consistently available courses), and how many are underfunded, undersupported, or mostly fictional veneers? I can kind of judge this from long experience reading departmental webpages and course catalogs, but it takes work and a lot of insider knowledge (so I’m less good at it for some kinds of institutions).

  • Are there some kinds of rough outcomes data, based in part on visibility of alumni networks and on some of the analysis of the connections between inequality and different tiers of higher education? (E.g., I can’t tell you the really specific differences in outcomes between two similar institutions in the same rough tier of institutional wealth and selectivity, but I can tell you the difference between them and an institution in another tier.)

  • How accessible are programs of study to students with a range of prior preparations and qualifications, and how often there are “weed-out” classes in the curriculum? How many students enter expecting to do a particular major and end up having to settle for something else? How much paracurricular assistance is there for students trying to make up for a preparation gap? Can students take an easier version of a required course at another institution and get credit for it in their intended major? 

Really hard (requires ethnographic knowledge, insider knowledge, or is really subjective or shifts constantly):

  • Is this college or university well-run? Will the average student experience with administrators at any rank be positive and helpful? How easy is it to resolve problems? How transparent is the administration, and how easy is it to get information about a problem or issue, especially regarding financial aid and costs? Are the services I might need any good, and run at a high professional standard? (Career placement, health care, mental health, information technology support, academic support services, etc.)

  • Are the faculty secure in their situation, confident of their place in the institution, and accessible to undergraduates? Are they inspiring? Do they reflect the diversity of the student body? Are there mentors to connect to? Is there trust between faculty and students? Between faculty and administration? Are the faculty quick to address issues with the program of instruction, including availability of required or important classes? Are they being given the resources to address those situations?

  • What’s the student culture really like? Is the student body arrogant, pretentious, exclusive, snobby, intolerant? Friendly, accepting, genuinely diverse? Are there lots of subcultures? 

  • What are the typical outcomes for graduates and how much are they a result of things the institution is intentionally doing? (E.g., the vexed question of how much value the institution is actually adding as a result of its distinctive design, culture, and services versus how much is just the general outcome of being of a certain age, having a certain existing quality of academic preparation, and being in any program of study at the general level offered by most colleges?)

Advertisement

The problem is that I think much of what’s in the “medium hard” and “really hard” categories above accounts for much of what actually makes a difference in student experiences between specific institutions. Chambliss and Takacs’ two most important measures (making friends and finding mentors) can be found somewhere in those two sections.

At the least, it might be good to see some organization attempt to provide a better kind of evaluation of American colleges and universities that might meaningfully alert incoming students and their families to the kinds of factors that make a difference. U.S. News’ rankings are just useless noise, confounding that evaluation. There is room to do better.

Advertisement