Higher education has not been immune to the gradual and recent evolution to a more customer-centric focus in nearly all aspects of life. Some schools have embraced this movement and have worked to improve the user experience in higher education. For example, not only are courses offered online, but course selection, modality, location, pricing, and even student support services have been tailored to support the student perspective and student desire. However, driven in large measure by the college rankings published by U.S. News & World Report, most colleges and universities remain locked in a fierce published ranking arms race of sorts, where many factors other than those that are of interest or significance to adult students are pursued. A few institutions, like University of Maryland University College, have taken the courageous step of dropping out of this arms race, rooted in the belief that a college or university should be measured by learning outcomes—what students learn and truly master—and whether they can apply that in the workplace.
Other institutions are taking things into their own hands and are going beyond the standard rankings with data that are of particular interest to this segment. For example, Capella University promotes CapellaResults.org and shines a light on learning outcomes, student satisfaction data, and cost information, along with student, faculty, and employer testimonials. Prospective students can easily find affordability data—costs, average debt, and average time to completion and career outcomes that are linked to a degree program. A prospective student can then act as an educated consumer and personally conduct the cost/benefit analysis.
Our lives today are governed by ratings. We are using Yelp to find places to eat and services to purchase. Angie’s List offers personalized and localized ratings for just about any home service available. And CarFax has put knowledge, and power, in the hands of the used car consumer as Consumer Reports has done for the new car purchaser. So it seems reasonable that market demand would seek to apply similar methodology to higher education. The issue for our consideration is not whether this market demand is appropriate, but rather whether the ranking system as constructed is appropriate and whether it holds up to scrutiny. I suggest it is lacking.
We know that the first-time, full-time, 18-22 year-olds who pursue residential higher education programs immediately after high school is a relatively small percentage of students, and growing smaller. Nearly three quarters of students pursuing higher education have at least one “non-traditional” student characteristic that exempts them from the standard Federal reporting requirements. The National Center for Education Statistics (NCES) has identified seven characteristics that are common to non-traditional students (Choy, 2002). They are:
- Delayed enrollment post high school
- Attend part time
- Work full time
- Considered financially independent
- Has dependents other than a spouse
- Is a single parent
- Does not have a high school diploma
Thus, the size and scope of this segment is underreported. The situation is further exacerbated by the fact that this group drops out nearly twice as often as the “traditional” students. So, a majority segment of students in higher education is under reported and at higher risk when it comes to successful completion, and school rating systems typically don’t measure factors that would help them succeed. Yet these same students rely on these rankings to determine a “good university” to attend, even though many factors in the rankings are not meaningful for part-time adult students, and factors that could make a difference are undervalued. Yet prospective students use the rankings to determine the better university fit for their academic pursuits.
Consider these comparative questions that underlie the U.S. News ratings in evaluating what makes for a better university:
- How many students does the university allow in, or how many students does it keeps out—in other words, how selective is the university?
- How many journal articles has a faculty member published, resulting in lifetime job security known as tenure, or how many students has a faculty member helped to graduate on time, land a good job, or advance a career, with or without tenure?
- How well is a faculty member prepared and supported to teach an online class?
- How much indebtedness has a student accumulated?
While these may at first seem like unconnected, even spurious, questions and connections, a deep dive into the Best Online Programs ratings by U.S. News reveals some interesting weighting anomalies, especially if one seriously considers the needs, experiences, and success stories of part-time adult students. Another article in this issue of UPCEA Unbound, “U. S. News and World Report’s Rankings of Online Programs,” has already dissected the technical aspects of the ratings, how they are compiled, and the criteria used. This article attempts to tease out some aspects that may not be readily apparent at first blush. It will also likely give pause to the reflective adult education practitioner who serves students in online programs. This is not meant to disparage admissions selectivity, faculty research and publishing productivity, or student debt reality. Rather, I strongly suggest that a rebalancing of some of the weighting of elements in the U.S. News survey is required if we want to better demonstrate true effectiveness for part-time adult students. By way of analogy, if we place a high priority on seat time for an online student, then we are worried about the wrong end of the student. Likewise, if our ranking system does not consider what matters and values what does not matter, then it is out of balance and is not serving our students well.
For example, one of the key issues historically used to rank institutions and used as an indicator of prestige is selectivity—how many students are rejected, turned away, not granted admission. On the issue of selectivity, relative selectivity is rewarded in the U.S. News & World Report rankings. In other words, who an institution keeps out is judged to be more important, as in it results in a higher score, than who an institution lets in and helps to be successful. While not all methodologies rate selectivity equally, that issue is still important to evaluate as it perhaps reflects a bias towards traditional notions of educational quality. By way of example, 20% of the scoring for the Best Online MBA & Graduate Business Program ratings is related to admissions selectivity. For MBA students, that percentage jumps to 25%, yet nursing, engineering, and computer information technology all consider admissions selectivity as 10% of the scoring for ranking. Selectivity is not even a factor in the ranking methodology for the Best Online Bachelor’s Programs, but faculty credentials shape 20% of the scoring for this area. Adult educators know that returning adult students who are working full time and studying part time are highly motivated and regularly exceed expectations for their academic performance, based on their past standardized test scores and undergraduate GPA. It seems counterintuitive to reward institutions that do a better job of keeping students out rather than reward those institutions that allow more students in and help them to be successful, as measured by learning outcomes, retention, degree completion, and career advancement.
With respect to faculty who instruct these students, the weighting system is even more pernicious. In ratings of the Best Online Bachelor’s Degree programs, for example, the faculty credentials and training category accounts for 20% of the weighting. However, having a terminal degree, such as a PhD, JD or MFA , or tenure makes up nearly 60% of this category. And yet faculty preparedness for teaching online is less than 30% of this category, and staff support for these faculty is a mere 13% of the category. In the rankings of the graduate programs, the situation is even worse from the student perspective. The rankings of the Best Online MBA & Graduate Business programs consider faculty credentials and training as 11% of the score, and for other business degrees, this factor is 13.5% of the scoring. Nursing, education, engineering, and computer information technology award a full 25%, and criminal justice comes in at a very odd 26%. However, for all of these graduate degrees, having a terminal degree and tenure accounts for 60% of the composition of the scoring for faculty credentials and training, while being prepared to teach online students is only 30% of the category. Thus, for an online MBA program, faculty preparation to teach online students accounts for 3% of the overall score in the rankings. Even at the highest level, criminal justice faculty preparation to teach online is a measly 7.8% overall. Students beware!
So, in other words, students who use the U.S. News & World Report rankings would be led to believe that the better university for them is the one that has engaged more tenured and tenure-track faculty (usually based largely on academic publishing prowess), especially those with a terminal degree, rather than those who have really been thoroughly trained to teach fully online students with a full staff support cadre behind them. Even the most casual observer of online learning knows what every online student knows—faculty preparation and back-up pedagogical and technical support for faculty is critical to student success and a satisfactory student experience.
These are brief examples of disproportionate weighting of factors that matter and those that don’t for part-time adult students. What about factors that are missing? For example, one of the relatively recent and increasingly adopted strategies for supporting online student persistence, retention and success is personal coaching. This has proven to be an effective strategy for student success while also a cost-effective use of university resources. A 2013 study of the impact of the work of InsideTrack, arguably the largest and most successful coaching company in the United States, found that during the first year of coaching, students were five percentage points more likely to persist in college. The same study, which was released by the American Educational Research Association and conducted by Stanford University researchers, found the effects of coaching didn’t disappear (Bettinger and Baker, 2014). Coached students were three to four percentage points more likely to persist after 18 months and 24 months, respectively. When we look at the value system of the U.S. News & World Report survey, we find that coaching is absent and is not counted at either the undergraduate or graduate program level. This is a glaring omission, given that personal student coaching may be one of the most significant student support services that an institution can implement for fully online students.
Another factor that is glaringly absent from the criteria for both undergraduate and graduate programs is online course development that is in alignment with national standards such as Quality Matters. While the aforementioned faculty preparedness has a very minor role in the weighting system, around 5 or 6%, and the use of instructional designers is mentioned to an even lesser degree, there is no attention paid to course development that aligns with national standards as an assurance of quality across courses in a full degree program. I would argue that such course design and development considerations far outweigh the priority given to other important factors such as multifactor online test authentication and University copyright and plagiarism policies when it comes to factors that matter for the academic success and satisfaction of part-time adult online students.
Finally, there is a litany of other metrics that are far more important and meaningful for an adult audience and the U.S. News & World Report survey never comes near this territory. These include metrics that are foreign to most traditional higher education reviews: long-term graduation rates, employment and salary data, career advancement after degree completion, full cost of attendance, and student satisfaction with instructor in a course as well as with the entire user experience. Even if we assume pure and good intentions by all involved, there are many aspects of the U.S. News & World Report ratings that are a disservice to part-time online adult students. Anyone who has been engaged in this enterprise knows this to be true. Granted, some of the shortcomings are inherently related to the methodology of only using a quantitative measure to produce the rankings. But even within the limitations of this quantitative system, as noted above, the weighting system could be adjusted to more accurately represent those criteria that make a difference for part-time adult students. We know how to do this. We owe it to our students to make the effort and improve the system.
References
Bettinger, E. P. and Baker, R. B. (2014). The effects of student coaching: An evaluation of a randomized experiment in student advising. Educational Evaluation and Policy Analysis. 36(1), 3-19.
Choy, S. P. (2002). Nontraditional undergraduates. Findings from The Condition of Education 2002 (NCES 2002-012). Washington, DC: U.S. Department of Education, National Center for Education Statistics.
Richard Novak, Ed.D., is Vice President for Continuing Studies and Distance Education at Rutgers, The State University of New Jersey, providing executive leadership for the University-wide Division of Continuing Studies (DoCS) – 20 distinct business units, 140 full-time employees in 20 locations, offering credit and non-credit educational opportunities across the lifespan and across the full calendar year, in face-to-face and online formats. In addition, the Division includes a University hotel and conference center, a high definition broadcast television studio and the largest Makerspace in the region. Novak is also an associate member of the graduate faculty for the Rutgers Graduate School of Education, regularly teaching online graduate courses in educational technology. He is a Past President of UPCEA and recipient of UPCEA’s Walton S. Bittner Award for Imaginative Leadership in the Advancement of Continuing Education and Distinguished Service to the Association. In April 2011 he was awarded the national Excellence in Online Administration award.