“It is hard to improve what isn’t measured.” So said Andreas Schleicher, who runs a programme to identify the abilities of school children in different countries on behalf of the OECD. Eight years ago, a similar measurement for higher education was proposed. The Assessment of Higher Education Learning Outcomes (AHLEO), as it is known, was due to be discussed this week at the sixth annual international symposium on university rankings and quality assurance in Brussels, but has been dropped from the programme. Where did AHELO go?
A clue could be found in where it came from. The idea was first mooted in 2006 when more than 30 ministers of education gathered in Athens to discuss higher education reforms worldwide. In his opening speech, Angel Gurria, the secretary general of the OECD, suggested undertaking “a comparable survey of the skills and abilities of graduates that might measure learning outcomes in higher education and help families, businesses and governments develop an evidence-based understanding of where and how higher education systems are meeting their expectations of quality and where they are not”. The OECD had developed the methodology to do it, he said. All they needed was a mandate. They got it.
This needs to be seen in the context of discussions about rankings that were taking place at the time. In the early 2000s international rankings in higher education were a new phenomenon. In 2003 the first edition of the Academic Ranking of World Universities was published by Shanghai Jiao Tong University. In 2004 the British magazine Times Higher Education published the first edition of the Times Higher Education-QS World University Rankings, which was later split up into the THE World University Ranking and the QS World University Ranking. Then followed the Taiwan Ranking and the Leiden CWTS Ranking and those of a plethora of newspapers.
The international higher education community was critical, to say the least. It disliked what these rankings measured, how they measured it and the possible consequences. There was too much focus on research and, in particular, on those aspects of research that could be quantified and counted: publications, citations and Nobel prizes. The resulting rankings and scores were presented as quality indicators for all activities in universities, including teaching, and an institution’s position in these rankings appeared to be important in the decision-making processes of international students. As a result, rankers searched for indicators that could express the quality of education, preferably in a single, globally comparable number—something that most experts and practitioners in education knew was an impossible goal.
Those education indicators that were included were poor proxies for the quality of teaching and learning. Teachers’ highest degrees or the ratios of staff to students merely said something about the input, not necessarily about quality. Other indicators, such as reputation or employer surveys as a proxy for teaching quality, were highly controversial. Highly selective institutions—which tended to lead the international rankings—might have many graduates in leading positions, but was that due to the institution’s efforts or the quality of the students it admitted? Is the best school the one that delivers the best graduates or the one that adds most value through the quality of its teaching?
Another problem was the homogenising tendency of international rankings. Having a limited set of indicators could cause institutions to focus on them, instead of the underlying objectives. This could mean specific local contexts would be neglected and local demands would not be met, which could lead to decreasing diversity. The only benchmark in the global knowledge economy—according to the international rankings—seemed to be the world-class, research intensive university.
AHELO—together with transparency tools such as U-Map, which looks at which activities individual institutions across Europe are involved in, and U-Multirank, which examines how well each institution does in these activities—emerged as the answer. The aim was to restore the teaching and learning mission of the university to a central position, look at the added value of institutions rather than their graduates, and show that a range of institutions—not just the so-called world-class research universities—could add value. Although the OECD repeatedly stressed that AHELO was never envisaged as a ranking tool, the rankings issue was frequently mentioned in its favour.
But can AHELO really provide an alternative for international rankings? It seems unlikely. Shortly after Schleicher addressed the conference in March 2013, the OECD published the results of a feasibility study. A staggering three volumes containing more than 500 pages concluded: “It is feasible to develop instruments with reliable and valid results across different countries, languages, cultures and institutional settings.” After that, came a deafening silence. The feasibility study may have proved that it is scientifically possible, but it is probably not financially or politically practical—or desirable.
According to the New York Times, the OECD spent about $13 million on the study. How much time and money was spent by the 17 governments and 248 universities involved is unknown—not great for an organisation that values and advocates transparency.
And AHELO does not solve the problem that rankings only look at input and output rather than the value added. In the feasibility study, the OECD notes that value-added analysis is possible but the challenges are considerable. Furthermore, it claims that the difficulties inherent in making comparisons across jurisdictions, mean the main use of the results of any value-added analysis would be for institutional self-study and comparisons between programmes within a jurisdiction. Although this might be helpful for a university’s internal quality assessment, it still only assesses learning outcomes after three or four years of study and, as has been shown with other instruments, this might be more an indicator of the quality of the admitted students and the quality of secondary education than the quality of the teaching and learning process within the university.
Finally, as critics have suggested, AHELO risks being at least as homogenising as other ranking systems, while standardisation of learning outcomes could lead to conformity and stifle innovation in teaching even more than the existing rankings do. A one-size-fits-all approach to learning outcome assessments, that originates largely from the Euro-American world, could undervalue local knowledge.
Schleicher may well have been right about the difficulties of improving what isn’t measured. But being able to measure something is no guarantee of improvement and measurement is not worth pursuing at all costs.