Although there are quite some differences in the results of the rankings, a few common observations can be made. First of all, it is clear that the United States still is home to the best universities. In all rankings the US universities are dominant and Harvard is the undisputed leader. Only in the QS ranking it was a non US university  – Cambridge – that topped the list.

Another observation is that non of the rankings manage to sufficiently capture the quality of teaching in their assessment. The THE ranking made an attempt to do so, but most of their indicators still reflect research quality and prestige more than the quality of teaching. The Shanghai, Leiden and Taiwan rankings put most emphasis on research.

Even though the ranking predominantly assess research – although in different ways – the results are very different. To illustrate this point I have mapped the results of the Dutch research universities in the different rankings. The results are shown in the graph below (click to enlarge).

The results for the twelve universities (the thirteenth, Tilburg University somehow doesn’t appear in the rankings) show a substantial variation for all universities. For universities like Eindhoven, Twente and Maastricht, the variation seems exceptionally large. Eindhoven for instance was ranked as the best university in the THE ranking while performing worst in the Shanghai ranking. Leiden shows the least variation but here the difference between its rank in the Shanghai ranking (70th) and the THE ranking (124th) is still enormous.

Earlier this week at the OECD/IMHE conference Charles Reed, Chancellor of California State University, critiqued the rankings (“rankings are a disease”) and argued that all universities add value. I guess he’s right. And the value measured by one ranking seems to be quite different than the value measured by the other…

For the EIS 2008 the methodology has been revised and the number of dimensions increased to 7 and grouped into 3 main blocks covering enablers, firm activities and outputs (Figure 1). The purpose of this revision is to have dimensions that bring together a set of related indicators to give a balanced assessment of the innovation performance in that dimension. The blocks and dimensions have been designed to accommodate the diversity of different innovation processes and models that occur in different national contexts. A summary of the European state of innovation is given in the picture below (click to enlarge)

Based on their innovation performance across 29 indicators, EU Member States fall into the following four country groups:

• Sweden, Finland, Germany, Denmark and the UK are the Innovation leaders, with innovation performance well above that of the EU average and all other countries. Of these countries, Germany is improving its performance fastest while Denmark is stagnating.
• Austria, Ireland, Luxembourg, Belgium, France and the Netherlands are the Innovation followers, with innovation performance below those of the innovation leaders but above that the EU average. Ireland’s performance has been increasing fastest within this group, followed by Austria.
• Cyprus, Estonia, Slovenia, Czech Republic, Spain, Portugal, Greece and Italy are the Moderate innovators, with innovation performance below the EU average. The trend in Cyprus’ innovation performance is well above the average for this group, followed by Portugal, while Spain and Italy are not improving their relative position.
• Malta, Hungary, Slovakia, Poland, Lithuania, Romania, Latvia and Bulgaria are the Catching-up countries with innovation performance well below the EU average. All of these countries have been catching up, with the exception of Lithuania.  Bulgaria and Romania have been improving their performance the fastest.

The full report can be downloaded here

No, it’s not that ENS Lyon is a lousy university. It’s just that they ended up on rank number 157 of the Times Higher Education Ranking in 2007. And – as I feared before – nowadays in the Netherlands this means that you are not qualified as a skilled migrant. As is stated in the new immigration policy – entered into force this year – only some knowledge workers are eligible to enter the Netherlands:

From abroad, immigrants are only eligible if they graduated from a university in the top 150 of the two recognised university rankings, the league tables published in 2007 by the ‘Times Higher Education Supplement‘ and the ‘Jiao Tong Shanghai University’.

You might argue that ENS Lyon improved its ranking to 140 in 2008. But no…, the Dutch government uses the 2007 league tables. Period!

Or what if I had graduated from some of the world’s best liberal arts colleges? From Amherst, Williams, Swarthmore, Wellesley?

Not good enough…

And what if I had graduated from the University of Hokkaido (Japan)? The University of Notre Dame (USA)? Shanghai Jiao Tong University (China)? Universität Stuttgart (Germany)? University of Calgary (Canada)? Macquarie University (Australia)? Helsinki University of Technology (Finland)? RWTH Aachen (Germany)? University of Surrey (UK)? University of Barcelona (Spain)? Chalmers University of Technology (Sweden)? ….Bergen, Ottawa, Frankfurt, Brussels, Stockholm, Coimbra, Delaware?

No…not good enough…

The graph below shows that the United States and the United Kingdom are again superior in the Times rankings, followed by Australia and Canada. The Netherlands is the first non English speaking country, followed by Japan and Germany. The main difference however compared to last years results is that the number of countries represented in the top 200 has increased. The group is now joined by countries like Greece, Argentina, Thailand, Russia and India.

But of course…size matters and it’s easier to have many well performing universities in a large country than in a small country. So here is the result if we take population into account.

This of course works well for the small states like Switzerland, Hong Kong and Singapore. The Netherlands again comes fifth in line. If we control for GDP instead of population we get a similar picture. Here however, Hong Kong clearly outperforms the rest.

For what it’s worth….

Yes, it’s time for the fifth edition of the Times Higher Education World University Ranking of 2008. Quacquarelli Symonds Ltd, the company responsible for the ranking, claims (again) that the methodology is improved. They are even so blunt to say that ‘the rankings have established themselves as an accepted benchmark of quality’. I beg to differ…

One issue at least seems to be resolved, that is the volatility of the THE ranking (compared for instance with the relatively stable Shanghai Ranking):

The final results will see more countries represented among the top 200 institutions, with Continental Europe beginning to make more of a mark than in previous editions. But there will be less volatility this year, thanks to the change in statistical methodology introduced in 2007. Single outliers no longer have a disproportionate effect on the overall ranking.

The world university ranking will be published here and here tomorrow morning…

This conference (Transparency in Diversity – Towards a European Classification of Higher Education Institutions) presented the results from the second stage of the project Classifying European Institutions for Higher Education, a project that might turn out to have a major impact on European higher education policy. This project was initiated in 2005 (see this previous post) and is now supported by for instance the European Commission (DG Education) and the German Hochschule Rektorenkonferenz. It’s run by an international team led by Frans van Vught.

The project can be seen as a response to two trends (at least, that’s my interpretation). First of all, there is the emergence of the European higher education area, the objective of the Bologna process. If there’s one space, we need to know what types of institutions are occupying that space and hence, we need a classification or typology.

Secondly, there is the proliferation of ranking and league tables. As I’ve discussed many times before, these rankings present a very uni-dimensional view of the contemporary higher education institution. Basically they only look at the – science heavy – traditional research university. Through this they neglect the quality of a very wide range of other institutions which might be very good at the things they are supposed to do. Here one can think of mono-disciplinary institutions (e.g. colleges of fine arts; schools of economics and business), teaching oriented institutions (like the American liberal arts colleges) ore more professionally and vocationally oriented institutions (like the German and Austrian Fachhochschulen, the Dutch Hogescholen, etc.).

A multidimensional classification of European higher education institutions can on the one hand create more transparency in European higher education, while at the same time clarify which institutions can be compared with each other (so we can compare apples with apples and pears with pears). If you are interested in how they intend to do this, I suggest you have a look at the presentations of the conference. See Frans van Vughts presentation (PDF) to get a better idea about the background of the project and have a look at Frans Kaiser’s presentation (PDF) for the technical aspects of such a multidimensional classification.

What the classification will look like exactly is not yet clear. If it will remain limited to the web tool and the resulting radar graphs, I expect the effects to be rather limited. The question is whether the various stakeholders related to the project will ultimately define real categories of institutions (like the old Carnegie classification did). This however might give the project a more political character. Even though the project-team stresses that they will not create a hierarchical classification, it is interesting to see whether some categories will be perceived as more prestigious than others.

Nevertheless, the classification project seems to be widely supported by institutions throughout Europe and their representative organisations. The feeling that Europe needs to create more transparency is widely shared and at the same time, many institutions are looking for benchmarking opportunities with like-minded institutions. After all, comparisons with Harvard, Oxford and Yale are not very useful for most higher education institutions in Europe…

This exactly is the strength of the SJT ranking. After all, universities are robust organisations and don’t change a lot in a years time. I guess it therefore corresponds with reality that the top 10 of 2008 is exactly the same as the one of 2007. Actually, not much has changed at all (although I of course did notice that the University of Sydney – my former employer – entered the top 100; the top 500 list is here).

2008(2007)		University
1 (1)		Harvard University
2 (2)		Stanford University
3 (3)		University California – Berkeley
4 (4)		University Cambridge
5 (5)		Massachusetts Inst Tech (MIT)
6 (6)		California Inst Tech
7 (7)		Columbia University
8 (8)		Princeton University
9 (9)		University of Chicago
10 (10)		University of Oxford

The main critique on the SJT rankings is that they only give an indication of a university’s research quality. They have only one proxy for teaching quality and that one isn’t exactly saying much about teaching quality at all. I have already pointed to some alternatives for these research biased rankings and league tables, for instance the new ranking being develop by CCAP (Center for College Affordability and Productivity).

This last one has now been published by Forbes Magazine. And yes…the criteria are very different than the ones we are used to:

1. Listing of Alumni in the 2008 Who’s Who in America (25%)
2. Student Evaluations of Professors from Ratemyprofessors.com (25%)
3. Four- Year Graduation Rates (16 2/3%)
4. Enrollment-adjusted numbers of students and faculty receiving nationally competitive awards (16 2/3%)
5. Average four year accumulated student debt of those borrowing money (16 2/3%)

And what’s the result?

2008	University
1	Princeton University
2	California Institute of Technology
3	Harvard University
4	Swarthmore College
5	Williams College
6	United States Military Academy
7	Amherst College
8	Wellesley College
9	Yale University
10	Columbia University

Compared with the SJT rankings, it are especially the liberal art colleges and the military colleges that are evident in the Forbes ranking. The high quality liberal arts colleges in the US (and elsewhere) are unfortunately lacking in nearly all international rankings. The reasons for this is of course again that these rankings are so research biased.

Another thing that I noticed after looking through the rest of the list is the relatively low standing of the public research universities. University of Virginia is the first one on 43, University of North Carolina at Chapel Hill at 66 and UC Berkeley at 73.This is probably due to another flaw in most rankings, that is that they measure the quality of the graduates without looking at the quality of the inputs. For more criticism on this ranking, see the comments on Vedder’s article in Inside HigherEd and the critical contribution of Patricia McGuire.

This challenge of actually measuring the added value provided by the university is taken up by the OECD’s AHELO project: assessing learning outcomes in higher education (sometimes referred to as the PISA for higher education). This exercise is still in it’s early stages and currently they are at the stage of studying the feasibility of such an exercise. And although the OECD explicitly does not want to promote it as a ranking, it might provide an alternative for the league tables.

Anticipating the implementation of the new migration system, the government will at the latest in the first half of 2009 introduce a regulation for highly skilled immigrants. On the basis of the regulation, foreigners can stay in the country for a maximum of one year to find a job as a ‘knowledge migrant’ or to start an innovative company.

The objective of the regulation connects well to the ambition of the innovation platform to attract 1000 extra knowledge migrants. It is also in line with the advice on knowledge migrants of the Commission on Labour Participation in its report ‘towards a future that works‘.

The target group consists of foreigners that are relatively young and received their Bachelor, Master or PhD degree not longer than three years ago. Migrants are eligible if they received their degree from a university that is in the top 150 of two international league tables of universities. Because of the overlap, the lists consists of 189 universities…

And guess what the two league tables are. Yes, the Shanghai ranking and the Times Higher Education Ranking. Now…this will mean that firms like this have influence on who is eligible to come and work in the Netherlands. Something is not right here…

It looks like it, however I must admit it’s slightly different. It won’t be a meta-ranker, but more an accreditor of rankings. I’m talking about the establishment of the IREG–International Observatory on Academic Ranking and Excellence:

On April 18, 2008 an important decision was reached by the International Ranking Expert Group (IREG) to consolidate its partnership arrangement with the   creation of the IREG-International Observatory on Academic Ranking and Excellence.

Bob Morse, director of data research of one of the first and one of the most influential rankings – US NEWS & World Report – is one of the Executive Committee members and he writes on his blog Morse Code:

The International Observatory, headquartered in Warsaw, will conduct reviews of various ‘academic rankings’ and measures of ‘academic excellence’ to assess how well they serve higher education stakeholders and the general public. The observatory will use the recommendations formulated in the Berlin Principles on Ranking of Higher Education Institutions. Members of the body also will meet at the request of various ranking agencies to review their particular methodology criteria and standards. Ranking entities that receive observatory approval will be able to declare themselves ‘IREG Recognized’.

Especially that last item seems to point to an accreditor of rankings. But then, what gives IREG the authority to declare a ranking recognized or not? Well… at least they have some ‘recognized’ persons in their Executive Committee. Next to Bob Morse there are Gero Federkeil (CHE, Germany), Liu Nian Cai (Shanghai Jiao Tong University) and Alex Usher (Education Policy Institute, Toronto, Canada). The Committee is chaired by Jan Sadlak, the Director of UNESCO-CEPES in Romania. I wonder how this all will develop. And I wonder who will first get the ‘IREG-disapproved stamp’. Plenty of candidates…

The researchers looked at five rankings in particular, three national ones (Sunday Times Good University Guide, The Times Good University Guide, The Guardian University Guide) and two international rankings (Shanghai Rankings and the Times QS Ranking). The report itself and the background data are all available on HEFCE’s website.

Roughly, the study is divided into three parts. The first looks at rankings and their shortcomings in general. The second at the impact of rankings on universities in the UK. And the final part discusses alternative ranking methods such as the CHE ranking.

One of the most interesting questions posed in the first part is actually the same as the title of the report: counting what is measured or measuring what counts? In other words, are the criteria used in these league tables used because they are the most important determinants of quality or because those indicators are simply the ones that are (most easily) measurable? Not surprisingly, they find that:

The measures used by the compilers are largely determined by the data available rather than by clear and coherent concepts of, for example, ‘excellence’ or ‘a world class university’. Also the weightings applied do not always seem to have the desired effect on the overall scores for institutions. This brings into question the validity of the overall tables.

Several other points of critique – many of which have been discussed before, also in this blog – are confirmed in this part of the study. But the real value of the study is that it doesn’t stop here. It continues with an analyses of the survey and case studies to identify the ways in which these rankings actually shape policies. They find that institutions are indeed strongly influenced by league tables. One finding that I confirmed my expectations (see here and here) was about the link – and often contradiction – between league table criteria and other missions of the university:

League tables may conflict with other priorities. There is perceived tension between league table performance and institutional and governmental policies and concerns (e.g. on academic standards, widening participation, community engagement and the provision of socially-valued subjects). Institutions are having to manage such tensions with great care.

These are just a few quick observations. Read the full report! I will and probably post more about it at a later stage.

The most ambitious project to date to rank universities on education quality is the plan of the OECD to rank according to learning outcomes. Andreas Schleicher, the OECD’s head of education research explained this in the Economist in November last year:

“Rather than assuming that because a university spends more it must be better, or using other proxy measures for quality, we will look at learning outcomes”

Just as the OECD assesses primary and secondary education in their PISA assessment, it will sample university students to see what they have learned. Once enough universities are taking part, it may publish league tables showing where each country stands, just as it now does for compulsory education. This of course is a very ambitious project, if not over-ambitious. But at the same time, the OECD is probably one of the few international organisations that have the capacity and experience to assess educational outcomesat a (near) global level. Or not?

The Center for College Affordability and Productivity (CCAP) at the University of Ohio recently proposed an alternative ranking of US colleges and universities:

We at CCAP have long complained that most rankings of colleges are largely based on inputs used in providing services, things like the faculty-student ratio or the average SAT score of entering students. Better would to evaluate schools on either consumer satisfaction (like we evaluate most other things) or on the post-graduate achievements of the products of the education –the alumni.

The data for measuring consumer satisfaction come from the popular website ratemyprofessors.com. Today in their blog, CCAP research associates show what the ranking would look like if it would be solely based on the data from this site:

rankings are calculated by taking a weighted average of all faculty members at a university in the categories of: overall quality, average easiness (with ease treated as a negative quality), and average “hotness.”

And what is the result for the US national universities?

1. Boston College
2. Northwestern
3. Harvard
4. California Tech
5. Princeton
6. Samford
7. U. of Chicago
8. MIT
9. Wake Forest
10. Brigham Young University (BYU)
11. Brown
12. Yale
13. U. of Pennsylvania
14. Emory
15. Stanford

What surprised me in the list is that it is composed only of private universities. None of the prestigious public universities such as Berkeley, Michigan, Wisconsin, Virginia, North Carolina, etc., appear in the list. In addition to a ranking for national universities, they also present the top 15 of Liberal Arts Colleges. Interestingly, these show a higher consumer satisfaction than the research universities.

Of course…ratemyprofessors.com has received a lot of criticism and using it for a ranking brings along many problems. Nevertheless, the idea of including student satisfaction as a measure of quality, is not that strange. Student surveys have been used by the Dutch magazine Elsevier in their rankings and also is the basis for part of the data behind the CHE/Die Zeit rankings in Germany. And after all, it has been applied – albeit indirectly – as a measure since the very beginnings of the university:

One of the oldest universities in the world, the University of Bologna chartered in 1158 by Frederick I Barbarossa, was designed to cater to student desires. Students collectively hired professors, set tuition rates, evaluated and even dismissed low-quality instructors. While we have moved away from that model (perhaps somewhat for the better) student instruction remains the primary function of a university. Faculty research and other things distract from this goal. Our findings that students at liberal arts colleges are more satisfied with their professors than those at national research institutions is not surprising. Perhaps it is time that our national research universities shifted priorities more toward satisfying high paying customers. Since students are the main consumers of the university product, any ranking of schools should include a student satisfaction variable.

It will be interesting to see how the inclusion of teaching and learning in international rankings will develop. And in the longer term, it will even be more interesting to see how this will change universities. Will it indeed cause a shift in priorities toward satisfying (high paying) customers?

I gave a score of 200 for the number one university (Harvard) and 1 for the number 200 (RMIT; U of Cape Town) etc., and than aggregated these scores for every country. The graph below shows that the United States (with 57 universities in the top 200) and the United Kingdom (with 32 universities) are clearly superior to all other countries:

But of course, that can be expected for a country with more than 300 million inhabitants. So lets control for the population of each country. Clearly, the smaller countries like Hong Kong (ok…not exactly a country), Singapore and Switzerland are doing well, while the United States falls to a position somewhere in the middle. Populous Asian countries like Japan and China also fall behind.

Another way to look at country performance is by controlling it for GDP (PPP; IMF Data). The picture remains rather similar, although here New Zealand leads the pack.

So what can we conclude from all this? Well, considering the methodology of the ranking, not too much. My main conclusion would be that MS Excel 2007 makes nicer graphs than MS Excel 2003.

The full top 100 can be found here

For what it’s worth…

Update: Richard Holmes at the University Ranking Watch has plenty of coverage on the issue. See also University World News for a special issue on the THES ranking and rankings in general.

Related Posts:

International Rankings: a self-fulfilling nightmare?

Counting what is measured and measuring what counts

SJT World University Rankings 2008

Times Higher Education World University Rankings 2008

Topping the list this year was Alexander Rinnooy Kan, Chairman of the Social and Economic Council of the Netherlands (SER). His current profile lists chairmanships/memberships of a wide variety of boards and councils, ranging from multinationals to business schools and from museums to hospitals.

The list is assembled according to a network analysis by the Erasmus University in Rotterdam and uses a database of over 8,000 persons. The ranking is based on more than 5000 positions in 1098 bodies in 518 organisations in various sectors. The Queen, Dutch Ministers and Members of Parliament are excluded from the list. For a further explanation of the methodology, have a look here (in Dutch).

I decided to have a closer look at the top 100 and their connections to the Dutch higher education landscape. If we first have a look at the Alma Maters of these powerful individuals, the graph below shows that Leiden University (the oldest university in the Netherlands) has awarded the most degrees to the list. 19 out of 100 attended Leiden University. Interesting detail is that 15 of them studied Law in Leiden (making this club the low countries’ equivalent of the Skull & Bones??).

Of the 100 persons, 17 continued to get their PhD degree. Four of them were awarded in Rotterdam (three in economics, one in sociology), three in both Amsterdam and Leiden and the rest in Groningen, Delft, Tilburg and Twente.

But of course these graphs don’t control for the position in the ranking and they do not include the academic and leadership positions in universities or membership of the Board of Trustees (or Supervisory Boards / Board of Governors). So I decided to take these factors into account (for as far as they were available) in order to assess which university in the Netherlands is most powerful.

In addition to their degrees and PhD’s, I looked whether they work or used to work in a university as an academic or as a member of the executive board and whether they were a member of the Board of Trustees (Supervisory B./B. of Governors). In addition I used a multiplier according to the position in the ranking. Stats can be found here (pdf).

And the most powerful university is… Leiden University. With Rotterdam as a close runner up and Amsterdam and Utrecht in third and fourth position. The results (indexed for Leiden = 100) are in the graph below. For the people currently connected to these universities it of course remains the question whether they are there because of their power or wether they are so powerful because they are there…