This month, the eighth edition of the European Innovation Scoreboard was published. The European Innovation Scoreboard (EIS) has been published annually since 2001 to track and benchmark the relative innovation performance of EU Member States.

For the EIS 2008 the methodology has been revised and the number of dimensions increased to 7 and grouped into 3 main blocks covering enablers, firm activities and outputs (Figure 1). The purpose of this revision is to have dimensions that bring together a set of related indicators to give a balanced assessment of the innovation performance in that dimension. The blocks and dimensions have been designed to accommodate the diversity of different innovation processes and models that occur in different national contexts. A summary of the European state of innovation is given in the picture below (click to enlarge)

Based on their innovation performance across 29 indicators, EU Member States fall into the following four country groups:

• Sweden, Finland, Germany, Denmark and the UK are the Innovation leaders, with innovation performance well above that of the EU average and all other countries. Of these countries, Germany is improving its performance fastest while Denmark is stagnating.
• Austria, Ireland, Luxembourg, Belgium, France and the Netherlands are the Innovation followers, with innovation performance below those of the innovation leaders but above that the EU average. Ireland’s performance has been increasing fastest within this group, followed by Austria.
• Cyprus, Estonia, Slovenia, Czech Republic, Spain, Portugal, Greece and Italy are the Moderate innovators, with innovation performance below the EU average. The trend in Cyprus’ innovation performance is well above the average for this group, followed by Portugal, while Spain and Italy are not improving their relative position.
• Malta, Hungary, Slovakia, Poland, Lithuania, Romania, Latvia and Bulgaria are the Catching-up countries with innovation performance well below the EU average. All of these countries have been catching up, with the exception of Lithuania.  Bulgaria and Romania have been improving their performance the fastest.

The full report can be downloaded here

Filed in: Europe, Innovation, Ranking | 2 Comments »

What if I graduated from Amherst College or the Ecole Normale Supérieure de Lyon, I was not a European citizen and I would like to pursue a career in the Netherlands? Well, the Dutch government would say I was not educated sufficiently to enter the Netherlands. You might ask yourself why? Isn’t the ENS de Lyon a good school? Actually it is, according to its 72nd place in the 2006 THES rankings.

No, it’s not that ENS Lyon is a lousy university. It’s just that they ended up on rank number 157 of the Times Higher Education Ranking in 2007. And – as I feared before – nowadays in the Netherlands this means that you are not qualified as a skilled migrant. As is stated in the new immigration policy – entered into force this year – only some knowledge workers are eligible to enter the Netherlands:

From abroad, immigrants are only eligible if they graduated from a university in the top 150 of the two recognised university rankings, the league tables published in 2007 by the ‘Times Higher Education Supplement‘ and the ‘Jiao Tong Shanghai University’.

You might argue that ENS Lyon improved its ranking to 140 in 2008. But no…, the Dutch government uses the 2007 league tables. Period!

Or what if I had graduated from some of the world’s best liberal arts colleges? From Amherst, Williams, Swarthmore, Wellesley?

Not good enough…

And what if I had graduated from the University of Hokkaido (Japan)? The University of Notre Dame (USA)? Shanghai Jiao Tong University (China)? Universität Stuttgart (Germany)? University of Calgary (Canada)? Macquarie University (Australia)? Helsinki University of Technology (Finland)? RWTH Aachen (Germany)? University of Surrey (UK)? University of Barcelona (Spain)? Chalmers University of Technology (Sweden)? ….Bergen, Ottawa, Frankfurt, Brussels, Stockholm, Coimbra, Delaware?

No…not good enough…

Filed in: Higher Education, Netherlands, Ranking | 1 Comment »

Like last year, I tried to look at the Times Higher education university league tables from a national perspective. I gave a score of 200 for the number one university (Harvard) and 1 for the number 200 (the university of Athens) etc., and than aggregated these scores for every country.

The graph below shows that the United States and the United Kingdom are again superior in the Times rankings, followed by Australia and Canada. The Netherlands is the first non English speaking country, followed by Japan and Germany. The main difference however compared to last years results is that the number of countries represented in the top 200 has increased. The group is now joined by countries like Greece, Argentina, Thailand, Russia and India.

But of course…size matters and it’s easier to have many well performing universities in a large country than in a small country. So here is the result if we take population into account.

This of course works well for the small states like Switzerland, Hong Kong and Singapore. The Netherlands again comes fifth in line. If we control for GDP instead of population we get a similar picture. Here however, Hong Kong clearly outperforms the rest.

For what it’s worth….

Filed in: Higher Education, Ranking, World | 2 Comments »

Tomorrow’s that day that many university leaders dread. Have they gone up in the rankings or not? For some, rankings may even determine whether they will receive their bonuses or not. But most of all it’s the day for your Vice Chancellor or university president to criticize league tables even though secretly it’s the first thing he or she will check in the morning…

Yes, it’s time for the fifth edition of the Times Higher Education World University Ranking of 2008. Quacquarelli Symonds Ltd, the company responsible for the ranking, claims (again) that the methodology is improved. They are even so blunt to say that ‘the rankings have established themselves as an accepted benchmark of quality’. I beg to differ…

One issue at least seems to be resolved, that is the volatility of the THE ranking (compared for instance with the relatively stable Shanghai Ranking):

The final results will see more countries represented among the top 200 institutions, with Continental Europe beginning to make more of a mark than in previous editions. But there will be less volatility this year, thanks to the change in statistical methodology introduced in 2007. Single outliers no longer have a disproportionate effect on the overall ranking.

The world university ranking will be published here and here tomorrow morning…

Filed in: Higher Education, Ranking, World | 3 Comments »

I’m on my way back to The Hague, returning from the EAIR conference in Copenhagen. Although lots of interesting new studies and findings have been presented there (some of them I’ll discuss in later posts), I actually want to talk about a conference I visited last July in Berlin.

This conference (Transparency in Diversity – Towards a European Classification of Higher Education Institutions) presented the results from the second stage of the project Classifying European Institutions for Higher Education, a project that might turn out to have a major impact on European higher education policy. This project was initiated in 2005 (see this previous post) and is now supported by for instance the European Commission (DG Education) and the German Hochschule Rektorenkonferenz. It’s run by an international team led by Frans van Vught.

The project can be seen as a response to two trends (at least, that’s my interpretation). First of all, there is the emergence of the European higher education area, the objective of the Bologna process. If there’s one space, we need to know what types of institutions are occupying that space and hence, we need a classification or typology.

Secondly, there is the proliferation of ranking and league tables. As I’ve discussed many times before, these rankings present a very uni-dimensional view of the contemporary higher education institution. Basically they only look at the – science heavy – traditional research university. Through this they neglect the quality of a very wide range of other institutions which might be very good at the things they are supposed to do. Here one can think of mono-disciplinary institutions (e.g. colleges of fine arts; schools of economics and business), teaching oriented institutions (like the American liberal arts colleges) ore more professionally and vocationally oriented institutions (like the German and Austrian Fachhochschulen, the Dutch Hogescholen, etc.).

A multidimensional classification of European higher education institutions can on the one hand create more transparency in European higher education, while at the same time clarify which institutions can be compared with each other (so we can compare apples with apples and pears with pears). If you are interested in how they intend to do this, I suggest you have a look at the presentations of the conference. See Frans van Vughts presentation (PDF) to get a better idea about the background of the project and have a look at Frans Kaiser’s presentation (PDF) for the technical aspects of such a multidimensional classification.

What the classification will look like exactly is not yet clear. If it will remain limited to the web tool and the resulting radar graphs, I expect the effects to be rather limited. The question is whether the various stakeholders related to the project will ultimately define real categories of institutions (like the old Carnegie classification did). This however might give the project a more political character. Even though the project-team stresses that they will not create a hierarchical classification, it is interesting to see whether some categories will be perceived as more prestigious than others.

Nevertheless, the classification project seems to be widely supported by institutions throughout Europe and their representative organisations. The feeling that Europe needs to create more transparency is widely shared and at the same time, many institutions are looking for benchmarking opportunities with like-minded institutions. After all, comparisons with Harvard, Oxford and Yale are not very useful for most higher education institutions in Europe…

Filed in: Europe, Higher Education, Politics, Ranking | 1 Comment »

Tomorrow, the new 2008 Academic Ranking of World Universities will be officially published. Not surprisingly, it’s an almost all American affair. It’s rather interesting that the publication of the Shanhai Jiao Tong rankings almost goes by unnoticed, especially if you compare it to the publication of the Times Higher Education Supplement/QS World University Rankings (the THES-QS rankings 2008 will be published on 9 October).

This exactly is the strength of the SJT ranking. After all, universities are robust organisations and don’t change a lot in a years time. I guess it therefore corresponds with reality that the top 10 of 2008 is exactly the same as the one of 2007. Actually, not much has changed at all (although I of course did notice that the University of Sydney – my former employer – entered the top 100; the top 500 list is here).

2008(2007)		University
1 (1)		Harvard University
2 (2)		Stanford University
3 (3)		University California – Berkeley
4 (4)		University Cambridge
5 (5)		Massachusetts Inst Tech (MIT)
6 (6)		California Inst Tech
7 (7)		Columbia University
8 (8)		Princeton University
9 (9)		University of Chicago
10 (10)		University of Oxford

The main critique on the SJT rankings is that they only give an indication of a university’s research quality. They have only one proxy for teaching quality and that one isn’t exactly saying much about teaching quality at all. I have already pointed to some alternatives for these research biased rankings and league tables, for instance the new ranking being develop by CCAP (Center for College Affordability and Productivity).

This last one has now been published by Forbes Magazine. And yes…the criteria are very different than the ones we are used to:

1. Listing of Alumni in the 2008 Who’s Who in America (25%)
2. Student Evaluations of Professors from Ratemyprofessors.com (25%)
3. Four- Year Graduation Rates (16 2/3%)
4. Enrollment-adjusted numbers of students and faculty receiving nationally competitive awards (16 2/3%)
5. Average four year accumulated student debt of those borrowing money (16 2/3%)

And what’s the result?

2008	University
1	Princeton University
2	California Institute of Technology
3	Harvard University
4	Swarthmore College
5	Williams College
6	United States Military Academy
7	Amherst College
8	Wellesley College
9	Yale University
10	Columbia University

Compared with the SJT rankings, it are especially the liberal art colleges and the military colleges that are evident in the Forbes ranking. The high quality liberal arts colleges in the US (and elsewhere) are unfortunately lacking in nearly all international rankings. The reasons for this is of course again that these rankings are so research biased.

Another thing that I noticed after looking through the rest of the list is the relatively low standing of the public research universities. University of Virginia is the first one on 43, University of North Carolina at Chapel Hill at 66 and UC Berkeley at 73.This is probably due to another flaw in most rankings, that is that they measure the quality of the graduates without looking at the quality of the inputs. For more criticism on this ranking, see the comments on Vedder’s article in Inside HigherEd and the critical contribution of Patricia McGuire.

This challenge of actually measuring the added value provided by the university is taken up by the OECD’s AHELO project: assessing learning outcomes in higher education (sometimes referred to as the PISA for higher education). This exercise is still in it’s early stages and currently they are at the stage of studying the feasibility of such an exercise. And although the OECD explicitly does not want to promote it as a ranking, it might provide an alternative for the league tables.

Filed in: China, Ranking, USA, World | 6 Comments »

Just before going to a meeting on rankings I saw this. It is from the proposed new immigration policy: Blueprint for a modern migration policy (pdf; in Dutch). As in so many other immigration countries, it contains a chapter on skilled migration. Here is a translation of the passage that surprised me:

Anticipating the implementation of the new migration system, the government will at the latest in the first half of 2009 introduce a regulation for highly skilled immigrants. On the basis of the regulation, foreigners can stay in the country for a maximum of one year to find a job as a ‘knowledge migrant’ or to start an innovative company.

The objective of the regulation connects well to the ambition of the innovation platform to attract 1000 extra knowledge migrants. It is also in line with the advice on knowledge migrants of the Commission on Labour Participation in its report ‘towards a future that works‘.

The target group consists of foreigners that are relatively young and received their Bachelor, Master or PhD degree not longer than three years ago. Migrants are eligible if they received their degree from a university that is in the top 150 of two international league tables of universities. Because of the overlap, the lists consists of 189 universities…

And guess what the two league tables are. Yes, the Shanghai ranking and the Times Higher Education Ranking. Now…this will mean that firms like this have influence on who is eligible to come and work in the Netherlands. Something is not right here…

Filed in: Globalisation, Higher Education, Netherlands, Ranking | 3 Comments »

After the proliferation of accreditation bodies in the 1990s and 2000s, the sector witnessed the appearance of meta-accreditation. Do we – after the proliferation of rankings in the past 10 years or so – witness the first meta-ranking?

It looks like it, however I must admit it’s slightly different. It won’t be a meta-ranker, but more an accreditor of rankings. I’m talking about the establishment of the IREG–International Observatory on Academic Ranking and Excellence:

On April 18, 2008 an important decision was reached by the International Ranking Expert Group (IREG) to consolidate its partnership arrangement with the   creation of the IREG-International Observatory on Academic Ranking and Excellence.

Bob Morse, director of data research of one of the first and one of the most influential rankings – US NEWS & World Report – is one of the Executive Committee members and he writes on his blog Morse Code:

The International Observatory, headquartered in Warsaw, will conduct reviews of various ‘academic rankings’ and measures of ‘academic excellence’ to assess how well they serve higher education stakeholders and the general public. The observatory will use the recommendations formulated in the Berlin Principles on Ranking of Higher Education Institutions. Members of the body also will meet at the request of various ranking agencies to review their particular methodology criteria and standards. Ranking entities that receive observatory approval will be able to declare themselves ‘IREG Recognized’.

Especially that last item seems to point to an accreditor of rankings. But then, what gives IREG the authority to declare a ranking recognized or not? Well… at least they have some ‘recognized’ persons in their Executive Committee. Next to Bob Morse there are Gero Federkeil (CHE, Germany), Liu Nian Cai (Shanghai Jiao Tong University) and Alex Usher (Education Policy Institute, Toronto, Canada). The Committee is chaired by Jan Sadlak, the Director of UNESCO-CEPES in Romania. I wonder how this all will develop. And I wonder who will first get the ‘IREG-disapproved stamp’. Plenty of candidates…

Filed in: Higher Education, Ranking, World | Comment »

The Higher Education Funding Council published a report on the impact of rankings in the United Kingdom. It is probably one of the most extensive studies on ranking today. The study was conducted by the Centre for Higher Education Research and Information (CHERI) and Hobsons Research and is based on a survey of 91 higher education institutions in the UK and six institutional case studies.

The researchers looked at five rankings in particular, three national ones (Sunday Times Good University Guide, The Times Good University Guide, The Guardian University Guide) and two international rankings (Shanghai Rankings and the Times QS Ranking). The report itself and the background data are all available on HEFCE’s website.

Roughly, the study is divided into three parts. The first looks at rankings and their shortcomings in general. The second at the impact of rankings on universities in the UK. And the final part discusses alternative ranking methods such as the CHE ranking.

One of the most interesting questions posed in the first part is actually the same as the title of the report: counting what is measured or measuring what counts? In other words, are the criteria used in these league tables used because they are the most important determinants of quality or because those indicators are simply the ones that are (most easily) measurable? Not surprisingly, they find that:

The measures used by the compilers are largely determined by the data available rather than by clear and coherent concepts of, for example, ‘excellence’ or ‘a world class university’. Also the weightings applied do not always seem to have the desired effect on the overall scores for institutions. This brings into question the validity of the overall tables.

Several other points of critique – many of which have been discussed before, also in this blog – are confirmed in this part of the study. But the real value of the study is that it doesn’t stop here. It continues with an analyses of the survey and case studies to identify the ways in which these rankings actually shape policies. They find that institutions are indeed strongly influenced by league tables. One finding that I confirmed my expectations (see here and here) was about the link – and often contradiction – between league table criteria and other missions of the university:

League tables may conflict with other priorities. There is perceived tension between league table performance and institutional and governmental policies and concerns (e.g. on academic standards, widening participation, community engagement and the provision of socially-valued subjects). Institutions are having to manage such tensions with great care.

These are just a few quick observations. Read the full report! I will and probably post more about it at a later stage.

Filed in: Higher Education, Ranking, UK | Comment »

One of the main criticisms of international rankings is that they measure research quality rather than teaching quality. This is especially the case in for the Shanghai Jiao Tong Ranking. The THES Ranking uses proxies like employer surveys, student staff ratios and the number of international students in order to indicate education quality. The best known national university ranking is probably the one of the US News and World Report.  However, their proxies for educational quality (such as selectivity) can not be applied in a standardised global setting.

The most ambitious project to date to rank universities on education quality is the plan of the OECD to rank according to learning outcomes. Andreas Schleicher, the OECD’s head of education research explained this in the Economist in November last year:

“Rather than assuming that because a university spends more it must be better, or using other proxy measures for quality, we will look at learning outcomes”

Just as the OECD assesses primary and secondary education in their PISA assessment, it will sample university students to see what they have learned. Once enough universities are taking part, it may publish league tables showing where each country stands, just as it now does for compulsory education. This of course is a very ambitious project, if not over-ambitious. But at the same time, the OECD is probably one of the few international organisations that have the capacity and experience to assess educational outcomesat a (near) global level. Or not?

The Center for College Affordability and Productivity (CCAP) at the University of Ohio recently proposed an alternative ranking of US colleges and universities:

(more…)

Filed in: Ranking, USA, World | 6 Comments »

Ok…I seriously had the intention not to pay too much attention to the THES ranking this year. So this will be the last post about it (of course not the last post about rankings in general and their dynamics). I played around a bit with the data in Excel and had a look at it from a country perspective.

I gave a score of 200 for the number one university (Harvard) and 1 for the number 200 (RMIT; U of Cape Town) etc., and than aggregated these scores for every country. The graph below shows that the United States (with 57 universities in the top 200) and the United Kingdom (with 32 universities) are clearly superior to all other countries:

(more…)

Filed in: Higher Education, Ranking, World | 10 Comments »

I have probably written more than I should about rankings, and especially the Times Higher Education Supplement list and its flaws and shortcomings, but I just couldn’t resist… Here is a preview of this years results [last year between brackets]:

The full top 100 can be found here

For what it’s worth…

Update: Richard Holmes at the University Ranking Watch has plenty of coverage on the issue. See also University World News for a special issue on the THES ranking and rankings in general.

Related Posts:

International Rankings: a self-fulfilling nightmare?

Counting what is measured and measuring what counts

SJT World University Rankings 2008

Times Higher Education World University Rankings 2008

Filed in: Higher Education, Ranking, World | 19 Comments »

Dutch newspaper the ‘Volkskrant’ has published its annual list of the 200 most influential Dutch people. The list portrays the Dutch ‘old boys network’ that rules the Netherlands through its interpersonal networks and interlocking boards and directorates. Clearly, the strength of weak ties at work; but how are these ties connected to the Dutch universities?

Topping the list this year was Alexander Rinnooy Kan, Chairman of the Social and Economic Council of the Netherlands (SER). His current profile lists chairmanships/memberships of a wide variety of boards and councils, ranging from multinationals to business schools and from museums to hospitals.

The list is assembled according to a network analysis by the Erasmus University in Rotterdam and uses a database of over 8,000 persons. The ranking is based on more than 5000 positions in 1098 bodies in 518 organisations in various sectors. The Queen, Dutch Ministers and Members of Parliament are excluded from the list. For a further explanation of the methodology, have a look here (in Dutch).

I decided to have a closer look at the top 100 and their connections to the Dutch higher education landscape. If we first have a look at the Alma Maters of these powerful individuals, the graph below shows that Leiden University (the oldest university in the Netherlands) has awarded the most degrees to the list. 19 out of 100 attended Leiden University. Interesting detail is that 15 of them studied Law in Leiden (making this club the low countries’ equivalent of the Skull & Bones??).

(more…)

Filed in: Higher Education, Netherlands, Politics, Ranking | 1 Comment »

In the latest issue of the American Journal of Sociology, Wendy Nelson Espeland (Northwestern University) and Michael Sauder (University of Iowa) present an impressive paper on Rankings and Reactivity: How Public Measures Recreate Social Worlds. The paper shows how the rise of public measures change social behaviour by looking at the law school rankings of the US News and World Report (USNWR). It struck me how many of their findings and arguments can be applied to international rankings as well. In some cases, their arguments might even be stronger for international rankings and expose additional complications.

Through processes of what the authors call ‘reactivity’, the independence between the measures and the social world they target are threatened. Rankings thus not just measure the current situation; they define it as well by changing behaviour. They identify two mechanisms of reactivity that are important in this respect: ‘self-fulfilling prophecies’ and ‘commensuration’. Here, I will discuss the self-fulfilling prophecy mechanism and make an attempt to ‘translate’ this to the level of global higher education and global rankings.

(more…)

Filed in: Globalisation, Higher Education, Ranking | 6 Comments »

Here’s a typical example of how international university rankings directly influence university policies. This is a newspaper article about a national university, reported in a major national newspaper somewhere in the world:

“The university has started recruiting international undergraduate students in an effort to boost its image on a global scale. About 300 international students from various countries registered at the university here yesterday.

The Minister for Higher Education said recruiting international students was to improve its rank in world university rankings, such as the Times Higher Education Supplement’s (THES) World University Rankings.”

You could argue that it doesn’t make too much sense to focus on an indicator which only is 5% of the total score. But even more you could argue that a minister should base his objectives on national needs and national circumstances. Not base them on a widely disputed ranking which is characterised by flaws and errors.

Filed in: Higher Education, Ranking, World | Comment »