On the use of rankings and league tables

Posted by Eric on July 1st, 2008

Just before going to a meeting on rankings I saw this. It is from the proposed new immigration policy: Blueprint for a modern migration policy (pdf; in Dutch). As in so many other immigration countries, it contains a chapter on skilled migration. Here is a translation of the passage that surprised me:

Anticipating the implementation of the new migration system, the government will at the latest in the first half of 2009 introduce a regulation for highly skilled immigrants. On the basis of the regulation, foreigners can stay in the country for a maximum of one year to find a job as a ‘knowledge migrant’ or to start an innovative company.

The objective of the regulation connects well to the ambition of the innovation platform to attract 1000 extra knowledge migrants. It is also in line with the advice on knowledge migrants of the Commission on Labour Participation in its report ‘towards a future that works‘.

The target group consists of foreigners that are relatively young and received their Bachelor, Master of PhD degree not longer than three years ago. Migrants are eligible if they received their degree from a university that is in the top 150 of two international league tables of universities. Because of the overlap, the lists consists of 189 universities…

And guess what the two league tables are. Yes, the Shanghai ranking and the Times Higher Education Ranking. Now…this will mean that firms like this have influence on who is eligible to come and work in the Netherlands. Something is not right here…

Is the UK going Down Under?

Posted by Eric on June 18th, 2008

During my years in Sydney, the issue of language skills and foreign students has come up repeatedly. The claim was that the financial reliance on foreign students had forced Australian higher education to accept students that lack even the basic English language and communication skills.

Most critical on this issue is probably Bob Birrell, Director of the Centre for Population and Urban Research at Monash University in Melbourne. Last year he published a study finding that one in three overseas students which were granted permanent residency after graduating from an Australian university does not have good enough English to handle a professional job.

An analysis of government visa testing, the first of its kind, found 34 per cent of 12,116 graduating international students who received permanent residency in 2005-06 did not have the English standard needed to be admitted to university, let alone to be awarded a degree. For students from China, the fastest growing international student market for Australian universities, the proportion with poor English leapt to 43 per cent.

The question is of course: how did they get into an Australian university anyway? And even more: how did they ever get a degree? With respect to the second question, Birrell claims that universities dealt with the poor English language skills of their students by lowering teaching and assessment standards. On the question of how they get in, Birrell has another explanation.

Applicants for a higher-education student visa must score at band-six level, rated as “competent”, under the International English Language Testing System, if based overseas when they apply. But international applicants can avoid the testing by basing themselves in Australia earlier to complete either year 12 or an intensive language course. Dr Birrell found that about 40 per cent of overseas students followed this path.

Professor Peter Abelson – a visiting scholar at the University of Sydney at that time – summarised the issue correctly:

“These figures are a very stunning result, but not entirely surprising to people who are in tertiary education.”

Former Minister of Education, Julie Bishop, and former president of the Australian Vice-Chancellors Committee Gerard Sutton basically denied the problems. Sutton didn’t accept that there is a problem in universities in terms of soft marking of international students. Julie Bishop rejected claims that a large number of foreign students graduating from Australian universities have poor English skills:

“Australian universities only enroll foreign students once they have achieved international standards of language proficiency. This has been an extraordinary attack by Professor Birrell on our universities. International students must meet international benchmarks in English language in order to get a place at a university in Australia.”

The denial of the problem is astonishing. Yes of course, there is a lot of money involved and the stakes in international education are high. If the international student market would plummet, so would much of the Australian higher education sector. But denying the problem while more and more foreign graduates fail in their job search because of their language skills, does obviously not help in the long run.

And now the debate has moved up north…

>> continue reading >>

Metaranking

Posted by Eric on June 17th, 2008

After the proliferation of accreditation bodies in the 1990s and 2000s, the sector witnessed the appearance of meta-accreditation. Do we - after the proliferation of rankings in the past 10 years or so - witness the first meta-ranking?

It looks like it, however I must admit it’s slightly different. It won’t be a meta-ranker, but more an accreditor of rankings. I’m talking about the establishment of the IREG–International Observatory on Academic Ranking and Excellence:

On April 18, 2008 an important decision was reached by the International Ranking Expert Group (IREG) to consolidate its partnership arrangement with the   creation of the IREG-International Observatory on Academic Ranking and Excellence.

Bob Morse, director of data research of one of the first and one of the most influential rankings - US NEWS & World Report - is one of the Executive Committee members and he writes on his blog Morse Code:

The International Observatory, headquartered in Warsaw, will conduct reviews of various ‘academic rankings’ and measures of ‘academic excellence’ to assess how well they serve higher education stakeholders and the general public. The observatory will use the recommendations formulated in the Berlin Principles on Ranking of Higher Education Institutions. Members of the body also will meet at the request of various ranking agencies to review their particular methodology criteria and standards. Ranking entities that receive observatory approval will be able to declare themselves ‘IREG Recognized’.

Especially that last item seems to point to an accreditor of rankings. But then, what gives IREG the authority to declare a ranking recognized or not? Well… at least they have some ‘recognized’ persons in their Executive Committee. Next to Bob Morse there are Gero Federkeil (CHE, Germany), Liu Nian Cai (Shanghai Jiao Tong University) and Alex Usher (Education Policy Institute, Toronto, Canada). The Committee is chaired by Jan Sadlak, the Director of UNESCO-CEPES in Romania. I wonder how this all will develop. And I wonder who will first get the ‘IREG-disapproved stamp’. Plenty of candidates…

Peer reviewing as a hidden subsidy

Posted by Eric on May 29th, 2008

Even though I left academia recently, I try to keep in touch with the academic community and keep up with academic publications in the field. Because of this, I still accept most requests from journal editors to review their contributions. What annoys me however, is the fact that I do not have access to such journals anymore. My current employer has subscriptions on some journals, but definitely not on all the ones I am interested in. Considering the subscription fees of most journals I can’t blame them.

Today I read an interesting article in the Times Higher Education on the costs of publishing and… the actual costs of peer reviewing. It does make you wonder about the current system of peer reviewing, especially if you don’t have access to the articles you review. I used to agree with the observation of the THE that the advancement of the academy’s collective body of knowledge was reward enough for the time and effort put into peer review and therefore academics needn’t get paid for doing this. But now I see what we are missing out on, I am not so sure anymore:

But a new report has attempted to quantify in cash terms exactly what peer reviewers are missing out on. It puts the worldwide unpaid cost of peer review at £1.9 billion a year, and estimates that the UK is among the most altruistic of nations, racking up the equivalent in unpaid time of £165 million a year.

The report says there would be a “significant transfer” of funds to academics if peer reviewers were paid. But such a move would drive up journal prices, with the estimated “breakeven price” of a major discipline journal jumping 43 per cent, leaving libraries with a bigger bill.

My first reaction to this was that there must be other possibilities with all the new technologies available. And there are. The report also shows that a move to electronic-only publishing would bring a fall of about £1 billion (12 per cent) in global costs. A system of author-pays open access publishing could add another saving of £556 million. I don’t have to get paid to review articles. But some changes to the system would be appreciated.

Market share and competition

Posted by Eric on May 14th, 2008

In the Dutch weekly journal ESB (Economic and Statistical Reports), economists from the universities of Groningen and Rotterdam presented an interesting article. Their starting assumption is that high student evaluations will have a positive effect on the market share of universities. After all, if a programme in a particular university is highly ranked by students, more students will chose this particular university to attend that programme.

The authors collected six year of student evaluations where students rate their programmes on a scale of 1-10  (as published annually in the Dutch weekly magazine Elsevier). Market share for each programme/university combination was calculated by dividing the number of students in programme X in university Y by all students in the Netherlands in programme X. When the evaluation of the programme is compared with the market share, we get the following graph:

marketshare

On the basis of this finding (and the results of a simulation that they run), they present some interesting conclusions. I won’t go in details, but one of them is that there is no clear relation between evaluation and market share. Hence, students have other criteria than quality in the choice of where they will attend university (especially location, and in particular the distance to their place of residence).

In their discussion, the authors indicate that this shows that competition on the basis of quality is not really taking place in the Netherlands. One reaction could be that the government should enable universities to compete on the basis of price. In other words, universities should be able to set their own tuition fees. This is currently not allowed for Dutch and EU students and the government decided last week that this will not be possible in the near future.

I could pose an alternative hypothesis on the interpretation of the results. The results show that students don’t take quality of education into account in their decision-making process. This could indicate that students don’t purchase a service (a high quality education), but a product (a degree). Universities should therefore emphasise the quality of their degrees, not the quality of their education. If the value of the degree differs per university (reflected in the opportunities to enter the labour market in higher positions or on a higher remuneration), the students will take this into account. This then would mean that universities are better of improving their position in rankings, attracting more prestigious scholars (like Nobel laureates) and increase their budget for marketing. This will bring them a in a better competitive position. On the other hand… maybe economics doesn’t provide answers for everything…

Erjen van Nierop, Peter Verhoef, Philip Hans Franses. Studie Evaluaties en marktaandelen van universiteiten (subscription required). Economisch Statistische Berichten, 4 April 2008.

The Trisakti Tragedy: 10 years later

Posted by Eric on May 12th, 2008

It’s been exactly ten years today since the Trisakti tragedy took place in Indonesia and the last  remaining foundations of Suharto’s 30 year rule started crumbling down. During the month of May in 1998, student demonstrations against Suharto were organised everywhere and several incidents occurred on campuses all over the Indonesian archipelago. But the events on May 12 at Trisakti university shocked many and directly led to the fall of Suharto nine days later.

On Tuesday May 12, at around 10.30, thousands of students gathered for a peaceful demonstration on the campus of Trisakti University, located between the airport and downtown Jakarta. On campus, a free speech forum was organised where students, academics and other speakers voiced their opinions. Students were allowed to demonstrate on campus but Suharto had explicitly prohibited all public demonstrations. Nevertheless, the students left campus and marched into the direction of the parliament at 1.30 in the afternoon.

trisaktiAlthough most students demonstrated peacefully, the rally turned violent later in the afternoon. The students were blocking the traffic at one of the main arteries of Jakarta. The security forces stopped the thousands of students while they were on their way to the Parliament building. For hours there was a stand off between the students on the one hand and the police and security forces on the other. Somewhere around 5 PM, the students negotiated a solution. They agreed that one row of students would back off for every row of police that did the same. But then suddenly the security forces started shooting with rubber and live bullets to the students who were running back to the campus grounds. At the end of the day, four students lost their lives. Elang Mulya, Hafidin Royan, Hendriawan Sie and Hery Hartanto later became ‘the heroes of the reformation’.

The tragedy remains surrounded by mysteries. There has been talk about infiltrators of the security forces, disguised as Trisakti alumni, who provoked the peaceful demonstrators. Rubber bullets were used by the security forces, but it remains unclear where the live ammunition came from. The fact that the air was filled with teargas only added to the confusion. The Trisakti Tragedy was followed by one of the cruelest events in Indonesian history. In the two days after the events at Trisakti University, thousands were killed in the May 1998 riots. The mysteries surrounding these events and the involvement of government forces also remain unsolved.

Many investigations have been conducted into the Trisakti tragedy and the events afterwards, but many questions remain. Calls upon the Indonesian president Susilo Bambang Yudhoyono and on the Indonesian judiciary to reopen the investigations have until now remained unanswered. Maybe the tenth anniversary of these events would be a good occasion to review this decision…

Counting what is measured or measuring what counts?

Posted by Eric on April 9th, 2008

The Higher Education Funding Council published a report on the impact of rankings in the United Kingdom. It is probably one of the most extensive studies on ranking today. The study was conducted by the Centre for Higher Education Research and Information (CHERI) and Hobsons Research and is based on a survey of 91 higher education institutions in the UK and six institutional case studies. hefce

The researchers looked at five rankings in particular, three national ones (Sunday Times Good University Guide, The Times Good University Guide, The Guardian University Guide) and two international rankings (Shanghai Rankings and the Times QS Ranking). The report itself and the background data are all available on HEFCE’s website.

Roughly, the study is divided into three parts. The first looks at rankings and their shortcomings in general. The second at the impact of rankings on universities in the UK. And the final part discusses alternative ranking methods such as the CHE ranking.

One of the most interesting questions posed in the first part is actually the same as the title of the report: counting what is measured or measuring what counts? In other words, are the criteria used in these league tables used because they are the most important determinants of quality or because those indicators are simply the ones that are (most easily) measurable? Not surprisingly, they find that:

The measures used by the compilers are largely determined by the data available rather than by clear and coherent concepts of, for example, ‘excellence’ or ‘a world class university’. Also the weightings applied do not always seem to have the desired effect on the overall scores for institutions. This brings into question the validity of the overall tables.

Several other points of critique - many of which have been discussed before, also in this blog - are confirmed in this part of the study. But the real value of the study is that it doesn’t stop here. It continues with an analyses of the survey and case studies to identify the ways in which these rankings actually shape policies. They find that institutions are indeed strongly influenced by league tables. One finding that I confirmed my expectations (see here and here) was about the link - and often contradiction - between league table criteria and other missions of the university:

League tables may conflict with other priorities. There is perceived tension between league table performance and institutional and governmental policies and concerns (e.g. on academic standards, widening participation, community engagement and the provision of socially-valued subjects). Institutions are having to manage such tensions with great care.

These are just a few quick observations. Read the full report! I will and probably post more about it at a later stage.

University rankings and customer satisfaction

Posted by Eric on April 1st, 2008

One of the main criticisms of international rankings is that they measure research quality rather than teaching quality. This is especially the case in for the Shanghai Jiao Tong Ranking. The THES Ranking uses proxies like employer surveys, student staff ratios and the number of international students in order to indicate education quality. The best known national university ranking is probably the one of the US News and World Report.  However, their proxies for educational quality (such as selectivity) can not be applied in a standardised global setting.

The most ambitious project to date to rank universities on education quality is the plan of the OECD to rank according to learning outcomes. Andreas Schleicher, the OECD’s head of education research explained this in the Economist in November last year:

“Rather than assuming that because a university spends more it must be better, or using other proxy measures for quality, we will look at learning outcomes”

Just as the OECD assesses primary and secondary education in their PISA assessment, it will sample university students to see what they have learned. Once enough universities are taking part, it may publish league tables showing where each country stands, just as it now does for compulsory education. This of course is a very ambitious project, if not over-ambitious. But at the same time, the OECD is probably one of the few international organisations that have the capacity and experience to assess educational outcomesat a (near) global level. Or not?

The Center for College Affordability and Productivity (CCAP) at the University of Ohio recently proposed an alternative ranking of US colleges and universities:

>> continue reading >>

International Student Tribes and Territories

Posted by Eric on March 19th, 2008

As you might know, I changed countries and positions in the last month, hence my lack of posts (now and probably in the very near future). As some of you might know, I left academia to work at the Nuffic, the Netherlands Organisation for International Cooperation in Higher Education. This week I started in the department of studies of their Knowledge and Innovation Directorate. One issue I’m currently looking at is the way foreign students choose their preferred study destinations. Market research organisation i-Graduate seems to have found all my answers already… or hasn’t it?

The Guardian reports on a study that looks at what motivates international students to study abroad and what influences their choice of study. I haven’t seen the study and I can’t link to it because it is not available on their website, but looking at the article of the Guardian, it seems to be a case of over-simplification and over-generalisation. According to i-Graduate, the international student population can be divided into five tribes:

igraduate

>> continue reading >>

Sydney Places I Liked

Posted by Eric on February 20th, 2008

One more week in Sydney and then I’ll return to the Netherlands. It’s been three years since I first set foot on Australian soil to start my postdoc at the University of Sydney.  And those were three good years. That was of course because of the great Aussie people, but it also had to do with Sydney’s great places. Here are a few that I will definitely miss…

First an outer Sydney location. Actually, I love all of them, simply because Sydney is surrounded by beauty. At the East there are of course Sydney’s world famous beaches. Manly, just north of Sydney Harbour and Bondi and Coogee south of it. And many more, between kuringgaiPalm Beach in the north  and Cronulla in the South. South of Sydney is the Royal National Park, with a beautiful scenic coastal walk (and some scary snakes). Inland, on the Western fringes of Sydney, the flat land turns into mountains, …Blue Mountains. Just a two hour train ride from bustling Sydney. But my favorite outer side of Sydney must be in the North: beautiful Kuringgai Chase National Park, with its lovely islands and scenic bays.

operahouse

From outer Sydney to inner Sydney. Not surprising, nor original; my favorite spot here seems to be the favorite spot of every tourist visiting the city: Sydney Harbour. With its beautiful Australian icons: the Opera House and the Harbour Bridge. It’s a place I’ve seen many times of course, but somehow I am still amazed by the stunning views, especially Sydney Harbour by night.

quadrangle Closer to home is a site I’ve seen many many times: the campus of the University of Sydney. Australia’s oldest university, founded in 1850. Despite a few horrible postwar makeshift buildings, it is a beautiful campus with the lovely quadrangle as its Oxbridge-like center. Unfortunately, It’s a bit of a construction site at the moment, but by 2010 it should all be up and running again.

newtown But the place I will miss most, without  any doubt, is Newtown. Three years ago, I was immediately captured by this suburb in Sydney’s Inner West and by its bustling artery, King Street. Daily morning walks to Sydney Uni through King Street always are a lively start of the day.  Despite the noisy traffic, King Street’s many coffee shops fill up every morning with Newtown’s Latte lovers, enjoying their big brekky or Vegemite sandwich. In the evenings Newtown’s chardonnay socialists seek refuge in King Street’s many bars and countless restaurants, appreciating their Thai, Lebanese, Greek, Italian, African, Turkish, Vietnamese, Korean, Malay, Macedonian, Indian, Mexican, Japanese, Chinese, Spanish or Nepalese dishes.

In an earlier post I cited a lonely planet quote on Newtown, one that is still very true today:

“a melting pot of social and sexual subcultures, students and home renovators. King Street, its relentlessly urban main drag, is full of funky clothes stores, bookshops and cafes. Newtown comes with a healthy dose of grunge, and harbours a decent live music scene.”

“a swag of funky cafes and restaurants lining King Street offer an interesting introduction to the suburbs community life”

If you ever visit Sydney, don’t forget these places. I definitely won’t!

Weird Science: Spouses & Physical Similarities

Posted by Eric on February 17th, 2008

wsConvergence in the physical appearance of spouses

by Zajonc, R.B., Adelmann, P.K., Murphy, S.T., & Niedenthal, P.M. (1987) 

Full Text Available in Motivation and Emotion

This study attempted to determine whether people who live with each other for a long period of time grow physically similar in their facial features. Photographs of couples when they were first married and 25 years later were judged for physical similarity and for the likelihood that they were married. The results showed that there is indeed an increase in apparent similarity after 25 years of cohabitation. Moreover, increase in resemblance was associated with greater reported marital happiness. Among the explanations of this phenomenon that were examined, one based on a theory of emotional efference emerged as promising. This theory proposes that emotional processes produce vascular changes that are, in part, regulated by facial musculature. The facial muscles are said to act as ligatures on veins and arteries, and they thereby are able to divert blood from, or direct blood to, the brain. An implication of the vascular theory of emotional efference is that habitual use of facial musculature may permanently affect the physical features of the face. The implication holds further that two people who live with each other for a longer period of time, by virtue of repeated empathic mimicry, would grow physically similar in their facial features. Kin resemblance, therefore, may not be simply a matter of common genes but also a matter of prolonged social contact.

Weird Science is a new item on Beerkens’ Blog. It presents a peculiar, remarkable, eccentric, extraordinary, unconventional, atypical, strange, funny, odd or bizarre study. In other words: a case of Weird Science.

Another Campus Shooting…

Posted by Eric on February 15th, 2008

Once again, there has been a shooting at a university campus in the US. On February 14, a gunman killed five students at Northern Illinois University. The killer died of a self-inflicted gunshot wound. He had been a graduate student in sociology at the university but was no longer enrolled. Sadly, the Northern Illinois shooting is part of a long list of random or semi-random shootings on university and college campuses:

USA / 2008 - February 14: Five people are killed when a man opens fire in a classroom at Northern Illinois University near Chicago, including the gunman who killed himself.

USA / 2007 – September 21: eighteen-year old student Loyer D. Braden shot two seventeen year old Delaware State University students from Washington, D.C.

USA / 2007 - April 16: A gunmen kills 32 people and himself and wounds 15 others at Virginia Tech University in the deadliest campus shooting in U.S. history.

Canada / 2006 - September 13: Kimveer Gill opens fire on the street and inside the college in Montreal’s Dawson College, killing one student and injuring 19 others. Gill kills himself after a battle with police.

USA / 2002 - October 28: Robert Flores, a forty year old failing student of the University of Arizona Nursing College, walks into an instructor’s office and fatally shoots her. A few minutes later, he enters one of his nursing classrooms and kills two more of his instructors before fatally shooting himself.

Australia / 2002 - October 21: Huan Xiang, 37, an honors student at Monash University in Melbourne, shoots and kills two students and wounds five other people.

>> continue reading >>

Meanwhile in Czar Putin’s Russia

Posted by Eric on February 12th, 2008

Meanwhile in Czar Putin’s Russia a university was forced to close down. The St. Petersburg European University - a non-governmental institution offering post-graduate programs in the humanities and social sciences - was told to close its doors because of safety concerns. Officially the closure was because of violation of fire protection regulations.

But university employees tell that the university has been closed because of political reasons, reports Kommersand, Russia’s daily online newspaper. The fire inspection came to the university after deputies of the State Duma, members of United Russia party, and the general Prosecutor’s office had taken an interest in an EU-funded program for training election observers.

The respective three-year program was launched in early 2007 and funded by the EU grant of €673,000. Authorities lashed out at it from the beginning, saying the money was appropriated not for some research work but for creating a net of observers here and viewing it as an attempt of direct interference into Russia’s election campaigns of 2007 to 2008.

putin Andrei Yurov, a human rights and education expert, said that closing a university for breaching fire regulations looks at least strange. According to him, Russia hardly has any universities meeting all standards of fire protection.

A week ago, Putin placed limitations on foreign election because he perceived it as foreign influence on the upcoming presidential election. This led the OSCE (Organization for Security and Co-operation in Europe) to pull out of monitoring the March 2 elections.

Global Classrooms in the Desert

Posted by Eric on February 11th, 2008

Both the International Herald Tribune and the New York Times bring an article by Tamar Lewin on universities rushing to set up outposts abroad. It presents an illustrative overview of the risks, benefits and the viability of institutional globalisation in higher education. If, after reading the article, you are left with any pressing questions, the NYT gives you the opportunity to pose them dirteclty to Charles E. Thorpe, the dean of Carnegie Mellon in Qatar (ht: globalhighered). To get you started, here are some interesting quotes that provide food for thought:

Howard Rollins, the former director of international programs at Georgia Tech, which has degree programs in France, Singapore, Italy, South Africa and China, and plans for India:

“Where universities are heading now is toward becoming global universities. We’ll have more and more universities competing internationally for resources, faculty and the best students.”

Susan Jeffords, vice provost for global affairs of the University of Washington, about the increase in demand for higher education from overseas students:

“It’s almost like spam”

>> continue reading >>

Free Press and Democracy

Posted by Eric on February 10th, 2008

Here’s an example of how democracy will not function without free press. In the wake of the upcoming elections, Second Finance Minister of Malaysia, Tan Sri Nor Mohamed Yakcop, explains in the government controlled media why Malaysians should again put their trust in the Barisan Nasional government led by Prime Minister Badawi:

“the country’s per capita income had risen by 40% between 2004 and 2007, from RM15,819 (US$4,163) to RM22,345 (US$6,452). The Barisan Nasional Government is confident that we will get the people’s mandate again, based on the improved economic resilience”

A good thing there is something called the internets, where people can voice other truths. Tony Pua over at Philosophy Politics Economics explains:

Nor Mohamed Yakcop must either be completely out of his mind, or can no longer perform simple Mathematics or worse, attempting to insult the intelligence of ordinary Malaysians. Malaysia’s Gross Domestic Product (GDP) grew by 5.0%, 5.9% and an estimated 6.0% in 2005, 2006 and 2007 respectively according to the Government’s official statistics.

Based on the above growth rates over the past 3 years, Malaysia’s GDP grew by approximately 17.9% from 2004 to 2007. Therefore, it is completely inconceivable that our per capita income increased by 40% when our GDP grew by only 17.9%. Unless of course, the honourable Minister believes that our population shrunk by some 16%!

But, then again, there will always be people who use blogs on the Internet to criticise the rapid economic growth achieved by the Government.

UPDATE: Prime Minister Abdullah Ahmad Badawi dissolved the parliament on Wednesday 13 February. The election is likely to be held in early March

UPDATE 2: Elections are called for 8 March. According to the Economist:

No one expects Mr Badawi to repeat his storming debut in 2004, when he led the ruling coalition to a 90% sweep of 219 seats in Parliament. Defeat is unthinkable: the coalition has won every election since independence in 1957.