Beerkens' Blog » Social Science

Does past performance influence success in grant applications?

Eric — Mon, 13 Apr 2009 09:33:59 +0000

Last week, the Dutch Volkskrant reported on an interesting study on the distribution of research funding by the Netherlands Research Council (NWO). Loet Leydesdorff (one of the researchers that introduced the Triple Helix concept) and Peter van den Besselaar – both of the Amsterdam School of Communications Research of the University of Amsterdam – conducted a study on the grant allocation decisions of the Netherlands Research Council in the Humanities and Social Sciences in the Netherlands.

Besselaar and Leydesdorff tested whether the grant decisions correlate with the past performances of the applicants in terms of publications and citations, and with the results of the peer review process organized by the Netherlands Research Council

In their paper they show that the Council is successful in distinguishing grant applicants with above-average performance from those with below-average performance, but within the former group no correlation could be found between past performance and receiving a grant. When comparing the best performing researchers who were denied funding with the group of researchers who received it, the rejected researchers significantly outperformed the funded ones. Within the top half of the distribution, neither the review outcomes nor past performance measures correlate positively with the decisions of the Council.

The authors conclude with some questions for further research. They suggest a network analysis of applicants, reviewers, committee members, and Council board members. This might provide an answer to the question whether funding is correlated to the visibility of the applicants within these networks. After all, in the social process of granting proposals many processes play a role, apart from scholarly quality: bias, old-boys’ networks and other types of social networks, bureaucratic competencies, dominant paradigms, etc., all play an important role in selection processes.

If my reading of the paper is correct, it might also point to a discrepancy between the grant decision makers and the international academic community. If we consider that metrics (past performance) and peer review very much emerge in international networks and the grant distributors make decisions contradicting the metrics and peer review, what does that tell about the Council members’ involvement in these international networks?

The paper will be published later this year in the journal Research Evaluation.

The Business School Business in Europe, Asia and the US

Eric — Sun, 20 Jul 2008 11:34:09 +0000

The Financial Times featured an interesting article from business guru Charles Baden Fuller. Professor at the Cass Business School of the City University, London. He observes a decrease in the gap between management research between the US and other regions like Europe and Asia. Although he acknowledges the supremacy of the US in the field, he says that the US share of management research will fall below 50 percent the next few years:

Research output in management is still concentrated: less than 3 per cent of the world’s universities produce more than 70 per cent of global output. Of these 214 universities, 126 are in the US, 13 in Canada, 57 in Europe and 18 in Asia and elsewhere. But comparative world positions have been changing quickly. My research** reveals how the world’s academic business research output has become more dispersed.

While Wharton and Harvard are still the best by a margin, Europe now accounts for 25 per cent of international research output. Its best schools – London Business School, Rotterdam School of Management (Erasmus), Insead and Tilburg are in the global top 30. Asian schools – in China and Singapore especially – are further behind but their stock is rising even faster.

While some of the best US schools admit privately to being worried, publicly they stress their continued dominance – at least, according to their data. But their measurements overemphasise past successes, ignore current trends and importantly use narrowly based research measures, looking only at material published in US journals and ignoring the fact that important new ideas are increasingly being published in highly regarded, peer-reviewed non-US publications.

The interesting part is his explanation for the rise of European and Asian management research. He claims they are more innovative in their approaches and engage more in cross border comparative work than their Colleagues in the US. Another factor is that European and Asian researchers seem to focus more on micro issues where US academics emphasise macrostatistical trends.

I have always admired the US management research and think they have produced some of the most interesting and sophisticated social science studies in the past decades. Not just in the field of economics but especially in sociology where many of the recent breakthroughs have come out of business schools. At the same I indeed found them to be very US centric. I think this is related to their emphasis on macrostatistical trends. If the priority is on the cleanliness of data sets and the complexity of the modeling, than comparative studies are just a nuisance. But of course, social sciences can not be just about data and models, it’s also about reality. And the reality is after all becoming less tidy, more global and less US centred…

Eronomics 101

Eric — Thu, 08 Nov 2007 01:38:07 +0000

Isn’t economics wonderful? It gives answers to all important questions in life. It even provides the tools for ‘understanding the preferences underlying the search for a mate’. Or in other words, an economist goes to a bar and solves the mysteries of dating.

At a local bar just off the Columbia campus, Raymond Fisman ran a speed-dating experiment with two psychologists, Sheena Iyengar and Itamar Simonson, and fellow economist Emir Kamenica. Some of their findings confirm the well known clichés, stereotypes and prejudices, other findings are more surprising:

We found that men did put significantly more weight on their assessment of a partner’s beauty, when choosing, than women did. We also found that women got more dates when they won high marks for looks from research assistants, who were hired for the much sought-after position of hanging out in a bar to rate the dater’s level of attractiveness on a scale of one to 10.

By contrast, intelligence ratings were more than twice as important in predicting women’s choices as men’s. It isn’t exactly that smarts were a complete turnoff for men: They preferred women whom they rated as smarter—but only up to a point. In a survey we did before the speed dating began, participants rated their own intelligence levels, and it turns out that men avoided women whom they perceived to be smarter than themselves. The same held true for measures of career ambition—a woman could be ambitious, just not more ambitious than the man considering her for a date.

When women were the ones choosing, the more intelligence and ambition the men had, the better. So, yes, the stereotypes appear to be true: We males are a gender of fragile egos in search of a pretty face and are threatened by brains or success that exceeds our own. Women, on the other hand, care more about how men think and perform, and they don’t mind being outdone on those scores.

A forthcoming paper by the authors is also taking race into account:

Women of all the races we studied revealed a strong preference for men of their own race: White women were more likely to choose white men; black women preferred black men; East Asian women preferred East Asian men; Hispanic women preferred Hispanic men. But men don’t seem to discriminate based on race when it comes to dating. A woman’s race had no effect on the men’s choices.

Two wrinkles on this: We found no evidence of the stereotype of a white male preference for East Asian women. However, we also found that East Asian women did not discriminate against white men (only against black and Hispanic men). As a result, the white man-Asian woman pairing was the most common form of interracial dating—but because of the women’s neutrality, not the men’s pronounced preference. We also found that regional differences mattered. Daters of both sexes from south of the Mason-Dixon Line revealed much stronger same-race preferences than Northern daters.

Summarising: men don’t discriminate on race in their search for female companionship, as long as the woman looks good and does not pose a threat to their ego.

EIT and Policy Research

Eric — Sun, 06 May 2007 00:49:00 +0000

A few weeks ago, I discussed a study of Luc Soete and Peter Tindemans on the feasibility of the European Institute of Technology. On the basis of a comprehensive analysis, they concluded that the decentralized EIT that has been proposed by the Commission was not feasible. It is too dispersed; it would not increase significantly the research output in a field; it cannot match a top tier university in providing an environment for training graduates; and a dispersed institute cannot adequately organize technology transfer. As an alternative, they suggested a clustered model for an EIT. Food for thought, you would think…

In the last weekend of April, EU competitiveness ministers backed a German EU presidency initiative on gradual progress towards a European Institute of Technology. In a public hearingCommissioner Figel said that it was time for the initial EIT plans to reach a conclusion. He claimed that there is a positive momentum now: “either we get it now or it’s lost”.

Obviously I was surprised to read nothing about the Soete/Tindemans study in the report of the hearing. As far as I could see, the design and organisation of the EIT presented in the hearing was exactly the same as the one suggested by the Commission before the study was published. This is all the more surprising considering that the research was conducted for a committee of the European Parliament. Of course government bodies are not obliged to follow the recommendations of reports that they have commissioned. But you would expect that it would at least be taken into consideration, especially since the authors are well known and respected researchers in this field.

This seems to be a typical example of the political (ab)use of policy research and policy analysis. If the results and recommendations are politically opportune and correspond with the politicians objectives they are praised and heralded as ground breaking landmark studies. If not, let’s just neglect them and get on with what we planned.

You would at least hope that decision makers on research policies in Europe would take research seriously…

Reith Lectures 2007

Eric — Tue, 10 Apr 2007 23:51:00 +0000

Today the BBC starts another episode in their Reith Lecture Series. The BBC has broadcasted the series since 1948. The Reith lecture series were initiated by Sir John Reith, the first director general of the BBC. He maintained that broadcasting should be a public service which enriches the intellectual and cultural life of the nation. In its long history the series have covered a wide range of topics in the sciences and social sciences. The first Reith lecturer was philosopher Bertrand Russel, speaking about the Authority and the Individual. In economics and the social sciences it has featured names like Arnold Toynbee, John Kenneth Galbraith and, more recently, Anthony Giddens on the Runaway World. Lectures are available online since 1999, but the BBC has also put some historic lectures online.

This years Reith lecturer will be Jeffrey Sachs, Director of the Earth Institute at Columbia University. He will give a series of five lectures related to global challenges that the world is facing:

Lecture 1: Bursting at the Seams
The 21st century will be marked by severe natural resource limits, the rise of new economic powers and the threats of failed states. These are tectonic changes with the potential to unleash global-scale upheavals. Global cooperation of an unprecedented depth and scale will be needed but we are not yet prepared for such cooperation.

Lecture 2: Survival in the Anthropocene
The biggest challenges that we face – climate change, alleviation of hunger, water stress, energy – are translated in the shadow of ignorance into “us versus them” problems, with only the weakest links to underlying scientific principles and technological options.

Lecture 3: The Great Convergence
Power and America have seemed synonymous for the last fifty years. No longer. Power in the 21st Century is shifting to the East: to India and above all to China. Facing up to the end of centuries of North Atlantic dominance – first Europe then the U.S. – will pose huge challenges.

Lecture 4: Poverty in the Midst of Plenty
This lecture considers the challenges of extreme poverty and the extreme worry of the rest of the world which fears for its own prosperity. It spells out the limits of the free market to solve these problems and proposes a plan of action which presents choices to those listening.

Lecture 5: A New Politics for a New Age
The key political novelty of our age is mass political awareness and mobilization. Mass mobilization has brought the Age of Empire to an end, and accounts for the failures in Iraq. No society any longer tolerates being ruled by another. Social mobilization can be a dramatic force for positive change.

You can listen to the lectures in streaming audio or download the mp3 files or transcripts. Lecture audio and transcripts will be available after each broadcast. Each lecture will be available as an MP3 download for 7 days after the first broadcast.

Still Bowling Alone

Eric — Tue, 11 Jul 2006 04:21:00 +0000

While staying in the US last month, I heard an interview on the radio with Lynn Smith-Lovin, a sociology professor at Duke. The interview was about a study she co-authored: Social Isolation in America: Changes in Core Discussion Networks over Two Decades.

The study of 1,467 adults showed that one-fourth of all Americans report that they have nobody to talk to about personal matters. Another quarter reports they are just one person away from nobody. It seems that, in addition to bowling alone, more and more Americans now also have to cope alone with their personal problems.

My first thought was that this would especially be the case for the higher educated part of society since they in general spend more time on work and also tend to be more mobile. I was therefore surprised to hear that non-whites and people with less education tend to have smaller networks than white Americans and those with higher educational levels.

It also made me wonder about the ‘American-ness’ of this isolation problem. Is it a trend that can mainly be seen in the US, or would other countries and regions show similar outcomes (do welfare states lead to more social cohesion and less isolation or not; is this problem also apparent in developing countries)? Does anyone know about a comparative study on this issue?

The full article, published in the American Sociological Review, can be found here.

Publishing & Open Access

Eric — Mon, 15 May 2006 03:09:00 +0000

Two related issues on the US academic publishing business were widely reported upon in the media in the last 2 weeks. The first was the National Institutes of Health policy on public access to research findings. The second, the proposal of a bill by Republican Senator Cornyn (Texas) and Democratic Senator Lieberman (Connecticut) requiring public access to federally funded research.

On February 3, 2005, the National Institutes of Health (NIH) announced a Policy on Enhancing Public Access to Archived Publications Resulting from NIH-Funded Research. Although the NIH strongly encourages that a manuscript be made available to other researchers and the general public immediately after it has been published in a journal, the Policy allows an author to delay the manuscript’s release for up to 12 months. Participation in the Public Access Policy is voluntary. The rate of submission to the system in the first 8 months has been less than 4 percent of the total number of articles estimated to be eligible.

The Chronicle however reports that momentum continues to build outside the NIH, and outside the United States, for mandatory posting of manuscripts in centralized free online repositories. In April, the European Commission released a report (pdf) calling for a guarantee of free access to all publicly sponsored research.

But in May, the two senators from Connecticut and Texas introduced a bill that would require every federal agency that sponsors more than $100-million annually in research to establish an online repository and make its grantees deposit their articles within six months of publication. The bill would apply to 11 agencies, including the NIH, the National Science Foundation, and NASA.

“It will ensure that US taxpayers do not have to pay twice for the same research – once to conduct it and a second time to read it,” Senator Cornyn told Congress.

Obviously, this proposal ignited a fierce reaction from the scientific publishing industry. Representatives from the publishers come with all kind of reactions:

Science addresses this issue:

Some publishers argue that there’s no evidence the public is as interested in, say, high energy physics papers as in health research. “You’re just expanding this willy-nilly on the assumption that there’s the same clamor,” says Allan Adler, vice president for legal and governmental affairs for the Association of American Publishers. Martin Frank, executive director of the American Physiological Society, argues that if the bill became law, it could be especially damaging to “small niche area” journals in disciplines such as ecology that have not yet experimented much with open-access journals that recoup publication costs from authors rather than subscribers.

And so does the New York Times:

Scientific data is easily misinterpreted, said Joann Boughman, executive vice president of the American Society of Human Genetics, publisher of The American Journal of Human Genetics. “Consumers themselves are saying, ‘We have the right to know these things as quickly as we can.’ That is not incorrect. However, wherever there is a benefit, there is a risk associated with it.”

And the Washington Post:

Patricia S. Schroeder, president and chief executive of the Association of American Publishers, promised a fight. “It is frustrating that we can’t seem to get across to people how expensive it is to do the peer review, edit these articles and put them into a form everyone can understand,” Schroeder said. [Isn't the peer review something that academics do...for free...? Ed.]

And the Guardian:

But the Association of American Publishers warned that the law would jeopardise the integrity of the scientific publishing process. Association member Brian Crawford warned it “would create unnecessary costs for taxpayers, place an unwarranted burden on research investigators, and expropriate the value-added investments made by scientific publishers, many of them not-for-profit associations who depend on publishing income to support pursuit of their scholarly missions”.

I guess there are a lot of vested interests here.. The bill will probably discussed later this year. It would be about time for some fundamental changes in the publishing industry. To me it remains a strange phenomenon that an academic writes an article or book for free, then his or her colleagues do the peer review for free and then (often after 2 years or so) they have to pay to get (on-line) access to the articles or books. Or do I fail to see something here?

All in 1 week

Eric — Sun, 07 May 2006 13:25:00 +0000

In the past week, three remarkable men have passed away. The best writer of all times, one of the most innovative artists of all times and one of the most influential economists of all times.

Pramoedya Ananta Toer (1925-2006) passed away last Sunday (30 April).

For me, his numerous books, short stories and essays are the most remarkable works I have ever read. Both his use of language and his choice of topics make that his books and stories portray a lively picture of Indonesian societies and cultures. Toer brought history to live, from the early Majapahit kingdom to the first stages of colonialism, from the first movements towards independence to the repression of the Suharto regime. I wrote a short post on is work before. Here is my top 5 of his work:

1. Gadis Pantai (The Girl from the Coast, 1962)
2. Buru Quartet: Bumi Manusia (Earth of Mankind, 1980); Anak Semua Bangsa (Child of all Nations, 1980); Jejak Langkah (Footsteps, 1985) and Rumah Kaca (The Glass House, 1988)
3. Korupsi (1954)
4. Keluarga Gerilya (The Guerrilla Family, 1950)
5. Arus Balik (1995)

—-

Karel Appel (1921-2006) passed away last Wednesday in Zurich.

Appel was probably the best known contemporary Dutch painter. He was one of the founders of the COBRA group, a group of painters from Copenhagen, Brussels and Amsterdam, allied with abstract expressionism.

—-

John Kenneth Galbraith (1908-2006) passed away on Saturday 29 April.

From the Economist:

“At six foot eight, he was a giant. Intellectually he was equally towering, a man who spent more than seven decades either on the stage of American public policy – as a bureaucrat in Franklin Roosevelt’s New Deal, a confidante of John Kennedy and adviser to countless other Democrats – or loudly lambasting Washington from offstage left, as a Harvard professor.”

And a well known quote:

“There are two classes of forecasters: those who don’t know, and those who don’t know they don’t know”

Academic Champions League?

Eric — Wed, 29 Mar 2006 12:11:00 +0000

In the latest Higher Education section of The Australian it is all about research assessment. The Australian Government has planned to introduce a Research Quality Framework (RQF) which is largely based on the UK Research Assessment Exercise (RAE). The RAE is a peer review exercise to evaluate the quality of research in UK higher education institutions. The introduction of the RAE has improved universities’ research performance (in terms of impact of publications) and created greater research concentration.

It’ s a rather strange moment to introduce the RQF because the UK has plans to abolish the RAE and return to a system that looks more like the current Australian model. In the words of Snitch:

“So Britain is scrapping its research assessment exercise just as Australia prepares to introduce one. What’s more, Britain is returning to a metric system of measuring quality, just like the one Australia uses now.”

One very visible result of the RAE is the concentration of research. Obviously this gives less concerns to Australia’s leading universities (joined in the Group of 8) than to the other players. It’s expected that most of the research funding will be concentrated in these 8 research intensive universities.

Although the framework has not been implemented yet, some of the consequences are already visible in anticipation of the RQF:

“In a significant loss for RMIT University, a leading expert in biomedical sciences has left the campus, taking his entire staff of 15, his laboratory and research grants worth nearly $1 million a year to a research quality framework-free medical institute. As universities prepare for greater competition under the framework, global diabetes specialist Mark Febbraio has announced he will leave RMIT for the Baker Heart Research Institute in Melbourne, blaming the impending introduction of the RQF and its effect on universities outside the Group of Eight.”

Many of the universities outside the group of 8 complain that criteria for societal and economic relevance are missing in the framework, and this will even increase the diversion of funds away from the technological universities to the Group of 8. The concentration of research will likely lead to a ‘bidding war on stars‘. An Australian equivalent of a European Champions League, where the Barcelonas, the Milans, the Arsenals and the Inters will always be in the semi-finals because they can afford to buy the best players?

On the other hand, maybe not:

“The plan to introduce a national assessment system for research quality has stalled after federal Education Minister Julie Bishop announced yesterday she was setting up another advisory group to consider it.”

Globalisation: 99 Definitions & Perspectives

Eric — Tue, 14 Feb 2006 01:58:00 +0000

While I was looking for a file in my computer I stumbled upon an old document. It’s a file with a list of different perspectives and definitions of globalisation that I assembled for my doctoral research some years ago. I thought it might be of useful for students and scholars that are trying to grasp the possible meanings of the term.

It is a list of 99 (give or take a few) views from different disciplines and different sectors. Most are from academics, ranging from anthropologists to economists and from philosophers to business gurus. It includes statements from people as diverse as Bill Gates, Karl Marx and Vandana Shiva and organisations ranging from Greenpeace to the World Bank.

I converted the list into a website that can be found here (pdf also available).

If you think any perspectives should be added, let me know..