Go to original

Science stifled? Why peer review is under pressure
By Clive Cookson and Andrew Jack
Financial Times (U.K.)

June 11, 2008

When the US National Institutes of Health unveiled plans last week to spend $1bn over the next five years on "high-risk, high-impact transformative research", it marked the latest effort by academic funders and publishers alike to tackle growing concerns about a process at the core of scientific progress: peer review.

At least since the Syrian physician Ishaq bin Ali al-Rahwi described more than a thousand years ago how a local medical council verified doctors' compliance with treatment standards, peer review has played a role in science. It has become increasingly important in recent years as a means of rationing grants and publications in the face of a proliferation of academic research.

Peer review guards the gates at both ends of the research process – obtaining money to carry out a project and publishing the results in a journal. Specialists in the field, working individually or as panels, identify flaws and assess the importance of the work; the reviewers' identity is normally withheld from the author.

But the process is under assault from critics who say it is ineffective at filtering out poor research, while it perpetuates predictable work at the expense of more imaginative thinking. In the long run we all suffer, argues Don Braben of University College London, because economic growth depends on unpredictable scientific advances.

In response to the perceived constraints of the process, researchers are using the internet to open up new ways of publishing that streamline peer review – or do away with it altogether – while several of the worworld's biggest science funding bodies are making radical changes in the way they assess research proposals.

Last week 25 distinguished scientists wrote to the Financial Times lamenting funding agencies' failure to back research "at the margins where unpredictable and transformative discoveries are made". They called for a global fund to support inspired scientists, free of peer review.

Then the Royal Society, Britain's national academy of sciences, said it was putting together a pilot scheme for a "blue skies" research fund, which would avoid the constraints of conventional peer review by using a generalist panel to consider proposals from any field, on the basis of their novelty and potential to open up new areas of science and technology.

This spring, the Bill & Melinda Gates Foundation, the world's richest charity, unveiled a $100m (€64m, £51m) "agile, acccelerated grant-making process" called Grand Challenges Explorations. "We've got to get around peer review – it'ss anathema to innovation," says Tachi Yamada, the foundation's head of global health. "Innovation has no peers, by definition."

Criticism of peer review in funding and publishing is nothing new. Drummond Rennie, professor of medicine at the University of California, San Francisco, wrote an article in the Journal of the American Medical Association in 1986 that triggered the birth of an entire sub-discipline scrutinising peer review, with regular conferences and a stream of papers.

He wrote at that time: "There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or unjustified, and no grammar and syntax too offensive for a paper to end up in print."

"Peer review is a rather arbitrary black box," agrees Richard Smith, author of a book on its failures and former editor of BMJ, the British medical journal. "It's very slow, expensive, a considerable lottery, completely hopeless at detecting errors and fraud, and there's evidence of bias."

He says scrutiny by editors and peer reviewers reinforces "publication bias", with articles claiming a clinical trial is effective getting into print while those showing a negative result do not. That in turn creates a problem for "meta-analysis", which attempts to understand whether a medicine works by combining all data.

Dr Smith now edits the recently launched online journal Case Reports, which provides minimal pre-publication scrutiny of medical articles. The classic peer review criteria of originality or importance are avoided, with all submissions accepted if they are understandable and complete. "We simply have not begun to use the possibilities of the web," he says.

Other journals have taken a similar route, such as several published online by the Public Library of Science (Plos) that assess technical content but do not judge the importance of a new study.

But few are willing to abandon the process entirely. "Peer review is the gold standard," says Mark Patterson, Plos director of publishing. "It's still the best process we have for judging the rigour of the scientific process. There are efficiencies to be made, but you can't skimp on it if you want to publish serious science."

What a growing number of journals do online – including Plos One andd BMJ through its " rapid responses" – is to encourage readers to comment after publication, providing a form of retrospective peer review and rating.

Many scientists in physics, maths and related subjects go still further. They post papers without any peer review on arXiv.org, an international archive of electronic papers maintained by Cornell University, New York, which now contains nearly 500,000 "e-prints" – and then allow others to challenge them.

However, John Dainton, physics professor at Liverpool University, concedes: "Sometimes we want the fastest publication possible, but there is always a certain pride in publishing in the most highly cited, highly regarded journals."

Linda Miller, executive editor of Nature, agrees that scientists continue to seek publication in prestigious journals to enhance their own standing. They also concentrate on reading the best-regarded ones, precisely because their time is precious. "You want to be directed, to use the best journals as a filtering device," she says. "I have been an editor for more than 20 years and I have handled a lot of papers. Every single one has been improved by peer review."

Although many publishers, authors and readers are reluctant to dispense with peer review entirely, there are several more practical ways under discussion to improve the process. The first is to eliminate bias as far as possible from reviews, which may otherwise recommend against publication because of personal rivalries – or snobbery about researchers working outside thhe most prestigious institutions.

Some journals such as BMJ identify the names of reviewers in an effort to hold them accountable, although this may deter them from being frank if they risk "retaliation" from authors. Others have tried "double blind" reviews that shield the identity of both sides, greater discussion between reviewers, and feedback or even ratings on their comments.

A second concern is speed of publication. When six volunteers testing an experimental medicine at London's Northwick Park Hospital suffered severe side-effects in March 2006, there was enormous interest in understanding what had gone wrong. But it took five months before the details were made public when the doctors who saved their lives published their experiences in the New England Journal of Medicine – itself a relatively rapid turnaround in academic publishing.

Mr Patterson argues that delays are often the result of "publishing inefficiency" as much as flaws in peer review: a draft article may be submitted to a prestigious journal, rejected and then be offered sequentially to others, each with new reviewers further delaying publication.

The same thing happens increasingly with grant applications that are rejected by one funding agency, rejigged slightly and then sent on to a chain of others. The solution here must be more collaboration and referral between journals and between funders, and greater sharing of the same reviews to speed the process.

A final issue is the capacity of reviewers to ensure both high quality and swift opinions. "One reason why the whole system is creaking is 'referee fatigue'," says Alf Game, deputy head of science at Britain's Biotechnology and Biological Sciences Research Council. In most fields, numbers of research proposals and scientific papers are rising faster than the number of active scientists qualified to review them. The UK research councils receive more than twice as many proposals as they did 20 years ago.

"The demand for referees is increasing," says Dr Game. "The BBSRC is a relatively small funding body, yet we need 10,000 individual referee comments every year to look at 2,500 applications. We have to ask many more people to comment before we get one to agree."

Today's referees are much more likely than their counterparts 20 years ago to be working in the same narrow area as the author of the research proposal or paper, and have an interest in keeping in touch with competitors' activities. But these self-interested referees cannot take such a broad and objective view of the work.

Since funds are limited, some evaluation of people or proposals is necessary. The question is how to do it. "The key is to use peer review more creatively, because you can't get away from having to make choices," says Mark Walport, director of the Wellcome Trust, Britain's largest research charity. "One thing is to give researchers more time. We have extended the length of our [fellowships] to five years. The important thing then is to choose the best scientists, on the basis of the spark they show on interview rather than the details of their proposed projects."

The Howard Hughes Medical Institute has just committed $600m to give 56 scientists "the opportunity to tackle their most ambitious, risky research plans" after holding its first open, web-based competition. Thomas Cech, its president, says: "They will receive unfettered long-term funding of about $1m a year, free of reporting requirements."

Even the European Union, once notorious for its bureaucratic funding process, has come up with a simplified reviewing procedure for the new European Research Council, which will spend 7.5bn Euros over seven years to support scientific excellence regardless of nationality.

"We have thrown away a lot of the old baggage and we are being experimental, rather than accepting the way things were done in the past," says Fotis Kafatos, ERC president. "For example, we have insisted that our review panels are small and largely generalist. There is a general sense among the heads of major research organisations ... that the peer review process is too conservative."

After more than two decades debating peer review, Prof Rennie says the issue is far more complex than he had originally thought. He believes that no better alternative has emerged, nor is there proof that it has stifled innovation. "I'm the most sceptical person, but what would we do if we had no peer review?"

Peer review may not be immortal, and may be experimenting with different forms, but it looks set to guard the gates of research for some time to come.

(In accordance with Title 17, Section 107, of the U.S. Code, this material is distributed without profit to those who have expressed a prior interest in receiving the included information for research and educational purposes. New Energy Times has no affiliation whatsoever with the originator of the original text in this article; nor is New Energy Times endorsed or sponsored by the originator.)

"Go to Original" links are provided as a convenience to our readers and allow for verification of authenticity. However, as originating pages are often updated by their originating host sites, the versions posted on New Energy Times may not match the versions our readers view when clicking the "Go to Original" links.