Radio Freethinker

Vancouver's Number 1 Skeptical Podcast and Radio Show

The Peer Review Process: Science by Committee

Posted by Jenna Capyk on June 24, 2011


When we talk about “real” or “rigorous” scientific research, most of the time we’re talking about “peer reviewed” work. This is not, in fact, work done under the supervision of some stodgy British gentleman, but rather results that have been scrutinized by the ever-venerated “peer review process.” But what exactly is the process of peer review and why is it a valuable tool to use both as a publishing scientist and as a consumer of that material? Read on and discover an important sieve to separate the baby from the bathwater.

The general idea behind peer review is to have scientific findings scrutinized by a separate group of researchers before being published. The theory is that by having scientists in the appropriate field of study critically review the methodology, results, and conclusions presented in a paper, this work can be brought to its best possible form. The peer review process is also designed to act like a filter, allowing quality work to be published and more dubious findings to be thrust back onto the desktop of some desperate graduate student.

In practice, the peer review process operates at the level of the academic publication; individual scientists submit papers for publication and the editors give it a (prolonged) glance and send it out for peer review. The “peers” in this process are just that, two to three principal investigators (read: head scientist in the lab) whose work is related to that under review. These reviewers then make polite comments, cutting remarks, or irrelevant musings on the document and send these reviews back to the editor. The author is forwarded the comments, makes revisions and responses, and sends everything back to the editor who makes the final decisions. There are different models of peer review such as open review, review following online publication, and non-anonymous review, all of which are currently under review themselves. The predominant model in use right now, however, involves anonymous, invited review before (and acting as the deciding factor in) publication of the manuscript.

This level of scrutiny by so many experienced and learned minds must always result in publication of the very best science, right? Right? Well, most of the time. Our more astute and realistic readers may have already spotted several flaws in this process. These stem almost exclusively from the fact that although each person involved in this process is a bona fide scientist, they are also humans. This means that there are human aspects, not only of error, but of baser instincts and emotions that can creep into the practice. Conflict of interest is a major culprit. With so much of the scientific community dominated by intense competition for funding and notoriety, it can be hard to give one’s greatest rivals a fair kick at the can, and that’s often whose work will come across your desk. Scientists are encouraged to declare such conflicts of interest and decline to review research too closely related to their own, but the anonymity of the reviewers can sometimes provide enough leeway that such issues slip through the cracks. Confirmation bias can also be a major factor, with researchers subconsciously unwilling to accept results contrary to their own hypotheses.

Despite the potential pitfalls, the peer review process is still a very effective way of assessing the quality of work presented for publication, and ensuring that substandard research, or papers making claims beyond the reach of the experimental evidence, never see the glossy cover of a respected journal. It also acts like a seal of quality for a reader. There are plenty of ridiculous papers that are published in non-peer reviewed places, and knowing the process a piece of work went through to reach your eyeballs can be a valuable way to judge its likely merit.

Like everything we H. sapiens come up with in society, the peer review process has problems. It is, however, normally a very effective way to ensure quality, well-supported research is separated from less stellar attempts or plain fiction. Not everyone can be an expert in everything, so at the end of the day it’s pretty nice to have a handful of true experts acting as our personal scientific skeptics.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s