Last year, a paper came out in the journal Science that made waves in the scientific community and the public at large (it was even voted the #5 most publicly cited paper of 2015 by Altmetric).

So why am I writing about a paper over a year old and one that has already received plenty of public attention? Well, because this paper is a huge deal. Why you may ask? One of the most important words in the entire scientific enterprise: Replication.

In brief, this study was done by the Open Science Collaboration (OSC), a massive consortium of hundreds of researchers. The OSC team attempted to replicate 100 research papers in the experimental psychology field. The scientists decided which studies were to be included in the analysis, shared methods and tools, and agreed on criteria for how they would critically evaluate the studies (in some cases, the original authors provided the original test materials).

Unfortunately, the results were not good. The authors found that 2/3 of the original findings could not be replicated with any degree of statistical confidence. When taken as group, the effect sizes of the 100 replication studies compared to the originals were only about half as a great. In other words, more than half of the 100 papers could not be replicated.

But what does this mean and why is it a big deal?

Science is all about uncovering “how stuff works” but at a far more fundamental level, what science really is, is a method or system to figure out “how stuff works” and “what causes what” and uncovering the underlying principles of nature, etc.

And how does a scientist know that why they discovered about how this or that works is actually real? Well, one way is for other scientists to run the same experiment. If they get the same result, then it’s a pretty good chance then the discovery is “real”.

One of the biggest challenges the scientific research field has been grappling with is this issue of replication and ability to replicate other people’s work. If what people are reporting in papers is real, then why are so many findings so difficult to replicate?

There’s about a million ways to answer that question but the simplest answer is that doing science is really, really hard. Even if you think you designed the perfect experiment, collected the best data you could, and analyzed it the best way you know how, you might have gotten something wrong. You might have forgotten to control for a variable that you never thought of or even more mundanely, you forgot to report some crucial detail that other scientists need to know but you have taken for granted.

The failure to replicate is NOT about making up results (though a few bad apples have done that) is about not having time and money to thoroughly consider the results of the field. Science has a way of weeding out ideas that just don’t hold water but it requires other scientists to delve into the work of their colleagues and try to expand on their initial colleagues.

And just to be clear, there’s plenty of outstanding work being done that has been replicated and is scientifically solid.

Regardless, scientists need to resolve how to solve this problem with replication.

As bad as it is can be in biology, it’s a whole lot worse for a “soft science” like psychology. Many psychological studies have either been discredited or shown to be outright frauds (one of the more sensational stories involved years of forged data by the psychologist Diederik Stapel).

Thankfully, the field as a whole is trying to acknowledge their past failings and improving the integrity of their discipline. It would be a huge step forward for other fields, such as in the biomedical research field, to also take on such an endeavor.

And in the end, this is why this paper is so ground breaking and worth talking about (again). The field acknowledged they had a problem, did a systematic analysis of all available studies, and tried to replicate which ones are good and which are bad.

But there’s one more layer to this too. There’s also no incentive to replicate findings either. The pressure to publish only “sexy” results and get the big research grant almost prohibits scientists from trying to replicate each others work.

As someone who has spent that last 10+ years in academic research labs, I’ve heard the concerns from friends and colleagues about how quickly they need to publish their results out of fear of being “scooped” by a competing lab working on the same topic. And I and anyone in the academic research knows the the near constant anxiety about how to come up with new exciting ideas for the big grant that your entire livelihood is dependent upon (maybe a little over-dramatic but seriously, only a little).

If a scientist is under pressure to publish a new finding as quickly as possible, sometimes mistakes are made or a critical control was overlooked on accident. One facet of the replication crisis may be this competitive drive between labs. In business, competing tech companies are pressured to release a product that may be cheaper or more appealing to the public. However, competition has the exact opposite effect in scientific research. Increased competition may actually hurt scientists. And of course, the root cause of competition to publish is competition for a limited pool of grant money, without which there would be no basic research at all.

The replication study is an important milestone and idealizes the self-correcting tradition of the scientific enterprise in general. Scientists are supposed to be the most critical of their own work and the community should be able to recognize if an initially exciting finding cannot be replicated.

With increased funding and reduced pressure to publish only “sexy” results in top-tier journals, perhaps the scientific community will turn away from competition and prestige and return to the spirit of openness, sharing, and collaboration. Maybe then the failure to replicate will become unable to be replicated.