46

I read and heard a lot about the Reproducibility Initiative recently, claiming that the data of many scientific studies cannot/was not/is not be reproduced.

“In the last year, problems in reproducing academic research have drawn a lot of public attention, particularly in the context of translating research into medical advances. Recent studies indicate that up to 70% of research from academic labs cannot be reproduced, representing an enormous waste of money and effort,” said Dr. Elizabeth Iorns, Science Exchange’s co-founder and CEO. “In my experience as a researcher, I found that the problem lay primarily in the lack of incentives and opportunities for validation—the Reproducibility Initiative directly tackles these missing pieces.”

Unfortunately I was not able to find those studies (where these reproduced!?) proving this statement. I want to know where these studies where carried out, medicine, biology, psychology, but couldn't find anything. I'm also somehow skeptical that science is in that bad shape, considering that studies are often used/mandatory here on skeptics.se for good answers and to get license for pharmaceutical products. 70% looks a bit too high to me.

Oddthinking
  • 140,378
  • 46
  • 548
  • 638
Hauser
  • 1,933
  • 1
  • 17
  • 25
  • 7
    I really don't find this all that shocking, no ones pays money for people to go find out things that are already known. There is no fame in being the fact checker, nobody gets a Nobel Prize for "best recreation of previous research." – Ryathal Nov 05 '12 at 20:59
  • 3
    Ryathal, while I am sympathetic to your view it is not completely true. When the Italians claimed that neutrinos traveled faster than light a flurry of experiments were done to reproduce the results. See here for the details: http://news.sciencemag.org/scienceinsider/2012/02/official-word-on-superluminal-ne.html?ref=hp – thisfeller Nov 05 '12 at 21:24
  • 3
    You normally become the fact checker if you have solid claims that the results of a given study are either erroneous or incomplete. It's no wonder everyone tried to reproduce the italians' experiment, since faster than light speeds are thought to be impossible (further tests revealed that there are no significant differences between the speed of light and that of neutrinos either). Some discoveries are just so potentially groundbreaking that they are immediately going to be verified. Others, not as flamboyant, can be somewhat expected by scientists and will often be taken at face value. – Dungarth Nov 05 '12 at 21:50
  • 1
    @thisfeller The OPERA guys never believed that number and they phrased the paper *very* carefully (it's worth reading to see *exactly* what the claim). They'd been sitting on it while they tried to figure it out, but it leaked and then they *had* to say something. In any case, in particle physics we tend to reproduce the results of the nth generation machine as part of commissioning the n+1st generation, so there is a strong expectation that your work will be put to the test. On the other hand we have some anomalous results in our history, too. – dmckee --- ex-moderator kitten Nov 05 '12 at 23:39
  • 1
    We do heavy-ion research, Tier-2 for the latest from the LHC. – thisfeller Nov 06 '12 at 02:34
  • There is a related question, http://skeptics.stackexchange.com/questions/1973/are-most-published-results-relying-on-measures-of-statistical-significance-likel, an the answers here have a strong bearing on the answers to the other question. – matt_black Nov 06 '12 at 08:27
  • @Dungarth Interesting story in this context: http://en.wikipedia.org/wiki/Sch%C3%B6n_scandal http://www.salon.com/2002/09/16/physics/ – Hauser Nov 07 '12 at 12:37
  • The title of this question is misleading/lurid. It is speculative to base the claim that up to 70% of all scientific studies would not be reproducible on the claim that 70% of medical studies were irreproducible. – Konsta Nov 08 '12 at 14:22
  • 1
    Some of the references below and more [are listed on Wikipedia](https://en.wikipedia.org/wiki/Reproducibility#Reproducible_research). See in particular [Why Most Published Research Findings Are False](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/) – Relaxed May 02 '14 at 06:13
  • 1
    "Up to" is weasel wording. They're saying that anything from 0% to 70% of results can't be reproduced. That's trivially true -- and utterly meaningless. If you're looking for the actual number, ask that question. – keshlam Aug 29 '14 at 03:23
  • A scientific study found that the results of about 70% of all scientific studies could not be reproduced. Subsequent studies could not reproduce this percentage. – SQB Feb 24 '16 at 14:16
  • Wouldn't "lack of incentive or opportunities" as a reason mean that 70% ARE NOT, instead of "can not"? – PoloHoleSet Apr 26 '17 at 22:11

3 Answers3

50

ALS Therapy Development Institute re-tested 70+ drugs from 221 independent studies:

  • 0 reproduced (1)
  • Minocycline: effective in four separate ALS mouse studies worsened symptoms in a clinical trial of more than 400 patients (2)

National Institute of Neurological Disorders and Stroke (NINDS) conducted sponsored replication of 12 spinal cord injury studies:

  • 2/12 successfully reproduced (3)

Bayer conducted in-house target validation studies

  • 14/67 reproduced (4)

Amgen attempted to reproduce 53 “landmark” oncology publications:

  • 6/53 reproduced (5)

References

  1. Scott et al. Amyotroph Lateral Scler. 9, 4-15 (2008).
  2. Gordon et al. Lancet Neurol. 6, 1045–1053 (2007).
  3. Stuart et al. Experimental Neurology 233, 597–605 (2012).
  4. Prinz et al. Nat Rev Drug Discov. 10, 712 (2011).
  5. Begley and Ellis. Nature. 483, 531-3 (2012).
Community
  • 1
Elizabeth Iorns
  • 524
  • 5
  • 2
  • Thanks Dr. Iorns for the quick and detailed reply, that are the references I was missing on many blogs and news sites explaining your initiative recently. I would put those references on your website, so it becomes clear it's esp. a major problem in medical sciences. – Hauser Nov 05 '12 at 22:04
  • This would also make a good answer to http://skeptics.stackexchange.com/questions/1973/are-most-published-results-relying-on-measures-of-statistical-significance-likel – matt_black Nov 06 '12 at 08:28
  • 2
    Also raises questions about evidence-based medicine, if the evidence has been cooked: http://skeptics.stackexchange.com/questions/3111/do-physicians-that-use-the-paradigm-of-evidence-based-medicine-achieve-better-he – mmr Nov 06 '12 at 22:07
  • 2
    @mmr: I don't think that anyone is alleging widespread falsification or deliberate distortion of evidence (which the term "cooking" seems to imply). Rather the problem is a combination of systemic biases in science as a whole; pressure to produce positive results, post-hoc analysis and poor understanding of statistics often seem to combine to produce "statistically significant" results out of noise, while at the same time negative results tend to remain unpublished. – Paul Johnson Feb 24 '16 at 18:08
14

There are several studies that document the problems of reproducibility, and thoroughly review the issue in indications of medicine, oncology, and neuroscience.

The two separate publications below detail how over 2/3 of landmark oncology studies were found not to be reproducible.

Asadullah: http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.html

Glenn Begley / Lee Ellis: http://www.nature.com/nature/journal/v483/n7391/full/483531a.html

Bilal Mahmood
  • 141
  • 2
5

Yes, and the problem exists across many areas of science

The problem of reproducibility is attracting a lot more attention. A recent Nature News Feature discusses the possible causes of bad results in the literature and many examples where they have been exposed. Its headline sums up the problem:

Humans are remarkably good at self-deception. But growing concern about reproducibility is driving many researchers to seek ways to fight their own worst instincts.

One huge problem is that results that sound plausible get a lot less scrutiny than unusual results so wrong but plausible results get a free pass on error checking far too often.

The article summarises a range of different reproducibility experiments and the results suggest that the majority of results across many fields are not reproducible (it is worth reading the article for all the links to these studies).

Failure to understand our own biases has helped to create a crisis of confidence about the reproducibility of published results, says statistician John Ioannidis, co-director of the Meta-Research Innovation Center at Stanford University in Palo Alto, California. The issue goes well beyond cases of fraud. Earlier this year, a large project that attempted to replicate 100 psychology studies managed to reproduce only slightly more than one-third. In 2012, researchers at biotechnology firm Amgen in Thousand Oaks, California, reported that they could replicate only 6 out of 53 landmark studies in oncology and haematology. And in 2009, Ioannidis and his colleagues described how they had been able to fully reproduce only 2 out of 18 microarray-based gene-expression studies.

On a more positive note the article recommends a number of ways to de-bias scientific answers. Some of these are summarised in the picture below:

Image (c) Nature

So there are plenty of known reasons why bad results get published and there are a lot of them. But it is something that good scientists can fight.

matt_black
  • 56,186
  • 16
  • 175
  • 373
  • 1
    This doesn't actually answer the question. The real problem why bad results get published is that there are too many journals for the available pool of competent reviewers and too much "publish or perish" pressure on the researchers. The cognitive bias thing isn't as much of a problem as is being portrayed and is distracting attention from where the problem really lies. Scientists are already trained to deal with these biases, the most successful scientists are generally those who are good at doing so. –  Feb 24 '16 at 12:18
  • 2
    @DikranMarsupial You are right that the proliferation of journals is part of the problem, but you are wrong that scientists are good at avoiding these biases. Even good scientists screw up because of them (see the Andrew Gelman story in the Nature article). And the evidence says that a lot of crap is published. Moreover, scientists are incentivised to advance their careers by publishing and this is synergistic with poor quality peer review to generate bad, published results. – matt_black Feb 24 '16 at 13:35
  • "Even good scientists screw up because of them" you are missing the point, yes we all have these biases and none of us is perfect. ***However*** scientists are already trained in avoiding them and are **vastly** better at avoiding them than the general public (e.g. compare public and scientific reactions to Prof. Salby's carbon cycle arguments). Sure top scientists still make errors, but part of the reason they are top scientists is that they are better at dealing with their biases than the rest. –  Feb 24 '16 at 13:44
  • Also related https://www.sciencedaily.com/releases/2016/03/160303145733.htm –  Mar 04 '16 at 17:52
  • 1
    @DikranMarsupial On that very topic, [a defence of the original study](http://andrewgelman.com/2016/03/05/29195/) and some more general criticism of the state of scientific evidence in psychology. – matt_black Mar 06 '16 at 14:46
  • Gelman is always an interesting writer, and he makes a good point here about why p-values are not necessarily informative. Ironically the way to prevent this from allowing bad ideas to get into the journals is to consider the a-priori plausibility of the hypothesis, which is essentially the sort of cognitive bias that you criticize in your answer! In otherwords reviewers need to apply assymetric attention to weed out the spurious whacky claims. Of course scientists generally know how to balance these things, although it is less of a problem in some science than psychology. –  Mar 06 '16 at 16:02
  • "1. Lots of bad stuff is being published in top journals and being promoted in the media." is an important part of it. Publication is only the first step in acceptance of an idea, but that is usually the only time the media are interested, which explains why there are lots of stories about this sort of thing in the media. A lot of the problem is caused by media interest that doesn't investigate the level of support from the scientific community, e.g. ice free arctic by 201X etc, which gets in the media, but climatologists almost uniformly reject. –  Mar 06 '16 at 16:06