Speaking out on the ‘replication crisis’

Psychologist Dr Simone Schnall (University of Cambridge) has spoken about her own personal experience of the so-called 'replication crisis' within psychology at an event organised by the online debating forum Edge.  

Dr Schnall began her talk speaking about the complicated nature of replications in social science. She said: 'When it comes to social psychology it's a little more complicated because what we usually do is test a specific question with various different experiments… These are called conceptual replications. Our entire literature is built on those conceptual replications, but those are not the ones people are now discussing. They're different. They're called direct replications.'

According to Schnall, replicating social psychology experiments is not as simple as in hard sciences such as medicine where a direct replication can be done simply by administering the same dosage of a drug to a patient. She moved on to talking about her work, which looks at the links between physical and moral disgust. Pointing to a paper by David Stanley and Jeffrey Spence where they discuss the expectations for replications and used computer stimulations, in which they ran experiments thousands of times under perfect conditions with nothing but measurement error, she said: 'Even then one gets a great variability of results. The conclusion is that any one given study is not conclusive. That's why normally we do lots of studies to see if there's a general pattern.'

Dr Schnall suggested that after it emerged that social psychologist Diederik Stapel had fabricated data, leading to scores of papers being retracted in 2011, it made the community adopt a mind-set where people felt they needed to do things differently and look out for fraudsters and false positives. As well as this shift in attitude towards replication, Dr Schnall explored the lack of a system for choosing which studies should be selected for replication and suggested that a disproportionate number of studies had been singled out for replication because findings may be counterintuitive if one is unfamiliar with the literature and studies appear easy to conduct.

She also said there were issues with the quality of some replication studies and the conclusions drawn from them. 'Often the way these replications are interpreted is as if one single experiment disproves everything that's come before. That's a bit surprising, especially when a finding is negative, if an effect was not confirmed. We don't usually do that with positive findings… we don't say this now proves once and for all that such and such effect is real. It perhaps comes with that idea that it intuitively seems like this is the real study because we repeated exactly what had been done before.'

Some of the issues with this increase in direct replications have affected Dr Schnall: one special issue of the journal Social Psychology, with 15 replication papers covering 27 earlier reported effects, went to print without having undergone any peer review, and one of her own findings was reported to have not been replicated by some researchers. She added: 'I looked at their data, looked at their paper and found what I consider to be a statistical problem. What was really interesting, though, was that when I alerted the editors, they were not very interested. They were not interested at all. In fact, they denied me the right to a published response. I had to fight tooth and nail to get that response.

'Let's think about it in the legal context. This is to declare a verdict on the quality of people's work without a judge and without having given the people whose work is concerned any right to even look at the verdicts, never mind to defend themselves.'

A recent paper reported several successful replications of Schnall's work, and a re-analysis of the claimed failed replication actually revealed the predicted effect.

Meanwhile there has been a growing concern among academics about the legality of post-publication peer review, and the general tone of discussion surrounding their work outside of the formal journals publishing process (see Tom Stafford's Mind Hacks article on this subject). For example, Times Higher Education has reported on a scientist in the US who has started legal action after, he claims, anonymous comments questioning his science cost him a job offer.

- See the video and the full transcript of Dr Schnall's talk, and our special issue on the topic.