From crisis to cornerstones of culture

In August 2005 a paper by John Ioannidis, 'Why most published research findings are false', ignited a new debate across psychology and the biomedical sciences about the reliability and robustness of many published journal articles. This reproducibility 'crisis', while at first troubling for researchers in many areas of science, has taken a more positive turn in recent years: with psychology leading the way.

Many concerns emerged: the validity of antiquated methods of publishing; peer review; and the biases that occur in journals favouring the publication of novel, positive findings over null results or replication attempts. Within psychology specifically many classic findings failed to replicate, and concerns were raised around widespread research practices such as 'p-hacking' (trawling datasets for significance).

We spoke to some of the psychologists working to change the culture of academia. Professor of Biological Psychology Marcus Munafò (University of Bristol) emphasised that psychologists are uniquely placed to study the inherent cognitive biases and many other issues prevalent in all areas of academic publishing. 'I think any focus on research integrity and fraud is missing the point,' he said. 'The vast majority of scientists are motivated to do good work, but however hard we train to be objective scientists, we're still human. The reason psychologists have been at the forefront of the debate is not because psychology has a particular problem as a science but because many of the issues are issues of human behaviour.'

Psychologists, and psychology journals, have been instrumental in coming up with innovative ways to change publishing; the 'Registered Reports' model of publishing, where a research question and study protocols are pre-registered prior to data collection, has flourished in psychology (see www.thepsychologist.org.uk/vaccine-against-bias) and has now been taken up by 50 journals. Psychological Science has introduced open science badges for articles whose authors have shared data, materials and/or pre-registered their studies, and even some undergraduate psychology courses ask students to pre-register dissertation projects.

Munafò pointed out that since this debate came to the fore we've moved from a more negative stance into a phase of thinking about how to improve the way we do science. He said: 'It feels like we're in the middle of a rapidly evolving natural experiment where people are trying lots of different things with much of the focus on improving the quality of how we do science, and hopefully some of that will stick and some of that will make a difference.' In fact, some of the results can already be seen: before Psychological Science introduced its badge for open data, rates of data sharing were below 5 per cent, just a few months after its introduction this shot up to more than 40 per cent.

As Editor-in-Chief of Nicotine and Tobacco Research, and thanks to a partnership with Cancer Research UK, Munafò is piloting a scheme to bring funding and Registered Reports together. The Registered Reports format allows journals to accept a study's research question and methodology for publication in principle, before any data has been collected, and funding panels work in a very similar way. Munafò explained: 'Logically it makes sense to say once you're committed to funding research on the basis of a good research question and methodology, that we'll also guarantee publication.

If applicants opt in to the pilot and they're successful in their application to funding, the journal will take over and go through a second phase of peer review, but as part of the same process, try and fine tune the protocol and the methods, and offer in-principle acceptance of that protocol. Then the applicants have the funding to do the work, and guaranteed publication at the end of that process. Whatever their results they will be published almost immediately after their work is complete.'

Lecturer Katherine Button (University of Bath) instils good research practices in her students at all stages of teaching, and worked with colleagues from Cardiff and Exeter to allow a group of students to collaborate on, and pre-register, a third-year project. Eight students across the three universities pre-registered with the Open Science Framework before data collection began. Dr Button explained: 'They each collected data following standardised procedures, and we held a mini-conference to discuss results and mutually agree the conclusions for the paper in preparation, on which all students will be co-authors. By working together, the students and we academics were able to conduct a rigorous piece of research that hopefully stands a good chance of being published regardless of the results, thus mutually satisfying both the need for increased methodological rigour and the career pressure to publish.'

How might these messages of doing better, more robust science, filter throughout the academic community? As well as teaching this approach at undergraduate level we need to have higher expectations of ourselves and others, Button said: 'Expect to see that a study has been pre-registered, and if not there should be a reasonable explanation as to why not. Give more weight to results from studies which have followed best-practices – i.e. sample size calculation, protocol and analysis plan pre-registered, data and material made open-access – and be more cautious of results from studies which have not. Be more aware of the role of chance in statistical analysis, with enough flexibility in analysis and reporting it is easy to find a "statistically significant" result. Be cautious.'

Similarly, Dr Pete Etchells, Senior Lecturer in Biological Psychology, has made changes at Bath Spa University, where students will soon pre-register their final-year projects. He said: 'There's been lots of discussion of how we can change things from the point of view of journals or grants or general culture in already established researchers, but there's a lot we can do at the ground level. From the very first day undergraduate psychology students come in we can start teaching these things, as they should be, as normal standard practice.'

More broadly, is the culture among academics changing? In an exclusive extract from his new book The Seven Deadly Sins of Psychology on our website Chris Chambers (Cardiff University) recalls that his proposals for Registered Reports were met with accusations 'of being "self-righteous," "sanctimonious," "fascists," "a head prefect movement," "Nazis," "Stasi," "crusaders" on a "witch hunt," and worse.' Etchells said he hoped we were moving away from the vitriol and anger thrown around at the start of the replication crisis. However, he admitted: 'I find it constantly fascinating that psychologists seem to be so thin-skinned about some of this stuff, when replications and good methodology are the cornerstones of science. For some reason people take these discussions about replication very personally. I don't know why that's the case, it certainly doesn't need to be, and I think people trying to promote open science practices and replication are doing it for the good of science, not to get at anybody.'

Many of these methodological debates take place on blogs and social media, with some of those whose work has been questioned accusing others of bullying. Etchells said having people with opposing opinions put in a room together to speak as adults may be a better approach (with a workshop at the Society's Annual Conference perhaps a good example: see tinyurl.com/repwork). He added: 'People can get pretty snarky online, and that's unhelpful. If you just talk to people in a sensible way, that'll go a long way to helping allay people's fears rather than firing off a snarky line on social media – with a caveat that I don't think that that happens very often. A lot of people talk about online bullies, I see very little evidence of that in this debate. I think we need to talk to each other more.'