Sharing across fields

The robustness of published scientific research, including psychology, has been in the news for several years now, and is something I have a long-standing interest in. This stems from my early experiences as a PhD student, where I was unable to replicate a high-profile finding and was fortunate enough to be reassured by a senior academic that I wasn't alone. However, the debate has moved on from the 'epidemiology' of reproducibility, to the development of interventions – how can we learn from this debate and do better? And here my own experience resonates.

I've been lucky enough to work with a wide range of colleagues, across disciplines and departments, learning as I went. My undergraduate degree was in psychology and philosophy in a psychology department with a strong experimental focus, whilst my MSc and PhD were in the much more applied area of health psychology. So far, so (relatively) conventional for a psychologist. But after my PhD I worked in a public health and primary care department, a clinical pharmacology department, and a psychiatry department. I worked on systematic reviews and meta-analyses, clinical trials, molecular genetic studies…

A key lesson from this experience was that every field does something well (often very well). Clinical trials have led the way in protocol pre-registration, systematic reviews and meta-analysis have provided much of the foundational material that has informed the reproducibility debate, genomics has a strong culture of data sharing, and so on. Sharing these ways of working, and integrating them into a common model of best practice (which may not apply fully to every discipline, but may at least form a starting point) is, I think, something that will be key to improving research quality.

One area that I have worked in illustrates how we can achieve a step change in research quality in a relatively short period. During my postdoc career I worked on candidate gene studies – looking for associations between specific genetic variants (typically assayed manually) and behavioural outcomes. This field was notoriously unreliable – in fact, my early forays into what is now known as meta-research used systematic review and meta-analysis skills, learned whilst working with the Cochrane Collaboration, on the candidate gene literature. This suggested the evidence was weak at best, and effect sizes very small.

I was certainly not alone in questioning the robustness of evidence from candidate gene studies. The advent of genome-wide association (GWA) technology, which allowed the simultaneous interrogation of 500k+ genetic variants, coupled with a growing awareness that the effect sizes of common genetic variants for complex traits would be small, meant two things. We would need very stringent statistical standards to correct for multiple testing, and (in part given this, and in part because of the likely small effect sizes) we would need very large sample sizes. And this meant collaboration.

The result was a radical change in the approach to genomics research. Consortia were formed, partly because researchers themselves realised this would be necessary, partly because funders had the will to encourage and support these. Approaches to data analysis had to be agreed across groups, new norms of authorship were agreed to ensure fair recognition of the contribution of those involved in data collection, data analysis and so on. The very large sample sizes that could be achieved through collaboration, together with the potential for replication, meant that the results of GWA studies were astonishingly robust.

What is perhaps even more remarkable is that the culture of data sharing, which was a consequence of this collaborative approach and the need to share summary data for synthesis across consortia, has led to other, unexpected benefits. There is now a thriving literature that makes use of summary data generated by GWA studies and posted publicly by these consortia. Mendelian randomisation and genetic correlation analyses are providing new insights into causal pathways and shared genetic architecture, bringing together yet more disciplines across the social and biomedical sciences, all using these summary data.

This integration of different approaches can also benefit in deeper ways. One limitation on the focus on reproducibility is that we may arrive at very robust observations that nevertheless give us the wrong answer (if the results are an artefact of the experimental design, for example). One solution to this problem is triangulation – the prospective application of different methods to address the same question, each resting on different (and unrelated) assumptions, and subject to different (and again unrelated) sources of bias.

We have produced an animation that describes the logic of triangulation [https://youtu.be/JEQ_tcweqz8]. Triangulation relies on team science and collaboration. This will make for richer, deeper (and perhaps longer) articles. We may need to re-think our models of scientific publication, as well as how we attribute different contributions – from an authorship model to a contributorship model, for example.

These issues are being discussed already, but illustrate how efforts to improve research quality intersect, and need to be considered together in a coordinated way. I have benefited enormously from the rich and varied experiences I've enjoyed throughout my career. This should be the norm.

-  Marcus Munafò is Professor of Biological Psychology at the University of Bristol. [email protected]

Illustration: Sofia Sita