FAIR’s fair – are interesting articles sufficiently transparent?

Research in Psychology can be high quality, reported transparently, and interesting to other Psychologists and hopefully a wider audience. 

In an ideal world, the Venn diagram of these three factors would be a single circle. But we don't live in an ideal world, and – more importantly – we don't really know what the overlap is.

Open, well organised and intelligible research datasets provide greater research transparency and enable reproducibility and reuse. Similarly, adhering to the FAIR (Findable, Accessible, Interoperable, Reusable) principles for data management and stewardship promotes enduring digital data access and reuse, values central to promoting integrity and enabling reproducibility. Many funders now expect primary datasets, code and materials generated during the course of research to be made as open as possible (and closed as necessary).

Of course, not all research datasets can be made open – perhaps due to the ethical and/or legal concerns surrounding sensitive research – but a detailed data availability statement included in articles can help provide clarity on why data remain closed or restricted, and the information required to access controlled datasets. Simply saying 'available upon request' is not an acceptable data availability statement (not least because, in practice, data is very often not made available on request!).

In addition to data availability, statements on ethics approval – if applicable – and any potential conflicts of interest amongst the authors of a study provide greater transparency, and support research integrity. This is particularly true when combined with a detailed breakdown of the contributions of the authors – the Contributor Roles of Taxonomy (CRediT) is the leading taxonomy to provide greater authorship transparency and accountability.

Finally, pre-registering study hypotheses, methodologies and analytical plans before the generation or collection of data also contributes to the transparency of a study. It may also help to reduce the prevalence of Questionable Research Practices – things like P-hacking and HARKing – that are now widely acknowledged to have contributed to broader issues of reproducibility in psychology and other areas.

With all this in mind, how do the articles and studies highlighted by the British Psychological Society's Research Digest fare when it comes to FAIRness (and the rest)?

We conducted an article statement, dataset intelligibility and FAIRness check on 30 journal articles given the Research Digest treatment from April-June 2024. We assessed the inclusion of statements on data and code availability, author contributions, ethics approval and conflicts of interest; whether the study was pre-registered; and the FAIRness and intelligibility of datasets underpinning the articles if available.

What did we find?

Approximately two-thirds of the articles we reviewed contained an acceptable data availability/access statement, with 61 per cent either making the complete primary dataset available (47 per cent) or justifying why the primary dataset was restricted or controlled (14 per cent).

However, 25 per cent included an 'available upon request' statement, and 11 per cent had no statement or mention of data availability or access at all. In other words, around one third of articles did not have an adequate data availability statement.

Code availability statements showed a similar pattern, with 72 per cent of the articles containing an acceptable statement, but 17 per cent of articles contained an 'available upon request' statement, leaving 11 per cent with no statement or mention of code availability.

The majority of articles included an appropriate statement of ethics approval (85 per cent) and conflict of interest (97 per cent). Over half (52 per cent) included author contributions using CRediT, and a further 10 per cent included a custom breakdown of author contributions.

Slightly less than half of the articles (41 per cent) were pre-registered.

The 'FAIRness' of available research data was tested using FAIR-checker v1.3, an online FAIR assessment tool designed to assess how well digital resources comply with the FAIR principles. It can help evaluate and improve the quality of open data by checking the data and metadata against recognised standards and ontologies, and provides an overall FAIR score, and a sub-score for each of the four principles.

The average overall FAIR score returned was 65 per cent (SD 19 per cent), with Findability highest at 78 per cent and Interoperability lowest at 46 per cent. This interoperability of data – how easily it can be recognised by, shared and used across different systems – can be improved by using recognised, open and common data formats and protocols (e.g., CSV, JSON and HTTP) and machine-readable metadata using controlled vocabularies and ontologies.

One initiative that has recognised the need for improving the interoperability of research data is The Behavioural and Social Sciences Ontology Foundry, an initiative promoting the development, adoption and use of ontologies and controlled vocabularies within the behavioural and social sciences.

In third-party repositories such as the Open Science Framework, many elements of the supporting metadata exist in the background of the repository and can't be amended by the user. However, the user can add contextual information that the FAIR-checker assesses.

One example is access and reuse license information – attributing a license to an OSF deposit, by simply choosing the appropriate license from a drop-down menu, improves both the interoperability and reusability score returned by FAIR-checker because it provides clarity on the access and reuse terms for researchers.

Of course, the dataset needs to be intelligible to a human to promote reproducibility and reuse. Refreshingly, the majority of the available datasets (79 per cent) were intelligible (to the author, at least!), with the remaining 21 per cent in the main understandable but with one or two elements that would require further clarification from the authors (e.g., asking authors about data variable names that have been abbreviated or acronymised without explanation).

But what does this tell us?

Is the open research glass half full or half empty?

Overall, I think the results are encouraging. Obviously we haven't compared against previous years, but the fact that around two-thirds of articles chosen for Research Digest coverage have an acceptable data and/or code sharing statement feels pretty good. There's still work to be done – ideally every article would have such a statement. But the fact that the majority of articles deemed interesting enough to be highlighted do is great.

Recognising author contributions is also now done in the majority of cases (again, in the limited and non-representative selection of articles we assessed). That is perhaps especially important in terms of individual authors – particularly early career researchers – being able to demonstrate their unique contributions to research outputs. Again, still some way to go, but encouraging nevertheless.

Open research is not new, but the emphasis placed on it by funders and publishers is only increasing. This may be behind the positive results we've seen, and shows how policy changes can perhaps positively impact on researcher behaviour. It will be interesting to perform a similar assessment in a few years to see how things have changed (or not!).

But for now we can be reassured that the Venn diagram of high quality, transparently reported and interesting research overlaps more tightly than we might have anticipated.

BOX: Methods annex

Article PDFs were retrieved and read, and the presence or absence of the relevant statements was recorded. One of the 30 articles could not be accessed because at the time of assessment, the University of Bristol, where the analysis took place, did not have a subscription in place for the journal in question. This article was excluded from subsequent analysis.

Any available primary data was accessed, and raw and processed datasets (if applicable) were matched to the corresponding results in the article. If the raw data underlying all of the article results were not available, then any justification for absent data was recorded.

To determine dataset intelligibility, the following guiding questions were used:

• Is the data deposit easy to navigate with logical, consistent and understandable file and folder naming and structure?
• Is there contextual information describing the study, data and analysis (i.e., in a descriptive readme file, data dictionary and/or codebook, containing for instance the variable names, coding schemas, data validation rules etc.)?
• Is the analysis pipeline understandable?
• Is any code sufficiently annotated and documented so that a competent reader could confidently attempt to computationally reproduce or reuse the results without contacting the authors?

Article datasets were recorded as either: intelligible, partially intelligible or not intelligible. A partially intelligible dataset was one in which there was an element to the dataset or analysis pipeline that was not clear and contact with the authors for clarification would be necessary to reproduce or reuse the data.

Marcus Munafò is Professor of Biological Psychology and MRC Investigator, and Associate Pro Vice-Chancellor - Research Culture, at the University of Bristol. 
[email protected]

Editorial comment

Part of the reason we asked Nick and Marcus to do this, with a selection of Research Digest output, is that we have often worried that a main focus on the 'interesting' part of the Venn diagram, to reach wider audiences and change something, might come at the expense of the other two sections. So it's good to hear from Nick that there's perhaps more overlap than we might have expected.

Of course, our editorial processes at the Research Digest might impact how reflective the findings are of the Psychology literature more broadly. While we have a sense of the kind of quality we prefer to cover, we're not doing a deep dive on how open the paper is in every case. If a new study appears to meet a reasonable standard for robustness and clarity, alongside the usual things like being in a journal that seems reputable, for example, we tend to consider it.

Emma Barratt, Research Digest Editor
Jon Sutton, Managing Editor