Keep your friends close, your adversaries closer

For at least the last 10 years, psychology has been shaken by the replication crisis. Many 'sound' findings have proven shaky when re-examined using more rigorous approaches (Open Science Collaboration, 2015), and the spotlight has fallen on 'false positives' – finding something in the data which isn't truly there. Publication bias, incentives for publishing hypothesis-supportive findings, the perception that flashier results may attract more grant funding or newspaper headlines; these are all problems in psychology. But there's an even more fundamental issue – researchers are human and often favour certain results over others.  

This bias may be particularly present in highly emotional topics in social science, which may unwittingly be an extension of an individual's advocacy efforts or ideological beliefs. In such an environment, scholars may push either for or against outcomes according to their perceived utility for a particular cause or belief, or unconsciously develop study methodologies that are likely to favour a desired outcome. One research area that has been notoriously emotionally valenced, and where researcher beliefs both for and against an effect are often highly impassioned, is media and body image.

In this area, some scholars are very passionate about advocating for media effects, while others are more skeptical (Ferguson, 2018). Two issues can come to the fore. First, it may be easy to dismiss the findings of a particular research group as 'they are just skeptics/believers'. Second, ambiguous results can be selectively interpreted. Consider for example the case of Whyte et al. (2016), an excellent study with specific innovations such as careful matching of media conditions, standardised outcomes and distractor tasks designed to minimise false positive results. For two outcomes, one result was non-significant, the other of threshold significance (p = .045). Removing a covariate from the latter result rendered this outcome also non-significant. Thus, even a well-designed study returned an ambiguous result, easily interpreted according to a scholar's a priori beliefs about effects.

One way to reduce these problems is through adversarial collaboration. This doesn't mean scholars need to hate each other (indeed, adversarial collaboration may be difficult in an environment lacking in trust and respect). Rather, two or more scholars with differing beliefs about a research field come together and design a study they mutually agree would be convincing with clear criteria for interpreting results. By combining viewpoints, the involved scholars can challenge each other both to develop better, more convincing designs and also limit selective interpretation of ambiguous results.

Recently, we came together to examine the potential impacts of sexualisation in a video game on women player's body satisfaction and aggression toward other women (Lindner et al., in press). Our collaboration was adversarial not because we loathe each other (we don't), but simply because we had different views on media effects. This presented an excellent opportunity to test our views using a rigorous preregistered design.

I [Chris] was aware of developing a reputation as a skeptic in the area of media effects. Approaching media effects as an adversarial collaboration offered an opportunity to challenge my perspectives and test them in a more rigorous format. Adversarial collaborations can be one element of open science and improving the rigour of data more generally.  

I [Danielle] chose to pursue this project for two reasons. First, I had published primarily correlational studies (e.g. Lindner et al., 2012). As an early career faculty member building a research program, broadening the research agenda to include experimental studies made sense. More importantly, I was interested in the intellectual exercise of designing a rigorous study with a colleague who had a different viewpoint given that null findings can be just as interesting and meaningful as significant findings.

For us, the actual process of the adversarial collaboration was in fact very collaborative. Our research questions emerged out of a series of conversations about the field. In designing the study, we were able to bring together knowledge from our previous research experiences (i.e. rigorous experimental study of media effects and knowledge of video games for Chris, deep understanding of objectification theory for Danielle) to develop methodology that avoided many of the pitfalls common in experimental studies of media effects and body image (e.g. Thompson, 2004; Want, 2014). We worked together to determine how best to operationalise our variables and develop study procedures that would minimise demand characteristics. Pre-registration required us to clearly establish not only our study design, but also our statistical procedures, prior to running the study. This level of decision making, while important in any collaboration, is especially important in an adversarial collaboration to ensure there is no allegiance to either researcher's agenda. We were fortunate to have undergraduate students help with data collection, and two students helped prepare the final manuscript.

The results of our study showed that exposure to a sexualized female video game protagonist had no effect on college women's self-objectification, body shame, body dissatisfaction, aggression, and negative attitudes toward women (operationalised as negative evaluations of the female research assistant). These findings were consistent with Chris's other research on the topic, and inconsistent with Danielle's research. I [Danielle] wasn't especially bothered by this finding. In my teaching, I talk about how theories can't possibly explain all behaviour. I also educate students about the lower rates of publication for null findings, and how this can have a negative impact on a field of study. Being open to the possibility of null findings and being willing to publish them when they arose is part of 'walking the talk'. We did talk at length about our findings and their implications. Particularly in the Discussion section, it is possible for researchers to present findings in ways that reflect their own scientific agendas rather than what the data actually show. It was important that our findings were represented in a way that we both felt was scientifically accurate. To this end, we explained how our findings were consistent with other findings from Chris's area of research, and utilised findings from my area of research to explain why our results turned out the way they did.

Did the results change either of our minds?  Definitely not! We still send zingers at each other across the hallway! Kidding… we still have our own views of media and sexualisation and that's fine. Results like these help us both to shape our views and consider them from new angles and, of course, design new research studies. We're already working on our follow-up examining impacts on male players. Who knows what we'll find, but it will be exciting however it turns out!

It's worth noting that not every adversarial collaboration is likely to work. In some research fields, debate may have descended into outright acrimony, possibly even legal proceedings! Remember, you'll likely have to live with this person for months at least, quite often years (our collaboration took about three years). Such collaborations can be rewarding and fun, so long as there is mutual respect, but a nightmare if that is lacking.

- Chris J. Ferguson is a Professor of Psychology at Stetson University and lives in Orlando, Florida

- Danielle Lindner is an Assistant Professor Psychology at Stetson University in DeLand, Florida

Illustration: Sofia Sita

Key sources
Ferguson, C.J. (2018). The devil wears Stata: Thin-ideal media's minimal contribution to our understanding of body dissatisfaction and eating disorders. Archives of Scientific Psychology 6, 70-79.
Lindner, D., Tantleff-Dunn, S. & Jentsch, F. (2012). Social comparison and the 'circle of objectification'. Sex Roles, 67, 222-235. 
Lindner, D., Trible, M., Pilato, I. & Ferguson, C.J. (in press). Examining the effects of exposure to a sexualized female video game protagonist on women's body image. Psychology of Popular Media Culture.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), 1-8.
Thompson, J.K. (2004). The mis(measurement) of body image: Ten strategies to improve assessment for applied and research purposes. Body Image, 1, 7-14. 
Want, S.C. (2014). Three questions regarding the ecological validity of experimental research on the impact of viewing thin-ideal media images. Basic and Applied Social Psychology, 36(1), 27-34. 
Whyte, C., Newman, L.S. & Voss, D. (2016). A confound-free test of the effects of thin-ideal media images on body satisfaction. Journal of Social and Clinical Psychology, 35(10), 822-839.