Invisible workers, hidden dangers

Roughly two billion people use Facebook each day, while 500 million navigate Instagram. Approximately 500 hours of video are uploaded to YouTube every minute, and 6,000 tweets are posted every second. The ever-increasing amount of user-generated content means more people are needed to evaluate and remove some of it. 'Content moderation' involves analysing material flagged as offensive, potentially harmful, or in violation of platform policy. 

These content moderators (CMs) can be exposed to depictions of extreme pornography, graphic violence, hate speech, self-harm, child abuse and gore. There's an increasing awareness that individuals who 'witness' traumatic events, or are exposed to the impacts of trauma second-hand through victim accounts, are also at increased risk of disorder. The four most commonly recognised conditions associated with indirect trauma exposure are secondary traumatic stress (STS), vicarious trauma, burnout and compassion fatigue. 

Although these terms are used somewhat interchangeably, they refer to different clusters of related but distinct symptoms or behaviours. STS has similar symptoms to post traumatic stress disorder such as intrusive thoughts, hyperarousal and hypervigilance, as does vicarious trauma, although it is also associated with a transformation of self-image and worldview.  Burnout is characterised by emotional exhaustion, cynicism and reduced feelings of accomplishment. Compassion fatigue refers to the emotional and physical fatigue that occurs as a result of the chronic use of empathy, and is often considered to be a combination of burnout and STS.

The impact of these syndromes has been widely studied in other professions, but currently there are no baselines for rates in CMs. Psychological expertise is badly needed to determine the effects of the work and establish what support is effective in this area.

What are the impacts of the work?

Anecdotal evidence suggests the work is psychologically damaging; CMs report anxiety, depression, nightmares, fatigue and panic attacks, with their relationships and physical health also affected. Both Microsoft and Facebook were sued by CMs who developed PTSD on the job and held the companies responsible. Academic research into the scale and range of potential problems associated with content moderation, however, remains lacking (although see Steiger et al., 2021). 

In this vacuum of evidence, we look to other related areas. Could the type of work-stress CMs experience be comparable to that faced by internet child exploitation (ICE) investigators? Their job also involves looking at digital material featuring disturbing violent or sexual imagery. Evidence suggests these law enforcement officers report high rates of sick days, increased absenteeism, relationship problems, and increased substance use, amongst the development of other mental health difficulties, including developing STS. 

However, the work of CMs and ICE investigators differ in several important ways that may change how the nature of the work impacts them. For instance, Perez et al. (2010) made a conservative estimate that ICE investigators view 25,000 disturbing images per work-year, including pornography, extreme violence and unusual sexual activity. CMs can view hundreds of videos every day, and although many of these are benign, they are frequently and chronically exposed to violence, abuse and murder in video content, with some viewing explicit material repeatedly over nine hour shifts with few breaks. Regular exposure appears more likely to lead to STS and burnout and there is some evidence that longevity in role may increase the likelihood of experiencing trauma – although many ICE investigators choose to stay in the role, many others move department. The ability to transfer to other roles within their organisations is not one that many CMs have. They instead face the choice of finding new employment or staying in a role with the potential to cause traumatic responses.  

Unlike police, CMs are also not all directly assigned caseloads. Instead, they are placed on work queues where they sort through flagged material, determining what action needs to be taken on each. Once a decision is made, the next piece of content is queued up. The accuracy and volume of material they are able to review is measured, and workers feel pressure to meet targets as their performance is regularly audited (Dwoskin et al., 2019). However, material is generated worldwide and therefore many are moderating flagged material that may not be in their native language, and trying to assess offense using cultural norms and colloquial terminology unfamiliar to them. Additionally, policies are subject to frequent changes, including ad hoc and spontaneous alterations. Despite this, CMs are expected to be 'accurate' (i.e. agree with their manager or another quality assessor) 95 per cent of the time. However, what counts as indecent is difficult even for trained experts; and levels of agreement can appear arbitrary and ultimately be inaccurate. Additionally, unlike ICE investigators, CMs are frequently separated from the results of their labour, often not being told what happened after they flagged or escalated certain material to authorities. With ICE investigators, feedback on cases is considered an important aspect of working, helping to reaffirm the broader meaning of their work. 

So while some aspects of content moderation work is similar to that undertaken by ICE investigators, CMs are also subject to unique work pressures with less institutional support, the impact of which is currently unknown. Research is needed to explore what aspects of the job are most damaging and why. That needs to include different cultural contexts and what factors may mitigate effects – CMs are based worldwide, especially in non-western countries, yet most of the research on STS and burnout is based on western participants. Even within a western context, levels and predictors of STS have been found to be different amongst investigators working on child exploitation cases in the UK and USA (Bourke & Craun, 2014). 

How can CMs be supported?

Psychologists can provide expertise on supporting CMs, although again there are few studies exploring areas of resilience and best practice. 

If we look again to studies of ICE investigators, we find the increased awareness of the emotional demands of police work generally affords them more flexibility within their working and better support than CMs might receive. Studies exploring coping strategies found ICE investigators take breaks from the work and go to their colleagues for support. However, many CMs are micro-managed, with their time required to be accounted for, making the psychological challenges of the work harder to cope with. Combined with high targets, this can lead to a culture where CMs feel unable to take breaks or spend too long away from their desk. This also reduces their ability to engage in casual conversation with colleagues or build up informal supportive relationships in the workplace. Working conditions that reduce the ability to access informal supports could decrease staff resilience. Many CMs are also required to sign non-disclosure agreements, inhibiting their ability to talk about their work to others, compounding the problem. Added to this is the sometimes-dispersed nature of the CM workforce, where many individuals work remotely from home or on a temporary or contract basis, removing them from a wider collegial network. 

Psychologists know that a supportive work environment generally helps protects employees from burnout, vicarious trauma and STS. However, currently it is not known how many CMs receive emotional support such as debriefs, and the practice will vary across the sector. Often though, CMs are often treated as 'second class' workers, hired by contractors rather than the companies themselves, and provided with low benefits and pay. This disparity is most notable for contractors who work on-site, side-by-side with employees but without the same salary or access to perks.  

Although companies have implemented a degree of CM autonomy in controlling their workflow through use of blurring, audio removal and looking at thumbnails rather than videos, these interventions are not available to all CMs and there still remain aspects where CMs fundamentally lack any safeguards over their exposure to harmful content. For instance, they are not involved in policy decisions that affect the composition of their roles and duties, how long they can do it for or what should be removed as offensive. Employees who perceive their workplace situation as having insufficient rewards, unfairness, and having a lack of control over organisational issues are more susceptible to burnout.

Burnout, vicarious trauma, and compassion fatigue are cumulative over time (STS can be cumulative or occur after one exposure). This can make it difficult for individuals to realise what is happening, sometimes only becoming aware once leaving their employment. Where there is a realisation something is wrong, there can be barriers to help-seeking, such as stigma associated with needing psychological help. Mental health practitioners may also not be perceived as competent to deal with the specific issues raised or there can also be worries around confidentiality, with reports that counsellors working for social media platforms have been pressured to share information from private sessions (Biddle, 2019). Additionally, there can be practical barriers such as restricted time to access mental health or a lack of wellness services altogether. Alternatives such as remote support could help individuals get help if they cannot access face to face services. 

Even where there are interventions, these tend to be untested in, and not specific to, this population; developed on an ad-hoc basis (for example via a company providing counselling); the impact is not formally evaluated, if evaluated at all; and the content moderators 'voice' is lost in terms of what they want and how they want it implemented. However, better psychological understanding of what works in supporting CMs is crucial, as organisational support can act as a buffer against compassion fatigue, STS and burnout.

What individual factors make a difference?

Even with supportive work environments, some individuals will inevitably be more susceptible to trauma than others and the degree of negative impact will vary. Several studies of ICE investigators note they feel positively about their work and generally cope well. However, many CMs are young college graduates who take up the role straight out of education. Younger age is associated with increased burnout and, as workers age, they use more positive emotion regulation strategies and possess higher emotional competencies to stave off negative consequences. The ability to compartmentalise and use more problem-focused coping rather than emotional responding helps protect individuals against the adverse effects of viewing disturbing material. There is evidence that experiencing childhood adversity is linked to greater reliance on emotion-focused and lower problem-focused coping as well as work-related stress and burnout. Ongoing adversity, such as financial worries or family illness, is also associated with increased work stress and the development of PTSD in police. Currently, there does not seem to be any research exploring the effects of childhood adversity or ongoing stressors on CMs, although it seems likely these would affect their vulnerability.

Mitigating risks – an evidence base of what works and for whom

Content moderation is an important and psychologically demanding job, and more psychological research and intervention is needed to help workers in this field. Anecdotal accounts suggest rates of psychological problems could be high and working conditions may exacerbate the likelihood of developing difficulties, yet there is not a lot of empirical research and practice to draw on. We look to related areas to understand what problems CMs might experience, what factors increase or mitigate risk, and for whom. 

It is encouraging that in equally difficult roles such as ICE investigators, the majority of employees appear be coping well. However, some of the risks and outcomes may be hidden. Psychological expertise is badly needed to both research and intervene to help workers in an area likely to be rife with secondary trauma and stress. 

-       Ruth Spence, Jeffrey DeMarco and Elena Martellozzo are researchers at the Centre for Abuse and Trauma Studies at Middlesex University. They gratefully acknowledge the financial support provided for this programme by the End Violence Fund.

References

Biddle, S. (2019, August 16). Trauma counselors were pressured to divulge confidential information about Facebook moderators, internal letter claims. The Intercept. 

Bourke, M.L. & Craun, S.W. (2014). Coping with secondary traumatic stress: Differences between U.K. and U.S. child exploitation personnel. Traumatology: An International Journal, 20(1), 57-64. 

Dwoskin, E., Whalen, J., & Cabato, R. (2019, July 25). Content moderators at YouTube, Facebook and Twitter see the worst of the web – and suffer silently. The Washington Post.

Perez, L.M., Jones, J., Englert, D.R., et al. (2010). Secondary traumatic stress and burnout among law enforcement investigators exposed to disturbing media images. Journal of Police and Criminal Psychology, 25, 113-124.  

Powell, M., Cassematis, P., Benson, M., et al. (2015). Police officers' perceptions of their reactions to viewing internet child exploitation material. Journal of Police and Criminal Psychology, 30, 103-111. 

Roberts, S.T. (2014). Behind the screen: the hidden digital labor of commercial content moderation. PhD Thesis, University of Illinois, Chicago, IL. 

Steiger, M., Bharucha, T.J., Venkatagiri, S., et al. (2021). The psychological well-being of content moderators: The emotional labor of commercial moderation and avenues for improving support. In CHI Conference on Human Factors in Computing Systems (CHI '21), May 8-13, 2021, Yokohama, Japan.