Faces in the wild

If you look hard enough at someone's face, can you tell what they are feeling? Let's say they seem puzzled, frustrated or relieved. How much information do you get from the face itself and how much comes from what else is happening: what the person is up to, where they are, and who else is around? Imagine that you don't know the person and have no idea what they are doing. Could you still work out what their expression meant?

In everyday life it is highly unusual to see faces without any knowledge of the person or their current situation. However, this is exactly what happens in some of the most famous research into facial expression, such as the classic studies led by Paul Ekman in the 1960s. Participants try to work out which of six emotions is represented by each of a series of carefully selected mug-shots, showing six distinctive expressions (see over for examples). The response options are happiness, sadness, anger, fear, disgust and surprise. Even if you didn't already know, you could probably work out which emotion each of these faces is intended to show. And most English-language speakers would agree with your answers.

Consistent judgements of these faces confirm that some faces can convey emotion-related meanings even without any information about the surrounding context. But what does this tell us about how faces operate under less constrained circumstances? What if there are other sources of information to consider? What if the face is moving rather than static? And what if it does not look like any of these six photographs? (See 'Are some emotions more basic than others?')

Outside psychology experiments, conversations are where we see facial expressions most often. We gauge other people's reactions while speaking, and their expressions help us understand what they are saying (or not saying) in return. Their face responds to our face, and our face responds back, with both of us matching, complementing or countering each other's changing smiles, grimaces or glances. Faces also respond to events happening outside conversations, and can communicate information about these events to other people. However, directly investigating these complex dynamic and interactional processes can be highly challenging. Instead, psychologists have begun to investigate how adding further sources of information affects how faces are perceived and how they influence other people's responses. Piecing together their results may help us appreciate how the meaning of interpersonal facial activity depends crucially on the context of its production.

Gaze direction

The faces typically shown in psychology studies are looking directly at the camera. It is as if they are responding to something right in front of them, around the place where the viewer is positioned. We cannot see whatever it is that might be causing the reaction, but its apparent location may provide clues about what might be happening.

A wealth of research, going back to Michael Argyle and Mark Cook's 1976 book Gaze and Mutual Gaze, shows that eye gaze in an important social cue. We track each other's eye movements during interaction and are acutely sensitive to differences in their direction of focus (e.g. Farroni et al., 2003). Gaze direction also affects perceptions of emotional expressions. For example, Sander and colleagues (2007) found that 'angry' faces were seen as more angry when looking directly at the perceiver, but 'fearful' faces were seen as more fearful when gaze was diverted (see also Adams & Kleck, 2005). One possible explanation is that a front-facing scowling gaze suggests that the person is angry with you as viewer, and this self-directedness increases its emotional impact. By contrast, fear directed away from you indicates that you may need to be alert to some frightening event happening somewhere close by, which makes it more frightening. More generally, gaze direction provides clues about what the emotional reaction is about, changing our perception of what the face is doing. When the object of attention is also visible, this provides much clearer information about what is happening and how the person feels about it.

Object-directed expression

Gaze can change perceptions of the objects at which it is directed as well as the faces doing the gazing. A 2007 study led by Andrew Bayliss showed that 'disgust' faces directing gaze at household objects worsened perceivers' evaluations of those objects. Gaze directed at other faces has similar effects. For examples, Mumenthaler and Sander (2012) found that a fear face was seen as more fearful when a scowling face gazed at it. Perceivers evidently registered that someone at whom anger is directed is likely to feel more afraid. This effect occurred even when the scowling face was presented subliminally (Mumenthaler & Sander, 2015), suggesting that perceivers are highly attuned to the relational properties of facial expressions.

More generally, these findings show that facial expressions convey information about objects at which they are directed. In social referencing, faces also regulate other people's behaviour towards those objects. For example, Sorce and colleagues' classic 1985 study showed that toddlers approaching an apparent precipice covered by glass were unwilling to cross this 'visual cliff' if their mothers showed fear faces rather than smiles, presumably because they concluded that venturing beyond the edge was too risky to attempt.

Similar processes can also operate in interactions between adults. My colleagues and I (Parkinson et al., 2012) recruited pairs of friends to complete a collaborative task involving real-time nonverbal interaction across a video link (with the sound turned off to avoid verbal communication). One friend (the player) inflated a simulated balloon by pressing a key on the computer (the Balloon Analogue Risk Task or 'BART', designed by Carl Lejuez and colleagues). Each pump earned additional points, but the balloon might pop at any time, losing all the points earned so far. Meanwhile, we surreptitiously instructed the other friend (the observer) either to express or suppress anxiety as they watched the player's progress from a separate cubicle. Players were less willing to inflate the balloon to the point of popping if observers expressed, rather than suppressed, anxiety. In other words, observers' ongoing facial reactions changed players' levels of risk-taking.

These results also show that facial expressions can communicate object-directed information by means other than gaze orientation. The real-time coordination between facial movements and a simultaneously unfolding sequence of events made it obvious what they were communicating. In fact, dynamic expressions often provide clearer information than static expressions even when no further contextual information is available (e.g. Krumhuber et al., 2013), perhaps because their temporally contoured changes provide stronger clues about the nature of the events to which they refer.

Facial expressions as communications

So far, I have focused mainly on the perception of facial expressions and its effects on other people's judgements and behaviour. But what about the production of facial expressions? Does that also depend on context? Indeed, do our faces move in certain ways precisely to exert effects on other people?

Although many psychologists assume that facial expressions are symptoms of internal states, other researchers see them instead as ways of displaying our orientation to what is happening to the other people around (e.g. Alun Fridlund's 1994 Human Facial Expression: An Evolutionary View). For example, an 'anger' face conveys the message 'Back off or I'll attack!' and a 'fear' face serves as an appeal for consolation or support.

Clear evidence that the production of facial expression is specifically attuned to interpersonal responses comes from studies into audience effects (see Parkinson, 2005). These studies consistently demonstrate that many kinds of expression are more common or more intense when an appropriate addressee is available. For example, viewers laugh more when watching comedy films in the presence of others, especially friends who are close by (e.g. Chapman & Wright, 1976). Similarly, bowlers smile less when the skittles first fall than after turning to face their friends (Kraut & Johnston, 1979). Their expressions thus seem to be communications directed to other players rather than symptoms of a direct emotional reaction to success. Indeed, Bavelas and colleagues (1986) showed that empathic winces in response to an observed injury were precisely timed to capitalise on eye contact with the victim. Observers apparently made sure that their facial response would be seen before showing vicarious pain.

Interpersonal emotion regulation

The face's capacity to alter other people's responses to events permits us to use facial expression strategically as a means of social influence. In the social referencing studies described above, participants regulated their other-directed fear expressions, thus affecting the other person's risk-taking. Outside the laboratory, it seems likely that people make similar regulatory attempts in order to influence other people's emotional reactions. For example, I might try to look more worried because I want you to be less blasé about something that is about to happen (interpersonal alerting). Thus, regulating my emotion can also serve to regulate your emotion.

Worry expression not only communicates that a concern may need to be addressed, but also indicates a potential need for support or comfort (Parkinson & Simons, 2012). Correspondingly, interpersonal responses to someone else's expressed worry may involve either expressing worry yourself (acknowledging a perceived alerting attempt) or displaying the contrasting emotion of calmness (in response to a perceived request for comfort). But what happens when the interpersonal response does not match the intention of the person expressing worry in the first place?

When your partner's worry expression is intended to solicit comfort, providing this comfort is likely to alleviate the worry. However, when your partner's worry expression is intended to alert you to a cause for concern, any attempt to provide reassurance or comfort is likely to backfire. Attempting to minimise the seriousness of the situation is likely to encourage intensified worry expression in a redoubled effort to persuade you that the impending concern should not be treated so lightly.  

Some preliminary evidence for the operation of these interpersonal dynamics was provided by my research group's recent study of romantic partners' conversations about shared concerns (Parkinson et al., 2016). Our specific focus was on how one partner's emotion-regulation style affected the other partner's use of emotional expression. For example, does selectively reinterpreting negative aspects of the situation to reduce unpleasant emotional reactions (reappraisal) necessarily make your partner feel better about things too? In fact, we found that partners of habitual reappraisers were more likely to express worry in order to make those reappraisers appreciate the seriousness of events (interpersonal alerting). In other words, your partner may try to counteract your attempt to see things as less worrying by communicating greater worry themselves. In some circumstances, this might lead to a self-defeating interpersonal spiral, where one partner increasingly tries to alert the other partner to the worrying nature of concerns, only to be met with increasing calmness. In our study, it was also evident that the female partner was significantly more likely to do the alerting and the male partner was more likely to adopt a calming demeanour, suggesting that gender-role stereotypes relating to masculine mastery impact on interpersonal emotion regulation in heterosexual relationships.

Faces in social interaction

As we have seen, facial expressions can communicate information about the person expressing them, the object or person at which they are directed, and the relation between that person and object. We have also seen how facial communication is attuned to the other person's anticipated and actual reactions. How then do these processes fit together when faces interact with other faces in ongoing interactions? Intensive analyses of how such interpersonal processes play out are still relatively rare, but recent methodological advances are likely to make them increasingly common (e.g. Butler, 2011).

A few earlier studies explored temporal contingencies between partners' emotional behaviours, but usually lumped together verbal and nonverbal expressions making it difficult to isolate specific effects of facial communication. For example, Biglan and colleagues (1985) focused on problem-solving interactions between wives suffering from depression and their husbands. Wives were more likely to express depression immediately following aggressive behaviour from their husbands, and husbands were less likely to behave aggressively immediately following their wives' expression of depression. It therefore seems possible that one interpersonal function of expressing depression is to solicit less aggressive treatment from your partner. Indeed, some theorists have suggested that depression is partly sustained by the favourable responses it can solicit from others. However, Coyne (1976) argued that these initially positive interpersonal effects are likely to diminish over time as partners become susceptible to growing compassion fatigue. Indeed, studies of interactions between depressed partners in longer-term relationships show different interpersonal contingencies (e.g. Johnson & Jacob, 2000). Recent advances in automated coding of facial expressions may permit a closer focus on interdependencies between partners' communication patterns and how they may change during an ongoing interaction and across the wider time-frame of a developing relationship.

Pulling faces out of context

Given that the social context has such consistent effects on the perception and production of facial expressions, how is it that some decontextualised faces still seem to convey consistent emotional meanings? One answer is that the highly selected facial stimuli used in most judgement studies provide clear clues both about the nature of the object at which they are directed and about the person's relation to that object. For example, the scrunched nose of the 'disgust' face suggests that something smells bad, prompting metaphorical associations with the related emotion. Similarly, the focused stare of an 'angry' face coupled with the furrowed brow and pressed lips suggests that the person is directing attention at something that they want to resist or push away. In fact, perceivers who are not directly asked to infer emotional meanings often describe these facial expressions in situational and behavioural terms.

The fact that faces trigger inferences about what else might be happening allows them to serve as a kind of shorthand for emotional situations in everyday conversation. I can show that I empathise with your pain by wincing, or sympathise with your moral disapproval by scrunching my nose, for instance. However, not all associations between situations and emotions are consistent across cultures, meaning that the same face may come to stand for different emotions.

One striking example of such a dissociation is provided by recent research conducted in a small-scale society that remains relatively isolated from external influence. In a study published last year by Carlos Crivelli's team, investigators showed photographs of facial expressions to participants from the Trobriand Islands in Papua New Guinea, and asked those participants to point to the picture corresponding to the relevant 'basic emotion'. Unexpectedly, islanders pointed most often to the Western 'fear' face when asked to select the anger face. They were also most likely to point to this same 'fear' face when asked to select the face communicating the message: 'Back off or I will attack you'. In interpreting these findings, Crivelli and colleagues point out that some societies also use similar 'gasping' facial configurations as ritualised symbols of threat (for example the Maori threat face). In other words, a supposedly universal expression of fear in fact communicates a different emotion in societies where it is associated with a different kind of behavioural intention or situational context.

Conclusion

In The Expression of Emotions in Man and Animals Charles Darwin rejected the idea that facial muscles were 'specially created for the sake of expression'. Instead, he argued that facial movements performed more direct adaptive functions such as biting or stimulus rejection ('serviceable associated habits'), and that their connections with emotion arose as a secondary consequence. Because someone getting ready to attack is often angry, the ready-to-attack face also became associated with anger. Paradoxically, however, many emotion researchers inspired by Darwin soon reverted to precisely the view that he had set aside. Just as eyes are seen as windows to the soul, faces continue to be investigated as if they are direct readouts of internal emotions.

Here I have taken a different tack by focusing on how faces communicate information about the expresser's orientation to external objects and events, and on how their interpersonal effects depend on the relational meanings they convey. But what if we pull the face out of this context and strip away all the relational processes surrounding it? Does that leave an unadulterated expression of underlying emotion? Or do we instead read context into static pictures of expressions, by imagining what kind of situation the person must be in to be showing a face like that, looking in that intent way, wanting to approach or withdraw? Perhaps the fact that some highly selected faces imply context so strongly explains the consistent emotional inferences people make when judging them. And perhaps people learn to show faces of a similar kind precisely because of their capacity to prompt contextual inferences in others.

Psychological research into facial expression has moved beyond the restrictive tasks used in early judgement studies. Investigating faces in relation to their social context permits us to develop more sophisticated theories that unsettle some of our previous assumptions and distinctions. For example, it is no longer clear that the distinctive facial configurations associated with six or so basic emotions correspond to spontaneous facial reactions produced when experiencing those emotions. The same emotion may be expressed in a variety of different ways (or not at all) depending on context. Further, the interpersonal orientation of facial expressions means that they may share some of the characteristics of more deliberate forms of facial communication, including conversational gestures. Faces do a lot of things in addition to communicating emotional meanings. It is important to understand their operation in the context of these other activities.

Box: Are some emotions more basic than others?

Most studies investigating facial expressions have focused on the six so-called 'basic' or 'primary' emotions, which many psychologists believe have a special status because they have evolved to meet important adaptive or reproductive challenges. For example, fear prepares the body to deal with impending environmental threats and attunes sensory systems to relevant stimuli. Other 'nonbasic' or 'secondary' emotions – such as guilt, embarrassment, or pride – are often seen as emerging later in infant development as combinations or articulations of these basic emotions, and as being less likely to have distinctive facial expressions.

However, the distinction between basic and nonbasic emotions may not be as clearcut as once believed (e.g. Draghi-Lorenz et al., 2001). Further, some studies have begun to suggest that even nonbasic emotions can be inferred from nonverbal information if facial stimuli are dynamic rather than static, if vocal cues are added, or if the pictures include bodies as well as faces (e.g. see Dacher Keltner and Daniel Cordaro's work on understanding multimodal emotional expressions).

Meet the author

'Although I'm usually willing to give others the benefit of the doubt in everyday conversations, something about the scientific claims they make always brings out my sceptical side. That's why I've always resisted the idea that emotions are simply and entirely private and subjective feelings. For me, they operate in the spaces between rather than within people. And facial movements are part of this process, not just expressions of something inside. Once you start to look at faces from that perspective, it's hard to stop appreciating the subtlety, dynamism and flexibility in how they work.'

- Brian Parkinson is at the University of Oxford
[email protected]

References

Adams, R.B. & Kleck, R.E. (2005). Effects of direct and averted gaze on the perception of facially communicated emotion. Emotion, 5, 3–11.

Argyle, M. & Cook, M. (1976). Gaze and mutual gaze. Cambridge: Cambridge University Press.

Bavelas, J.B., Black, A., Lemery, C.R. & Mullett, J. (1986). 'I show how you feel': Motor mimicry as a communicative act. Journal of Personality and Social Psychology, 50, 322–329.

Bayliss, A.P., Frischen, A., Fenske, M.J. & Tipper, S.P. (2007). Affective evaluations of objects are influenced by observed gaze direction and emotion expression. Cognition, 104, 644–653.

Butler, E.A. (2011). Temporal interpersonal emotion systems: The 'TIES' that form relationships. Personality and Social Psychology Review, 15, 367–393.

Chapman, A.J. & Wright, D.S. (1976). Social enhancement of laughter: An experimental analysis of some companion variables. Journal of Experimental Child Psychology, 21, 201–218.

Coyne, J.C. (1976). Toward an interactional description of depression. Psychiatry, 39, 28–40.

Crivelli, C., Russell, J.A., Jarillo, S. & Fernández-Dols, J.M. (2016). The fear gasping face as a threat display in a Melanesian society. Proceedings of the National Academy of Sciences, USA, 113, 12403–12407.

Darwin, C. (1998). The expression of the emotions in man and animals (3^rd edn, P. Ekman, Ed.). London: HarperCollins. (Original work published 1872)

Draghi-Lorenz, R., Reddy, V. & Costall, A. (2001). Rethinking the development of 'nonbasic' emotions: A critical review of existing theories. Developmental Review, 21, 263–304.

Ekman, P., Sorenson, E.R. & Friesen, W.V. (1969). Pan-cultural elements in the facial displays of emotion. Science, 164, 86–88.

Farroni, T., Mansfield, E.M., Lai, C. & Johnson, M.H. (2003). Infants perceiving and acting on the eyes: Tests of an evolutionary hypothesis. Journal of Experimental Child Psychology, 85, 199–212.

Fridlund, A.J. (1994). Human facial expression: An evolutionary view. San Diego, CA: Academic Press.

Keltner, D. & Cordaro, D.T. (2016). Understanding multimodal emotional expressions: Recent advances in basic emotion theory. Emotion Researcher. Retrieved from http://emotionresearcher.com/understanding-multimodal-emotional-expressions-recent-advances-in-basic-emotion-theory

Kraut, R.E. & Johnston, R.E. (1979). Social and emotional messages of smiling: An ethological approach. Journal of Personality and Social Psychology, 37, 1539–1553.

Krumhuber, E.G., Kappas, A. & Manstead, A.S.R. (2013). Effects of dynamic aspects of facial expressions: A review. Emotion Review, 5, 41–46.

Lejuez, C.W., Read, J.P., Kahler, C.W. et al. (2002). Evaluation of a behavioral measure of risk taking: The Balloon Analogue Risk Task (BART). Journal of Experimental Psychology: Applied, 8, 75–84.

Mumenthaler, C. & Sander, D. (2012). Social appraisal influences recognition of emotions. Journal of Personality and Social Psychology, 102, 1118–1135.

Mumenthaler, C. & Sander, D. (2015). Automatic integration of social information in emotion recognition. Journal of Experimental Psychology: General, 144, 392–399.

Parkinson, B. (2005). Do facial movements express emotions or communicate motives? Personality and Social Psychology Review, 9, 278­–311.

Parkinson, B., Phiri, N. & Simons, G. (2012). Bursting with anxiety: Adult social referencing in an interpersonal Balloon Analogue Risk Task (BART). Emotion, 12, 817–826.

Parkinson, B. & Simons, G. (2012). Worry spreads: Interpersonal transfer of problem-related anxiety. Cognition and Emotion, 26, 462–479.

Parkinson, B., Simons, G. & Niven, K. (2016). Sharing concerns: Interpersonal worry regulation in romantic couples. Emotion, 16, 449–458.

Sander, D., Grandjean, D., Kaiser, S. et al. (2007). Interaction effects of perceived gaze direction and dynamic facial expression: Evidence for appraisal theories of emotion. European Journal of Cognitive Psychology, 19, 470–480.

Sorce, J.F., Emde, R.N., Campos, J. & Klinnert, M.D. (1985). Maternal emotional signaling: Its effect on the visual cliff behavior of 1 year olds. Developmental Psychology, 21, 195–200.

Tracy, J.L., & Robins, R.W. (2004). Show your pride: Evidence for a discrete emotion expression. Psychological Science, 15, 194–197.

Yik, M.S.M. & Russell, J.A. (1999). Interpretation of faces: A cross cultural study of a prediction from Fridlund's theory. Cognition and Emotion, 13, 93–104.