The multisensory perception of flavour

'No animal can live without food. Let us then pursue the corollary of this: namely, food is about the most important influence in determining the organization of the brain and the behavior that the brain organization dictates.' J.Z. Young (1968, p.21).

'Cooking is probably the most multi-sensual art. I try to stimulate all the senses.' (Ferran Adrià, El Bulli restaurant, Spain)

Cognitive psychologists and neuroscientists interested in the topic of multisensory integration have, until recently, largely kept away from the study of flavour perception, preferring instead to focus on the integration of auditory, visual and, to a lesser extent, tactile signals. However, the last few years have seen a rapid growth of interest by researchers trying to apply the insights uncovered in the psychophysics and neuroimaging laboratory to the study of multisensory flavour perception (Stevenson, 2009; Verhagen & Engelen, 2006).

As the above quote from J.Z. Young makes clear, food acquisition has likely played a critical role in shaping brain development throughout the course of human evolution. Indeed, food is one of the most effective stimuli in terms of modulating brain activity in hungry participants; with the sight and smell of appetising food leading to a 24 per cent increase in whole-brain metabolism in one PET study (see Wang et al., 2004).Our enjoyment of food and drink comes not only from the unified oral sensation of taste and smell (both orthonasal and retronasal), but also from the sound it makes, not to mention what it looks like. The oral-somatosensory qualities of foods are also very important; the texture, temperature, and even the pain, as in the case of eating chilli peppers, all contribute to the overall multisensory flavour experience (or gestalt: Verhagen & Engelen, 2006). The pleasure of food is critically dependent on all these sensory attributes being right, and so food can, for instance, be ruined simply by serving it at the wrong temperature, or if it has an inappropriate colour.

For the last 30 years or so, psychologists and cognitive neuroscientists have been investigating how our brains combine what they see, hear and feel in order to generate the rich and varied multisensory experiences that fill our everyday lives (see Calvert et al., 2004, for a review). Researchers have, for example, spent a lot of time trying to work out why it is that people perceive the voice of a ventriloquist as coming from the moving lips of the dummy, and why people hear better when they put their glasses on. By studying such cross-modal illusions in the laboratory, scientists have started to uncover some of the fundamental rules governing multisensory perception. For example, they have discovered that people's perception is typically dominated by what they see (as in the case of the ventriloquist's dummy: Alais & Burr, 2004).

There is now a growing body of empirical evidence to show that Bayesian decision theory, incorporating the principle of maximum likelihood estimation, does a remarkably good job of predicting our perceptual experiences under conditions of intersensory conflict (see Ernst & Bülthoff, 2004, for a review). Meanwhile, neurophysiologists have shown that individually weakly effective (i.e. near-threshold) stimuli sometimes combine in a superadditive manner, giving rise to multisensory experiences that are more intense, and richer, than would be predicted by the simple linear combination of their individual parts (Stein & Meredith, 1993; Stein & Stanford, 2008). Similar superadditive brain responses have now been documented in the human orbitofrontal cortex (the part of the brain that controls our perception of the pleasantness and reward value of food) in response to congruent combinationsof olfactory, gustatory and visual stimuli (de Araujo et al., 2003).

The latest evidence now shows that many of the same rules of multisensory integration/perception (such as sensory dominance and superadditivity) also help to explain why food and drink taste the way they do, and why what tastes nice to one person may taste awful to another. While much of the research in this area requires the use of computer-controlled olfactometers and gustometers, it is important to note that fundamental studies on flavour perception can be carried out with nothing more complex than a tub of ice-cream or a packet of sweets from the corner shop (e.g. Gallace et al., submitted; Levitan et al., 2008; Shankar et al., 2009; Yeomans et al., 2008).

Taking the confusion out of fusion

Flavour perception is, however, not only of interest to psychologists and cognitive neuroscientists but also to an increasing number of philosophers (e.g. Smith, 2007), given the conceptual uncertainty surrounding how exactly we should define flavour (Spence et al., 2010). Should flavour be considered as a separate sensory modality, like vision and hearing? Or is it instead more appropriate to consider flavour as a kind of perceptual system (Auvray & Spence, 2008; Stevenson, 2009)? One thing is certain, though, we all find it extremely difficult to distinguish between tastes and smells. Think only of how hard it is to taste food when your nose is blocked; under such conditions, it is, of course, only your sense of smell that is actually impaired. Some researchers have argued that the ubiquitous nature of such taste–smell confusions may reflect a form of synaesthesia that is common to us all (Auvray & Spence, 2008; Rozin, 1982; Stevenson & Tomiczek, 2007). What is clear it that the extremely tight coupling between the senses of taste and smell means that terms such as
'a sweet smell' reflect much more than a merely metaphorical use of language (Stevenson & Boakes, 2004).

For example, placing a drop of sucrose on the tongue (be it at a sub- or supra-threshold level) leads to a dramatic increase in the ability to detect the presence of a congruent odorant such as benzaldehyde (which has an almond-like odour familiar to those who like Bakewell tart; Dalton et al., 2000). Interestingly, however, this particular multisensory effect appears to be culture-/experience-specific: While Europeans and North Americans, for whom the combination of almond and sugar is very common (in marzipan, for example), are able to integrate this particular combination of stimuli, they show no such multisensory enhancement effect when an almond odour is paired with a salty taste (monosodium glutamate). By contrast, the Japanese, for whom almond odor is associated with salty tastes in pickled condiments, typically show the reverse effect. That is, they exhibit enhanced responding to the combination of almond and salt, but not to almond and sugar. Results such as these suggest that our brains learn to combine just those tastes and smells (i.e. flavours) that commonly co-occur in the foods we eat (Blumenthal, 2008; Spence, in press). In fact, we apparently start learning our responses to flavours before we have even left the womb (Schaal et al., 2000).

The ventriloquist in your mouth

While many people might think that the ventriloquism illusion only occurs in the cinema or when watching a ventriloquist's dummy, the latest evidence suggests that a similar effect also occurs in the mouth. Ventriloquism may help explain why it is that we always seem to perceive flavours in the mouth (rather than in the nose, where the majority of flavour perception originates): olfactory and gustatory stimuli, which are difficult to localise, are perceived as emanating from the mouth because they are referred to the highly localisable tactile sensations associated with mastication. The ventriloquism illusion may also play a crucial role in one of the classic dishes on the tasting menu at Heston Blumenthal's restaurant in Bray, The Fat Duck. When bacon-and-egg ice-cream was first created, it was only moderately pleasant; the flavours did not appear to stand out from one another. The breakthrough came when a piece of crispy fried bread was added to the plate. While the bread does not, in-and-of-itself, impart much flavour to the dish, its addition brought the dish alive, seemingly helping to separate the bacon and egg flavours. It appears as though the bacon
is 'ventriloquised' towards, and hence becomes perceptually localised within, the crispy bread, while the eggy flavour stays behind in the more texturally appropriate soft ice-cream.

Sound bites

Try eating a crisp (or potato chip) without making a noise. It's simply impossible! Do such food-related sounds exert any influence on our perception? Experiments conducted here at the Crossmodal Research Laboratory in Oxford have demonstrated that food-eating sounds contribute to the perception of crispness and freshness in foods such as crisps, biscuits, breakfast cereals and vegetables (Zampini & Spence, 2004). The participants in our study (for which we were awarded the 2008 IG Nobel prize for nutrition) had to bite into 180 potato chips and rate each one on its perceived crispness and freshness. Pringles were ideal for this research because they are all more-or-less identical in size, shape and consistency. The crisp-biting sounds were picked up by microphone, modified and then immediately played back over headphones. The crisps were rated as tasting significantly crisper (and fresher) if the overall sound level was increased, or if just the high-frequency components of the crisp-biting sound were boosted. People's perception of the carbonation of a beverage served in a cup is also modulated by what they hear (Spence & Shankar, 2010). Companies such as Unilever, Proctor & Gamble and Nestlé have now started to use this psychologically inspired technique in-house when developing their new dry-food products.

Grape expectations

Many studies published over the last 75 years have shown that visual cues (especially those concerned with a food or drink's colour) can have a profound effect on both the sensory-discriminative and hedonic aspects of multisensory flavour perception (Spence et al., 2010). So, for example, people find it particularly difficult to correctly identify fruit-flavoured drinks that have been coloured incongruently, as when an orange-flavoured drink is artificially coloured green. Similarly, many people will reportedly run to the bathroom to be ill if they find out that the steak that they have been eating is actually coloured blue. More seriously, multisensory research on flavour may turn out to have important health implications, given that, for example, beverages can be made to taste as much as 11 per cent sweeter simply by adding the appropriate amount of red food colouring. Colours associated with the ripening of fruits are particularly effective in modulating perceived sweetness, while foods and drinks with a green hue (i.e. associated with unripe fruits) are often judged to be sourer. As yet, however, researchers have been unable to modulate people's perception
of saltiness using food colouring. The problem here is that salty foods come in all manner of colours, and hence there is no natural colour–taste association on which to build.

Wine is particularly interesting from the point of view of flavour research because it allows one to look at the effects of expertise. In what is perhaps the most famous study in this area, 54 students enrolled on a university oenology course in Bordeaux were given two glasses of wine to sniff, one white, the other red (Morrot et al., 2001); They had to indicate which of the wines presented a range of characteristics most intensely. Descriptors such as lemon, honey and straw were chosen for the white wine aroma, and another set of terms, such as prune, chocolate, tobacco, etc. to describe the red wine. A week later, the students were given two further glasses of wine (one white, the other red, just as before). This time, however, what looked like red wine was actually a white wine that had been artificially coloured red. The budding oenologists – who hadn't been made aware of the deception – all chose the red wine odor descriptors when evaluating the inappropriately coloured white wine.  When it comes to wine, people appear to smell what they see!

Professional wine tasters and wine makers also fall prey to this visual dominance effect (see Parr et al., 2003). If anything, they appear to be even more susceptible to such cross-modal effects than social drinkers. This is presumably because wine experts' expectations concerning the taste, aroma and flavour characteristics that are likely to be associated with a particular colour, and/or by any other visual appearance cues associated with the wine, are that much stronger than in the non-expert (Shankar et al., 2010; Spence, 2010).

Many fast-moving consumer goods companies have already started to play with the mapping between food colour and flavour. Think only of green tomato ketchup, or the recently introduced Fruity Smarties or Confused Skittles, where the colours and flavours of the individual candies have been deliberately mixed up. Similarly, some of the chefs that I work with, such as Heston Blumenthal, have also started to play with their diners' expectations through dishes such as the 'beetroot and orange jelly' dish. When the waiter brings this dish to the table they will recommend that the diner should start with the beetroot. People naturally go straight for the purple-coloured jelly. However, beetroots turn orange on cooking, while Heston uses blood oranges that naturally have that dark purple colour.

Environmental contributions

Thus far, I have focused on the contributions to multisensory flavour perception of the various sensory cues – gustatory, olfactory, oral-somatosensory, auditory and visual – that are intrinsic to food and drink. However, the packaging in which a food is presented, the name/label it is given (Gallace et al., submitted; Shankar et al., 2009; Yeomans et al., 2008), and even the environment in which it is served, play an equally important role in modulating our perception of, and responses to, food and drink (Spence & Shankar, 2010). When, after all, was the last time you enjoyed airplane food (see Green & Butts, 1945)? Everything from the red-and-white checked tablecloth in a Mediterranean restaurant through to the colour of the lighting and the background music all affect both the foods and drinks we choose to order and how much/quickly we consume them (Bell et al., 1994; see North & Hargreaves, 2008; Spence & Shankar, 2010, for reviews). But could environmental cues also influence the flavour and hedonic qualities of food and drink? The latest research suggests that they can.

In one study, Heston Blumenthal and I demonstrated that people rate bacon-and-egg ice-cream as tasting significantly more bacony when listening to the sound of sizzling bacon than to a farmyard of clucking chickens. In a second experiment, people rated oysters eaten while listening to the 'sound of the sea' (i.e. the sound of seagulls squawking and waves crashing gently on the beach) as tasting significantly more pleasant than oysters eaten while listening to the farmyard noises. Taken together, these results (Spence & Shankar, 2010) highlight just how dramatically environmental sounds can influence (or bias) people's perception of food.

These findings led directly to the introduction of the 'Sound of the Sea' dish at The Fat Duck. Diners are presented with a plate of food that is reminiscent of a beach (with foam, seaweed, and sand all visible on the plate). The dish also comes with a mini iPod hidden inside a sea-shell, with the earphones poking out. Diners are encouraged to insert the headphones (whereupon they hear the 'Sounds of the Sea' soundtrack) before starting to eat. The dish is currently the signature dish on the tasting menu. Wearing headphones also has the advantage that is focuses the diners' attention on the food itself.

Elsewhere, Oberfeld et al. (2009) have shown that the ambient lighting can also affect people's perception of the flavour, taste and market value of wine. They changed the colour of the ambient lighting at a winery on the Rhine from white to blue, red, or green, while 200 wine buyers rated Riesling (white) wines served in an opaque black glass (to ensure that the room colour did not impact on the colour of the wine itself). The customers liked the wine significantly more when it was tasted under blue and red lighting than under green or white lighting. What is more, the maximum price that they were willing to pay was also nearly 50 per cent higher for wine tasted under red lighting than when the same wine was evaluated under green or white lighting. Follow-up experiments revealed that blue and green room lighting made wines taste spicier and fruitier, while a Riesling was rated as tasting nearly 50 per cent sweeter under red lighting than under blue or white lighting.

In another recent study, Gal et al. (2007) demonstrated that the brightness of the ambient lighting can also affect people's consumption of coffee. People who like strong coffee were found to drink more of the stuff under bright lighting than under dim lighting, whereas the reverse was true for those who preferred their coffee weak. Taken together, these results therefore demonstrate that both the intensity and colour of the ambient lighting can affect people's perception of, liking for, and even their consumption of, drinks such as wine and coffee. Our perception of flavour involves not only the multisensory integration of all of the available cues in the food itself, but can also be influenced by the environment in which we happen to eat and drink (North & Hargreaves, 2008; Spence & Shankar, 2010).

Conclusions

The last few years have seen many important advances in our understanding of the multisensory perception of flavour at both the psychological and neural levels (see Stevenson, 2009; Verhagen & Engelen, 2006, for reviews). Cognitive neuroscientists have uncovered many key rules (such as sensory dominance and superadditivity) used by the human brain in order to combine the different sensory signals when we eat and drink. It should come as no surprise then to learn that chefs and other food producers are now starting to prepare dishes that more effectively stimulate the senses of the customer/consumer, because they have been designed on the basis of these recently discovered rules of multisensory flavour perception (see Blumenthal, 2008). The hope is that psychological insights can be harnessed in order to make food healthier (by reducing sugar, fat, salt, etc.) while not sacrificing flavour.

Charles Spence is head of the Crossmodal Research Laboratory at the Department of Experimental Psychology, Oxford University
[email protected]

References

Alais, D. & Burr, D. (2004). The ventriloquist effect results from near-optimal bimodal integration. Current Biology, 14, 257–262.
Auvray, M. & Spence, C. (2008). The multisensory perception of flavor. Consciousness and Cognition, 17, 1016–1031.
Bell, R., Meiselam, H.L., Pierson, B.J. & Reeve, W.G. (1994). Effects of adding an Italian theme to a restaurant on the perceived ethnicity, acceptability, and selection of foods. Appetite, 22, 11–24.
Blumenthal, H. (2008). The big Fat Duck cookbook. London: Bloomsbury.
Calvert, G.A., Spence, C. & Stein, B.E. (Eds.) (2004). The handbook of multisensory processes. Cambridge, MA: MIT Press.
Dalton, P., Doolittle, N., Nagata, H. & Breslin, P.A.S. (2000). The merging of the senses. Nature Neuroscience, 3, 431–432.
de Araujo, I.E.T., Rolls, E.T., Kringelbach, M.L. et al. (2003). Taste–olfactory convergence, and the representation of the pleasantness of flavour, in the human brain. European Journal of Neuroscience, 18, 2059–2068.
Ernst, M.O. & Bülthoff, H.H. (2004). Merging the senses into a robust percept. Trends in Cognitive Sciences, 8, 162–169.
Gal, D., Wheeler, S.C. & Shiv, B. (2007). Cross-modal influences on gustatory perception. Manuscript submitted.
Gallace, A., Boschin, E. & Spence, C. (submitted). On the taste of 'Bouba' and 'Kiki': An exploration of word–food associations in neurologically normal participants. Cognitive Neuroscience.
Green, D.M. & Butts, J.S. (1945). Factors affecting acceptability of meals served in the air. Journal of the American Dietetic Association, 21, 415–419.
Levitan, C., Zampini, M., Li, R. & Spence, C. (2008). Assessing the role of color cues and people's beliefs about color-flavor associations on the discrimination of the flavor of sugar-coated chocolates. Chemical Senses, 33, 415–423.
Morrot, G., Brochet, F. & Dubourdieu, D. (2001). The color of odors. Brain and Language, 79, 309–320.
North, A. & Hargreaves, D. (2008). The social and applied psychology of music. Oxford: Oxford University Press.
Oberfeld, D., Hecht, H., Allendorf, U. & Wickelmaier, F. (2009). Ambient lighting modifies the flavor of wine. Journal of Sensory Studies, 24, 797–832.
Parr, W.V., White, K.G., & Heatherbell, D. (2003). The nose knows: Influence of colour on perception of wine aroma. Journal of Wine Research, 14, 79–101.
Rozin, P. (1982). 'Taste–smell confusions' and the duality of the olfactory sense. Perception & Psychophysics, 31, 397–401.
Schaal, B., Marlier, L. & Soussignan, R. (2000). Human foetuses learn odours from their pregnant mother's diet. Chemical Senses, 25, 729–737.
Shankar, M.U., Levitan, C.A., Prescott, J., & Spence, C. (2009). The influence of color and label information on flavor perception. Chemosensory Perception, 2, 53–58.
Shankar, M.U., Levitan, C. & Spence, C. (2010). Grape expectations: The role of cognitive influences in color–flavor interactions Consciousness & Cognition, 19, 380–390.
mith, B.C. (Ed.) (2007). Questions of taste: The philosophy of wine. Oxford: Oxford University Press.
Spence, C. (2010). The colour of wine – Part 1. The World of Fine Wine, 28, 122–129.
Spence, C. (in press). Multisensory integration & the psychophysics of flavour perception. In J. Chen & L. Engelen (Eds.) Food oral processing: Fundamentals of eating and sensory perception. Oxford: Blackwell.
Spence, C., Levitan, C., Shankar, M.U. & Zampini, M. (2010). Does food color influence taste and flavor perception in humans? Chemosensory Perception, 3, 68–84.
Spence, C. & Shankar, M.U. (2010). The influence of auditory cues on the perception of, and responses to, food and drink. Journal of Sensory Studies, 25, 406–430.
Stein, B.E. & Meredith, M.A. (1993). The merging of the senses. Cambridge, MA: MIT Press.
Stein, B.E. & Stanford, T.R. (2008). Multisensory integration. Nature Reviews Neuroscience, 9, 255–267.
Stevenson, R.J. (2009). The psychology of flavour. Oxford: OUP.
Stevenson, R.J. & Boakes, R.A. (2004). Sweet and sour smells. In G.A. Calvert, C. Spence & B.E. Stein (Eds.) The handbook of multisensory processing (pp.69–83). Cambridge, MA: MIT Press.
Stevenson, R.J. & Tomiczek, C. (2007). Olfactory-induced synesthesias. Psychological Bulletin, 133, 294–309.
Verhagen, J.V. & Engelen, L. (2006). The neurocognitive bases of human multimodal food perception. Neuroscience and Biobehavioral Reviews, 30, 613–650.
Wang, G-J., Volkow, N.D., Telang, F. et al. (2004). Exposure to appetitive food stimuli markedly activates the human brain. NeuroImage, 212, 1790–1797.
Yeomans, M., Chambers, L., Blumenthal, H. & Blake, A. (2008). The role of expectancy in sensory and hedonic evaluation: The case of smoked salmon ice-cream. Food Quality and Preference, 19, 565–573.
Young, J.Z. (1968). Influence of the mouth on the evolution of the brain. In P. Person (Ed.) Biology of the mouth. Washington, DC: American Association for the Advancement of Science.
Zampini, M. & Spence, C. (2004). The role of auditory cues in modulating the perceived crispness and staleness of potato chips. Journal of Sensory Science, 19, 347–363.