PII: S1364-6613(00)01626-0 TRENDS in Cognitive Sciences Vol.5 No.5 May 2001 http://tics.trends.com 1364-6613/01/$ – see front matter © 2001 Elsevier Science Ltd. All rights reserved. PII: S1364-6613(00)01626-0 211Review James Tanaka* Daniel Weiskopf Dept of Psychology, Oberlin College, Oberlin, OH 44074, USA. *e-mail: jim.tanaka@ cs.oberlin.edu Pepper Williams Dept of Psychology, University of Massachusetts-Boston, 100 Morrissey Blvd, Boston, MA 02125, USA. In a memorable scene from The Wizard of Oz, Dorothy is catapulted by a ferocious tornado from a drab, grey-scale Midwest existence into the land of Oz, a world that is portrayed in full-scale, technocolor vision. In the film, the transition from black-and-white to color images is a striking cinematographic technique alerting Dorothy and the audience that we are indeed ‘not in Kansas anymore’. However, whilst the aesthetic value of color in the visual arts is undeniable, the contribution of color to everyday vision, especially the stage of vision concerned with object recognition, is more controversial. Does the presence of color help us recognize objects in our world? Are we any faster or more accurate in recognizing an apple when it is seen in color versus black-and-white? This article focuses on the functional contributions that color makes to the way humans perceive and recognize objects. Drawing upon converging evidence from behavioral, neuroimaging and neuropsychological methodologies, we present a framework that emphasizes the interaction between object color perception and object color knowledge in recognition. Color in low-level vision Neurophysiological research has revealed that a significant amount of visual processing is dedicated to the analysis of color information. When light enters the eye, the composition of wavelength energy is captured by specialized retinal photoreceptors (cones), which in turn send their outputs to specific cells in the lateral geniculate nucleus (LGN) (Ref. 1). Wavelength- coded signals from the LGN, which are essential for the processing and perception of color information, are then transmitted along specialized pathways to cortical visual areas V1, V2 and V4 (Ref. 2). Given that the brain has developed specialized mechanisms to handle the color information in the visual environment, it is a fair question to ask what functional role color might play in everyday vision. Whereas other mammals possess dichromatic or monochromatic color vision, it is only primates that are endowed with three types of cone photoreceptors and thereby have trichromatic color vision. What ecological advantage does this give primates over animals with dichromatic or monochromatic vision? Recently, Sumner and Mollon found that the photopigments of primates are optimized for differentiating edible fruits and young leaves amongst a background of mature leaves3. In this case, the additional dimension of trichromatic color vision gives primates a behavioral advantage when having to select edible fruits and plants from a complex scene. Similarly, during the early stages of low-level visual processing, it has been shown that color is a useful cue for segregating and organizing visual input into three-dimensional objects and scenes4–6. For example, very brief presentations (16 ms), of natural scenes are matched more accurately by subjects when shown in color than when shown as luminance-controlled grey-scale images7. Thus, studies in low-level vision indicate that color provides an important source of information in the pre-recognition stage of visual processing. Color in high-level vision Although keen color vision might give humans an adaptive edge in the early stage of visual processing, the role that color plays in later stages of object recognition has been a point of contention in the literature. On one side of the issue, ‘edge-based’ theories, such as Biederman’s recognition-by- components model8, claim that objects are recognized solely on the basis of their shape properties. According to the edge-based approach, representations mediating initial object recognition contain information about an object’s shape, but no information about the surface properties of an object, such as its color or texture. By contrast, ‘surface-plus-edge-based’ theories allow for object representations to include information, not only about an object’s shape, but also about its surface properties, such as color and texture9,10. The competing claims of the two approaches should be testable in behavioral experiments by examining whether there is an advantage for recognizing the chromatic version of an object over its achromatic version. However, this relatively straightforward test has yielded mixed results. Some studies have shown that recognition times are essentially unaffected when objects are presented in their appropriate colors (e.g. a yellow banana), inappropriate colors (e.g. a purple banana) or in black-and-white11–13, which would support the Traditional theories of object recognition have emphasized the role of shape information in high-level vision. However, the accumulating behavioral, neuroimaging and neuropsychological evidence indicates that the surface color of an object affects its recognition. In this article, we discuss the research that examines the conditions under which color influences the operations of high-level vision and the neural substrates that might mediate these operations. The relationship between object color and object recognition is summarized in the ‘Shape + Surface’ model of high-level vision. The role of color in high-level vision James Tanaka, Daniel Weiskopf and Pepper Williams TRENDS in Cognitive Sciences Vol.5 No.5 May 2001 http://tics.trends.com 212 ReviewReview edge-based theories. However, other studies have found that appropriately colored objects are recognized faster than monochrome objects and inappropriately colored objects14–16, which is consistent with the surface-plus-edge-based position. There are several reasons why color might influence recognition in one situation and not another. First, the relative contribution of color to object recognition will greatly depend on the structural properties of relevant comparison objects. For example, when shopping for fruit, ‘yellow’ is an a important cue for differentiating lemons from limes, but yellow would not be critical for selecting lemons from pineapples. Thus, shape and color interact; color facilitates recognition of objects within structurally similar categories (e.g. animals, birds), but not necessarily structurally dissimilar categories (e.g. body parts, musical instruments, tools)16. Color diagnosticity in object and scene recognition Color can provide useful information for the recognition of some, but not all, objects. For example, although ‘red’ might be an informative cue for identifying fire engines, red is certainly not a very useful cue for identifying automobiles or bicycles. Objects that that are high in color ‘diagnosticity’ – that is, objects such as fire engines and lemons that appear in a consistent color – are the most likely candidates to show the effects of color in recognition (see Fig. 1). By contrast, objects such as cars and hammers, that do not consistently appear in a characteristic color are low in color diagnosticity and should be little affected by the availability of perceptual color information. Consistent with this prediction, Tanaka and Presnell showed that the presence or absence of color information has a significant impact on the recognition of isolated objects with high color-diagnosticity and little effect on the recognition of objects with low color- diagnosticity17. In a control condition, when the high-and low-color-diagnostic objects were matched for structural complexity, reliable color effects were still found, indicating that color made a unique contribution to recognition independently of shape. In addition to the recognition of isolated objects, color can also be diagnostic for recognition of everyday scenes7,18. For example, one experiment showed that scenes that are rich in color-diagnostic content (e.g. coast, canyon, desert, forest) are best recognized in their normal colors18. When the same scenes were shown in a luminance-only condition, they were recognized more readily than when shown in inappropriate colors (see Fig. 2). Non-color- diagnostic scenes (e.g. city, shopping area, road and bedroom), on the other hand, showed no difference in recognition across the normal-color, luminance-only and abnormal-color conditions. Thus, the concept of color diagnosticity generalizes to the recognition of color-diagnostic scenes as well as color-diagnostic objects (see also Box 1). What are the ecological benefits of representing objects and scenes in terms of both color and shape? Objects represented by color and shape might show a recognition advantage over objects represented by shape only in conditions where access to edge information is limited. For instance, given its distinctive yellow color, identifying a partially occluded banana should be easier than identifying a partially occluded can opener. That is, under less than ideal viewing conditions, multi-coded objects will suffer less than objects that are coded by a single dimension. It could be argued that in the real world, recognition-under-occlusion is more the rule than the exception, and hence, color might play an critical role in everyday object recognition. Perception and knowledge of object color in the brain Perceiving that an apple is red versus knowing that an apple is red are distinct cognitive operations. Above perception, the knowledge of object color requires an association between the color ‘red’ and the object ‘apple’. At the neuroanatomical level, distinct neural regions appear to be differentially engaged during the processes of color perception and the retrieval of visual color knowledge19–21. In color perception, when human subjects passively view Mondrian color displays (arrays of different colored patches), areas of the lingual and fusiform gyrus are differentially activated relative to when viewing the same patterns shown in grey-scale19,20. On the other hand, if subjects are asked to generate the color associate to an achromatic object (e.g. responding ‘yellow’ to a line drawing of a bulldozer) the left inferior temporal, frontal and Review Fig. 1. Images of high color-diagnostic objects shown in their appropriate colors (left) and inappropriate colors (right). For such objects, color plays an important role in recognition. Fig. 2. Images of a high color-diagnostic beach scene shown in appropriate colors (left) and inappropriate colors (right). Images provided courtesy of Aude Oliva and Philippe Schyns. TRENDS in Cognitive Sciences Vol.5 No.5 May 2001 http://tics.trends.com 213ReviewReview posterior parietal areas of the brain are differentially activated relative to naming the object. It is noteworthy that these are the same brain structures that have been closely linked to visual object perception and recognition22, which suggests that modality-specific, visual representations are activated by the object-color association task. Moreover, because the left inferior temporal area showed increased activation during retrieval of color knowledge relative to color naming of a colored version of the same picture (e.g. responding ‘yellow’ to a picture of an achromatic bulldozer versus responding ‘yellow’ to a picture of a yellow bulldozer), this region would seem to be related to the access of object color knowledge rather than simple lexical access (the word ‘yellow’). Similarly, the right anterior fusiform gyrus, hippocampus and parahippocampal gyrus showed increased activation during the mental imagery of high color-diagnostic objects (e.g. apple, banana)23. Collectively, these studies make two important points. First, retrieving information about an object’s color activates many of the same visual brain areas that are known to be involved in object recognition. Second, the neural areas activated by the retrieval of object color knowledge are separable from those areas activated by color perception. A more direct link between color knowledge and object recognition processes is demonstrated by a neuroimaging study by Zeki and Marini24. In their Review Artists are often well ahead of cognitive scientists in their experimentation with perception. For centuries artists have been aware that color can play a key role in object and scene recognition as a result of humans’ strong associations between colors and certain objects or scenes. Artists in the Impressionist era relied heavily on viewer’s color knowledge, realizing that color is sometimes the most integral element of scene recognition. If the colors and the interaction between colors are captured, then accurate shape representations of a scene are not necessary – and may even detract from the experience of the painting. For example, a painting from Monet’s famous series of water lilies (Fig. I) depicts forms that are so strongly associated with particular colors, such as blue water and green vegetation, that the colors themselves convey the objects and mood of the scene without the need for detailed shape representations. In direct contrast to the Impressionist artists, who exploited color associations, artists from the Fauvist tradition tried to ‘free color from its descriptive role in representation’ (Ref. a, p. 779). Finding unique applications for color was one of the main goals and achievements of artists such as Henri Matisse, André Derain, and Maurice Vlaminck. The Fauves used brilliant blocks of colors, often in ways that seemed inappropriate, to disrupt our traditional associations between colors and objects. Water was not necessarily blue for the Fauves, and trees were not always green. For example, the colors used in Derain’s The Pool of London (Fig. II) are vibrant and unnatural, such as the bright pink steamboat, the blue mountains, and the orange faces of the workers. This audacious use of color forces the viewers’ knowledge of color and shape to be at odds, bringing a new emotional dissonance to scenes that might otherwise be mundane. Artists have long been experimenters in perception, using color and shape to suggest, to reinforce, to violate, or simply to complicate what they assume to be their viewers’ traditional associations between color and form. Reference a Honour, H. and Fleming, J. (2000) The Visual Arts: A History, Prentice-Hall Fig. II. Derain The Pool of London. (Reproduced with permission of the Design and Artists Copyright Society). Fig. I. Monet Waterlilies. (Reproduced, with permission, from the Art Institute of Chicago.) Box 1. Color knowledge from an artist’s perspective study, participants viewed normally colored objects, abnormally colored objects and grey-scaled Mondrian displays. As expected, the naturally and unnaturally colored objects activated visual areas known to be involved in color processing – V1 and V4. However, naturally colored objects (e.g. red strawberries) also engaged the same brain structures, specifically, the fusiform gyrus, hippocampus, and ventrolateral portion of the frontal cortex, that were found to be activated by other tasks requiring the retrieval of object color. Curiously, unnaturally colored objects (e.g. purple strawberries) activated the dorsolateral area of the frontal cortex. It is not clear why different pathways should be activated by appropriately and inappropriately colored objects. However, the critical finding was that object color knowledge was demonstrated by the differential activity of specific brain regions during the recognition of high color- diagnostic objects. Neuropsychology of object color knowledge Neuropsychological studies have indicated that brain injury can result in a separation of object color knowledge from shape knowledge. For instance, patient JB demonstrated preserved shape knowledge in the absence of object color knowledge25. This patient could easily discriminate real objects from ‘nonsense’ objects (composites of other real objects) and could discriminate perceptual differences between colors. However, JB was not able to judge whether objects had appropriate or inappropriate colors suggesting that his color- knowledge system was compromised. Similarly, Luzzatti and Davidoff 26 described two patients who demonstrated an intact ability to name colors, but an impaired ability to associate the appropriate color with an object. Despite a preserved ability to perceive and name colors, these patients experienced a selective impairment in their knowledge of object color. Neuropsychological studies also suggest that within the system of object color knowledge, a finer division exists between visual color knowledge and verbal color knowledge. For example, patient RV could perform verbal tasks that involved the color naming of abstract terms (e.g. ‘What color name would you give communists?’) and concrete objects (e.g. ‘What color is a gherkin?’)27. However, despite intact color perception, when asked to point to the correctly colored picture of an object, RV was severely impaired unless he was allowed to rely on verbal mediation. Thus, his verbal knowledge of object color seemed to be intact whereas his visual knowledge of color was compromised. By contrast, patient MP demonstrated intact visual and verbal knowledge about colors, but was not able to link the two types of knowledge. For instance, MP could point to the correct color of an imagined or visually presented line drawing of an object, as well as verbally report the color associated with an object (e.g. an apple is ‘red’). Thus, within the verbal or visual modality, he demonstrated intact object color knowledge. However, he was unable to report verbally the color of a visually presented picture or point to the color of a verbally spoken object (e.g. ‘point to the color of a strawberry’). Thus, MP seemed to lack the ability to integrate information across the visual and verbal domains of object color. In summary, the neuropsychological evidence indicates that knowledge of color is stored in both a verbal format and a visual format with close ties that normally bind the two types of knowledge. The Shape ++ Surface model of object recognition According to the ‘Shape + Surface’ model of object recognition (Fig. 3), color provides one of the perceptual inputs into the object representation system. As depicted by the larger ‘shape’ input box, the model acknowledges that object recognition is primarily a shape-driven system (e.g. blue rabbits are still recognized as rabbits). However, the model maintains that the color plays a supporting role in the recognition of high color-diagnostic objects and scenes. Thus, in contrast to other theories of object recognition (e.g. Ref. 28), the Shape + Surface model allows for objects to be represented in terms of both their shape and color. As indicated in Fig. 3, an unanswered question that we are currently pursuing in our laboratory is whether other types of surface information, specifically texture information, can influence the recognition process. The Shape + Surface model also draws a distinction between the perceptual object color at the input level versus stored visual color knowledge. According to the model, visual color knowledge can be triggered either by the perceptual object during object recognition or by its lexical label during mental imagery. The bi-directional arrow between color knowledge and the object representation (Fig. 3) indicates that prior color associations can have TRENDS in Cognitive Sciences Vol.5 No.5 May 2001 http://tics.trends.com 214 Review TRENDS in Cognitive Sciences Color Shape Texture? Object name Visual color knowledge Verbal color knowledge Object representation P e rc e p tu a l i n p u ts Fig. 3. The ‘Shape + Surface’ model of object recognition. As seen from the inputs on the left, this model allows for objects to be represented in terms of both their shape and color (and possibly texture). Visual color knowledge can be triggered either by the perceptual object during object recognition or by its lexical label during mental imagery (right). top-down effects on the perceptual processes involved in recognition. For example, it has been shown that, in an object recognition task, stronger interference effects were produced by inconsistent semantic color associations than by inconsistent perceptual color associations29. Similarly to other interactive models of letter and object perception30, the Shape + Surface model posits that object recognition is jointly determined by the bottom-up influence of perceptual color and the top-down influence of color knowledge. Finally, to take into account the neuropsychological evidence, the model maintains a separation between linguistic and visual representations of object color. For example, it is possible to know that apples are red without having to consult some kind of visual representation. Conclusion In summary, the converging behavioral, neurophysiological and neuropsychological evidence demonstrate that color plays a critical role in both low-level and high-level vision. At the lower level, color segments the complex visual input into coherent regions, thereby helping to differentiate objects from the background. At the higher level of recognition, objects and scenes imbued with characteristic colors are recognized more readily when seen in their natural colors than when not. Thus, beyond its aesthetic qualities, color enhances the manner in which we perceive and recognize objects in our everyday world. TRENDS in Cognitive Sciences Vol.5 No.5 May 2001 http://tics.trends.com 215Review • The converging evidence suggests that objects are recognized by virtue of their color as well as their shape. Do other types of surface information also influence object recognition processes – specifically, is recognition affected by an object’s surface texture? If so, what are the neural substrates related to the processing of an object’s texture? Like object color, can the perception of object texture be neurologically distinguished from knowledge of object texture? • What is the relationship between object color and the expertise of the perceiver? One hallmark of expert recognition is that experts initially recognize objects in their domain of expertise at a more specific level of categorization than novices. For example, expert birdwatchers recognize birds at the subordinate level of ‘greenfinch’ or ‘sparrow’ whereas novices recognize bird objects at the basic level of ‘bird’. To what extent does knowledge of object color facilitate the rapid subordinate-level recognition of the expert? Does the presence (or absence) of accurate color information affect the recognition performance of experts more than novices? • For the recognition of rigid and semi-rigid objects, the encoding of shape information takes precedence over the encoding of surface information. However, for the recognition of mass objects (e.g. water, sand), the converse seems true: surface information is more important than shape information. For example, blue rabbits are still recognized as rabbits, but are clouds still recognized as clouds if they are blue rather than white, or smooth rather than fluffy? What are the mechanisms underlying the recognition of mass objects and how might they differ from the mechanisms governing the recognition of rigid and semi-rigid objects? Questions for future research References 1 Livingstone, M.D. and Hubel, D.H. (1987) Psychophysical evidence for separate channels for the perception of form, color, movement and depth. J. Neurosci. 7, 3416–3468 2 Zeki, S. et al. (1991) A direct demonstration of functional specialization in the human visual cortex. J. Neurosci. 11, 641–649 3 Sumner, P. and Mollon, J.D. (2000) Catarrhine photopigments are optimized for detecting targets against a foliage background. J. Exp. Biol. 203, 1963–1986 4 Callaghan, T. (1984) Dimensional interaction of hue and brightness in preattentive field segregation. Percept. Psychophys. 36, 25–34 5 Cavanagh, P. (1987) Reconstructing the third dimension: interactions between color, texture, motion, binocular disparity and shape. Comput. Vis. Graphics Image Process. 37, 171–195 6 Troscianko, T. and Harris, J. (1988) Phase discrimination in compound chromatic gratings. Vis. Res. 28, 1041–1049 7 Gegenfurtner, K.R. and Rieger, J. (2000) Sensory and cognitive contributions of color to the recognition of natural scenes. Curr. Biol. 10, 805–808 8 Biederman, I. and Gerhardstein, P. (1993) Recognizing depth-rotated objects: evidence and conditions for three-dimensional viewpoint invariance. J. Exp. Psychol. Hum. Percept. Perform. 19, 1162–1182 9 Bruner, J. (1957) Contemporary Approaches to Cognition, Harvard University Press 10 Gibson, E. (1969) Principles of Perceptual Learning and Development, Appleton-Century-Crofts 11 Biederman, I. and Ju, G. (1988) Surface versus edge-based determinants of visual recognition. Cognit. Psychol. 20, 38–64 12 Davidoff, J. and Ostergaard, A. (1988) The role of color in categorical judgments. Q. J. Exp. Psychol. 40, 533–544 13 Ostergaard, A. and Davidoff, J. (1985) Some effects of color on naming and recognition of objects. J. Exp. Psychol. Learn. Mem. Cognit. 11, 579–587 14 Humphrey, G. et al. (1994) The role of surface information in object recognition: studies of visual form agnosic and normal subjects. Perception 23, 1457–1481 15 Wurm, L. et al. (1993) Color improves object recognition in normal and low vision. J. Exp. Psychol. Hum. Percept. Perform. 19, 899–911 16 Price, C.J. and Humphreys, G.W. (1989) The effects of surface detail on object categorization and naming. Q. J. Exp. Psychol. 41A, 797–828 17 Tanaka, J.W. and Presnell, L.M. (1999) Color diagnosticity in object recognition. Percept. Psychophys. 61, 1140–1153 18 Oliva, A. and Schyns, P. (2000) Diagnostic colors mediate scene recognition. Cognit. Psychol. 41, 176–210 19 Chao, L. and Martin, A. (1999) Cortical regions associated with perceiving, naming and knowing about colors. J. Cogn. Neurosci. 11, 25–35 20 Martin, A. et al. (1995) Discrete cortical regions associated with knowledge of color and knowledge of action. Science 270, 102–105 21 Wiggs, C. et al. (1999) Neural correlates of semantic and episodic memory retrieval. Neuropsychologia 37, 103–118 22 Ungerleider, L. and Mishkin, M. (1982) Two cortical visual systems. In Analysis of visual behavior (Ingle, D.J., ed.) pp. 549–586, MIT Press 23 Howard, R.J. et al. (1998) The functional anatomy of imagining and perceiving colour. NeuroReport 9, 1019–1023 24 Zeki, S. and Marini, L. (1998) Three cortical stages of colour processing in the human brain. Brain 121, 1669–1685 25 Riddoch, M. and Humphreys, G., eds (1987) Picture Naming, Erlbaum 26 Luzzatti, C. and Davidoff, J. (1994) Impaired retrieval of object-color knowledge with preserved color naming. Neuropsychologia 32, 933–950 27 Beauvois, M.F. and Saillant, B. (1985) Optic aphasia for colours and colour agnosia: a distinction between visual and visuo–verbal impairments in the processing of colours. Cognit. Neuropsychol, 2, 1–48 28 Davidoff, J. (1991) Cognition Through Color, MIT Press 29 Joseph, J.E. and Proffitt, D. (1996) Semantic versus perceptual influences of color in object recogniton. J. Exp. Psychol. Learn. Mem. Cognit. 22, 407–429 30 Rumelhart, D.E. et al. (1986) A general framework for parallel distributed processing. In Parallel Distributed Processing Explorations in the Microstructure of Cognition (Vol. 1), pp. 45–109, MIT Press