Artful media - interactive and informative art - Multimedia, IEEE Neil de Grasse Tyson, director of the HaydenPlanetarium, and I were wandering around Central London. He spotted an antique pen shop, and wanted to stop in. It was 1998, and we were in town to meet with some folks helping to build the digital dome system for the Rose Cen- ter for Earth and Space at the American Museum of Natural History, to open in 2000. It turns out that Tyson is an antique pen buff. He’s also a lateral thinker, and looking at the old fountain pens and pencils got him going. There isn’t much difference, he said, between these old writing implements and the state-of-the-art star projector for the planetarium; the dome is the paper, and the projector is no different than the modified quill pen that Leonardo da Vinci used to render the astronomical phenomena he observed. Like da Vinci, we had to have the eyes of an artist as well as a scientist to communicate our ideas most effectively. One of my goals in designing interactive museum exhibits has been to do just that— bringing art to the service of science, to make art that informs. Before my postgraduate work in interactive media, I studied traditional art, using my own share of pens and paper; and journalism, in which the TV camera is called the “400-pound pencil.” I also studied anthropology, which devotes itself to observation and description. What storytellers in all media share is the need to make the complex understandable while reducing, compressing, and editing to fit space and time restrictions. Computers and art Computers were, of course, created by the sci- entific establishment, and the continued infusion of scientific knowledge—most recently from biol- ogy—now enable the nearly complete simulation of nature. Genetic algorithms let software and hardware evolve, self-modifying and selecting versions with desired traits (see for example http://www.cogs.susx.ac.uk/users/adrianth/). Flocking and swarming behaviors in animals show animators how to make crowds move. Cel- lular automata based on simple rules can recreate the intricate, complex patterns found in nature.1 A hundred years ago, art was at a similar stage, with photography and film having attained a pretty accurate representation of reality. In response, artists such as Pablo Picasso, Paul Klee, and Vasily Kandinsky began to experiment with different sorts of visualizations—of the world, of the mind, of subjects not grounded in reality. Some sought a vague spiritual aspect while oth- ers wanted merely to make the invisible visible, instead of merely reproducing reality. Coincidentally, the concepts of relativity and cultural relativism arose at around the same time. It could be, as Kandinsky argued, that contem- plating the abstract enabled us to see “the rhythm in-between.”2 Just as in the physical world, emergent properties arise from the inter- actions between things; just as music is what’s between the notes (as postulated by Kandinsky’s contemporary, Claude Debussy); just as you can’t break down a chocolate cake back into its origi- nal ingredients—a completed artwork has under- gone a change of state. Does abstract art convey information? All of these artists would certainly say so. So too would contemporary artists such as Gerhard Richter. Just as a work of realism is a model of reality, he said, an abstract work is a fictive model, a sort of model of an invisible reality. What better way to describe something incomprehensible?3 Digital media Given their limitless possibilities, perhaps dig- ital media could do an even better job. It seems too soon to tell, but I believe that it holds promise. Although I haven’t seen any digital work with the invisible, emotional power of a Bach sonata or a Picasso painting, the media are beginning to break out of the early stages of experimentation. 4 1070-986X/03/$17.00 © 2003 IEEE Published by the IEEE Computer Society Artful Media Editor: Dorée Duncan SeligmanAvaya Labs Kevin Walker Exhibit Research Interactive and Informative Art One problem is that, with the computer screen (as with the canvas), everything is reduced to a 2D plane, no matter how accurate the 3D rendering or how sophisticated the display tech- nology. That’s why, in my opinion, the most interesting things are happening in the ever- expanding periphery as computers gain senses and are becoming inserted, component by com- ponent, into the world. This is the world of ubiq- uitous computing in which computers are everywhere and nowhere, and smart walls, sur- faces, and objects interact with us or do their work autonomously. When I completed graduate school in 1995, hype about our interactive future was running high, the Internet was beginning to take off, and there was a mad rush into the uncharted virtual world. Unlimited bandwidth and completely photorealistic graphics would soon make possi- ble a world much cleaner than the real one, where anything was possible. At the Interactive Telecommunications Pro- gram at New York University, we were taught to maintain a skeptical view of all this and to remain grounded in the real world with some- thing called physical computing—using comput- ers to augment real environments and facilitate human interactions. It’s essentially art with elec- tronics, and considering the tools available to artists today, Buckminster Fuller was right in his belief that as technology advances, art begins to resemble science.4 Digital media mirror how we understand the world, breaking it down into small parts that can be reordered. Audio, video, text, and images are all equally reduced to bits, and as an artistic medium, computers have mostly been used used to process other media. I began, however, exploring the computer as a medium in itself, emphasizing the computer’s original (scientific) purpose—computation. In my master’s thesis (see an example in Figure 1), I disregarded high-resolution graphics, instead using simple color and sound to try and create emergent, improvisational sorts of experiences, giving up some control over composition to the computer, and some to the user. It was inspired by Piet Mondrian,5 jazz, the pioneering work of Myron Kreuger,6 and Jaron Lanier’s idea that computers should be more like musical instru- ments (one of Lanier’s talks is transcribed at http://www.nilescanyon.net/nyc/jaron). I was fortunate to escape the Internet hype in 1996, at the American Museum of Natural History, where I was the senior software designer for Exhi- bitions until 2000. There, I was able to apply some of my ideas and experience with art to science. Relating art and science Understanding science is different than understanding art, but perhaps not as different as it would seem. Today, we can see deep into the universe and down to a molecular level. We can preserve images of people long dead, and we can control time using recording and playback tech- nologies. We can simulate the motions of galax- ies over billions of years. The world is awash in data, coming at us from every direction, and sci- entists, like poets, attempt to reduce chaos down to digestible bites.7 However, where the scientist sees only the observed in the material world, the artist sees the spiritual.2 While the scientist describes and pre- dicts, the artist distills and presents. Interpreting science for the public means balancing the desire of scientists to spew data with the artistic urge to create a transcendent experience. Content and presentation are everything: form and function must be fused. Leonardo’s Codex Leicester One of the first exhibitions I worked on at the museum was “Leonardo’s Codex Leicester: A Mas- terpiece of Science.” Bill Gates had just purchased the manuscript and the exhibition would launch its first US tour. Being in the gallery with the doc- ument was an eerie experience—each page in its own darkened, climate-controlled case, each case 5 Jan u ary–M arch 2 0 0 3 Figure 1. Screen shot from random abstract by Kevin Walker, 1996 (http://www. nilescanyon.net/ random). brightening slightly at staggered intervals. It was primarily an art exhibition and an odd way to explore the ideas in the codex. But Gates’ com- pany, Corbis, created some excellent magnifying- glass software, which not only translated from the 15th-century Italian, but also flipped Leonar- do’s mirror-image writing back into readable ori- entation. The exhibition had this software installed in some adjacent computer desks and Corbis included it in a CD-ROM released concur- rently in 1996 (called Leonardo da Vinci). My own task was to try and bring the docu- ment to life in a video theater. In trying to distill 72 pages into 8 minutes, we recreated some of Leonardo’s experiments, using new digital simu- lations from the planetarium, as well as real water (the manuscript’s primary subject). I also picked up the (digital) pen, to trace and animate his original drawings. I tried adding a third and fourth dimension to the manuscript by using it as a surface upon which additional layers of information float, illuminating the words and pictures on the page. It hardly needed it—the manuscript is quite animated already, filled with drawings showing motion and process, and da Vinci’s arguments with an imaginary adversary. At the heart of it, da Vinci shows that the birth of both science and art is observation. Exploring a World at Risk With current technology, our powers of observation are magnified many times, but sci- ence still contains a great deal of art. Not only is this evident in the mathematical beauty of equa- tions, but in visual presentation as well. For example, images from high-powered micro- scopes and telescopes are typically monochrome and are colorized for informational and aesthet- ic reasons. We took advantage of the use of false color in our 1999 exhibition, “Epidemic! Exploring a World at Risk” (see Figure 2). To help visitors make sense of the intricate complexities of infec- tious viruses, bacteria, and protozoa, we gave each organism its own identity, not only in dis- tinctive shape and color, but with sound as well. In a series of simple touchscreen kiosks, each organism carried its own abstract sound, and the sounds were designed to “infect” the space indi- vidually and work together harmonically, creat- ing a subtle ambient soundtrack to the exhibition instead of a cacophony. The simplified use of color and sound drew upon my previous experience with abstract art, combining it with scientific content. The infor- mation was rich, but was edited and simplified enough to facilitate understanding. What’s more, it transcended the screen and employed the space and visitors in the knowledge-creation process. To complete the sense of immersion, real models were designed to match the virtual mod- els of the microbes and to merge seamlessly with large projections of the organisms. Barrett Klein (an entomologist and model-maker) and James Stoop (trained as a 3D character animator) col- laboratively created the microbe models. Far from cartoon characters, the microbes are prob- ably the most scientifically accurate microbe models yet created. Merging the real and virtual in a “microbe forest” was just one attempt to integrate digital media into the gallery’s physical form. To accomplish the opposite—making the space the computer—I tapped Joseph Stein, a New York University colleague I had just hired to use physical computing techniques, and he used floor sensors to create an interactive shadow (see Figure 3), as a way of letting visitors peek into the body. The overall goal was to break down the barriers between information and architecture. Scaling the universe Exhibition designer Ralph Appelbaum is adept at merging information and architecture, and the following year, he used the giant sphere in the 6 IE E E M u lt iM e d ia Artful Media Figure 2. Part of the “microbe forest” in the exhibition—“Epidemic! Exploring a World at Risk” (http:// research.amnh.org/ exhibitions/epidemic/ microbes.html). Ph ot o b y Ke vi n W a lk er Rose Center (see Figure 4) to demonstrate the vast differences in size scales in the universe—postu- lating that if the sun was the giant sphere, the earth is shown as ball- sized; similarly with molecules and atoms. He also succeeds in trans- forming space into time, making places into timelines of billions of years, entire halls into ideas to be explored. As information spaces they approach the virtual world. This is a broader notion of architec- ture that encompasses the electron- ic as well as the physical.8 No amount of information or exhibitry could capture the cre- ation and evolution of the uni- verse, so to show this in the Rose Center’s Hall of the Universe, we turned to the skilled animators at Funny Garbage. They created a simplified, not-quite-cartoonish animation that’s scientifically accurate (at least as far as scientists currently understand it) and visually minimal such that a child can comprehend it. We projected this onto three satellite-dish-shaped screens (see Figure 5), with an interactive version in a touchscreen kiosk below each. Astrophysicist Frank Summers is another Leonardo-esque individual who brings an artistic background to science—in his case, dance. He choreographed what can only be called a dance of two galaxies interacting, in another projected visu- alization in the Hall of the Universe (Figure 6). 7 Jan u ary–M arch 2 0 0 3 Figure 3. Interactive shadow for the “Epidemic” exhibition (http:// research.amnh.org/users/jstein/ shadowman/shadowupdate.html). Ph ot o co ur te sy o f A m er ic a n M us eu m o f N a tu ra l H is to ry Figure 4. Size scales at the Rose Center for Earth and Space, American Museum of Natural History. Figure 5. Formation and Evolution of the Universe, an animation by Funny Garbage (http://www.funnygarbage.com/ client/case_study.php3?proj=17). © 2 0 0 2 A m er ic a n M u se u m o f N a tu ra l H is to ry Ph ot o co u rt es y of S te ve G od u n When we look up into the night sky, we see a flat, black mat, dotted with points of light—not unlike Leonardo’s 2D page, or, for that matter, a screen full of pixels. Standing on Earth, we have no sense of the infinite depth of space. For anoth- er Hall of the Universe visualization, Summers and I showed, for the first time, some recently discovered planets in other star systems—which aren’t visible unless you travel some distance into space. He generated some animated flights through a massive 3D star database, and I over- laid a layer of information, as I had with Leonar- do’s page, using just enough line and color to aid the viewer. The result (Figure 7) is that you see a slow pan across the sky, as you would see it from Earth. But then it stops and you leave the flat- land, not zooming in, but traveling past stars to reach the destination planets. Multiple viewpoints and ambient media For yet another visualization, Active Galactic Nuclei (see Figure 8), we traded traditional per- spective for something like the Cubist model used by Picasso—looking at an object from more than one viewpoint simultaneously. In our case, we showed galaxies for the first time in both vis- ible and radio frequencies simultaneously. I employed color to inform, as I had with microbes in the “Epidemic” exhibition, and in the resulting images, each galaxy appears radi- cally different in the two frequencies, with the radio image looking almost like a smudge of paint on a field of light. The images might be described as collaborative artworks, created in equal parts by human hands, imaging and com- puting technology, and the distant galaxies themselves. Lately, I’ve revisited astronomical data—this time for purely artistic reasons. I’ve been exper- imenting with abstract methods of depicting real-time data, using astronomical information (see Figures 9 and 10), atmospheric conditions, traffic patterns, or sound to create minimal, screen-based, or physical displays. It’s at an early stage, but I have found that minimalism is key, particularly when introducing an unfamiliar landscape. In an art exhibition, we’re prepared for confrontation with the incomprehensible, but we have come to associate the computer screen with the rational and “user friendly,” with help readily accessible. If we use a more abstract piece over time, however, we can learn a new language. This is sometimes called ambient media or 8 Artful Media Figure 6. Galaxy collision by Frank Summers (http://terpsichore.stsci.edu/ ~summers/viz/mhs/). Figure 7. Other Planetary Systems, a visualization by Frank Summers and Kevin Walker for Rose Center/Hall of the Universe. © 2 0 0 2 A m er ic a n M u se u m o f N a tu ra l H is to ry © 2 0 0 2 A m er ic a n M u se u m o f N a tu ra l H is to ry informative art, and Hiroshi Ishii’s group at the Massachusetts Institute of Technology is doing some interesting projects (see http:// tangible.media.mit.edu/), as is Tobias Skög of Sweden’s Play Interactive Institute (http:// www.viktoria.se/play/publications/2000/ ainfoart.html). These are akin to what Umberto Eco calls “visual icebergs”—minimal points of information on the surface, hinting at a wealth of data below.9 In installations such as these, the goal isn’t complete accuracy, because the outcome is often a behavior, not a number (for more on this con- cept, see Carver Mead’s work at http://www. pcmp.caltech.edu/). Besides, our unmediated representations of reality are always false anyway because our brains aren’t perfect; thus we’re always in a sort of virtual reality, as David Deutsch argues.10 In physics, we define informa- tion as a measure of order of a physical system, and work such as this is merely a way of describ- ing physical phenomenon using different phys- ical means. Linking technology to the real world I’m not interested in using technology to sub- stitute for the real world. Instead, I’d rather link it to the natural world, to mediate between real- time, real-world data and physical objects or 9 Figure 8. Active Galactic Nuclei, a composite image combining visible and radio wavelengths. © 2 0 0 2 A m er ic a n M u se u m o f N a tu ra l H is to ry Figure 10. Real-time visualization of the sun’s position, a prototype for installation by Kevin Walker. Figure 9. Music of the Spheres, a prototype for installation using real-time planetary data by Kevin Walker. spaces. (After all, computing—whether in biology or in silicon—is a physical process.) Most of these installations aren’t interactive, but autonomous— computers interacting with other computers, and with the physical world. Hardware prices and power requirements are falling sufficiently enough that using a computer, monitor, or projector for a single-purpose artwork is becoming more feasible. Even the humble, ancient technology of paper has more fidelity than the most advanced display hardware. In museums and on the screen, we’re used to seeing science and information visualized in pixel-accurate, simulated 3D. However, I think that deeper emotional power is held by a few piano notes, or in a tiny da Vinci drawing. That’s why the most interesting frontier of digital visu- alization is in the minimal and the physical. Whether on a planetarium dome or piece of paper, art can move raw scientific data to a new level of knowledge and experience. MM References 1 S. Wolfram, A New Kind of Science, Wolfram Media, 2002. 2. V. Kandinsky, Point and Line to Plane, Dover Publica- tions, 1979, p. 11. 3. G. Richter and H.-U. Obrist, The Daily Practice of Painting: Writings 1960–1993, MIT Press, 1995. 4. V. Vesna, “Bucky Beat: Buckminster Fuller, the Anticipatory Design Scientist, Foreshadows the Complex Persona of a Contemporary Media Artist,” ArtByte, vol. 1, no. 3, Aug./Sept. 1998. 5. P. Mondrian, The New Art—The New Life: The Col- lected Writings of Piet Mondrian, DaCapo Press, 1993. 6. M. Kreuger, Artificial Life II, Addison-Wesley, 1990. 7. I. Calvino, Six Memos for the Next Millennium, Vin- tage Books, 1993. 8. T. Horan, “The Paradox of Place,” Comm. ACM, vol. 44, no. 3, Mar. 2001, p. 60. 9. U. Eco, Travels in Hyperreality, Harcourt Brace Jovanovich, 1990, p. 199. 10. D. Deutsch, The Fabric of Reality, Pengiun USA, 1998. Readers may contact Kevin Walker at kevin@ nilescanyon.net. Contact Artful Media editor Dorée Duncan Seligmann at Avaya Labs, 666 Fifth Ave., 11th Floor, New York, NY 10103, email dorree@avaya.com. 10 IE E E M u lt iM e d ia Artful Media O cto b e r–D e ce m b e r 2 0 0 2 January/February Web Graphics With the popularity of the Internet, we’re seeing a migration of tradition- al applications to run on the Web environment and a growing demand for more powerful Web-based applications. Fused by the increasing availability and dramatic reduction in the cost of 3D graphics accelera- tors, a new direction of research, called Web Graphics, is emerging. This includes developing graphics applications as well as tools to support these applications in the Web environment. March/April Graphics Applications for Grid Computing Grid computing allows access to distributed computing resources with the same ease as electrical power. In recent years, graphics application tools that take advantage of distributed computing, or grid environ- ments, have emerged. New methodologies and techniques that harness resources for graphics applications are critical for the success of grid environments. May/June Advances in Computer Graphics This issue covers an array of advances in computer graphics such as organic textures, lighting, and approximation of surfaces. Also, you’ll find out about new developments in virtual reality, novel approaches in visualization, and innovative CG applications. The range of topics highlights the usefulness of computer graphics for everyone. July/August Nonphotorealistic Rendering Nonphotorealistic rendering (NPR) investigates alternatives that lever- age techniques developed over centuries by artists and illustrators to depict the world. One goal of this research is to broaden the achievable range of image styles and thereby embrace new applications. Additional- ly, NPR has the potential to open a new line of attack on one of the central problems of 3D computer graphics today: content creation. September/October Perceptual Multimodal Interfaces This issue focuses on recent advances in methods, techniques, applica- tions, and evaluations of multimodal interaction. Learn how researchers’ cross-disciplinary approaches helped develop multimodal interfaces from interaction-centered prototypes to user-oriented and application- tailored solutions. This issue also explores the notion of moving toward transparent user interfaces. November/December 3D Reconstruction and Visualization Models based on 3D data will ultimately include everything associated with the environment, such as trees, shrubs, lampposts, sidewalks, streets, and so on. The main mode of exploration for this massive collec- tion will be through interactive visualization. Ultimately, you should be able to fly continuously from overviews of a large city to centimeter-size details on the side of any building. Smoothly joining these different scales may require integrating rendering techniques in new ways. 2003Editorial Calendar http://computer.org/cga Index: CCC: 0-7803-5957-7/00/$10.00 © 2000 IEEE ccc: 0-7803-5957-7/00/$10.00 © 2000 IEEE cce: 0-7803-5957-7/00/$10.00 © 2000 IEEE index: INDEX: ind: