The Problem:
Representing and Archiving Live Performance
Manuscripts, paintings, sculptures, films, and recordings are artifacts that can be preserved and archived for subsequent generations to appreciate and analyze. Live theatre, however, is ephemeral. Is it possible to archive a live performance? One can use film or videotape to document a present day performance and, with some creative interpretation and speculation, recreate a performance from the past. But films and videotapes are incapable of conveying the experience of attending a live performance. A filmed performance offers only a single perspective on the action: the camera decides exactly where to look at each moment. Spectators at a live event, by contrast, act as their own camera operators, selecting their own point of focus Ð which may not even be on stage. Films omit a vital dimension of live performance: the viewer's immersion in the world of the theatre, and the crucial role that the community of spectators plays in constituting a performance event.
The underlying problem here extends beyond the theatrical performance. Precisely the same challenges arise with any kind of performative event, such as dance performances, rituals, political congresses, coronations, parades, festivals, battles, riots, etc.
One strategy to address this problem has been to build a physical reconstruction of an historic structure and to stage performances in it, as has been done, for example, with the Globe Theatre in London and with numerous structures in Colonial Williamsburg in Virginia. This solution requires an extraordinary, continuing investment of money and land, and so is feasible only in a very limited number of cases. Moreover, such physical reconstructions are available only to people at one geographic location and implement only one interpretation, and so cannot be used to evaluate conflicting scholarly interpretations of the historical evidence. Perhaps the deepest problem with such historical constructions is that, while painstaking efforts may be undertaken to achieve historical accuracy in the physical environment, performers and perhaps even the support personnel, the audience itself Ð and so, ultimately, the context of reception Ð remains resolutely contemporary.
Our Solution:
The Live Performance Simulation System
In January 2002, I began work as Principal Investigator on a project designed to address this problem: "A Live Performance Simulation System: Virtual Vaudeville." Our strategy is to recreate historical performances in a virtual reality environment. Virtual Vaudeville is, in effect, a single-user 3D computer game that allows users to enter a virtual theatre to watch a simulated performance. The objective is to reproduce a feeling of "liveness" in this environment: the sensation of being surrounded by human activity onstage, in the audience and backstage, and the ability to choose where to look at any given time (onstage or off), and to move within the environment. A vital concern is to find a way to bring the nuances of great stage performances into this virtual environment. To this end, we are using optical motion and facial capture technology to capture real-world performances by professional, highly skilled actors, singers, dancers, acrobats and musicians.
This three-year project is supported by a $900,000 grant from the National Science Foundation, supplemented by an additional $110,000 from the State of Georgia. I am leading a team of researchers from seven universities, including University of Georgia, University of Pittsburgh, Georgia Tech and the Naval Postgraduate School in Monterey, that includes historians specializing in 19th century American theatre, music and culture, computer scientists specializing in high-performance 3D game design, and theatre practitioners.
Our long-term goal is to develop a flexible set of techniques and technologies that scholars and theatre practitioners can use to simulate a wide range of performance traditions, from classical Greece to Japanese Noh. Our short-term objective is to complete a fully-functional simulation of 19th century American vaudeville theatre.
Vaudeville
American vaudeville is an especially apt test case for Live Performance Simulation. Vaudeville was the most popular form of entertainment in the United States from the 1880s through the 1920s, functioning in its day much as television does today. Many vaudeville acts both reflected and helped to constitute the enthusiasms and anxieties of their time, especially those concerning the integration of new immigrant groups into mainstream American culture. Consequently, a rich simulation of a vaudeville performance will be a useful resource, not just for those interested in theatre history, but for scholars and students of American history generally.
A vaudeville performance was divided into many short, self-contained segments. A typical vaudeville bill encompassed a wide variety of acts Ð contortionist performances, dance numbers, juggling acts, singing groups, comic monologues, blackface comedy, condensed versions of full-length plays Ð with particular acts in the lineup appealing differently to different groups in the audience. As a result, simulating different acts of a vaudeville show and exploring the likely responses of different groups of spectators opens up for historical investigation a wide range of ethnic, gender, class, and racialized interactions during America's industrial age.
Our simulated performance takes place in B. F. KeithÕs Union Square Theatre, a typical Vaudeville house seating approximately 1200 spectators, in the year 1895, fifteen years after the first Vaudeville theatre opened in New York. We are recreating four of the most popular and representative acts on the vaudeville circuit during that time: (1) the strongman Sandow the Magnificent; (2) the Irish singer, Maggie Cline; (3) the ethnic comic, Frank Bush, most famous for his "Stage Jew" character, which sets the pattern for Jewish stereotypes in Vaudeville and popular entertainment well into the 20th century; and (4) the sketch comedy of the four Cohans, whose youngest member, George M. Cohan, went on to become one of the great stars of early 20th century Broadway.
As we approach the midpoint of the project, we have completed archival research into all of these acts, and are creating the models and motion-capturing the performances. Our presentation featured animated video clips showing parts of the Sandow and Frank Bush act, along with a section of the Frank Bush act in the real-time game engine.
Design
Virtual Vaudeville allows the user to switch between two very different ways of experiencing the simulated performances. In what we call "invisible camera" mode, viewers fly through the 3D space to observe the performance from any position in the theatre and zoom in as close to the performers as they please. In this mode, the viewer can scrutinize such details as a performer's shoes, the carvings on the theatre walls or the upholstery on the chairs.
Most importantly, the viewer can study the audience's response to the performances. One of the greatest challenges of the project has been to find a way to represent the audience within the technical constraints of a real-time environment. The theatre, according to our estimates, would have held approximately 800 spectators during a typical afternoon performance. Our goal has been to reflect the demographics of the audience as precisely as possible, showing the distribution of gender, class and ethnicity in various parts of the theatre. Not only do we need to fill the seats with historically accurate and convincing faces and costumes, but we need to animate all the spectators so they respond to every moment of every act in a way consistent with their demographic profile. For example, when Frank Bush portrays his Irish character, the Irish spectators in the gallery Ð a notoriously boisterous group Ð should respond very differently from the WASP characters in the boxes.
Similarly, when Florenz Ziegfeld comes onstage to announce that women may pay to come backstage after the performance for a private reception with Sandow, the women should respond differently from the men. However, to create 800 distinct models, with 800 distinct animations, would not only be an awesomely onerous task, but would be impossible for the program to handle in real-time. Our solution has been to define 32 basic audience groups, for example, one group of upper class WASP men, and another of middle class African-American women, and we have scripted and animated responses for each of these groups. We then created from three to five physical variations for each group (i.e., three to five different faces and costumes), and produced even more variation by adding and subtracting hats and facial hair.
As an alternative to the quasi-objective perspective of invisible-camera mode, the viewer can adopt an embodied perspective, watching the performance through the eyes of a particular member of the audience. Virtual Vaudeville allows the viewer to select one of four spectators, each representing a different socioeconomic group in 19th century America: (1) Mrs. Dorothy Shopper, a wealthy socialite attending the performance with her young daughter; (2) Mr. Luigi Calzilaio, an Italian immigrant fresh off the boat, attending the performance with his more Americanized brother; (3) Mr. Jake Spender, a young "sport" sitting next to a Chorus Girl (he may or may not strike up a relationship, depending on the viewer's choices); and (4) Miss Lucy Teacher, an African-American schoolteacher watching the performance with her boyfriend from the second balcony, where she is confined by the theatre's segregation policy.
Viewers can switch between any of the avatars at any time, and can move the avatar's head to focus on different areas of the stage or auditorium, and can trigger a limited set of avatar responses, for example, applauding or laughing. Some of these responses are verbal, such as cracking a joke or heckling the performance. In these cases, the viewer selects only the generic response type, and the system produces a specific response appropriate to what is happening onstage and off, taking into account the viewer's previous interactions with other members of the virtual audience. Because the surrounding spectators respond interactively to the viewer's avatar, each viewer has a different experience of the performance event.
Significance
Virtual Vaudeville offers scholars in all disciplines of the humanities a model for a new kind of "critical edition." A conventional published monograph can pick and choose details to examine, and so lacuna and even contradictions in the historical analysis are easy to overlook. The imperative of precisely recreating both onstage and offstage events will demand an unprecedented degree of scholarly thoroughness and rigor.
Key to our project is the depth of the collaboration between technology, scholarship, pedagogy and art. This project is conceived to make a significant contribution to all four domains simultaneously, rather than merely using any one in the service of the others. The end result, we hope, will represent an important advance in the design and implementation of virtual environments, building on recent successes in creating photo-realistic simulations of real 3D environments by introducing a large quantity of complex human performance data. It will constitute an invaluable work of applied scholarship, an unprecedented resource for visualizing past performances and testing hypotheses about historical performance practices. It will provide an unprecedented resource for students to engage with historical performance traditions as performance (and not as literature or film). Finally, from an artistic perspective, the Virtual Vaudeville project will test the potential of virtual reality technology to provide truly nuanced and engaging theatre experiences.