Preservation and representation of digital art presents a significant challenge for curators, archivists and artists. The most notable problem is the fundamentally interactive nature of this evolving art form. Because there are no clearly defined frameworks for the authentic representation of the variations resulting from interactions, curators and archivists are currently deciding on a case by case basis which interactions should be preserved, who should be able to interact with a given piece, and how to represent those interactions in a meaningful and objective way. Not only is this method problematic from a representational standpoint, it's very time-consuming. My thesis will attempt to resolve some of these complexities by making connections between this new art form and an established one, namely music; assessing the representational characteristics of music's notation system and exploring the methods by which musicians and conductors handle musical interaction.
One of the most significant challenges for the new media art community is development of a representation framework for preservation purposes. There are currently three preservation models under consideration. The first two have technical origins, and should be familiar to the general digital preservation community: migration, the premeditated upgrade of file formats; and emulation, which focuses on development of ur-operating systems able to run obsolete media. The third option, much more radical, and developed by and for the new media art community, is re-interpretation (Depocas, Ippolito & Jones); a method intimately related to the presentation, exhibition, and performance of an interactive new media art object.
While re-interpretation is essential to success in the performing arts, in the fine arts and electronic preservation communities, the idea of re-interpretation as a valid preservation strategy challenges beliefs that are at the heart of these fields. Specifically, for the purpose of archival preservation, the characterization of a 'reliable' or 'authentic' object will need to undergo a significant transformation if these new media art objects; highly variable, interactive, and data-rich cultural artifacts, stand a chance of meaningful survival. Although radical, curators, archivists and conservators could regard the idea of re-interpretation as a way to honestly address the realities of presenting a variable work over time. Both migration and emulation effect subtle re-interpretations of a work, whether we perceive those changes or not. For example, the processor speed of an emulated computer will display an interactive work from the mid-1970s differently than the original operating system would. Whether that difference fundamentally changes the work's meaning can only be determined if a representational framework exists in which the creator/cataloger has the ability to address the realities of this medium's artistic creation and representation process: specifically, the highly interactive nature of the work, the variable quality of output, and technically complex interactions within the work itself must be addressed. By embracing the paradigm that includes re-interpretation as a valid preservation strategy, digital preservationists will need a thorough and consistent representation of the original from which to work.
In mid-2004 Richard Rinehart delivered a paper arguing that new media art is more like a musical performance than it is like a painting or a book, and therefore more appropriately represented by a scoring system than the text-centric methods used today (Rinehart). His proposal of a media art notation system (MANS), based on the MPEG-21 framework, is a welcome step forward in the development of a viable preservation schema for these highly variable and ephemeral objects. Rinehart's system, however, is more of a metadata framework or ontology than a scoring system, and therefore runs into the same problems inherent in any text-based representational framework that attempts to describe or define a non-textual entity (Svenonius), and is based on the ‘conduit-metaphor' model of communication (Davis).
There are numerous examples of existing art forms that employ re-interpretation as the de facto means of delivery and representation: namely; drama, for which scripts represent the work; dance, which often makes use of a notational system to describe and record body movements across space and time (Labanotation being the primary example); and music, which is usually represented in the West using the common notation system (CNS). In order to develop a less text-centric representational model for new media art, I chose to explore the representation and interaction techniques of these art forms, which have interpretable or variable output.
Music, drama and dance each share characteristics with new media art. They're temporal: meaning they take place and change over time; they're performance based and ephemeral: the performance is the instantiation of the work, and once that performance is over, it's gone; and they're open to interpretation within some pre-determined set of values: although it is possible and expected to interpret freely, each form has a framework within which the director /choreographer/conductor must work. However, music has traits uniquely shared with new media art. For example, music and new media art both share a 'unity of artistic vision', basically meaning that all instruments, in the case of an orchestra; and libraries, programs, data sources, in the case of a new media art object, interact in a specific way to produce the final product. Another shared trait between music and new media art is the existence of a level of abstraction that is not necessarily present in drama or dance. Both the composer and the programmer use instruments, or tools, to achieve his or her artistic vision, whereas a playwright or choreographer works with the more immediately available bodies in motion and/or words as their means of expression. Finally, a play or dance does not inherently depend on the availability of tools to exist, whereas an orchestral / new media art performance most often does.
In the hopes of developing a set of characteristics to include in a new media art notation system, I have started exploring the information contained in musical scores. In addition to recording the fixed musical elements like pitch, rhythm, tempo, dynamics, and articulation; this research also seeks to understand musicians' interpretative decisions, as well as their interactions with each other, and with the score. Interpretation and interaction are particularly interesting for the purpose of this research, because these are the primary means by which musicians achieve artistic, reliable performances, and that is the ultimate goal of any new media representation framework. Although it might be difficult to completely understand musicians' interpretative choices and interactions, we believe that the personal notes (annotations) musicians' make on the scores themselves can provide valuable information regarding these transient and personal decisions (Marshall).
Some of the questions this phase of the research seeks to answer: Which musical elements must be regulated, which can be improvised, and which must be freely interpreted? Is this dependant on context of presentation? Can any of these musical elements transfer usefully to representation of new media art? Under what circumstances does a musician decide to do something that is not in the published score? How do composers communicate over time, space, and cultures, their intent and goals regarding performance? Are there different models of interpretation based on different musical styles or genres? How do composers, conductors, and musicians react to the idea of official representation of interpretation?
With these issues in mind, we developed an experimental framework consisting of a musician/score collection methodology; a coding schema, which will help categorize the annotations; and a method for the systematic exploration of annotations.
The musician/score hierarchy defines from whom and what kinds of musical scores this study will consider. It is a structural arrangement of four parallel interests and skills of each musician: first is the three 'levels' of music-makers: musician, conductor, and composer; second is the level of proficiency: amateur, college-level, and professional; third, we take into consideration the presence of a conductor: orchestras versus quartets, for example; and finally, the hierarchy includes a consideration of style of music – jazz, classical, and musical theater.
After collecting the scores, we are marking up any annotations the musicians/composers/conductors might have made on them. There are two types of mark-up: structural, and content-based. At the structural level, we decided to mark up at the bar level, delineating the extent of bars and phrases, where appropriate. At the content level, there are three types of written notes: textual, where the musician has actually written a word in the margins ("Less Bow!!!" or "FROG!"); symbolic, where the musician has written non-textual symbols (stars, exclamation points, and glasses); and numeric, where the musician has put numbers above or below notes for fingering or timing instructions, or numbered the bars if that information isn't included in the published score.
At this point in the process we conduct interviews with the participating musicians, asking questions regarding the process, and context of annotation behavior (MacMullen). The purpose of the interview is to get a deeper understanding of musicians' attitudes toward interpretations and whether their annotation behavior is in fact an important element in understanding that interactive quality of musical performance.
The final step in this process is to analyze the annotated scores, looking for n-way consensus, investigating any 'important' or consistently documented sections or elements within a piece. I will normalize at the basic unit of annotation (in this case at the bar and phrase level); I'll record all instances of annotation: who, where, what kind; and all count all instances of annotation to provide percentages at the bar and phrase level between and among musicians and musical types. Finally, I'll conduct consensus analysis to determine how often annotations concur on selections.
After concluding a pilot study, exploring the initial data from a university-level orchestra, and interviewing several musicians and conductors, initial findings indicate that the premises upon which this study are based are valid. Annotation of the score provides insight into a number of different issues relevant to the development of a notation schema for new media art: annotation analysis identifies those characteristics of musical notation important across musician types and skill level, allowing me to make recommendations for the inclusion of certain characteristics in the new notation system. Annotation also evidences performance-related interaction between and among musicians. Finally, annotation is a physical sign of an individual musician's interaction with, and interpretation of, the score itself.
This work is ongoing. Data collection will end in April 2005, when more comprehensive findings will be available.
This work was partially funded by an unrestricted research gift from
Microsoft Research to the
Annotation of Structured Data research team in the School of Information and Library Science at the University of North Carolina at Chapel Hill, whose members contributed to this work: Gary Marchionini, Paul Solomon, and Catherine Blake, co-PIs; with team members Tom Ciszek, Xin Fu, Lili Luo, W. John MacMullen, Cathy Marshall, Mary Ruvane, and David West. The project's website is available at:
http://ils.unc.edu/annotation/.