Modelling Complex Multimedia Relationships in the Humanities Computing Context: Are Dublin Core and FRBR up to the Task? J. Stephen Downie jdownie@uiuc.edu University of Illinois at Urbana-Champaign Allen Renear renear@uiuc.edu University of Illinois at Urbana-Champaign Adam Mathes adam@adammathes.com University of Illinois at Urbana-Champaign Karen Medina kmedina@alexia.lis.uiuc.edu University of Illinois at Urbana-Champaign David Dubin ddubin@uiuc.edu University of Illinois at Urbana-Champaign Jin Ha Lee jinlee1@uiuc.edu University of Illinois at Urbana-Champaign Introduction It is now widely recognized that the creation, management, and analysis of content other than text is extremely important if the digital humanities are to deliver access to, and provide an analytical purchase on, the full range of human culture. However it is not clear to us whether the cataloguing and classification systems for digital content are up to the task. Difficulties in this area threaten to impede both the development of tools and techniques — and the production of sound theoretical results. In our paper we discuss some of these problems, focusing on relationships amongst the various cultural modes of expression. With the intention of convening a larger discussion of how these confusions might be remedied, we then propose directions for some clarification and improvement. However, the larger issues here are not merely terminological and resist any easy resolution. The Problem Within the humanities computing community it has been a commonplace that while the emphasis on representing and analyzing textual content may be understandable, it is important to support the other kinds of content as well. We agree. The digital humanities must support the full range of human cultural products: text, music, images, dance, cinema, architecture, design, and so on. At present there are many different research communities looking into the organization of, and enhanced access to, these various modes of cultural expression. There is a text retrieval community (see Baez-Yates & Ribeiro-Neto), a growing music information retrieval community (see Futrelle & Downie), an image retrieval community (see Hsin-liang & Rasmussen), and so on. Notwithstanding the real progress being made by each of these, very astonishingly little work has yet been done to comprehensively address the issue that each of these individual modes of expression interact with each other in the ordinary course of production, management and use, as well as how formats at varying level of abstraction interact within a single modality. First, to illustrate how the modes of expression interact with each other, let us consider the Othello corpus. An incomplete inventory of the Othello corpus includes the novella by Giraldi Cinthio (1565) "upon which Shakespeare based his play" (Hunt), Shakespeare's play (1604), the operas by Rossini (1816) and Verdi (1887), Dvorak's concert overture, Op. 93 (1892), and the ballet by Lubovitch (2002). If we are going to create a digital humanities repository worthy of use by humanities scholars and their students, it is incumbent on us to build a system that can collocate, or gather up, all extant digital representations of Othello: all recordings, all scores, all movies, all choreographies, all libretti, all scripts, all set and costume designs, all critiques, and so on. To aid in this collocation, we need to clearly express the relationships between each of these things at both the specific and generic levels. On the specific level, we need to indicate that, for example, Othello choreographic labanotation W is directly based on Othello score X, which was specifically used in Othello movie Y, and also released in Othello soundtrack recording Z. On the generic level, we need to indicate that all Othello scores have some generic relationship to all Othello recordings, to all Othello movies, etc. in such a way that explicates that the works are all members of the Othello corpus. Second, to illustrate interactions between formats within a single mode, consider only the music mode of the Othello corpus. For each musical realization there usually exists a symbolic score and its individual parts. These symbolic representations can, in turn, be represented in a variety of digital formats: MusicXML, TIFF, Finale, etc. The aural aspect of the music is represented in another variety of digital formats: WAV, MP3, Ogg Vorbis, etc. Again, complex relationships exist between the symbolic and aural representations at both the specific (e.g., recording X used score Y) and generic levels (e.g., a fakebook score used to generate different recordings of improvised renditions). Other potentially complex relationships exist because many of these formats can be used to generate the others. For example, a TIFF scan of the original score can be fed through an Optical Music Recognition (OMR) system to create a MusicXML score file which can generate a MIDI file which then can generate any of the audio file formats. Further complicating matters, research is also underway to backwards create scores from audio recordings which would capture, symbolically, the nuances of a given performance (e.g., Plumbley et al.). Standards for Expressing Relationships Among and Within Modes There is, of course, a body of work — standards and related research — within the cataloguing and classification communities that holds some promise for supporting the relationships described above. The Dublin Core (DC) is perhaps the most widely used within the digital humanities. IFLA's Functional Requirements for Bibliographic Records (FRBR) is becoming increasingly important. Work by organizations devoted to specific modalities such as the Federation Internationale des Archives du Film (FIAF) [Note 1: ] , and the International Association of Sound and Audiovisual Archives (IASA) [Note 2: ] , as well as work by such researchers as Martha M. Yee (moving pictures — see Yee), and Richard Smiraglia (music — see Smiraglia), etc., are also contributing insights and theory to this research domain. Are We There Yet? We have reviewed results from projects and analyses that suggest there is still much work to do before the functionality envisaged above is a reality. Here we describe one such project that attempts to use FRBR and the DC to support inter- and intra-modal relationships. The DC does in fact hold the most promise for representing these relationships in a way that enables computer supported exploitation for retrieval, navigation, analysis, and so on. Ayres describes a project at MusicAustralia to use FRBR and DC to create a digital repository that explicates the complex relationships between the works, expressions, manifestations and items of a collection of music and lyrics found that: "The DC.Relation element can be used to display and support navigation between items with flat, horizontal relationships [i.e., inter-modal relationships like those between some music and its text]. However, the kinds of relationships MusicAustralia wants to expose are a combination of vertical [i.e., intra-modal relationships like those between a score and its recording] and horizontal relationships, and rely heavily on abstract but well understood and demonstrable concepts of the Work and the Expression or version. At this stage, DC does not offer support exposure of navigational pathways that explicitly acknowledge both vertical and horizontal relationships. [Bracketed injections are ours.]" Indeed, a close look at Dublin Core format and type elements suggests that the level of precision, and subtlety required is probably not yet available there. For instance the DC type vocabulary includes such disparate things as sound, text and physical object, and examples for sound include music playback file format and an audio compact disc (DCMI Usage Board). Next Steps: Exploring Ayres' Open Questions Because the work of Ayres and her colleagues represents the most thorough examination of the combination of FRBR modelling and Dublin Core encoding to build a comprehensive multimodal repository, we are taking it as the starting point for our present work. The Ayres study uncovers a series of unresolved open questions associated with FRBR and the modelling of real-world multimodal information. In the Ayres case, the two modes are music (i.e., scores, recordings, etc.) and text (i.e., lyrics, poems, etc.). These two modes come together to create what we commonly consider to be songs. To paraphrase Ayre's first open question: 1. Should we model as the primary work: 1. the music; 2. the text; or, 3. the combination of text and music? Ayres clearly illustrates that each modelling approach above clarifies a specific set of relationships between the music compositions and the texts while at the same time obscuring other relationships. The examination of this question has implications beyond the simpler music-text modelling case. For example, what are the implications when we attempt to model more complex cases (e.g., the Othello corpus, a Hollywood musical, etc.) with their exponentially growing relationships between text (novellas, plays, libretti, etc), music (i.e., notations, recordings, etc.), choreography (i.e., notations, video), and so on? Our paper examines this very question. We also explore the broader ramifications of Ayre's three related subsidiary open questions: 1. Should all notated and performed expressions of music [or dance, or text, etc.] be modelled as a single expression category? 2. Should expressions themselves be further modelled to include sub-categories for notated and performed expressions? 3. Should performed expressions based on particular notated expressions be modelled as expressions of expressions? By examining these fundamental questions, we intend to encourage a long-overdue conversation within the humanities computing community. Unless our representation schemes do justice to the multidimensional complexity of cultural content in all its modes of expression, we will not realize the full potential of digital humanities repositories. Bibliography Ayres, Marie-Louise MusicAustralia: Experiments with DC.Relation Presented at DC-ANZ (Dublin Core in Australia and New Zealand) Conference in Canberra February 2003 Baez-Yates, R. Ribeiro-Neto, B. Modern information retrieval1st ed. Addison-Wesley Reading, MA 1999 DCMI Usage Board DCMI Type Vocabulary 2004 Futrelle, Joe Downie, J. Stephen Interdisciplinary Research Issues in Music Information Retrieval: ISMIR 2000-2002 Journal of New Music Research 32.2 121-131 2003 Hsin-liang, Chen Rasmussen, Edie M. Intellectual access to images Library Trends 48.2 291-302 1999 Hunt, Mary Ellen Review of San Francisco Ballet, "Othello". War Memorial Opera House, San Francisco, CA criticaldance.com 2002 Functional Requirements for Bibliographic Records (FRBR) UBCIM Publications, 19 Plumbley, M.D. Abdallah, S.A. Bello, J.P. Davies, M.E. Monti, G. Sandler, M.B. Automatic Music Transcription and Audio Source Separation Cybernetics & Systems 33.6 603-627 2002 Smiraglia, Richard The Nature of "a work": implications for the organization of knowledge Scarecrow Press Lanham, MD 2001 Yee, Martha M. What is a Work? Weihs, Jean The Principles and Future of AACR: Proceedings of the International Conference on the Principles and Future Development of AACR, Toronto, Ontario, Canada, October 23-25, 1997 Canadian Library Association Ottawa 1998 62-104