The Media Library for Dance and Theatre (MTT) is a centre for artistic and scientific research and discussion for those involved in the performing arts. It is one of the most extensive publicly accessible audiovisual documentation centres for the performing arts in Germany. A variety of video materials cover documentation of performances within various disciplines of performing arts, genres such as dance films and documentaries, as well as documentation of discursive formats like conferences, lecture performances and symposia. This diversity enabled a broad range of use cases of the annotation tool, as it opened possibilities to test various combinations of approaches. MTT connects with an audience consisting of researchers and artists, many of whom seek to go beyond simply viewing the available materials. One of the first impulses while viewing the videos is to take notes and add in information that connects observations to specific points either in spatial or temporal sense.
Within the cooperation ITI ♥ MOTION BANK, the video annotation tool developed by Motion Bank was tested in a frame of how MTT visitors could use it for different types of research as well as from the perspective of working in the archive to facilitate and support the tool’s usage. The working process was documented through writing and screenshots, while the videos were selected to simulate the various possible applications, ranging from dance film to movement- or discourse-oriented material. Each of these formats and approaches offered different annotation processes and could serve distinct purposes according to the working needs. The try-outs are documented and published within this evaluation with an aim both to familiarise archive users with the possibilities of a video annotation tool and to serve as an example for the implementation of similar workflows in other institutions and collections. All examples are taken out of the stocks of the Media Library and were carefully chosen to make use of different options of the tool of Motion Bank.
1. chapter segmentation and indexing
Der Gehängte im Garten der Venus is a dance film by Lutz Gregor, Claudia Feest and Dieter Heitkamp, which premiered on 12.12.1988 in Akademie der Künste in Berlin. Being a dance film, it was chosen because of the specificity of the medium – to annotate the movement, but also the filmic frame in which it unfolds.
Chapter segmentation was one of the first things that was tried out. It came down to marking the scenes with a title, which comes with a timestamp and a mark on the timeline. This proved to be useful when it came to locating specific moments within the video and getting familiar with it. The bookmarking feature was used for this action, as it is separately shown on the timeline and differently in the annotation list.
Marking transitions, subchapters and editing techniques lead to an observation - when annotating with a specific task in mind and while rewatching the video, more and more details come to attention, which consequently deepens the understanding of the video at hand and prompts more annotation tasks. All these annotations were differentiated visually in the list and different layers of information were sorted using comments and a highlighting feature. Annotations highlighting a specific moment in time compared to those annotating a timespan are visually distinct in the annotation tool’s timeline representation.
First outcomes: The Annotation list can give an overview of important moments and make visible the structure of the video. Coming back to specific points in the video is much easier, while naming and describing the contents of the video opens up the pathway to analysis and interpretation.
2. annotating content, cues, choosing details to focus on, qualities
The second video that was annotated was Turning Solo, the portrait for Naïma Ferré, in which whirling practice is brought into dialogue with Isabelle Schad’s ongoing movement research. The video was chosen as it is a solo work with only one body to observe and capture, and because of its distinctive trait - a slowly but constantly transformational movement quality – which offered itself to annotation.
The process of annotation with a determined research question added to the quality of observing – the emergence and changes of movement patterns and elements that interact with the details of dance (like props, light and costume) also became important. To capture these complex relations that might be perceived differently by different people, a personal terminology for capturing the choreography emerged and developed during annotation.
The Machine Learning-based feature for pose recognition automatically recognizes basic segments of the human body and draws a coarse skeleton over the body. It’s then possible to select body parts or single joints and assign annotations to them, which proved useful for capturing targeted details within the choreography.
First outcomes: Dance and physical practices contain a lot of information which might not be apparent at first view. Deciding on a central research question while annotating can lead to a more focused annotation process. From there, each person can find their own way of deciding which moments are worth capturing and how to connect the annotations. The tool also revealed a knowledge repository potential, where people with diverse information specifically related to the videos can share it with other users, thus enriching the archive.
3. voice annotation, categorising and filtering annotations
Turning Solo 2 is a continuation of the work Turning Solo, with two dancers interpreting the same choreographic material. This video was annotated experimentally during the first watch, using a voice recording feature throughout the whole duration of the piece, which was then transcribed, processed and converted into an annotation.
The approach to annotation shifts depending on the medium – writing text calls for a more structured analysis, while audio recording allows more spontaneous observations. The recorded audio can be saved as annotation and played over the video, but the transcript of it can be best seen as a thought mediation tool between speech and writing.
After editing the notes, a need to categorize them came up, and a “hashtag system” was applied, where custom categories were created and marked with a symbol # for easier searchability. It can be helpful in locating the large number of annotations and categories in the list. Filtering of the annotations is possible by keywords, annotation type (highlights/comments), media type (image/video/text/audio) or analysis type (people recognition/speech to text).
Furthermore, for the purpose of highlighting a single person in group or ensemble pieces, the Machine Learning-based feature for people detection automatically identifies bodies in the video image and draws boxes around them. It’s then possible to select these boxes and assign annotations to a specific box/person.
First outcomes: The hashtag categorization system is just one of many possibilities for structure and recall. There is a lot of creative room to apply such methods according to the annotation objectives. Audio recordings turned out to be useful to gather initial thoughts, and then develop further reflective annotation stemming from there, may it be textual or not. A possible future archival use could be that choreographers (or any other artist involved) watch their own videos and record audio reflections or discuss the concepts they worked with and share insights into their creative process.
4. referencing external sources, subtitles
The material for this use case was a recording of an event that was part of the series Zwischenrufe/Interventions, which is centred around dialogues on stage between the body and language, where artists Yvonne Rainer and Xavier Le Roy met to discuss and exchange current concepts and projects. This video was chosen to try out how annotation can be of use for more discursive formats, with an aim to link references outside the video recording itself and build connections between them.
The process began with transcribing the audio by using the automated speech-to-text feature, which generated subtitles and enabled automatically transcribed annotations of interesting quotes.
The annotation list was used for referencing people, places and objects at the times that were mentioned within the recording, as well as for marking key points of conversation. At the time there was no singular database which could provide for the references, therefore links to information that is easily searchable on the internet were added.
First outcomes: An annotation list can be used as a resource for providing further context and other relevant references about the artistic work or points of a discussion. By inviting a person who was somehow involved in the video at hand, annotations can be used to add background information that is not evident from the video alone. When it comes to information sources and the extensiveness of the references, the implementation of Motion Bank Systems would include making sources from the Media Library for Dance and Theatre catalogue easy to find directly from within the tool. A question emerged when it comes to annotating information that is not bound to a specific time within the video but to the video content in general, and it’s something Motion Bank is currently working on.
Implementation of Motion Bank Systems in an archive
When imagining how Motion Bank Systems could be implemented in the archive, the discussions always ended up in the direction of activating the archive. The annotation list together with the archival material could become interfaces where people can leave their own reflections and then interact with each other's thoughts as well. Additionally, the annotation tool within an archive context can serve for linking different materials, documents, places and people within a video, creating a cartography and structuring a network of information.
In addition to personal use, sharing annotations could introduce a new dimension of interaction which could be between collaborators on a project (which is already possible) or future visitors (which is planned to be possible) as well as between archive users and the archive itself. Planning for annotations aimed at an unknown future audience could be an intriguing aspect to consider. Annotations themselves could become part of the archive, adding layers of complexity and capturing a spectrum of reactions where different viewers can contribute with varying perspectives (e.g. one might focus on technical movement analysis, while another might reflect personal memories or interpretations from seeing the performance live). Annotations capturing multiple views for the same video could offer contrasting views and reveal how people remember or interpret the same moment differently over time.
The whole idea of the archive is shifting as well, it's not just about documenting chronology and factual information. It also delves into the idea of post-history, questions what the truth really is and consequently creates a multi-layered concept of archive. Instead of just focusing on who the director and choreographer were, when the work was made, and which production house was involved – it becomes a question of personal and interpersonal knowledge. The idea of putting assumptions into databases exists and is something archivists are aware of, knowing that there's no linear history and that there's a need for multi-personal and non-hierarchical knowledge which should be followed. The annotation tool suggests itself to be a fitting tool for this process.
Additionally, the use of the annotation tool from an artistic perspective could help blur the often-rigid boundaries of scientific and scholarly vocabulary by finding and coming up with own descriptions and words. A more contextual, personal or nuanced understanding of documentation and reflection about dance might arise from that.
As many of the videos from the MTT collection cannot be viewed outside the archive itself due to copyright regulations, the subject of “taking home” what users do in the archive was also addressed during the discussions, and how annotations can serve a purpose in that case. Exporting annotations Is currently possible in formats like JSON and spreadsheets; the first being directed towards exporting data in a coding-friendly format, while the latter is a more accessible format that can also be printed out and preserved. The key idea was to use non-proprietary and standard formats which ensure that the annotations are readable in the future, even though the original software or digital tools are no longer available.
In summary
ITI ♥ MOTION BANK is a unique collaboration with the Motion Bank Systems being applied to a public archive of the performing arts. It’s an ongoing process full of trials and experimentation. It is crucial to note that this is a pilot project with no fixed product as a result and the implementation of the tool at the MTT to make it available for the users is still in progress and aims to be finished in early 2025.
If you are interested in trying out the current version of Motion Bank Systems yourself, you can directly contact the Motion Bank team at office@motionbank.org. For more information about the Media Library for Dance and Theatre or about this project, feel free to contact c.henniger@iti-germany.de and come visit next year to try out the annotation tool.
Powered by Froala Editor