Gamifying Sonic Interfaces: an Interactive Music Engine as a Music Production Tool
By Maria Kallionpää & Hans Peter Gasselseder
Augmented- and virtual reality environments (and instruments) are playing an increasingly important role in the classical music culture of today. Even the music genres leaning on a fixed performance tradition have been affected by them. For example, the art of contemporary opera has been influenced by composers´ and stage directors´ search for new modes of expression. The use of augmented reality technologies in a stage performance is part of this development. An illustrative example is Van der Aa´s opera “Sunken Garden”, in which the live action on stage is combined with a 3D projection. Moreover, human-computer interaction has become a vital part of composing: various composers design their own music systems. For example, Karlheinz Essl has created “Sequitur Generator” which he uses in a whole series of interactive compositions. Moreover, his “Lexicon Sonate” is an independent system that can generate music by itself almost infinitely. The purpose of this paper is to provide information on how the interactive music techniques usually associated with computer game music could benefit various music professionals, such as, for example, composers, performers and music producers. We will focus on techniques and technologies used in procedural music. Certain computer game scores and sound installations represent this genre, as well as electronic real-time-based compositions that may or may not require a human performer. In the context of interactive computer games, dynamic music systems directly react to the gamers´ actions. Automatisation challenges the form, rhythm, and harmony in a musical work. Instead of a closed entity, a dynamic music composition is a never-ending story with an infinite number of alternatives; it gets created again in every performance.
Maria begins by outlining the abstract, and states that the project can be applied to two types of object – a ‘fixed’ musical object (where the goal is for the player simply to play the piece accurately) and ‘process’ objects, where a higher degree of interactivity and creativity is required. She also gives us a (long) history of generative music, observing that Mozart and CPE Bach wrote music for dice.
We are given some examples of composers who design their own software and instruments, include Karlheinz Essl, and then there is a discussion of multimedia contexts, with an excerpt of Sunken Garden by Michel van der Aa.
Hans’s contribution, he says, contrasts with Maria’s composition perspective; he is a psychologist. His task is to look at the problems facing performers in realising a performance – the question of how do you bring a piece of music to life, and what state of mind is needed to do so effectively?
He hypothesises a ‘bad musician’ and asks how we would empower them to be flexible and reactive on stage. We need to find ways to represent ‘acting objects’ to enable musicians to react to the changing environment. He coins the term ‘artefact literacy’, and gives the example of the crackles in vinyl being acceptable to a listener who is familiar with vinyl. We see a an abstract diagram that focuses on the theoretical communication of meaning/agency from the performer/creator to the listener.
We now get to specifics of the experiments, where headphone mixes were altered in various ways to see how the performers would respond. They used a ‘Doppler system‘ in-ear microphone/headphone with its own DSP, meaning that performers could alter their own acoustic situations in real time. He suggests a real-world example, where a performer working on the early stages of a piece might prefer a drier acoustic, and as they get closer to the date of the concert, might add more reverb to simulate the hall.
Maria now summarises the next steps in the research, whereby methods of game research and arts research might be combined. She describes the technical realisation, explaining the need for acoustic and MIDI-controlled piano (Yamaha Disklavier, Bosendorfer CEUS).
Hans summarises the gaming aspect: it is called ‘Climb’ and the ‘player’ has to climb in a virtual environment (the game controlled by the piano), and can ‘stumble’ and ‘defend against bird attacks’ (!) while playing provided notation according to prompts from the game interface. We see a video of the game, with a musician playing along and the game interface responding.
[JB note: this is a really exciting project but I have not summarised it well here, and the video doesn’t seem to be publicly available. Recommend that interested researchers or musicians follow Maria’s and Hans’s research, as it emerges through publication].