Experiences of remote ‘virtual’ real-time performance #arp2016

Jamming in The 3rd Room: Experiences of remote ‘virtual’ real-time performance and recording.

Zack Moir, Paul Ferguson & Gareth Dylan Smith

inside-front-coverIncreasingly, many of our daily tasks are carried out ‘virtually’ via digital networks, including Skype-calls, video tutorials, and collaborative editing of documents via the ‘cloud’. While these tasks can be undertaken using normal domestic internet connections, issues of latency and poor internet connection make meaningful, real-time musical collaboration problematic and impractical to the point of impossible. However, using Gigabit connections onto National Research and Education Networks such as JANET and GEANT engineers are able to establish extremely high bandwidth and low latency links. This, coupled with LOLA (a low-latency, videoconferencing system) means that engineers and musicians are beginning to find ways to facilitate real-time live performances with remote performers, across long distances. While this has been achieved successfully in a number of cases, the process is still nascent and more research is required to understand the implications, functionality, and limits of such a workflow. This is particularly important, given that companies such as AVID are leading us towards cloud-based music production.
The authors, in their capacities as musicians (Moir and Smith) and sound-engineer/producer (Ferguson) are currently conducting research into the the experience of collaborating musically using LOLA. Our research investigates the impact of this means of working on the musical experiences of collaborators, in a variety of musical contexts. We are exploring the potential for live performance, audio/video realism, integration into future workflow for record production, and teaching/education applications. Additionally, we are interested in exploring the limits of this
system by way of understanding how it may be better deployed and developed for future use. This paper will report on a qualitative study in which the authors present accounts of their musical experiences of remote rehearsal (in Edinburgh, London, and mainland Europe), pre-production, and recording using LOLA, and will discuss implications for future use in remote, real-time, collaborative record production.

Zack begins with a short video of a jazz band jamming live over a LOLA connection. The session was live over 500 miles or so, between the Royal College of Music and Edinburgh Napier University. Paul Ferguson talks about their project (which, as shown above, uses a special low latency fast academic network), and cites a 15ms latency delay. This, he says, can be ‘felt’ by musicians playing together, but it’s enough to stay just about in time with each other. Paul speculates that network speeds that are currently only available to universities (and, I infer, governments) will one day be available to consumers. Given the history of the Internet this is a reasonable assumption.

Gareth Dylan-Smith, one of the researchers and the project’s drummer, gives us the all-important musician’s perspective on the latency experience. Gareth says that while the band were in time with each other, he felt that the groove was not symbiotic across the band, so he adapted to the latency by leading more, and setting his own groove for the musicians to play along to. The best results, he says, were for the final tune of the 3-hour session, which had the whole band playing from click track. Finally, Gareth reports a personal observation that he felt socially disconnected from the band the moment he removed his headphones.

Zack gives his own perspective as a sax player, and reported that the first tune (the standard ‘Equinox’) felt rather workmanlike as a blues, and the band didn’t feel inspired (we infer due to the 15ms latency). The other two (original) songs ‘different from here’ and ‘halfheartfelt’ were more successful, and in the case of the latter tune, the ensemble gelled much  better when a click track was present. Minor tech problems (like a broken headphone connection) undermined ensemble confidence early on, and camera/screen placement and eye contact made a psychological difference. More importantly, the physical layout of the space and cameras needed to take better account of the need to see visual cues. Overall, though, Zack describes the experience as ‘amazing’, especially considering that this was the first time this ensemble had ever played together online.

Technical network issue workaround: although there was minor packet loss (heard as audio dropout or clicks and pops) the session generated robust clean audio, by running separate local Pro Tools rigs at each end.

Paul now talks from the technical perspective, and notes that as the session progressed and headphone mixes improved, the band became more integrated and effective over time. He notes that he needed to juggle two roles – audio engineer and network engineer – although he speculates that over time these roles could come together more naturally, and it may in future be a reasonable expectation that a session engineer runs the network. By 1 hour into the session he suggests the band was generating good useable performances.

Visually, frame rate is an issue, with lower rates compromising musician communications and audio/video sync. This has bandwidth (and therefore latency) implications (they were using a 1GB connection). He notes that in high schools the connection would be slower and the problems greater.

The presentation ends with speculation about the future. Next, Zack says, they want to test the connection (more) internationally, for example with a cross-Atlantic or pan-European hookup. Paul’s next step is to figure out what can be achieved with the kind of (lower) bandwidth that is currently available to pro studios.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: