The virtual orchestra: pioneering immersive entertainment in classical music
“Creative artists for a long time have been dreaming about creating a world where they would create the laws, the physics, the grammar and the syntax. Now for the first time in human history, we have the technology available to do it.”
— Esa-Pekka Salonen, Principal Conductor and Artistic Advisor Philharmonia Orchestra
The Virtual Orchestra is a pioneering work of immersive entertainment in the world of orchestral performance, offering classical music lovers a world-first – the chance to ‘step inside’ the Philharmonia Orchestra. It ran as a 10-day festival at the Southbank Centre in 2016 and included the premiere of a ground-breaking VR film with the same name. The film was a co-creation between the internationally-renowned Philharmonia with its technology-focused Principal Conductor Esa-Pekka Salonen, Southbank Centre and London-based VR specialists Inition.
Audiences were treated to 360° 3D video and audio performances through VR headsets in the foyer of the centre’s Royal Festival Hall and the experience is now available on PlayStation®VR. Alongside the VR experience, the Festival Hall’s Clore Ballroom was transformed into a 10-room immersive exhibition Universe of Sound, using speakers and up to 30 projection screens to allow audiences to walk through the orchestra.
Founded in 1945 by EMI producer Walter Legge, the Philharmonia is the most recorded orchestra in the world and was conceived to capitalise on the then cutting-edge technology of recorded music. Now, as the orchestral leaders in VR technology, The Virtual Orchestra has been showcased around the world in partnership with the British Council, GREAT and DIT. Its international showcases include the WIRED conference in Tokyo, as well as events at California’s Berkeley University and the Rio Olympics, where it played at British House. This project is the VR successor to the Philharmonia’s earlier large scale immersive installation projects, RE-RITE and Universe of Sound, which toured for almost 10 years across the UK and to Germany, Portugal and China.
Using giant visual displays, touch screens, projecting surfaces, movement-based interaction and planetarium-style projections, Universe of Sound allowed its audience take part as musicians, conductors, arrangers and composers in a performance of Holst’s The Planets. It opened at the Science Museum in London in 2012, and won the 2013 Royal Philharmonic Society Award for Audiences and Engagement. RE-RITE was a digital installation based on The Rite of Spring, playing 27 HD video and 18 audio feeds in frame-accurate synchronisation with custom-built playback system using multiple screens and projections. And, with the orchestra’s iPad app The Orchestra launching in 2012, the organisation has firmly positioned itself as a global leader in the sector for ground-breaking digital work.
Gearing up for VR
Luke Ritchie, Head of Digital Innovation and Partnerships at the Philharmonia, explains that there were three main reasons for the organisation’s move to embrace new immersive technologies; to build audiences, generate alternative revenue streams for the orchestra and create new experiences that were committed to making new and engaging content in different ways, for both educational and artistic purposes.
“Creatively, VR felt like a natural progression for us, and I believe it’s a fantastic medium for orchestral music. 2D video and stereo audio can struggle to convey the complex dynamics and visceral energy of a live symphony orchestra.”
With one of his primary objectives to build business cases for new immersive projects, across physical installations as well as VR and AR experiences, his remit includes securing commercial investment to expand the orchestra’s portfolio of digital work for international audiences.
Preparing to take on the challenge
The organisation is unusual in the sector in having a dedicated in-house digital department, reporting directly to the MD, with the active involvement of the Principal Conductor, a creative visionary when it comes to using digital technologies in pioneering ways.
“At conferences, I’m always asked by people looking to move into the digital sphere, ‘where do I start?’” says Luke. “I believe it’s important to have in-house experience and that’s something that’s often overlooked. You don’t have to start with a multi-million pound project, just start by employing one person.”
The Philharmonia also uses a network of freelance experts, which is necessary to support projects when working with a small in-house team, says Luke. “It’s important to find the right people, create relationships and keep them going – none of it is easy. Keep it small to start with, maybe buy a VR camera – there’s no need to have lots of 360° film experience.”
Developing partnerships
He also stresses the importance of building partnerships within the wider industry with other organisations or tech firms. The Virtual Orchestra was conceived of as a partnership with the Southbank Centre and Inition, which reduced cost and risk for the orchestra. Inition had one of the few cameras that could film live action in full 360° in stereoscopic video at the time, Luke explains, and both teams were particularly interested in developing 3D audio.
The project turned out to be much more however than creating a live orchestral performance in 3D. Peter Collis is the head of 360/3D Film Production at Inition, who filmed the production. He explains that in addition to capturing the power of being immersed in the music, they also wanted to engineer a back story and make it more of a narrative, documentary-style piece.
The final result documents the day of performance at the Southbank, the trains going over Hungerford Bridge as the audience take their seats, the atmosphere building as the musicians arrive backstage and the anticipation rising as conductor Esa-Pekka Salonen stands in the wings before stepping out to conduct Sibelius’ Fifth.
Re-creating an orchestra: building the technology
“At Inition there is huge focus on and interest in 360° film-making and 3D within 360°,” says Peter. “When the Philharmonia approached us, we felt this was a technical challenge worth pursuing – to try to create a 360° film in 3D that went beyond anything that was being developed at that time.”
Starting with a Go-Pro off-the-shelf camera which was stripped down and rebuilt, the team added a bespoke lens system and created a custom 360° 3D camera rig which was incorporated into a digitally-printed rig designed in-house. That was a big step in getting to the next level, reveals Peter. The end production combined audio-visual solutions with stereoscopic and CGI skills, as well as high levels of dexterity. Placing the viewer in the centre of the orchestra to give the unique point of view only normally experienced by professional musicians, entailed filming on two cameras, one in the middle of the stage, one hanging right over the balcony. Which created its own challenges.
But filming an orchestra in 3D was always going to be challenging. “One of the ultimate difficulties in 360° 3D filming is the close proximity to the objects being filmed,” explains Peter. “It’s tricky to film straight lines that move in complex ways in this space for example. And we had to film violin bows.”
Other challenges came from the equipment itself, including accommodating the lateral motion detection system on Sony PSVR headsets, which had to be disabled by building a bespoke app. Inition was an early adopter of the 3DIO microphone and the experience is captured in 3D spatial audio using an array of microphones set in silicone moulded ears. The challenge was to capture sounds from specific places that rose and faded as the viewer’s head turned, feeling the audio move around them. The technology allowed them to focus on a particular player or section and even hear details such as the score pages turning. Altogether, eight microphones made up the binaural recording system and a bespoke app was developed to manipulate sounds by head position.
“Tools are now available to manage spatial sounds and in Facebook 360 you can plot sounds in space,” says Peter. “Back then we were pioneering this technology, we had to build it from scratch.”
And while Luke believes that VR is still viewed with a lot of scepticism in the industry, people who have seen the work of the Philharmonia are beginning to see the potential of it. “This isn’t 3D TV. It isn’t a fad. We’re just at the very beginning of this technology.”
Results and feedback
The Orchestra’s full range of immersive installations have reached 350,000 people over the last decade, and generated commercial income for the orchestra. The Orchestra app for iPad, which costs £10, has had nearly 100,000 purchases since launch, and was at the centrepiece of an international marketing campaign by Apple.
“One of the most rewarding aspects of the project was seeing the positive response from classical music lovers, as they often have the most emotional reactions, experiencing a piece they love, like Sibelius’ Fifth, from a completely new perspective.”
— Luke Ritchie, Head of Digital Innovation and Partnerships, Philharmonia
“An extraordinary glimpse inside the sights and sounds of a symphony orchestra,” said The Guardian while the Evening Standard claimed that “Violinists’ bows are so close you flinch”.
While the Philharmonia’s VR projects are on a smaller scale than their major audience development projects, iOrchestra and The Virtual Orchestra Festival, the development of a portfolio of immersive experiences has has changed the way they work. “As our projects have grown and evolved, they have increasingly required a step-change in the way our whole organisation works as they demand a much more joined-up approach. We have continued to refine and develop our audience development methodology, while also trying to make our projects more cohesive, with a clear pathway for the audience from our immersive installations to the concert hall.”
One of the direct outcomes from their first VR project was the creation of a new partnership with Google, who commissioned a VR production of Philharmonia’s Beethoven’s Fifth, in partnership with NASA, which won the Raindance Film Festival Award for Best VR Music Experience in September 2017. The Orchestra have joined Google’s Jump VR Camera loan programme, which allowed them to embark on their third VR project, a filming a performance of Mahler’s Third Symphony, which is due for release in 2018.
And what of the future for VR in this sector?
“The truth is, nobody knows,” he says. “VR is not a gimmick but it will have a smaller market than AR or MR. It does however have huge artistic potential.”
He sees three major obstacles for arts organisations who want to embrace cutting-edge VR technologies: cost, quality and distribution. VR is expensive to make, the camera rigs are still in the early stages and, while the experiences are genuinely compelling, it’s still early days for VR headsets which are generally bulky and not yet of optimum quality. However, as the technology is progressing extremely rapidly, he believes this situation will change over the next two to three years and feels that London has the potential to have as much clout as Silicon Valley in this performance space.
Words by Bernadette Fallon