By Jan Skorupa, Wojciech Raszewski, Bartlomiej Idzikowski and Maciej Głowiak
Experimenting with technologies that support 3D audio in binaural domain, we immersed ourselves in ambisonic audio.
Nowadays 3D audio is gaining popularity as it is used in movies or computer games, among others. Combined with high resolution video or VR it provides a complex and rich experience. Any improvement in this area is almost immediately absorbed by commercial producers of content connected with entertainment.
Binaural 3D audio is based on HRTF (head-related transfer function) – which reflects how our ears receive sounds from any point in a given space. Therefore, if we implement that function into audio mixing, it ought to give us possibility to make an illusion of spatial sound in binaural domain.
The aim of our experiments was to find some free external plugins or devices which would allow us to achieve that effect. We chose two from each category to make comparison. The first software was a set of free plugins made in the Institute of Electronic Music and Acoustics at Kunst Uni Graz (as it was mentioned in our previous article). One of the plugins from the set allows for binaural decoding of ambisonic mix. We wanted to extend the possibilities of that software, so we started looking for some external device allowing for wireless, real time control of the plugin. We used simple application for android that sends data to each sensor built in a smartphone. This application transmitted information encoded as OSC (open sound control) data. Then we used the Max MSP software where we could easily analyse and scale the delivered data and then send it to Reaper. That connection allowed us to get really accurate information about rotation of the phone. Delivered and processed data was assigned to appropriate parameters of external plugins. As a result we can use a phone as a controller of 3D audio panning parameters.
Second product that we wanted to try was Klang hardware that is used as stage monitoring system that allows for mixing in binaural domain. This product has its own external software that is available on any tablet (ios and android); it supports Dante protocol, so one can send and receive any audio data, and with the use of tablet it is possible to spatialize and mix any audio material. Finally we made a short video that shows the results of our tests. The audio content we used was composed by a jazz band called Anomalia.
The results of our experiments related to immersive sound and 360° video was shown at Network Performing Arts Production Workshop in Prague on April 4th, 2019. The NPAPWS (http://www.npapws.org) is an annual conference and workshop on technologies utilizing advanced networks to enable arts instruction and performance, experience master classes and live performances leveraging these technologies.
