Sound Design pipeline for cross-platform 360 virtual productions
Project partners: Edinburgh Napier University, VRTONUNG
XR Network+ supported Edinburgh Napier University and immersive audio experts, VRToung, to develop a pipeline for sound design in 360 virtual productions.
The collaboration formalised a pipeline to edit 360 video content to a standard 16:9 widescreen version with downmixed binaural audio (converting audio taken from multiple channels into a single stereo output).
The work addresses a major industry issue relating to the cropping of 360 videos to resize content for different platforms, such as TV or mobile. Once a video is cropped, different sounds (which before were visually represented in the scene) are still audible but with no visual representation. This can result in a negative experience for the listener, causing disorientation or distraction.
Whilst software such as Insta360 Studio enables cropping of 360 videos to match the resolution of the preferred device, there is currently no automated solution or established pipeline to facilitate the cross-platform spatial sound design which reflects the cropping of the video.
The project explored a first step towards defining an industry pipeline for spatial audio production in 360 video across multiple platforms.
The research team also undertook a listening study to investigate how variations in audio elements within the refined videos impact upon user behaviour, engagement, and immersive experiences.
Whilst study participants favoured 360-degree videos over 2D versions, audio clarity, quality, and integration with visuals were paramount regardless of platform, with well-positioned sound contributing to a sense of realism and engagement.
The project is one of seven collaborations supported by the XR Network+ Prototyping, Impact and Acceleration (PIA) funding call. Researchers at UK universities were awarded up to £10,000 to develop new ideas and complete existing research related to VP between September 2023 and March 2024.