Performing Non-Human Characters in Live Mixed-Reality Performance

Project partners: The Royal Central School of Speech and Drama, University of London (CSSD), Ruth Mariner, Maggie Bain, Freyja Sewell

XR Network+ provided funding for The Royal Central School of Speech and Drama (CSSD) and industry partners to explore how Virtual Production (VP) technologies can be used to empower actors working in immersive, mixed-reality performance. 

The collaboration accelerated the development of a mixed-reality immersive performance, ShapeShifter. The piece features a live actor embodying a non-binary, shapeshifting nature spirit who ventures into the digital realm to explore the wonders of biodiversity and challenge our ideas of what it means to be ‘natural’.

The project team developed and tested novel strategies using motion capture data to map human movement onto a non-human avatar with multiple arms and expressive ‘horns’ on its head. The team enabled the avatar to dynamically morph between different forms while under the control of the mocap actor and explored the expressive potential of these transformations, looking at how the gestural expression of the actor blends with potential VFX or animation to communicate this intention.

Using Unreal Engine, the free Livelink Face app, and a low cost facial capture helmet (from Rokoko), the team reduced costs in the technology pipeline, demonstrating how other lower-budget productions could employ these techniques. 

Research and development undertaken as part of this project has helped to advance the creative vision of the production. It has provided all team members with hands-on experience in XR technologies and software workflows and has contributed to work that the creative partners have undertaken with other industry members. 

The work has also contributed to teaching at the CSSD, helping to introduce new skills and creative techniques to future creative professionals. 

Performing Non-Human Characters in Live Mixed-Reality Performance’ is one of seven projects supported by round one of XR Network+ Prototyping, Impact and Acceleration (PIA) grants. Researchers at UK universities were awarded up to £10,000 to develop new ideas and complete existing research related to VP between September 2023 and March 2024.

Image credit to videographer, Amy Dyduch.

Categories: Performance, Research, Technology