AI Composure: Leveraging Deep Neural Networks for Composing Physical Actors and Virtual Environments in Multi-Camera Green Screen Virtual Production
Project partners: University of York, dock10
Researchers at the University of York are exploring the transfer of knowledge between academia and industry as part of a collaboration with Manchester-based television facility, dock10.
The project brings together leading AI researchers from the university and Virtual Production (VP) experts from dock10 to tackle a major challenge for VP – simulating realistic light-interactions in multi-camera green screen studios.
Light interactions are a key element of VP. They can manifest in various ways, such as actors being illuminated by light sources in the virtual environment. In single-camera movie productions (where one primary camera is used to capture scenes), these effects can be achieved through a combination of large walls of LED panels (known as LED volumes) situated behind the actor and visual effects added in post-production.
These techniques are unsuited, however, for live or ‘as live’ TV productions that use multi-camera green screen studios, including sports reporting, music performances, or game shows. In this context, the production needs to capture the performance from multiple angles simultaneously, making LED volumes unsuitable.
The AI composure project is exploring an entirely new approach for VP practice in the form of an AI-powered compositing pipeline that seeks to remove constraints on current light interactions in green-screen productions.
The proposed system will simulate bi-directional light interactions between physical actors and the virtual environment in multi-camera virtual studios, without the need for expensive LED walls. The project sets the foundations for novel AI-powered compositing techniques that ultimately will be applicable in real-time, making them suitable for live or ‘as live’ productions.
The collaboration is one of five projects supported by the XR Network+ Embedded R&D funding call, with grants of up to £60,000 awarded to researchers at UK universities to explore the transfer of knowledge between academia and industry in areas aligned with Virtual Production.
Categories: Film, Games, Performance, Research, Technology, TV